Explainable Mixture Models through Differentiable Rule Learning
Authors: Matthias Wilms
Sascha Xu
Jilles Vreeken
Venue: ICLR 2026 Poster (2026)
Keywords: Mixture modeling
Interpretability
Conditional Density Estimation
Primary area: interpretability and explainable AI
TL;DR: We introduce explainable mixture models, a framework that pairs each mixture component with a human-interpretable rule over descriptive features.

Abstract: Mixture models excel at decomposing complex, multi-modal distributions into simpler probabilistic components, but provide no insight into the conditions under which these components arise. We introduce explainable mixture models (XMM), a framework that pairs each mixture component with a human-interpretable rule over descriptive features. This enables mixtures that are not only statistically expressive but also transparently grounded in the underlying data. We formalize the problem and examine conditions under which an XMM exactly captures a target distribution. We then propose a scalable, differentiable learning procedure for discovering sets of rules. Experiments on synthetic and real-world datasets demonstrate that our method discovers interesting sub-populations in both univariate and multivariate settings,  offering interpretable insights into the structure of complex distributions.

Published as a conference paper at ICLR 2026
EXPLAINABLE MIXTURE MODELS THROUGH DIFFERENTIABLE RULE LEARNING
Matthias Wilms", Sascha Xu", Jilles Vreeken CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
(mwilme s sascha. xu, evv wec.ssp.c... .pa.de
ABSTRACT
Mixture models excel decomposing complex, multi-modal distributions into simpler probabilistic components, but provide no insight into the conditions under which these components arise. We introduce explainable mixture models (XMM), a framework that pairs each mixture component with a human-interpretable rule over descriptive features. This enables mixtures that not only statistically pressive but also transparently grounded in the underlying data. We formalize the problem and examine conditions under which XMM exactly captures target distribution. We then propose scalable, differentiable learning procedure for dis- covering sets of rules. Experiments on synthetic and real-world datasets demon- strate that our method discovers interesting sub- sab-populations in both univariate and multivariate settings, offering interpretable insights into the structure of complex distributions.
INTRODUCTION
Mixture models represent complex, multi-modal data as combinations of simpler distributions [Mclachlan et al. 2019). On a dataset of insurance charges (Choil 2017) for example, Gaus- sian mixture model (GMM) identifies subpopulations with distinet modes as shown Fig. D In many applications, however, we also have access to descriptive features, e.g. age or BMI. Classical mixture models the marginal distribution of the target and therefore cannot leverage such features to explain when different sub-distributions arise.
To overcome this limitation, conditional density estimation (CDE) extends mixtures by modeling the conditional distribution of outcomes given features. In particular, mixture density networks (Bishop 1994) and kernel mixture networks (Ambrogioni et al.] 2017) parameterize mixture weights and components functions of descriptive features. However, these dependencies typically mod- eled with neural networks that do not yield human-inteppretable explanations. More broadly, CDE methods tend to prioritize predictive accuracy over interpretability (Sugiyama al. 2010). While tree-based approaches (Cousins & Riondato 2019 Yang & van Leeuwen, 2024) offer insight, we observe in experiments they can overfit and lack support for overlapping regions.
'Equal contribution.
Age
BMI
Children
Smoker
Donsity
5.0
2.5
20000
40000
60000
25
50
25
$0.0
Insurance Charges
Figure 1: The XMM recovers similar modes to the GMM (left), and also explains when each mode observed using simple rules over deseriptive features.

---

Published as a conference paper at ICLR 2026
To this end, propose Explainable Mixture Models (XMM), framework that directly pairs each mixture component with human-interrs iimaa-interpretable rule over descriptive features. The XMM framework defines each mixture component ular parametric family, e.g., Gaussian, and naturally allows for overlapping components. On the insurance dataset, a fitted XMM in Fig. [ recovers near identical modes to the GMM (see Appx. A but add the same time also provides simple, interval-based rules that explain when each mode is observed. The subpopulation with the lowest insurance charges, for example, corresponds young, non-smoking individuals without children, whereas the highest-charge component comprises older, smoking individuals with high BMI. Our main contributions are as follows:
data-induced distribution rather than restricting it to partic-
1. Concept. We propose XMM, which both characterize the subpopulations of the global distribu- tion, whilst accurately estimating the local conditional density given any feature vector.
2. Theory. We derive exact-recovery conditions for marginal and conditional densities and intro- duce regularizers to steer learning towards these regimes.
Practice. We propose a scalable, differentiable training procedure and show that the XMM accu- rately models the underlying distribution whilst discovering interesting subpopulations.
2 RELATED WORK
Mixture models are a classical tool for density estimation and clustering. There exist many variants based on parametric families such as Gaussians (Reynolds) 2015) or t-distributions (Peel & McLach- lan 2000) as well as nonparametric approaches (Antoniak 1974). In general, unconditional mixture
models however are limited to modeling latent component variables (Viroli & McLachlan 2019).
Mixture of Experts (MoE) (Jacobs et al. 1991) are a general class of models in which gating network determines the weighting of local experts. While MoEs typically rely on black-box neural networks for gating, recent surveys identify interpretability as a critical open challenge (Mu & Ling
2025). Interpretable variants have been proposed (Ismail et al. 2023 Pradier et al.) 2021). however, these approaches focus primarily on classification or deferral to human experts. EMMs share the
high-level conditional mixture structure of MoEs but differ fundamentally by targeting conditional density estimation through differentiable rule learning. Similarly, Conditional VAEs (CVAE) (Sohn lt al. 2015) can model complex conditional distributions (z), However, they rely on a latent prior and deep neural networks, resulting in black-box model. In contrast, EMMs explicitly model the conditional density through rule-based components, providing direct insight into structure of the data without latent variables.
Conditional density estimation (CDE) aims to estimate the full conditional distribution of a target variable y given input features Approaches range from kernel and RKHS-based estimators (Hyn- dman et al.) 1996 Sugiyama et al. 2010). random forest (Watson et al.) 2023), to normalizing flows (Winkler al. 2019). Another relevant line of work extends mixture models to the conditional setting. In Mixture Density Networks (Bishop. 1994) a neural network outputs mixture parameters as functions of xc, while Kernel Mixture Networks (Ambrogioni al. (017) replace the parametric mixture components with nonparametric kernels. However, both methods use black-box neural net- works for gating and thus do not provide insight into when each component is active, a limitation shared by all the aforementioned CDE methods.
To address this issue, interpretable, tree-based approaches have been proposed. Density Estima- tion Trees (Ram & Gray 2011) use interpretable tree structures but only target the unconditional density. CADET uses trees model conditional densities with exponential family distributions in the leaves (Cousins& & Riondato 2019). but tends to learn very deep trees that are hard to interpret. Most similar to our approach is CDTREF (Yang van Leeuwen! (222), which learns a minimum description length regularized decision tree with histogram in each leaf. Both approaches however are primarily aimed at fitting densities, and not for discovering its components. Consequently, these approaches can yield trees with many fine-grad fine-grained leaves, limiting interpretability in practice.
Subgroup discovery a line of work where interpretability is key but with an objective orthogonal to mixture modelling (Atzmueller 2015). The goal is to identify subpopulation, e.g. single mixture component, that is statistically interesting with respect to a target variable and describe through a iumernnnterppeetable rule. Using combinatorial (Lavraé et al. 2004 Atzmueller & Puppe.
2

---

Published as a conference paper at ICLR 2026
2006) continuous optimization (Xu et al. 2024). rule is learned that maximizes the measured deviation the subgroup from the global population (Todorovski et al. 2000).
The main difference to XMM is that subgroup discovery is inherently local, focusing isolating an interesting subset the data rather than modeling the entire population. While there exist ap- proaches that learn multiple subgroups (Van Leeuwen & Knobbe 2012 |rooeçç et al. 2022), they typically do not attempt to model the full conditional distribution. XMM extend the notion a subgroup mixture models: each component represents subgroup of the data characterized by an interpretable rule, while the model collectively captures the entire conditional distribution.
Summary. Explainable Mixture Models bring together ideas from all three areas: In contrast
neural gated mixture models, XMM provide interpretable rules for each component; Compared to subgroup discovery, we model the entire domain; And compared to tree-based CDE, we allow for mixture of components rather than single tree. In the following, we will formally define Explain- able Mixture Models and show how learn them from data.
EXPLAINABLE MIXTURE MODELS
)) (0)) of d real-valued features and a target value y E V.. We assume each sample (x, y) to realization of pair of random variables (X, Y) ~ Px.., drawn i.i.d. We write p to denote probability density functions and P to denote probability distributions.
We consider a dataset of pairs {xx(0,
consisting of descriptive feature vector E Rd
Our goal is to explain the distribution of the target variable Y mixture of simpler components. bumma-interpretabll explanation over the de-
The idea is to use components that are grounded scriptive features X instead of being latent factors. An explainable mixture model (XMM) therefore not only provides a decomposition of the target into simpler sub-distributions, but also explains the conditions under which these sub-distributions observed.
Definition (Marginal-XMMM An explainable mixture model M = [ei) of the marginal den- sity ply) is defined as set of k feature-bart based explanations e; I2d - {0,1} with non-zen support, iee. (X), > For each respective explamation eg define the mixture weight w; as
B(e((X)]
(1)
ΣØ=1 Ele, (X)
where it holds that 2 0 and Σ Wi = 1. The induced density PM((y) is finite mixture of l: components as per
(M Σ wi (1))
p(()) = PY (v)
The marginal XMM as a weighted sum of simpler component densities p((y). based on the standard finite mixture model (McLachlan et al., 2019). The differentiating factor of an XMM compared to conventional mixture models lies the explainability of the individual components Pr(y). Instead of restricting these to parametric family, e.g. Gaussians, XMM is based on non-parametric, data- induced densities ps (y). Each component reflects the conditional distribution of the target Y given that the explanation 6 over the descriptive features X holds. The choice of human-interpretable explanation E (c.g., logical rules) is application dependent and agnostic to the definition.
Proposition 1 Let M {ei}t & be an XMM with marginal density per Def. If the set of explanations e; form partition of the feature space R.d, i.e. Σi_ ex(x) = 1 for all x in the support of Px, then the induced density (M) equals the true marginal density py (v))
Proposition Dis a direct consequence of the law of total probability (See for the proof Appx. B.I The result shows that we cannot rely on maximization of the marginal likelihood to learn an XMM. Setting all components to the same constant function, e.g. ex) = 1 for all i, leads to a perfect fit of the marginal distribution, provided that the component densities Pı()) are sufficiently flexible. Therefore, we cannot expect to learn a meaningful XMM by maximizing the marginal likelihood. To address this issue, next introduce conditional interpretation of the XMM.
3

---

Published as a conference paper at ICLR 2026
3.1 CONDITIONAL EXPLAINABLE MIXTURE MODELS
The issue of treating an XMM as a purely marginal model is the degeneracy of maximum likelihood solutions. To address this, leverage the ability of an XMM explain under which conditions dis- tinct sub-distributions occur and formally introduce the conditional XMM to model the conditional density py (y) x).
Definition (Conditional-XMa) An explainable mixture model M = {eiKi=z of the conditional density ply x) is defined as a set of explanations 6, R2d - [0,1] with non-zero support,
i.e. [X); > 0, and complete coverage, i.e. Σi ex(x) > 0 for all x in the support of Px. For
each feature vector x, we define the conditional mixture weights wx(x) as
(x((x) = (x)
(3)
Σラ=1 ex(x)
The induced conditional density (M x) is a finite mixture of l components, where
PM x)
Σ wi (x(x))
py(y) PY ((x()))) (v)
(4)
Similar to marginal models, the conditional XMM is finite mixture of simpler component densities Py(y). In addition, the mixture weights wx(xx) are now dependent on the descriptive features, similar to mixtures-of-expeee
(MoE) model (Jacobs et al. 1991). The main difference to MoEs is that an XMM consists of explanation- any gating mechanism is permissible and with arbitrary parametric experts that need not represent an underlying demographic group. Next, we examine what conditions are needed so that a mixture M faithfully represents the true conditional distribution.
components, which are derived from the data, while in MoF,
Let M = {eilt be an XMM with conditional density as per Def. 2 explanations 6, form a partition of the feature space Rd into homogeneous regions with respect to the target variable Y. i.e. for every explanation e, and its induced sub-distribution pv(y). holds that (y x) Pı(y) for all x with e exxx) = 1 and px px(x) >> 0. then the induced density (M x) equals the true conditional density py (y x).
Proposition 2
If the set of
We provide a proof of Proposition in see Appx. B.2 This result implies a structural constraint on any valid set of explanations. To guarantee set of explanations induces the true conditional density sufficiently well, the explanations must partition the feature space, such that within the scope of each explanation e,, the target variable Y is i.i.d.
While this is a stronger requirement than for the marginal XMM, where a partitioning alone is suf- ficient, it also serves eliminate degenerate solutions. By maximizing the conditional likelihood, the XMM is encouraged to find set of explanations that capture where distinct, yet locally homoge- neous sub-distributions occur. Therefore, we propose to fit an XMM M by minimizing the negative log-likelihood (NLL) given a dataset ((x(1) (1)) (
NLL(M) = Σ log Σ (x(())))) p2(o())) (xx)))))))))))
In practice, we estimate each P: from the subset {1 re (((xx(1) 1} with appropriate smoothing
(e.g., KDE bandwidth selection or Dirichlet priors for discrete Y) and add a small = > 0 inside the logarithm for numerical stability. This now provides principled objective to learn an informative XMM using likelihood maximization. Once obtained, an XMM M gives insight into the global distribution Py through its explainable components, and can also be used make local, conditional density inferences P. (y) x) for given descriptive feature vector
3.2 OPTIMIZATION OBJECTIVE
Lastly, we discuss how to optimize the NLL objective in Eq. 1 to learn an XMM. In Propositions and 2 we have seen that appropriate partitioning can achieve perfect fit of the true density. On
|
4

---

Published as a conference paper at ICLR 2026
Zuckk
13
MCRT 10.5.0.31
000
145
0.
64
-0.
40
-05
10 013
-15.
11.0
(a) Condition
(b) Conjunction
(c) Explainable Mixture Model
Figure 2: The building blocks of XMM: Learnable thresholding conditions are placed each feature z. E IR (a). They are combined into a conjunctive, differentiable rule (b). Each rule acts as gating function for an expert density, with mixture in the overlap (c).
the other hand, we also want to allow a certain degree of overlap between explanations to improve interpretability, e.g. by providing broader, more general explanations.
To balance these two objectives, we propose to learn an XMM by minimizing a regularized NLL objective. We introduce an overlap penalty R(M) that penalizes explanations e, that frequently hold together. It is defined as
1 R(M)
n Σ Σ (x((x(1)2)
(6)
For particular sample (4), the term in parentheses is minimized when exactly one explanation ej holds, i.c. (xx()) = 1 for some i and ux(x(") = 0 for all j # i. To penalize overlap, we square constant by definition. Squaring ensures the penalty gets smaller as the distribution of weights becomes more sparse. and minimized when converging to single active component. The overall optimization objective with hyperparameter X that controls the strength of the overlap penalty is given by
the weights wy because the sum Σ (x(x())
min NLL(M) XR(M). M
LEARNING EXPLAINABLE MIXTURE MODELS
In this section, describe concrete instantiation of Explainable Mixture Models for tabular data, which uses conjunctive rules as class of explanations, e.g. "18 < Age < 65 AND BMI > 25". Such rules, familiar from decision trees and subgroup discovery, offer compact and human-interpretable description of the regions in which mixture components apply. Unlike combinatorial rule learning approaches, our method requires directly via gradient descent (see Eq. (9) for both continuous and discrete features. In particular, we consider rules e Rd {0. 1} that map input features x Rd to binary activations as per
pre-discretization; instead, the thresholds aj, Bj are learned
e(x; = A (a);aa,8,)
Learning a set of such rules challenging combinatorial problem, as the number of possible predicates, thresholds, and their conjunctions grows exponentially with the number of features d. To handle this complexity, we introduce smooth relaxation that enables multiple rules be learned jointly via gradient-based optimization.
4.1 A DIFFERENTIABLE RULE-BASED MIXTURE
We now show how to learn a rule- based mixture using gradient- based optimization. To avoid com- binatorial search we extend the differentiable formulation single rule by Xu et al. (2024) to learn a mixture of multiple rules jointly using traseent-based optimization.
exponential search space (Lavrač et al.] 2004 Atzmueller & Puppel 2006).
5

---

Published as a conference paper at ICLR 2026
We briefly summarize the key components of the differentiable rule learner's architecture. Firstly, the conditions a(zz;az, B,) = 1[a, < 2j < 3y] placed on individual features 2, E R. E
{1, (I.....,]) d], are approximated as
3jj - 03 * ( Bj) = 0
(
7 01) B3 23 T
where o is the sigmoid function and T > 0 is temperature parameter that controls its steepness. During training, we anneal the temperature gradually to zero, transitioning from soft constraints tt IR - [0, 1] to hard constraints, i.e. lim # (1z;
(j (8)) T(z;; 33) for all x * 0j, 88-
We show an example in Fig. 2a where the condition becomes steeper as T + 0.
We use the weighted harmonic mean to combine multiple conditions into rule that approximates the logical AND operator. It is defined as
Σj=1 0j
e(x: A)
with aj 20,
(10)
Σj= aj + (2); (3.
where we denote the parameters of rule as 0 havior of a logical conjunction whilst being fully differentiable: If any condition (zyz is close to zero, then the reciprocal fty (x,z)) grows, and thus the overall rule activation é(x) becomes small. Conversely, the rule activation ē(x) = only if all conditions ły (xx) = are high. The learnable, non-negative weights aj represent the importance of feature within the rule. By setting aj O, the corresponding condition ât has no effect on the rule activation é(x), allowing the optimizer to effectively prune unnecessary conditions.
This function mimics the be-
Last, we combine multiple differentiable rules with their local densities. To obtain an XMM, follow- ing Definition use conditional gating function follows
(x 0c) +e with e = (0)....,
(x: Q)
(11)
Σラ (x)
for given input x, where we add an e floor to avoid numerical instability. This formulation ensures that the mixture weights w; (x; θ) are non-negative and sum to one.
Density Estimation. To estimate the target density (v()) for each component i, we need density estimator p. v²)). We outline a parametric and non-parametric solution that then evaluated in the experiments. As a parametric alternative, we use unconditional Gaussian mixture model (GMM). As we learn sub-distributions of the marginal distribution, we parameterize each component density (v()) with the same set of means and covariances learned on the marginal distribution, but allow for different mixture weights v; for each component i. This has the advantage of being much more computationally efficient, and aligns with our goal of describing distinct modes in the data.
As the non non-paraettric variant, we use a Neural Spline Flow (NSF) (Durkan et al.) 2019). A normal- izing flow transforms simple base distribution into a complex target distribution through series of invertible mappings. NSFs are parameterized by a cubic spline neural network, whose param- eters t/s are learned by maximizing the likelihood. NSFs powerful density estimators, but computationally expensive and may overfit on small subgroups.
4.2 OVERSPECIFICATTION AND PRUNING
general challenge in rule learning navigating the combinatorial search space of all possible rules. Previous approaches are limited to recursive partitioning or greedy schemes, but our differentiable approach allows for parallelized optimization of large quantities of rules. That is, we overspecify the initial number of rules A ensure sufficient coverage of the feature space.
To ensure that the initial rules effectively cover the feature space, the initialization of each rule key. Random initialization of the rule parameters 0, often leads to poor coverage (Fig 3a) while choosing random samples from the training set as anchors improves coverage but can still leave gaps (Fig, 33b). We opt for guided initialization, where select anchoring points k- k-means++ centroids AAthhr & Vassilvitskii (007)) (Fig. (cp). This way, we ensure that each initial rule è; is anchored on distinct region of the feature space, improving the likelihood of discovering meaningful explanations.

---

Published as a conference paper at ICLR 2026
(a) Randomly initialized.
(b) Randomly anchored.
(c) k-means++ anchoring.
Figure 3: Initialization: k-means++ anchoring ensures a thorough coverage of the feature space.
Pruning and Model Selection Our initialization ensures broad coverage of the feature space, but overspecification inevitably introduces redundant explanations. The primary pruning mechanism is the optimization itself: a rule é₂ can be disabled by learning an inverted interval (aij > Biy) for any feature j with non-zero weight dzj > 0, forcing (x) = 0 everywhere and removing its gradient signal. This allows the optimizer to discard uncompetitive rules. For efficiency and stability, we periodically check for such inactive rules during training and disable them completely. If several neighboring rules converge to nearly identical densities (.))), they may all survive pruning; we address this with post-hoe merging procedure (see Appx.
While initializing with more components can reveal more specialized explanations, the gain in like- lihood often comes the cost of interpretability. To avoid dataset-specific tuning of the initial number of rules k we use the Bayesian Information Criterion (BIC) to balance expressiveness and complexity. After training, we compute
BIC(M) 2 NLL(M) + |0] log(n);
(12)
where | is the number of active parameters in the rule network. This criterion ignores parameters of the local density estimators P. (v; 2h,), as our framework models them to be data-induced, instead focusing model selection on the complexity of the explanations. We train multiple models from range of lo and select the one with the best BIC score (see Appx.
EXPERIMENTS
We empirically validate XMM, using NSF and GMM respectively density estimators. As base- lines we include the interpretable CDE methods CDTREE (Yang & van Leeuwen] (2024) and CADET (Cousins& & Riondato 2019). which partition the feature space via decision trees, and non- interpretable methods MDN (Bishop) 1994). KMN (Ambrogioni et al. (007)), NF (Rezende & Mo-
hamed 2015), CVAE (Sohn et al.) 2015) and LSCDE (Sugiyama et al. 2010 We provide the code
for XMM, experiments and data generators in the Supplementary Material
5.1 SYNTHETIC DATA
We first test synthetic data with known ground truth. We generate d independent uniformly distributed features X,, partition the space into & disjoint hyperrectangles, and assign each region randomized density (Gaussian, Uniform, etc), resulting in piecewise-constant p(y x) (see Appx.
https://edes grwup/prorrrr xmm/
Test NMAt
CADEY
0.0
13
15
50
05
10
00
02
04
True Composents
Noise Features
Noise Scale
Overlap
(a) True Components
(b) Noise Features
(c) Target noise
(d) Overlap
Figure 4: NMI between true and learned components variety of settings.
7

---

Published as a conference paper at ICLR 2026
Unless varied the experiment's parameter per component, overlap 3 = 0.1 and no noise on Y, averaging results over 4 datasets.
D.1.
d 5, k 5 components, 600 samples
Accuracy. We first measure the accuracy of XMM in recovering the ground-truth. We report the nor- malized mutual information (NMI, which compares the cluster similarity between true component labels and those by learned rules (Appx. D.4). Fig. 4a shows that both XMM instantiations reliably recover ground-trutt truth components, with only slight performance drop for many components. CADET struggles due unregularized large trees, while CDTREE regularization aids it in recovering good solution. KMN, a black-box method from which we extract labels as that of the component with highest weighted likelihood, performs well on large numbers of components, but poorly on few.
Robustness. Second, we evaluate robustness to noise in the features and target, shown respectively Figures4bland Ac XMM is largely unaffected by feature noise and only slightly degrades under high target noise. CDTREE RRE performs similarly but is less accurate at high target noise, while CADET and KMN are consistently weaker in both settings.
addition, we measure the effect of increasing overlap between the component densities in Fig. 4c] XMM remains stable under moderate overlap but degrades when overlap is large, KMN shows similar trend, whereas CDTREE declines more gracefully and surpasses XMM at high overlap. CDTREE's advantage is its tendency to create many small leaves, which approximate overlapping densities well but are not penalized by the NMI metric.
Model Complexity. Next, assess model complexity by compar- ing the number of learned components to the ground truth. In Figure 5 we plot the number of excess components, i.e., the difference be- tween learned and true components. Fig. 2eoo5sshwws that after pruning, both XMM variants recover component counts close to the ground truth, with GMM slightly underfitting and NSF slightly overfitting. In contrast, the gap between CDTREE and the true number com- ponents widens complexity increases, reflecting the limitations of greedy top-down splitting, while CADET'S number of excessive rules consistently exceeds the limits of the plot. On the other hand, XMM precisely identifies the correct number of components no matter if we have 5, 10, 20 true components.
20
Xuni Cluss
Excess Rules
Xyit-No Nis
10
CDTRE
10
15
20
True Components
Figure 5: Excess rules vs. true components.
Sensitivity to A. We further investigate the effect of the overlap penalty weight À (Eq. 7 on model complexity using the real datasets (Sec- tion 5.2). Figure [a| shows the change in test NLL (NLLA- - NL and Figure 6b shows the ratio of active rules relative to the unregu- larized baseline (A- 0) As shown in Fig. in- creasing À effectively regularizes XMM-GM using up to 16% fewer rules at À = 0.3 than the
0.4
Xus-Giua XousNNu
Deless /N stales,
1.1
3.3
0.2
NLL
0.8
0.25
0.30
0.75
1.0
0.38
5.25
0.50
0.25
110
(a) Test NLL delta
(b) Rule ratio
baseline. The likelihood cost is negligible, in- dicating the components were redundant. This confirms that the penalty successfully steers the optimization towards a concise partitioning for XMM-GMM. For XMM-NSF the benefit is less clear. The number of rules only decreases significantly À and incurs a larger likelihood cost. Con- sequently, we recommend the use of the overlap penalty primarily for the XMM-GMM variant.
Figure 6: Sensitivity A
Rule Scaling. Finally, we analyze the robustness to overspecification by increasing the initialized rules l on synthetic datasets with 5 and 10 true components, and show the results in Figure 7 On these datasets we see in Figure 7a that once k is sufficiently large to capture the true structure, NLL plateaus. In Figure 7b and Figure 7c we see that XMM-GMM is very stable in this setting even when k is much larger than the true components, as no excess rules are discovered and NMI remains high. XMM-NSF achieves better NLL because it is more flexible, but this flexibility makes it more prone to retain excess rules when k is large. This indicates that the inductive bias of restricted model class (XMM-GMM) allows for more effective pruning of excess rules through likelihood objective.
8

---

Published as a conference paper at ICLR 2026
Interpretable
Dataset
XMM-NNS
XMM--NaP Bic
XMM--GM XMM-GGM Bic
CDTREE
CADET
CVAE
KMN
LSCDE
MON
NF
SkillCran
-3.58
-3.56
-1.11
-4.19
-1.03
2.23
1.61
-0.91
1.57
2.73
1.47
Themoggaphy
1.21
1.26
1.00
0.63
0.56
1.50
1.63
0.90
0.57
1.33
abalone
1.06
1.80
air qualitt
0.27
0.27
-0.19
-0.19
0.53
0.15
0.25
0.91
0.18
0.15
bike
8.81
8.90
8.06
8.94
8.66
0.30
8.62
9:48
8.67
8.59
9.71
boston
2.98
2.50
2.60
2,58
2.81
1.51
3.20
9.17
3.07
2.67
7.32
3.64
3.61
3.73
3.58
3.33
2.96
3.45
energy
insurunce
8.53
8.95
9.06
9.06
9.11
20.06
8.03
8.72
9.53
8.03
7.44
life
2.40
2:35
2.25
2.42
2.08
0.21
2.27
2.18
2.65
1.91
3.71
obesity
33.00
-3.33
-.53
-3.45
-0.18
-1.79
1.12
2.76
-0.30
synchoonous
-2.16
-2.41
lodicity
1.57
1.62
1:41
1.44
1.51
1.71
1.34
1,90
1.37
1.44
1.55
wages
10.88
11.13
10.89
10.80
11.20
11.90
11.33
11.68
11.45
11.50
11.53
wine
4.15
-2.61
-4.11
4.89
4..01
3.15
1.37
1,20
3.20
-0.38
5.20
6.33
Table I: NLL of interpretable and black-box models on real-world datasets. Bold values indicate the best NLL among interpretable models, underlined values indicate the best overall NLL.
5.2 REAL-WORLLD DATASETS
We next evaluate XMM on real-world datasets from the UCI Machine Learning Repository (Dua & Graffi 2017). Since ground-truth components unavailable, performance is measured by negative log-likelihood (NLL) on a held-out test set. We report results using the full A = 100 starting components, as well as with BIC regularization for automatic model selection (Section 4.2
We report the NLL in Table I interpretable baselines, while the BIC-regularized variant achieves the second best rank but with substantially fewer and simpler rules (see Table 2). Among tree-based methods, CDTREE outper- forms CADET and falls between our GMM and NSF instantiations. Non-interpretable methods vary in performance, with MDN and CVAE the strongest, but still trailing XMM-GMM. Overall, XMM achieves state-of-the-ar accuracy with full interpretability. The XMM-GMM consistently outper- forms XMM-NSF, suggesting that the simpler parametric estimator is better suited for this setting. BIC regularization typically incurs small loss in accuracy but yields models with fewer, shorter rules, offering a practical trade-off between accuracy and interpretability.
XMM-GMM ranks highest across both interpretable and non-
5.3
CASE STUDY
We conclude with a case study on data-driven materials science. In particular we consider gold nan- oclusters, whose electronic and catalytic properties relevant photovoltaics and medicine (Gold- smith et al. 2017). We fit an XMM to understand which molecular configurations lead to desirable properties. We target the HOMO-LUMO energy gap, key indicator of photovoltaics performance, and visualize the learned densities and explanations in Fig. [ We recover the known relationship that clusters with an odd number of atoms exhibit smaller gaps than those with an even number of atoms, but more importantly uncovering finer distinctions based on planarity, cluster size, and bonding structure. Compared CDTREE, which requires 64 components for weaker fit, XMM achieves a lower NIL (-1.706 vs. - 1.683), with far fewer explanations (19.7 vs. 58.7) and orders- of of-magnitude lower runtime (29s vs. 1782s).
5-8
Xuge-Chou Xau--ar
Xuto-Guu Xats-Ner
The NM
Bucon Bules
6.6
10
28
40
100
180
23
40
100
Number
Nomber rales
Nomber
miles
(a) Test NLL
(b) Test NMI
(c) Excess rules
Figure 7: Robustness to rule overspecification (large k). While XMM-NSF achieves lower NLL (a), it retains redundant rules (c). XMM-GMM successfully prunes excess components, maintaining high NMI (b) and recovering the exact number of ground-truth rules k increases.

---

Published as a conference paper at ICLR 2026
Planarity Now
Density
05
0.0
HOMO-LUMO
Figure 8: Densities and explanations for 18 mixture components learned by XMM. Continuous intervals represented bars relative the feature domain, discrete values as boxes. Blue bars indicate active rule constraints (aj > 0), gray ones indicate inactive features(a, $ 0).
Multi-Target Learning. distinctive feature XMM its capacity explain multivariate tar- gets. XMM identifies visible clusters in the joint distribution of relative gyration Rgo and van der Waals energy A in Fig. 9 revealing clear separation in gyration Rgo between planar (2D, Planarity 0) and non-planar (3D, Planarity 1) clusters. This matches the physical intuition that planar clusters are less compact and therefore have larger radius of gyration. Our results further corroborate previous studies showing that non-planar clusters have higher intermolecular Waals interactions than planar ones (Goldsmith et al.) (017). For example, explanations and 15 correspond to clusters of the same size but different planarity, yielding distinct A Eeaww values. Our results on real-world datasets, including a study on Abalone (see Appx. [D8), highlight the ability of XMM explain meaningful interactions behind interesting subpopulations.
der
CONCLUSION
We introduced Explainable Mixture Models, a framework that pairs each mixture component with a human-interpretable rule. We established conditions for the exact recovery of the underlying data distribution, and proposed scalable, differentiable learning algorithm with automatic model lection. Experiments show that XMM reliably recovers ground-truth components, while achieving state-of-the-art performance in CDE on real-world datasets. Case studies on materials science fur- ther illustrate the utility of XMM in Al for Science. Overall, XMM accurately models complex distributions whilst providing meaningful, interpretable explanations.
Limitations. A primary limitation of our approach is the need for fixed number of mixture compo- nents & start of training. We mitigate this through our initialization strategy and the BlC-based model selection, but in practice & must be tuned for optimal results. Furthermore, we consider limited class of explanations in the form of conjunctive rules over intervals. Future work will ex- plore more expressive rule classes, such disjunctive normal form rules, and extend explanations to different modalities such as images or text. Lastly, XMM is dependent on the performance of the underlying density estimator, which may need to be adapted to the specific data domain.
N
AT
T
12
34
46
62
Planarity
Neve
0.3
15
23
24
R0
80
Rye
Figure 9: XMM over joint distribution of radius of gyration Rgo and van der Waals energy A
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work aims to increase the transparency and interpretability of complex data distributions. The rules generated by our model are based on statistical correlations in the data and cannot be used to make definitive statements about causality or generalizability. The results must thus be used with caution, especially when sensitive data is involved.
REPRODUCIBILITY STATEMENT
To facilitate reproducibility we provide all code necessary to replicate the experiments. addition the method itself, this includes code to generate the synthetic data for our experiments, as well as code to reproduce the evaluation results on synthetic data, real data, and case studies.
REFERENCES
Luca Ambrogioni, Umut Güçlü, Marcel A. J. van Gerven, and Eric Maris. The kernel mixture network: A nonparametric method for conditional density estimation of continuous random vari- ables. arXiv: 10211705501111 2017.
Charles E. Antoniak. Mixtures of dirichlet processes with applications to bayesian nonparametric problems. The Annals of Statistics, (66):1152-111 1974.
David Arthur and Sergei Vassilvitskii, k-means++ the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAN symposium on Discrete algorithms, pp. 1027-1035, 2007.
Martin Atzmueller. Subgroup discovery. Wiley Interdisciplinary Reviews: Data Mining and Knowl- edge Discovery, 5(1):35 2015.
Martin Atzmueller and Frank Puppe. Sd-map-a fast algorithm for exhaustive subgroup discov- ery. In European Conference Principles of Data Mining and Knowledge Discovery, pp. 6-17. Springer, 2006.
Christopher M. Bishop. Mixture density networks. Technical Report NCRG/94/004, Neural Com- puting Research Group, Aston University, 1994.
Mi Choi. Medical cost personal dataset ("insurance"). Kaggle, 2017. Dataset with age, sex, bmi, children, smoker, region on, charges.
Cyrus Cousins and Matteo Riondato. Cadet: Interpretable parametric conditional density estimation with decision trees and forests. Machine Learning. 1080-110):111-16334 2019. doi: 10. 1007/ 510994-019-058144
Dheeru Dua and Casey Graff. Uci machine learning repository, 2017. URL httt::t/arcciiivee ics. uci eeeimedumml
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. Ad- vances neural information processing systems, 32, 2019.
Bryan R Goldsmith, Mario Boley, Jilles Vreeken, Matthias Scheffler, and Luca M Ghiringhelli. Uncovering structure- property relationships of materials by subgroup discovery. New Journal of Physics, 19(1)001303, January 2017. ISSN 1367-2630. doi: 111088//137-2233//aa57
structure-property
Rob Hyndman, David M. Bashtannyk. and Gary K. Grunwald. Estimating and visualizing condi-
tional densities. Journal of Computational and Graphical Statisties, 5(4):315-336, 1996.
Aya Abdelsalam Ismail, Serean Ö. Arik, Jinsung Yoon, Ankur Taly, Soheil Feizi, and Tomas Pfister. Interpretable mixture of experts, 2023. /tt.tttss:///axiiii http::::///xwwvvovr/e.oo abs /20/622066222 02107
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991.

---

Published as a conference paper at ICLR 2026
Nada Lavrač, Branko Kavšek, Peter Flach, and Ljupčo Todorovski. Subgroup discovery with en2-sd. Journal of Machine Learning Research, 5(Feb): 188, 2004.
Geoffrey J McLachlan, Sharon X Lee, and Suren Rathnayake. Finite mixture models. Annual review of statistics and its application, 6(1):355-378, 2019.
Siyuan Mu and Sen Lin. comprehensive survey of mixture-of-expent Algorithms, theory, and applications, 2025.
wttp:///.a./a...aa///////// 07137
David Peel and Geoffrey McLachlan. Robust mixture modelling using the distribution. Statistics and computing, 10(4):339-344) 2000.
Melanie F. Pradier, Javier Zazo, Sonali Parbhoo, Roy H. Perlis, Maurizio Zazzi, and Finale Doshi- Velez. Preferential mixture-of-experts: Interpretable models that rely on human expertise as much as possible, 2021. URL lhttps://aaxiite org/ abs/ 05360
Hugo M Proença, Peter Grünwald, Thomas Bäck, and Matthijs van Leeuwen. Robust subgroup discovery: Discovering subgroup lists using mdl. Data Mining and Knowledge Discovery, 36(5): 1875-1970, 2022.
Parikshit Ram and Alexander G Gray.
Density estimation trees. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 627-635, 2011.
Douglas Reynolds. Gaussian mixture models. In Encyclopedia biometrics, pp. 827-832. Springer, 2015.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows In Interna- tional conference on machine learning, pp. 1530-1538. PMLR, 2015.
Jonas Rothfuss, Fabio Ferreira, Simon Walther, and Maxim Ulrich, Conditional Density Estimation with Neural Networks: Best Practices and Benchmarks. URL
19C3.00952
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation us- ing deep conditional generative models. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28, Cur- Associates, Inc., 2015. URL http:httts: nwwriss.eerppp... wwtpssseccppppppppppeeeeep files/ prppspasp77522222222rappppppppp PPa.er.----pppppd...
Masashi Sugiyama, Ichiro Takeuchi, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Daisuke Okanohara. Least-squares conditional density estimation. IEICE Transactions on In- formation and Systems, 93(3):583-594, 2010.
Ljupčo Todorovski, Peter Flach, and Nada Lavrač. Predictive performance of weighted relative accuracy. In European conference on principles of data mining and knowledge discovery, pp- 255-264. Springer, 2000.
Matthijs Van Leeuwen and Arno Knobbe. Diverse subgroup set discovery. Data Mining and Knowl- edge Discovery, 25(2):208 22220222222222 2012.
Cinzia Viroli and Geoffrey J McLachlan. Deep gaussian mixture models. Statistics and Computing, 29(1):43-51 2019.
David S Watson, Kristin Blesch, Jan Kapar, and Marvin N Wright. Adversarial random forests for density estimation and generative modeling. In International Conference on Artificial Intelligence and Statistics, pp. 3357-5375. PMLR, 2023.
Christina Winkler, Daniel E. Worrall, Emiel Hoogeboom, and Max Welling. Learning likelihoods with conditional normalizing flows. CoRR, abs/ 0bs/1912.00042. 2019, URL http::/arxivv
org/abs/19/ 30042
Sascha Xu, Nils Philipp Walter, Janis Kalofolias, and Jilles Vreeken. Learning exceptional sub- groups by end-to-end maximizing kl-divergence. In Proceedings of the 41st International Con- ference Machine Learning pp. 55267-55288 2024.
Lincen Yang and Matthijs van Leeuwen. Conditional density estimation with histogram trees. Ad- vances in Neural Information Processing Systems, 17:117315-117338 2024.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A GMM VS. XMM ON INSURANCE DATASET
On a widely used dataset of insurance charges CCoi 2017) for example, Gaussian mixture model (GMM) identifies subpopulations with distinct modes as shown in Fig. Toal In many applications, however, we also have access to descriptive features, e.g. age BMI. A fitted XMM in Fig. [Ob] re- covers similar modes the GMM, and additionally provides simple, interval- based rules that explain when each mode is observed. For example, the subpopulation with the lowest insurance charges corresponds to young, non-smoking individuals without children, whereas the highest-charge com- ponent comprises older, smoking individuals with high BMI.
Age
BMI
Children
Smoker
Region
Dreasty
Deasity
50000
48000
50000
20000
40000
Lnsurnacc Changes
Instrance Charges
(a) The GMM recnvers distinct modes, but no explanations.
(b) The XMM recovers similar modes to the GMM (left), and also explains when each mode is observed using simple rules over descriptive features.
Figure 10: Comparison Gaussian Mixture Model (GMM) and an Explainable Mixture Model (XMM) dataset of insurance claims (Choi 2017).
B
PROOFS
We provide the proofs for the propositions stated in the main text.
B.1 PROOF OF RECOVERY OF MARGINAL DISTRIBUTION
Proposition 1 Let M = {ei} be an XMM with a marginal density as per Def. 4 If the set of explanations e; form partition of the feature space Rá, i.e. Σiz cx(x) = 1 for all x in the support of Px, then the induced density pm)(y) equals the true marginal density PY (3).
Proof: By Definition 7 the induced marginal density of an XMM is
k
(X)]
PM(U)= Σ
wi p.(y)
with
w.
Pi(v) = -M\((((X))))) Σ EXe((X))'
1=1
If the explanations {es) i-1 form a partition of the support of Px, then Σk cx(x) for all x in the support of Px, and hence
/ Σ e(())x(()dzz= px(ee) dx
Σ Bje((X)]
i=1
Therefore we = Ele,(X))/1 (e(e,()),, and substituting this yields
PM(()= Σ Ele,(X)) (1(())=)(1))
By Bayes rule we rewrite
* Σ (x))
Py, (x)
pm(y)= =
(13)
P(e,(X) =)
13

---

Published as a conference paper at ICLR 2026
As (X)] (X),(X) = 1), cancel terms to obtain
PM(U) = Σ PY, (x)))=1()))
Finally, since the events {x: of total probability implies
measurable partition of the support of X, the law
Σ (x)) (p)(()
Thus (v) py (v), proving the claim.
B.2 PROOF OF RECOVERY OF CONDITIONAL DISTRIBUTION
Let M = feilt be an XMM with conditional density as per Def. 2 If the set of explanations e; form a partition of the feature space R into homogeneous regions with respect to the target variable Y, i.e. for every explanation e, and its induced sub-distribution p,ly). holds that Py x) pely) for all x with ex(x) = and px > 0, then the induced density PM x) equals the true conditional density py x (y x).
Proposition 2
Proof:
By Definition
2
k PM x)) Σ px()),
cx(x) Σラ=1 exx))
wx(x)
pr(v) = Py (o(()))))()))
If (ealt forms a partition of the feature space, then for every x the support of Px there exists unique index i" it (x)) such that ex (I) = 1 and ex(x) = 0 for all j * it. Consequently,
le (e((x)=1 1=1
(y-(x) = and wz(x) = for j t
and thus
PM(y x) = pp (u).
By the homogeneity assumption of the proposition, for all a with e, (x) we have
Pr x)) = - (v).
Combining the two displays yields PMlly x) pri x)) for all such in the support of Px. Hence the induced conditional density equals the true conditional densiry.
C LEARNING AND OPTIMIZATION DETAILS
This appendix provides supplementary details on the training, optimization, and rule extraction pro- cedures for XMM.
C.I ONLINE PRUNING AND POST-HOC MERGING
Online Pruning. During training, some rules may fail to specialize any subset of the data. The optimizer can effectively disable such rules by learning an inverted interval (aij > B,₃)) for one more of its predicates, which drives its activation (x(x) towards zero. We periodically identify rules whose average mixture weight [xx] over the dataset falls below small threshold (e.g., 10 10-3). These components are considered inactive and permanently removed from the computa- tion for the remainder of training by fixing é(x) = and skipping density computation. This saves computational resources and improves stability by fully removing the gradient.
14

---

Published as a conference paper at ICLR 2026
Post-Hoc Merging. The maximum likelihood objective is invariant to splitting homogeneous data region into multiple sub-regions modeled by functionally identical experts. This can result in a fragmented solution. To improve interpretability, adjacent explanations j, l we compute the pairwise similarity of the densities (v) and (V) using Sensen-Shannon divergence. We consider explanations adjacent if their data-based intervals (see Appendix C.2) touch (= some tolerance) on one feature and are similar on all others with non-zero weight If the divergence between pair of densities is below predefined threshold, merge their corresponding rules by taking the union of their redundant experts.
merge such components after training. For all
intervals and retain only one of the
C.2 TEMPERATURE ANNEALING AND RULE EXTRACTION
To produce final, human-readable set of rules, the soft, differentiable model must be converted into discrete, logical representation.
Temperature Annealing, The temperature parameter T in the soft predicate (Eq, P controls the trade-off between smooth gradients for effective optimization and sharp boundaries for interpretabil- ity. We begin training with higher temperature to allow for broader exploration of the solution space. As training progresses, we gradually anneal T towards a small positive value. This process encourages the model to converge towards solution with crisp, well-defined decision boundaries that closely approximate hard logical rules.
Data-Based Rule Extraction. Simply reporting the learned interval parameters [asj. 9.y] can be misleading, as optimization may push boundaries towards infinity in uncontested regions of the feature space. We therefore derive more faithful representation of the learned partition from the empirical properties of the data governed by each rule.
For each explanation e;, we first identify its corresponding data partition, D,. This partition consists of all samples assigned to component i based on the maximum responsibility criterion, as defined for label extraction in Section D.4 That is,
D; = v) | arg max w/ (x)]
(14)
5
The final, human-readable rule for component is then defined by the empirical range of the data in D, for each feature j: [millxeD, Ty; maxxeD, xy]. This data-derived bounding box is valid representation because our predicate design ensures that if explanation i has maximum responsibility for the empirical minimum and maximum values in D,, it also does so for all values between. We report these ranges for all features, visually distinguishing those the model deemed unimportant (i.e., a. S 0) communicate both the model's concise logic and the data's full distributional properties.
We use this this rule extraction to create the rule visualizations (see for example Fig B). The bars indicate the range, categorical features show segments. The segments can be partially colored multiple values are present in an explanation. Features that are active (a > 0) are blue, others are grey. The empirical intervals are computed for all features, active or not.
C.3 MODEL SELECTION
Since the true number of components & is unknown, we treat it as a hyperparameter. We train set of models with range of values for i (e.g., k (10, 100)) and select the best one using the Bayesian Information Criterion (BIC). The BIC score is calculated after the online pruning and post-hoc merging steps have been applied. The penalty term in the BIC score considers only the number of active parameters in the gating network (the rule bounds aij, 3ij, and weights a:j). This choice reflects our goal of finding the most parsimonious partitioning of the feature space, rather than penalizing the complexity of the expert density estimators, which could otherwise dominate the score. This automatic balancing of model complexity and fit provides an alternative to manually chosing k.
15

---

Published as a conference paper at ICLR 2026
D EXPERIMENTS
All experiments are performed on an Intel 15-12400 and Nvidia RTX 3070. GPU acceleration was used for methods that support it, which is true of XMM.
D.1 SYNTHETIC DATA GENERATION DETAILS
We generate synthetic data from a process that mirrors our model's core assumption that the data arise from mixture of components, where each component corresponds distinct subregion of the feature space with an associated conditional density. We define a collection of disjoint, axis- aligned hyperrectangular regions { that partition the feature-space Rd, For each region Ht, define an unconditional target density Py (v) on The resulting ground-truth conditional density pienewise-constant over IR., taking the value Py (v) for any feature vector E Hz-
}
Recursive Binary Partitioning. The regions are constructed by recursively splitting an initial hyperrectangle in manner analogous decision tree. This procedure ensures that the resulting set of regions forms a true partition and avoids creating excessively thin regions. We also generate empty leaves that will not get any samples to make the data more realistic.
1. Initialization. Start with the full domain as the root of a tree.
2. Recursive Splitting. Iteratively select leaf node and split it along randomly chosen fea- ture dimension. A split is permitted only if the node's width along that dimension exceeds a minimum threshold. The tree grows until a target number leaves reached.
3. Component Selection. From the set of leaf nodes, select exactly & serve the active components, defining the regions {H /
We show a full partitioning in Fig. 11a and one that contains 50% empty leaves Fig. 11b
Conditional Density Assignment and Sampling. Once the feature space is partitioned, sign target densities and generate samples. For each active region Hj. we draw an unconditional density Py(y) from randomized family of standard distributions (Gaussian, Exponential. Gamma, Uniform) to induce diverse shapes. We show an example such densities in Fig. [Tc To generate the dataset, specify a fixed number of samples n, for each region. For each of the n, samples in region Hj. we first sample the feature vector x uniformly from within the hyperrectangle defining Hy. and then sample the target value y from its corresponding density, y (y)) The resulting ground-tree -trnh conditional density is
l x) Σ Txx [(xeHH)))())
Task difficulty can be tuned by controlling the overlap between the densities {p)(r) via a parameter [0, 0.5). A small / yields well-separated densities, while 3 0.5 implies that all densities share the same median.
D.2 BASELINE DETAILS
For all baseline methods, we utilized the authors' publicly available implementations and followed their recommended parameter settings unless otherwise specified.
CDTREE. state-of-the-art interpretable model that greedily builds a decision tree with non- parametric histogram densities in the leaves, regularized by the Minimum Description Length (MDL) principle. We use the authors' original R implementation with default parameters.
CADET. An intrinsically interpretable CDE method that fits decision tree with parametric dis- tributions in the leaves. We use the authors' implementation with BIC regularization. The method requires specifying the parametric family for leaf distributions. We use Gaussians, other fami- lies led to numerical failures on our test data. We further add very small Gaussian noise (standard deviation 0.001) the target feature duplicate values cause the method to fail.
16

---

Published as a conference paper at ICLR 2026
1.0
1.0
25.00
Ph.91
0.4
0.4
0.50
1.00
0,00
0.75
1.00
-2.5
11.0
2.5
271
(a) Exact partitioning into ac- tive regions.
(b) Partitioning with active
(c) The constant densities py (v) asso-
and 4 inactive regions.
ciated with each region H,
Figure 11: Illustration of the steps involved in the synthetic data generation.
Mixture Density Networks (MDN). A neural network- based approach where the network outputs the parameters (mixture weights, means, variances) of a Gaussian mixture model for the target variable, conditioned on the input features.
Kernel Mixture Networks (KMN). Similar to MDN, but models ply x) as mixture of fixed kernel functions whose mixture weights are determined by neural network conditioned on
Least-Squares CDE (LSCDE). A non-parametric method that directly models the conditional density without assuming specific functional form, using kernel-based approach.
Normalizing Flows (NF). This method combines conventional neural network with multi- stage Normalizing Flow, where the neural network outputs the flow parameters.
For MDN, KMN, NF, and LSCDE, we use the implementations from the Python CDE package by Rothfuss et al.) We apply noise regularization of 0.01 to both features and targets, and otherwise use default parameters, On synthetic data we 3-fold cross validation to select the number of kernels of KMN to improve label quality.
Conditional Variational Autoencoder (CVAE). We implement CVAE (Shnn al. 2015) with learned conditional prior, where the encoder q(= x, v. decoder p(y =), and prior p(= x) are parameterized by multi-layer perceptrons with ReLU activations. We employ a latent dimension of 16, with hidden layer sizes of (128, 64, 32) for the encoder, (32, 64, 128) for the decoder, and (64, 32) for the prior. The decoder models the conditional likelihood as a Gaussian distribution. We optimize the Evidence Lower Bound (ELBO) with KLD weight of 0.5 using Adam and apply early stopping based on validation set performance. Both features and targets are standardized during training. We estimate the test NLL by approximating the marginal likelihood p(y x) via Monte Carlo sampling with 2000 latent samples.
D.3 IMPLEMENTATION AND PARAMETERS
We implement XMM in Python using standard machine learning libraries.
For the experiments we additionally apply standard entropy loss regularizer to the feature impor- lance weights. This mainly serves to make rules more concise for interpretability by encouraging the optimizer to actually reduce for redundant features. Let a; (a.1,.. 0ia) be the vector of non-negative feature importance weights for rule é, Negative weights are set to 0 for this calcula- tion. Rules with no support are ignored. First, these weights are normalized form probability distribution
az
Øig
(15)
11=1
17

---

Published as a conference paper at ICLR 2026
Ride Complesty
Ruks
Daner
XVM-NNS
Xati-Na
NVV-GUN XMV-GUU BIC
(131456
XUM-NSF
XиАААНИ BIC
CD7TES
SollCwh
10:17
5.83
7.07
6.00
6.00
15.00
6.00
1.00
166
Themography
10 h4
1::3
0:11
3.33
1.00
1.58
17:00
15.10
16.00
0.00
1.00
00
shalone
131
3:31
10:00
110
11:00
7.10
1.00
pequuity
Nkz
531
3.39
12:00
16.00
31.00
11.00
Tonsen
7:0
7/17
133
1/00
2.00
3.5
15:00
15.10
23.00
18.18
6.80
75.08
consict
6.54
420
4.34
4.30
0.32
4.20
1630
16.00
12.00
10.00
15.00
65.10
cacras
65.10
lif
0.73
15,00
7.35
7112
2.50
4.50
11.00
12.00
15.00
5,00
20.00
302 m
dlssly
0.00
7.56
5.87
3.88
0.00
4.23
7.00
25.00
7.30
1.00
127 22:.0
syachnooss
17.00
7.00
18.10
wages
0.07
1.50
1.65
3.80
1.00
4.07
9.00
5.00
25,
30.00
2.00
55. m
wase
5.00
3.00
1.67
3.70
0.00
6.58
0.00
3.00
0.00
4.70
1.00
100.00
Table 2: Rule complexity of interpretable models on real-world datasets.
The entropy regularization term is then the average Shannon entropy over all & rules
1 1: Ra(M)
Tey log(ā.;)
(16)
Adding this to the objective we get
min NLL(M) + AR(M) + (M
(17)
MM
where Aa is a hyperparameter.
For all experiments we use x 0.1. We use \ 0.05 for synthetic experiments, and A 0.1 for the real data experiments. All synthetic experiments are ran with BIC selection of k E {10, 100), except the scaling experiment with = (Fig. 4aj. where we use k E {10, 2003. We always use starting temperature of r 0.2 and smoothly anneal it to = 0.005 during the middle 80%
of training epochs. The first and last 10% are reserved to encourage initial competition and final settling of the borders. For online pruning we use threshold of Ex (x)] < 0.005.
D.4 METRICS
Component Label Extraction. On synthetic data, we can compare the predicted component la- bels to the ground truth. For XMM, we assign each sample (Xn, Vn) the component with the highest responsibility, which corresponds to the most active explanation for that sample's features:
En arg max (x((x)) je(l....,kk
(18)
For Kernel Mixture Networks (KMN), which models the conditional density as ply | x) Eee (x) o;), we cannot obtain feature-based rules. Instead, we assign a label based on the most probable kernel component for the full data point:
2n arg max
Mjj o,)
(19)
(E.......,MM
D.5 ADDITIONAL. RESULTS
D.5.1 MODEL FIT ON SYNTHETIC DATA
Pseudo R2 (.ermete) We report a normalized log- likellhoooo score to ensure comparability across different experimental settings. This metric measures the fraction of improvement a model achieves over an unconditional baseline, relative the improvement achieved by the ground-truth data- generating model (oracle).
D.6 DESTRUCTIVE NOISE
We perform an additional robustness experiment with destructive noise, showing the results in Fig- 13 We replace the Y value for increasing fraction of samples with noise sampled from
18

---

Published as a conference paper at ICLR 2026
Interprerable XMM-QQM
Dalaset
XMM-NSF
XMM-NSP BiC
XMM-GMM Bic
CDTREE
CADET
CVVE
KMN
LSCDE
MON
NF
SkillCralt
305.9
255.4
29.5
39.7
3056.
0.2
3.5
36.4
5.0
6.5
8/4
Thermography
abalone
5
335.3
22.3
35.1
9625. 1
0.2
5.1
35.0
3.1
6.7
9.01
air quality
502.5
352.1
29
43.9
1385 8
0.5
12.6
39.8
5.7
9.0
11.7
bike
890.5
521.6
49.5
66.8
36.7
0.0
1.3
32.1
2.2
5.2
5.2
43.9
concrcte
772.4
524.5
51.3
59.6
96.2
0.0
3.3
32.3
2.0
5.0
5.5
601.1
40012
24.8
32.4
201 1
0.5
9.D
41.4
7.9
9.4
11.8
lile
611-4
436.6
38.2
0.1
4.1
33.8
1.8
5.7
obesity
317 1
268.2
38.0
13.7
63L.
0.1
3.5
3L.T
2.8
5.8
6.3
synchronous
608.4
456.6
27.3
35.6
45.8
0.0
5.2
31.9
1.4
4.4
5.2
toxicity
611.6
wages
588.8
419.0
42.1
50.8
62.5
0.1
2.5
333.5
2.4
5.4
5.6
wine
315.4
236.3
22.3
28.0
2168.6
0.3
6.7
36.1
5.3
6.9
9.9
Rank
Table 3: Runtime in seconds.
g0.
1.2
20.6
82
0.4
0.5
0.2
Xoue-Lous CDTia
8:2
Xaot Nap
XaN
04
Bd
10
28
60
0.85
0.1
0.
055
68
62
86
Ovetlap
Noise Frocunes
Notoe Scale
Overlap
(a) True Components
(b) Noise Features
(c) Target noise
(d) Overlap
Figure 12: Likelihood fit (R² ( on synthetic data for varying (a) number of true components, (b)
number of noise features, (c) target noise level, and (d) overlap between components.
Normal distribution (M,, 1) where μ IE()) This tests robustness when noise introduces significant outliers relative to the true conditional distributions. Figure 13b we see that the condi- tional structure is recovered accurately the NLL. Due to the increased presence of outliers that are modeled by the same number of density estimators, the likelihood degrades when maintaining the true conditional structure.
when 30% of Y values are destroyed. Figure 1alssows
D.7 RUNTIME
Finally evaluate the scalability data dimensionality increases. For show the results for in- creasing d in Fig. 14a and for increasing number of samples Fig. 14b] We observe that neural methods like XMM and KMN are consistently fast even large datasets. Our NSF instantiation takes longer to run due to increased parameter count, but exhibits stable scaling. The runtime of CDTree increases very quickly even for moderate dimensions due its iterative nature. CADET is comparatively very fast because of its small search space.
D.8 ABALONE CASE STUDY
We apply XMM to the popular abalone dataset which contains various size and weight measurments of abalones, kind sea snail. Typically this dataset used for regression or classification using Age as the target variable. We apply XMM using 28 Gaussian density components as there are 28 unique values in Age. In Fig. [15] we show that XMM can recover reasonable explanations and distributions. The explanations show that larger and heavier abalones have higher mean Age. But because we estimate the entire conditional distribution we can further see exactly how Age is distributed for these subgroups. For example explanations consisting mostly (1) entirely (2) of infants are distributed in relatively low and narrow age range. Explanation 6 contains the largest and heaviest ones, which distributed the upper end with wider distribution. We interpret this explanation to describe abalones that have reached their maximum size but continue to age. CDTree does not find any conditional structure in the data, returning a tree consisting only of the root node.
19

---

Published as a conference paper at ICLR 2026
1.5
1.0
Ted NL.
1.0
Test NMI
Xasae-Nir
0.4
XasreGoux
Mbx
0.2
Xasr--iar
CDTure
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
Destructive fraction
Destructive fraction
(a) Test NLII
(b) Test NMT
Figure 13: NLL and NMI for increasing fraction of Y samples replaced with destructive noise.
Xast Nar
7000
2000
Yuntinn
Bustime
Xast
Samples
Samples
Runtime vs Features
(b) Runtime vs Samples
Figure 14: Runtime of all methods on synthetic data with increasing number of features (left) and samples (right).
Whole weight
Shooked weight
Viscem
Shell weight
Length Diameter
Height
weight
Female
Infont
Male
0.15
Probibiity Mass
0.05
0.00
Age
Figure 15: XMM results Abalone. Probability masses weighted by explanation size.
D.9 GOLD HOMO-LUMO CDTREE
We provide a visualization of the CDTree density estimates on the Gold nano clusters dataset with target variable HOMO-LUMO in Figure 16
E LLM USAGE
LLM usage did not play significant role in research ideation or writing of the paper itself. However LLMs and assistants were used during the implementation of the method.
20

---

Published as a conference paper at ICLR 2026
Leares
Population
25
Teats
20
1.0
0.5
0.0
1.5
2.
HOMO-CUN CUMO
Figure 16: CDTree result on Gold dataset with HOMO-LUMO target. Densities are scaled by weight (relative number of samples per leaf). Legend abbreviated, colors repeat.
21


Reviews:
[{"review_id": "9BCkgeX5E1", "reviewer": "Reviewer_ZarE", "summary": "Given a set of data-value pairs $(x_i,y_i)$, where the covariates $x_i \\in \\mathbb{R}^d$, the paper proposes an explainable mixture model (EMM) approximation of the conditional density $p(y|x)$. The mixture model is conceptually proposed as follows:\n- first, the $p(y|x)$ is approximated by a weighted mixture of $k$ components\n- each component ideally represented as a box (intersection of intervals) in $\\mathbb{R}^d$ or in a lower dimension. The authors show that if the components are supported on a partition of $\\mathbb{R}^d$, then there is no error in the approximation\n- In order to apply standard optimisation to learn the parameters, each component is approximated by a  smoothed \"box\" (equation 10)\n- the mixture model parameters are learned by minimising a regularised negative log likelihood\n- some further pruning is done to reduce the number of rules learned\n\nThe authors demonstrate that the proposed EMM model provides good fit of data, while providing fewer learning rules for the explainable model.\n\nThe authors show", "questions": "Could the authors explain how to read the explanation plots (for example, right hand side of Fig 7)? What do the bars and colours mean?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The overall EMM model uses a combination of simple ideas, but learns an explainable rule-based model that consistently has fewer rules.\nThe experiments look convincing (with the caveat that I not an expert in this specific problem)", "weaknesses": "The paper's theoretical contribution is limited. A simple result is provided in the case where the supports of the components form a partition. However, there are no guarantees regarding the severity of the approximation error in the general case.\nI was tempted to give a low score due to this, but it turns out that this is not a concern in relevant literature (papers like CADET, CDTree).  \n\nI found parts of the papers a bit difficult to follow, which could be because I am not familiar with this line of work (and generally, less familiar with explainability literature).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MYoIGNo1BL", "reviewer": "Reviewer_LLHS", "summary": "This paper proposes EMM, explainable mixture model, a principled learning procedure that discovers each mixture component with a human-interpretable rule over descriptive features. Experiments on synthetic and real-world datasets demonstrate the effectiveness of the proposed approach.", "questions": "N/A", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed approach is based on a principled machine learning procedure.", "weaknesses": "The experiments are restricted to mixture models, and don't reflect the paradigm shift in the era of deep learning. The experiments should include the results that compare with deep learning models such as VAE, and whether the proposed can be extended deep learning models; the data sets are UCI which are too small, don't support the claim that the proposed method is scalable and applicable to more complex, multi-model distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cQ7aWbNHF0", "reviewer": "Reviewer_2m51", "summary": "This paper maximizes the conditional likelihood, where the conditional likelihood is parameterized using a mixture model, and adds a regularizer to the objective to encourage each data point to follow a unique path within the mixture. The authors design the mixture model as a rule function, providing explainability, and learnt by gradient descent.", "questions": "Can you perform an ablation on $\\lambda$, and also explain how increasing the number of mixtures affects the conditional likelihood?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a novel objective function that maximizes conditional density estimation in mixture-models.", "weaknesses": "How is this different from Mixture of Experts (MoE)?\nMoEs use a gating function to decide which expert/block to activate, and are typically trained under a maximum likelihood objective for generative modeling. I see EMMs as a special case of MoEs where the gating logic is determined by learned rules. There are many possible gating mechanisms that can yield explainability, for example by restricting the gating function to be simple. Calling these models “EMMs” and not acknowledging them as a special case of MoEs is, in my view, misleading.\n\nDefinitions are not complete\nHow do you define “human-interpretable explanation”? This is subjective. You need to either define it explicitly or give application-specific criteria before introducing Definition 1 (Marginal-EMM). Otherwise, you cannot really call it a “definition.”\n\nAlso, from Definition 1, any tree-based method would qualify as an EMM if you take the path to the node as, $e_i$. Does that mean all tree-based models are EMMs?\n\nComparison to [1]\nThis paper modifies the objective proposed in [1] for conditional density estimation. Can you provide a direct comparison with [1]? The only clear difference I see is that your objective removes the entropy term. But [1] can still be used to compute a conditional density. How does the current method compare to [1] in terms of (a) explainability and (b) test-set likelihood?\n\nI also disagree with the claim that tree-based approaches are “prone to overfitting.” Any machine learning method, deep learning models, are prone to overfitting.\n\nFinally, in Eq. (6), since $w_i$ is constrained to be positive, why do you need to square it?\n\n[1] Sascha Xu, Nils Philipp Walter, Janis Kalofolias, and Jilles Vreeken. “Learning Exceptional Subgroups by End-to-End Maximizing KL-Divergence.” ICML, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bjt1mxWVkT", "reviewer": "Reviewer_utZR", "summary": "They propose a method for explainable mixture models. The explanations created are a conjunctions of different features. The mixture model created can fit the model well and able to generate logical rules to explain the mixtures created.", "questions": "* Are the definitions in section 3 part of contribution or is it background?? It is a bit unclear. if it is background, then it is good to add ref. for them.\n* What kind of noise is added in the robustness experiments and how much??\n* In synthetic dataset will the performance change if the data is generated from a random distribution rather than a gaussian and uniform?\n* Given the rules are logical statements can the features be continuous or do they have to be discrete/categorical for the approach to work?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Well written paper.\n* Good exploration with synthetic dataset to showcase the capability of the models.\n* Approach also fairly robust based on the synthetic dataset noise.", "weaknesses": "* The motivation of the problem with application can be improved.\n* Many variants of their method is being introduced and it is a bit unclear on what the conclusion is between them. While they say that over NLL on real dataset is best for EMM-GMM, i am not sure how accurate it would be to claim that to be the best, as different datasets yielded different best model.\n* Some human evaluation for some real-world dataset would have been good to evaluate the goodness of the explainability.\n\nNOTE: i did not read the appendix section and my comments are purely based on main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)