Conditional Density Estimation with Histogram Trees
Authors: Lincen Yang
Matthijs van Leeuwen
Venue: NeurIPS 2024 poster (2024)
Keywords: Conditional density estimation
MDL principle
Decision Tree
Histogram
Primary area: probabilistic_methods
TL;DR: Intrinsically interpretable models for conditional density estimation, employing decision trees equipped with histogram density estimation

Abstract: Conditional density estimation (CDE) goes beyond regression by modeling the full conditional distribution, providing a richer understanding of the data than just the conditional mean in regression. This makes CDE particularly useful in critical application domains. However, interpretable CDE methods are understudied. Current methods typically employ kernel-based approaches, using kernel functions directly for kernel density estimation or as basis functions in linear models. In contrast, despite their conceptual simplicity and visualization suitability, tree-based methods---which are arguably more comprehensible---have been largely overlooked for CDE tasks. Thus, we propose the Conditional Density Tree (CDTree), a fully non-parametric model consisting of a decision tree in which each leaf is formed by a histogram model. Specifically, we formalize the problem of learning a CDTree using the minimum description length (MDL) principle, which eliminates the need for tuning the hyperparameter for regularization. Next, we propose an iterative algorithm that, although greedily, searches the optimal histogram for every possible node split. Our experiments demonstrate that, in comparison to existing interpretable CDE methods, CDTrees are both more accurate (as measured by the log-loss) and more robust against irrelevant features. Further, our approach leads to smaller tree sizes than existing tree-based models, which benefits interpretability.

Conditional Density Estimation with Histogram Trees
Lincen Yang Matthijs van Leeuwen LIACS, Leiden University
Einsteinweg 55, 2333CC Leiden, The Netherlands
(1. yang, m. van leeuwen] @al iacs deidenuniv.cr
Abstract
Conditional density estimation (CDE) goes beyond regression by modeling the full conditional distribution, providing richer understanding of the data than just the conditional mean in regression. This makes CDE particularly useful in criti- cal application domains. However, interpretable CDE methods understudied. Current methods typically employ kernel-basee based approaches, using kernel functions directly for kernel density estimation contrast, despite their conceptual simplicity and visualization suitability, tree-based methods-whieh which are arguably more comprehensible- have been largely over- looked for CDE tasks. Thus, we propose the Conditional Density Tree (CDTree), fully con-aarametric model consisting of decision tree in which each leaf is formed by histogram model. Specifically, we formalize the problem of learning a CDTree using the minimum description length (MDL) principle, which eliminates the need for tuning the hyperparameter for regularization. Next, we propose an iterative algorithm that, although greedily, searches the optimal histogram for every possible node split. Our experiments demonstrate that. in comparison to existing interpretable CDE methods, CDTrres are both more accurate (as measured by the log-loss) and more robust against irrelevant features. Further, our approach leads to smaller tree sizes than existing tree-based models, which benefits interpretability.
basis functions in linear models. In
Introduction
Conditional density estimation (CDE) is a crucial yet challenging task in modeling the associations between features and a continuous target variable, which has received a lot of research interest since the 1970s 44. By modeling the full conditional distribution, CDE is useful when the datasets are multi-modal, heavily skewed, or heteroscedastic. As a result, it is widely applied in various fields, including genomies [8], astronomy
B3, wind power forecasting [19), and computer networks [47].
CDE provides richer information for data understanding than regression, which only models the conditional mean. This makes CDE desirable for well-informed decision-making in critical areas, such healthcare
32
which calls for interpretability.
Despite its importance, recent CDE research has predominantly focused on black-box models, such neural networks [10
46 50) and tree ensembles [13. In contrast, intrinsically interpretable models for CDE are understudied. Specifically, decision tree-based methods are largely neglected, with CADET being the only existing method to the best of knowledge. However, CADET's assumption of Gaussian distribution for the target variable in each leaf node limits its ability to model complex conditional densities.
55
39
As result, kernel-based models became the standard among 'shallow" models for CDE, including methods based on kernel density estimation (KDE) 44), and linear models with the basis functions chosen as Gaussian kernels [44] II. Nevertheless, kernel-based models are arguably less interpretable than decision trees: conditions in decision trees directly readable, they are comprehensible to
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

humans without statistical expertise (e.g., individuals affected by data-driven decisions rather than professional data analysts).
To address these limitations, we propose the Conditional Density Tree (CDTree), a flexible tree-based CDE model in which each leaf node consists of a histogram model. For illustration, Figure shows the CDTree learned from a dataset about personal medical costs with demographic features [24]. The figure visualizes the histogram models for the conditional densities on three selected leaf nodes, along with the unconditional density. The readable rules, extracted from the tree paths, and the histogram visualizations, make the results easily comprehensible to humans without statistical expertise. For instance, the difference in medical costs between smokers and non-smokers is evident when comparing the plots. The full results with histograms on all leaf nodes, as well as further descriptions of the dataset, are provided in Appendix E
Learning a CDTree from data is a challenging task, it requires simultaneously optimizing the decision tree structure and the number of bins for histograms on all leaf nodes. While often used for either task, cross- vessdaalio is too time-consuming if we need to search for the optimal number of bins for every possible node split. Thus, we formalize the learning problem using the minimum description length (MDL) principle 16 41, which we briefly review in Section 4. Adopting MDL eliminates the need for tuning the hyperparameter, by cross-validation, for both regularizing the
decision trees 5 and for choosing the number of bins for histograms.
Our main contributions are as follows. First, we introduce the CDTree for the CDE task and formalize the learning problem with the MDL principle. Second, we propose iterative algorithm that can search the optimal histogram for all possible node splits. Third, we benchmark against a wide range of competitors and demonstrate that CDDTrre is highly competitive. Specifically, CDTree is more accurate than existing interpretable CDE methods (as measured by the log-loss). Meanwhile, CDTree has smaller tree sizes than other tree-based methods, which benefits interpretability. In addition, CDTree is extremely robust to irrelevant features, which is noteworthy as irrelevant features are known to harm the convergence rate for CDE [77. Further, we argue that the (intrinsic) explanations of CDTree are trustworthy only if the CDTree is robust against irrelevant features.
Unconditiona density
age age smokeer
age age 30 TTue
age &bmi 5- emoceer
20-
Ioqqueney
800-
100-
50-
20-
2000D
au0co medical cost
20000
40000
2000
20000
medicall 0ost
mecical coGl
mecical cosl
Figure 1: Three selected leaves from the CDTree modeling the conditional density of the medical costs given demographic features, together with the unconditional density for medical costs.
2 Related Work
KDE-based CDE. Several CDE methods have been developed based on kernel density estimation (KDE). The most straightforward approach, known as conditional-KDE (CKDE) 44, involves separately estimating the joint and marginal densities using unconditional KDE and then taking their ratio to obtain the conditional density. Another approach, e-neighborhood KDE (NKDE), estimates the conditional density by applying KDE to the subset of data points within the e-neighborhood of the target point, with the range of the neighborhood controlled by the parameter e.
However, KDE-based methods have several limitations. First, KDE is arguably less interpretable than bree- models, as understanding KDE requires statistical knowledge, making it less accessible to domain experts and the general public. Additionally, while decision trees can naturally handle both discrete and continuous feature values, discrete features pose significant challenges for kernel-based methods. Specifically, CKDE requires estimating the joint density of the features and target variable, which necessitates the use of discrete kernels for discrete variables. These discrete kernels are often difficult to interpret (31. As for NKDE, when both discrete and continuous feature values exist, the choice of the scale for the continuous variables unavoidably introduces a certain degree of arbitrariness in defining the distances used to characterize the neighborhood.
2

---

Regression-based CDE. Motivated by several issues of CKDE, including high variance as plug-in estimator, the exponentially growing search space for bandwidth tuning, and the curse of dimension- ality for estimating the joint density function, the method named least-squares CDE (LSCDDE) [54] was proposed. LSCDE aims to directly estimate the ratio between the joint and marginal densities by assuming this ratio is a linear combination of several basis functions, chosen to be Gaussian kernels. Similarly, Fan et al. [ proposed double-kennel local linear regression, which transforms the conditional density function into the conditional expectation of the (unconditional) density function of the target variable, leading to least-squares approach as well. However, both methods face the challenge of bandwidth selection, as the value of the Gaussian kernel function must be calculated with the entire feature vector as imput. This issue becomes particularly problematic as the search space for the bandwidth grows exponentially with the number of features.
Tree-based CDE. The only existing CDE method based on single trees that we are aware of is CADET ( which improves CART regression trees ( by designing a node-splitiing heuristic specifically for CDE. However, CADET assumes a Gaussian distribution for the target variable on each leaf, which is far less flexible than our non-paraett histogram models.
Black-box CDE. Neural networks have been shown to perform well in a wide range of tasks. For CDE, the most well-known methods include NF [39] and MDN 4 As for tree ensemble models, RFCDE 35| first fits a standard random forest, and then estimates the CDE by the weighted average of the unconditional density estimates obtained via KDE, with weights determined by the random forest. Furthermore, the recently proposed LinCDE [3] method Iearns a boosted tree model based on Lindsey's 's method (25). While black-box models are highly accurate, we argue that interpretable CDE methods are valuable for applications in critical domains and for understanding the data.
3 Conditional Density Tree with Histograms
Tree-based models divide the feature space into disjoint (hyper-)boxes by recursively splitting on individual feature variables (for which we consider binary splits only). Consequently, the induced partition with K subsets (leaves) can be represented as M = {SA [kk]k where each leaf Sk represents a subset of the feature space and [K] == {1,.... K}. Further, each leaf is equipped with a single (unconditional) density estimator, denoted as f.)... Thus, for a dataset D (x",y") y²) with sample size n. the tree-based model M first identifies the leaf to which each (I, y) E D belongs, and then estimates the conditional density fy]zx) as fi (v) (assuming x E Sz).
We choose histograms as our model class for each f.)..). Unlike previous parametric models 13 6). which carry the risk of misspecification, histograms are non non-parametric yet efficient. Additionally, in comparison to KDE, histograms do not require selecting a kernel or tuning the bandwidth.
We next describe our notations and formally present the histogram as a probabilistic model. Formally, a (fixed) histogram model partitions the domain of the target variable Y into equal-width bins, and then approximates the density of Y by piece-wise constants estimated from data. Thus, we can denote histogram model as a tuple H = (Bi, Bu, h). with B1 and B, respectively representing the lower and upper boundary of the histogram, and h the number of bins. A histogram model /l can parameterize a family of distributions by a = (a11..... 011), with the probability density function in the form of fu = 3) B.)/k Vy E [B| + (j- R3,- B, Bi+j Pu- -B-) )., in which (P,-Bi)/l
3;
-
the bin width and [Br + (j B. Br + j _ denotes the interval for the jth bin. In practice, the histogram boundaries Bi and B,, are often set based on the dataset at hand, by prior knowledge, or by the range of the values plus/minus a small constant. Meanwhile, the parameters a can be estimated by the maximum likelihood estimator (i.c., by the empirical frequencies in each bin): d;- -Σ [B++(-1) By B By 21 (æ:), in which 1()) is the indicator function.
4 The MDL-optimal CDTree
We briefly review the minimum description length (MDL) principle, and then formalize the problem of learning a CDTree that consists of histograms as MDL-based model selection problem.
3

---

4.1 Preliminary: the MDL principle for model selection
Rooted in information theory, the minimum description length (MDL) principle states that the optimal model is the one that compresses the data most [41 15 Precisely, given the dataset D = (x", v²) the optiooptimal model is defined as
M arg min MEM
log, PM (y""""""" (x*) + L(M).
in which M is a CDTree and MM the model class of all possible CDTrees for our learning task, mean- while ((M) is the code length (in bits) needed to transmit the model in a lossless manner. According prior probability distribution defined on
to the Kraft's inequality 17). ((M) can also be regarded the model class. Further, (.) is the so-called universal distribution [5] as M parameterizes a fam- ily of probability distributions, denoted as PM, 8, (-) be regarded as "representative" distribu- tion such that the maximum regret, defined as maxy" mmax log, (y=,"(" - log, Pas (y" [=") [15, in which the maximum is defined over all possible
can be bounded by ne > 0, Ve > 0 as n values for y" in the domain of the target variable. Intuitively, the regret is the difference between the log-likelihood of Pas [x") (which does not depend on 0) and the maximum log-like likeelolooodd maxe logg Par.o (y" ). ) For our learning task, is the parameter vector that contains the histogram
parameters (i.e., the a' for all leaf nodes.
The MDL principle has been successfully applied to various data mining and machine learning tasks [12. and specifically to partition-based models, including histograms and classification trees/rules Bi 57 36 22]. The MDL framework provides a principled way of regularizing model complexity without any regularization hyperparameter to be tuned. Moreover, the Bayesian marginal distribution is one specific type the universal distributions, the MDL-based model selection can also be regarded as a generalization of Bayesian model selection [16].
4.2 Normalized maximum likelihood for CDTrees
The optimal universal distribution under the MDL framework is the so-called normalized maximum
likelihood (NML) distribution, defined as [5 49 16
maxo PM. PM,0(y"(x")
PM(""(x")
Jyn maxe Pas,o (ym(z*)
under the condition that the denominator is finite. It can be shown that the NML distribution is the only distribution that leads to the minimax regret minp,, max," { (maxe log, PM. (z")
log, (y""""(2")) [z"]"""))) in which the denominator in Eq. ils exactly equal to the regret [5]
The denominator in Eq. is in general prohibitively expensive to compute 15 exceptions including the cases when the probabistic model represents categorical distributions 21. decision rules for classification [57]. and one- and multi-dimensional histogram models 122 28 58). We extend these previous results and prove that, for CDTrees with histogram models, the denominator the regret terms of the NML distributions for one- dimensional histogram models, as shown in Proposition [ This result is useful for efficiently calculating the denominator (regret) in Eq. Eq.22 for CDTree models, as the regret terms for histogram models are known to be equal to those of categorical distributions (20 21). for which efficient algorithm exists with sub-linear time complexity [30).
with a few
(regret) is finite and equal to the products
Proposition 1. Let 0 (a'...., " be the histogram parameters for histograms on all leaves, and let 0 = arg maxę P'M,(y"((T") Then Sy maxş PM.e(y"(<")) [[EE[K]] R(Nk, hx), in which R(N,, fux) is the regret (denominator) of the NML distribution of the histogram model the kth leaf node that contains Nş data points and hy bins.
We defer the proof Appendix dluee to space limitations.
4.3 Code length for the model
To encode CDTree model in a lossless manner, we need to sequentially encode 1) the number of nodes in the decision tree, 2) the structure of the tree, 3) the splitting condition for each internal node in predetermined order (c.g., depth first), 4) the number of bins for the histogram each leaf

---

node, and 5) the boundaries B, and B, of the histograms. That is. let L(.) denote the function that calculates code length in bits, the code length needed to encode a model M can be decomposed
L(M) =L leaves) + [trree structure)
Σ
L(splitting condition of the jth internal node)
JEIK-1]
T(# bins of histogram on the kth leaf node) + L((Bi, Bu)));
RE(K]
next describe them in order.
Encoding the tree size and structure. As we consider full binary trees only, it is sufficient encode the number of leaves K, which also determines the number of total nodes. As K is positive integer, we adopt the standard Rissanen's integer universal code 422 ,, denoted as (N(K), (K), for which the code length is equal to (K) + logg ( (K) + log2 logg K) + ...; the summation continues until small enough precision is reached, and с z2 2.865 is a constant.
Further, for full binary trees, the number of all possible tree structures for trees with K leaves is equal the Catalan number, denoted as Cg - [52]. Hence, specifying one certain structure costs log, Cx 1 bits. Thus, (K( + log2 Cx-1 bits are required for encoding the tree size and structure.
Note that while there exist multiple ways of encoding tree size and structure (e.g., one alternative is to leverage the joint probability of the tree size and the tree structure, which can be defined by treating the tree realization of Galton-Watson process 33 21,, our encoding scheme adheres to the principle of achieving conditional minimax 166] by explicitly putting a prior on the number of nodes.
Encoding the splitting conditions. An individual splitting condition on single tree node consists of a variable name and a splitting value, which we encode sequentially.
To begin with, if the dataset contains m feature variables, encoding the name of certain variable X costs log, m bits. Further, the code length needed for encoding the splitting value depends on the variable type. First, for a discrete variable X with |A' unique values, specifying single value cost log2(1 bits. Second, encoding the splitting value for a continuous variable can be achieved by sequentially encoding 1) the granularity level for the search space, denoted as the positive integer d, and 2) the exact value within the granularity level Specifically, with fixed d. we consider as the search space the C 2d-1 quantiles that can partition the values of X into equal-frequency bins. Note that the values of X are based on the subset of data points locally contained in this internal node.
Hence, encoding d costs LN(d) bits with Rissanen's 's code [42) and encoding one specific splitting value (quantile) costs log2 (C 22-1)= = (og2(C) + d 1 bits. That is, we treat the granularity level d as a parameter to be optimized when learning CDTree from the data, which avoids (arbitrarily) specifying the granularity in advance, a shortcoming in previous MDL-based methods (for other tasks
instead of CDE) 22 36 58 57 28 In contrast, the parameter C' can be used express the prior belief about the hierarchical structure of the search space, as further discussed in Appendix C..3
Encoding the histograms. One subtle choice made is set the boundaries for histograms all leaves to be the same, and we set them as the global boundary of the target variable. This avoids unseen (test) data points falling outside the boundaries of the histograms is because while it may be common to assume that the boundaries for the histogram are known in unconditional density estimation, assuming the same for CDE is hardly realistic.
the leaf nodes. This
Therefore, the code length needed to encode the boundary L((Br, B.)) Eq. a constant that does not affect the model selection result. Consequently, it suffices to encode the number of bins for each histogram: for the histogram on the kth leaf with by bins, costs I (hε) bits by Rissanen's
integer code 42
4.4 Model selection criterion
Combining Eq. 12 and together with the results from Proposition I MDL-score as our final model selection criterion:
we present the following
M = arg min MEM
- -log2 max PM. (M<l))/²²))
+
Σ kE[K]]
loga R(N,, (4) + I(M)
(4)
5

---

which has the form of the regularized maximum likelihood, yet without the need to tune the regu- larization hyperparameter. Note that the exact form of L(M) depends on the types of the feature variables; e.g., by assuming all variables are continuous, we obtain
L(M) = Hlgg Ck-It
Σ (log2(m) + LN(d) log2 + + 1)+ Σ LN(he), JEIK-1]
KE(K]
where, as defined previously, R()) denotes the regret term of each histogram, (.) the Rissanen's integer universal code [42], the number of feature variables, dj the granularity level for the search space for the splitting values corresponding to the jth internal node, the constant that controls the hierarchical structure of the search space of the splitting values, and hy the number of bins for the histogram on the kth leaf node.
Algorithm
While finding the optimal tree- based model with the saanch-ndd-boume approach is possible for regression [59] and classification [18], this does not apply to our task for two reasons. First, our MDL-based regularization term differs from traditional penalty terms based on tree size. Second, our task resembles optimizing model trees [27 38 rather than classification and regression trees, we aim to find the optimal histogram for each candidate split.
Algorithm Learn CDTree from data
Input: Training dataset /)
Output: CDTree M
M {
// One leaf node only
while True do
for S E M do
Search the condition that splits S into two nodes and minimizes the MDL-score (Eq,
// Described in detail in Algorithm end
Splitting any M cannot further decrease the MDL-score then
return CDTree M
else
Among all S E M, find the single S* to be split that minimizes the MDL-score. Update M by replacing S* with its two child nodes that minimize the MDL-score. end
10
end
greedy approach for tree construction. We thus take a heuristic approach to optimize our MDL- score in Eq. As summarized in Algorithm [ start with a tree with one leaf node M = { ): next, we iteratively update M by replacing one of the leaf nodes with its 'best' two child nodes. Specifically, to achieve the lowest MDL-score at each iteration, we simultaneously search for 1) which node to split, 2) the splitting condition for that node, and 3) the optimal number of bins for the histograms. That is, we iterate over all leaf nodes in M; for each leaf node, search for the splitting condition, along with the number of bins for the histograms on the (potential) child nodes, which whole minimizes the MDL-score. Notably, while exhaustive search for the 'best' models on all potential child nodes is considered infeasible in traditional model tree methods 27 38), we empirically demonstrate Section 6.5 that our algorithm is comparable to KDE-based methods.
Finding the child nodes. We next elaborate on the search for the tree-splitting condition at each iteration, for which the pscudo-code is provided in Algorithm 2 For simplicity, assume that all feature variables are either continuous or binary (i.e., categorical features are one-hot encoded in our implementation).
Further, we iterate over all columns the feature matrix. For each column, we start with the granularity level d = and generate candidate split points as the C. 2d-1 quantiles that lead to equal frequency binning of the values. Note that the quantiles are generated based on the subset of data points covered by the node to be split, rather than the entire dataset. We search for the best split point

---

at the fixed d that yields the minimum MDL score among all generated candidates, We then proceed the next granularity level 1 and repeat the process, which stops if no split point in the next granularity level results in better (smaller) MDL. score.
Last, we also need to search for the optimal number of bins for the histogram that leads to the minimum MDL score, which we defer to Appendix
Algorithm 2 Find the best split for node S
Input: Training set Ds (z(5), y)5) covered by node S, with x5()) (0) ... (i,...mmmm
Tmm having
columns, and constant C'
Output: The splitting condition of S that minimizes the MDL-score
13
for (1, ..... m do
14
d+1
while True do
16
candidate_splete + The C- C-24-1 quantiles for equal-frequenc, binning for Z.
17
for s E candidate splits do
18
Di +- (z. E Dsla S 8} Ds|ay > 8}
// Always 8= 1/2 for binary features
19
D, +- ((x,
20
Construct histograms for Dj and D, that minimize the MDL-score conditioned on the // Described in detail in Algorithm
fixed j and d. :
21
Score +- Calculate the MDL-score assuming Ds is split into D; and D.
end 22
23
if The best Score is worse than that of the previous d then Break; else d +
end
25
Record the best tuple (d, for this column index
26
end
return The tuple (j, d, s) that minimizes the MDL-score
Experiments
We present our experiments to demonstrate the empirical performance of CDTree from various perspectives. Specifically, we aim answer the following research questions: 1) Does CDTree provide more accurate conditional density estimation compared to existing interpretable (tree-based and kernel-based) methods? 2) As proxy for interpretability, does CDYTree have smaller tree sizes in comparison to other tree-based models? 3) Is CDTrree robust against irrelevant "noisy" features? 4) Are the runtimes of our algorithm comparable those of kernel-based methods?
6.1 Experiment setup
We use 14 datasets with numerical target variables from the UCI repository [. These datasets, summarized in Table [ cover wide range of sample sizes and dimensionalities. We benchmark our method against a variety of competitors, with all results obtained on the test sets using five-fold cross-validation Our competitors include 1) NKDE and CKDE 44 46), which are based on kernel density estimation, with bandwidth and (for NKDE only) tuned by cross-validation; 2) LSCDE (54), which directly estimates the ratio of joint and marginal density using linear models with Gaussian kernel basis functions; 3) tree-based method CADET 6 which fits a Gaussian distribution on each leaf node; 4) two more tree-based baselines introduced by us, CART-K and CART-h, which respectively fit a KDE model and a histogram after CART regression tree [5) is learned from data.
Furthermore, we compare against three black-box models "upper" baselines, including 1) neural network methods NF 39 and MDN 4. which apply both dropout and noise regularization [45), and 2) the recently proposed tree boosting method LinCDE [13.. Nonetheless, goal with CDTree not to be more accurate than black-box models, but to introduce a CDE method that is both interpretable and accurate.
For reproducibilltty we provide further details about implementation and parameter choices in Appendix C We made source code public:
7

---

Table 1: Datasets from the UCI repository, with the numbers of rows and columns
datasct
energy 9568
synchronou
localizati
toxicity
concrete
slump
foresthiee
# rows
557
107
908
1030
103
517
# cols
5
5
7
9
10
13
datasct
navalpropo
skillcraft 3338
sm|2010
thermograp
support2 9105
studentmat
supercondu
# rows
11934
2764
1003
395
21263
# cols
17
19
21
26
27
44
82
Table 2: Negative log-likelihoods (smaller is better) on test sets. The best results among interpretable methods shown in hold, and the best results among all interpretable and black-hox models marked by underlines. The datasets are ordered by their numbers of columns (ascending).
Interpretable models
Black-bol models
Datasets
CADET
CART-
CARE-K
CKDE
LSCDE
NKDE
Ours
LieCDE
MDN
NF
energy
3.55
3.09
3.06
2.47
3.38
3
2.93
2.93
2.78
2.86
synchrono
-2.93
-1.63
-1.86
-3,59
-1.25
-1.57
-2.11
-1.85
-2.94
-2.64
localizat
-0.23
-0.55
-0.01
-0.26
-0.61
-0.28
-0.66
-0.95
-0,68
-0.43
toxicity
1.8
1.5
1.38
1.32
1.34
1.55
1.53
1.29
1.24
1.23
concrete
4.17
3.75
3.93
3.32
3.66
3.91
3.72
3,47
2.97
3.18
slump
3.42
3.55
3.43
2.35
2.91
3.08
3.34
2.98
2.23
2.39
forestfir
134
3.96
4.39
4.85
4.68
5.55
3.43
4.35
3.26
3.23
navalprop
-3.53
-3.3
-3.66
-2.8
-2.88
-3.19
-3.6
-3.36
-4.12
-3.75
skillcraf
94.4
0.46
-0.42
1.54
1.61
1.56
-1.02
1.26
0.35
1.11
sml2010
6.52
2.85
2.89
1.61
3.14
3.12
2.7
2.97
2.15
2.61
thermogra
2.21
0.66
0.72
0.66
0.94
0.94
0.64
0.59
0.56
0.52
support2
97.3
0.51
0.32
2.09
2.46
2.13
0.29
1.48
1.53
1.24
studentma
3.83
2.65
2.66
2.89
4.19
3.11
2.66
2.59
3.85
3,54
supercond
9.6
3.84
4.36
4.55
4.17
4.19
3.48
3.87
3.33
3.5
rank (all)
8.79
5.68
6.04
5.11
7.46
7.68
4.46
2.57
3.21
rank (intp.)
6.07
3.43
3.86
3.14
4.57
4.86
2.07
Conditional density estimation accuracy
In Table 2
report the average negative log- log-liielihoods (NLL) the test sets, which can be regarded as an approximation to the expected log-loss E log(u/<:)) the standard loss function for measuring CDE accuracy.
Comparison to interpretable models. The NLL of the CDTree models are the best (lowest) in 6 out of 14 datasets among all interpretable models, which are shown in bold in the table. Further, among all interpretable models, our average rank is the best, reported in the (last row) of Table As the datasets are increasingly ranked based on their dimensionalities, we observe that CKDE also achieving the best NLL in 6 datasets-only performs well on datasets with low dimensions. Additionally, other two kernel- methods LSCDE and NKDE have in general worse accuracy.
Further, CDTree has better accuracy (lower NLL) than all free-based competitors on almost all datasets (13 out of 14 datasets for CADET, and 12 out of 14 datasets for CART-h and CART-k). The superiority of CDTree highlights the advantages of 1) using non-parametric histogram models, and 2) conducting exhaustive search for optimal histograms for all node splits when learning the tree structure, rather than fitting the model after the tree structure is fixed, as in CART-h and CART-k.
Comparison to black-box models. Neural network models generally exhibit better accuracy than interpretable models, as indicated by their average ranks. However, their non- non-transparency limits their applicability in critical areas. We argue that studying interpretable models and introducing CDTree paves the way for developing local surrogate models 40) 26]. an essential approach for generating post-hoc explainability for black-box models,
such method currently exists for CDE.
Further, the performance of CDTree is surprisingly on par with that of LinCDE. Since CDTree is based on single tree while LinCDE is a tree ensemble model, we conjecture that the on-par performance caused by the fact that CDTree adopts non-parametric histograms whereas LinCDE takes parametric approach (with much more flexible model class than Gaussian though).

---

We also report the standard deviations of NLL for all methods in Appendix D.I The standard deviations for our method remain low, indicating stable performance across different datasets.
CADET
CART
Ours
CXDE
LSCDE
NKDE
Ours
looves (tag10)
1000
Aandno (ceoordia)
100000
100
10-
conevete
energy
lorestiie
looaizau
nevalpropo
akiliran
dummp smiti
iupporti
gynetronod
teemsggra toxccly
concrete
energy
foresiire
locaizali
navejeropo
aclicnift
elamp
emizi10
studentraal
superiznnii
supportt
pprtorronu
thermogi ap
Eoxcity
Figure 2: Left: the number of leaves for tree-based methods. Right: Runtimes of CDTree and kernel-based methods. Note that the y-axes are scaled by logio(-)
6.3 Complexity of trees
For tree-based models, the size of the trees is an important proxy for the degree of interpretability 29 We hence compare the number of leaves of CDTree against the other tree-based models CADET and CART (note that CART-h and CART-K have the same tree structures). We demonstrate in Figure (left) that CDTree has smaller tree sizes than the competitors on most datasets.
Noley features: Gaussian
Notey features:
added teatures
Randomiy selected feature Gaussian
Number of splits on acded features
400-
800
200
10
20
100
Algorithm
GADET
concrete
energy
foresttiee
locatizoo
navelpropo
sellbora
alomp
6miz010
studentraal
support2
gynchroncu
themsgrap
tocisty
conertte
energy
foresttive
locelicao
nevalpros
pkilzzatt
stump
6mi2010
studentinat
soperconati
aupport2
syndhsonat
themsgrap
toucty
CART
Ours
Figure 3: Number of internal nodes with split conditions that contain irrelevant features. The y-axis is scaled by the squared-root for better visualization.
6.4 Robustness irrelevant features
We next investigate whether CDTree is robust against irrelevant features, which is particularly important for the interpretability of tree-based models, since the (intrinsic) explanation contained in the CDTrre would not be trustworthy if such robustness did not hold.
Specifically, we generate 'noisy" features two different ways. The first way is add w irrelevant features randomly drawn from the standard Gaussian distribution to each dataset. The second way is to first randomly select w features from each dataset; then, for each selected feature Xj, we generate "noisy" feature by adding a Gaussian noise to it, i.e., X X; + N (0, s(Xr)/2), where s( s(X,) denotes the estimated standard deviation. We refer to the irrelevant features generated by the first (second) approach independent (dependent) noisy features. where E {3, 5, 10, 20) in both cases. Note that adding X to the dataset does not change the true conditional density as the target variable is conditionally independent of X given the original feature X,.
Next, we train the tree- based models on expanded datasets that include irrelevant features. We count the number of nodes with splitting conditions that involve these added irrelevant features. As demonstrated in Figure 3 the number of splits on irrelevant features for our CDTree is almost always zero for both ways of generating noisy features. contrast, the bree- models learned by CART and CADET contain many more nodes with irrelevant features.

---

on4rgg
4.0
3.0-0:0
Nagstive Lag-ikalhesn
9.5-
-04
-06
15
sispert
Enerroppp
Algontihm
NADE
CARTT
10
15
15
20
10
20
Number added feetures, which are generated by Randoniy Se ected Feelures Gevasian dish buuions
Figure 4: Negative log-likelihoods with different number of irrelevant 'dependent noisy' features. The results of CDTree (shown in blue lines) are stable on all datasets.
We also examine whether the negative log- log-likelshoods (NLL) remain stable when irrelevant features are added. For 'dependent noisy' features, the results are shown in Figure 4 where the NLL of CDTree remains nearly identical regardless of the number of added features- -te only exception being the dataset "slump" (with only 103 rows). In contrast, the NLLs obtained by competitor methods are much less stable: the NI.L of NKDE varies for most datasets, and CART-k shows visible changes in 6 out of 14 datasets. Similar results are observed when 'independent noisy" features are added, as shown in Appendix D.2
6.5 Runtimes
The idea of fitting separate models on the leaves of decision tree has existed for a long time (38] Nevertheless, is (still) often believed that fitting separate models for all possible node splits when growing a tree is infeasible. However, we show in Figure 2 (right) the runtime of CDTree and demonstrate that its runtimes are in general lower than the runtimes of CKDE (which requires intensive parameter tuning). While NKDE and LSCDE are in general faster than CDTree, the accuracy of their conditional density estimates are sub-optimal, as discussed previously. We exclude the comparison with other tree-based methods whose implementations are based on CART, as CART highly optimized and known to be extremely fast. Further, as these tree-based methods either model the conditional means only (CART-h and CART-K) or assume Gaussian model (CADET), they have far smaller search space, and hence are fast but less accurate.
7 Discussion
In this paper, we studied the interpretable conditional density estimation (CDE) models. Motivated by the fact that free-based methods are arguably more interpretable than kernel-baser methods yet have been largely disregarded for interpretable CDE, we introduced the Conditional Density Tree (CDTree)). We formalized the learning problem under the MDL framework, proposed an iterative algorithm, and demonstrated its competitive empirical performance
wide range of datasets.
Limitations. As histograms are used in the CDTree, we implicitly assume that the support of the target variable is bounded. Hence, the boundaries for the histograms need to be chosen in an ad-hoc way, possibly based on prior knowledge. Further, in practice, may happen that the unseen data points fall outside the histogram boundary, for which the predicted (conditional) density will be O, unless the full model is re-trained. We realize that this may cause some issues when using CDTree in practice, but meanwhile, we argue that 1) this is a limitation of the histogram model itself, 2) similar issues could happen in modelling quantiles by the empirical cumulative probability function, as in well-known methods like quantile regression trees and forest, and 3) all probability models have their own (implicit) assumptions on the probability tails, including those specifically designed for modeling the extreme value distributions 9 23
10

---

Acknowledgement
We thank the anonymous reviewers who have provided constructive and valuable reviews. We also thank Prof.dr. Alexander Marx (TU Dortmund) for his feedback in developing the initial idea.
References
[1] The uci machine learning repository. URU/atpp:://rrchiieeeeee.. edu/
[2] K. B. Athreya, P. E. Ney, and P. Ney. Branching processes. Courier Corporation, 2004.
[3] N. M. Ball, R. J. Brunner, A. D. Myers, N. E. Strand, S. L. Alberts, and D. Tcheng. Robust machine learning applied to astronomical data sets. iii, probabilistic photometric redshifts for galaxies and quasars in the sdss and galex. The Astrophysical Journal, 683(1):12, 2008.
[4] C. M. Bishop. Mixture density networks. 1994.
[5] L. Breiman. Classification and regression frees. Routledge, 2017.
[6] C. Cousins and M. Riondato. Cadet: interpretable parametrie conditional density estimation with decision trees and forests. Machine Learning. (08:1613-1634 2019.
[7] T. M. Cover and J. A. Thomas. Elements of information theory. John Wiley & Sons, 2012.
[8] C. DeSantis, J. Ma, L. Bryan, and A. Jemal. Breast cancer statistics, 2013. CA: a cancer journal for clinicians, 64(1):52-62, 2014.
[9] G. Durrieu, Grama, Jaunatre, Q.-K. Pham, and J.-M. Tricot. Extremefit: a package for extreme quantiles. Journal of Statistical Software, 87:1-20, 2018.
[10] V. Dutordoir, H. Salimbeni, J. Hensman, and M. Deisenroth. Gaussian process conditional density estimation. Advances in neural information processing xystems, 31, 2018.
[11] J. Fan. Q. Yao, and H. Tong. Estimation of conditional densities and sensitivity measures in nonlinear dynamical systems. Biometrika, 83(1):189 1996.
[12] E. Galbrun. The minimum description length principle for pattern mining: A survey, Data mining and knowledge discovery, 177 2022.
[13] Z. Gao and T. Hastie. Lincde: conditional density estimation via lindsey's method. Journal of machine learning research, 23(52):1-55 2022.
[14] D. B. Gilleskie and T. A. Mroz. A flexible approach for estimating the effects of covariates on health expenditures. Journal of health economics, 23(2):391-418, 2004.
[15] Grünwald. The minimum description length principle. MIT press, 2007.
[16) P. Grünwald and T. Roos. arXiv: /908. 08484, 2019.
Minimum description length revisited. arXiv preprint
[17] Hall, J. Racine, and Q. Li. Cross-validation and the estimation of conditional probability densities. Journal of the American Statistical Association, 191468):1055-1026, 2004.
[18] X. Hu, C. Rudin, and M. Seltzer. Optimal sparse decision trees. Advances in Neural Information Processing Systems, 32, 2019.
[19] J. Jeon and J. W. Taylor. Using conditional kernel density estimation for wind power density forecasting, Journal of the American Statistical Association, 107(497):66-99, 2012.
[20] P. Kontkanen and P. Myllymäki. Analyzing the stochastic complexity via tree polynomials. Unpublished manuscript, 2005.
[21] Kontkanen and Myllymäki. A linear-time algorithm for computing the multinomial stochastic complexity. Information Processing Letters, 103(6):227-233, 2007.

---

[22] P. Kontkanen and P. Myllymäki. MDL histogram density estimation. In M. Meila and X. Shen, editors, Proceedings of the Eleventh Interaational Conference on Artificial Intelligence and Statistics, volume 2 of Proceedings of Machine Learning Research. PMLR, 2007.
[23] S. Kotz and S. Nadarajah. Extreme value distributions: theory and applications. world scientific, 2000.
[24] B. Lantz. Machine learning with R: expert techniques for predictive modeling. Packt publishing Itd, 2019.
[25] J. Lindsey. Comparison of probability distributions. Journal of the Royal Statistical Society: Series B (Methodological), 36(1):38 47, 1974.
[26] Q. Luo and T. M. Khoshgoftaar. Unsupervised multiscale color image segmentation based on mdl principle. IEEE Transactions on Image Processing, 15(9):2755- 2761, 2006.
[27] D. Malerba, F. Esposito. M. Ceci, and A. Appice. Top-down induction of model trees with regression and splitting nodes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(5):612-62) 625, 2004.
[28] A. Marx, L. Yang. and M. van Leeuwen. Estimating conditional mutual information for discrete- continuous mixtures using multi-dimensional adaptive histograms. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pages 387-395. SIAM, 2021.
[29] C. Molnar. Interpretable machine learning. Lulu.com, 2020.
[30] T. Mononen and P. Myllymäki. Computing the multinomial stochastic complexity in sub-linear time. In Proceedings of the 4th European Workshop on Probabilistic Graphical Models, pages 209-216, 2008.
[31] H. Y. Mussa. The aitchison and aitken kernel function revisited. Journal of Mathematics Research, 5(1):22, 2013.
[32] S. Nikolova, A. Sinko, and M. Sutton. Do maximum waiting times guarantees change clinical priorities? a conditional density estimation approach. Technical report, HEDG, c/o Department of Economics, University of York, 2012.
[33] I. Papageorgiou and I. Kontoyiannis. Posterior representations for bayesian context trees: Sampling, estimation and convergence. Bayesian analysis, 19(2)5501-529, 2024.
[34] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer. R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-288 2011.
[35] T. Pospisil and A. B. Lee. Rfede: Random forests for conditional density estimation. arXiv preprint arXiv: 1804 05753, 2018.
[36] H. M. Proença and M. van Leeuwen. Interpretable multiclass classification by mdl-based rule lists. Information Sciences, 512:1372-1393, 2020.
[37] J. R. Quinlan. C4. 5: programs for machine learning. Elsevier, 2014.
[38] J. R. Quinlan et al. Learning with continuous classes In 5th Australian joint conference on artificial intelligence, volume 92, pages 343-348. World Scientific, 1992.
[39] D. Rezende and S. Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
[40] M. T. Ribeiro, S. Singh, and C. Guestrin. Why should trust you? explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016.
[41] J. Rissanen. Modeling by shortest data description. Automatica, 14(5):465-471, 1978.
12

---

[42] J. Rissanen. A universal prior for integers and estimation by minimum description length. The Annals of statistics, 11(2):416-433) 1983.
[43] T. Roos, T. Silander, Kontkanen, and P. Myllymaki. Bayesian network structure learning using factorized nml universal models. In 2008 Information Theory and Applications Workshop, pages 272-276. IEEE, 2008.
[44] M. Rosenblatt. Conditional probability density and regression estimators. Multivariate analysis 11, 25:31, 1969.
[45] Rothfuss, F. Ferreira, S. Bochm, S. Walther, M. Ulrich, T. Asfour, and Krause. Noise regularization for conditional density estimation. arXiv: 2019.
[46] J. Rothfuss, F. Ferreira, S. Walther, and M. Ulrich. Conditional density estimation with neural networks: Best practices and benchmarks. arXiv preprint arXiv: 1rxXiv:/93.000 003.00954, 2019.
[47] S. Samani, R. Stadler, C. Flinta, and A. Johnsson. Conditional density estimation of service metrics for networked services. IEEE Transactions Network and Service Management, 18 (2):2300-2364, 2021.
[48] S. Seabold and J. Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.
[49] Y. M. Shtar'kov, Universal sequential coding of single messages. Problemy Peredachi Informat- sii, 23(3):3-17, 1987.
[50] R. Shu, H. H. Bui, and M. Ghavamzadeh. Bottleneck conditional density estimation. In International conference on machine learning, pages 3164-3172. PMIR, 2017.
[51] T. Silander, J. Leppä-Aho, E. Jääsaari, and T. Roos. Quotient normalized maximum likelihood criterion for learning bayesian network structures. In International conference artificial intelligence and statistics, pages 948-957, PMLR, 2018.
[52] R. P. Stanley. Catalan numbers. Cambridge University Press, 2015.
[53] E. V. Strobl and S. Visweswaran. Dirac delta regression: Conditional density estimation with clinical trials. In The KDD'21 Workshop on Causal Discovery, pages 78-125. PMLR, 2021.
[54] M. Sugiyama, I. Takeuchi, Suzuki, T. Kanamori, H. Hachiya, and D. Okanohara. Con- ditional density estimation via least-squares density ratio estimation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 781-788. JMLR Workshop and Conference Proceedings, 2010.
[55] B. L. Trippe and R. E. Turner Conditional density estimation with bayesian normalising flows. arXiv preprint Kiv:180220440
2018
[56) P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski,
P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, F. Jones, R. Kern, F. Larson, C. J. Carey, i. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde. Perktold, R. Cimrman, I. Henriksen, A. Quintero. C. R. Harris, A. M. Archibald, A. H. Ribeiro. F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing Python. Nature Methods, 17:261-272. 2020. doi: 10.
[57] I.. Yang and M. van Leeuwen. Truly unordered probabilistic rule sets for multi-class classi- fication. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 87-103. Springer, 2022.
[58] L. Yang, M. Baratchi, and M. van Leeuwen. Unsupervised discretization by -wo-dimensional mdl-based histogram. Machine Learning, pages 1-35, 2023.
[59] R. Zhang, R. Xin, M. Seltzer, and C. Rudin. Optimal sparse regression trees. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11200-11279, 2023.
13

---

A Proof
Proposition 1. Let 0 (a ...,...akk be the histogram parameters for histograms all leaves, and let 9 arg maxş PMr (y" (="). Then Sy maxy Par,o(y"(=") =[[e[K]] RRN hg), which RNN,, hy.) is the regret (denominator) of the NML distribution of the histogram model on the kth leaf node that contains N& data points and h2 bins.
Proof. Consider the dataset = (x".y" } and a CDTree with K leaf nodes M = {S1,.... Skh each leaf S₁ equipped with histogram model H& (Bi, Bus hy), in which h2 denotes the number of bins. Denote the histogram density estimator associated with each H3 as fHz (.), and short f.). which contains a family of distributions parameterized by a Further, denote the subset of data points that end up the kth leaf as D2 = (a:(k), ((+)) = [(x, v) Dj E Sa.. Thus,
max PM, PM,o("((2") [ y
f max KE[K]
II
II max fo (y (2otk)
kE[R]
[ yyl)) yla)
(Fubini's Theorem)
II max Sa (x)(k) (xx(
(5)
ylk) RE(K]
at
[ max
(re-arranging)
II kE[K] yln)
II R(Ng,h) kE[K]
because the NML distribution for D2 given the histogram H3 is equal to 22
max far (y (+))
Ph: (y (2
(6)
Syt max,+ for (y [x[]]]
Further, it has been shown that the denominator (regret) of the NML distributions for histogram models a function that only depends on N₁ and h4 [22]
B Algorithm Details for Optimizing Histogram Bins
We further discuss the detail process of finding the number of histogram bins that minimizes the MDL score, for which the pseudo-code is provided in Algorithm
While the search space for the number of bins, denoted as h, for the histogram can range from to the number of data points, this is computationally expensive in practice. To mitigate this, first narrow down the range of h by fixing a step size and searching among h E {1. + 1, 2g + 1....). The MDL-score will initially decrease we increase the number of bins, since the the goodness-of-ft of the histogram increases. However, when the number of bins become too large, the MDL-bbaedd regularization terms will start dominate, and hence the MDL-score starts increase.
Assuming the MDL-score starts to increase at h = eg + 1 (where e is a positive integer), we can then narrow the range to (e 2)g < h < eg + 1, since the maximum so far is reached at (e 1)g + 1.
Finally, we iterate over all h within this narrowed range, and we select the number of bins with the smallest MDL score.
C Full Experiment Details for Reproducibility
C.1 Experiment compute resources
The runtimes reported in Section 6.5 all algorithms are recorded on the CPU machines with the AMD 7702 cores.
14

---

Algorithm 3 Learn the MDL-optimal histogram
Target values و covered by leaf node S, step size Output: Histogram H that minimizes the MDL-score
Input:
h' < beeb_sssorr +00
29
while True do
30
mdll calculate the MDL-score given h' if mall < best_scor then
31
32
best cest_score-o mdl
33
N +
34
else
break end
36
37 end
bound low +- (1, 22g)
1/ Since h 29 may be smaller than
39 bound sandddhhhhee + h'
h" <- the optimal within the range bound low boonddoudii high that minimizes the MDL-score
41 return ht
C.2 Data preparation
We noticed that duplicated values in datasets may cause some of our competitor methods to fail. For instance, for those methods that adopt the Gaussian kernel, duplicated values lead to the standard deviation estimated as 0 for some subsets of data points). Thus, we add a very small noise generated by N (0, 10-3) all data columns, which are used for all methods.
C.3 Parameters for CDTree
The parameter C. We first discuss the parameter C 'hhtt controls the hierarchical structure for the search space of the splitting values, for which we C in our experiments.
The parameter C can be used to express a prior belief on the candidate splitting values. For instance, when C = 5, the code length needed for encoding the quantiles when d 1 are the same. Equivalently, the prior probabilities of these quantiles are the same.
As these 5 quantiles divide the values into 6 subsets, essentially, are saying that the first quantile among these 5 quantiles, i.e., the 1 1/--quantiee, and the third quantile, i.e., the 1/2-quantile (3/6- quantile), have the same prior probability.
However, if we set C 1, there will be only quantile when 1, i.e., the 1/2- //2-quantile, In this case, the prior probability for the 1/2- 1/2-quantile will be different (larger) than the 1/6 1/6-quantile.
Thus, the parameter C can be used to express the prior belief in the splitting values. That is, ask herself that, is it equally likely, a priori, to observe a node containing the splitting value equal to the 1 1/100-quantile, and to observe node containing the splitting value equal to the 1/2-quantile (i.e., the median)? If the answer is yes, C' might be set as 99. Otherwise, a smaller C' should be considered.
The step size g. We set g= = 30 searching the number of histogram bins in Algorithm Note that if we set very large, we only need very few iterations to "narrow" down the range for the number bins for histograms; nevertheless, the resulting range will not be very narrow in this case. On the other hand, if we set very small, it may cost a lot of time to obtain the narrower range. In that sense,
30 seems a rather balanced choice.
The histogram boundary. Further, as discussed Section we assume that range of the target variable for each dataset is known. Thus, we set the global range of the histograms based on the range of the full dataset (before train/test split the cross-validation) plus/minus small constant, chosen 1
15

---

C.4 Details for competitor algorithms
We use the implementation of CKDE, NKDE, LSCDE, NE, and MDN from from the Python package "cde"" 46) and the implementation of LinCDE and CADET from the original authors.
CADET. No specific guidelines for regularization is provided in the original paper [6]; further, in the author's original implementation, the standard regularization based on the tree size is not available. Instead, only choose the regularization mode in 'constant', 'AIC', and 'BIC'. We pick the 'BIC' as behaves similarly to MDL asymptotically [5
The bandwidth is tuned by optimizing the cross- validation maximum likelihoods, which is implemented in 'statsmodels' [48, the standard Python package for multivariate statistical analysis, and directly used in the 'cde" package [46]. For the three largest datasets ( ('support2', 'navalprop', and "supercond"), we fail to obtain the cross-validation tuning results within 10 hours for a single fold, and hence we adopt the 'normal reference" for the bandwidth selection.
CKDE.
Note that it required to specify the range for searching the bandwidth, as the implementation is essentially based on the "optimize" function (for continuous optimization) in the well-known Python package 'scipy' [56]
NKDE. The bandwidth and the c (the parameter that controls the neighbor range) are tuned by optimizing the cros-validatiio maximum likelihoods, which is implemented in the Python package 'cde' [46]. Similar to CKDE, the search space for these two parameters are not required. For two very large datasets (" ('support2" and 'supercond") ) we fail obtain the cross-validation tuning results within 10 hours for single fold, and hence we adopt the 'normal reference' for the handwidth selection.
CART-K and CART-h. In these methods, CART regression tree is first trained, with the regulariza- tion hyperparameter for pruning tuned by cross-validation. After fixing the tree structure, a kernel density estimation model and a histogram model are fit to the subsets of data points each leaf node for CART-K and CARThh, respectively. The bandwidth for the former is tuned by cross-validation. while the number of bins for the latter is picked to optimize the MDL score (given the fixed tree structure). Specifically, we use the CART implementation from the scikit-learn (34] Python package.
LSCDE. No tuning for bandwidth is discussed in the original paper, nor implemented. Thus, we stick to the default setting.
LinCDE. The tree depth is set as 3, which is common setting for training boosted tree models.
NF and MDN. As suggested by Rothfuss et al. 45), we 'noise regularization' for both NF and MDN. Specifically, we choose the standard deviation of added noise for both the features and the target as 0.01. We further noticed that adding the standard dropout with dropout rate equal 0.1 gives more stable results. Other hyperparameters (e.g., the number of hidden layers) are kept as default.
D More experiment results
D.1 Standard deviation of negative log-likelihoods
In Table we report the negative liglliielihoods together with the standard deviations obtained by the five-fold cross-validation. In comparisons to other competitors, we observe no anomalously larger standard deviations for our method.
D.2 Robustness against irrelevant features
In Figure 5} we show the results that demonstrate the stability of the negative log-likelihoods when independent gaussian features are added to the 14 UCI datasets, in which we observe similar results as in Figure 4 i.e., our proposed method CDTree extremely stable across all datasets except for the very small dataset 'slump'.
16

---

Table 3: Negative log-likelihoods (smaller is better) on test sets, together with the standard deviation,
obtained using five-fold cross-validation.
Imerpestahic madels
Blick-hox muskis
Gainse
CADET
CARTA
CARTK
CXDE
LSCDE
NKDE
Chu
LACDE
MDN
energy
(.25
(.09
3.06
2.47 (0.02)
3.38 (0.01)
(0005)
2.03 (0:01)
(00))
2:78 (002)
2,56
synchranou
(0.24)
-1.63-0..7)
-1.86 (0.1)
3359 (0,09)
-1.25 (0.02)
-1.57 (0.04)
-2.11 (0.02)
-1.33 (0.03)
2.94 (0.05)
-2.64 (0.15)
localizati
0.23 (0.36)
(0.29)
0.01(1.63)
0.26 (1.79)
0.61 (0.76)
0.28 (0.95)
0.66 (0.27)
-1.9 (0.11)
0.58 (0:46)
-0.43 (0.95)
loticity
1.8.0..19
(.500
1.38 (0.09)
1.32 (1:45)
(0.02)
155 (0.08)
1.53 (0.19)
1.29 (0.06)
1.24 (0.08)
1.23 (0.07)
concrete
4.17 (0.7)
(0.0)
3,93 (0.57)
3.32
3,66 (0.03)
391 (0.07)
3.72 00.06
347 (0.03)
2.97 (0.06)
(..1
3,55
(0.2)
2.91
3.0
2.39
forexifire
20)
(1.9(
5.55
3:23 5.23(0074)
rovalpcupo
3.66 3.3:00.44)
(0)
(0.01)
(0.06)
4.12
(02)
skilkratt
94.42
0.46 (1.7)
.0.42 (8.53)
(1:01)
(0,02)
1.56 (0:11)
-1,02 (0.13)
(0.1))
(1.26)
1.11
smi2010
(2,6))
2,85 (0:05)
(0.17)
3.14 (0.91)
3.12
(0.06)
(0102)
2:15 (0,02)
266 (0.17)
themograp
(0.62)
(0.05
0.72 (0:35)
(0.06)
0.94 (0:05)
0.94 (0.05)
(0.07)
(0.06)
(.52(000)
suppin2
(81.6)
0.51 (0.04)
0.32 (0.06)
2.09 (0.01)
2.46 (0.06)
2.13 (0.02)
0.29 (0.04)
1.48 (0.07)
1.53 (0.86)
(022)
modestaat
3.83 (0.399)
2.65 (0.05)
2.66 (0:05)
2.89 (0.05)
4.19 (0.41)
3.11 (0.11)
2.66 (0.17)
2.59 (0.05)
3.85 (0.34)
3.54 (0336)
seprooondo
(.773
3.84 (0.01)
4,36 (0.14)
4.55 (0)
4.17 (0.02)
4.19 (0.15)
3.48 (0.02)
3.57 (0.02)
3,33 (0.0))
(,00
00:10
eT4rg
Icsecthe
10 Irat
2100
-3.25
-2:50
Silort
00120100
Negette Log shghhho
3.40
Algorthm
stadertiaal
supercondu
suppor2
sensfronce
CART
5.0
45-
35
Stemoprap
todiity
1.0
07
10
10
15
Numbar added features, which Are generated indepenndant Gaussian distriuutoos NJ0,1)
Figure 5: Negative-log likeliiiisssodss with different number of added features, which are generated by independent Gaussian distributions. The results of CDTree, shown in blue solid lines, are extremely stable on all datasets expect for the very small 'slump" dataset.
E Medical costs visualizations
We show in Figure 6 the full results of applying CDTree to the medical expenses dataset used in Section D As sanity check for the goodness-of-fit, we report the average cross-validation negative log- likelliod (on test sets) for CDTree is 9.03 (with the standard deviation 0.13), which slightly better than that of the black-box boosted tree model LinCDE, with the average negative log- ligliiklihood 9.55 (standard deviation 0.09). Each Histogram corresponds to a single leaf that describes a meaningful subgroup of patients, in which we observe 'age", 'BMI' and 'smoker' are the most distinguishing features to characterize each subgroup. Last, although the dataset contains features and only 1338 samples, the illustrations show that CDTrre does not require a very large dataset to reveal meaningful and comprehensible information.
17

---

Uncondikoral denaty
24 5 2
47 6 0.00 9F
1go 54.57 57 s.g bm 30
aroosst hasse
E1i1 30 smoker Tius
356
100-
50-
20000
40000
50000
200C0
40000
50000
20000
-0000
50000
20000
-0000
50000
medical oost
medicaal 0251
medical 0:42
mecical cosl
c:39 & brr-c:30.9
60 A smolor- Fa
51
35 & De 25
54 47
Smo1or= Tlue
c1i0ten x 25 6 SmoOeY Fabte
chidren x 28 smoksr Faze
00-
20
60-
12-
30-
26-
vocco
00001
2000C
400CO
20000
40000
20000 medical
madical
bmi 5
30 5
92
28 5
2/a//er Thue
s1akker TIUM
S1rcker = Faline
40-
30-
30
20-
10-
20005
6000G
2000
20000
50000
20000
1ge
1ge
1ge- age
2m0122 50
Cnidion x ampkir - Fase
C18dicn & smokkr Fabs
Smokeer Tru
30-
40-
15-
20
0-
20000
400LD
60000
20000
<0000
60000
200000
40.00
60000
20000
6000
madical
medical
medical
medical
500 &
2010/00115w & ertoker
40-
30-
20-
10
2500c medical Cool
ADCCO
KEOOG
Figure 6: All leaves from the CDTrre that describes the conditional density of medical costs, conditioned on demographic features. Root-to-leaf conditions are presented after removing logically redundant conditions.
18

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] " "provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our mains claims and the paper's contributions are both 1) to introduce tree-based CDE method, and 2) we show that our proposed method is both more accurate (as measured by the log-loss), has smaller tree sizes, and is robust against irrelevant features.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have separate 'limitation' subsection within the Discussion Section in
Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
19

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-texl system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: our proof has no additional assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Full descriptions provided both in the main context and in the appendix. Also, the source code provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
20

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The source code is provided. The datasets are all available on the website of the UCI repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guides/Codest wttpess/oooeSuummmssooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ mnips.cc/public.//
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
21

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Full descriptions are provided both in the main context and in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: we report the standard deviation of the negative log likelihoods, obtained by five-fold cross-validation, in Appendix D.I
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma eтror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the compute resources in Appendix C.I
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
22

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses the positive societal impacts about 1) interpretable CDE methods have advantages in critical domains, 2) CDTree is more interpretable than kernel- based methods because it is also interpretable to the general public without Statistics knowledge. As far as we are concerned, our method does not have the risk of being misused in a malicious way.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
23

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and code free for research purposes according the licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, phcorswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Source code is provided, together with the details of how to get the code running.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "HvkB00Q50S", "reviewer": "Reviewer_QurN", "summary": "This paper proposes a new conditional density estimation algorithm based on histogram trees. \nThe base model corresponds to a full binary tree, where each internal node is associated with a split of the feature space in one coordinate, and each leaf node is associated with a histogram density estimator for the response variable for the feature variable falling into the leaf node.\nTo find the best among all possible models that best fit the data, the proposed algorithm invokes the MDL principle, which avoids any hyperparameter tuning, to derive the model selection criterion.\nSince the optimization is infeasible to be solved exactly over all possible models, the paper proposes a greedy algorithm.\nExperiments support that the resulting estimator CDTree is competitive against the state-of-the-art models.", "questions": "**Suggestions**\n- The definition of a conditional density tree with histogram in Section 3 seems to be incomplete. I was able to understand the complete model after I read Section 4.3. I believe that this should be properly defined as a model in Section 3, since without a proper definition Section 4.1 and 4.2 read awkward.\nFor example, what is the dimension of $x$ and do you assume $y\\in\\mathbb{R}$? Somehow this is not explicitly defined.\n- And I think the sentence in lines 108-110 is quite confusing, in the sense that while the model $M$ may depend on the dataset $D$, a covariate $x$ needs not be from the dataset to evaluate the conditional density.\n- In Section 4.1, the definition of MDL optimal model is rather abrupt as the \"universal model\" is defined only in Section 4.2. I believe that the NML code should be defined before (1) or at least there should be a pointer to (2) in that paragraph. The notation $P_M(.)$ is also misleading. Please consider $P_M(\\cdot | \\cdot)$.\n\n**Questions**\n- Why did the authors define $\\hat{\\theta}$ in Proposition 1 while it is not used?\n- Why are CKDE and NKDE put under \"interpretable models\"? I was confused as the authors emphasized at multiple points that kernel-based methods are not interpretable.\n- A silly baseline is to consider a Gaussian density estimator as CADET assumes in the proposed MDL criterion of CDTree. This would result in a faster algorithm as the search space is simple. I am curious how this simple baseline would work in practice, as this can help understand the benefits of using the flexible histograms in place of Gaussians and using MDL principle separately. I would appreciate additional experiments, if time permits.\n- (Minor) Regarding the discussion on runtime: Can you provide a bit more quantitative remark, is possible? For example, the authors mention that CART is highly optimized while the proposed algorithm is not. In which part of CDTree could be more optimized?\n- Please revise the bibliographic info. For example, \"mdl\" in the title of [41] should be capitalized.", "limitations": "The authors note its limitation regarding the boundedness assumption of target variable.\nOn top of that, the algorithm is greedy and there is no guarantee on closeness between the actual estimator and the optimal estimator defined by the learning criterion.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a nice algorithm for an important problem based on basic principles. This seems to be a nice practical application of the MDL principle.\n- Though the resulting estimator is a single tree-based conditional density estimator, which is inherently \"interpretable\" thanks to the tree structure, experimental results show that CDTree is competitive against the SOTA algorithm based on ensemble. This is quite surprising.\n- Experimental results are sufficiently thorough to understand the pros and cons of the proposed method.", "weaknesses": "While I like the paper overall, the manuscript is not spotless, especially in terms of presentation, grammatical errors, and typographical mistakes.\nI believe that after a careful revision of the paper, this would be a nice addition to the community. (Though the fit might be much better with an applied statistics journal.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the detailed, constructive, and very helpful comments on my writing! \n\n## Presentational issues fixed\n\nI have fixed the presentational issues and typos you mentioned. Among others, one extremely helpful comment is that \"lines 108-110 are confusing\": in these lines I wrote \"... $f (y|x)$ as $f _ k (x)$ ... \", but it should actually be  \"... $f (y|x)$ as $f _ k (y)$ ... \". \n\n## Rebuttal about \"model definition not complete\"\n\nWe respectfully disagree that \"The definition of a conditional density tree with histograms in Section 3 seems to be incomplete\", and you seem to indicate part of the model definition is in Section 4. However, Section 3 contains all information about how to calculate the probability/likelihood of data given a fixed conditional density tree (regardless on whether it is actually a good model), which is sufficient for defining a probabilistic model. In contrast, Section 4 is all about the definition of the model selection criterion. \n\n## Rebuttal about \"applied statistics journal may be a better fit\"\n\nThe machine learning community has been recently highly interested in both conditional density estimation (CDE) (e.g., the references [9, 37, 48, 12] in the paper are from NeurIPS/ICML/JMLR) and interpretability, yet interpretable CDE methods are neglected. Second, the lack of CDTree, as the counterpart to the regression tree and classification tree, hinders the development of XAI methods (e.g., local surrogate models) for CDE. \n\n## Reply to your questions:\n\n- **Regarding the first and last point**: you are correct. Thanks so much again!\n- **Whether CKDE/NKDE is interpretable model**: CKDE/NKDE (and other kernel-based models) are currently the standard (if not the only) choices if someone is looking for a \"shallow\" model for CDE. We argued in the paper that these models are \"arguably less interpretable to (single) trees\", yet we are also aware that these models may be considered more interpretable than tree ensembles/neural networks. (After all, there is still no precise definition of \"interpretable models\". )\n- **About comparison to the baseline with Gaussian assumption**: The MDL criterion we proposed CANNOT be used for Gaussian assumptions, as it is known that the regret term for the NML will be infinity (see page 298 of the book [14], {Grünwald 2007, The minimum description length principle MIT press, Example 11.1, Chapter 11}). I agree it would be interesting to do some sort of this kind of comparison though, as kind of an ablation study for the histograms and a supplementary to CART-h in the paper. However, the closest related work I can think of is (Proença, Hugo M., et al. \"Discovering outstanding subgroup lists for numeric targets using MDL.\" ECML PKDD 2020), but they 1) consider rule lists (not trees), 2) they consider subgroup discovery for numeric targets (so not regression or CDE) and 3) they use the Bayesian encoding, not NML (which is not possible as discussed). \n- **Runtime of CART**: it depends on the search space of the regularization hyper-parameter. In our experiments, we set the range by “ccp_alphas = np.logspace(-5, 2, 30)”, and for all datasets, a single run should not exceed 100 seconds from what I remember.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kNEZEZXykX", "reviewer": "Reviewer_yfPy", "summary": "This paper proposed to use decision tree with leaves formed by histogram models for conditional density estimation to gain interpretability. Characteristics of the proposed method, along with the density estimation accuracy and run time, are evaluated with numerical experiments.", "questions": "- If p(y|x) changes smoothly with respect to x, is the proposed method capable of capturing such smoothness, given it is based on partition of X? \n- How to prevent overfitting?", "limitations": "The authors discussed some limitations of their method. There is no potential negative societal impact of their work.\nI suggest providing some discussion on the questions raised above.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "Extensive experiments are conducted to evaluate multiple aspects of the proposed method, including comparison of the density estimation accuracy with related methods, as well as computational efficiency and interpretability-related aspects.", "weaknesses": "The proposed method can be viewed as a conditional density estimator with partitions on both X and Y space. There is limited novelty as far as I read from this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, \n\nAlthough I understand that the judgment about novelty can be a subjective matter, I would appreciate it if more information is provided regarding the motivation for this judgment, which would be helpful for me to improve the paper. \n\nRegarding your two questions: \n1. I am not sure what exactly you mean by \"capturing such smoothness\". Could you give a formal definition or an example of \"capturing smoothness\"? In general, partition-based models (including histograms, decision trees, regression trees, etc) approximate smooth functions (e.g., decision boundaries) in a piecewise manner. \n2. As elaborated in the paper, the overfitting is prevented by formalizing the learning problem as a MDL model selection problem.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NKAcm2FFRF", "reviewer": "Reviewer_n5cC", "summary": "This paper addresses the problem of conditional density estimation, i.e. given a conditioning variable x estimate the whole distribution of y, with special emphasis on interpretability.\nFor this interpretability requirement, the authors resort to classical decision trees allowing to partition the conditioning space in an interpretable manner. The conditioned distributions of y are modelled with equally-spaced histograms with a variable number of bins.\nThis construction is optimised in an heuristic manner using an objective function that is built using a Minimum Description Length (MDL) approach, using different codes for the different parts (size and structure of the tree, splitting conditions and histograms) of the construction to be determined.\nExperiments show log loss performance that is globally better than interpretable competitors on the selected datasets. Empirically, the trees obtained with this method are shallower than other interpretable tree-based methods and they are more robust to irrelevant features, in the sense that these are less often used for splitting.", "questions": "paragraph explaining the role of C and d  (lines 191 to 194) is a bit confusing, can you provide an example showing what are the splitting options that are encoded when d>1 ?", "limitations": "Limitations were properly discussed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper is generally  well written and easy to read.\nThe construction seems useful in practice as shown by the experiments and could be easily adopted by practitioners.", "weaknesses": "This work combines in a straightforward manner classical building blocks from the machine learning and information theory literature: decision trees, histograms, MDL, universal codes for integers, ... so there is little novelty in the construction\n\nAlthough it is advertised that no hyperparameter tuning is needed thanks to MDL, there are still arbitrary choices for the priors and also the hyperparameter C (C=5 in the experiments).\nFor example, the tree size and structure could be encoded using Willems' approach in Context Tree Weighting  (preorder traversal of the tree, for each node use one bit to say if it is internal or leaf, which is equivalent to putting a 1/2 prior probability of splitting a node).\n(F. M. J. Willems, Y. M. Shtarkov, and T. J. Tjalkens. The context-tree weighting method: Basic properties. Information Theory, IEEE Transactions on, 41(3):653–664, 1995.)\nThis should be discussed.\n\nThe MDL approach used in this work is called \"crude\" in [14]  since there is some arbitrariness in the choice of priors instead of some optimality criterion.\nThe only part in this work that is \"refined\" [14] is the NML  used for the histograms themselves but this is something that is well know (Proposition 1 is a simple extension of existing results). \nThis should be discussed.\n\nThe optimisation method is heuristic and thus its interest for the NeurIPS audience is limited (in fact, most of the algorithm is in the appendix, which indicates that it is not an important contribution).\n\nFor these reasons, I think that the novelty and the significance for NeurIPS is quite limited. \n\nIn my opinion, this work would fit better in a more applied venue.\n\nMinor issues:\ncitations to books (e.g. [6,14]) should be more precise\n\nline 186: \"variable name\" can be misleading, I suggest to say \"variable index\"\n\ntypo: line 231 \"we iterative\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments and advice. \n\nWe now address **each paragraph in the \"Weakness\" section** of your review.\n\n## Factual Error in Paragraph 2:\n- The statement in the review that \"no hyperparameter tuning is needed\" is incorrect. Our paper highlights that the advantage of using MDL is the elimination of the need for a REGULARIZATION hyperparameter. We specifically use the term **\"regularization hyperparameter\"/\"hyperparameter for regularization\"** throughout the paper, in which we mentioned the word \"hyperparameter\" 4 times in total. We specifically emphasized this point because the “standard” way of regularization in decision tree learning is to penalize on the number of nodes times a \"regularization hyperparameter\", which needs extensive parameter tuning. \n\n## Rebuttal about the statement that we are using \"crude\" MDL (Paragraph 3):\n- The claim that our MDL approach is \"crude\" is incorrect according to Chapter 14.3 of the cited book [14]. **See page 426. Here I quote: \"In the end, we encode data D using a hybrid two-part/one-part universal model, explicitly encoding the models we want to select between and implicitly encoding any distributions contained in those models\", which defines our approach as “the refined MDL”**. Additionally, while the NML regret for histograms is well-known, our contribution extends these results to a supervised setting with histograms and demonstrates that the NML regret for the CDTree is the product of the regret terms for all histograms on all leaves.\n\n## Rebuttal about choosing priors (Paragraph 2):\n- Although there is indeed some flexibility in choosing the model encoding scheme within the MDL framework, similar to selecting priors in Bayesian model selection, encoding the model is not arbitrary but follows requirements and guidelines. This includes ensuring the prior probability defined on the model class sums to 1, preventing an excessively large penalty term that would require larger sample sizes to converge to the true model. The encoding scheme you mentioned is suboptimal because the corresponding prior probability is not proper (consider, e.g., the code word that corresponds to the case that every node is a leaf node, which cannot form a decision tree). Notably, many “old” methods that leverage MDL used various intuitive yet “crude” encoding schemes like this one, and it is hardly possible to review all of them (e.g., the one used in the C4.5 decision tree is different from the one you mentioned above). \n\n## About choosing hyperparameter C (Paragraph 2): \n- The hyperparameter C establishes a hierarchical structure on the search space of continuous feature variables, setting levels for granularity. This hierarchy helps express prior beliefs about model complexity. For instance, intuitively, if a decision tree gives a split condition with a highly precise value like $X > 1.0001$, domain experts may ask, is this level of granularity (up to the 4 decimal places) necessary? Why not just $X > 1$? Thus, from the perspective of model selection, the former condition is \"more complex\" (more bits required to encode) than the latter one. However, as different feature variables may have very different ranges, instead of considering the granularity in terms of the decimal precision, in this paper we consider the granularity in terms of the quantiles (e.g., splitting on the 1/2-quantile may be considered \"simpler\" than splitting on the 1/10000-quantile). As explained in Appendix C.2, just like we have different numeral systems (e.g., decimal/binary), which set up different hierarchical structures for the levels of granularities, the hyperparameter C plays a similar role here in setting up the hierarchical structures for the quantiles. \n- We have shown that a global ad-hoc choice for $C = 5$ works well for **all datasets in our experiments section**. By contrast,  it is impossible to have a single value for the regularization hyperparameter used in traditional decision tree learning that works well for different datasets, which also does not carry this natural meaning of the $C$ in our paper. \n\n## About our algorithimic innovations (Paragraph 4): \n- While our algorithm is greedy, it addresses the challenging task of optimizing the tree structure and separate models (on leaves) simultaneously. Recent algorithmic advancements for classification/regression trees have been reviewed, highlighting the challenges for CDE tasks, which motivates the choice for the heuristic algorithm. \n- Our main algorithmic innovations are described in Section 5 (not the appendix), including 1) no pruning for the tree, which favors the model complexity, and 2) searching the histograms directly (without using any \"guessing\" heuristics that are common in traditional \"model tree\" methods). We also include in Section 5 a high-level yet complete description of the algorithm process. The pseudo-code and other details are put to the appendix for reproducibility due to the page limit. \n- Due to the fact that improving on the heuristic algorithms for classification/regression trees often require a whole research paper to describe (as reviewed in our paper), we consider further improving on our proposed algorithm for CDE as future research. \n\n## Rebuttal about the characterization of our method as a \"straightforward combination of classical building blocks\". \nWe respectfully disagree on this statement as it overlooks our contributions:\n- We propose the first single tree-based model specifically designed for the CDE task (as CADET essentially used the same model as CART due to the equivalence between the MSE loss and the Gaussian assumption);\n- We are the first to formalize the decision tree learning under the (modern) MDL framework as a model selection problem, introducing new encoding schemes for data (with NML) and the decision tree itself.\n- Our extensive experiments demonstrate that with CDTree, kernel-based methods are no longer the only choice for \"shallow\" models for CDE.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "de7oFwy0Uw", "reviewer": "Reviewer_cqKh", "summary": "This paper proposes a tree-based conditional density estimation model. Previously, conditional density estimation involved: 1) kernel density estimation-based method (requires expensive bandwidth tuning); 2) neural network (black-box approach); 3) a tree-based model which focuses on a Gaussian distribution for the target; 4) regression-based approach. In contrast to these approaches, the proposed method is 1) non-parametric; 2) uses a MDL principle which eliminates the need for hyperparameter tuning for regularization; 3) an iterative algorithm which searches for the optimal histogram in each leaf (thus modeling the full distribution). The authors claim that the proposed method is both more accurate (as measured by the log-loss) and more robust against irrelevant features. Further, the method is smaller in size and improves interpretability (which comes from a tree-based model). The experimental results are provided against a varying range of datasets from the UCI repository. The comparison is provided against other tree-based models and blackbox-approach methods.", "questions": "1. The method denoted as \"Ours\" should be formally labeled as \"CDTree.\"\n2. Are all of the attributes in the datasets continuous? How would the method handle discrete/categorical attributes?\n3. What model selection criteria is used for CART models? Is it the same as Equation 4 which is used for CDTree?\n4. Is it possible to move the pseudocodes into the main writeup?\n5. Figure 3 may contain some presentational issues. What do \"+\", \"circle\" ,\"square\", etc mean?", "limitations": "Overall the paper is reasonable well-written. However, the experimental results are limited to fairly small datasets which unfortunately do not highlight potential scalability of the proposed method. While the focus of the method is on interpretability and accuracy, the authors would have to provide some empirical results on the training phase. In addition, please move the pseudocode description of the algorithm to the main writeup section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Experimental results are comprehensive, providing superior accuracy and more compact model compared to other approaches.\n2. The proposed method does not need extensive parameter tuning and is empirically fast.\n3. The algorithm seems to be fairly novel and easy to understand.", "weaknesses": "1. The dataset used in the experiments is fairly small. To highlight empirical performance in training, the authors are advised to use larger datasets.\n2. The method assumes that the support of the target variable is bounded.\n3. Some presentational issues in the figure are present. See below in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nThanks for the constructive advices! I will now reply to each point in the Weaknesses and Questions sections. \n\n# Regarding Weaknesses in the review: \n## 1. Rebuttal to Weakness 1: In comparison to previous research for CDE, the datasets we used are NOT smaller.\n- We included 14 datasets in total, with various sample sizes (from 104 to 21263) and dimensionalities (from 5 to 82). We aim to test on both small and large datasets since 1) the sample size is often not too large in critical areas like healthcare; 2) (as mentioned in the Introduction) CDE is often used to understand the data collected from scienfitic experiments, in which data size is often small too, as data collection is expensive. \n    \n   **Discussion about the data sizes in previous research:** for instance, in the most recent related work [12] *(Gao, Zijun, and Trevor Hastie. \"LinCDE: conditional density estimation via Lindsey's method.\" Journal of machine learning research 23.52 (2022): 1-55.)*, which is a tree ensemble method, all datasets considered are below $2 000$, indicating that smaller datasets are indeed interesting to the CDE task. Further, the deep learning model proposed in [9] *(Dutordoir, Vincent, et al. \"Gaussian process conditional density estimation.\" Advances in neural information processing systems 31 (2018)) {see page 8, figure 4}* considers benchmark datasets with similar sizes (number of rows) to ours (and the dimensionalities of datasets we consider are even in general higher). \n\n- Despite what has been discussed, we tried on a larger dataset about NYC taxi duraion (sample size $N=203686$), which we take from Kaggle. This is a dataset that can be used to predict the taxi trip durations based on the pick-up and drop-off locations (longitutde and latitude). We only use a subset of this data that corresponds to 1 month. However, most interpretable competitor methods fail on this dataset, and specifically **CKDE/NKDE/LinCDE fail to return any results because their (widely used) implementations caused that the memory limit is reached**. \n\n   Meanwhile, our method and CADET are the only interpretable methods that can be applied to this dataset. However, as shown in the table below, the negative log-likelihoods of our method are pretty close to those of the two neural network methods (NF and MDN), significantly outperforming the negative log-likelihoods of CADET. **Although the main goal of this paper is NOT to (just) introduce a CDE method that scales better than existing \"shallow\" models, this result indeed shows the potential of the scalability of CDTree to larger datasets**. \n\n| Methods | Mean (Negative Log-likelihoods)  | Standard Deviation |\n|-----------|-------|--------------------|\n| Ours (CDTree)     | 7.015 | 0.0038             |\n| CADET     | 8.290 | 0.0870             |\n| NF        | 6.921 | 0.1031             |\n| MDN       | 6.835 | 0.0056             |\n\n\n## 2. Reply to Weakness 2:\n- We are aware of this limitation as we already discussed it in the Limitation of the paper. \n\n## 3. Rebuttal to Weakness 3: Figure 3 does NOT contain the presentational issue mentioned in the review as the legend shows the meanings of different shapes. \n- As shown by the legend on the upper part of the figure, each method is shown by a different color TOGETHER with a different shape. We find that, in comparison to using different colors only (with the same shape), our visualization makes it easier to show the general picture for each method accross all datasets.\n\n\n\n\n# Reply to Questions: \n## 1. The term \"Ours\" / \"CDTree\" for proposed method / model respectively\nWe deliberately use “Ours” to refer to our method (the model we proposed, the learning criterion, and the algorithm, as a whole). By contrast, we use the term “CDTree'' to refer to the probabilistic model (learned from the data), which can be seen already from our abstract. \n\n## 2. Our method can handle both discrete/continuous features. \nAs discussed in Appendix B.2, we do not require the feature variables $X$ to be continuous. That is, while the target variable $Y$ needs to be continuous (otherwise there is no need for conditional density estimation), the feature variables can be both categorical and continuous. Just like the decision tree for classification/regression, the splitting conditions on the internal nodes can be defined for both discrete and continuous variables. We do assume that categorical variables are preprocessed and one-hot encoded though. \n\n## 3. About CART model selection. \nThe CART we use is the original version from the inventors. That is, the tree is first grown by iteratively maximizing the gini index, and then pruned by the so-called “cost complexity pruning”, i.e., by optimizing the impurity (MSE for regression) plus a regularization term defined as the number of nodes times a regularization hyperparameter, in which the hyperparameter is tuned by cross-validation in our experiment. These are (already) discussed in Appendix C.4.\n## 4. Put (more) content about Algorithm in the main content. \nYes, if the paper is accepted and then one more page is granted. \n## 5. Resolved (see above).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents an interesting tree-based, interpretable, conditional density estimator. All the reviews seem to agree that the method is of practical utility and is an interesting application of MDL principle. Interpretability in CDE is indeed under-studied. In case the paper is accepted, I request the authors to incorporate all the changes suggested by the reviewers.
Decision: Accept (poster)