Treeffuser: probabilistic prediction via conditional diffusions with gradient-boosted trees
Authors: Nicolas Beltran-Velez
Alessandro Antonio Grande
Achille Nazaret
Alp Kucukelbir
David Blei
Venue: NeurIPS 2024 poster (2024)
Keywords: trees
gradient boosting
diffusion
probabilistic predictions
uncertainty quantification
Primary area: probabilistic_methods

Abstract: Probabilistic prediction aims to compute predictive distributions rather than single point predictions. These distributions enable practitioners to quantify uncertainty, compute risk, and detect outliers. However, most probabilistic methods assume parametric responses, such as Gaussian or Poisson distributions. When these assumptions fail, such models lead to bad predictions and poorly calibrated uncertainty.  In this paper, we propose Treeffuser, an easy-to-use method for probabilistic prediction on tabular data. The idea is to learn a conditional diffusion model where the score function is estimated using gradient-boosted trees. The conditional diffusion model makes Treeffuser flexible and non-parametric, while the gradient-boosted trees make it robust and easy to train on CPUs. Treeffuser learns well-calibrated predictive distributions and can handle a wide range of regression tasks---including those with multivariate, multimodal, and skewed responses. We study Treeffuser on synthetic and real data and show that it outperforms existing methods, providing better calibrated probabilistic predictions. We further demonstrate its versatility with an application to inventory allocation under uncertainty using sales data from Walmart. We implement Treeffuser in https://github.com/blei-lab/treeffuser.

Treeffuser: Probabilistic Predictions via Conditional Diffusions with Sradient-Boosted Trees
Nicolas Beltran- Beltran-Velez" Alessandro Antonio Grande" 2,3 Achille Nazaret 1,3 Alp Kucukelbir
David Blei
Department of Computer Science, Columbia University, New York, USA Halvorsen Center for Computational Oncology, Memorial Sloan Kettering
Cancer Center, New York, USA
3 Irving Institute for Cancer Dynamics, Columbia University, New York, USA 'Fero Labs, New York, USA
Department Statistics, Columbia University, New York, USA
Abstract
Probabilistic prediction aims to compute predictive distributions rather than sin- gle point predictions. These distributions enable practitioners to quantify uncer- tainty, compute risk, and detect outliers. However, most probabilistic methods assume parametric responses, such as Gaussian or Poisson distributions. When these assumptions fail, such models lead bad predictions and poorly calibrated uncertainty. In this paper, we propose Treeffuser, an casy-to-use method for probabilistic prediction tabular data. The idea is to learn conditional diffu- sion model where the score function is estimated using gradient- boosted trees. The conditional diffusion model makes Treeffuser flexible and non-parann non-parametric, while the gradient- trees make it robust and easy to train on CPUs. Treef- fuser learns well-calibrated predictive distributions and handle wide range of regression dasks-including those with multivariate, multimodal, and skewed responses. We study Treeffuser on synthetic and real data and show that it out- performs existing methods, providing better calibrated probabilistic predictions. We further demonstrate its versatility with application inventory allocation under uncertainty using sales data from Walmart. We implement Treeffuser in attps://gg thub. com/blmmeeeeeoe weeeeelllaaeeeeee lab/treeffuuet.
Introduction
this paper, we develop a new method for probabilistic prediction from tabular data. This problem is important many fields, such power generation [1], finance [2], and healthcare [3]. It drives decision processes such as supply chain planning [4], risk assessment [5, 6), and policy- -making [7].
Example: Manufacturing plants measure information as they operate. This information propppttionnppooertteess of raw materials, operational flows, temperatures, and level measurements- cessureements-colllccieefeee determine the output of the plant [8]. From this data, how can manufacturers adapt operations to reduce emissions while maximizing profits? The answer requires both good predictions and estimates of uncertainty, to trade off the risk of failure with the reward of lower emissions and higher profit. More broadly, such industrial workflow problems often rely on predictions from vast amounts of tabular dbse-oobes of variables arranged in table [9, 10].
'Equal contribution, authors listed in alphabetical order.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

0.2
0.2
M
= 0.5
M
0.8
= 0.8
Figure Samples y from Treeffuser vs. true densities, for multiple values of under three different scenarios. Treeffuser captures arbitrarily complex conditional distributions that vary with æ.
Our method builds on two ideas: diffusions [11] and gradient- boosted trees [12]. Diffusions accurately estimate conditional distributions, but existing methods are not designed for tabular data. Decision trees excel at analyzing tabular data, but do not provide probabilistic predictions. Our method, which we call Treeffuxer,, combines the advantages both formalisms. defines an accurate tree-based diffusion model for probabilistic prediction which can easily handle large datasets.
Fig. shows Treeffuser action. We feed it data from three complex distributions-a multimodal response with varying components, an inflated response with shifting support, and a multivariate response with dynamic correlations. For each task, Treeffuser uses trees to learn diffusion model and then outputs samples from the target conditional. These samples fully capture the complexity of the response functions.
Treeffuser exhibits several advantages:
It is nonparametric. makes few assumptions about the form of the conditional response distribution. For example, it can estimate response distributions that are multivariate, multimodal, skewed, and heavy-tailed.
It is efficient. Diffusions can be slow to train [13]. Treeffuser relies on trees and is fast. For instance, in section 5.3, Treeffuser trains from table with 112,000 observations and 27 variables 53 seconds on a laptop.
It is accurate. On benchmark datasets, Treeffuser outperforms NGBoost [14), IBUG [15], quantile regression [16] and deep ensembles [17]. It provides better probabilistic predictions, including more precise quantile estimations and accurate mean predictions.
The rest of the paper is organized as follows. Section reviews diffusions and gradient- boosted trees. Section 3 integrates these two ideas into Treeffuser and justifies the method. Section 4 discusses related work. Section studies synthetic and standard benchmark datasets.
2
Background
Treeffuser combines two ideas: diffusion models and gradient-boosted trees (GBTss. This section provides gentle introduction to both topics. Familiar readers can skip ahead to Section 3.
2.1 Diffusion Models
A diffusion model is generative model that learns an arbitrarily complex distribution ( It consists of two processes: forward and reverse diffusion.
The forward process takes the target distribution iT (g) and continuously transforms it into a simple distribution. It does so by progressively adding noise samples from n:
Draw y()) ~ π.. Evolve (t) until T according to: dy(() f()(t), g(t)dur(t),
(I)

---

where Eq. (I) defines a stochastic differential equation (SDE) with standard Brownian motion w (f), and drift and diffusion functions and 9. The time horizon T is large, such that the resulting marginal distribution Psimple pr (y = J ( v(0) = yy'
y ()')(((' ddy'' is agnostic to π.
The reverse process transforms the simple distribution back into the target distribution by denoising perturbed samples from Psimple- posits the following model, which runs backward from T to O:
{ (T) ~ Psimple, dÿ = [f(((t).t) g(t)² ((1) log pt (ū(())dt + gtt)
(2)
where (t) a standard Brownian with reversed time. Anderson [18] shows that (t) d y(t) for each time f, and so ū0) This means that by drawing sample from Psimple and then numerically approximating the reverse process in Eq. (2), we obtain samples from æ.
However, the score function, + Vy (o is usually unknown. Vincent [19] shows it can be estimated from the trajectories of the forward process as the minimizer of the following objective:
+ Vy (v) = arg min E,E,
| y(t) Pog ((())) - s (t)), 1)
(3)
#ES
where S is the set of all possible functions of y indexed by time t. and Por denotes the conditional distribution of (() given y0). In practice, we set f Uniform(() (0, 1]) and choose the drift f and diffusion function g so that por is Gaussian. We detail our choice of f and y in Appendix C.
We approximate the expectations in Eq. (3) with the empirical distribution of the observed y for Eg, and with the known Gaussian trajectories yt) (0) for Epo. The objective can then be minimized by parametrizing the score with any function approximator, such a neural network or, as we do, with trees. By estimating the score function, we effectively learn the distribution π.
For more detailed introduction to diffusion models, provide expanded version of this section in Appendix A.
2.2 Gradient Boosted Trees
Consider the following common machine learning task. Given variables (a, b) n,, where is scalar, learn the function F* (a) that best approximates b from a:
F
arg min ((a,t)> (L((()))))
where L is loss function measuring the quality of F, such as the squared loss L(u, v) (w-ua u)2 maadient-boosted trees (GBTs) are widely used parametretic machine learning model for this task [12]. GBTs use decision trees [20] as simple building block functions f R IR form good approximation f of F*. GBTs sequentially build approximations F; defined as
A( Σ (a),
(5)
m==
where (0, 1) is a parameter akin learning rate.
Each decision tree fi is constructed to minimize the squared error between the current approximation F; and the negative gradient of the loss function:
(Fla)+ ( 27
ft = arg min Ea, fETreas
(6)
(F.(a)
As the number of trees increases, the approximation F. becomes better minimizer of Eq. (4). Various modifications to this basic algorithm have been proposed, including different loss functions in Eq. (4), higher order optimizations for Eq. (6) and general heuristics for faster training [21-23].
3 Probabilistic Prediction via Treeffuser
Treeffuser tackles probabilistic prediction by learning diffusion model with gradient- trees. It is particularly well-suited to modeling tabular data. Trees offer useful inductive biases, natural
3

---

handling of categorical and missing data, and fast and robust training procedures. Diffusions eliminate the need for restrictive parametric families of distributions and protect against model misspecification.
Denote an independently distributed set observations as D = { y=) Treeffuser forms the predictive distribution (y a)) as function of inputs æ. We first introduce conditional diffusion models and discuss the conditional score estimation problem for both univariate and multivariate outcomes. We then outline the training and sampling procedures for Treeffuser.
3.1 The Conditional Diffusion Model
Treeffuser produces a distribution over y for each value of æ using conditional diffusion models. Unlike approaches that guide an unconditional model to achieve conditionality, Treeffuser follows the line of work that directly fits the conditional score function [24-26].
Conditional diffusion models. The diffusion models introduced in Section 2.1 target the marginal distribution of y. Here, we extend them to conditional distributions T (3) == π ( x).
To model nw assign diffusion process (4) to each value of æ. These processes share the same diffusion equation but have different boundary conditions corresponding to their target conditionals:
{
yz (0) ~ nr (y); dy (t) (t),,)),))))))) t)dt g(t)dwo(t)
(7)
As before, f and g are simple functions such that ya (T) ~ Psimple for all x. Denote the marginal distribution of (t) as Par.. For two time points t and u, where t > denote the conditional distribution ym(()) (n(u) as Put. (Note how Put does not vary in æc)
To match each x de-epeedent forward SDE we have an x-deppedent reverse SDE of the form:
dÿ (t) = [f())()))) - g(t)2V (t(()) log Px, (ū)) dt + ((t)du(())
(8)
where the score function V Ps, Pa., now also depends on æ. Similar unconditional diffusions, by estimating the conditional score, we can sample from Yee by first sampling (T" from Psimple (y) and then solving the corresponding reverse SDE.
Conditional score estimation objective. Estimating the conditional score follows similar strategy as the unconditional version, but with the added requirement of simultaneously estimating the score for all æ. Recall that by Eq. (3), for fixed æ, the function s defined by
S = arg min E, (v (0)) Epos (ve (.) (0)) (t) log Por (or (t) yo) s(y (t). t)
#ES
satisfies * (ym, () V (1))) log Pw, (yn (()) for all y. l and the fixed æ. Intuitively, if we allow E S to also take æ as input, we can gather all of these individual optimization problems into a single large problem by taking additional expectation over π, that is:
arg min E (o) (0)) SES+
[|| (ya por Por (t) ye (0)
(t),,
(9)
Here, S+ represents the set of functions that take æ as an extra input along y and t. The uppercase S emphasizes that S takes as input, contrary to lowercase The validity of this objective is given by the following result.
Theorem (Optimal Conditional Objective). Define S* as the solution of Eq. (9). Then, for almost all x, y, t with respect to π (x y) and the Lebesgue measure on t E [0, T), we have
S (y. t, x)) = V log (3))
(10)
We refer Eq. (9) the conditional score objective. The proof is provided in Appendix B.
3.2 The Trees
Treeffuser uses gradient-boosted trees (GBTs) to minimize Eq. (9). As GBTs work on scalar outputs, separate the conditional score objective into an equivalent set of dy independent scalar- valued

---

Algorithm 1: Treeffuser Training
Algorithm 2: Treeffuser Sampling
Data: D, R, SDE-specific (h, T)
Data: GBTss (U1, ...: Ua,), ), imput instance æ,
Result: GBTs (#1,-.,,4) optimizing Eq. (12)
discretization steps na, SDE-specific
D' Ø
(Psimlle T, T.J.9.
for (x y (0) E do
Result: A sample I ( x))
for r 1, ,..., R do
< T/na
t ~~ Uniform(0, 7] < ~ La, D' D' ((h(s,t,
Psimple
- T
for i 1,..., nd do
w (O, Ia,)
for k 1,...,., dy do
f + f 1) ((1)9((y,,,,)//(tt))
De + {(a, ) (a, b) DD} Uk TrainGBI(D**)
y - y (f8 g(t) t - A
return (Ug....,Uu Ud)
return
sub- cobb-prbblmss
( (t)(
(olog por
S arg min E, (x.)) (y (t)lv)
(11)
Sp EGBT
where ag denotes the k-th component of vector
Recall that the drift and diffusion functions of the forward process are chosen such that Po: is Gaussian. Let m mtt: y) and a o(t) denote the corresponding mean and standard deviation, respectively. Treeffuser replaces the partial derivative in Eq. (11) with its closed-form expression as function of m and σ. Treeffuser further reparametrizes S(y. f, x) with Uy, f. (t)) and defines MCC, L, y) = m([; 3) + (1),,, the process by which a sample y at time gets diffused into a sample at time / with Gaussian noise Ç. The optimization problems in Eq. (11) are then:
Vk E (1,...,d)) dg). UH = arg min E( E (((. [(C+ +
(12)
Us
The next theorem justifies how the individual problems in Eq. (12) estimate the conditional score. Theorem 2 (Treeffusee One- One-Dimensional Objectives). Denote U* (Uī UZ,)) Then for almost all x. y.t with respect to ((x y) and the Lebesgue measure E [0, T), we have
U (,
V P.,
(13)
(t)
Each problem in Eq. (12) a GBT problem where the notation within Eq, (4) corresponds to F Us. : (M(C, y), x)) Rdy+1+d, b = -Ca R and L is the square loss. We note that the noise scaling reparametrization, = U/o, is key to stabilizing the learning process; see the ablation study in Appendix G.
Finally, Treeffuser approximates the expectations in Eq. (12) with Monte Carlo sampling. For each sample (æx, y) from the dataset D, Treeffuser samples R pairs of (t, ~ Uniform([0, 1)) &.(0, Ta,) and creates new datasets & containing R datapoints of the form ((h(S, t, y), f, -(k), one D per dimension of y. Then, each of these datasets is given a standard GBT algorithm, such as LightGBM [22] or XGBoost [21]. Our implementation of Treeffuser uses LightGBM. Algorithm details this procedure.
3.3 Sampling and Probabilistic Predictions
Treeffuser provides samples from the probabilistic predictive distribution ( æ). It does so as in standard unconditional models by plugging in the GBT-estimated conditional score from Eq. (12)
5

---

into a numerical approximation of the SDE in Eq. (8). While Treeffuser is compatible with any SDE solver, our implementation leverages Euler-Maruyama [27] due its good balance between accuracy and the number of function evaluations. In general, we found good performance with as few as 50 steps. Algorithm implements this procedure.
The samples generated by Treeffuser can then be used to estimate means, quantiles, probability intervals, expectations, or any other quantity of interest that depends on the response distribution.
3.4 Limitations
The design of Treeffuser offers advantages terms of usability, versatility, and robustness. But also comes with a few limitations. First, the diffusion process is theoretically defined to only model continuous responses y. However, count and other forms of discrete responses are common in the probabilistic modeling of tabular data. While our experiments show that this limitation does not prevent Treeffuser from outperforming comparable methods on these kinds of data, there may be opportunities for further improvement with direct modeling of discrete outcomes. Second, Treeffuser does not offer a closed-fore forommffommmmoo density and must solve an SDE to generate samples. This sampling process, in contrast with the fast training, become expensive when many samples per datapoint are required.
Related Work
Treeffuser builds on advances in diffusion models to form probabilistic predictions from tabular data.
Diffusion models. Diffusion models excel at learning complex unconditional distributions on a range of data, such as images [28-30], molecules [31], time series [32], and graphs [33]. A common task conditional generation, where the goal to generate samples from a distribution conditioned on features. There are two approaches to this objective. One approach is to use guidance methods by which the score of an unconditional diffusion model is altered during generation to mimic the score of the conditional distribution [11. 34, 35J. This approach is especially popular for inverse problems [11, 36, 37j. Another approach is to train conditional model from the start, incorporating the conditioning information during training [24-26, 38, 39j. This is the approach adopted by Treeffuser.
Treeffuser contributes to a recent line of work that applies diffusions to tabular data. This includes deep learning approaches for data generation [40], probabilistic regression [41], and missing data imputation [42]. Among these methods, CARD [41] uses neural-net based diffusions for probabilistic predictions and thus is most similar to Treeffuser in scope. We attempted to include it in our experiments using the implementation from Lehmann [43], but returned very poor results. We therefore excluded it from our testbed. The imputation of missing data has been recently extended to gradient-boosted trees [44].
Probabilistic prediction for tabular data Probabilistic prediction for tabular data can be classified into parametric and non-parametric methods based on their assumptions about the likelihood shape. Parametric tree-based methods include NGBoost [14] and PBGM [45]. NGBoost uses natural gradients to optimize a scoring rule, while PBGM sequentially updates the mean and standard deviation for predictions. DRFs obtain maximum likelihood estimates by using this criteria to choose splits. Neural-based parametric methods include Bayesian Neural Networks [46], MC Dropout [47], and Deep Ensembles [17]. Notably these methods are all indirectly or directly Bayesian. Another approach, normalizing flows, transforms a latent distribution via an invertible neural network [48] and has been applied to tabular data [49]. Non-parametric methods are often tree-based, such as Quantile Regression Forests [16]. Distributional Random Forests (DRF) [50], and IBUG [15]. Quantile Regression Forests approximate the inverse cumulative distribution function by minimizing pinball loss, while DRF and IBUG use a tree-based similarity metric to weight training data for predictions. These methods are baselines in our empirical studies, with detailed descriptions in Appendix F.I.
5 Empirical studies
We demonstrate Treeffuser across three settings: synthetic data, standard UCI datasets [51], and sales data from Walmart. We find that Treeffuser outperforms state-of-the-art methods [15, 17, 50];

---

Dataset
N. dl
Deep ensembles
Quantile regression
Treeffuser
Treeffuser (no uninn)
(Gaussian)
(XGBoest)
17.379.12.1
0.11
1.85
2.154
1.60
0.05
eaergy
768. 8.
5.00 + 0.71
178 0.49
KA
K4
5.13 2
3.07 = 0.40
3.32 + 0.48
X16"
kin8om
8,192.
3.59
0
0.44
6.3 0.16
9.44
5.89
0.14
movics
7.415,
2.94 0.33
x
3.12 0.52
7.90
5.57 #.67::0611
2.68 -
2.69 0.28
K107
naval
11,934,17.1
4.11 0.33
0.19
3.20 0.17
16.86 = 2.46
0.16
2.02
2
x16-1
news
19,644,58.1
0.27
3.89
1.19
2.32
0.117
0.16
0.17
power
9,568, 4. 1
2.06 1. 0.10
2.01
1.133
1.61
1 0.07
5.40
0.1.2
1.10
1.49
1 0.07
1.52 1: 0.07
x16"
supere. 11,263,81.1
+ 0.31
6.24
0.50
4.14
0.28
3.79 0.14
4.32
3.52
3.60
+
wine
6,497,
12,
3.59
1. 0.10
3.82
0.11
3.25 1. 0.14
3.24 0.14
3.30
2.59
1. 0.13
2.67
1. 0.13
x1o
yacht 308,
4,86 4: 1.58
3.67
1,47
3.75
3.53
119:1155 1,30
7.73
3.11
= 0.99
3.39
1: 0.97
x10
Table 1: CRPS (lower is better) by dataset and method. x indicates the method failed to run, and NA that the method not directly applicable multivariate outputs. Standard deviations are measured with 10-fold cross-validation. For each dataset, the two best methods are bolded. Treeffuser provides the most accurate probabilistic predictions, even with default hyperparameters (no tuning). Powers of ten are factorized out of each row in the rightmost column.
it can capture complex distributions with multimodal, inflated, or multivariate responses; it achieves better probabilistic predictions on many real-world datasets and it produces more ac- curate sales forecasts on Walmart data. We also find that Treeffuser can outperform other (tuned) methods without dataset-specific hyperparameter tuning. We provide the code for Treeffuser at https:/ thub. wwwm///eellabbeaaeeeeeeeeeeeeeee lab/treeffueer.
5.1 Probabilistic prediction synthetic data
We show the flexibility of Treeffuser on three probabilistic regression tasks that are difficult model with standard parametric models. We also show its competitive performance on Gaussian data when compared to methods that posit Gaussian likelihood.
Treeffuser vs. complex response functions. We design three difficult probabilistic regression tasks: multimodal response where the number of modes changes with æ; an inflated response with support that changes with æ, and a multivariate response with nontrivial covariance structure. The data generation processes for these datasets are detailed in Appendix E. For each dataset, we generate 10,000 observations (z,, 34) and train Treeffuser with its default parameters (see Appendix C).
Fig. 1 shows the histograms of 1,000 Treeffuser samples for three different values of æ and compares them with the true conditional densities of the response. Treeffuser captures the conditional distribu- tion for all values of the distribution without knowing their number priori. For inflated data, it recovers the peak at the inflation point and the x-dependent support of the response. For multivariate data, recovers complex covariance structures.
across all settings. For multimodal data, Treeffuser correctly detects the modes
Treeffuser vs. parametric oracles. We simulate Gaussian responses with a linear response function and compare Treeffuser to other parametric methods that assume Gaussian likelihood. These methods serve as oracles in these experiments as they are informed by the true functional form the conditional response distribution. Results and further details are reported in Appendix E. Among all methods, Treeffuser consistently ranks second best, only behind the Deep Ensemble oracle.
5.2 Probabilistic prediction on real-world datasets
We compare Treeffuser with state-of-the-arr methods for probabilistic predictions standard UCI datasets [51], A detailed description of the baseline models can be found in Appendix F.I.
Metries. We evaluate probabilistic predictions with the continuous ranked probability score (CRPS) and standard accuracy metrics.
CRPS is defined as CRPS(F, distribution function of the predicted distribution p(y x) and is the true observed value [52]. CRPS proper scoring rule, in that its expected value is minimized under the true conditional distribution of the response [53]. CRPS is usually preferred over the log- ligllioel,,ood, which is also
(F())) 12y S d'y) where F' is the cumulative
7

---

proper scoring rule, but is sensitive to the estimation of the tail densities [54]. Also, CRPS can readily be estimated from samples of ply z), which is our setting with the non-parametric methods we evaluate. We evaluate CRPS by generating 100 samples from p(y x)) for each æ. For evaluating multivariate responses, we report the average marginal CRPS over each dimension.
We also measure the quality of point predictions for each model. This is the ability to predict conditional means EEly æ). We approximate IEly æ+] using 50 samples and evaluate the accuracy using the mean absolute error (MAB) and the root mean squared error (RMSE).
Experimental setup. We performed 10-folds cross- vooss-vaiiitiion For each fold, we tuned the hyper- parameters of the methods using Bayesian optimization for 25 iterations, using 20% of the current fold's training data as a validation set. Additional Bayesian optimization's 's iterations did not change the results. We detail the search space of hyperparameters for each method in Appendix F.2.
Results. Table presents CRPS by dataset and method. Treeffuser consistently provides the most accurate predictions across datasets, measured by CRPS. Notably, it outperforms other methods even when initialized with its default parameters. There no consistent runner-up: among parametric methods, deep ensemble does overall better than NGBoost and IBUG; quantile regression does well on some datasets but underperforms in others.
We find that Treeffuser also returns the best point predictions of Ely æ). as reported in Table 4. For comparison, we report the accuracy of point predictions from vanilla XGBoost and LightGBM in Table 6 in Appendix F.3. These methods do not provide probabilistic predictions but are tailored for point predictions. As expected, XGBoost and LightGBM outperform or tie with all the probabilistic methods. In particular, they often tie with Treeffuser, suggesting that Treeffuser provides probabilistic prediction without sacrificing average point predictions.
Finally, we conducted an ablation study to investigate the impact of the noise-scaling reparametrization of the score function on Treeffuser's performance. As detailed in Appendix G, noise scaling is key to achieving top accuracy and stability.
5.3 Sales forecasting with long tails and count data
model metric
Deep ensembles
IBUG
NGBoost Poisson
Quantile regression
Treeffuser
Treeffuser (no luning)
CRPS 7.05
7.75
6.86
7.11
6.44
6.62
x10
RMSE 2.03
2.16
2.33
2.88
2.09
2.09
x10"
MAE
0.97
1.04
0.99
1.01
0.99
0.99
x10"
Table 2: Walmart dataset metrics (lower is better). The evaluation is on the last 30 days of data. Treeffuser provides the best probabilistic predictions alongside NGBoost Poisson. Deep ensembles excel at point predictions. Powers of tens are factorized out of each row in rightmost column.
We further demonstrate the applicability of Treeffuser on a publicly available dataset [55] for sales forecasting under uncertainty. The goal is to forecast the number of units sold for product given features such as its price, its type (e.g., food, cloths), and past sales.
This task is challenging due to zero inflation and long tails in the distribution of item sales. (E.g., umbrella sales are typically low but can spike during rainy weeks.) It even more challenging for diffusion model like Treeffuser, which is designed for continuous responses and not count data.
We use five years of sales data from ten Walmart stores (a large American retail chain). We randomly select 1,000 products, training on 112,000 data points from the first 1,862 days and evaluating 10,000 other data points for the remaining 30 days.
In addition the previous baselines, we include NGBoost Poisson, a parametric model specifically designed for count data. We evaluate the predictions returned by each method in three ways.
CRPS and accuracy metrics. First, we compute the same evaluation metrics as in the previous experiments. We benchmark against the methods in the experiment and the methods in section 5.2.
8

---

The results are reported in Table 2. We find that Treeffuser again proves competitive, achieving better CRPS than all methods and comparable MAE and RMSE.
Posterior predictive checks. Second, we perform held-out predictive checks [56], examining six statistics the number of items sold: the total count of zero sales, the highest sales figure, and sales figures at the 0.99, 0.999, and 0.9999 quantiles (lower quantiles are well captured by all methods). We produce probabilistic predictions of these quantities by returning their empirical distributions as induced by the samples generated by the models. Fig. 2 compares the observed values against the probabilistic predictions. Treeffuser best captures the proportion of zeros and performs as well as NGBoost IGBoost-Poisson in modeling the behavior of the tails.
Proportion of zeros
Quantile 0.9
Quantile 0.99
Max
Quantile 0.989
Quantile 0.9999
NGBoost
Quantile Regression
100
200
300
60
50
100
150
Figure 2: Posterior predictive checks for Treeffuser, NGBoost Poisson, and quantile regression. Red dashed line shows the realized value on the test set. Treeffuser best captures the inflation point at zero and performs well on the tails.
Newsvendor model. Finally, we illustrate the practical relevance of accurate probabilistic predic- tions with an application to inventory management, using the newsvendor model [57]. Assume that every day we decide how many units g of an item to buy. We buy at cost and sell at price However, the demand y is random, introducing uncertainty in our decision. The goal to maximize the expected profit:
maxp p [min(q. y) cq.
The solution to the newsvendor problem is to buy F-1 p function the distribution of y.
units, where F-1 is the quantile
We apply this model to evaluate the inventory decisions induced by each method on the Walmart dataset. To compute profits, we use the observed prices and assume margin of 50% over all products. We let Treeffuser, NGBoost-Poisson, and quantile regression learn the conditional distribution of the demand of each item, estimate their quantiles, and thus determine the optimal quantity buy.
Fig. plots the cumulative profits over the last 30 days of data. Treeffuser outperforms quantile regression by large margin and performs comparably NGBoost-Poisson This is coherent with our PPC results in Fig. 2, showing better quantile estimations for Treeffuser. This demonstrates that Treeffuser delivers competitive probabilistic predictions even for count data responses, scenario was not specifically designed to handle.
5.4 Runtime Performance Overview
We measure Treeffuser's performance in terms of training and inference speed across different datasets. On the M5 dataset, using default parameters, Treeffuser completed training in 53.2 seconds on a MacBook Pro M3 Max and generated 10,000 samples (one test data point) in 2.53 seconds. conducted additional benchmarking experiments on both the UCI and M5 datasets, with results detailed in Appendix H. (Further discussion on the model's 's time complexity is also provided in that appendix.) Details about the computational resources used in of our experiments are available Appendix D.

---

2500
Model
Hyperparameters
Treeffuser
Tuned
Cumulative profit
2000
NGBoost-Poisson
Default
Quertiieeegressoon
1500
1000
500
0
5
10
15 Day
20
25
30
Figure 3: Cumulative profits by method on inventory management problem. Treeffuser produces more accurate probabilistic predictions yielding higher profits.
Conclusion
We have introduced Treeffuser, model for probabilistic prediction from tabular data.
Treeffuser combines conditional diffusions models with gradient-boosted trees. can capture arbitrarily complex distributions without requiring any data-specific modeling tuning. It is amenable to fast CPU learning and naturally handles categorical data and missing values. have demonstrated that Treeffuser outperforms state-of-the-an methods in probabilistic regression across datasets and metrics. These characteristics make Treeffuser a flexible, easy-to-use, and robust model for probabilistic predictions.
One limitation of our diffusion- based approach is the need numerically solve an SDE generate samples, which can be costly when producing many samples. Recent advances, such progressive distillation [58] and consistency models [59], address this issue. Applying these methods Treeffuser is a direction for future work.
References
[1] Aristeidis Mystakidis, Evangelia Ntozi, Konstantinos Afentoulis, Paraskevas Koukaras, Paschalis Gkaidatzis, Dimosthenis loannidis, Christos Tjortjis, and Dimitrios Tzovaras. Energy generation forecasting: Elevating performance with machine and deep learning. Computing, 105(8):1623-1645 2023.
[2] Jillian M. Clements, Di Xu, Nooshin Yousefi, and Dmitry Efimov. Sequential deep learning for credit risk monitoring with tabular financial data, 2020. arXiv preprint 133.v.
[3] Snigdha Somani, Adam J Russak, Filippo Richter, Simon Zhao, Akhil Vaid, Farah Chaudhry, Jason K De Freitas, Niyati Naik, Riccardo Miotto, Girish N Nadkarni, Jagat Narula, Edgar Argulian, and Benjamin S Glicksberg. Deep learning and the electrocardiogram: Review of the current state-of-the-arr. Europuce, 23(8):1179-119) 2021.
[4] Anshuman Gupta and Costas D Maranas. Managing demand uncertainty in supply chain planning, Computers & Chemical Rngineering, 2718-9):1219-1227, 2003.
[5] Yacov Haimes. Risk Modeling, Assessment, and Management. John Wiley Sons, 2011.
[6] Philippe Jorion. Value at Risk: The New Benchmark for Managing Financial Risk. McGraw-Hill, 2007.
[7] James Taylor and Kimberly S. Taylor. Combining probabilistic forecasts of covid-19 mortality in the united states. European Journal of Operational Research, 304(1):25-44) 2023.
10

---

[8] Ziqiu Kang, Cagatay Catal, and Bedir Tekinerdogan. Machine learning applications in produc- tion lines: systematic literature review. Computers & Industrial Engineering. 149:106773, 2020.
[9] Anna L. Buczak and Erhan Guven. A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Communications Surveys and Tutorials, 18(2): 1153-1176, 2016.
[10] Thi-Thu-Huong Le. Yustus Eko Oktian, and Howon Kim. XGBoost for imbalanced multiclass classification-basee industrial Internet of Things intrusion detection systems. Sustainability, 14 (14), 2022.
[11] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[12] Jerome H Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29(5):1189-110
2001.
[13] Zhendong Wang. Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, and Mingyuan Zhou. Patch Diffusion: Faster and more data-efficient training of diffusion models, 2023. arXiv preprint arXiv:2304.12526.
[14] Tony Duan, Anand Avati. Daisy Yi Ding, Khanh K. Thai, Sanjay Basu, Andrew Y. Ng, and Alejandro Schuler. NGBoost: Natural gradient boosting for probabilistic prediction, 2020. arXiv preprint arX1v:1910.03225.
[15] Jonathan Brophy and Daniel Lowd. Instance-based uncertainty estimation for gradient-booster regression trees. In Advances in Neural Information Processing Systems, volume 35, pages 11145-11159, 2022.
[16] Nicolai Meinshausen and Greg Ridgeway. Quantile regression forests. Journal of Machine Learning Research, 7(6), 2006.
[17] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, 2017.
[18] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
[19] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):166111674 2011.
[20] Trevor Hastie, Robert Tibshirani, and Jerome H Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, volume 2. Springer, 2009.
[21] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '16. ACM, 2016.
[22] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[23] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features, 2019. arXiv preprint
arXiv:1706.00516.
[24] Georgios Batzolis, Jan Stanczuk, Biroia-Bibbine Schönlicb, and Christian Etmann. Conditional image generation with score-based diffusion models, 2021. arXiv preprint 111::2111.11600
[25] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mo- hammad Norouzi. Image super-resolution via iterative refinement, 2021. arXiv preprint
arXiv:2104.07366
11

---

[26] Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. In Advances in Neural Information Processing Systems, volume 34, pages 24804-24816 24816. Curran Associates, Inc., 2021.
[27] Peter E. Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations. Stochastic Modelling and Applied Probability. Springer Berlin, Heidelberg, 1992.
[28] Stability AI
Introducing stable diffusion.
https://stabbiitt..u
-rabla--diffuussoo--il---------- 2022.
[29]
MidJoumey. Midjourny. httpp::/www.miijounee com/, 2022.
[30]
OpenAl. Dall-e
2022.
[31] Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen Eisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile I. M. Wicky, Nikita Hanikel. Samuel J. Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torre,, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Sergey Ovchinnikov, Regina Barzilay, Tommi Jaakkola, Frank DiMaio, Minkyung Baek, and David Baker. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976)
2023.
[32] Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denois- ing diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning, pages 8857-8868, PMLR, 2021.
[33] Chenhao NIu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In International Conference Artificial Intelligence and Statistics, pages 4474-4484, PMLR, 2020.
[34] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In Advances in Neural Information Processing Systems, volume 34, pages 8780-8794, 8794, 2021.
[35] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. arXiv preprint arXiv:2207.12598.
[36] Hyungjin Chung, Jeongsol Kim, Michael Mccann, Mare L. Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems, 2023. arXiv preprint arXiv:2209.1 14687.
[37] Chenlin Meng, Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations, 2022. arXiv preprint 2Xiv:2108.01073
[38] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. arXiv preprint arXiv:2204.06125.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition, pages (0684-10695. 2022.
[40] Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. TabDDPM: Mod- elling tabular data with diffusion models. In International Conference Machine Learning, pages 77564-17577 PMLR, 2023.
[41] Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. CARD: Classification and regression diffusion models. In Advances in Neural Information Processing Systems, volume 35, pages 18100-18115, 2022.
[42] Shuhan Zheng and Nontawat Charoenphakdee, Diffusion models for missing value imputation tabular data, 2022. arXiv preprint arXiv:2210.17128.
[43] Nils Lehmann. lightning-uq-bon 0.1.0. http:/ppp...ogg//////// 2024.
http:://pyp..org//ppjeet/t////// wtg/ppoojje///iggnning -u-- box/,
12

---

[44] Alexia Jolicoeur-Mant Martineau, Kilian Fatras, and Tal Kachman. Generating and imputing tabular data via diffusion and flow-based gradient-boosee boosted trees. In International Conference on Artificial Intelligence and Statistics, pages 1288-1296. PMLR, 2024.
[45] Olivier Sprangers, Sebastian Schelter, and Maarten de Rijke. Probabilistic gradient boosting machines for large-scale probabilistic regression. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '21. ACM, 2021.
[46] Radford M Neal. Bayesian Learning for Neural Networks, volume 118. Springer Science & Business Media, 2012.
[47] Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pages 1050-1059, PMLR, 2016.
[48] Ivan Kobyzev, Simon J.D. Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Puttern Analysis and Machine Intelligence, 13:39))994433 2021.
[49] Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. Semi-supervised learning with normalizing flows, 2019. arXiv preprint arXiv:1912.1302:
[50] Domagoj Cevid, Loris Michel, Jeffrey Näf, Peter Bühlmann, and Nicolai Meinshausen. Distri- butional random forests: Heterogeneity adjustment and multivariate distributional regression. Journal of Machine Learning Research, 23(333):1-79, 2022.
[51] Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The UCI machine learning repository, 2023. URL https http//rcciivvvi ve. ics. uci.edu.
[52] James E Matheson and Robert Winkler. Scoring rules for continuous probability distributions. Management Science, 22 22110: 1976.
[53] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-378 2007.
[54] Mathias Blicher Bjerregård, Jan Kloppenborg Møller, and Henrik Madsen. An introduction to multivariate probabilistic forecast evaluation. Energy and AI, 4:100058, 2021.
[55] Spyros Makridakis Addison Howard. M5 Forecasting - Accuracy, 2020. URL https:// kaggle. com/ hatp/competitio iepeetiiiioss/et https///ffooooooionoeeeaaee -ww.eaaatiigggaccccccc.......
accuracy.
[56] Andrew Gelman, Xiao-Li Meng, and Hal Stern. Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6:733-880 807, 1996.
[57] Kenneth Arrow, Theodore Harris, and Jacob Marschak. Optimal inventory policy. Economet- rica: Journal of the Econometric Society, pages 250-272, 1951.
[58] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022. arXiv preprint mtXiv:2202.00512.
[59] Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023. arXiv preprint arXiv:2303.014:
[60] Roger Koenker and Kevin F Hallock. Quantile regression. Journal of Economic Perspectives, 15(4):143-18) 156, 2001.
Acronyms
CRPS continuous ranked probability score. 8,
GBT gradient-boosted tree. 2, 3, 4, 5, 16
MAE mean absolute error. 8,
RMSE root mean squared error. 8, 9
SDE stochastic differential equation. 3, 4, 6, 14, 15, 16
13

---

A A primer on unconditional diffusion models
A diffusion model is a generative model that consists of two processes, forward diffusion, and reverse diffusion. The forward process takes an unknown target distribution x (3) and continuously transforms it into simple distribution Psimple (3). typically Gaussian. It does by progressively adding noise to samples from т, such as our data.
The reverse process transforms the simple distribution back into the target distribution by denoising any perturbed samples from Primple- If we can learn the reverse process, we can effectively access the target distribution æ: first draw samples from Psimple and then use the reverse transformation, as guided by the score, to map them back to π.
One way to learn this noising-denoisee process is to estimate the score function, the gradient of the log probability of the noisy data with respect to the data itself.
We detail the three main components of score- based diffusions the forward process, the reverse process, and the estimation of the score function-below.
The forward diffusion process.
A diffusion model is described by the following generative process:
Draw (()) ~ n. Evolve (t) until T accorddng to: dy(t)= (()(t) t)dt g(t)dw(tt)
(14)
where the process of evolving the stochastic differential equation (SDE) involves a standard Brownian motion wt) and model parameters f Rdd x [0, T + R and g : [0, T] + R. respectively called the drift and diffusioo functions.
The SDE in Eq. (14) determines the evolution of yt) over time, thereby inducing a distribution of (t)) at each t, that write (c)((t). We further write the distribution of y(t) conditional another (u) as Pue (B(() y())). . In practice, f and g are simple functions such that pae (v)() y)0)) has a closed form. In fact, if the drift f is affine in (t) and the diffusion function g is scalar that does not depend on y(t), the distribution por is always Gaussian, with a mean we write m(t; (0)) and covariance multiple of the identity we write a (1)=1.
W choose the time horizon T such that the resulting marginal distribution pr (T) = Sy Pr (T) y() = y' (1y ddy' is simple distribution Psimple agnostic to æ. The intuition is that as the diffusion progresses, the initial density ι is forgotten.
We detail our choice of f. 9 and T in Appendix C, which is the variance exploding SDE (VESDE) in Song et al. [11]. This choice induces a trivial closed form expression for the functions m and a.
There are no model parameters to learn in the diffusion model, and the conditional distributions Poe (3( (0)) are trivial to compute. The challenge is to learn how to compute (or sample from) the conditional distribution pre (ty(t) (T)).. This is called reversing the diffusion process. Knowing pre is key to learning the target distribution æ,, which can be written π (3)( = Epr [Pro (v(0) y (T))] where pr = Psimple is known.
The reverse diffusion process.
Consider the following SDE that is reversed in time from T to 0,
{ ū(T) Prample dÿ. = ()(((),)) g())2 @ce) lo + (()))
(15)
where ti(t) is standard Brownian with reversed time and p; is the density of yt) defined by the
forward SDE (1). Similar to the forward case, the reverse SDE induces a distribution p. for g(t) at each t. Anderson [18] shows that, if we denote by Pry the conditional distribution of ÿtt given
(T),, then Prv- = Pre for any time t. In other words, (t) (T)) = a and (t) (T) = a have the same distribution for all Since designed the diffusion such that pr Psimple pr, we have in particular (t) ~ yt).
SDE solvers let us sample from distributions that are solutions of an SDE; hence, we can obtain samples of rgy) by solving Eq. (15). However, the function y + V, log) called the score, usually unknown and needs to be estimated.
14

---

Estimating the score. Vincent [19] shows that the score can be estimated from trajectories of the SDE as the minimizer of the following objective function:
y + V. log pe(y) = arg min E,E,,, SES
V v(e) Por por (3(() ((0) s(y((), t)1) (16)
where S {s: Rdd [0,7] + R is the set of all possible score functions indexed by time t, and Eg can be an expectation over any distribution of whose support is exactly [0, 7". Since por is normal distribution (mm(t: (00),,
have
V Vyce) Por por (y(1) y(0) m(t: y(0)) y(t)
(17)
(1))
In practice, the expectations in Eq. (16) are approximated using the empirical distribution formed by the observed y for E,, and by sampling the simple and known forward SDE trajectories y ( y for Epec The objective can then be minimized by parametrizing s with any function approximator, e.g., neural network. The score estimator can then be used to reverse the diffusion process and estimate the target distribution (1y).
B Proofs of the main theorems
In this section, we provide the proofs for the main theorems in the text. Theorem (Optimal Conditional Objective). Define S as the solution of Eq. (9). Then, for almost all x, y, L with respect to π (a. y) and the Lebesgue measure on E [0, T), we have
S (.. t, Vy log Pæ.
(10)
Proof. For conciseness, define for any function s Rd, [0, Rdd, the quantity: r(æ: s)
(g)) E. pm.o= Via (e)log Pou (t) (0)) s yy (t) 21 Also write for any function
S: RRd [0, Rd By and Red the function S (y, () S(y,,,,)) (.
The function S* is characterized as, S* E arg minges: (r(m) Sz) where S+ = {S : x [0, d. +Rdd
Define Ω = (xc E Rdd (y, (, x)) @ arg min, r(æ,, s)), we will show that TS2) = 0. By [19], we know that Vy Pa, (y) arg min, r(æ, 8) for all so by definition of Ω we have: Vo Ω ((, r(æ, (y, 1) V. log)).(())).
(18)
We further have for any other x & Q2 ræx 5 = r(x, (v. 1) + V (o.)))
If (S2) > then integrating Eq. (18) over (xx will yield Ir(x, St)]
Ex [r(x, ((v, t, x)) Vy Pp))) which is not possible by definition of S Hence
(Ω)) 0.
Hence we have S E argmin, r(x, s) for almost any x, so by [19] again, we can conclude that S (y, = v, log (yy) for almost any x, f, y.
Theorem 2 (Treeffuser One- One-Dimensional Objectives). Denote U (UT, U2,). Then for almost all æ, y, l with respect to x(æ y) and the Lebesgue measure on l E [0, T), we have
U t,
V log (mg
(13)
o(1)
Proof. By definition in Eq. (12), we have
U arg min E. yo E, E,Ee [Gx - -3((((A)))))
15

---

With Theorem 1, we have (y, (y,t, x) - Vy log (y) almost everywhere, with S defined as
S E arg min lE (o Boa,on (o (yx yn (0)) S(y (1), SES+
We have:
E (xx) (ebr pe
(o) (3 yz) 5 (1),
(19)
m(t yk (0)) (t)
(2
= (2)) E (u) Bo
S (t), x
(20)
(t))
m(t: yoo (0))s (t)x ((t))
((x)) EgL (0))
(21)
(t))
o(t)
Ba(
(B Be
(o(t) (- - (t))oo((((+++ (())),))00))))))))
1
(22)
where we used the following facts:
from Eq. (19) Eq. (20): the closed-form expression of the score S from Eq. (17),
from Eq. (20) to Eq. (21): expanding the norm and switching the expectations with the finite sum over k;
from Eq. (21) to Eq. (22): reparametrizing the expectation ya (t) yo (0) which by definition is normal distribution with mean m(t; (0)) and variance ot)
The final manipulation remember that theorem and the theorems from Vincent [19] are valid with expectations E, against any strictly positive measure of f over [0. T7. In particular, we can absorb o(B) as a reweighted non -onneetaevve measure, and we obtain exactly the definition of Ug by defining (yy,t, a St) + ott) which concludes the proof.
C Treeffuser default configuration
We the following configuration defaults for Treeffuser.
Forward diffusion. We use the variance exploding SDE [11] for all experiments, defined by setting:
(t)
fly, = and g(t) V where is given increasing function defined by,
(23)
dt
O'max
(tt) = Crmin
(24)
O'min
and Cenin = 0.01 and Crmax = 20. For all our experiments we let t E [0, (i.e. T == 1).
Gradient- boosted tree (GBT) parameters and dataset repetitions. We provide a short description of each hyper-parameter the model alongside the default value. Treeffuser uses LightGBM [22] to learn the GBTss
estimators (3000): Specifies the maximum number of trees that will be fit, regardless whether the stopping criterion is met.
learning rate (0.1): Specifies the shrinkage use for every tree.
num leaves (31): Specifies the maximum number of leaves tree can have.
early stopping rounds (50): Specifies how long wait without a validation loss improvement before stopping.
n repeats (30): Specifies how many Monte Carlo samples to draw per data point to estimate E,, in equation Eq. (9).
16

---

03
-10
(a) Simulation I: branching Gaussian mixture.
(b) Simulation 2: shifted and inflated Gamma.
Figure 4: Visualization of ground-trun samples for the one-dimensional synthetic datasets used in the empirical studies.
D Experiments: description of computer resources
All benchmark tasks presented in Section 5.1 and Section 5.2 were run cluster with one job per triplet of (model, dataset, split index).
The real world experiments totalled 1135h for 700 tasks (10 datasets, 7 methods, 10 splits). Each task was allocated cpus.
The synthetic experiments totalled 290 hours, for 800 tasks (16 datasets, 5 methods, 10 splits). Each task was allocated CPU.
The M5 experiments were run on a single MacBook Pro over a couple hours. Other figures such as Fig. 1 were also generated MacBook pro in few seconds.
E Experiments on synthetic data (supplement)
E.1 Arbitrarily complex synthetic data
We provide details on the three synthetic data experiments presented in Section 5.1 and illustrated in Fig. 1. Each experiment introduces different distribution of the response variable y given the covariate x. The first experiment generates multimodal responses from a branching Gaussian mixture, the second experiment generates inflated responses from mixture of shifted Gamma distribution and an atomic measure, and the third experiment generates two-dimensional responses from nonlinear multioutput regression.
provide additional visualization of samples the one-dimensional datasets in Fig. 4.
We assume that x ~ Uniform(O0 (0, 1]) and model the conditional distribution of the response y given differently for each of the synthetic dataset. Scatter plots of synthetic data from these distributions are provided in Fig. 4
Models
Branching Gaussian mixture. We generate multimodal responses from a mixture of equally-weigheee Gaussians. our experiments, we the scale = 0.05 and the number of components and their means scale with as follows:
y x GaussianMixture,,C (I,, -x),
0 x 1/3;
GaussianMixture, 2x,2/3 I, -x);
1/3 $ 2/3;
(25)
GaussianMixture, (r/4/3 x, 2/3 3/3 -x,- -.-x);
2/3 $ 1
where Mixtusan .......; , denotes Gaussian mixture distribution with K equally- weighted components, each with mean μa and scale

---

Covariate inflated GaMma. We generate inflated responses through the following mixture of a shifted Gamma distribution and an atomic measure:
ShiftedCanma(k,,.. 0,x), 2 0;
Bernouli(pp),
(26)
2 1;
where ShiftedGamma(k; 0, 2) denotes Gamma distribution with scale and shape 0 shifted by æ. In words, given
y to be equal to with probability p, and to be drawn from Gamma shifted by z otherwise. Hence, the conditional distribution of y given æ is r-infflted and its support [z, 00) changes with In our experiments,
p 0.15, k = 2, A 1.
Nonlinear multioutput regression. We generate two-dimensional responses from a density p(y 0.05, dist(a, B) measures the shortest Euclidean distance from E IR to the set B C RR2 and (ax)) circular centered (0, 0) with radius radius( and angles of extremities 01, 02 evolving as functions of z. Intuitively, they interpolate the circular are represented in Section 1.
2) X exp ((a))) where
20a
The functions are defined as follows.
If x S 0.5
radius(x) = ((x) 1+r(z)
0o(x)
(z) = (())
where (xx) (0.5 - x0)6-x//005-000 0.17), r(x) ((x).
If x 0.5
radius(z) x)) = {(x) 0.1 0.1+
(x(x) {(x) 0.120..1 0.125 0.45,
(x)) f(x) 1.12))
where f(x) (0.83 =x)((0833 0.5). r(z) = 11-((x)
E.2 Data with normal likelihood
We study the performance of Treeffuser in a setting where a simpler model is better adjusted to the generating process. In particular, we let N be number of data points sampled and d, the dimension and use a linear generative model
8 Na.,
Ia,, iE [N]
Y N ((10082,00) ie [N]
where ²² = 1. The results are provided in table 3 using the same configuration described in section
section 5.2.
Results From Table 3, we observe that while Treeffuser remains competitive, Deep Ensembles and NGBoost now demonstrate either equal or very close performance. This is expected since both models assume a normal distribution for the outcomes and the model assumed by Deep Ensembles closely resembles the real generative model. Interestingly, Treeffuser outperforms both iBUG and quantile regression (QReg), suggesting that its superior performance is not merely due to using LightGBM XGBoost. Finally, we note that Treeffuser continues to perform well in low-data situations compared to other models.
F Experiments on standard datasets (supplement)
F1 Baseline methods.
We briefly describe the methods used baselines in our empirical studies.
18

---

N
dx
Deep Ens.
IBUG
NGBoost (oracle)
QReg
Treeffuser
(oracle)
(oracle)
100
1
2.62+1..7
8.65=10.30
4.08
5.00+4.50
4.01 +4.11
x10
100
5
3.72+1.60
1.45
1.75
4.69+2.02
3.86++1..6
x10-1
100
10
1.04
5.31+2223
1.58
2.13+22.20
4.90:22.08
x10
100
20
4.27:2.05
5.15#22.9
4.49 =2000
5..04
4.63++2.19
x10
500
1
0.84+0.41
1.58+0.0B
0.87=
1.36++0.73
1.11#0.44
x10
500
5
3.19+0.27
4.00=0.35
3.82
4.44+0.74
3.85+0051
x10
500
10
3.44+0.71
4.25=#0.87
4.03++0.00
1..1
3.98++0.88
X10
500
20
3.6910.81
1..04
4.50
4.80
1.3210.91
X10
1000
1
2.51+1008
10.59=2.57
7.10#2.09
11.85+2..4
9.72+2.47
X10
1000
5
3.18+0.26
3.67
3.73+t0.33
3.66+0.34
3.52
x10
1000
10
3.29:0.42
4.01:0.63
4.01
7.23
3.95:1.59
X10
1000
20
3.4010.29
4.25.1034
4.1110.47
4.27:00.9
4.0010.38
X10
5000
1
0.5610.25
9.2511271
3.87
10.36
10.0012.49
X10
5000
5
3.2710.26
3.60
3.63
3.63
3.63
5000
10
3.18:10.18
3.82+0.26
3.6022.25
3.65+0.27
3.59:0.25
X10
5000
20
3.21:1.20
3.8610.23
3.90
3.80+0.24
3.69:0.22
x10
Table CRPS (lower better) by dataset and method. Treeffuser provides comparable predictions to methods with parametric assumptions that match the generating process. The standard deviations are measured with 10-fold cross-validation.
NGBoost (parametric, tree-based). NGBoost [14] models the conditional distribution of the target variable using parametric family of distributions whose parameters predicted by gradient- boosting algorithm. uses natural gradients for more stable, accurate, and faster learning. In our experiments, set the parametric model to be Gaussian.
Deep ensembles (parametric, nnet-based). A Deep ensemble [17] is a collection of neural networks that are individually trained to model a parametric conditional distribution p(y[zz). The neural networks are then combined to obtain a mixture. We set the parametric model to be Gaussian.
Quantile regression (nonparametric, tree-based), Quantile regression [60] estimates the quantiles of the conditional distribution p(y)z)) We implemented GBT-based version of the method by fitting trees with inputs and where q is the probability of the desired quantile. During training, we optimized the objective F.,, Fz. (Q((z. y), 4)], where the pinball loss and g sampled from a zero-one uniform distribution. In our experiments, we used LightGBM as GBT method.
IBUG (parametric, tree-based). IBUG [15] extends any GBT into a probabilistic estimator. Given an input instance it outputs distribution around the prediction using the k-nearest training data points. The distance between training instance Vy and depends on the number of co-occurrences of and z across the leaves of the ensemble. In our experiments, we used XGBoost and Gaussian likelihood. following the default specifications of the method.
DRF (nomparametric, tree-based). DRF [50] grows a random forest by maximizing the differences in the response distributions across split groups with distributional metric. Repeated randomization induces a weighting function that assesses how relevant a training data point is to a given input x. These weights are used return weighted empirical distribution of the response.
F.2 Hyperparameters of the methods.
Unless specified otherwise, tuned the hyperparameters of each method using Bayesian optimiza- tion for 25 iterations. We used the following search space for each method (where [a, 6] denotes the of integers from a to b):
Treeffuser
estimators [100, 3000]
repeats [10,50)
learning rate [0.01, 1] (log-uniform)
early stopping rounds E [10, 100]
num leaves E [10, 100]
19

---

Quantile regression (QReg)
estimators E [100, 3000] (log-uniform)
n repeats [10, 100]
learning rate [0.01,
early stopping rounds E [10, 100]
num leaves [10]
IBUG
- k E [20, 250]
n estimators 1000]
learning rate [0.01, 0.5] (log-uniform)
max depth 100
DRF
minn node size E [5. 30]
num trees E [250, 3000]
NGBOOST
estimators [100, 10000]
learning rate [0.005, 0.2
Deep ensemble
layers E [1,5]
hidden size E [10,500 500
learning rate [10-5, 10-2) (log-uniform)
ensembles E [2, 10]
For the methods that only return point predictions, we used the following search spaces:
XGBoost
estimators 1000
-laanning rate [0.01, 0.5] (log-uniform)
max depth E [1, 100]
LightGBM
estimators 1000]
-learning rate [0.01, 0.5] (log-uniform)
num leaves E [10, 100]
early stopping roundsE [10, 100]
20

---

F33
Accuracy and calibration results.
dataset
N. N. dy
Deep ensembles
NGBoost (Gaussian)
iBUG (XGBoost)
Quantile regression
DRF
Treeffuser
Treeffuser
(no tuning)
bike
17379,12,1
3. 0.
11.52
4.11
4.63::02
4.86
3.69:0.14
3.81
X10'
energy
768, 8.
11.34:22.8
11.41
NA
18.7321.08
8.32
8.79-1446
x10-1
kin8am
8192,
0.64:0.02
1.71
1.38+0.0s
1.18:0.04
1.7240.04
1.06=0.08
1.06=0.02
movies
7415,
9.35
X
9.83
18.23
15.87
8.85
8,63
x10"
naval
11934.17.1
10,42=1.00
8.66=0061
6.46+1.07
148.80
38.880=36.50
11.49:0.98
4.40=0.26
4.92=0.57
news
19644,58,1
1.95
X
1.18
1.12::.46
1.08:0.41
1.10:0.10
1.09
X10*
power
9568,
3.81
3.66=0.as
3.06++..2
36.27+4.56
3.65+0.55
2.93=0.30
3.02:0.30
x10"
supere.
21263,81,1
11.26:a.74
11.25
9.67
9.36
10.7940.58
9.08
9.21=0.53
x10"
wine
6497, 12. 1
6.54=0017
6. 6.85=00.18
6,02+0.25
6.28
6.81#0.21
5.88=0.17
5.98=0.19
x10
yacht 308,
6,
1.06a046
0. 0.83:10.31
0.83#0.81
1.22:0.74
2.30
2.10=1.08
2.08+0.45
Table 4: RMSE (lower is better) by dataset and method. indicates the method failed to run, and
NA that the method is not directly applicable to multivariate outputs. The standard deviations are measured with 10-fold vols-valiiation. For each dataset, the two best methods are bolded. Treeffuser provides the most accurate probabilistic predictions, even when initialized with defaults.
dataset
N, dz. dg
Deep
NGBoost
iBUG (XGBoost)
Quantile regression
DRF
Treeffuser
Treeffuser tuning)
ensembles
(Gaussian)
bike
17379.12.1
3.28+
17.79
5.77=11.0
2.55
3.45+a.48
1.73-0.62
0.86=0.7
energy
768. 8,
8.08:1.41
5.82=1.74
NA
NA
4. 4.54:0.74
4.11=1.14
5.53+1.42
x10-
kin8nm
8192.
2.58#1 so
4.7020.44
2.90
8.47.0.50
2.79:0.82
3.85
3.34
x10
movies
7415,
5.75+1138
7.56=3.a1
49. 09.
1.31:047
2.71-1.0
5.17+1.22
naval
11934,17,1
18.31
5.50 m5
3. 3.866122.5 59
15.05
17.3720-40
7.7520.41
9.1020.39
x16-2
news
39644,58.1
11.36+a0.s
X
14.43+1.82
18.9620.42
1.14:0.25
3.50=0.70
4.87+0.28
x10
power
9568. 4.
2.52#1.18
2.53+1000
3.67=1142
4.75
2.37=0.58
1.36+1a1
1.77=0.74
superc.
21263,81,1
3.19
2.75
5.39
5.47100.44
3.87
1.34_0.10
1.33-016
x10
wine
6497, 12,
1.92=0.s
2.68:2..5
4.72:00.s
4.36
22.02:1:15
5.70+0.55
5.22+0.09
x10
yacht
308.
13.35
9.01=241
9.06#8.26
8.26+2..0
7.15#2.60
7.27-268
10.02:2.0
Table 5: MACE (lower is better) by dataset and method. X indicates the method failed to run, and
NA that the method not directly applicable to multivariate outputs. The standard deviations are measured with 10-fold cross-validation. For each dataset, the two best methods are bolded. Treeffuser has a competitive calibration error across most datasets.
dataset
N, dy
iBUG (XGBonst)
XGBoost
LightGBM
Treeffuser
Treeffuser (no tuning)
bike
17379.12,1
4.11 + 0.19
3,72 = 0.13
3.79 1 0.12
3.69 + 0.14
3.81 + 0.15
x10'
kin8nm
8192, 8,
1.38 + 0.08
1.12 302=00m 0.04
1.04 + 0.03
1.06
+ 0.03
1.06 + 0.02
x10-1
movies
7415, 9.
9,83 1.44
8.96 1.60
9.08 + 1.76
8.85 + 1.66
8.63 + 1.38
x107
naval
11934.17,1
6.46 1.07
5.11 111=0.45 0.45
6.68 + 1.53
4.40 + 0.26
4.92 + 0.57
x10-
news
39644.58.1
0.40
1.09 0.40
1.09 1 0.40
1.10 - 0.40
1.09 + 0.41
x10*
power
9568, 4,
3.06 4.06-0.32
2.97 0.33
2.99 + 0.32
2.93 + 0.30
3.02 + 0.30
x10"
supere.
21263,81,1
9.67 0.54
9.04 0.48
9.21 + 0.44
9.08 0.48
9.21 + 0.53
x10"
wine
6497, 12,
6.02 = 0.26
5.95
0.23
5.90 1 0.27
5,88 1 0.17
5.98
4 0.19
x10-1
yacht
308, 6,
0.83 = 0.31
2. = 1.82
0.67 + 0.40
2.10
+ 1.08
2.08
0.45
x10-1
Table 6: RMSE (lower is hetter) by dataset and method. This table extends Table 4 by includes vanilla XGBoost and LightGBM, alongside iBUG and Treeffuser. The standard deviations are measured with 10-fold cross-validation. For each dataset, the two best methods are bolded.
G Ablation study on noise scaling
We evaluate the impact of including noise scaling the score parametrization of Eq. (13), specifically S(y. z.) U Uty I, 1, ((()) where S(y, ([y,,,a) .)) denotes the score function of y(() xx, U the gradient- boosted trees, and a (t) the noise schedule of the diffusion process. We compare Treeffuser with and without noise scaling under the same experimental setup of Section 5.2.
21

---

Table 7 presents the CRPS by dataset and method. Treeffuser without noise scaling fails to achieve any reasonable CRPS. We believe Treeffuser without noise scaling may require extra model capacity (e.g.. deeper trees and more training iterations) to handle the varying noise levels adequately. However, we did not further optimize it, as noise scaling provides state-of-the-an results without the need for additional complexity.
Treeffuser
bike
energy
kin8nm
naval
power
superc.
yach!
with scaling 1 1.00a0.0
3.0 3.07
5. 88 14
2.02 os
1.98 62.1
1.49+0.07
3.52=0
2.59
3.11 186822-1106
scaling
176
52581
204
154
1124162
30
x30
x10
xiu
x30
XII
x
X1I
x10
Table 7: CRPS (lower better) by dataset for Treeffuser with and without the noise scaling
reparametrization in Eq. (13). Without scaling, Treeffuser does not produce meaningful results.
H Runtime and Complexity
m5
movics
bike
energy
kin8nm
naval
news
power
superc.
wine
yacht
1.33
0.68
1.33
0.55
1.23
1.46
1.31
1.15
0.85
0.56
0.28
sample
Table 8: Average running time in milliseconds for producing a sample from the model.
To demonstrate the efficiency of Treeffuser, we present the results of the following experiments and benchmarks:
1. We run Treeffuser on subsets of the M5 dataset (section 5.3) of various sizes and report the training time Fig. 5.
2. We report the training time for rumning Treeffuser the datasets used in the paper in Fig. 6.
3. We report in Table 8 the average runtime for generating single sample after training Treeffuser. The average calculated by sampling five thousand points after training the model with default settings and 50 discretization steps.
We conducted all these experiments on a 2020 MacBook Pro with 2.6 GHz 6-Core Intel Core i7 processor.
From these results we find that:
Treeffuser is very fast at fitting moderately sized datasets, with runtime increasing linearly with dataset size.
Sampling single points is very fast (ss 10-3 seconds) yet, drawing many samples can become significant, e.g., with large test set.
With respect to the time complexity of the model we note the following. If the time complexity of fitting a single GBT is of order (FF(D))) where F R+ - R4 and [D] is the size of the dataset, then the time complexity of fitting Treeffuser is O x F(n_repeats x [D)),, where is the dimension of y and n_repeats is the parameter that determines how many noisy versions of the dataset are sampled, Similarly, assuming constant time per GBT evaluation, the complexity of sampling is O(n_ samples x d_discretization steess, where n n_samples is the number of samples drawn from the model, and n_ n_discretizationsster steeps the number of function evaluations used by the SDE solver.
22

---

m5
100
(90000, 27)
Mean Training Time (seconds)
(81000, 27)
m5
80
m5
(72000, 27)
(63000, 27)
m5 (64000. 27)
(46000, 27)
m5
(36000. 27)
40
(27000, 27)
m5
(18000,
m5
(9000,
Number of Samples in Dataset
Figure 5: Dataset size vs training time on subsets of the M5 dataset. Error bars are computed over runs. Treeffuser training speed grows linearly with the size the training points.
Mean Training Time (seconds)
100
m5
(90000,
(39644,
(7415, 9)
(9568, 4)
bike
superc
40
(17379, 12)
(21263, B1)
energy
wine (6497,
kin8nm (B192,
(308,
naval (11934, 17)
20000
60000 Number of Samples in Dataset
40000
100000
Figure 6: Dataset size vs training time on benchmark datasets. Error bars are computed over 5 runs. Treeffuser trains under 120 seconds for all datasets.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose a method for probabilistic prediction from tabular data. In our abstract and introduction, make two central claims:
Treeffuser is accurate, in that it outperforms state-of-the-ar art methods in probabilistic prediction from tabular data on standard UCI datasets. This is discussed detail in Section 5.2 and Appendix F. Table 1, Table 4, and Table present the results.
Treeffuser is flexible, in that it adapts to complex distributions. This is shown in Fig. I, which presents the results of our synthetic experiments, and in Section 5.1 and Appendix E, which provide the details.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our method in Section 3.4. Treeffuser has two limitations: the sampling process can become expensive when many samples are required for given input instance the diffusion model is specifically designed to generate only continuous responses, which may not be suitable for count data. However, in Section 5.3, demonstrate that Treeffuser outperforms other methods
this kind of data.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We state two theorems, Theorem and Theorem 2, and discuss their implica- tions in Section 3.1 and Section 3.2, respectively. Proof's are given in Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The details needed to reproduce our experiments are stated in Section and Appendices C, E and F. We also provide an implementation of Treeffuser in the supplemental material. The notebooks with the experiments can be found in wwtpo/ooooooppppppeeeeee
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
25

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide an implementation of Treeffuser in the supplemental material. The notebooks with the experiments can be found in wwwebe/d//ttbooos///oooooooo pttpt//eeeepeeeeoooosssseeeee _nooebooks. For our experiments on real data, we use publicly available datasets and point to their references [51, 55].
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr ca/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //iips.co. wttpccccpppppeeeeeecccecccccccet / cettmmisssooooiiyoyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details needed to reproduce our experiments are stated Section 5 and Appendices C, E and F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
26

---

7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All the accuracy and calibration metrics reported in our tables present standard deviations computed with 10-fold cross validation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we provide details Appendix D: the number tasks, the number of cpus and the total time taken. The memory is not reported as our work has no particular needs for memory and no particular configuration was necessary.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the ines?
NeurlPS Code of Ethics https: //tearr////eisss// httpp://nerrpp..cpppllpcc///////
Answer: [Yes]
Justification: We provide full transparency in our methodologics and support reproducibility through shared code and datasets. There no conflicts of interest to declare.
Guidelines:
27

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., dep.oymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We not releasing pre-trained models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
28

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: For experiments real data, use publicly available datasets and point to their references [51, 55J.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi wwwoee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide an implementation Treeffuser the supplemental material, which includes an anonymized zip file containing notebooks that document the usage of our method.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
29

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "hnviGmIFHH", "reviewer": "Reviewer_iWzb", "summary": "This paper proposed a method called \"Treeffuser\" for probabilistic prediction (density regression) of tabular data.", "questions": "What are the potential applications of the proposed method, especialy given that (1) it's designed for tabular data, and (2) it is generative but cannot provide density evaluation?", "limitations": "My main concern about this proposed method is that it cannot provide exact density evaluation. This is also discussed in the limitations section by the authors.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and clearly presented the core idea as well as the main results.", "weaknesses": "There should be more related methods to compare with in the experiment in section 5.3, especially more recent ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal reviewer iWzb\nWe thank reviewer iWzb for their comments. We appreciate that the reviewer found our paper to be well-written and clearly presented. We appreciate the great questions and suggestions by the reviewer. We provide answers and discussions below.\n\n\n## Weaknesses\n\n**W1. There should be more related methods to compare with in the experiment in section 5.3, especially more recent ones.**\n\n\nWe thank the reviewer for the suggestion. We designed Section 5.3 to be a concrete example showing how simple it is to use Treeffuser for real-world applications. This section was not intended to be an extensive comparison like Section 5.2.\n\nWe selected the methods in Section 5.3 because they performed the best in Table 2, including the recent iBUG [1]. They also represent two types of methods with interesting properties for comparison with Treeffuser: a “hand-crafted” log-likelihood specially suited  for the problem (NGBoost Poisson), and a  “likelihood-free” tree-based model (amortized quantile regression). We omitted other methods in Figures 2 and 3 to enhance the figures’ legibility.\n\n## Questions\n**Q1. What are the potential applications of the proposed method, especialy given that (1) it's designed for tabular data, and (2) it is generative but cannot provide density evaluation?**\n\nThank you for your question:\n1. Methods for predictions from tabular data are one of the most frequent uses of machine learning in practical industrial applications (e.g. sklearn [2], or methods such as xgboost [3]). Uncertainty estimation in the form of modeling $p(y \\mid x)$ is equally important in those applications and for risk-aware decision making [4-7]. \n\n2. We do not focus on providing density evaluation because it is unnecessary for our purposes. In practice, we aim to model $p(y \\mid x)$ to make real-world decisions based on quantities such as $\\mathbb{E}[f(y, x) \\mid x]$. These quantities are purely evaluated from samples of $y \\mid x$. For example, a user might be interested in the expected output of their factory given specific settings, the standard deviation, the expected profit, or the probability that profit falls below a certain threshold $c$, where their profit function $f(y, x)$ is a black-box function. With Treeffuser, we can model and compute quantities by using the samples to compute $\\mathbb{E}[y \\mid x]$, $\\mathbb{E}[y^2 \\mid x] - \\mathbb{E}[y \\mid x]^2$, $\\mathbb{E}[f(y, x) \\mid x]$, and $\\mathbb{E}[\\mathbb{1}_{f(y, x) \\leq c} \\mid x]$, respectively. In contrast, with density estimation $p(y \\mid x)$, evaluating $\\mathbb{E}[f(y, x) \\mid x]$ or even $\\mathbb{E}[y \\mid x]$ would likely require advanced techniques to sample from $p(y \\mid x)$ and use Monte Carlo estimators to compute these quantities. Treeffuser performs these computations directly.\n\n\nTo re-emphasize, while having an exact density evaluation could be useful in some scenarios, we believe having sampling is as important if not more important. In fact, the ability to sample from a density is an active research area (inverse transform sampling, rejection sampling), and Treeffuser enables sampling directly without modeling the density.\n\nWe have now expanded the paper’s introduction with the discussion above to emphasize that the focus of the paper is sampling from $p(y \\mid x)$ instead of approximating the value $p(y\\mid x)$. Thank you for helping us clarify any confusion about the goal and impact of Treeffuser.\n\n## Limitations\n\n**L1. My main concern about this proposed method is that it cannot provide exact density evaluation. This is also discussed in the limitations section by the authors.**\n\n\nAs detailed above, we did not aim to do exact density evaluation because the downstream applications we are focusing on do not need it. We focused on designing a method that provides high-quality samples, which is more important in practice. We believe that focusing on sampling is a strength, not a weakness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4f6WHz2ZRg", "reviewer": "Reviewer_3Apn", "summary": "The authors propose a methodology for computing probabilistic predictions in regression problems using diffusion and gradient-boosted trees. In particular, their methodology can generate samples from p(y | x) from which statistics such as estimated quantiles could be recovered. They find that their method can learn flexible and non-standard distributions well and also works well on real-world datasets.", "questions": "I'm left with some basic questions about the method. What part of the contribution is due to the specific way the diffusion process was modeled and what part is due to the use of trees? How does the method differ from other diffusion-based approaches to probabilistic prediction such as the cited CARD paper? What were the key breakthroughs (if there are differences) in how you set up the diffusion problem?\n\nTagasovska and Lopez-Paz's Single-Model Uncertainties for Deep Learning (NeurIPS 2019) is a relevant work to compare to here. If I understand right it's essentially the same as your quantile regression comparison except you use trees instead of neural networks to learn f(x, q). How would the method perform with neural networks?\n\nHow easy is the SDE solver to stitch with the outputs of the trees?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method results in a tractable standard supervised-ML problem, where the dataset is an augmented version of the original dataset.\n\nThe method produces good results on a range of datasets.\n\nThe paper goes into detail on the relevant diffusion equations governing the process.\n\nI liked the clever application with the newsvendor problem.", "weaknesses": "The work is not clear about its contribution and what parts of the diffusion-based setup are new vs borrowed from existing works.\n\nThere could be more and better empirical comparisons.\n\nNotation and clarity could be improved.\n* Line 129 defines a vector a but that is not used in equation (11).\n* Not obvious how eqn (11) simplifies to eqn (12).\n\nThe proposed method may not be easily used or widely applicable because of the complex process needed to generate samples involving solving differential equations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal Reviewer 3Apn\n\nWe thank Reviewer 3Apn for their valuable feedback. We are pleased to hear that they appreciate our application to the newsvendor problem. Based on their comments and questions, we clarify our contributions, the applicability of our method, and its implementation below.\n\n## Weaknesses\n\n**W1. The work is not clear about its contribution and what parts of the diffusion-based setup are new vs borrowed from existing works.**\n\nPlease see our answer to Q1. \n\n**W2. There could be more and better empirical comparisons.**\n\nThank you for the comment. We selected five strong approaches that practitioners might choose for probablisitc predictions. However, we look forward to comparing the performance of Treeffuser (our method) against other methods.\n\nWhat are some specific datasets or methods you think would be valuable for comparison in our paper?\n\n**W3. Notation and clarity could be improved**\n\nThank you for the feedback.\n- The vector **a** is a dummy vector to explain notations, it is normal to not find it in the equation. We rephrased the sentence to avoid confusion.\n- The derivation of Eq. 12 was shown in Appendix Eqs. 19 to 22. We updated the paper to make sure this is also clear in the main text.\n\n**W4. The proposed method may not be easily used or widely applicable because of the complex process needed to generate samples involving solving differential equations.**\n\nThank you for raising this point, usability is very important to us. But we are surprised by this comment, because Treeffuser is actually _straightforward to use and widely applicable_. This is a key contribution of our work. We argue this is the case for the following reasons:\n\n- Concretely, we designed a simplified user experience by wrapping Treeffuser in a plug-and-play Python package that solves the differential equations for the user. With the accompanying package in the supplementary material, using Treeffuser is as simple as: \n    ```\n    model = Treeffuser()\n    model.fit(X,y)\n    samples = model.sample()\n    ```\n    \n- Offering such a simple user interface is only possible because we built Treeffuser with a flexible estimator (trees) and flexible model (diffusions). As a result, this simple training procedure is the same regardless of the type of data (e.g. heavy tailed, multi-modal, heteroskedastic, categorical, missing).\n- In practice, Treeffuser is accurate on diverse datasets, even without tuning (see our studies in Table 1, 2, and 4). \n\n    \nTo reiterate, while solving differential equations is indeed complex, it is not more complex than fitting boosted trees or optimizing thousands of neural network parameters. Efficient packages such as `xgboost`, `sklearn`, and `torch` made these tasks accessible to any user. With our `treeffuser` companion package, we aim to make diffusion-based probabilistic predictions for tabular data equally accessible.\n\n## Questions\n\n**Q1.1 What part of the contribution is due to the specific way the diffusion process was modeled and what part is due to the use of trees?**\n\nThank you for the question. We think that both parts cannot really be separated. The main contribution and goal is to use trees, but we had to adapt the diffusion model process to do so. (For example, trees only have a 1d output.)\n\nYet, you are right that the way we modeled the diffusion is important. For example, the score reparametrization on line 133 (and Eq. 13) is crucial for performance.\n\nTo highlight the importance of our design choices, we added this discussion to the main text, and included an ablation study on the score reparametrization in the Appendix.\n\n**Q1.2 How does the method differ from other diffusion-based approaches to probabilistic prediction such as the cited CARD paper?**\n\nThe differences between CARD[9] and Treeffuser are:\n1. CARD uses a specialized neural network architecture, while we use out-of-the-box gradient-boosted trees (e.g. XGBoost, LightGBM).\n2. CARD uses a discrete time approach [13-14], while we formulate our conditional diffusion problem with continuous SDEs [8], which we believe renders the theory easier to follow.\n3. Our approach requires much fewer function evaluations (i.e. 50 vs 1000) to generate a sample.\n4. CARD requires two training steps: a first model is trained to predict the mean of the target only, then a second model learns the distribution of the data using a diffusion that takes the prediction from the first model as input. Treeffuser directly trains the diffusion model on the target. It doesn't require training any auxiliary model for conditioning and is thus simpler.\n5. We attempted to use CARD in our experiments for comparison but were unable to obtain good results.\n\n**Q2. Tagasovska and Lopez-Paz's Single-Model Uncertainties for Deep Learning (NeurIPS 2019) is a relevant work to compare to here. [...] How would the method perform with neural networks?**\n\nThank you for sharing this reference. We were not aware of it but now include it in the text. Indeed you are correct, the baseline quantile regression used in our paper implements the same idea using trees instead of neural networks.\n\nIn fact, we originally tried a similar version of quantile regression with neural networks, and we found that on all our experiments, the tree based quantile regression performed better. This is in line with the observed dominance of GBTs over neural networks on tabular data [10-12].\n\n\n**Q3 How easy is the SDE solver to stitch with the outputs of the trees?**\n \nIntegrating the SDE solver with the outputs of the GBT is easy, akin to neural network based diffusions. Concretely, the trees are wrapped into a class with a `predict` function that returns the learned score, which is then fed to the SDE solver. \n\nWe ensured our code is modular to support any SDE solver, while hiding all the SDE complexity from the users. For example, `model.sample(n_samples)` calls the standard Euler-Maruyama SDE solver that handles the numerical integration automatically.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RGDJuNA90U", "reviewer": "Reviewer_knMV", "summary": "The paper proposes Treeffuser, a nonparametric method for modeling the output distribution. Treeffuser learns a diffusion model using gradient-boosted trees, and uses conditional diffusion models to produce a distribution over the response variable given an input vector. Since Treeffuser user gradient-boosted trees, it is adept at providing accurate predictive distributions for tabular data problems. Experiments on toy examples, real-world datasets, and a case study highlight the flexibility and effectiveness of Treeffuser for accurately modeling the output distribution for a wide range of use cases.", "questions": "- Do the authors have negative log likelihood (NLL) results?\n\n- I'm surprised Treeffuser performs better than iBUG for point predictions (Table 4) since iBUG is simply using the underlying GBDT model, do the authors have a hypothesis as to why Treeffuser performs better than XGBoost in terms of point performance?", "limitations": "Yes, the authors address the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The experimental results section demonstrates Treeffuser's effectiveness on complex contrived examples, real-world tabular regression datasets in comparison with existing methods, and a realistic case study.\n\n- The paper is well-written and relatively easy to follow.\n\n- The proposed approach is very flexible and nonparametric, enabling it to model any type of response distribution. This aspect is especially valuable for tabular regression problems, in which GBDTs do not inherently provide any type of uncertainty in their predictions.\n\n- Treeffuser works well even with default hyperparameters.", "weaknesses": "- As the authors note, Treeffuser must solve a stochastic differential equation to generate samples, which can become expensive if there are a large number of example to predict and each prediction requires the generation of a large number of samples.\n\n- No complexity or empirical runtime analysis is provided. This type of analysis would be greatly beneficial to readers and practitioners to help them decide if and when to use Treeffuser for their particular problem.\n\n- The comparison to other methods (Section 5.2) could be improved. For example, iBUG's $k$ hyperparameter is tuned using values [20, 150]; however, $k$ is a critical hyperparameter, and increasing the number of potential values may significantly increase iBUG's performance. Additionally, iBUG can non-parametrically model the response using kernel density estimation (KDE) which may improve its performance on datasets in which the output is not expected to be Gaussian.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal Reviewer knMV\n\nWe thank knMV for their review. We are encouraged that the reviewer found the paper well-written and easy to follow and that the author found our approach very flexible and valuable for tabular regression problems with UQ. Below, we provide additional results and discussion that answer the questions and concerns that the reviewer presented. The new experiments definitely strengthened the paper!\n\n## Weaknesses\n**W1. Treeffuser must solve a stochastic differential equation to generate samples, which can become expensive**\n**W2. No complexity or empirical runtime analysis provided**\n\nThank you for the great comments. In the paper, we provided some runtime analysis of Treeffuser lines 257-261. Yet, we agree that more runtime analyses are helpful for readers and practitioners. So we generated three new figures running Treeffuser with default parameters on a 2020 MacBook Pro.\n1. [Figure 1](https://imgur.com/a/Szo5ihU): runtime in seconds for training Treeffuser on subsets of different sizes of the M5 dataset (section 5.3). The shape of the training data is indicated on the plot. Error bars represent the standard error over five runs of Treeffuser.\n2. [Figure 2](https://imgur.com/a/CzsPy1Z): runtime in seconds for training Treeffuser on the other datasets used in the paper. Error bars represent the standard error of over five runs of Treeffuser.\n3. [Figure 3](https://imgur.com/a/5RQbM77): average runtime in seconds for producing one sample after training Treeffuser. The average is computed by sampling five thousand points from Treeffuser using its default settings of 50 discretization steps.\n\nFrom these results, we find that:\n- Treeffuser is very fast at fitting moderately sized datasets, with runtime increasing linearly with dataset size;\n- sampling a single point is very fast ($\\approx 10^{-4}$ seconds) and can be parallelized;\n- yet, drawing many samples can take more time, e.g., with a large test set.\n\nWe also agree with the suggestion of including a complexity analysis and will add a detailed discussion in the appendix. To be brief here, the complexity for training is the same as for LightGBM with the subtelty that the dataset size is multiplied by `n_repeats` (see Alg 1 and Appendix C), and a different model is trained for each dimension of $y$. The complexity for sampling is the same as evaluating the trees `n_sampling_steps * y_dim` times. \n\n\n\n**W.3 The comparison to other methods (Section 5.2) could be improved. For example, iBUG's hyperparameter [...] which may improve its performance on datasets in which the output is not Gaussian.**\n\nThank you for your suggestion. Following your advice, we\n1. increased the search space of $k$ using the method for optimal tuning of $k$ (Algorithm 3 in iBUG's paper)\n2. Implemented iBUG with KDE likelihood -- by tuning the kde bandwidth in [0.0001, 1000].\n\nThe results are available [here](https://imgur.com/a/D1x4k9S).\n\nThe conclusions are unchanged in both cases: Treeffuser outperforms iBUG (with and without KDE). Thank you for suggesting these great baselines; we have updated section 5.2 of our paper with them. We hope that this addresses your concerns.\n\n## Questions\n\n**Q1. Do the authors have negative log likelihood (NLL) results?**\n\nWe thank the reviewer for this question. We do not provide NLL results. While NLL can be approximated using KDEs on the samples outputted by Treeffuser, we found that KDEs are sensitive to the choice of bandwidth. We also note that exact likelihood computations are possible for diffusion models when the score estimator is continuous everywhere, as with neural-based diffusions (see section D.2 [8]). However, this is not applicable for GBT, because trees are not continuous.\n\nFortunately, this apparent limitation has no practical implications:\n- For evaluation, we use CRPS, which can be evaluated from samples. CRPS is a proper scoring rule [16] with advantages over NLL, such as better handling of tail probabilities [17]. CRPS is commonly used for probabilistic predictions [16,18].\n- For concrete downstream applications, we argue that being able to generate samples from $p(y\\mid x)$ is more important than raw values of $\\log p(y\\mid x)$.\n\n\n**Q2. I'm surprised Treeffuser performs better than iBUG for point predictions [...] Do the authors have a hypothesis as to why Treeffuser performs better than XGBoost in terms of point performance?**\n\nThank you for your question. We also anticipated that iBug would excel in point prediction since it relies on XGBoost. However, we are not surprised by the results.\n\n1. First, we nuance that Treeffuser only slightly outperforms iBug for point prediction, with iBug often very close behind.\n2. Second, we point out that iBug's point predictions do not mathematically coincide with XGBoost's. Indeed, the point predictions of a probabilistic method are computed as the expected mean: the empirical average of 50 samples from the modeled conditional probability distribution $p(y\\mid x)$. So if the conditional probability returned by iBUG has errors, then the point prediction might also have errors.\n\nIn light of your comment, we conducted additional experiments for point predictions against XGBoost and LightGBM directly (with hyperparameter tuning optimization).\n\nThe results are shown in [this table](https://imgur.com/a/4gNpSWd), where we also updated iBUG's point predictions with your suggested hyper-parameter tuning improvements which yielded similar but slightly improved results.\n\nAs expected, vanilla XGBoost and LightGBM outperform or tie with all the probabilistic methods. In particular, they are often comparable with Treeffuser, suggesting that Treeffuser provides probabilistic prediction without sacrificing point predictions of the mean.\n\nWe updated Table 4 with these results and added further discussion in the paper for clarity. Thank you for raising this point and suggesting the addition of more baselines; they strengthened the paper!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General rebuttal

We are grateful to the reviewers for their great feedback! Here, we summarize the key points from our rebuttals.

## Contributions

We clarify that our main contribution is to successfuly combine diffusion models and gradient-boosted trees (GBT), and use them to provide state-of-the-art probabilistic predictions on tabular data. Our approach combines the robustness and accuracy of GBT with the nonparametric flexibility of diffusion models. 

Treeffuser is straightforward to use and widely applicable, as it comes with a plug-and-play Python package and requires little to no tuning.

Using Treeffuser with default parameters requires three lines of code:
```
model = Treeffuser()
model.fit(X, y)
samples = model.sample(n_samples)
```
Thanks to the use of diffusions and trees, this approach works with both multi-dimensional and uni-dimensional responses, regardless of whether $p(y \mid x)$ is multi-modal, heteroskedastic, heavy-tailed, or follows a simple normal distribution. 

## Empirical experiments

We incorporated the reviewers feedback to expand our empirical experiments and strenghten our paper. More specifically,

 - We implemented a KDE-based version of IBUG and extended its hyperparameter search space. This improved the method’s performance, but Treeffuser still outperforms IBUG across most datasets.
 - We added vanilla XGBoost and LightGBM to the point prediction benchmark tables. As expected, they outperform or tie with all the probabilistic methods. In particular, they often tie with Treeffuser, suggesting that Treeffuser provides probabilistic prediction without sacrificing average point predictions.
 - We added an ablation study of Treeffuser where we removed the standard deviation reparametrization (line 133 and Eq. 13) and show that this design choice was important.
 - We conducted an empirical runtime analysis of the training and sampling processes. The results show that training is fast (less than 2 minutes for the datasets in the paper) and scales linearly with dataset size; sampling is also efficient ($\approx 10^{-4}$ seconds per sample) but slow for large sample sizes, yet it can be scaled with parallelization.


## Other feedback
We clarify that our focus is on generating samples rather than approximating the density $p(y|x)$. Practical applications require quantities such as expectations, standard deviations, quantiles, among others; while these quantities are hard to compute with densities, they can be readily computed with samples.

We also improved the clarity of our notation and derivations in our method section.

## References
These are the references cited in our rebuttals.

[1]  Jonathan Brophy, & Daniel Lowd. (2022). Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Trees.

[2]  Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, E. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12, 2825–2830.

[3] Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM.

[4] Gneiting, T., & Katzfuss, M. (2014). Probabilistic forecasting. Annual Review of Statistics and Its Application, 1(1), 125-151.

[5] Begoli, E., Bhattacharya, T., & Kusnezov, D. (2019). The need for uncertainty quantification in machine-assisted medical decision making. Nature Machine Intelligence, 1(1), 20-23.

[6] Jillian M. Clements, Di Xu, Nooshin Yousefi, & Dmitry Efimov. (2020). Sequential Deep Learning for Credit Risk Monitoring with Tabular Financial Data.

[7] Taylor, J. W., & Taylor, K. S. (2023). Combining probabilistic forecasts of COVID-19 mortality in the United States. European Journal of Operational Research, 304(1), 25–41.

[8] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, & Ben Poole (2020). Score-Based Generative Modeling through Stochastic Differential Equations. CoRR, abs/2011.13456.

[9] Xizewen Han, Huangjie Zheng, & Mingyuan Zhou. (2022). CARD: Classification and Regression Diffusion Models.

[10] Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2024). Deep Neural Networks and Tabular Data: A Survey. IEEE Transactions on Neural Networks and Learning Systems, 35(6), 7499–7519.

[11] Léo Grinsztajn, Edouard Oyallon, & Gaël Varoquaux. (2022). Why do tree-based models still outperform deep learning on tabular data?.

[12] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, & Artem Babenko. (2023). Revisiting Deep Learning Models for Tabular Data.

[13] Jonathan Ho, Ajay Jain, & Pieter Abbeel. (2020). Denoising Diffusion Probabilistic Models.

[14] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, & Surya Ganguli. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics.

[15] Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.Y. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Advances in Neural Information Processing Systems. Curran Associates, Inc..

[16] Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378.

[17] Blicher Bjerregård, M., Kloppenborg Møller, J., & Madsen, H. (2021). An introduction to multivariate probabilistic forecast evaluation. Energy and AI, 4, 100058.

[18] Tony Duan, Anand Avati, Daisy Yi Ding, Khanh K. Thai, Sanjay Basu, Andrew Y. Ng, & Alejandro Schuler. (2020). NGBoost: Natural Gradient Boosting for Probabilistic Prediction.

Meta-review: Treeffuser is a diffusion model in which the score function is approximated by gradient-boosted trees (unlike some other diffusion models that use neural networks).  Both the diffusion process and the GBTs are fairly standard, though the diffusion was adapted to the use of trees. The combination appears to be novel, and was clearly explained.  From the experimental results, it also seems to be quite effective for both synthetic and real-world tabular data sets, even if the hyper-parameters are not tuned.  There is also a software package which seems straightforward to use from the description.
Decision: Accept (poster)