Title: Treeffuser: probabilistic prediction via conditional diffusions with gradient-boosted trees
Venue: NeurIPS 2024
Accepted: yes
Keywords: trees

gradient boosting

diffusion

probabilistic predictions

uncertainty quantification
Abstract: Probabilistic prediction aims to compute predictive distributions rather than single point predictions. These distributions enable practitioners to quantify uncertainty, compute risk, and detect outliers. However, most probabilistic methods assume parametric responses, such as Gaussian or Poisson distributions. When these assumptions fail, such models lead to bad predictions and poorly calibrated uncertainty.  In this paper, we propose Treeffuser, an easy-to-use method for probabilistic prediction on tabular data. The idea is to learn a conditional diffusion model where the score function is estimated using gradient-boosted trees. The conditional diffusion model makes Treeffuser flexible and non-parametric, while the gradient-boosted trees make it robust and easy to train on CPUs. Treeffuser learns well-calibrated predictive distributions and can handle a wide range of regression tasks---including those with multivariate, multimodal, and skewed responses. We study Treeffuser on synthetic and real data and show that it outperforms existing methods, providing better calibrated probabilistic predictions. We further demonstrate its versatility with an application to inventory allocation under uncertainty using sales data from Walmart. We implement Treeffuser in https://github.com/blei-lab/treeffuser.

Treeffuser: Probabilistic Predictions via Conditional Diffusions with Gradient-Boosted Trees

Abstract

Probabilistic prediction aims to compute predictive distributions rather than single point predictions. These distributions enable practitioners to quantify uncertainty, compute risk, and detect outliers. However, most probabilistic methods assume parametric responses, such as Gaussian or Poisson distributions. When these assumptions fail, such models lead to bad predictions and poorly calibrated uncertainty. In this paper, we propose Treeffuser, an easy-to-use method for probabilistic prediction on tabular data. The idea is to learn a conditional diffusion model where the score function is estimated using gradient-boosted trees. The conditional diffusion model makes Treeffuser flexible and non-parametric, while the gradient-boosted trees make it robust and easy to train on CPUs. Treeffuser learns well-calibrated predictive distributions and can handle a wide range of regression tasks-including those with multivariate, multimodal, and skewed responses. We study Treeffuser on synthetic and real data and show that it outperforms existing methods, providing better calibrated probabilistic predictions. We further demonstrate its versatility with an application to inventory allocation under uncertainty using sales data from Walmart. We implement Treeffuser in https://github.com/blei-lab/treeffuser .

1 Introduction

In this paper, we develop a new method for probabilistic prediction from tabular data. This problem is important to many fields, such as power generation [1], finance [2], and healthcare [3]. It drives decision processes such as supply chain planning [4], risk assessment [5, 6], and policy-making [7].

Example: Manufacturing plants measure information as they operate. This information-properties of raw materials, operational flows, temperatures, and level measurements-collectively determine the output of the plant [8]. From this data, how can manufacturers adapt operations to reduce emissions while maximizing profits? The answer requires both good predictions and estimates of uncertainty, so as to trade off the risk of failure with the reward of lower emissions and higher profit. More broadly, such industrial workflow problems often rely on predictions from vast amounts of tabular data-observations of variables arranged in a table [9, 10].

x

=

0

.

2

x

=

0

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

y

x

=

0

.

8

x

=

0

.

2

x

=

0

.

5

0

2

4

6

8

y

x

=

0

.

8

y

1

y

2

x

=

0

.

2

x

=

0

.

5

x

=

0

.

8

Our method builds on two ideas: diffusions [11] and gradient-boosted trees [12]. Diffusions accurately estimate conditional distributions, but existing methods are not designed for tabular data. Decision trees excel at analyzing tabular data, but do not provide probabilistic predictions. Our method, which we call Treeffuser , combines the advantages of both formalisms. It defines an accurate tree-based diffusion model for probabilistic prediction which can easily handle large datasets.

Fig. 1 shows Treeffuser in action. We feed it data from three complex distributions-a multimodal response with varying components, an inflated response with shifting support, and a multivariate response with dynamic correlations. For each task, Treeffuser uses trees to learn a diffusion model and then outputs samples from the target conditional. These samples fully capture the complexity of the response functions.

Treeffuser exhibits several advantages:

It is nonparametric. It makes few assumptions about the form of the conditional response distribution. For example, it can estimate response distributions that are multivariate, multimodal, skewed, and heavy-tailed.

It is efficient. Diffusions can be slow to train [13]. Treeffuser relies on trees and is fast. For instance, in section 5.3, Treeffuser trains from a table with 112,000 observations and 27 variables in 53 seconds on a laptop.

It is accurate. On benchmark datasets, Treeffuser outperforms NGBoost [14], IBUG [15], quantile regression [16] and deep ensembles [17]. It provides better probabilistic predictions, including more precise quantile estimations and accurate mean predictions.

The rest of the paper is organized as follows. Section 2 reviews diffusions and gradient-boosted trees. Section 3 integrates these two ideas into Treeffuser and justifies the method. Section 4 discusses related work. Section 5 studies synthetic and standard benchmark datasets.

2 Background

Treeffuser combines two ideas: diffusion models and gradient-boosted trees (GBTs). This section provides a gentle introduction to both topics. Familiar readers can skip ahead to Section 3.

2.1 Diffusion Models

A diffusion model is a generative model that learns an arbitrarily complex distribution π ( y ) . It consists of two processes: forward and reverse diffusion.

The forward process takes the target distribution π ( y ) and continuously transforms it into a simple distribution. It does so by progressively adding noise to samples from π :

where Eq. (1) defines a stochastic differential equation (SDE) with a standard Brownian motion w ( t ) , and drift and diffusion functions f and g . The time horizon T is large, such that the resulting marginal distribution p simple := p T ( y ( T )) = ∫ y ′ p T ( y ( T ) | y (0) = y ′ ) π ( y ′ ) d y ′ is agnostic to π .

The reverse process transforms the simple distribution back into the target distribution by denoising perturbed samples from p simple. It posits the following model, which runs backward from T to 0:

where ˜ w ( t ) is a standard Brownian with reversed time. Anderson [18] shows that ˜ y ( t ) d = y ( t ) for each time t , and so ˜ y (0) ∼ π . This means that by drawing a sample from p simple and then numerically approximating the reverse process in Eq. (2), we obtain samples from π .

However, the score function, y ↦→∇ y log p t ( y ) , is usually unknown. Vincent [19] shows it can be estimated from the trajectories of the forward process as the minimizer of the following objective:

where S is the set of all possible functions of y indexed by time t , and p 0 t denotes the conditional distribution of y ( t ) given y (0) . In practice, we set t ∼ Uniform ([0 , 1]) and choose the drift f and diffusion function g so that p 0 t is Gaussian. We detail our choice of f and g in Appendix C.

We approximate the expectations in Eq. (3) with the empirical distribution of the observed y for E π , and with the known Gaussian trajectories y ( t ) | y (0) = y for E p 0 t . The objective can then be minimized by parametrizing the score s with any function approximator, such as a neural network or, as we do, with trees. By estimating the score function, we effectively learn the distribution π .

For a more detailed introduction to diffusion models, we provide an expanded version of this section in Appendix A.

2.2 Gradient Boosted Trees

Consider the following common machine learning task. Given variables ( a , b ) ∼ π , where b is scalar, learn the function F ∗ ( a ) that best approximates b from a :

where L is a loss function measuring the quality of F , such as the squared loss L ( u, v ) = ( u -v ) 2 . Gradient-boosted trees (GBTs) are a widely used non-parametric machine learning model for this task [12]. GBTs use decision trees [20] as simple building block functions f : R d → R to form a good approximation ˆ F of F ∗ . GBTs sequentially build approximations ˆ F i defined as

where ε ∈ (0 , 1) is a parameter akin to a learning rate.

Each decision tree f i is constructed to minimize the squared error between the current approximation ˆ F i and the negative gradient of the loss function:

As the number of trees i increases, the approximation ˆ F i becomes a better minimizer of Eq. (4). Various modifications to this basic algorithm have been proposed, including different loss functions in Eq. (4), higher order optimizations for Eq. (6) and general heuristics for faster training [21-23].

3 Probabilistic Prediction via Treeffuser

Treeffuser tackles probabilistic prediction by learning a diffusion model with gradient-boosted trees. It is particularly well-suited to modeling tabular data. Trees offer useful inductive biases, natural

handling of categorical and missing data, and fast and robust training procedures. Diffusions eliminate the need for restrictive parametric families of distributions and protect against model misspecification.

Denote an independently distributed set of observations as D = { ( x i , y i ) } n i =1 . Treeffuser forms the predictive distribution π ( y | x ) as a function of inputs x . We first introduce conditional diffusion models and discuss the conditional score estimation problem for both univariate and multivariate outcomes. We then outline the training and sampling procedures for Treeffuser.

3.1 The Conditional Diffusion Model

Treeffuser produces a distribution over y for each value of x using conditional diffusion models. Unlike approaches that guide an unconditional model to achieve conditionality, Treeffuser follows the line of work that directly fits the conditional score function [24-26].

Conditional diffusion models. The diffusion models introduced in Section 2.1 target the marginal distribution of y . Here, we extend them to conditional distributions π x ( y ) := π ( y | x ) .

To model π x , assign a diffusion process y x ( t ) to each value of x . These processes share the same diffusion equation but have different boundary conditions corresponding to their target conditionals:

As before, f and g are simple functions such that y x ( T ) ∼ p simple for all x . Denote the marginal distribution of y x ( t ) as p x ,t . For two time points t and u , where t > u , denote the conditional distribution y x ( t ) | y x ( u ) as p ut . (Note how p ut does not vary in x .)

To match each x -dependent forward SDE we have an x -dependent reverse SDE of the form:

where the score function ∇ log p x ,t now also depends on x . Similar to unconditional diffusions, by estimating the conditional score, we can sample from y x by first sampling ˜ y x ( T ) from p simple ( y ) and then solving the corresponding reverse SDE.

Conditional score estimation objective. Estimating the conditional score follows a similar strategy as the unconditional version, but with the added requirement of simultaneously estimating the score for all x . Recall that by Eq. (3), for a fixed x , the function s ∗ x defined by

satisfies s ∗ x ( y x , t ) = ∇ y x ( t ) log p x ,t ( y x ( t )) for all y , t and the fixed x . Intuitively, if we allow s ∈ S to also take x as input, we can gather all of these individual optimization problems into a single large problem by taking an additional expectation over x ∼ π , that is:

Here, S + represents the set of functions that take x as an extra input along y and t . The uppercase S emphasizes that S takes x as input, contrary to lowercase s . The validity of this objective is given by the following result.

Theorem 1 (Optimal Conditional Objective) . Define S ∗ as the solution of Eq. (9) . Then, for almost all x , y , t with respect to π ( x , y ) and the Lebesgue measure on t ∈ [0 , T ] , we have

We refer to Eq. (9) as the conditional score objective . The proof is provided in Appendix B.

3.2 The Trees

Treeffuser uses gradient-boosted trees (GBTs) to minimize Eq. (9). As GBTs work on scalar outputs, we separate the conditional score objective into an equivalent set of d y independent scalar-valued

Algorithm 1: Treeffuser Training

sub-problems

where a k denotes the k -th component of vector a .

Recall that the drift and diffusion functions of the forward process are chosen such that p 0 t is Gaussian. Let m = m ( t ; y ) and σ = σ ( t ) denote the corresponding mean and standard deviation, respectively. Treeffuser replaces the partial derivative in Eq. (11) with its closed-form expression as a function of m and σ . Treeffuser further reparametrizes S ( y , t, x ) with U ( y , t, x ) /σ ( t ) and defines h ( ζ, t, y ) = m ( t ; y ) + ζσ ( t ) , the process by which a sample y at time 0 gets diffused into a sample at time t with Gaussian noise ζ . The optimization problems in Eq. (11) are then:

The next theorem justifies how the individual problems in Eq. (12) estimate the conditional score. Theorem 2 (Treeffuser One-Dimensional Objectives) . Denote U ∗ = ( U ∗ 1 , ..., U ∗ d y ) . Then for almost all x , y , t with respect to π ( x , y ) and the Lebesgue measure on t ∈ [0 , T ] , we have

Each problem in Eq. (12) is a GBT problem where the notation within Eq. (4) corresponds to F := U k , a := ( h ( ζ, t, y ) , t, x ) ∈ R d y +1+ d x , b := -ζ k ∈ R and L is the square loss. We note that the noise scaling reparametrization, S = U/σ , is key to stabilizing the learning process; see the ablation study in Appendix G.

Finally, Treeffuser approximates the expectations in Eq. (12) with Monte Carlo sampling. For each sample ( x , y ) from the dataset D , Treeffuser samples R pairs of ( t, ζ ) ∼ Uniform ([0 , 1]) ⊗N (0 , I d y ) and creates new datasets D k containing R · n datapoints of the form ( ( h ( ζ, t, y ) , t, x ) , -ζ k ) , one D k per dimension of y . Then, each of these datasets is given to a standard GBT algorithm, such as LightGBM [22] or XGBoost [21]. Our implementation of Treeffuser uses LightGBM. Algorithm 1 details this procedure.

3.3 Sampling and Probabilistic Predictions

Treeffuser provides samples from the probabilistic predictive distribution π ( y | x ) . It does so as in standard unconditional models by plugging in the GBT-estimated conditional score from Eq. (12)

into a numerical approximation of the SDE in Eq. (8). While Treeffuser is compatible with any SDE solver, our implementation leverages Euler-Maruyama [27] due to its good balance between accuracy and the number of function evaluations. In general, we found good performance with as few as 50 steps. Algorithm 2 implements this procedure.

The samples generated by Treeffuser can then be used to estimate means, quantiles, probability intervals, expectations, or any other quantity of interest that depends on the response distribution.

3.4 Limitations

The design of Treeffuser offers advantages in terms of usability, versatility, and robustness. But it also comes with a few limitations. First, the diffusion process is theoretically defined to only model continuous responses y . However, count and other forms of discrete responses are common in the probabilistic modeling of tabular data. While our experiments show that this limitation does not prevent Treeffuser from outperforming comparable methods on these kinds of data, there may be opportunities for further improvement with direct modeling of discrete outcomes. Second, Treeffuser does not offer a closed-form density and must solve an SDE to generate samples. This sampling process, in contrast with the fast training, can become expensive when many samples per datapoint x are required.

4 Related Work

Treeffuser builds on advances in diffusion models to form probabilistic predictions from tabular data.

Diffusion models. Diffusion models excel at learning complex unconditional distributions on a range of data, such as images [28-30], molecules [31], time series [32], and graphs [33]. A common task is conditional generation, where the goal is to generate samples from a distribution conditioned on features. There are two approaches to this objective. One approach is to use guidance methods by which the score of an unconditional diffusion model is altered during generation to mimic the score of the conditional distribution [11, 34, 35]. This approach is especially popular for inverse problems [11, 36, 37]. Another approach is to train a conditional model from the start, incorporating the conditioning information during training [24-26, 38, 39]. This is the approach adopted by Treeffuser.

Treeffuser contributes to a recent line of work that applies diffusions to tabular data. This includes deep learning approaches for data generation [40], probabilistic regression [41], and missing data imputation [42]. Among these methods, CARD [41] uses neural-net based diffusions for probabilistic predictions and thus is most similar to Treeffuser in scope. We attempted to include it in our experiments using the implementation from Lehmann [43], but returned very poor results. We therefore excluded it from our testbed. The imputation of missing data has been recently extended to gradient-boosted trees [44].

Probabilistic prediction for tabular data Probabilistic prediction for tabular data can be classified into parametric and non-parametric methods based on their assumptions about the likelihood shape. Parametric tree-based methods include NGBoost [14] and PBGM [45]. NGBoost uses natural gradients to optimize a scoring rule, while PBGM sequentially updates the mean and standard deviation for predictions. DRFs obtain maximum likelihood estimates by using this criteria to choose splits. Neural-based parametric methods include Bayesian Neural Networks [46], MC Dropout [47], and Deep Ensembles [17]. Notably these methods are all indirectly or directly Bayesian. Another approach, normalizing flows, transforms a latent distribution via an invertible neural network [48] and has been applied to tabular data [49]. Non-parametric methods are often tree-based, such as Quantile Regression Forests [16], Distributional Random Forests (DRF) [50], and IBUG [15]. Quantile Regression Forests approximate the inverse cumulative distribution function by minimizing pinball loss, while DRF and IBUG use a tree-based similarity metric to weight training data for predictions. These methods are baselines in our empirical studies, with detailed descriptions in Appendix F.1.

5 Empirical studies

We demonstrate Treeffuser across three settings: synthetic data, standard UCI datasets [51], and sales data from Walmart. We find that Treeffuser outperforms state-of-the-art methods [15, 17, 50];

Dataset

N,d

x

, d

y

Deep

NGBoost

iBUG

Quantile

DRF

Treeffuser

Treeffuser

ensembles

(Gaussian)

(XGBoost)

regression

(no tuning)

bike

17,379,12,1

1

.

61

±

0

.

05

7

.

15

±

0

.

18

1

.

88

±

0

.

11

1

.

85

±

0

.

07

2

.

15

±

0

.

06

1

.

60

±

0

.

05

1

.

64

±

0

.

05

×

10

1

energy

768,

8,

2

5

.

00

±

0

.

71

4

.

78

±

0

.

49

NA

NA

5

.

43

±

0

.

69

3

.

07

±

0

.

40

3

.

32

±

0

.

48

×

10

0

kin8nm

8,192,

8, 1

3

.

59

±

0

.

12

9

.

48

±

0

.

29

7

.

74

±

0

.

41

6

.

63

±

0

.

16

9

.

44

±

0

.

20

5

.

89

±

0

.

14

5

.

88

±

0

.

17

×

10

-

2

movies

7,415,

9, 1

2

.

94

±

0

.

35

×

3

.

42

±

0

.

52

7

.

90

±

0

.

68

5

.

57

±

0

.

61

2

.

68

±

0

.

31

2

.

69

±

0

.

28

×

10

7

naval

11,934,17,1

4

.

11

±

0

.

39

4

.

43

±

0

.

19

3

.

20

±

0

.

17

16

.

86

±

2

.

46

4

.

55

±

0

.

16

2

.

02

±

0

.

08

2

.

46

±

0

.

07

×

10

-

4

news

39,644,58,1

2

.

53

±

0

.

27

×

3

.

89

±

1

.

19

2

.

32

±

0

.

17

1

.

98

±

0

.

16

1

.

98

±

0

.

17

1

.

98

±

0

.

16

×

10

3

power

9,568,

4, 1

2

.

06

±

0

.

10

2

.

01

±

0

.

13

1

.

61

±

0

.

07

5

.

40

±

0

.

12

1

.

90

±

0

.

11

1

.

49

±

0

.

07

1

.

52

±

0

.

07

×

10

0

superc.

21,263,81,1

4

.

89

±

0

.

31

5

.

24

±

0

.

50

4

.

14

±

0

.

28

3

.

79

±

0

.

14

4

.

32

±

0

.

16

3

.

52

±

0

.

13

3

.

60

±

0

.

15

×

10

0

wine

6,497, 12, 1

3

.

59

±

0

.

10

3

.

82

±

0

.

11

3

.

25

±

0

.

14

3

.

24

±

0

.

14

3

.

30

±

0

.

12

2

.

59

±

0

.

13

2

.

67

±

0

.

13

×

10

-

1

yacht

308,

6,

1

4

.

86

1

.

38

3

.

67

1

.

47

3

.

75

1

.

24

3

.

53

1

.

30

7

.

73

2

.

43

3

.

11

0

.

99

3

.

39

0

.

97

×

10

-

1

±

±

±

±

±

±

±

it can capture complex distributions with multimodal, inflated, or multivariate responses; it achieves better probabilistic predictions on many real-world datasets and it produces more accurate sales forecasts on Walmart data. We also find that Treeffuser can outperform other (tuned) methods without dataset-specific hyperparameter tuning. We provide the code for Treeffuser at https://github.com/blei-lab/treeffuser .

5.1 Probabilistic prediction on synthetic data

We show the flexibility of Treeffuser on three probabilistic regression tasks that are difficult to model with standard parametric models. We also show its competitive performance on Gaussian data when compared to methods that posit a Gaussian likelihood.

Treeffuser vs. complex response functions. We design three difficult probabilistic regression tasks: a multimodal response where the number of modes changes with x , an inflated response with support that changes with x , and a multivariate response with nontrivial covariance structure. The data generation processes for these datasets are detailed in Appendix E. For each dataset, we generate 10,000 observations ( x i , y i ) and train Treeffuser with its default parameters (see Appendix C).

Fig. 1 shows the histograms of 1,000 Treeffuser samples for three different values of x and compares them with the true conditional densities of the response. Treeffuser captures the conditional distribution for all values of x across all settings. For multimodal data, Treeffuser correctly detects the modes of the distribution without knowing their number a priori. For inflated data, it recovers the peak at the inflation point and the x -dependent support of the response. For multivariate data, it recovers the complex covariance structures.

Treeffuser vs. parametric oracles. We simulate Gaussian responses with a linear response function and compare Treeffuser to other parametric methods that assume a Gaussian likelihood. These methods serve as oracles in these experiments as they are informed by the true functional form of the conditional response distribution. Results and further details are reported in Appendix E. Among all methods, Treeffuser consistently ranks second best, only behind the Deep Ensemble oracle.

5.2 Probabilistic prediction on real-world datasets

We compare Treeffuser with state-of-the-art methods for probabilistic predictions on standard UCI datasets [51]. A detailed description of the baseline models can be found in Appendix F.1.

Metrics. We evaluate probabilistic predictions with the continuous ranked probability score (CRPS) and standard accuracy metrics.

CRPS is defined as CRPS ( F, y ) = ∫ ∞ -∞ ( F ( y ′ ) -✶ ( y ≤ y ′ )) 2 dy ′ , where F is the cumulative distribution function of the predicted distribution p ( y | x ) and y is the true observed value [52]. CRPS is a proper scoring rule, in that its expected value is minimized under the true conditional distribution of the response [53]. CRPS is usually preferred over the log-likelihood, which is also

a proper scoring rule, but is sensitive to the estimation of the tail densities [54]. Also, CRPS can readily be estimated from samples of p ( y | x ) , which is our setting with the non-parametric methods we evaluate. We evaluate CRPS by generating 100 samples from p ( y | x ) for each x . For evaluating multivariate responses, we report the average marginal CRPS over each dimension.

We also measure the quality of point predictions for each model. This is the ability to predict conditional means E [ y | x ] . We approximate E [ y | x ] using 50 samples and evaluate the accuracy using the mean absolute error (MAE) and the root mean squared error (RMSE).

Experimental setup. We performed 10-folds cross-validation. For each fold, we tuned the hyperparameters of the methods using Bayesian optimization for 25 iterations, using 20% of the current fold's training data as a validation set. Additional Bayesian optimization's iterations did not change the results. We detail the search space of hyperparameters for each method in Appendix F.2.

Results. Table 1 presents CRPS by dataset and method. Treeffuser consistently provides the most accurate predictions across datasets, as measured by CRPS. Notably, it outperforms other methods even when initialized with its default parameters. There is no consistent runner-up: among parametric methods, deep ensemble does overall better than NGBoost and IBUG; quantile regression does well on some datasets but underperforms in others.

We find that Treeffuser also returns the best point predictions of E [ y | x ] , as reported in Table 4. For comparison, we report the accuracy of point predictions from vanilla XGBoost and LightGBM in Table 6 in Appendix F.3. These methods do not provide probabilistic predictions but are tailored for point predictions. As expected, XGBoost and LightGBM outperform or tie with all the probabilistic methods. In particular, they often tie with Treeffuser, suggesting that Treeffuser provides probabilistic prediction without sacrificing average point predictions.

Finally, we conducted an ablation study to investigate the impact of the noise-scaling reparametrization of the score function on Treeffuser's performance. As detailed in Appendix G, noise scaling is key to achieving top accuracy and stability.

5.3 Sales forecasting with long tails and count data

Table 2: Walmart dataset metrics (lower is better). The evaluation is on the last 30 days of data. Treeffuser provides the best probabilistic predictions alongside NGBoost Poisson. Deep ensembles excel at point predictions. Powers of tens are factorized out of each row in rightmost column.

We further demonstrate the applicability of Treeffuser on a publicly available dataset [55] for sales forecasting under uncertainty. The goal is to forecast the number of units sold for a product given features such as its price, its type (e.g., food, cloths), and past sales.

This task is challenging due to zero inflation and long tails in the distribution of item sales. (E.g., umbrella sales are typically low but can spike during rainy weeks.) It is even more challenging for a diffusion model like Treeffuser, which is designed for continuous responses and not count data.

We use five years of sales data from ten Walmart stores (a large American retail chain). We randomly select 1,000 products, training on 112,000 data points from the first 1,862 days and evaluating 10,000 other data points for the remaining 30 days.

In addition to the previous baselines, we include NGBoost Poisson, a parametric model specifically designed for count data. We evaluate the predictions returned by each method in three ways.

CRPS and accuracy metrics. First, we compute the same evaluation metrics as in the previous experiments. We benchmark against the methods in the experiment and the methods in section 5.2.

The results are reported in Table 2. We find that Treeffuser again proves competitive, achieving a better CRPS than all methods and comparable MAE and RMSE.

Posterior predictive checks. Second, we perform held-out predictive checks [56], examining six statistics on the number of items sold: the total count of zero sales, the highest sales figure, and sales figures at the 0.99, 0.999, and 0.9999 quantiles (lower quantiles are well captured by all methods). We produce probabilistic predictions of these quantities by returning their empirical distributions as induced by the samples generated by the models. Fig. 2 compares the observed values against the probabilistic predictions. Treeffuser best captures the proportion of zeros and performs as well as NGBoost-Poisson in modeling the behavior of the tails.

0.35

0.40

0.45

0.50

0.55

0.60

Proportion of zeros

1

2

3

4

5

Quantile  0.9

10

12

14

Quantile  0.99

100

200

300

Treeffuser

NGBoost

Poisson

Quantile

Regression

20

40

60

Quantile  0.999

50

100

150

Quantile  0.9999

Newsvendor model. Finally, we illustrate the practical relevance of accurate probabilistic predictions with an application to inventory management, using the newsvendor model [57]. Assume that every day we decide how many units q of an item to buy. We buy at a cost c and sell at a price p . However, the demand y is random, introducing uncertainty in our decision. The goal is to maximize the expected profit:

The solution to the newsvendor problem is to buy q = F -1 ( p -c p ) units, where F -1 is the quantile function of the distribution of y .

We apply this model to evaluate the inventory decisions induced by each method on the Walmart dataset. To compute profits, we use the observed prices and assume a margin of 50% over all products. We let Treeffuser, NGBoost-Poisson, and quantile regression learn the conditional distribution of the demand of each item, estimate their quantiles, and thus determine the optimal quantity to buy.

Fig. 3 plots the cumulative profits over the last 30 days of data. Treeffuser outperforms quantile regression by a large margin and performs comparably to NGBoost-Poisson. This is coherent with our PPC results in Fig. 2, showing better quantile estimations for Treeffuser. This demonstrates that Treeffuser delivers competitive probabilistic predictions even for count data responses, a scenario it was not specifically designed to handle.

5.4 Runtime Performance Overview

We measure Treeffuser's performance in terms of training and inference speed across different datasets. On the M5 dataset, using default parameters, Treeffuser completed training in 53.2 seconds on a MacBook Pro M3 Max and generated 10,000 samples (one per test data point) in 2.53 seconds. We conducted additional benchmarking experiments on both the UCI and M5 datasets, with results detailed in Appendix H. (Further discussion on the model's time complexity is also provided in that appendix.) Details about the computational resources used in all of our experiments are available in Appendix D.

0

5

10

15

20

25

30

Day

0

500

1000

1500

2000

2500

Cumulative profit

Model

Treeffuser

NGBoost-Poisson

QuantileRegression

Hyperparameters

Tuned

Default

6 Conclusion

We have introduced Treeffuser, a new model for probabilistic prediction from tabular data.

Treeffuser combines conditional diffusions models with gradient-boosted trees. It can capture arbitrarily complex distributions without requiring any data-specific modeling or tuning. It is amenable to fast CPU learning and naturally handles categorical data and missing values. We have demonstrated that Treeffuser outperforms state-of-the-art methods in probabilistic regression across datasets and metrics. These characteristics make Treeffuser a flexible, easy-to-use, and robust model for probabilistic predictions.

One limitation of our diffusion-based approach is the need to numerically solve an SDE to generate samples, which can be costly when producing many samples. Recent advances, such as progressive distillation [58] and consistency models [59], address this issue. Applying these methods to Treeffuser is a direction for future work.

References

Aristeidis Mystakidis, Evangelia Ntozi, Konstantinos Afentoulis, Paraskevas Koukaras, Paschalis Gkaidatzis, Dimosthenis Ioannidis, Christos Tjortjis, and Dimitrios Tzovaras. Energy generation forecasting: Elevating performance with machine and deep learning. Computing , 105(8):1623-1645, 2023.

Jillian M. Clements, Di Xu, Nooshin Yousefi, and Dmitry Efimov. Sequential deep learning for credit risk monitoring with tabular financial data, 2020. arXiv preprint arXiv:2012.15330.

Snigdha Somani, Adam J Russak, Filippo Richter, Simon Zhao, Akhil Vaid, Farah Chaudhry, Jason K De Freitas, Niyati Naik, Riccardo Miotto, Girish N Nadkarni, Jagat Narula, Edgar Argulian, and Benjamin S Glicksberg. Deep learning and the electrocardiogram: Review of the current state-of-the-art. Europace , 23(8):1179-1191, 2021.

Anshuman Gupta and Costas D Maranas. Managing demand uncertainty in supply chain planning. Computers & Chemical Rngineering , 27(8-9):1219-1227, 2003.

Yacov Y Haimes. Risk Modeling, Assessment, and Management . John Wiley & Sons, 2011.

Philippe Jorion. Value at Risk: The New Benchmark for Managing Financial Risk . McGraw-Hill, 2007.

James W. Taylor and Kimberly S. Taylor. Combining probabilistic forecasts of covid-19 mortality in the united states. European Journal of Operational Research , 304(1):25-41, 2023.

Ziqiu Kang, Cagatay Catal, and Bedir Tekinerdogan. Machine learning applications in production lines: A systematic literature review. Computers & Industrial Engineering , 149:106773, 2020.

Anna L. Buczak and Erhan Guven. A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Communications Surveys and Tutorials , 18(2): 1153-1176, 2016.

Thi-Thu-Huong Le, Yustus Eko Oktian, and Howon Kim. XGBoost for imbalanced multiclass classification-based industrial Internet of Things intrusion detection systems. Sustainability , 14 (14), 2022.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2020.

Jerome H Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics , 29(5):1189-1232, 2001.

Zhendong Wang, Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, and Mingyuan Zhou. Patch Diffusion: Faster and more data-efficient training of diffusion models, 2023. arXiv preprint arXiv:2304.12526.

Tony Duan, Anand Avati, Daisy Yi Ding, Khanh K. Thai, Sanjay Basu, Andrew Y. Ng, and Alejandro Schuler. NGBoost: Natural gradient boosting for probabilistic prediction, 2020. arXiv preprint arXiv:1910.03225.

Jonathan Brophy and Daniel Lowd. Instance-based uncertainty estimation for gradient-boosted regression trees. In Advances in Neural Information Processing Systems , volume 35, pages 11145-11159, 2022.

Nicolai Meinshausen and Greg Ridgeway. Quantile regression forests. Journal of Machine Learning Research , 7(6), 2006.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems , volume 30, 2017.

Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications , 12(3):313-326, 1982.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation , 23(7):1661-1674, 2011.

Trevor Hastie, Robert Tibshirani, and Jerome H Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction , volume 2. Springer, 2009.

Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '16. ACM, 2016.

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features, 2019. arXiv preprint arXiv:1706.09516.

Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann. Conditional image generation with score-based diffusion models, 2021. arXiv preprint arXiv:2111.13606.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement, 2021. arXiv preprint arXiv:2104.07636.

Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. In Advances in Neural Information Processing Systems , volume 34, pages 24804-24816. Curran Associates, Inc., 2021.

Peter E. Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations . Stochastic Modelling and Applied Probability. Springer Berlin, Heidelberg, 1992.

Stability AI. Introducing stable diffusion. https://stability.ai/blog/ stable-diffusion-public-release , 2022.

MidJourney. Midjourney. https://www.midjourney.com/ , 2022.

OpenAI. Dall·e 2. https://openai.com/index/dall-e-2/ , 2022.

Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile I. M. Wicky, Nikita Hanikel, Samuel J. Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Sergey Ovchinnikov, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung Baek, and David Baker. De novo design of protein structure and function with rfdiffusion. Nature , 620(7976):1089-1100, 2023.

Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning , pages 8857-8868. PMLR, 2021.

Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In International Conference on Artificial Intelligence and Statistics , pages 4474-4484. PMLR, 2020.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems , volume 34, pages 8780-8794, 2021.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. arXiv preprint arXiv:2207.12598.

Hyungjin Chung, Jeongsol Kim, Michael T. Mccann, Marc L. Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems, 2023. arXiv preprint arXiv:2209.14687.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations, 2022. arXiv preprint arXiv:2108.01073.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. arXiv preprint arXiv:2204.06125.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. TabDDPM: Modelling tabular data with diffusion models. In International Conference on Machine Learning , pages 17564-17579. PMLR, 2023.

Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. CARD: Classification and regression diffusion models. In Advances in Neural Information Processing Systems , volume 35, pages 18100-18115, 2022.

Shuhan Zheng and Nontawat Charoenphakdee. Diffusion models for missing value imputation in tabular data, 2022. arXiv preprint arXiv:2210.17128.

Nils Lehmann. lightning-uq-box 0.1.0. https://pypi.org/project/lightning-uq-box/ , 2024.

Alexia Jolicoeur-Martineau, Kilian Fatras, and Tal Kachman. Generating and imputing tabular data via diffusion and flow-based gradient-boosted trees. In International Conference on Artificial Intelligence and Statistics , pages 1288-1296. PMLR, 2024.

Olivier Sprangers, Sebastian Schelter, and Maarten de Rijke. Probabilistic gradient boosting machines for large-scale probabilistic regression. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '21. ACM, 2021.

Radford M Neal. Bayesian Learning for Neural Networks , volume 118. Springer Science & Business Media, 2012.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning , pages 1050-1059. PMLR, 2016.

Ivan Kobyzev, Simon J.D. Prince, and Marcus A. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(11):3964-3979, 2021.

Pavel Izmailov, Polina Kirichenko, Marc Finzi, and Andrew Gordon Wilson. Semi-supervised learning with normalizing flows, 2019. arXiv preprint arXiv:1912.13025.

Domagoj Cevid, Loris Michel, Jeffrey Näf, Peter Bühlmann, and Nicolai Meinshausen. Distributional random forests: Heterogeneity adjustment and multivariate distributional regression. Journal of Machine Learning Research , 23(333):1-79, 2022.

Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The UCI machine learning repository, 2023. URL https://archive.ics.uci.edu .

James E Matheson and Robert L Winkler. Scoring rules for continuous probability distributions. Management Science , 22(10):1087-1096, 1976.

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association , 102(477):359-378, 2007.

Mathias Blicher Bjerregård, Jan Kloppenborg Møller, and Henrik Madsen. An introduction to multivariate probabilistic forecast evaluation. Energy and AI , 4:100058, 2021.

Spyros Makridakis Addison Howard. M5 Forecasting - Accuracy, 2020. URL https:// kaggle.com/competitions/m5-forecasting-accuracy .

Andrew Gelman, Xiao-Li Meng, and Hal Stern. Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica , 6:733-807, 1996.

Kenneth J Arrow, Theodore Harris, and Jacob Marschak. Optimal inventory policy. Econometrica: Journal of the Econometric Society , pages 250-272, 1951.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022. arXiv preprint arXiv:2202.00512.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023. arXiv preprint arXiv:2303.01469.

Roger Koenker and Kevin F Hallock. Quantile regression. Journal of Economic Perspectives , 15(4):143-156, 2001.

Acronyms

CRPS continuous ranked probability score. 7, 8, 9

GBT gradient-boosted tree. 2, 3, 4, 5, 16

MAE mean absolute error. 8, 9

RMSE root mean squared error. 8, 9

SDE stochastic differential equation. 3, 4, 6, 14, 15, 16

A A primer on unconditional diffusion models

A diffusion model is a generative model that consists of two processes, a forward diffusion, and a reverse diffusion. The forward process takes an unknown target distribution π ( y ) and continuously transforms it into a simple distribution p simple ( y ) , typically a Gaussian. It does so by progressively adding noise to samples from π , such as our data.

The reverse process transforms the simple distribution back into the target distribution by denoising any perturbed samples from p simple. If we can learn the reverse process, we can effectively access the target distribution π : first draw samples from p simple and then use the reverse transformation, as guided by the score, to map them back to π .

One way to learn this noising-denoising process is to estimate the score function , the gradient of the log probability of the noisy data with respect to the data itself.

We detail the three main components of score-based diffusions-the forward process, the reverse process, and the estimation of the score function-below.

The forward diffusion process. Adiffusion model is described by the following generative process:

where the process of evolving the stochastic differential equation (SDE) involves a standard Brownian motion w ( t ) and model parameters f : R d y × [0 , T ] → R d y and g : [0 , T ] → R , respectively called the drift and diffusion functions.

The SDE in Eq. (14) determines the evolution of y ( t ) over time, thereby inducing a distribution of y ( t ) at each t , that we write p t ( y ( t )) . We further write the distribution of y ( t ) conditional on another y ( u ) as p ut ( y ( t ) | y ( u )) . In practice, f and g are simple functions such that p 0 t ( y ( t ) | y (0)) has a closed form. In fact, if the drift f is affine in y ( t ) and the diffusion function g is a scalar that does not depend on y ( t ) , the distribution p 0 t is always Gaussian, with a mean we write m ( t ; y (0)) and a covariance multiple of the identity we write σ ( t ) 2 I .

Wchoose the time horizon T such that the resulting marginal distribution p T ( y ( T )) = ∫ y ′ p T ( y ( T ) | y (0) = y ′ ) π ( y ′ ) d y ′ is a simple distribution p simple agnostic to π . The intuition is that as the diffusion progresses, the initial density π is forgotten.

We detail our choice of f , g and T in Appendix C, which is the variance exploding SDE (VESDE) in Song et al. [11]. This choice induces a trivial closed form expression for the functions m and σ .

There are no model parameters to learn in the diffusion model, and the conditional distributions p 0 t ( y ( t ) | y (0)) are trivial to compute. The challenge is to learn how to compute (or sample from) the conditional distribution p Tt ( y ( t ) | y ( T )) . This is called reversing the diffusion process. Knowing p Tt is key to learning the target distribution π , which can be written as π ( y (0)) = E p T [ p T 0 ( y (0) | y ( T ))] where p T = p simple is known.

The reverse diffusion process. Consider the following SDE that is reversed in time from T to 0 ,

where ˜ w ( t ) is a standard Brownian with reversed time and p t is the density of y ( t ) defined by the forward SDE (1). Similar to the forward case, the reverse SDE induces a distribution ˜ p t for ˜ y ( t ) at each t . Anderson [18] shows that, if we denote by ˜ p Tt the conditional distribution of ˜ y ( t ) given ˜ y ( T ) , then ˜ p Tt = p Tt for any time t . In other words, ˜ y ( t ) | ˜ y ( T ) = a and y ( t ) | y ( T ) = a have the same distribution for all a . Since we designed the diffusion such that p T ≈ p simple = ˜ p T , we have in particular ˜ y ( t ) ∼ y ( t ) .

SDE solvers let us sample from distributions that are solutions of an SDE; hence, we can obtain samples of π ( y ) by solving Eq. (15). However, the function y ↦→∇ y log p t ( y ) , called the score , is usually unknown and needs to be estimated.

Estimating the score. Vincent [19] shows that the score can be estimated from trajectories of the SDE as the minimizer of the following objective function:

where S = { s : R d y × [0 , T ] → R d y } is the set of all possible score functions indexed by time t , and E t can be an expectation over any distribution of t whose support is exactly [0 , T ] . Since p 0 t is a normal distribution N ( m ( t ; y (0)) , σ ( t ) 2 I ) , we have

In practice, the expectations in Eq. (16) are approximated using the empirical distribution formed by the observed y for E π , and by sampling the simple and known forward SDE trajectories y ( t ) | y (0) = y for E p 0 t . The objective can then be minimized by parametrizing s with any function approximator, e.g., a neural network. The score estimator can then be used to reverse the diffusion process and estimate the target distribution π ( y ) .

B Proofs of the main theorems

In this section, we provide the proofs for the two main theorems in the text.

Theorem 1 (Optimal Conditional Objective) . Define S ∗ as the solution of Eq. (9) . Then, for almost all x , y , t with respect to π ( x , y ) and the Lebesgue measure on t ∈ [0 , T ] , we have

Proof. For conciseness, define for any function s : R d y × [0 , T ] → R d y the quantity: r ( x , s ) = E t E π ( y x (0)) E p x , 0 t [ ∥ ∥ ∇ y x ( t ) log p 0 t ( y x ( t ) | y x (0)) -s ( y x ( t ) , t, x ) ∥ ∥ 2 ] . Also write for any function S : R d y × [0 , T ] × R d x → R d y and x ∈ R dx the function S x : ( y , t ) ↦→ S ( y , t, x ) .

Define Ω = { x ∈ R d x | S ∗ ( y , t, x ) ̸∈ arg min s r ( x , s ) } , we will show that π (Ω) = 0 .

The function S ∗ is characterized as, S ∗ ∈ arg min S ∈S + E π ( x ) [ r ( x , S x )] where S + = { S : R d y × [0 , T ] × R d x → R d y }

By [19], we know that ∇ y log p x ,t ( y ) ∈ arg min s r ( x , s ) for all x , so by definition of Ω , we have:

We further have for any other x ̸∈ Ω , r ( x , S ∗ x ) = r ( x , ( y , t ) ↦→∇ y log p x ,t ( y )) .

If π (Ω) > 0 , then integrating Eq. (18) over π ( x ) will yield E π ( x ) [ r ( x , S ∗ x )] > E π ( x ) [ r ( x , (( y , t, x ) ↦→ ∇ y log p x ,t ( y )) x )] , which is not possible by definition of S ∗ . Hence π (Ω) = 0 .

Hence we have S ∗ x ∈ arg min s r ( x , s ) for almost any x , so by [19] again, we can conclude that S ∗ ( y , t, x ) = ∇ y log p x ,t ( y ) for almost any x , t, y .

Theorem 2 (Treeffuser One-Dimensional Objectives) . Denote U ∗ = ( U ∗ 1 , ..., U ∗ d y ) . Then for almost all x , y , t with respect to π ( x , y ) and the Lebesgue measure on t ∈ [0 , T ] , we have

Proof. By definition in Eq. (12), we have

With Theorem 1, we have S ∗ ( y , t, x ) = ∇ y log p x ,t ( y ) almost everywhere, with S ∗ defined as

We have:

where we used the following facts:

from Eq. (19) to Eq. (20): the closed-form expression of the score S from Eq. (17),

from Eq. (20) to Eq. (21): expanding the norm and switching the expectations with the finite sum over k ,

from Eq. (21) to Eq. (22): reparametrizing the expectation of y x ( t ) | y x (0) which by definition is a normal distribution with mean m ( t ; y x (0)) and variance σ ( t ) 2 .

The final manipulation is to remember that theorem 1 and the theorems from Vincent [19] are valid with expectations E t against any strictly positive measure of t over [0 , T ] . In particular, we can absorb 1 σ ( t ) as a reweighted non-negative measure, and we obtain exactly the definition of U ∗ k by defining U ( y , t, x ) = σ ( t ) S ( y + σ ( t ) ζ, t, x ) which concludes the proof.

C Treeffuser default configuration

We use the following configuration as defaults for Treeffuser.

Forward diffusion. We use the variance exploding SDE [11] for all experiments, defined by setting:

where σ is a given increasing function defined by,

and α min = 0 . 01 and α max = 20 . For all our experiments we let t ∈ [0 , 1] (i.e. T = 1 ).

Gradient-boosted tree (GBT) parameters and dataset repetitions. Weprovide a short description of each hyper-parameter of the model alongside the default value. Treeffuser uses LightGBM [22] to learn the GBTs.

n estimators (3000): Specifies the maximum number of trees that will be fit, regardless of whether the stopping criterion is met.

learning rate (0.1): Specifies the shrinkage to use for every tree.

num leaves (31): Specifies the maximum number of leaves a tree can have.

early stopping rounds (50): Specifies how long to wait without a validation loss improvement before stopping.

n repeats (30): Specifies how many Monte Carlo samples to draw per data point to estimate E t,ζ in equation Eq. (9).

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

y

x

(a) Simulation 1: branching Gaussian mixture.

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

2

4

6

8

10

12

14

16

y

x

(b) Simulation 2: shifted and inflated Gamma.

D Experiments: description of computer resources

All benchmark tasks presented in Section 5.1 and Section 5.2 were run on a cluster with one job per triplet of (model, dataset, split index).

The real world experiments totalled 1135h for 700 tasks (10 datasets, 7 methods, 10 splits). Each task was allocated 4 cpus.

The synthetic experiments totalled 290 hours, for 800 tasks (16 datasets, 5 methods, 10 splits). Each task was allocated 2 CPU.

The M5 experiments were run on a single MacBook Pro over a couple hours. Other figures such as Fig. 1 were also generated on a MacBook pro in a few seconds.

E Experiments on synthetic data (supplement)

E.1 Arbitrarily complex synthetic data

We provide details on the three synthetic data experiments presented in Section 5.1 and illustrated in Fig. 1. Each experiment introduces a different distribution of the response variable y given the covariate x . The first experiment generates multimodal responses from a branching Gaussian mixture, the second experiment generates inflated responses from a mixture of a shifted Gamma distribution and an atomic measure, and the third experiment generates two-dimensional responses from a nonlinear multioutput regression.

We provide additional visualization of samples for the one-dimensional datasets in Fig. 4.

Models We assume that x ∼ Uniform ([0 , 1]) and model the conditional distribution of the response y given x differently for each of the synthetic dataset. Scatter plots of synthetic data from these distributions are provided in Fig. 4

Branching Gaussian mixture. We generate multimodal responses from a mixture of equally-weighted Gaussians. In our experiments, we fix the scale σ = 0 . 05 and let the number of components and their means scale with x as follows:

where GaussianMixture σ ( µ 1 , µ 2 , . . . , µ K ) denotes a Gaussian mixture distribution with K equallyweighted components, each with mean µ k and scale σ .

Covariate inflated Gamma. We generate inflated responses through the following mixture of a shifted Gamma distribution and an atomic measure:

where ShiftedGamma ( k, θ, x ) denotes a Gamma distribution with scale k and shape θ shifted by x . In words, given x , we set y to be equal to x with probability p , and to be drawn from a Gamma shifted by x otherwise. Hence, the conditional distribution of y given x is x -inflated and its support [ x, ∞ ) changes with x . In our experiments, we set p = 0 . 15 , k = 2 , θ = 1 .

Nonlinear multioutput regression. We generate two-dimensional responses from a density p ( y | x ) ∝ exp ( -dist ( y ,C ( x )) 2 2 σ 2 ) where σ = 0 . 05 , dist ( a, B ) measures the shortest Euclidean distance from a ∈ R to the set B ⊂ R 2 , and C ( x ) is a circular arc centered at (0 , 0) with radius radius ( x ) and angles of extremities θ 1 , θ 2 evolving as functions of x . Intuitively, they interpolate the circular arc represented in Section 1.

The functions are defined as follows.

If x ≤ 0 . 5

-

0

.

375)

,

If x > 0 . 5

45

,

E.2 Data with normal likelihood

We study the performance of Treeffuser in a setting where a simpler model is better adjusted to the generating process. In particular, we let N be number of data points sampled and d x the dimension and use a linear generative model

where σ 2 = 1 . The results are provided in table 3 using the same configuration described in section section 5.2.

Results From Table 3, we observe that while Treeffuser remains competitive, Deep Ensembles and NGBoost now demonstrate either equal or very close performance. This is expected since both models assume a normal distribution for the outcomes and the model assumed by Deep Ensembles closely resembles the real generative model. Interestingly, Treeffuser outperforms both iBUG and quantile regression (QReg), suggesting that its superior performance is not merely due to using LightGBM or XGBoost. Finally, we note that Treeffuser continues to perform well in low-data situations compared to other models.

F Experiments on standard datasets (supplement)

F.1 Baseline methods.

We briefly describe the methods used as baselines in our empirical studies.

NGBoost (parametric, tree-based). NGBoost [14] models the conditional distribution of the target variable using a parametric family of distributions whose parameters are predicted by a gradientboosting algorithm. It uses natural gradients for more stable, accurate, and faster learning. In our experiments, we set the parametric model to be Gaussian.

Deep ensembles (parametric, nnet-based). A Deep ensemble [17] is a collection of neural networks that are individually trained to model a parametric conditional distribution p ( y | x ) . The neural networks are then combined to obtain a mixture. We set the parametric model to be Gaussian.

Quantile regression (nonparametric, tree-based). Quantile regression [60] estimates the quantiles of the conditional distribution p ( y | x ) . We implemented a GBT-based version of the method by fitting trees with inputs x and q , where q is the probability of the desired quantile. During training, we optimized the objective E q E x , y [ L ( Q ( x , y ) , q )] , where L is the pinball loss and q is sampled from a zero-one uniform distribution. In our experiments, we used LightGBM as a GBT method.

IBUG (parametric, tree-based). IBUG [15] extends any GBT into a probabilistic estimator. Given an input instance x , it outputs a distribution around the prediction using the k -nearest training data points. The distance between a training instance x 0 and x depends on the number of co-occurrences of x 0 and x across the leaves of the ensemble. In our experiments, we used XGBoost and a Gaussian likelihood, following the default specifications of the method.

DRF (nonparametric, tree-based). DRF [50] grows a random forest by maximizing the differences in the response distributions across split groups with a distributional metric. Repeated randomization induces a weighting function that assesses how relevant a training data point is to a given input x . These weights are used to return a weighted empirical distribution of the response.

F.2 Hyperparameters of the methods.

Unless specified otherwise, we tuned the hyperparameters of each method using Bayesian optimization for 25 iterations. We used the following search space for each method (where J a, b K denotes the set of integers from a to b ):

Treeffuser

-n estimators 100 , 3000

-n repeats

-learning rate

-early stopping rounds 10 , 100

-num leaves ∈ J 10 , 100

Quantile regression (QReg)

-n estimators ∈ J 100 , 3000 K (log-uniform) -n repeats ∈ J 10 , 100 K -learning rate ∈ [0 . 01 , 1] -early stopping rounds ∈ J 10 , 100 K -num leaves ∈ J 10 , 100 K · IBUG -k ∈ J 20 , 250 K -n estimators ∈ J 10 , 1000 K -learning rate ∈ [0 . 01 , 0 . 5] (log-uniform) -max depth ∈ J 1 , 100 K · DRF -min node size ∈ J 5 , 30 K -num trees ∈ J 250 , 3000 K · NGBOOST -n estimators ∈ J 100 , 10000 K -learning rate ∈ [0 . 005 , 0 . 2] · Deep ensemble -n layers ∈ J 1 , 5 K -hidden size ∈ J 10 , 500 K -learning rate ∈ [10 -5 , 10 -2 ] (log-uniform) -n ensembles ∈ J 2 , 10 K

For the methods that only return point predictions, we used the following search spaces:

XGBoost -n estimators ∈ J 10 , 1000 K -learning rate ∈ [0 . 01 , 0 . 5] (log-uniform) -max depth ∈ J 1 , 100 K · LightGBM -n estimators ∈ J 10 , 1000 K -learning rate ∈ [0 . 01 , 0 . 5] (log-uniform) -num leaves ∈ J 10 , 100 K -early stopping rounds ∈ J 10 , 100 K

F.3 Accuracy and calibration results.

G Ablation study on noise scaling

We evaluate the impact of including noise scaling in the score parametrization of Eq. (13), specifically S ( y , t, x ) = U ( y , t, x ) /σ ( t ) , where S ( y , t, x ) denotes the score function of y ( t ) | x , U the gradientboosted trees, and σ ( t ) the noise schedule of the diffusion process. We compare Treeffuser with and without noise scaling under the same experimental setup of Section 5.2.

Table 7 presents the CRPS by dataset and method. Treeffuser without noise scaling fails to achieve any reasonable CRPS. We believe Treeffuser without noise scaling may require extra model capacity (e.g., deeper trees and more training iterations) to handle the varying noise levels adequately. However, we did not further optimize it, as noise scaling provides state-of-the-art results without the need for additional complexity.

H Runtime and Complexity

To demonstrate the efficiency of Treeffuser, we present the results of the following experiments and benchmarks:

We run Treeffuser on subsets of the M5 dataset (section 5.3) of various sizes and report the training time in Fig. 5.

We report the training time for running Treeffuser on the datasets used in the paper in Fig. 6.

We report in Table 8 the average runtime for generating a single sample after training Treeffuser. The average is calculated by sampling five thousand points after training the model with default settings and 50 discretization steps.

We conducted all of these experiments on a 2020 MacBook Pro with a 2.6 GHz 6-Core Intel Core i7 processor.

From these results we find that:

Treeffuser is very fast at fitting moderately sized datasets, with runtime increasing linearly with dataset size.

Sampling a single points is very fast ( ≈ 10 -3 seconds) yet, drawing many samples can become significant, e.g., with a large test set.

With respect to the time complexity of the model we note the following. If the time complexity of fitting a single GBT is of order O ( F ( |D| )) , where F : R + → R + and |D| is the size of the dataset, then the time complexity of fitting Treeffuser is O ( d × F ( n_repeats × |D| )) , where d is the dimension of y and n_repeats is the parameter that determines how many noisy versions of the dataset are sampled. Similarly, assuming constant time per GBT evaluation, the complexity of sampling is O ( n_samples × n_discretization_steps ) , where n_samples is the number of samples drawn from the model, and n_discretization_steps is the number of function evaluations used by the SDE solver.

0

20000

40000

60000

80000

100000

Number of Samples in Dataset

0

20

40

60

80

100

m5

(9000, 27)

m5

(18000, 27)

m5

(27000, 27)

m5

(36000, 27)

m5

(45000, 27)

m5

(54000, 27)

m5

(63000, 27)

m5

(72000, 27)

m5

(81000, 27)

m5

(90000, 27)

0

20000

40000

60000

80000

100000

Number of Samples in Dataset

0

20

40

60

80

100

120

m5

(90000, 27)

movies

(7415, 9)

bike

(17379, 12)

energy

(768, 8)

kin8nm

(8192, 8)

naval

(11934, 17)

news

(39644, 58)

power

(9568, 4)

superc

(21263, 81)

wine

(6497, 12)

yacht

(308, 6)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose a method for probabilistic prediction from tabular data. In our abstract and introduction, we make two central claims:

Treeffuser is accurate, in that it outperforms state-of-the-art methods in probabilistic prediction from tabular data on standard UCI datasets. This is discussed in detail in Section 5.2 and Appendix F. Table 1, Table 4, and Table 5 present the results.

Treeffuser is flexible, in that it adapts to complex distributions. This is shown in Fig. 1, which presents the results of our synthetic experiments, and in Section 5.1 and Appendix E, which provide the details.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our method in Section 3.4. Treeffuser has two limitations: the sampling process can become expensive when many samples are required for a given input instance x ; the diffusion model is specifically designed to generate only continuous responses, which may not be suitable for count data. However, in Section 5.3, we demonstrate that Treeffuser outperforms other methods even on this kind of data.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We state two theorems, Theorem 1 and Theorem 2, and discuss their implications in Section 3.1 and Section 3.2, respectively. Proofs are given in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The details needed to reproduce our experiments are stated in Section 5 and Appendices C, E and F. We also provide an implementation of Treeffuser in the supplemental material. The notebooks with the experiments can be found in testbed/notebooks/paper_notebooks .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide an implementation of Treeffuser in the supplemental material. The notebooks with the experiments can be found in testbed/notebooks/paper_notebooks . For our experiments on real data, we use publicly available datasets and point to their references [51, 55].

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The details needed to reproduce our experiments are stated in Section 5 and Appendices C, E and F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All the accuracy and calibration metrics reported in our tables present standard deviations computed with 10-fold cross validation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, we provide details in Appendix D: the number of tasks, the number of cpus and the total time taken. The memory is not reported as our work has no particular needs for memory and no particular configuration was necessary.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We provide full transparency in our methodologies and support reproducibility through shared code and datasets. There are no conflicts of interest to declare.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We are not releasing pre-trained models or datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: For our experiments on real data, we use publicly available datasets and point to their references [51, 55].

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide an implementation of Treeffuser in the supplemental material, which includes an anonymized zip file containing notebooks that document the usage of our method.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Treeffuser, a nonparametric method for modeling the output distribution. Treeffuser learns a diffusion model using gradient-boosted trees, and uses conditional diffusion models to produce a distribution over the response variable given an input vector. Since Treeffuser user gradient-boosted trees, it is adept at providing accurate predictive distributions for tabular data problems. Experiments on toy examples, real-world datasets, and a case study highlight the flexibility and effectiveness of Treeffuser for accurately modeling the output distribution for a wide range of use cases.\n\nSTRENGTHS:\n- The experimental results section demonstrates Treeffuser's effectiveness on complex contrived examples, real-world tabular regression datasets in comparison with existing methods, and a realistic case study.\n\n- The paper is well-written and relatively easy to follow.\n\n- The proposed approach is very flexible and nonparametric, enabling it to model any type of response distribution. This aspect is especially valuable for tabular regression problems, in which GBDTs do not inherently provide any type of uncertainty in their predictions.\n\n- Treeffuser works well even with default hyperparameters.\n\nWEAKNESSES:\n- As the authors note, Treeffuser must solve a stochastic differential equation to generate samples, which can become expensive if there are a large number of example to predict and each prediction requires the generation of a large number of samples.\n\n- No complexity or empirical runtime analysis is provided. This type of analysis would be greatly beneficial to readers and practitioners to help them decide if and when to use Treeffuser for their particular problem.\n\n- The comparison to other methods (Section 5.2) could be improved. For example, iBUG's $k$ hyperparameter is tuned using values [20, 150]; however, $k$ is a critical hyperparameter, and increasing the number of potential values may significantly increase iBUG's performance. Additionally, iBUG can non-parametrically model the response using kernel density estimation (KDE) which may improve its performance on datasets in which the output is not expected to be Gaussian.\n\nQUESTIONS:\n- Do the authors have negative log likelihood (NLL) results?\n\n- I'm surprised Treeffuser performs better than iBUG for point predictions (Table 4) since iBUG is simply using the underlying GBDT model, do the authors have a hypothesis as to why Treeffuser performs better than XGBoost in terms of point performance?\n\nLIMITATIONS:\nYes, the authors address the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a methodology for computing probabilistic predictions in regression problems using diffusion and gradient-boosted trees. In particular, their methodology can generate samples from p(y | x) from which statistics such as estimated quantiles could be recovered. They find that their method can learn flexible and non-standard distributions well and also works well on real-world datasets.\n\nSTRENGTHS:\nThe method results in a tractable standard supervised-ML problem, where the dataset is an augmented version of the original dataset.\n\nThe method produces good results on a range of datasets.\n\nThe paper goes into detail on the relevant diffusion equations governing the process.\n\nI liked the clever application with the newsvendor problem.\n\nWEAKNESSES:\nThe work is not clear about its contribution and what parts of the diffusion-based setup are new vs borrowed from existing works.\n\nThere could be more and better empirical comparisons.\n\nNotation and clarity could be improved.\n* Line 129 defines a vector a but that is not used in equation (11).\n* Not obvious how eqn (11) simplifies to eqn (12).\n\nThe proposed method may not be easily used or widely applicable because of the complex process needed to generate samples involving solving differential equations.\n\nQUESTIONS:\nI'm left with some basic questions about the method. What part of the contribution is due to the specific way the diffusion process was modeled and what part is due to the use of trees? How does the method differ from other diffusion-based approaches to probabilistic prediction such as the cited CARD paper? What were the key breakthroughs (if there are differences) in how you set up the diffusion problem?\n\nTagasovska and Lopez-Paz's Single-Model Uncertainties for Deep Learning (NeurIPS 2019) is a relevant work to compare to here. If I understand right it's essentially the same as your quantile regression comparison except you use trees instead of neural networks to learn f(x, q). How would the method perform with neural networks?\n\nHow easy is the SDE solver to stitch with the outputs of the trees?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

comments: SUMMARY:
This paper proposed a method called "Treeffuser" for probabilistic prediction (density regression) of tabular data.

STRENGTHS:
The paper is well-written and clearly presented the core idea as well as the main results.

WEAKNESSES:
There should be more related methods to compare with in the experiment in section 5.3, especially more recent ones.

QUESTIONS:
What are the potential applications of the proposed method, especialy given that (1) it's designed for tabular data, and (2) it is generative but cannot provide density evaluation?

LIMITATIONS:
My main concern about this proposed method is that it cannot provide exact density evaluation. This is also discussed in the limitations section by the authors.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12