Distributional regression: CRPS-error bounds for model fitting, model selection and convex aggregation
Authors: Dombry Clement
Ahmed Zaoui
Venue: NeurIPS 2024 poster (2024)
Keywords: distributional regression; error bounds; concentration inequality; continuous rank probability score; empirical risk minimization
Primary area: learning_theory
TL;DR: We provide concentration inequalities for the CRPS-error in distributional regression when empirical risk minimization is used for model fitting, model selection or convex aggregation.

Abstract: Distributional regression aims at estimating the conditional distribution of a target variable given explanatory co-variates. It is a crucial tool for forecasting when a precise uncertainty quantification is required. A popular methodology consists in fitting a parametric model via empirical risk minimization where the risk is measured by the Continuous Rank Probability Score (CRPS). For independent and identically distributed observations, we provide a concentration result for the estimation error and an upper bound for its expectation. Furthermore, we consider model selection performed by minimization of the validation error and provide a concentration bound for the regret. A similar result is proved for convex aggregation of models. Finally, we show that our results may be applied to various models such as EMOS, distributional regression networks, distributional nearest neighbours or distributional random forests and we illustrate our findings on two data sets (QSAR aquatic toxicity and Airfoil self-noise).

Distributional regression: CRPS-error bounds for model fitting, model selection and convex aggregation
Clément Dombry"
Ahmed Zaoui
Université de Franche-Comie CNRS, LmB (UMR 6623),
Université de FrancheCComte CNRS, LmB (UMR 6623),
F-25000 Besançon, France.
F-25000 Besançon, France.
clement. www.ddddnonv-eeoooooooooo
ahmed zaoui Suniv-fcomte fr
Abstract
Distributional regression aims estimating the conditional distribution of a target variable given explanatory co-variates. It is a crucial tool for forecasting when precise uncertainty quantification is required. A popular methodology consists in fitting a parametric model via empirical risk minimization where the risk is measured by the Continuous Rank Probability Score (CRPS). For independent and identically distributed observations, we provide a concentration result for the estimation error and an upper bound for its expectation. Furthermore, we consider model selection performed by minimization of the validation error and provide a concentration bound for the regret. A similar result is proved for convex aggregation of models. Finally, we show that our results may be applied to various models such as Ensemble Model Output Statisties (EMOS), distributional regression networks, distributional nearest neighbors or distributional random forests and we illustrate our findings on two data sets (QSAR aquatic toxicity and Airfoil self- self-noise).
Introduction
Motivation and related literature. We consider in this paper the distributional regression problem, where we want to estimate the conditional distribution of target random variable Y given explanatory co-variates X. We assume (X, Y) Rdd IR and we let
(w) S
x)
(1)
be the conditional cumulative distribution functions (c.d.f.). Estimation is built on a training sample D,, {(X,, Y,), 1 i n) of independent identically distributed (i.id.) copies of (X,Y).
Let us emphasize that distributional regression is much more challenging than standard regression where only point prediction for Y given X is provided which typically reduces of the conditional expectation. The conditional distribution provides a full account for the variability of Y given X and distributional regression is therefore crucial tool for forecasting when precise uncertainty quantification is required.
estimation
Distributional regression is of relevance in various applied fields. To name few, let us mention statistical post-processing of weather forecast (Matheson and Winkler 1976. Gneiting et al. 2005). forecasting of wind gusts (Baran and Lerch, 2015), of solar irradiance (Schulz al. 2021), l), of ICU length stays during COVID-19 pandemy (Henzi et al. 2021a). of breast cancer ODX score in oncology (Al Masry al. 2024)...
The methodology relies on probabilistic forecast where the forecaster produces a predictive distri- bution for the quantity of interest (Gneiting and Katzfuss 2014). Fitting distributional regression
clement, dedeeettdoombryuuniv www.:uuwwwwioooooeoeoooomo
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

model typically involves the minimization of a proper scoring rule, the most popular one being the Continuous Ranked Probability Score (CRPS, Matheson and Winkler 1976; Gneiting and Raftery 2007). It compares the actual observation with the predictive distribution in comprehensive manner.
Many different models have been proposed for distributional regression among which: Analog similar to a nearest neighbor method (KNN, [Toth] 1989), Ensemble Model Output Statistics similar to
Gaussian heteroscedastic regression (EMOS, Gneiting and Raftery) 2007). Isotonic Distributional
Regression (Henzi et al., 202 and more recent machine learning methods such as Distributional
Regression Network (DRN, Rasp and Lerch 2018 or Distributional Random Forest (DRF, Cevid ct al. 2022).
Despite these successful achievements in terms of methods and applications, a sound theory for distributional regression via scoring rule minimization still missing. Recently, minimax rates of convergence with CRPS-error have been considered for distributional regression (Pic et al. 2023). The aim of this paper is to provide statistical learning guarantees for model fitting, model selection and convex aggregation based on CRPS minimization.
Model selection and convex aggregation are very important techniques in the field of statistics and machine learning and have been used very successfully for regression function estimation, see (Tsybakvv] (2003); [Bunea et al. (007)). The methods two independent samples; the first sample, called training sample, is used to construct the initial estimators which are constituted as a dictionary (a collection of candidates). The second sample, called the validation sample, is used to aggregate them. Model selection enables the selection of the best candidate in the dictionary, while convex aggregation provides the optimal convex combination from these candidates.
Contributions. Our main results include a concentration bound for the theoretical risk when a parametric model is fitted via CRPS empirical risk minimization. We also consider model selection and model aggregation via CRPS minimization on a validation set and provide concentration bounds for the regret. Our results are first derived under Gab--aasssanity assumptions and then extended under weaker moment assumptions. We show that they apply to several popular models such as EMOS, DRN, KNN DRF and provide short illustration
different datasets.
Structure of the paper. Section 2 first provides some background on distributional regression and proper scoring rules and then presents precisely the main methods and goals. The main results are stated in Section 3 an oracle inequality for the estimation error in model fitting (Theorem I and concentration bounds for the regret in model selection (Theorem 2 and model aggregation (Theorem(3) 33). In Section g some specific models are introduced and the assumptions for our results to hold are checked. A short illustration two different data set is provided in Section Finally, an appendix contains all the proofs as well as some additional results.
Background on distributional regression and main goals
2.1 Probabilistic forecast and its evaluation with scoring rules
We first consider the simple setting of probabilistic forecast without co-variate where future observation Y is predicted by probability distribution F, called predictive distribution. Proper scoring rules are used in order to compare the predictive distribution observation which are objects of different nature. Let Po C. P(R) denote subset of the set of all probability measures on R. often identified with their c.d.f. A scoring rulef on Po is a function the error between the predictive
and the materializing
S: Po x IR + [0, +oo). The quantity S(F,)) y) is interpreted distribution F and the materializing observation y. The mean error when Y has "true" distribution G is denoted by
5(F, G) = Ey~c The following notion of proper and strictly proper scoring rule is central in the theory.
Definition 1. The scoring rule S is said proper on Po when
S(F.)) 2 S(G, G), for all E.G It is said strictly proper when equality in Eq. 2 implies F G.
FFrr the sake of simplicity. we consider only the case of real-valued observation Y and of non- negatieee scoring rule. A more general definition can be found in Gneiting and Raftery (2007).

---

Stated differently, the scoring rule S is strictly proper on Po when
arg min S(F, {G), for all G Po- PEP
The interpretation is that, in order to minimize its mean enror, the forecaster has to predict the "true" observation distribution G.
In this paper, we consider the Continuous Ranked Probability Score (CRPS, Matheson and Winkler 1976). This scoring rule is defined by the formula
S(F.y) (L(us))-F())} d=
for all finite absolute moment. Here the subset Po C ) is the Wasserstein space
Pi Pi(R) = E mi mi)
by F(dy) < xx}
of probability measures on R with finite first moment. One easily check from this definition that
S(F.G) ((EG))-G(()00) ((=) G(2) + / (F(z)- G(z)) dz,
which implies
- (F(:) (F(=) G(=)) d2.
This quantity is nonnegative and vanishes if and only if F G, ensuring that the CRPS is strictly proper scoring rule. For discrete predictive distributions of the form F Dirac mass at y), the CRPS be computed simply (Gneiting and Raftery 2007) by
w;yy, (with by the
S(F Σ
Wi yl
wowglyi M/1-
2.2
Model fitting, model selection and convex aggregation
We briefly present the methods and objectives that address in this paper.
2.2.1 Theoretical risk in distributional regression
In a regression framework, copies of (X E Rd x R. Distributional regression aims at estimating the conditional distribution Y|X = x characterized by its c.d.f. F" defined in Eq. (T]. The marginal distribution of X denoted by Px. The forecaster uses the training sample D,, and some algorithm to build functional estimator F= The accuracy of this estimator is here measured by its
observe sample P,, (I),, 1 s n} of independent
F; + Fr.z of the map F* theoretical risk
= [S [ M)
where expectation is taken with respect to the joint law of (X, Y). This quantity can be seen as the counterpart of the mean squared error in point regression. The excess risk of F,, is defined as
R((F.) R(F*) E [s(
S(F m)] = FX) 5(F F*) 2 0.
The nonnegativity ensured by the fact that a proper scoring rule according to Definition [ strictly proper, the excess risk is equal to if and only if FM..- = F; almost everywhere (with respect to Px). For the CRPS, Eq. 4 implies that the excess risk can be rewritten
R(F:) - R(F*) = E [ (BBxx(l)
FX du

---

2.2.2 Model fitting
Our first interest lies in model fitting by empirical risk minimization. Here we consider a parametric family (Fo) C RKK where Fo: z R H Fo.. E Pu C P(R). The empirical risk associated with Fg is computed on the training sample D, by
(Fo) = Σ S(Fo,x, Yi)
and is an empirical counterpart of the theoretical risk R(Fo). Empirical risk minimization consists in finding
,, = arg min (F.(F))
(5)
8ge
and proposing the estimator Fs,, which is thought as almost optimal within the family (Fo)ege. classical decomposition of the excess risk of the corresponding estimator is given by
- = (R(F
inf R(F)) + AE
inf R(Fo) -R(F)) 9Ee
where the two terms are called the estimation error and the approximation error respectively. The approximation error deterministic and depends on the ability of the family (Fe )eee approximate
F' The estimation error is random as it depends on the training sample Da. Our first goal is the following:
Goal 1: provide non asymptotic estimates for the estimation error R infoge R(Fo).
2.2.3 Model selection and convex aggregation
Our second interest lies model selection and convex aggregation via validation error minimization. Here suppose that validation sample D' {(X,, Y)), assumed independent of the training sample Dn--
1 s i s N) is available, which is
Model selection. A common situation in machine learning is that we have M algorithms at hand E.M.
that trained Dn, resulting in models F....., In order to select the best model, compute the empirical risks on the validation sample
N
Σ R'y N
S(F Y!)
and select the model
m arg min R/y (F
ISMEM
An oracle having access the theoretical risk would have selected
m arg min R(F ):
ISmSM
leading to the definition of the regret
R(F ((( = R(F min
R(Fm))
ISMSM
Goal 2: provide asymptotic estimates for the regret R (( minigmmM M R(Fm)).
Convex aggregation. Aty..., AM by
We define the convex aggregation of models F!..... AM with weights
M
AA
Am
Fu n.
m==1
Here == (A1...., As) is element of the simplex AM Am 0, Ei Am 1}. The best weights for convex aggregation obtained by minimization of the validation error, i.c.
arg min (fA).
AEAM
Goal 3: provide asymptotic estimates for the regret R R(FA) inf R(F.).

---

3
Main results
We present our main results for model fitting, model selection and convex aggregation in distributional regression. We first focus on concentration bound under sub-Gaussianit assumptions, In a second step, we extend our results under weaker moment assumptions. The definition of sub-Gaussian random variables and distributions is given in Appendix A as well as some useful concentration inequalities. All the proofs are postponed to Appendices B and
3.1 Estimation error in model fitting
We provide concentration results for the estimation error when fitting a parametric model via empirical CRPS-error minimization according to the framework described in Section 2.2.2 Our working assumptions are the following.
Assumption 1. Ksub-gaussiant
i) the variable is Si-sub-Gaussian,
ti) there exists 32 0 such that mi (FF,)) is
for all 0 E e.
Assumption I-i) is classical regression setting (Györfi et al., 2002 Biau and Devroye 2015) while Assumption ii) characterizes the sub-Gaussian behavior of the absolute moment of Fa,x- Importantly, Assumption lmplies that the variable Ze S(Fo, 2 + BuJ-sub-Gaussian for all 0 θ, see Proposition S in the appendix.
Our second assumption requires compactness of the parameter space and Lipschitz continuity of the model (Fo feee. We denote by W, the Wasserstein distance order 1 on the space Pi (R).
Assumption 2. (regularity) The parameter space C RK is compact and there exists constant L > O such that W1 Fez.)) s L||01 - B2 ll for all 01, 01,02 θ r E Rd.
This assumption ensures the Lipschitz continuity of the empirical risk 0 + R, (Fe), see Proposition 6 and, by compactness, the existence of 6n. the ERM estimator defined Eq. ( We provide in Section 4 examples of popular models verifying the two assumptions.
Our main result provides concentration bound on the estimation error. We let R be such that θ is included in the ball centered at 0 and with radius R.
Theorem Under Assumptions 7 2 for all d E (0, 1), the estimation error satisfies, with probability at least 1- 5,
Ca log Log(2nX/00
R(F inf R(Fe) S DE
(6)
n1 with cD = 64(39 + (B) and provided that n. is large enough so that log(2n** 2 /8) (48LR)//es
The inequality (6) is commonly known as an oracle inequality. The convergence rate depends on / /log(2 log((n /8)/n with K the dimension of the parameter space and n the sample size. A key point in the proof is the the combinatorial complexity of θ in terms of e-net, see Devroye et al. (199)). A bound expectation can easily be deduced from Theorem and its proof.
Corollary 1. Under Assumptions
7
C3 log(2n
(FF..) inf (Faa)
I 2
AEB
provided is large enough that (og(2n (48T.R)//cg.
In particular, the ERM approach is weakly consistent when n goes to infinity.
3.2 Estimation error in model selection and convex aggregation
We provide concentration results for the regret in model selection and convex aggregation. Recall that the methodology based on minimization the test error on a validation sample, see the precise framework in Section 2.2.3 We need the following assumption.
Assumption 3. (sub-Gaussianity)
5

---

1) the variable Y is B1-sub-Gaussian Gaussian; i1) conditionally on Dn, there exists 3n B(D,) 0 such that my (F is Bu- Gaussian for all m ,,..., M.
We will see in Section that for several popular models such as distributional nearest neighbors or distributional random forest, Assumption 3 is satisfied with Bn. maxi [Y]] which of order 31 /log /Verhhynin 2018 Exercise 2.5.8 p.25).
Under this assumption, a control of the regret in model selection is provided by the following theorem. Our results hold conditionally on the training set D,,-
Theorem 2. Under Assumption for all 5 E (0, 1), the regret in model selection satisfies
R(( P
min ISMSM
R(F S 4v cn log(2M/8)/N Pr. -
with C. = B7 + By Furthermore,
E [(((
min R(Fm I5MSM
D
8
On log(2M)
N
When selecting the hyperparameter & in nearest neighbor distributional regression mtry in distri- butional random forest, one has respectively M = n and M = if all possible values are considered Section for more details. Note that when the response variable Y is bounded, then on does not depend on n and is constant. We now state a bound for the regret in convex aggregation.
Theorem 3. Under Assumption 3 for all ó E (0, 1), the regret in convex aggregation satisfies
P(R(F2) - inf R(F.) I 8V (olgg(2N /8)/N D., 2 - 5, XEA
with c, = =B9 3h provided is large enough so that log(2N /8) Z / Furthermore, en (2g(2N
Dm S 2 (((())) inf R(F2)
AEA
N
provided N Nogg(2 2 482 /cn
3.3 Beyond sub-gaussianity
The preceding results hold under strong sub-Gaussianity. next adapt our results following weaker moment condition.
Assumption 4. There is p 2 and such that B[[y < D and [[m, S Dor all
0 E θ
We recall that, for p 2 the TP-norm of random variable Z is defined by || Eilzp/up. Theorem 4. Under Assumptions 2 and we have
[
inf 9Ee
R(Fo)ll Cn -P(((k)))
with constant > depending only on K,p.L, D. and possibly made explicit from the proof. This implies the bound for the expected estimation error
E B inf R(Fo) < Cn Cnn-p/(2ie -/(((44))) 6eB
For large, the rate of convergence tends to the parametric rate n 1/2 obtained (up to a logarithmic factor) in the sub-Gaussian case We propose additional results for model selection (Theorem 5 and convex aggregation (Theorem 6b which are, for the sake of brevity, postponed to Appendix D.2
Examples and popular models for distributional regression
We present the most popular models for distributional regression for which we want to apply our results. The first two (EMOS and DRN) are parametric, while the last two (distributional I--N and DRF) fully non-parametric.

---

4.1 EMOS and distributional regression networks
The EMOS model was designed by Greiiing et al. (005) for the purpose of statistical post-processing of ensemble weather forecast. In this framework, the predictive distribution takes the form of a discrete distributions m 1 Ii=1 oy with members y1...... ........9m.. 9m corresponding to different scenarios obtained from numerical weather predictions. Such forecast typically suffer from bias and underdispersion so that statistical post-process proopeccss is needed. The explanatory variable for distributional regression
EMOS.
Ensemble Member Output Statistics such as the ensemble mean and ensemble variance eg- In its simplest version, EMOS models the predictive distribution as a Gaussian distribution with parameters m 50 + Big and o² = P 31 u This is parametric model with 0 = (So, B1, Rn 84 in θ IR_ (0, o0) Minimum CRPS estimation is used for model fitting as described in 2.2.2. This simple yet successful method has encountered many generalizations (Scheuerer 2013 Scheuerer and Hamill 2015 Baran and Nemoda 2016) and we shall consider the following general setting. Given x E IR", the predictive distribution takes the form F(; F((- m)/o) with (a, a. R
Il+d R 4d and
(m(2;0) = a TTx 0² =oftplus) + 8 3TTz)
with softplus(u) = log(1 + c²). The predictive distribution belongs to a location/scale family and the location and log-scale parameters are linear in the covariate x. The softplus link function ensures positivity of the scale.
Distributional Regression Networks (DRN). order to consider higher dimension co-variates together with non-linear dependence, Raasp and Lerch (018)) introduce distributional regression networks. In a setting similar to EMOS, neural networks are used to model complex response funetions m(:; 0) and log 0² (2; 0). In the case of a single hidden layer with H units, the equations write
( (z; 0) = a 3 +
0² (x; 0) softplus(a' + 5' +( :
where g denotes the activation function acting componentwise, (a, a, 9') E ERIHH x R. RIIH the parameters for the output layers and (7, 5) pul+Hd the biases and weights of the hidden layer. The global parameter f = a, 8, a', 3 % 6) lies in dimension (d + 3) + 2. Note that in absence of hidden layers, DRN reduces to EMOS. Extension to a MLP structure with multiple hidden layers is
straightforward. see [Schulz and Lerch (2022) for a review of DRNs and their applications.
We next provide conditions ensuring that our results hold for the EMOS and DRN models.
Proposition 1. Let (Fa)oee be a EMOS or DRN model with parameters restricted to compact subset O Y is sub-Gaussian, X is bounded and the activation function g is Lipschitz continuous,
then Assumptions 7 2 and 3
are satisfied.
4.2 Distributional k-Nearest Neighbors and Random Forests
Distributional k-Nearest Neighbors (KNN). The predictive distribution built on a straightfor- ward extension of k-nearest neighbor regression: set
&
(y) = T: Σ 1l (X.ekun(o)) I (Xisw)
where knn(x) denotes the set of i nearest neighbors of in the training set (Xt) The main hyperparameter is the number in 2.2.3 This simple method is known the Analog method in the framework of statistical post- processing of weather forecast Toth 1989
neighbors, leading to a model selection problem as considered
Distributional Random Forests (DRF). Random forests for distributional regression have been introduced by Čevid et 2022) and are powerful nonparametric method. We describe only the main lines of the method. Recall that in standard regression, Breiman's 's Random Forest (Breiman
7

---

2001 estimates the regression function by
B B
(xx
Σ
Σ 1=1
Yu (MEL((0))>
B
(x)
b=1
where Ti... TB denote randomized regression trees built on bootstrap samples of the original data and L (z) the leaf containing x: in the tree T Interverting the two sums yields
n
B
(x) [((x)
A(x)
Y
Σ Wis (x)
B
b=1
Using these Random Forest weights, the predictive distribution writes
(v) Σ (ne(z)ll
This strategy based on Breiman's 's regression tree is used by Meinshausen (2006) to construct the quantile regression forest. Different splitting strategies in the tree construction have been considered to better detect changes in distribution rather than changes in mean (Taillardat et al. 2016)] Athey et al. 2019). Note that Cevid et al. (202)) minimizes a scoring rule to construct the splits of the trees. The main hyperparameter of this nonparametric method is the so-called mtry parameter that controls the number of co- co-variates tested at each split in the trees.
For the distributional KNN and DRF models, model fitting with ERM is irrelevant and we only consider model selection and convex aggregation.
Proposition 2. Let Fn be KNN or DRF model fitted on a training set Dn. Then Assumption satisfied with 3n = maxisign Y/|.
3 ii) is
Numerical results
In this section, illustrate how model selection and model aggregation work real data sets. The cross validation methodology, widely used in practice, is justified by the theoretical results obtained in Section 3 The source code for these experiments can be found at htttts://gith::
Datasets. We consider datasets used in the framework of heteroscedastic regression with reject option as detailed in /Zoui et al.] (2020). The first dataset, QSAR aquatic toxicity (Ballabio et al. 2019) is referred acute toxicity in Pimephales promelas. The toxicity output ranges from 0.12 to 10.05 with evidence of low heteroscedasticity, The second dataset, Airfoil Self-Noise (Brooks al. 2014) is referred to as airfoil and consists of 1503 observations with 5 measured features from aerodynamic and acoustic tests. The output represents the scaled sound pressure level in decibels, ranging from 103 to 140 with evidence of a strong heteroscedasticity.
qsar and comprises 546 observations with 8 numerical features used predict
Models. We consider the KNN and DRF models to predict the conditional distribution of the output variable. Our focus lies on hyperparameters selection via minimization of the validation error, where the main hyperparameter is the number of neighbors and the number mtry of variables considered at each split for KNN and DRF respectively. We utilize the implementation of these methods from the R packages KernelKnn and DRF For DRF, a preliminary exploration shows that sensible choices for the other parameters are num. trees= 1000, sample.. fraction=0.... min. node. size= and default values for others parameters. Finally, the R package ScoringRule is used for CRPS computation and the optim function based on the Nelder-Mead method is used for parameter optimization convex aggregation.
Methodology. We use the same methodology for the two datasets. We divide the data into three parts: 50% for training, 20% for validation and 30% for testing. In a first stage, the training set is used to train the model (KNN or DRF) for the various hyperparameters (k mtry) and the validation set is used to select best hyperparameters & or mtry. Furthermore, the validation set also used to

---

choose for model selection (MS), choosing between KNN and RF, and the best convex aggregation (CA). In a second stage, the different models (KNN, RF. MS and CA) are refitted on the union of training and validation sets (70% of data) and evaluated on the test set (30%) by computing the test CRPS-error. This process repeated 100 times for different random splits of the data into training, validation and testing sets. The distributions of the test error for KNN, DRF, MS and CA is then analyzed in terms of mean, standard error and boxplot.
Results. For the sake of brevity, only the results for the qaar dataset are presented, while the results for the airfoil dataset are postponed Appendix The first stage of the procedure where the hyperparameter (k or mtry) is selected by minimization of the validation error is shown on the left plot of Fig. T for KNN and on the middle plot for DRF. In both cases, the validation error curve shows a clear minimum allowing to select the hyperparameter k 8 and mtry = 4 corresponding to CRPS-error of 0.696 and 0.678 the validation set respectively. In the second stage, KNN, DRF, MS and CA are evaluated on the test set and the right plot of Fig. T shows the distribution of the test eπror over 100 repetitions. We see from the boxplots that the distribution of the test error is slightly larger for KNN than for DRF, that MS achieves almost the same performance as RF and that CA achieves slightly better performance than DRF. Hence model selection and convex aggregation accomplish their goal. The numerical values the means together with standard errors summarized in Table [ confirming analysis from the boxplots.
0.705
0.75
1.1
0.700
CRPS (validation)
(validaton)
0.70
0.685
CAPS (teet)
0.65
0.9
20
mtr
Figure 1: qsar data, Left and middle: selection of k for the KNN algorithm and of mtry for the DRF algorithm by minimization of the validation error. Right: test error evaluated with 100 repetitions for KNN, DRE, model selection (MS) and convex aggregation (CA).
KNN 0.643 (0.00-1)
DRF 0.624 (0.004)
MS 0.632 (0.004)
CA 0.612 (0.003)
Table qsar data. Mean of the test CRPS and its standard error (in parenthesis) over 100 repetitions.
6 Conclusion
In this work, we considered distributional regression with error assessed using the CRPS and investigated model fitting via empirical risk minimization. Additionally, we explored classical aggregation procedures: model selection and convex aggregation where two independent samples are used, the first for model construction, the second for model selection or convex aggregation. We derived oracle concentration inequalities for the estimation eтror and established an upper bound for its expectation within the sub-Gaussian framework and beyond, considering weaker moment assumptions. These new theoretical results are solid mathematical justifications for common practices the framework of distributional regression. In future work, we will study the minimax convergence rates for our approaches and consider the use of empirical process theory to strengthen results.

---

References
Al Masry, Z., Pic, R., Domby, C., and Devalland, C. (2024). A new methodology to predict the oncotype scores based clinicoppathllgiin data with similar tumor profiles. Breast Cancer Res Treat, 203:587-598.
Athey, S., Tibshirani, J., and Wager, S. (2019). Generalized random forests. Ann. Statist., 47(2):1 148 1178.
Ballabio, D., Cassotti, M., Consonni, V., and Todeschini, (2019). QSAR aquatic toxicity. UCI
Machine Learning Repository. DOI:
Baran, S. and Lerch, S. (2015). Log-normal distribution based ensemble model output statistics models for probabilistic wind-speed forecasting. Quarterly Journal of the Royal Meteorological Society, (41(691):2289-229) 2299.
Baran, S. and Nemoda, D. (2016). Censored and shifted gamma distribution based emos model for probabilistic quantitative precipitation forecasting. Environmetrics, 27.
Biau, G. and Devroye, L.. (2015). Lectures on the Nearest Neighbor Method. Springer Series in the Data Sciences. Springer New York.
Bobkov, S. and Ledoux, M. (2019). One-dimensional empirical measures, order statisties and Kantorovich transport distances. Memoirs of the Amer. Math. Soc., (66 2019. ISSN 0065-9266.
Boucheron, S., Lugosi, G., and Massart, P. (2013). Concentration inequalities. Oxford University Press, Oxford. A nonasymptotic theory of independence, With a foreword by Michel Ledoux.
Breiman, L. (2001). Random forests. Machine Learning, 45.
Brooks, T., Pope, D., and Marcolini, M. (2014). Airfoil Self-Noise. UCI Machine Learning Repository. DOI: https//bo..og//222333//cwww
Bunea, F., Tsybakov, A., and Wegkamp, M. (2007). Aggregation for gaussian regression. Annals of Statistics, 3774
Chhachhi, and Teng, F. (2023). On the I-wasserstein distance between location-scale distributions and the effect of differential privacy. Preprint.
Devroye, L., Györfi, L., and Lugosi, G. (1996). A Probabilistic Theory of Pattern Recognition. Springer, New York.
Gneiting, T. and Katzfuss, M. (2014). Probabilistic forecasting. Annual Review of Statistics and Its Application, 1(1):125-151.
Gneiting. T. and Raftery, A. (2007). Strietly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-375,
Gneiting, T., Raftery A., A.H., W., and Goldman, T. (2005). Calibrated probabilistic forecasting using ensemble model output statisties and minimum crps estimation. Monthly Weather Review, 13385):1088-111 1118-
Györfi, L., Kohler, M., Krzyžak, A., and Walk, H. (2002). A distribution-free theory of nonparametric regression. Springer Series in Statistics. Springer- Verlag, New York.
Henzi, A., Kleger, G.-R., Hilty, M. P., Wendel Garcia, P. D., Ziegel, J. F,, and on behalf of RISC-19, ICU Investigators for Switzerland (2021a). Probabilistic analysis of covid-19 patients' individual length of stay in swiss intensive care units. PLOS ONE, 16:1-14.
Henzi, A., Ziegel, J. F., and Gneiting, T. (2021b). Isotonic Distributional Regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(5):963 993.
Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58:13-30.
10

---

Matheson, J. E. and Winkler, R. L. (1976). Scoring rules for continuous probability distributions. Management Science, 22(10):087--196.
Meinshausen, N. (2006). Quantile regression forests. The Journal of Machine Learning Research, 7:983-999.
Petrov, V. (1995). Limit Theorrms of Probability Theory: Sequences of Independent Random Variables. Oxford science publications. Clarendon Press.
Pic, R., Dombry, C., Naveau, P., and Taillardat, M. (2023). Distributional regression and its evaluation with the crps: Bounds and convergence of the minimax risk. Journal of the American Statistical Association, 39(4):164-1157.
Rasp. S. and Lerch, S. (2018). Neural networks for post-processing ensemble weather forecasts. Monthly Weather Review, 146:3885-3900.
Scheuerer, M. (2013). Probabilistic quantitative precipitation forecasting using ensemble model output statistics. Quarterly Journal of the Royal Meteorological Society, 140.
Scheuerer, M. and Hamill, T. M. (2015). Statistical post-processing of ensemble precipitation forecasts by fitting censored, shifted gamma distributions. Monthly Weather Review, 143.
Schulz, B., El Ayari, M., Lerch, S., and Baran, S. (2021). Post-processing numerical weather
prediction ensembles for probabilistic solar irradiance forecasting. Solar Energy, 220:1016-1031.
Schulz, B. and Lerch, S. (2022). Machine learning methods for postprocessing ensemble forecasts of wind gusts: A systematic comparison. Monthly Weather Review, 150(1):235-257.
Taillardat, M., Mestre, O., Zamo, M., and Naveau, P. (2016). Calibrated ensemble forecasts us- ing quantile regression forests ensemble model output statistics. Monthly Weather Review, (44(6))2375-2399)
Toth, Z. (1989). Long-range weather forecasting using analog approach. Journal of Climate, 2(6):594-607,
Tsybakov, A. (2003), Optimal rates of aggregation. Learning Theory and Kernel Machines, 2777:303- 313.
Vershynin, R. (2018). High-Dimensional Probability: Introduction with Applications Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press.
Zaoui, A., Denis, C., and Hebiri, M. (2020). Regression with reject option and application to knn. In Larochelle, H., Ranzato, M., Hadsell, R.. Balcan, M., and Lin, H.. editors, Advances in Neural Information Processing Systems, volume 33, pages 20073-20082 Curran Associates, Inc.
Čevid, D., Michel, L., Näf, J., Bühlmann, P., and Meinshausen, N. (2022). Distributional random forests: Heterogeneity adjustment and multivariate distributional regression. Journal of Machine Learning Research, 23(333):1-79.

---

A Sub-gaussian distributions and concentration inequalities
The concept of sub-Gaussianity has been extensively studied and is a key tool in deriving concentration inequalities, documented in the monographs by /Vershynin (2018) and Boucheron al. (2013). For the convenience of the reader, we recall the definition and basic properties of Gaussan random variables as well as the useful Hoeffding inequality.
A.1 Sub-Gaussian random variables
Definition 2 (Sub-Gaul Sub-Gaussian random variable). Let 3 > O. The random variable X is said to be A-sub-Gaussicn if
E [[xI
< x
for all X
Proposition Let B,
i) X is A-sub-Gausside then [X] is also
ii) If X is bounded random variable such that X E [a, almost surely, then X is B-sub- Gaussian with 3 = =bb a2/2.
iti) If H,YY are random variables such that X is B-sub-Gaussian and 0 S Y S X, then Y is also 3-sub- Gaussinn
kv) If X; is Bi-sub- Gaussian for = 1,2, the sum Y X, + X2 E-sub-Gaussan with
3 = 22 32).
A.2
Concentration inequalities
In this section we gather several technical results which are used to derive the contributions this work. We start with the general Hoeffding's 's inequality (Vershynin 2018 Theorem 2.6.2 p. 28).
Proposition (Hoeffding 1963)). ) Let Zi...., Zn be independent random variables such that Z; is for all 1..., n. Then, for all 2 0 we have
Σ Z - (B12)) 2t
nil2
IP
S 2 eexp
2,3
The following Lemma is often useful to derive upper bound in expectation from concentration inequalities.
Lemma 1. Let 2 1, b and Z positive random variable such that
(Z 2 t) S exp(a bl2) for all t 2 0.
Then we have
v a 2
a b
B[Z] S 1
V
Proof. The assumption implies that P (Z 2 <
min (1, exp(a bt?)) for all t 2 0. Then,
E
P(Z 2 t)dl V a
to0
exp (-(bt² - a)) dl.
(8)
Varb
Since (u - v)² < 22 2² for u, we have
It exp
+00
to0
exp (-b
(a/V)))))) di VR
V
((bl - a)) dl <
(9)
Vafb
2vb
2
where the last inequality > 1. Combining Eq. and Eq. 9 yields the result.
12

---

A.3 Application to the empirical risk
The results from the last two sections are applied the CRPS and the empirical risk. Proposition S. Under Assumption 7 for all 0 E O the random variable S(Fo,x,Y) Y) is B-sub-
Gaussian with 3 = /2 + B2).
Proof. We use following alternative representation of the CRPS, see Gneiting and Raftery (2007): for fixed F E (I)) and E IR.
S(F.)) El - - Z
where Z, Z denote independent random variables with distribution F. The triangle inequality then implies
| + BliZ 2 IZ Z S(F v) E yll BiIz' -
2
1y + [|] l4v mm
with mi (F) the absolute moment of F. We deduce that, for all A E e,
S(Fe.x, Y) s (i
(10)
where, by Assumption Y and mi (Fa,x) are sub-Gaussian with parameter 31 and 32 respectively.
Then, Proposition 3 implies that S(Fe. Y) is B-sub-Gaussian with B 2 + 32)
L
Corollary 2. Under Assumption ! for all 0 E θ the empirical risk R., (F)) satisfies the concentra- tion inequality
nt²
P Rn (Fo) - (Fa)))(((F)))) 21) S 2exp
for all 2 0
2,3
with 3 = v 203 32).
Proof.
The random variables = Yi), Sis S n, are iid. with expectation
E[Za.] ((S(Fe,F)))) Y,) = -R(Pa)
and empirical mean
n
1 Σ S(Fo,x,: Y (Fo) n
Zo.
72
By Proposition 5 Zo.; is 8-sub-Gaussian with 3 2( + B2). We can then apply the general
Hoeffding's 's inequality (Proposition 4 and deduce the result.
B Proof of the results of Section 3.1
Before proving Theorem T we introduce preliminary results about the Lipschitz regularity of the CRPS and the empirical risk.
Lemma 2. For all F1, F2 E Pi (R) and IR, we have
|S(F1, y) S(F2. v) S (F,
This Lemma states that the CRPS is 2-Lipschitz the first variable with respect to the Wasserstein distance W1
Proof. By the definition 3 of the CRPS,
S(F1,)) - S(F2, = - ("(use) Fi()))) (I Fa(#))?) dz.
13

---

Using a² - = (a- () + b). we get
[S(Fi.u) - u)) - / [F((z)) - (2))) +=() 21 (=sw)ldz
s 2 / F
We recognize the expression of the Wasserstein distance of order 1 for probability measures on R as the -distance between their cdf, i.e. the formula
/
[i(()) d2 (F..F.F)).
see Bobkov and Ledoux (2019). The result follows.
We can deduce that, under Assumption 2 the theoretical and empirical risks are also Lipschitz continuous.
Proposition 6. Under Assumption
2 for all 01, 02 O, (F(FF) - R((Fe)) ) - 2 02ll
and
Rr (Fo,) - (Fo.). - 2L|16] 0211-
Proof. Using the definition of the theoretical risk and the Lipschitz continuity of the CRPS, have
[R(Fa,) R(Fo.)) = [F Y) S(Fe2.x, Y)]
< [F Y) S(Fez, Y)[l < 2 (Fo,,x, Fo2.x)]-
Then, Assumption 2 implies
(R(Fo.)) R(Fo,))l S - 021l-
Similarly for the empirical risk,
1
(Fo,) Rr (Fo2))
S(Fo,,x,> Yi) S(Fe Yi)l
n1
2
Wi(Fos.X,, Fos.x,)
1=1
< 2 - 821l.
We are now ready for the proof of Theorem
Proof of Theorem 7 By compactness of O and continuity of 0 + R(0), the theoretical risk reaches a minimum on è and we can define 0* arg mingee R(Fo). Similarly, Ú, = arg mingge R., Fb) is well defined. The estimation error can then be decomposed into three terms
R(F inf R(Fo)) #EB
=R(F R(Fo-)
= (RR(Fb.) - (Fo.)) + (Fo,)) - Rn(()) + (Rn (Fo-) -(o-)----(- R(For)
(11)
By definition of 6., the second term is non- non-positive and we deduce
R((F,))
inf R(Fo) seB
S 2 sup
Rr, (Fo) R (Fo)
(12)
14

---

Since ④ is compact, it can be included in some centered closed Euclidean ball BR with radius R > 0.
Therefore, for all S R, there exists an c-net + of such that card((,)) S (RR see Devroye al. 1996). The term e-net means that, for all 0 E θ, there exists 0₂ E O₂ such that ||e 8. S Now, for all A E 6,, we introduce the decomposition
RR, (Fa) - R (Fa) -
Rr, (Fe) - Rr (Fe.) + RR, (Fe.) - R (Fo.) IR (F&.) R (F+))
By the Lipschitz properties stated in Proposition 6 the first and third terms are bounded from above by 2Lc Therefore, we deduce
2 sup 8E6
Rr (Fo) R (Fo) 8Lc + sup B
RR, (Fo) R (Fo)
(13)
which implies, for t 2 161e,
P 2 sup AEG
Rr (Fo) R (Fb) 2t) S sup MEO.
Rr (Fo) R (F6) 2t/4)
(FF)
We then use Corollary
stating that, for all O
(- nt²
([ ( R(Fo) 2t) S exp
2,3
with 3 = 22 + 39). The union bound and the incquality card(O.) S $32R)
(3R)
imply. for
t 16LE,
IP 2 sup AE6
Rr (Fo) - R (Fo) 21) s
Ru (Fo) - R (Fa) 2 114
AEO.
3R
ntt²
2 2
exp
32,82
Setting c = 3R/n in this inequality and using Eq.
12
we deduce
p (( (R((F))) inf R(FO) 2t) 2n exp
nf.
48LR
t>
(14)
32,32
The right hand side is equal to 8 for t = /32,32 (2n 18)/n, yielding the result. Note that we need this specific choice of / to satisfy 2 48LR/n, or equivalently n log(2n 10) 2 (48LR)2/es /ca with C3 = 64(37 + (3))
Proof of Corollary 7 Setting a = log(2n*) and n/cc,, Eq. 14 can be rewritten as
[R(PO.)
inf R(Fo) 2 21) S exp (a bi²), 2 48LR/n.
BEE
Fort S Va/b, the inequality trivial because the right hand side is larger than 1. We deduce that the inequality holds for all 2 0 as soon as 48LR/n S Va/b which is equivalent to n log (og(2nk) 2 (A) /ca. which holds for n large enough. Then we can apply Lemma I and deduce the upper bound 2 a/b for the expectation of the estimation error.
C Proof of the results of Section 3.2
Proof of Theorem 2 A straightforward adaptation of the proof of Proposition
5 shows that As-
sumption implies that the random variables S(fm Y), m 1,..., M, are B-sub-Gaussian
conditionally on D, with 3 /2( + (2). Then, similarly as in Corollary 2 Hoeffding inequality implies
N+²
P (RR - R(PT)) 2 Dm) s 2 2exp
(-4(0p + (2)
(15)
15

---

for all t 2 0 and m = i....,,..... M. With similar reasoning as in Eq. 111 the regret in model selection
bounded from above by
R(F - min ((F..) 1 SMSM
R(F R(PmT))
(R(PT)) R'y (F + (R'( ((m)))))) + (frm") R((rr))
2 max I
R2 (F R (PT))
(16)
This together with the union bound and Eq. 15) implies
PP (R(F.))
min ISMSM
R(F:")
Dn
P
max ISMSM
R (P:)) - R((
t
Dr
( R( (F
/
Dr
2
m
N12
$ 2M exp
16(B} + (3)
The right hand side is equal 5 for f second claim follows by an application of Lemma with a log(2M) and = N/(16c,). L
4v Cn log (MM which proves the first claim. The
For the proof of Theorem 3 we need the following Lemma.
Lemma 3. Consider cumulative distribution functions (mm") and, for À E A, the convex aggregation G - Em=1 AmG.m Then the following Lipschitz property satisfied: for all X1, X2
A.
((GG', S
max m1(G" ISMSM
1| d2ll1
Proof. In dimension 1, the Wasserstein distance is given by
Wi
G^i ())(y)
By definition of GA we deduce
M
Wi Wi ,,²))=))>>00)
Σ (A1,m )2,m) - (o20))(dy
M
Di A2ml / (TT()) - (v207 dy
m=
M
Σ m=1
A2ml commmm( (G
< max mi 15m5
mi (GTh) || A21l1.
In the first line, we use that Em= (A1. A2,
that we can introduce the indicator function 1 (w20)- In the third line, we use J 1G (v) 1 (x20) Idy W1 Wi(Gm, 5o) m. (G
M
Proof of Theorem 3 We work conditionally on Dn so that fm S m S M, can be seen as deterministic. The empirical risk computed on the validation set D'y with cardinal N. Furthermore, Assumption implies that, conditionally on Dn, Y and F satisfy Assumption with θ replaced by A and constant (3 replaced by Bn. Lemma implies that, conditionally D,, Fè satisfy Assumption with θ replaced by Aas and constant L. replaced by VM maxi M my (Fm)). As consequence, apply Theorem I and deduce the result.
16

---

D Proofs and additional results for Section 3.3
D.1 Proof of Theorem 4
Without Gaussian assumptions, we cannot use Hoeffding inequality. Alternativel, we use consequence of Rosenthal inequality providing simple control of moments. For a reference. see Petroy (1995) Theorems 2.9 and 2.10).
Proposition Let Z, Z1,.... ........,. Z- be i..d. random variables such that my = 1RI < o for some p 2 2. Then,
E
(Z-
B[21)
c(p)mp(Z)m
n
with c(p) 0 depending only on p.
We deduce a simple moment bound for the empirical risk.
Proposition 8. Under Assumption 4 we have, for all 0 θ
E (R.(F R((F))
|"] s Dn
with (p)p 4p) depending only on
Proof.
Eq. 100 together with the upper bound + bl < 20 (|a|P + [b(²) imply
[S(Foxx,Y) s (IY) + m² m1 (Fo,x)) ))" (PY m2 (F),x)").
Taking the expectation and using Assumption 4 we deduce, with the notation Zo ((Foxx,
E| E| < 2 (DY + m. (Fo,x))) 2PD.
By Jensen inequality, this implies the upper bound
E I E[Zo] < 4PD.
The centered empirical risk can be written as
(Fo) - R((F)) 1 Σ Zo.5 B(Zo.s)
with Ze,; = S(Fa.x., Yj),
1 < i < i.idd random variables. Then the result follows from an
application of Proposition
The beginning of the proof follows exactly the same lines: according Eq. 12 13 for all e-net e O, we have
Proof of Theorem 4
We use the same notation as in the proof of Theorem
(R(Fs.)
inf R(Fo) I SLe + 2 sup R. (Fe) - R (FF) DE0
(17)
SEe.
This implies
R(Fo inf R(F)) |1] < 22 + 2
Rr, (Fo) - R (Foo)
sup .Ee.
8eB
The expectation in the right hand side is bounded from above by
R. (Fo) RR (Fb)
sup OEO.
oee.
(3R)* cl)) Dm p/2
where the last inequality follows from Proposition 8 and the bound card (+.) < (3R/e)K, Hence deduce
E
R(Fo.) inf R(Fo)
( (8Le)++2(31)) Dn Dnp/2) -/2))
2-
dee
17

---

Minimizing the right hand side with respect to e according to Lemma beelow and taking the p-th root, we obtain
E B inf R(Fb) DEG
1/P
I On Cn (/((+())))))
where the constant C= = C(K, L, D, R) does not depend on n and can be made explicit thanks to Lemma 4
Finally, Jensen's 's inequality yields the result.
Lemma 4. For all a,b > 0 and p, q > 0,
inf (ace + be b)= Cp.grtchofy <00
with Cp.1
2
Proof. A straightforward analysis of the function e + ac + be shows that its derivative vanishes where the minimum is reached.
by
at c
D.2 Additional results
We provide some additional results to Theorem 4 from Section 3.3 regarding model selection and convex aggregation. Our working assumption in this framework the following.
Assumption 5. For parp2 2,
i) 1 D:
ii) conditionnaly on Da,, Im (F ) S Dn D(D.) for all m 1, 1,..., M.
For instance, one can show with a reasoning similar to Proposition that for the EMOS and DRN models, Assumption 55 holds bounded. For the KNN and DRF models, a reasoning similar to Proposition shows that the assumption holds with D, maxigis. |.
as Y and F have finite moment of order p and X remains
Model selection. Here is our result model selection under moment assumption only. Theorem 5. Under Assumption
5 have
E B[R(F
R(Fm Pn] s ((c(()) ma D.)M) 1/p N-12/
min
ISM5M
with e' (p) the constant from Proposition
g
Before proving the Theorem, establish the following result.
Proposition 9. Under Assumption 5 we have, for all 1 S S M,
E R R(Fm))) ) Da s dp) ma D, D.)N p/2
Proof. We work conditionally on D, so that fm can be seen as deterministic quantity. The empirical risk is computed on the validation set D/y with cardinal N. Furthermore, Assumption S implies that, conditionally D,,, Y and i' satisfy 4 with constant D replaced by max DD D,) Given these remarks, Proposition folloows by an application of Proposition g where Fy replaced by Frmm Rr, is replaced by R'y and the expectation by the conditional expectation given Dn..
Proof of Theorem Thanks to Eq. 16)
R( R(F
min R(Em)) S 2 max
(F.))) - R((m)))
ISMEM
18

---

whence we deduce
E (((()))
min R((m)) ISMSM
Dn - E
max 15m M
('y(F - R(Frm)) / Dnn
M
Ri R(Fm)) Dn
x 2"
m==
2 Mc max (D, DD.
The result follows by Jensen's inequality.
Convex aggregation. Here our main result on convex aggregation under moment assumption
only.
Theorem 6. Under Assumption 5 we have
cc max(D, m/3) 1/p+K
E
R(FA)
inf R(FA) Dr
λEλM
with L = VM max gax my (FTT) and constant > depending only on and K.
Proof of Theorem The proof follows by an application of Theorem 4 exactly in the same way that
Theorem was derived from Theorem
E Proofs for Section
The following Definition and Lemma about location/scale families are useful for the proof of Section
Definition 3. Let be a probability distribution on R with zemo mean and unit variance. The associated location-scale family is defined as {Fmoor R, > 0} where Fm, is the law of m + o Z with Z F.
Proposition 10 (Chhachhi and Teng 2023). For all mı. m2 R and 01, > 0. we have Wi Wi(Fmaos F 02) S [m] 1 mal mP( 01 02/-
Proof of Proposition Denote by F ((x:))) the output distribution of the DRN with parameter 0 @ and input x, where m( 8) and (x;; () are the location and scale parameters given by Eq. (7p. Because the activation function g is assumed Lipschitz continuous, when and remain in bounded sets, the functions 0 + m(2; 0) and 0 + oz; () are Lipschitz continuous with a Lipschitz constant bounded by some constant C' uniformly in æ. Then, Proposition 10 implies that the output distributions satisfy
W1 (( F Fm(x;0))) I (2; 01) m(2; 02) + m (F) (o(;))))))))) (x; 02)
S SC mn (mmmm())))) (F)) 02ll-
for all 0 in @ compact and æ in the support of X bounded. It follows that Assumption 2 is satisfied.
Furthermore, this estimation of the Wasserstein distance implies that the absolute moment mi (Fe,x)
remains bounded for all 0 E θ and x in the support of X. Hence mi is sub-Gaussian (uniformly in 8) and this proves Assumption| | For the same reason, the absolute moment m, mi (f.))
mi (F& ",,,x x remains bounded and this implics Assumption 3 with (D,,, constant not depending on the training set.
Proof of Proposition The two models KNN and DRF can be written in the form
F = Σ
T=1l
for suitable probability weights (wnr(x)) Hence the absolute moment of the predictive distribution satisfies
my (Fn,)) =
Σ wii (V) < max [Y]]-
1si5n
We can see that the random variable my (f.,xx remains bounded and is hence 3u-sub--aaat Sub-Gausssin with 3n maxisisn [Y], which proves Assumption 3
19

---

Additional numerical results
The numerical results for the airfoil data is presented in Figure 2 and Table The left plot of Figure 2 corresponds to KNN, and the middle plot to DRF. In both instances, the validation curve exhibits a clear minimum, enabling the selection parameter values = 3 and mtry 5 associated with CRPS-error of 2.16 and 1.47 on the validation set respectively. Then the models KNN, DRF, MS and CA are tested on the test set, and the right plot of Figure 2 shows the test error evaluated with 100 repetitions. The boxplots reveal clear difference between the test errors for KNN and DRF in favor of DRF. In this case, MS achieves the same performance as DRF (in fact DRF is systematically selected), and a slight improve achieved with CA. These results are confirmed by the numerical values in Table 2
2.5
CRPS (validatio)
(velidation)
2,00
2.4
CRPS (teec)
1.75
1.50
222
20
50
KAN
ORF
MS
mtry
Figure 2: airfoil data. Left and middle: selection of for the KNN algorithm and of mtry for the DRF algorithm by minimization of the validation error. Right: test error evaluated with 100 repetitions for KNN, DRF, model selection (MS) and convex aggregation (CA).
KNN 1.864 (0.015)
DRF 1.366 (0.004)
MS
CA 1.344 (0.005)
1.366 (0.004)
Table 2: airfoil data. Mean the test CRPS and its standard error (in parenthesis) over 100 repetitions.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims presented in the abstract and introduction accurately reflect the contributions and scope of the paper. The abstract outlines the study's 's focus on distributional regression and its importance for uncertainty quantification in forecasting. mentions key methodologies such as fitting parametric models using empirical risk minimization with CRPS. The claims about providing concentration results for estimation error, bounds on expectation, model selection, and convex aggregation via validation error minimization, as well as application to various regression models, are all consistent with the paper's contributions and scope as described. Additionally, the examples cited (QSAR aquatic toxicity and Airfoil self-noise datasets) demonstrate practical applications of the discussed methodologies. Therefore, the abstract and introduction effectively summarize the paper's objectives, findings, and relevance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The approach is very general and may apply to parametric and parammrmtttic methods and does not assume any model assumption. Observations are assumed independent and identically distributed which is a very standard assumption in statistical learning. The results are derived under nub-Gaussian assumption and a weaker moment condition is also considered.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
21

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
the different assumptions are clearly stated in the suitable Assumption environ- ment. At the beginning of each Theorem, the relevant assumptions are recalled. Furthermore, the assumptions are discussed and checked variety of different examples. All the proofs are provided and carefully explained in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the details of the numerical experiment can be found in the Models and Methodology parts of Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
22

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Data and code are available on the Github repository https:///gthheii com/
Zaoui httplls2222____tri1____ wtrterrrbuuiinnaaeeeeeeee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 htJps.cc/public/guudee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the details of the numerical experiment can be found in the Models and Methodology parts of Section 5 (data description, methods and algorithms used, hyperpa- rameters, optimizers...)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
23

---

7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: standard error are provided in parenthesis in Table 1 and reveal the statistical significance the results (e.g. the convex aggregation is significantly better than the other models).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [NA]
Justification: The main purpose of the paper is theoretical and the numerical experiment very quick to run so that it is not useful to discuss the computer resource.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics / http://wwwrpp.cpp.u.o.cccccsttaa
Answer: [Yes]
Justification: Nothing specific declare.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The introduction remind the reader that uncertainty quantification is important for decision making.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
25

---

Answer: [Yes]
Justification: proper citations provided for data sets and algorithms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's ' creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
26

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "aHI5b0Kype", "reviewer": "Reviewer_GnPM", "summary": "The paper provides theoretical guarantees for distributional regression, which aims to estimate the conditional distribution of a target random variable Y given covariates X.    These theoretical guarantees hold when the regression is learned by minimizing a particular proper scoring rule, the Continuous Rank Probability Score (CRPS), in the presence of i.i.d. data (X,Y).    Concentration bounds on the estimation error are provided for model fitting, model selection, and model averaging.", "questions": "1.) The authors write: \"[...] distributional regression is way stronger and more challenging than standard regression, where only a point prediction for Y given X=x is provided which typically reduces to an estimation of the conditional expectation.\"  I found this statement a bit puzzling.    For instance, both frequentist and Bayesian GLM's would seem to estimate conditional distributions of a target given explanatory variables, rather than simply providing point estimates.    Could the authors clarify?\n\nEditing notes:\n\n1.) \"Way stronger and more challenging\" (line 20) is redundant, and the first phrase is too colloquial.  Streamline to \"more challenging\" or perhaps \"much more challenging\". \n\n2.) It is unclear why \"we will mostly use the CRPS for discrete predictive distributions\" (line 87).  Please clarify.\n\n3.) Can the Wasserstein space `P_1(\\R)` (defined between lines 83 and 84) be equivalently described as the set of probability distributions over `\\R` with finite first moment? If so, this might be worth stating explicitly.", "limitations": "No limitations were provided.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.) The paper gives perhaps the clearest presentation I've ever seen when reviewing for one of the top ML conferences.   Its acceptance is almost warranted from this fact alone; there is significance in the exceedingly clear overview of distributional regression. \n\n2.)  The provided concentration bounds cover multiple modeling goals (model fitting, model selection, and model averaging) and a wide range of popular models for distributional regression (both parametric and non-parametric).", "weaknesses": "1.) The experiments do not seem strongly related to the theory, other than that the experiments investigate distributional regression.  I expected the theoretical results to play a stronger role here.   Is there some way in which the theory guides the data analysis?  If not, what is the purpose of including an experiment at all? If nothing else, the experiments would seem to provide an opportunity to perform sanity checks.  For example, can the authors provide empirical evidence of consistency to the theoretical limiting values?  \n\n2.) Generalizing #1, the authors do not provide guidance on what their theoretical results can be used for.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the positive assessment of the quality of the paper presentation! Please find below answers to your questions and concerns.\n\nWeaknesses:\n1) We agree that the experiment is loosely related to the theoretical results; our aim is to illustrate that the methods of interest (model selection, model aggregation) work well in practice; ideally it would be interesting to illustrate the convergence to zero of the estimation error, but we have to take into account that the theoretical limit value for the error (oracle) is unknown in our setting. \n2) The main message is that the empirical risk minimization widely used in practice (be it for model fitting or model selection) is mathematically grounded. We have clarified and strengthened this message in the sections \"Numerical Experiments\" and \"Conclusion\". See in particular the sentence \"These new theoretical results are solid mathematical justifications for common practices in the framework of distributional regression\".\n\nQuestions:\n1) Indeed, generalized linear models can be seen as distributional regression techniques since the conditional distribution of Y given X=x is modelled by a distribution belonging to an exponential family.\n2) Editing notes have been taken into account in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J5gBUjT5da", "reviewer": "Reviewer_uh9Z", "summary": "The paper considers the problem of distributional regression, i.e., learning the distribution of Y conditional on X. In particular, the distributional regression problem is formulated as an empirical risk minimization problem, where standard concentration techniques are applied to obtain non-asymptotic bounds on the excessive risks.", "questions": "The current paper mainly considers the case of iid data, and correspondingly leverages concentration tools for iid data. I wonder if it could be generalized to adaptively collected data, since there are parallel concentration tools for such data.", "limitations": "The paper has discussed its limitation.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The question is well-motivated, and the technical development is solid.", "weaknesses": "After formulating the problem as an empirical risk minimization problem, applying concentration techniques appears to be very standard --- the technical contribution of the current paper is relatively weak.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for overall positive appreciation of the paper! Please find below answers to your questions and concerns.\n\nWeaknesses:\nWe agree that the technical development based on Hoeffding inequality is quite standard. Still, the paper provide the first detailed analysis of the CRPS risk for distributional regression and original contributions appear in this direction: Proposition 5, Lemmas 2 and 3 provide original results for the analysis of the CRPS, Propositions 1 and 2 provide original results for the analysis of various models (distributional nearest neighbours, EMOS, distributional neural networks). Our point of view is that the value of the work is to apply (standard) concentration techniques in an original context and to provide new insight into distributional regression, which is a fundamental task in statistics and machine learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RBNwWB7UUz", "reviewer": "Reviewer_TyYn", "summary": "This paper considers the problem of conditional distribution prediction. For covariate-response pair $(X, Y)$, the objective is to estimate the conditional distribution of $Y|X=x$ for all $x$. The paper provides concentration bounds for the empirical risk minimization (ERM) estimator with continuous rank probability score (CRPS) as loss. It also provides regret bounds for the model selection and model aggregation with CRPS minimization.", "questions": "As given in the \"Weakness\" section.", "limitations": "As given in the \"Weakness\" section.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is well-structured and clearly written. The proofs are rigorous, and the notations are well-organized. It offers theoretical guarantees that will benefit future research involving the CRPS minimization technique.", "weaknesses": "1. The proving technique for the theorems presented in the paper is pretty standard. The work lacks technical contribution.\n2. In Theorem 1, the definition of $R$ which denotes the boundary of the parameter space $\\Theta$ should be added in the main text. \n3. The theorems have not provided insights for the algorithm design. The paper has not proposed new algorithms, but in line 262 it says \"demonstrate the effectiveness of our proposed methods\".\n4. The entire experimental section does not utilize the theoretical guarantees established earlier. Additionally, there is a lack of experiments to validate the effectiveness and tightness of the theoretical bounds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for overall positive appreciation of the paper! Please find below answers to your questions and concerns.\n\nWeaknesses:\nWe agree that the technical development based on Hoeffding inequality is quite standard. Still, the paper provide the first detailed analysis of the CRPS risk for distributional regression and original contributions appear in this direction: Proposition 5, Lemmas 2 and 3 provide original results for the analysis of the CRPS, Propositions 1 and 2 provide original results for the analysis of various models (distributional nearest neighbours, EMOS, distributional neural networks). Our point of view is that the value of the work is to apply (standard) concentration techniques in an original context and to provide new insight into distributional regression, which is a fundamental task in statistics and machine learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fRXdr6Qa9Q", "reviewer": "Reviewer_kUjo", "summary": "The paper studies distributional regression, a statistical technique that models not just the conditional mean of the response variable given the predictors (as in traditional regression) but the entire conditional distribution. The learning task minimizes the risk function, measured by the Continuous Rank Probability Score (CRPS). The authors aim to estimate the error bounds for the expectation of the predictive distribution. They also investigate model selection and aggregation via CRPS minimization on a validation set.", "questions": "See Weaknesses.", "limitations": "The limitations have not been properly discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* Comprehensive Approach: The use of distributional regression provides a more holistic view of the data by modeling the entire conditional distribution, offering more insights compared to traditional regression methods. This work explores various aspects of this model and provides a thorough analysis.\n\n* Theoretical Contributions: The authors present a robust theoretical framework, including the estimation of error bounds for the expectation of the predictive distribution. This adds valuable understanding and reliability to the model's predictions.\n\n* Model Selection and Aggregation: The paper addresses practical aspects such as model selection and aggregation using CRPS minimization on a validation set, which is crucial for effectively applying the methodology in real-world scenarios.", "weaknesses": "* Typos in the Text: The text should be checked again by the authors to remove the typos (e.g., line 88: \"on can show\" should be \"one can show\").\n\n* Math Symbols and Abbreviations: Several math symbols or abbreviations are used in the text without prior definition, reducing the readability of the text. For instance, \"EMOS\" in the abstract or $\\mathcal{G}$ in line 69. Additionally, some symbols are used multiple times with different meanings. For example, $\\mathcal{D}$ in line 19 indicates the training set, while $\\mathcal{D}$ in line 71 denotes a subset of the set of all probability measures on $\\mathcal{R}$. It would be better to use symbols more carefully and ensure consistency throughout the text.\n\n* Comparison with Existing Methods: While the paper focuses on the advantages of the proposed CRPS-error bounds for distributional regression, it would be beneficial to include a comparison with existing techniques, both in terms of theoretical properties and empirical performance. Related works and available baselines have not been properly discussed in the paper. Including such comparisons would provide a clearer context for the contributions of the proposed method and its relative performance.\n\n* Complexity Analysis: A detailed complexity analysis is needed to understand the computational cost of the proposed method, especially in comparison with other techniques.\n\n\n* Numerical Analysis Clarity: The results presented in the numerical analysis should be clearer, with a more detailed explanation of what the numbers represent and how they support the claims made in the paper. The numerical analysis is not sufficient, and not enough sensitivity analysis has been provided. Only two small datasets have been used, and the simulation does not cover all aspects of the model. It would be better if the authors designed new experiments with different datasets and settings to comprehensively validate the proposed method. This would help demonstrate the robustness and applicability of the approach across various scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your work and time for reviewing the paper. We believe indeed that distributional regression is an important technique providing a more holistic view of the data and we stress that, despite its importance in practice, CRPS minimization in distributional regression has been very little investigated from a theoretical point of view. Please find below answers to your questions and concerns.\n\n1) We have run again a spell checker and corrected a few typos.\n2) Following your advice, we have used a different notation for $\\mathcal{D}$ (replaced by $\\mathcal{P}_0$) in order to avoid confusion with the training set $\\mathcal{D}_n$. We have also checked that abbreviations are correctly explained when introduced for the first time.\n3) We have chosen to focus in this work on the CRPS minimization technique because it is the most widely used in applications of distributional regression (see the references in the introduction), in particular in the context of ensemble forecast where the log-score and likelihood methods are not available. Despite it common application, there is a shortage of theoretical results regarding CRPS minimization for distributional regression which is the main motivation for the paper. Comparison with existing methods (e.g. minimization of other scoring rules) is left for further research.\n4) Strictly speaking, we do not propose a new method but rather provide solid theoretical ground for existing methods. The original part of the paper is thus the concentration bound establishing the consistency of commonly used method in distributional regression. This is why our paper does not provide a detailed numerical analysis (only an illustration) nor a complexity analysis, but rather focuses on the theoretical aspects.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We acknowledge the referees for their work and time spent on the paper. Overall, the comments are positive but two main concerns emerge from the reports that we comment below. All other (minor) suggestions have been taken into account.

1) Technical contributions --
We agree that the technical developments and concentration inequalities based on Hoeffding inequality are quite standard. Still, the paper provide the first detailed analysis of the CRPS risk for distributional regression and original contributions appear in this direction: Proposition 5, Lemmas 2 and 3 provide original results for the analysis of the CRPS, Propositions 1 and 2 provide original results for the analysis of various models (distributional nearest neighbours, EMOS, distributional neural networks). Our point of view is that the value of the work is to apply (standard) concentration techniques in an original context and to provide new insight into distributional regression, which is a fundamental task in statistics and machine learning. We believe the work will benefit future research involving CRPS minimization techniques in distributional regression.  

2) Numerical experiments --
We agree that the experiment is loosely related to the theoretical results; our aim is to illustrate that the methods of interest (model selection, model aggregation) work well in practice; ideally it would be interesting to illustrate the convergence to zero of the estimation error, but we have to take into account that the theoretical limit value for the error (oracle) is unknown in our setting. 

We very much hope that the reviewers will appreciate this point of view and support the work for presentation in Neurips.

Meta-review: The paper provides concentration inequalities for the empirical risk minimization (ERM) estimator with continuous rank probability score (CRPS) as loss in the context of distributional regression, which aims to estimate the conditional distribution of a target random variable $Y$ given covariates $X$. Concentration bounds on the estimation error are provided for model fitting, model selection, and model averaging. The proof techniques are standard but the paper contains interesting results. The authors argue that the main takeaway of their theoretical results is that "the empirical risk minimization widely used in practice (be it for model fitting or model selection) is mathematically grounded". However, it might improve the paper substantially if the author(s) could use the theoretical results to provide deeper insights for the ERM methods.
Decision: Accept (poster)