Probabilistic Forecasting: A Level-Set Approach
Authors: Hilaf Hasson
Bernie Wang
Tim Januschowski
Jan Gasthaus
Venue: NeurIPS 2021 Poster (2021)
Keywords: probabilistic forecasting
quantile regression trees
consistency

Abstract: Large-scale time series panels have become ubiquitous over the last years in areas such as retail, operational metrics, IoT, and medical domain (to name only a few). This has resulted in a need for forecasting techniques that effectively leverage all available data by learning across all time series in each panel. Among the desirable properties of forecasting techniques, being able to generate probabilistic predictions ranks among the top. In this paper, we therefore present Level Set Forecaster (LSF), a simple yet effective general approach to transform a point estimator into a probabilistic one. By recognizing the connection of our algorithm to random forests (RFs) and quantile regression forests (QRFs), we are able to prove consistency guarantees of our approach under mild assumptions on the underlying point estimator. As a byproduct, we prove the first consistency results for QRFs under the CART-splitting criterion. Empirical experiments show that our approach, equipped with tree-based models as the point estimator, rivals state-of-the-art deep learning models in terms of forecasting accuracy.

Probabilistic Forecasting: A Level-Set Approach
Hilaf Hasson
Yuyang Wang
Amazon Research waw.llaaaaaa....co com
Amazon Research
yuyasangdamazon.comm com
Tim Januschowski Amazon Research t tjnsch@amazon. com
Jan Gasthaus
Amazon Research gas thwuseamazon.cccmm com
Abstract
Large-scale time series panels have become ubiquitous over the last years in areas such as retail, operational metrics, loTT and medical domain (to name only few). This has resulted in a need for forecasting techniques that effectively leverage all available data by learning across all time series in each panel. Among the desirable properties forecasting techniques, being able to generate probabilistic predictions ranks among the top. In this paper, we therefore present Level Set Forecaster (LSF), a simple yet effective general approach to transform a point estimator into probabilistic one. By recognizing the connection of our algorithm to random forests (RFs) and quantile regression forests (QRFs), we are able to prove consistency guarantees of our approach under mild assumptions the underlying point estimator. As a byproduct, we prove the first consistency results for QRFs under the CART-splittnn criterion. Empirical experiments show that our approach, equipped with tree-based models as the point estimator, rivals cate-of-theear deep learning models in terms of forecasting accuracy.
Introduction
Tree-based methods are known to be robust, general-purpose and high-accuracy methods for general machine learning and data science tasks, particularly those that not image, video text based. This is exemplified by their popularity in competitions. In a recent interview, Anthony Goldbloom, the CEO of Kaggle, called the prominence of gradient-boosted trees Kaggle the "most glaring difference" between what is used on Kaggle and what is "fashionable in academia," a fact also reflected in the Kaggle Data Science and Machine Learning surveys [] In time series forecasting in particular, tree-based methods have performed solidly in public competitions over the years [IO]. but have had a recent boost in attention via the M5 accuracy competition [23], arguably the most influential forecasting competition, which was dominated by tree-based methods. However, this recent surge in interest is not accompanied by methodological advances. In particular, while being able to generate probabilistic predictions ranks among the top desirable properties of forecasting techniques, the main tree-based method that outputs predictions for multiple quantiles remains Quantile Regression Forests (QRFs), which does not take advantage of gradient boosted trees algorithms.
In this manuscript we provide such an advancement and show its theoretical and empirical soundness: we introduce novel algorithm, Level Set Forecaster (LSF), which can turn any point estimator algorithm into probabilistic one. Applied to XGBoost [I]] (in which case we refer to LSF as XLSF), and with a simple processing component that turns time series data into tabular data, this yields one of the first notable methodological advancements towards creating tree-based probabilistic forecasts
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

since the introduction of Quantile Regression Forests [27]. That said, the LSF algorithm itself is much more general: it applies to any tabular data, and can take any point estimator algorithm.
Our contributions can be summarized as follows:
1. We propose a novel algorithm, Level Set Forecaster (LSF),, which can turn any point estimator algorithm into a probabilistic one. At a high level, it groups training data whose predictions are sufficiently close, and then uses the resulting bins of true values in the training set the predicted distributions.
2. We prove the consistency of LSF (Theorem and as byproduct, prove the consistency of QRFs (Theorem 3p. By building on the methods in [32], we introduce a general framework for consistency results that goes beyond random forests, and apply it to LSF and QRF as special cases.
3. We compare LSF with the state-of-the-an models in both tabular (see Section 6.1 and forecasting tasks (See Section 6.2). and the empirical results verify the effectiveness of the proposed approach.
The rest of the article is structured as follows. In Section 2 we introduce the Level Set Forecaster algorithm, and present the consistency of the resulting estimator in Section 3] We continue by describing the method of proof of the result in Section 4 as well as presenting the novel result (Theorem for QRFs. We discuss related work in Section followed by the empirical studies for tabular and time series data in Section 6.
2 Level Set Forecaster
Given a dataset D : {x y:) C. Rd x R, train given point prediction algorithm A on 2> to arrive at model f E + I.. Our goal is to output probabilistic predictor based on f. such that one could query an arbitrary quantile at any testing point æ E Rd. In more precise terms, assuming (z,, y:)'s sampled IID from the joint distribution (X,Y), we wish to find an estimator for P(Y X)(X).
In Algorithm[] [ we propose a conceptually simple yet effective approach to turn f into a probabilistic forecaster.
high level, Algorithm creates groupings of training data points such that their predictions are "sufficiently close." These groupings are used to partition the feature space. With new testing data point z, we find the partition cell whose predictions are "close to" f(x), and the empirical samples that belong to the same partition cell yield the predictive distribution.
Forming the partition of the feature space. Algorithm [ Thas has a natural geometric interpretation. First, one trains the algorithm A on the data D to get model f Hd + E. The level sets (f=)( are disjoint, but they do not in general satisfy that their union covers Rd because may be that (f(x1,, f(xx)) is not equal the entire image of f. We therefore create Voronoi sets for (f(z1), f(r,)) }. and look at inverse images of these. Namely, writing
as {U1,..., } without repetition and with i1 <... < Uk the jth Voronoi set is:
Vn = {a E Bley = argmin ( mn) (1u; - al)}-
(1)
We now define f- (VG), with each B, being a disjoint union of level sets of f. The inverse Voronoi sets B, 's are disjoint, and their union covers all of Rd. This is not yet, however,
final partition: with the basic set-up described above, the cardinality of each bin Ky E (yı,.., Vo } |x, e B, may be too small (often equal to 1) to ensure the desired property of consistency. In the final part of the algorithm we merge the different B, together, by going sequentially over B,..., Bg (associated with the ascending values vI.... 22). grouping them until sufficient bin size ensured (repeated y, values are not ignored in this computation), and then proceeding to new grouping, arriving at the final partition Rd = Bj. where each Bj is union of B.'s. The intuition behind the partitioning algorithm that for test sample (X, Y) and a value y E R we expect P(Y S x) to not vary wildly as X runs over particular partition cell By.

---

Algorithm 1: Level Set Partitioning Algorithm
Input: D == { (2 Mi=)....... C RR x IR, a natural number min_ bin_ size,, point prediction algorithm A (e.g., XGBoost).
Train A on D, and call the resulting model : IRd + R.
Create dictionary pred_to to_bin whose keys are {z(x))) such that the value of f(xz) is the list of y, satisfying f(z = =f f(z))
Sort the distinct elements of {f(z1), .... (zn)} in ascending order, and denote them [U1,... Uk]- Initialize an empty dictionary res -preep to_bin, and an empty list current_bie
If the size of a bin is smaller than the minimum required size, successively merge with the bins on its right (higher value) until the minimum size constraint is satisfied.
for v in (0.....y U//] do Concatenate pred_to to_ _____t_______ie _bin[u] at the end of current binn res _ess__red_too [t__ii[[]]]] <- current_bing (pass current, bin by reference)
if ln (current_bing 2 sain_iinssiae then
L Point current_bin to new empty list object.
If the last bin is too small, merge to the left. if len(res_pree ptwp_to_ddbpppeeeeeeeeee to_bi < mii_miinnnstteeeeeeeeeemmmemiiem bin_sinee then
concatenate res res_pred_to boo[i res res_pred_to_ tt_ to_bil[ where i is the maximal index satisfying that res_
fess_pred_ttee toad__o____ bin_in * res trrnd_to_bin(ux.
tred_to_binee res_pred_ to_d_to_biin
Output: A partition such that w1, E Rd are said to be in the same partition if pred_to_bil maps argmin (u) (10; f(w1) II) and argmin ( (o (1v; f(w2)l) the same bin.
Using the partition to make inferences. In the notation of Algorithm the algorithm finds the closest element Vy in [e1.... Uk] to f(x) (by binary search), and then computes and caches the quantile of pred_to_ tinbi]
In more mathematical terms, Algorithm outputs partition IR = -, of the feature space into finitely many disjoint sets. Queried at feature vector E B₁ C Rd. and for any value E R, our probabilistic forecasting algorithm returns an estimator for P(Y S x) at =
ÍLSS (xr) H E L17 S w
[n,]
where L1 : Ly/ E "u.....,,aHel B₁} is the associated true target values for the partition cell B We shall study the consistency property of the estimator ÎLSF the next section.
Remark 1. Algorithm | resembles the classic Quantile Regression Forecasts (QRFs) (1277) in the following sense. In QRFs, the random forests are trained in the regular sense, i.e., one splits the nodes according to the CART-spliting criterion. The probabilistic predictions are generated by "opening" the leaf nodes of each tree, and empirically sampling the predictions instead of predicting with the average values in the leaves. Here, the second step is exactly the same while the "leaf nodes" are replaced by partitions induced by the level sets of an arbitrary point estimator. In fact, if one used decision tree for and let min_bin__t wwn_biinnn_iinee size be then Algorithm Areducs to a QRF with one tree.
We refer to the estimator described via Equation 2 with the partitioning algorithm in Algorithm ] as the Level Set Forecaster (LSF) associated to If is XGBoost, we refer to it as XLSF.
Main Theorem: LSF is Consistent
In this section, we aim provide quantitative understanding of the Level-set Forecaster (LSF), in particular, the consistency of LSF The key insight that motivated the LSF algorithm was the observation that the literature regarding the consistency of Random Forests (RFs) can be extended to apply in much more general contexts.
LSF is implemented in GluonTS: Luontss/modee//zotbauuu// http://ooobaamRRRADDDDDE. LSF .pyyb
3

---

We remark that it is precisely because the partitions of trees grown by the CART-spliting algorithm are data data-dependent that consisteney of RFs has only been proven recently [22 while most other work (77) [36]]] 6 (77 BII has focused on easier algorithms that are not used in practice. We can leverage these methods to prove consistency of L.FF as well (Theorem [] and as a byproduct we also prove consistency of QRFs under the CART-splittig algorithm (Theorem 3
In order have a consistency assurance, require mild assumptions on the data generating process. and some intuitive assumptions on the point forecasting algorithm A.
Assumption 1.
1. Assumption on the data generating process:
(a) There is a uniformly equicontinuous family of functions Py (2( (as varies) so that Py(z) integrates P(Y S x) ) for all y. Further assume that EY [X) is bounded. (b) For every > there is a > so that for every x in the image of X the probability that X lies in the ball of radius E centered at x is at least 6.
(c) There exists a function m(x) that integrates to E(Y so that 0 Ouch that for all x E Rd and 0 S y S 1: [m(x) m(x') < 5 implies (py(x) Py (x')l
2. Assumptions the point estimator A: Let Wn. (f(X)), f (Xx)). with h: [W.]- Assume that A has a choice (for every n) of hyperparameters that would satisfy:
(a) Collisions among the training data f(X,) f(X;) rare in the precise sense that there exists positive number C such that:
P p(<< +1.
(b) For an independent variable following the distribution of the X;'s the value f(X) does not tend to be extremal among the J(X.)'s. 's To be precise, for any sequence
dn + 0 of positive numbers we have that
{VE (,ll((x) < <u>>
(HUEW,ll W < f(X)H > dn + I.
dm + 1,
k
k
(c) The image of f is "dense in probability". To be precise,
(In(n))2
max 2
Vol(V,) +
in probability, where the V.' the Voronoi sets from Equation
(d) There's 's a constant e so that Vj, Vx E (VV(V 0 the set (-1 (x(x) a.s.
contains a ball of radius E.
(e) The base algorithm is mean square consistent estimator of the conditional mean:
[((((X) B(Y])) 0.
Theorem 1. Under Assumption [ letting min_bin_size- (In(n))², LSF is mean square consistent. That is, for any value E P. we have
B(Mnss((X) P(Y S x((i) + 0,
where the convergence is uniform in y.
As byproduct of proving this result, we also give the first consistency result (Theorem 3 for QRFs grown under the CART-splitting algorithm under an additive regression model assumption.
Remark 2. Assumptions Ta and Tb much weaker than the additive regression assumptions in
[32]. Assumption ic what allows the base algorithm to be informative about the conditional pdfs. In light of the empirical success of LSF (see Section 6). it appears that Assumption Ic holds quite
generally in real-world data. We expect that Assumptions 2a 2b 2c and 2d which simply ensure non-des non-degenerate behavior, hold for any reasonable choice of A. but the precise statement left as future work. (If if is locally constant on hyperrectangles then Assumption 2d holds with Va+i

---

Method of Proof: Extend Results on RF Consistency to More General Settings
Our strategy for proving Theorem[] generalize and strengthen existing results on the consisteney of RFs, in particular through the work [32]. We propose unified framework that subsumes both the standard RFs (and QRFs) and the proposed LSF estimator, allowing us to arrive at the consisteney
results. We begin by introducing additional notation and nomenclature.
4.1 Problem Setup and Existing Results
Define a data-based partitioning algorithu B as being any algorithm taking a training dataset, ie.. finite subset D = ((X,,Y)),.. (Xn, Yn)} of R x E and outputting a (potentially random) partition
27 of d into disjoint sets ( By satisfying that there's constant > 0 so that Vjvz E B, VE the set B, n (2) a.s. contains ball of radius CE. (This holds if the B,'s 's are hyperrectangles using e v Examples include Algorithm| 1 well as any tree-growing algorithm such as the CART-splitting algorithm (see Algorithm 2 recalled below).
We define mean-regression (resp. quantile-regression) B-estimator, trained on D as estimator of XX) where is the identity map E(Y) == Y (resp. {(Y) == ly given by first applying
to get partition R2d = U By; and then at inference taking a feature vector a E Bj to: Σxel, E(r)
[4]]
where L, (Y; E {Y1,..., Y, Yx, Bi). Now ready introduce the central object of our study, generalized RF/QRF,
Definition 1. Generalized RF (resp. Generalized QRF) grown using the partitioning algorithm B with M estimators is defined as:
?(x) = Σ/= (z)
(4)
M
where fly is mean-regression (resp. quantile-regressien B-estimator trained only part of the data: for the jth estimator subsample uniformly without replacement on many datapoints from D to train on, where on is a (non-ramd (oon-random) sequence of natural numbers satisfying S an S n.
We remark that LSF is simply generalized QRF grown using Algorithm | with one estimator (M 1) and full subsampling (a,, = n); and that RFs (resp. QRFs) are simply generalized RFs whose partitioning algorithm is induced by tree-growing algorithm such the CART splitting algorithm (see Algorithm[2.
To fix some notation, we let t, (a random variable that can depend on the data) be the number of partitions in the first B-estimator, and we let R2,,, be the partition the first B-estimator that contains feature vector x E I
The main question we consider is:
Problem 1. Under what assumptions the generalized RF/QRF estimator f(-)) constructed in (4) L (mean square) consistent? i.e.,
B(ń(() ((((Y)) +
where & is defined either as E(T) = T or E(T) = TTSy-
Our investigation starts with the recent work the consistency of random forests. In Theorem 1 of [32] the authors proved mean square consistency in the mean regression case (E(T) = T') under some mild assumptions, where the partitioning algorithm is the induced by grown via the CART splitting criterion.
We briefly recall the tree-growing algorithm for the CART splitting criterion:
5

---

Algorithm 2: Mean and Quantile Regression Forests with the CART-Splitting Criterion; CART loss is recalled in Appendix A Note that unlike Algorithm [ the number of partition cells bn here is a ayper-parameter and therefore not random.
Input: We fix natural number Mary S d; and sequences of natural numbers (In {anin such
that in S On n..; while not reaching in many leaves
For the subsampling of each tree, choose an points uniformly without replacement from the training set.
At each step, and for each leaf, choose the split that minimizes CART loss, considering only
mary split dimensions sampled uniformly without replacement from {1. (l..... d}.
Mean prediction: output the average value of the leaf nodes. (E(T) = T);
Quantile prediction:
output the average value of the leaf nodes. (5(T) = Irry)
In 322, the authors make the following assumptions:
Assumption (Assumptions in 32
1. Subsamples and tree growth
as in the CART-splitting algorithm.
2. Additive Regression Model Assumption: Y is normal with fixed variance o² and with mean 2 my mi(pe (X:)), where p; is the projection onto the tth coordinate, where each m, : 0 + IR is continuous function, and where the marginal distribution of X is uniform in the hypercube [0, 1]%
-d
tn In(a,))
3. Assume that lim
O, limm bn 0C.
On
The result primary interest in
32 the following.
Theorem 2. (132), Theorem 1) Under Assumptions
we have mean square consistency
E
( -
+ 0,
where E(T) ; T is the identity transformation.
Remark 3. The authors [32 phrase their result as the consistency of (()((X))X (X,, Y.)., i... the case of an infinite number of trees, but they in fact reduce first to the stronger statement that single tree is consistent.
In (27], the quantile regression case (E(T) TTEY) was addressed, but only for the case of label- independent weights, which precludes the case of the partitioning algorithm being induced by trees grown via the CART-spliting algorithm.
4.2 Consistency of Generalized RFs/QRFs
To begin with, we state novel result, the first consistency proof for QRFs under the CART-splittnn criterion.
Theorem 3. Under Assumption with the assumption 2 replaced by the weaker assumption that Ln In(ax) O, In = oo, we have that:
lim
an
E0 (X) < 5((x)13
uniformly in y, where E(T) ; ITSE
We remark that convergence in probability (the notion used in [27] and I,2 convergence coincide for QRFs because the estimator and the CDF are both bounded.
Both Theorems 3 and Theorem | are corollaries from the following generalization Theorem
In order to be rigorous one really ought to use random closed sets. We will ignore this subtlety for convenience, and assume that the clever reader deduce the proper adjustments.

---

Theorem 4. A generalized QRF with single estimator (a "generalized quantile regression tree") mean square consistent under the data generating assumptions la TB in Assumption 1 and the following conditions:
The expected variance of the conditional CDFs in the cell containing X goes to uniformly in the quantile. To be more precise, for (X', Y") IID copy of (X,Y) restricted to the sub probability space where X' E Rx.m assume that (PV S y(X))(x)) goes to uniformly in y.
tn In(an) 0 in probability.
2.
On
Namely, under these conditions, we have that:
E [[nn(X) P(Y // x3 0,
where the convergence is uniform in y, and where E(T) == ITS
Theorem follows from Theorem 4 as an immediate corollary:
Proof. (Theorem By Assumption 2a and our choice min siin_size=_sss (In(n))², follows that in probability t, has the same order of magnitude as (Ia(n))z It is therefore clear that Assumption in Theorem Iss satisfied.
Note that Rx,, is the union of at most (In(n))? many f (VV)). By Assumption 2b with probability converging to 1 none of these Vj's are V1 nor Vi,, Now Assumption 2e immedrately implies that (ol(f(Rx,,) goes to in probability. Letting (X', Y") be IID copy of (X, Y) restricted the sub probability space where X' E Rx,., this implies that ((VV((X*) - O. By Assumption
2el we have that V(J(X)= - E(YIX) - O, and thus (by the law of total variance) ((V(J(X')
ETY" X)) + 0. Therefore R( ((V(B(Y"
0. By an application of Chebyshev's
Inequality and the boundedness of (Y X)., Assumption lc implies that (X 0 uniformly in y.
s xl)x 7
Note that Assumptions 2a
2b and
2c could have been replaced simply by ((V(((X') going to 4 implies Theorem 3 The
in the notation of the proof above. It is less obvious to see why Theorem assertion boils down to proving that the assumptions of Theorem 3 imply Assumption T of Theorem
4
We refer to Appendix for the proof of Theorems and
Related Work
The classical conformal prediction algorithm (33), whose underlying principle is bootstrapping the residuals on a validation set, is another approach for turning point estimator algorithms into probabilistic ones. The natural setting in this approach is to create prediction intervals, rather than to make predictions for particular quantiles. Unlike LSF, the prediction intervals from conformal predictions always symmetric about the point prediction with fixed length. More sophisticated conformal prediction algorithms exist that allow for variable length prediction intervals (e.g. (29), which takes as input two quantile regression estimators), but they no longer satisfy that they take single point estimator algorithm to probabilistic one. We refer to Appendix C for more details.
The main available probabilistic tree-based algorithm other than LSF (applied to tree-based point estimator algorithm) that does not require a list of quantiles during training is QRFs [27]. We remark that LSF applied to decision tree is equivalent to QRF with single tree, and that therefore QRFs with multiple trees are an ensemble of these simple LSF algorithms.
Another approach for creating tree-based probabilistic predictions is to learn the parameters of family of conditional distributions (e.g., (9). For tree-based methods, this has been proposed [25 26 We do not further compare against this method in our experiments because the implementations are not available and instead focus on comparisons with ttaee-of-the-art methods for forecasting that use similar approach (c.g., DeepAR [30].
7

---

For probabilistic time series forecasting, a number of global [18] models have been proposed
(e.g., 228
12
22
44
37 For comprehensive review for neural network models for forecasting.
please refer to 16 5]. In contrast to classical forecasting models (e.g., [77) which are local
(parameters are estimated per time series independently), tree-based methods are best employed as
global models (learning parameters over the entire panel of time series, see e.g., [19] for discussion
and [35] for a concurrent contribution). In our empirical comparisons, we chose DeepAR [30] and CNN-QR (88 as global, deep-learning based models for their robust state-of-the performance.
6 Experiments
In Section
compare XLSF (LSF with XGBoost) against QRFs (as leading tree- rree-based probabilistic algorithm) and Conformalized Predictions (as a baseline for turning point estimators into probabilistic ones). In Section 6.2 we apply LSF time series data. All experiments done using Amazon Sagemaker (21) with instance type m11m4.. 16xlarge. All hyperparameters used are specified in Appendix
6.1
6.1 Experiments on Tabular Data
A main competitor for LSF for using tree-based methods to create probabilistic predictions is QRFs (277). In the experiments we use the skgarden implementation ((2)). QRFs are made up of multiple trees, but XLSF is one generalized tree made up of an XGBoost ensemble. It is therefore interesting to see how the two compare. By design, conformal predictions are phrased in terms of prediction intervals (a pair of quantiles) rather than range of quantiles. For example for an 0.1, conformal predictions aim to estimate PO5 and P95 so as to have an accuracy of the prediction intervals (percent of times that the true value is in the prediction interval) that is at least 90%. We therefore chose only the pair of quantiles PO5 and P95 in our experiments, as opposed to 3 quantiles as in Section 6.2
The results are shown in Table I and we see that XLSFF outperforms conformal predictions by wide margin on quantile prediction, though conformal predictions' prediction interval accuracy was more faithfully close to 90%. We remark that to reap the benefits of both algorithms, one can feed XLSF into the "Conformalized Quantile Prediction" " algorithm; see [29), and the discussion in Appendix C We also remark that QRF is much slower compared to XLSF, shown in Table T] The datasets in this section were small, but in Section 6.2 we will see that this makes classical QRF impractical for time series predictions, while XLSF is quite competitive.
XLSF
QRF
Conformalized Predictions
PO5
P95
accuracy time (s)
PO5
P95
accuracy time (s)
PO5
P95
accuracy time (s)
facebook;
0.103
0.288 94.31%
7.668
0.094 0.317 92.74%
29.918
0.471 0.428 89.98%
6.942
facebookg
0.097 0.293 95.32%
14.790
0.094 0.299 92.93%
103.61
0.391 0.445 89.89%
14.442
meps19
0.100 0.562 93.37%
5.874
0.111 0.687 89.45%
9.159
0.633 0.766 89.64%
5.305
meps2o
0.100 0.664 92.78%
6.384
0.107 0.674 88.37%
10.526
0.498 0.774 90.13%
5.884
meps21
0.100 0.555 92.68%
6.215
0.105 0.635 88.40%
9.111
0.525 0.802 89.94%
5.246
concrete
0.036 0.039 76.69%
0.627
0.036 0.037 82.52%
0.167
0.031 0.042 86.89%
0.089
star
0.011 0.012 78.52%
1.004
0.011 0.014 79.21%
0.500
0.011 0.011 87.99%
0.305
bio
0.082 0.132 87.63%
13.195
0.073 0.096 84.40%
33.153
0.103 0.128 89.99%
4.528
community
0.105 0.189 76.19%
1.120
0.079 0.187 86.71%
1.132
0.136 0.184 92.48%
0.955
bike
0.056 0.059 87.74%
3.403
0.043 0.044 80.53%
2.828
0.050 0.048 91.23%
0.734
Table 1: Benchmarking results: the datasets were taken from https: petts://githhue wwtp:occo.ooo/aaemme.. http://aattrdtaaeeeeeeeeeee Accuracy is percent of times the true value was in the prediction interval. (It should revolve around 90%.) PO5 and P95 are weighted quantile losses.
6.2 LSF for Time Series Prediction
In order to apply LSF to time series data, a few choices need to be made. First, the method for preprocessing the data into tabular data; second, whether to use single model for the entire forecast
8

---

quantile
wrapping of winning M5 solution
Winner
Runner-up
Third
Fourth
Fifth
0.005
0.01227
0.010
0.04201
0.01891
0.01225
0.01133
0.025
0.05484
0.05004
0.08649
0.06446
0.05483
0.05739
0.165
0.31677
0.31276
0.33782
0.33371
0.31566
0.35329
0.25
0,45108
0.44436
0.46158
0.46699
0.44937
0.49415
0.5
0.72434
0.69886
0.69044
0.74712
0.71279
0.7661
0.75
0.72848
0.68747
0.72231
0.69522
0.71071
0.70321
0.835
0.60748
0.58519
0.59897
0.59059
0.61292
0,59058
0.975
0.21801
0.18236
0.19664
0.19268
0.19348
0,20365
0.995
0.0863
0.0551
0.07689
0.07433
0.06225
0.06811
mean_wWl __QL
0.35551
0.33624
0.35702
0.35378
0.34714
0.36087
Table 2: The wSF--iapping of the winning M5 accuracy competition's solution on the top point forecast submission compared with top results in the uncertainty competition. Note that the non- monotonous nature of the metrics for the winning solutions due to the fact that all the numbers in this table are being evaluated only the bottom level of the hierarchy, and that the metric used is different from the competitions' metric.
horizon, or one model for each timestep; and finally, one has to decide on what point estimator algorithm A use and its hyperparameters. The winning solution the M5 accuracy competition has already made all of these decisions, and so we can simply wrap it with LSF, and see how it compares with the top solutions for the M5 uncertainty competition. We also propose a simple default choice (using only lag features, one model for each timestep, and using XGBoost) and apply multiple datasets.
M5 competition The M-competitions are a series of competitions led by Spyros Makridakis intended to evaluate and compare the accuracy of different forecasting methods. The most recent competition, the M5 competition (point forecasting) and an uncertainty competition (probabilistic forecasting) with the same training data. The winning solution (by YeonJun-IN. available in [T) to the M5 accuracy competition is tree-based solution that trains 220 lightgbm ((20) models (with different feature engineering set ups and/or trained on different parts of the data) and combines them to create a point forecast. The preprocessing involved more than mere lag features.
(23), was in fact twin competitions: an accuracy competition
We remark that the M5 uncertainty competition requires forecasts not only of the individual time series, but also of hierarchical aggregates. In reported metrics we only consider the bottom level of the hierarchy, since there is no trivial way to produce aggregate forecasts using the LSF-wrapping of the winning M5 accuracy competition's 's solution.
We can see that wrapping the M5 accuracy competition's 's winning solution is competitive among the top 5 solutions for the M5 uncertainty competition, showing the out-of-the-box were able to turn quality point-forecass into quality probabilistic forecasts.
Benchmarking datasets. We now turn to some benchmarking with certain default choices of preprocessing and inference on common time series datasets. To be precise, we preprocess the time series data into tabular data by sampling P context windows with only lag features, and then training a separate probabilistic forecaster for each time-step in the future. (See Algorithm 3 in the appendix for more details; see also Appendix[F]
We refer to this set-up as Tree- based Time Series Wrapper (TTSW). We remark that by design, TTSW with XLSF never predicts above (resp. below) the maximal (resp. minimal) value observed. We also include TTSW with QRFs instead of XLSF, and with lightgbm ([20)) with quantile loss, which we refer to as Quantile Regression. If one chooses XLSF QRF there is no need to specify quantiles training time, but for Quantile Regression, one specifies quantiles before training, and for each timestep in the forecast horizon it trains many models as the number of quantiles requested.

---

TTSW (XLSF)
TTSW (QR)
TTSW (QRF)
DeepAR
CNN-QR
electricity [13
0.0499
0.1478
0.0785
0.0592
0.0601
parts [4
0.9709
0.7584
0.9617
0.9877
0.7566
m4_ m4_daily 224
0.0148
0.0281
0.0188
0.0239
0.0154
traffic |13
0.1212
0.1062
0.1289
0.0913
0.1134
wikil0k
0.2926
0.2534
0.3069
0.2421
0.2482
dcrideshare
8
0.3381
0.3171
0.3929
0.2837
0.2820
Table 3: Mean Weighted Quantile Loss (across P10, P50, P90).
By default P 1000000 and the context window equal the forecast horizon. In Table since QRF is much slower to train, we had to reduce P to 10000, which means it had 100 times less data- points to train on. Even with this adjustment, it took considerably longer than the other methods. Our results in 3 show that TTSW (both XLSSF and with Quantile Regression) perform competitively with state-of-the-an deep learning forecasting methods (G0 38), while maintaining advantages of tree-based methods such as interpretability. For detailed results, please refer to Appendix F
10

---

References
[1] M5 solutions.
[2] scikit-garden.
https///gggtttt.ooo////aaaaaaaaa 2019.
[3] Kaggle survey http::/swww...ggg....../////aaaa
[4] Kaggle survey 2020.
survey- 2020, 2020.
[5] Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Bernie Wang, Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael Schnke-S David Salinas, Lorenzo Stella, et al. Neural forecasting: Introduction and literature overview. arXiv preprint arXiv:2004. 10240, 2020.
[6] Gérard Biau. Analysis of random forests model. Journal of Machine Learning Research, 13 106r):1063-1095, 2012.
[7] Gérard Biau, Lue Devroye, and Gábor Lugosi. Consistency of random forests and other averaging classifiers. Journal Machine Learning Research, 9(Spp):0015-2033, 2008.
[8] Capital Bikeshare. Capital bikeshare data, 2021. URL com/ hommssysemm-dee data.
[9] Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag. Berlin, Heidelberg, 2006. ISBN 0387310738.
[10] Casper Solheim Bojer and Jens Peder Meldgaard. Kaggle forecasting competitions: overlooked learning opportunity. International Journal of Forecasting, 2020. ISSN 0169-2070.
[11] Tianqi Chen and Carlos Guestrin. XGBoost: scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD 16, pages 785-794. ACM, 2016.
[12] Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke- Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. Advances in Neural Information Processing Systems, 33, 2020.
[13] Dua Dheeru and Efi Karra Taniskidou. UCI machine learning repository, 2017. URL URLhttp!
// /archive.ics. www.uci.eeu.. udd.mldemlmmm
[14] Jan Gasthaus, Konstantinos Benidis, Yuyang Wang. Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic forecasting with spline quantile function mas. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 1901-1910. PMLR, 16-18 Apr 2019. URL. http! #/proceedings.nt mlr. wwtps//w//gaasteeeeeeeeee
html
[15] Lászłó Györfi, Michael Kohler, Adam Krzyzak, and Harro Walk. A distribution-free theory of nonparametric regression. Springer Science & Business Media, 2006.
[16] Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. Recurrent neural networks for time series forecasting: Current status and future directions. International Journal of Forecasting, 37(1):388 2021.
[17] Rob Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
[18] Tim Januschowski, Jan Gasthaus, Yuyang Wang, David Salinas, Valentin Flunkert, Michael and Laurent Callot. Criteria for classifying forecasting methods. Interna- tional Journal of Forecasting, 2019.

---

[19] Tim Januschowski, Yuyang Wang, Kari Torkkola, Timo Erkilla, Hilaf Hasson, and Jan Gasthaus. Forecasting with trees. International Journal of Forecasting, 2021.
[20] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[21] Edo Liberty, Zohar Karnin, Bing Xiang, Laurence Rouesnel, Baris Coskun, Ramesh Nallapati, Julio Delgado, Amir Sadoughi, Yury Astashonok, Piali Das, et al. Elastic machine learning algorithms amazon sagemaker. Proceedingy of the 2020 ACM SIGMOD International Conference Management of Data, pages 731-735 2020.
[22] Bryan Lim, Sercan Arık, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horize time series forecasting. International Journal of Forecasting, 2021.
[23] Spyros Makridakis and Evangelos Spiliotis. The M5 Competition and the Future of Human Expertise in Forecasting. Foresight: The International Journal of Applied Forecasting, (60): 33-37, Winter 2021.
[24] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting, 34(4): 802-808, 2018.
[25] Alexander März. Xgboostlss extension of xgboost probabilistic forecasting, arXiv preprint arXiv:1907.03178, 2019.
[26] Alexander März. Catboostlss an extension of catboost to probabilistic forecasting, 2020.
[27] Nicolai Meinshausen. Quantile regression forests. Journal of Machine Learning Research, (Jun):983-999, 2006.
[28] Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Advances in Neural Information Processing Systems, pages 7785-7794, 2018.
[29] Yaniv Romano, Evan Patterson, and Emmanuel Candès. Conformalized quantile regression. Advances in neural information processing systems, 2019.
[30] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020.
[31] Erwan Scornet. On the asymptotics of random forests. Journal of Multivariate Analysis, 146: 72 83, 2016.
[32] Erwan Scornet, Gérard Biau, and lean-Philipee Vert. Consistency random forests. The Annals of Statistics, 43(4):1716-1744 2015.
[33] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008
[34] Ralph D Snyder, Keith Ord, and Adrian Beaumont. Forecasting the intermittent demand for slow-moving inventories: modelling approach. International Journal of Forecasting, 28(2): 485-496, 2012.
[35] Olivier Sprangers, Sebastian Schelter, and Maarten de Rijke. Probabilistic gradient boosting machines for large-scale probabilistic regression. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, Aug 2021. doi: 10.1 141114//447548.4472277 http:///d doi.. org/ 10. 14453444545 3467278
[36] Cheng Tang, Damien Garreau, and Ulrike von Luxburg. When do random forests fail? In Advances in neural information processing systems, pages 2983-2993, 2018.
12

---

[37] Yuyang Wang, Alex Smola, Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski. Deep factors for forecasting, In International conference on machine learning, pages 6607-6617. PMLR, 2019.
[38] Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi- horizon quantile recurrent forecaster. arXiv preprint 10X 2017.
13


Reviews:
[{"review_id": "LtDfETLKBLT", "reviewer": "Reviewer_Bgd7", "summary": "This paper proposes a level-set approach (LSF) to model the uncertainty in forecasting problems. It can process the result of any point forecaster into probabilistic forecasts. The main idea of LSF is to group the predictions of training data that are sufficiently close, which is also used to partition the feature space. One then obtains different bins of true values in the training set, and this can be used as the predicted distributions. The authors have proved the mean-square consistency of LSF and further extended to random forests in more general settings. Empirical evaluations on both tabular and time series competition datasets demonstrate the effectiveness of this method.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has a clear motivation and is well structured. The LSF partitioning algorithm is theoretically sound in terms of the mean-square consistency. Prior works have shown the consistency of random forests. The authors further prove that LSF is a generalized QRF with tree-growing partitioning algorithms. Overall, LSF can produce probabilistic forecasts at any specified quantile, which is distribution-free, but I am not sure if the binning procedure can be implemented more efficiently.\n\nIn addition, since the authors mentioned (line 33) that LSF can turn any point estimator into a probabilistic one, I wonder if the results (both LSF algorithm and theory) can extend to point predictors other than XGBoost. In that case, is preprocess to tabular data still necessary? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed social impacts but haven't discussed limitations. From the experimental section, the computational efficiency and coverage guarantee compared with conformal prediction approaches, as well as the generalization to sequential data with hierarchical aggregations might be good directions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "AHw_iDBrRjE", "reviewer": "Reviewer_Zoe4", "summary": "The authors propose a method for turning any prediction model into a probabilistic forecaster. Briefly, the approach works by grouping training data with similar outputs, and then considers the binned true values as a prediction distribution. They prove that their algorithm is consistent by extending proofs about the consistency of random Forests to more general contexts. To evaluate their method, the authors analyze publicly available tabular datasets and time series datasets. They also adapt use their method to adapt the M5 forecasting competition’s winner to perform probabilistically, and compare their results to the top entrants in the M5 uncertainty competition.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality- The method proposed to make probabilistic inferences from point wise forecasters is fairly intuitive, but the rigorous proofs and evaluations show that their method performs well and is theoretically sound. I do not know of any other work that turns point-wise forecasters into a probabilistic method with so much rigor. So overall it is a novel method.\n\nSignificance- Pointwise models are standard, but most forecast applications require only knowing where an output lies generally, ie knowing what range an output will be in is often as useful as exact estimates. Furthermore, explainability is a hot issue right now, and knowing a model’s confidence aids in that tremendously. Overall, a method that is able to take any high-quality forecasting method and utilize it to learn more interpretable models is quite useful.\n\nClarity- Many aspects of the paper were clear, for instance many of the high-level overviews were easy to follow and motivated the rest of the paper. The overall method was also easy to follow. Some of the implementation and evaluation details, particularly pertaining to the M5 competition and benchmark datasets, were difficult to follow, as were some of the assumptions.\n\nQuality-Overall, this is a high quality paper, with a useful method, strong proofs and extensive evaluations. What is most limiting is a lack of details and clarity in the evaluation section and in some other areas, which I will describe below.\n\n\n\nIssues-\n\n1) Algorithm 1- by always merging with the bin to the right (except at the end), it is possible to wind up with very lopsided bins. For instance, if your minimum bin size were 10, and your initial bins were: 10,2,3,5,40, you would wind up with two bins of size 10 and 50, where as if you merged to the left, they would be 20 and 40, which is much closer. Is there a reason why this doesn’t pose a big issue? \n\n2) Assumptions- these could be elucidated with some definitions-  for instance, what does volume mean in this setting, and the description of collisions is unclear. Overall, some high-level descriptions of the assumptions, along with a discussion regarding how likely they are to hold, would be quite useful.\n\n3) Overall evaluation- the main evaluation metric, weighted quantile loss, is not described anywhere. Also it is a little confusing that each table has different metrics and methods, although that is a minor issue. It is also not clear if XLSF is better compared to QRF in table 1- they seem to outperform each-other an equal amount across the different datasets and metrics. \n\n4) M5 data analysis- The authors only consider the bottom level of the hierarchy, but in the competition, they checked both. Are the results for the competitors in table 2 what was reported in the M5 competition, or the author’s updated metric? If they did not get the same metric for both datasets, that does not appear to be a fair comparison, otherwise they should clarify that the metric is the same for all methods.\n\n5) Benchmark datasets- where did these datasets come from? I understand that they are commonly available, but there are no references or descriptions that I could find. It also appears that their model is outperformed by deep baselines for most datasets, is this not a limitation?\n\n6) Limitations- the authors do not discuss many limitations. What happens in the case where any assumptions are violated? How likely is it that any of these assumptions would be violated? Building a model for each timepoint doesn’t take advantage of shared information between timepoints like a multi-step model does. Does this method generalize to small datasets, or skewed datasets? Any comments amount where and why it was outperformed?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss limitations in detail, as I described in the previous section. It is difficult to imagine a specific negative societal impact, beyond whatever negative societal impact any forecasting method has. Impacts would be more associated with potential applications.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {"ethical_concerns": "The authors do not discuss limitations in detail, as I described in the previous section. It is difficult to imagine a specific negative societal impact, beyond whatever negative societal impact any forecasting method has. Impacts would be more associated with potential applications.\n"}}, {"review_id": "-lSpVYjx4VK", "reviewer": "Reviewer_eTYD", "summary": "The authors propose a general method to derive probabilistic / conditional distribution predictions for any given trained base prediction model f(x) - i.e., CDF of y|x, by using an approach somewhat related to quantile regression forests (QRFs).  In this approach prediction function f(x_i) outputs are grouped with nearby values, so that groups have a minimum size from the training data.  Each group's true values {y_i} define a sample for that group that can be used to compute any CDF value - and, for instance provide a prediction interval via a set of quantiles.  A prediction for a new sample x is transformed into the probabilistic prediction by finding the nearest group to f(x), and using its samples to derive the distribution statistics.\n\nThe authors prove consistency under fairly reasonable assumptions of the prediction algorithm, as well as proving consistency for QRFs with CART splitting.\n\nThe authors also perform extensive experiments showing the proposed approach is effective at estimating distribution statistics / prediction intervals for a variety of datasets, compared to other approaches to deriving distributions from point predictions (conformal approach) as well as other probabilistic approaches (QRF, quantile regression approaches, DeepAR), showing it is both efficient and competitive in terms of accuracy, even out-performing other methods.  These include experiments with forecasting datasets showing how their approach applied to the point forecasting winner of M5 made it competitive for probabilistic forecasting with the top 5 probabilistic forecasting methods from the competition.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found this to be a very interesting paper that is also very useful.\n\n**Originality:** This seems like a pretty novel approach.  Although it has some similarity to QRF, it is a generic procedure applicable to any model, and much different to decide the groups representing the conditional distributions.  Unlike QRF that is based on a specific learning and prediction algorithm only, and much slower.   Also it includes novel proofs and experiments.\n\n**Significance:** This work seems very significant and useful.  The majority of models are point prediction models (and these are often easier and more efficient to create and train), but for many reasons its very desirable to have prediction intervals and distributional predictions - e.g., for making decisions based on predictions.  So having some procedure like this that is effective and efficient at transforming point predictions into probabilistic ones is extremely useful, across many domains.  This is a step in that direction, which could be useful in practice and spawn new research in this direction as well.\n\n**Clarity:**\nOverall the paper is well written and clear - there are a few points that are hard to follow, when describing the approach / algorithm - it takes some careful reading through to fully understand.  It might be helpful here to have a simple example to illustrate the approach - as this would immediately make it clear, or some visual illustration of the approach.\n\n**Quality:**\nThe approach and details seem technically sound (though I did not check the proofs carefully).  That being said, I think the work could definitely use more discussion around its limitations - including the practical implications of the assumptions and the approach, and under which cases it may not work well.  I can think of several cases where it wouldn't work, and also found this approach raised several questions about its viability (see below).  That being said, the thorough set of experiments does show that despite these assumptions and cases it may fail it is generally effective with a lot of real data and at least one common class of learning algorithms (gradient boosting) - but it is important for readers to have some idea of when it might fail.  \n\nIt would also be ideal to show results when using it with other algorithms as well besides just gradient boosting (such as deep learning models, linear models, SVM, random forest, etc.).\n\nAlso, On the claims: \nOne claim is for the proof of consistency of QRFs,  however consistency was already proven in [21] for QRFs – so what is the specific differentiating claim here?  Later text seems to make clear it is for a different splitting criteria than previously proven.  This should be made clear  up-front that the differentiating contribution is here and that consistency was already proven in the prior work.\n\n\nHere are some detailed questions and potential limitations that could use more discussion, and other issues:\n\n\n**1)\tPractical implications of assumptions:**\nMore should be discussed about the practical implications of the assumptions and when this works or when this doesn’t, based on the algorithm A and the data properties.\n\nFor example, how can this not depend on consistency of the algorithm A, or to converge to it predicting some particular statistic?  Clearly this is interestingly tied to the assumptions, such as Assumption 2a.\nE.g., in an extreme case, if A is an algorithm that returns a function always predicting the same value, then nothing can be gained with this approach.  In a more reasonable case, what if A always predicts a median of a sparse count distribution (like Poisson with mean << 1).  Again it will always predict 0, despite being completely accurate in estimating the median, and again the correct quantiles of the underlying distribution cannot be obtained with this method.  There could be cases in between these extreme cases, e.g., estimating the median but still having a good proportion of values > 0.  \n\nAnother related question along these lines, if the point prediction is for some other statistic, say itself target at the 0.1 quantile, then does this method still return the conditional CDF of y given x?  I.e., if f(x) is trained to predict quantile 0.1, e.g., with quantile loss. \n\nI also wonder what the implications are for the consistency of algorithm A itself, as this seems to suggest that algorithm A could basically be anything and need not be consistent.  Further, if the underlying data is stochastic, and algorithm A can at best predict a statistic – does this still hold?  Some experiments that would be good to show include using different algorithms as the base algorithm including ones that underfit (like linear models), and also even arbitrary functions.\n\nIn general, some interpretation and discussion of the implications of the assumptions, and analyses and experimentation around when the method works well or not, would be very helpful to include.\n\n\n**2) Heteroskedasticity:**\nAnother major concern is this approach seems to impose strong assumptions, that if the prediction function output values are similar, the underlying distribution is also similar, but in general this may not be the case and could lead to incorrect results, as it doesn’t depend on any grouping of the data (x_i) itself, only f(x_i).  In fact it’s fairly common for time series from real world applications to exhibit heteroskedasticity.  For example, going through time periods of cyclical or event-driven increased volatility/uncertainty.  In other words, for one set of inputs and/or at one point in time the mean (and correspondingly the prediction, f(x), of a good prediction model) could be exactly the same as for another set of inputs and/or another point in time, but the variance much different, and the underlying distribution much different (and thus the prediction interval much wider for one than the other).  But because the predicted mean value is the same, they will be grouped into the same group, and the prediction intervals estimated for both time period will be the same, and incorrect.\n\nE.g., consider commodity prices, right before government or other major reports are released, the mean prediction for the change in price might be the same as recent mean predictions, but the variance in that prediction should be much wider, as it depends largely on the yet unforeseen news.  In practice, the true value for the next time period will very much more than during other stable times.\n\n\n**3) Overfitting:**\nI wonder if the authors could comment on the case of models that typically overfit like neural nets.  These days in many cases neural nets will achieve 0 training error before the test error reaches its minimum.  This method would not seem to work at all for such models – which makes me question its utility when applied to modern DL algorithms.\n\n-->Again it would be good to see results with more algorithms besides just gradient boosting…\n\n**4)\tDefinitions of the metrics used are missing**\nIn particular quantile loss should be defined, at least in the appendix – and also explained what weights are used as Table 1 says “weighted quantile loss” but weights are never explained\n\n\n\n**Minor comments:**\n\nAs mentioned, another option for forecasts is to output a vector of results – i.e., f(x) is a vector  - how would you apply the approach in this case?\n\n\nLine 72-72 why is k not equal to n?  Should add an explanation here, that {v_j} is the set of unique values of {f(x_i)} – I’m assuming this is what is meant but it should be spelled out.  This becomes clear later but would make it easier to follow if this is explained when first introduced.\n\nIn Algorithm 1 – is the sufficient bin size based on all values of y_i (even if they are the same value)?  It seems to be so but it might help to note this in the text.  \n\n\n\n**--Update after response and discussion--**\n\nI appreciate the authors' responses and addressing the many reviewer comments - I think it will make it an even stronger paper.  Kept my rating, and looking forward to reading the final version!\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I feel more could be discussed around the limitations of the algorithm, and practical implications of the assumptions.  In particular, in which kind of cases, for what kind of datas or learning algorithms, would the method be unlikely to work?  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a novel approach for transforming point estimators into probabilistic estimators. Overall, reviewers appreciate the novelty and significance of the work. However, they also made several important recommendations regarding the presentation and specifically the discussion of limitations. The authors should carefully consider reviewer feedback when working on their revision.
Decision: Accept (Poster)