Probabilistic Forecasting: A Level-Set Approach

1 Introduction . Tree-based methods are known to be robust , general-purpose and high-accuracy methods for general machine learning and data science tasks , particularly those that are not image , video or text based . This is exemplified by their popularity in competitions . In a recent interview , Anthony Goldbloom , the CEO of Kaggle , called the prominence of gradient-boosted trees on Kaggle the “ most glaring difference ” between what is used on Kaggle and what is “ fashionable in academia , ” a fact also reflected in the Kaggle Data Science and Machine Learning surveys [ 3 , 4 ] . In time series forecasting in particular , tree-based methods have performed solidly in public competitions over the years [ 10 ] , but have had a recent boost in attention via the M5 accuracy competition [ 23 ] , arguably the most influential forecasting competition , which was dominated by tree-based methods . However , this recent surge in interest is not accompanied by methodological advances . In particular , while being able to generate probabilistic predictions ranks among the top desirable properties of forecasting techniques , the main tree-based method that outputs predictions for multiple quantiles remains Quantile Regression Forests ( QRFs ) , which does not take advantage of gradient boosted trees algorithms . In this manuscript we provide such an advancement and show its theoretical and empirical soundness : we introduce a novel algorithm , Level Set Forecaster ( LSF ) , which can turn any point estimator algorithm into a probabilistic one . Applied to XGBoost [ 11 ] ( in which case we refer to LSF as XLSF ) , and with a simple processing component that turns time series data into tabular data , this yields one of the first notable methodological advancements towards creating tree-based probabilistic forecasts 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . since the introduction of Quantile Regression Forests [ 27 ] . That said , the LSF algorithm itself is much more general : it applies to any tabular data , and it can take any point estimator algorithm . Our contributions can be summarized as follows : 1 . We propose a novel algorithm , Level Set Forecaster ( LSF ) , which can turn any point estimator algorithm into a probabilistic one . At a high level , it groups training data whose predictions are sufficiently close , and then uses the resulting bins of true values in the training set as the predicted distributions . 2 . We prove the consistency of LSF ( Theorem 1 ) ; and as a byproduct , prove the consistency of QRFs ( Theorem 3 ) . By building on the methods in [ 32 ] , we introduce a general framework for consistency results that goes beyond random forests , and apply it to LSF and QRF as special cases . 3 . We compare LSF with the state-of-the-art models in both tabular ( see Section 6.1 ) and forecasting tasks ( See Section 6.2 ) , and the empirical results verify the effectiveness of the proposed approach . The rest of the article is structured as follows . In Section 2 we introduce the Level Set Forecaster algorithm , and present the consistency of the resulting estimator in Section 3 . We continue by describing the method of proof of the result in Section 4 , as well as presenting the novel result ( Theorem 3 ) for QRFs . We discuss related work in Section 5 , followed by the empirical studies for tabular and time series data in Section 6 . 2 Level Set Forecaster . Given a dataset D : = { ( xi , yi ) } i=1 , ... , n ⇢ Rd ⇥ R , we train a given point prediction algorithm A on D to arrive at a model f : Rd ! R. Our goal is to output a probabilistic predictor based on f , such that one could query an arbitrary quantile at any testing point x 2 Rd . In more precise terms , assuming ( xi , yi ) ’ s are sampled IID from the joint distribution ( X , Y ) , we wish to find an estimator for P ( Y y|X ) . In Algorithm 1 , we propose a conceptually simple yet effective approach to turn f into a probabilistic forecaster . At a high level , Algorithm 1 creates groupings of training data points such that their predictions are “ sufficiently close. ” These groupings are used to partition the feature space . With a new testing data point x , we find the partition cell whose predictions are “ close to ” f ( x ) , and the empirical samples that belong to the same partition cell yield the predictive distribution . Forming the partition of the feature space . Algorithm 1 has a natural geometric interpretation . First , one trains the algorithm A on the data D to get a model f : Rd ! R. The level sets { f 1 ( f ( xi ) ) } are disjoint , but they do not in general satisfy that their union covers Rd , because it may be that { f ( x1 ) , ... , f ( xn ) } is not equal to the entire image of f . We therefore create Voronoi sets for { f ( x1 ) , ... , f ( xn ) } , and look at inverse images of these . Namely , writing { f ( x1 ) , ... , f ( xn ) } as { v1 , ... , vk } without repetition and with v1 < · · · < vk , the jth Voronoi set is : Vj = { a 2 R|vj = argmin { v1 , ... , vk } ( |vi a| ) } . ( 1 ) We now define B̃j = f 1 ( Vj ) , with each B̃j being a disjoint union of level sets of f . The inverse Voronoi sets B̃j ’ s are disjoint , and their union covers all of Rd . This is not yet , however , our final partition : with the basic set-up described above , the cardinality of each bin | { yi 2 { y1 , ... , yn } |xi 2 B̃j } | may be too small ( often equal to 1 ) to ensure the desired property of consistency . In the final part of the algorithm we merge the different B̃j together , by going sequentially over B̃1 , ... , B̃k ( associated with the ascending values v1 , ... , vk ) , grouping them until a sufficient bin size is ensured ( repeated yi values are not ignored in this computation ) , and then proceeding to a new grouping , arriving at the final partition Rd = S · Bj , where each Bj is a union of B̃i ’ s . The intuition behind the partitioning algorithm is that for a test sample ( X , Y ) and a value y 2 R we expect P ( Y y|X ) to not vary wildly as X runs over a particular partition cell Bj . Algorithm 1 : Level Set Partitioning Algorithm Input : D : = { ( xi , yi ) } i=1 , ... , n ⇢ Rd ⇥ R , a natural number min_bin_size , a point prediction algorithm A ( e.g. , XGBoost ) . Train A on D , and call the resulting model f : Rd ! R. Create a dictionary pred_to_bin whose keys are { f ( xi ) } i , such that the value of f ( xi ) is the list of yj satisfying f ( xj ) = f ( xi ) . Sort the distinct elements of { f ( x1 ) , ... , f ( xn ) } in ascending order , and denote them [ v1 , ... , vk ] . Initialize an empty dictionary res_pred_to_bin , and an empty list current_bin . # If the size of a bin is smaller than the minimum required size , successively merge it with the bins on its right ( higher value ) until the minimum size constraint is satisfied . for v in [ v1 , ... , vk ] do Concatenate pred_to_bin [ v ] at the end of current_bin . res_pred_to_bin [ v ] current_bin ( pass current_bin by reference ) if len ( current_bin ) min_bin_size then Point current_bin to a new empty list object . # If the last bin is too small , merge to the left . if len ( res_pred_to_bin [ vk ] ) < min_bin_size then concatenate res_pred_to_bin [ vi ] to res_pred_to_bin [ vk ] , where i is the maximal index satisfying that res_pred_to_bin [ vi ] 6= res_pred_to_bin [ vk ] . pred_to_bin res_pred_to_bin Output : A partition such that w1 , w2 2 Rd are said to be in the same partition if pred_to_bin maps argmin { v1 , ... , vk } ( |vi f ( w1 ) | ) and argmin { v1 , ... , vk } ( |vi f ( w2 ) | ) to the same bin . Using the partition to make inferences . In the notation of Algorithm 1 , the algorithm finds the closest element vj in [ v1 , ... , vk ] to f ( x ) ( by binary search ) , and then computes and caches the quantile of pred_to_bin [ vj ] . In more mathematical terms , Algorithm 1 outputs a partition Rd = S · Bj of the feature space into finitely many disjoint sets . Queried at a feature vector x 2 Bl ⇢ Rd , and for any value y 2 R , our probabilistic forecasting algorithm returns an estimator for P ( Y y|X ) at X = x : ⌘̂LSF ( x ) : = | { 2 Ll| y } | |Ll| , ( 2 ) where Ll : = { yi 2 { y1 , ... , yn } |xi 2 Bl } is the associated true target values for the partition cell Bl . We shall study the consistency property of the estimator ⌘̂LSF in the next section . Remark 1 . Algorithm 1 resembles the classic Quantile Regression Forecasts ( QRFs ) ( [ 27 ] ) in the following sense . In QRFs , the random forests are trained in the regular sense , i.e. , one splits the nodes according to the CART-splitting criterion . The probabilistic predictions are generated by “ opening ” the leaf nodes of each tree , and empirically sampling the predictions instead of predicting with the average values in the leaves . Here , the second step is exactly the same while the “ leaf nodes ” are replaced by partitions induced by the level sets of an arbitrary point estimator . In fact , if one used a decision tree for A and let min_bin_size be 1 then Algorithm 1 reduces to a QRF with one tree . We refer to the estimator described via Equation 2 with the partitioning algorithm as in Algorithm 1 as the Level Set Forecaster ( LSF ) 1 associated to A . If A is XGBoost , we refer to it as XLSF . 3 Main Theorem : LSF is Consistent . In this section , we aim to provide a quantitative understanding of the Level-set Forecaster ( LSF ) , in particular , the consistency of LSF . The key insight that motivated the LSF algorithm was the observation that the literature regarding the consistency of Random Forests ( RFs ) can be extended to apply in much more general contexts . 1LSF is implemented in GluonTS : https : //github.com/awslabs/gluon-ts/blob/master/src/ gluonts/model/rotbaum/README_LSF.ipynb We remark that it is precisely because the partitions of trees grown by the CART-splitting algorithm are data-dependent that consistency of RFs has only been proven recently [ 32 ] , while most other work ( [ 7 , 36 , 6 , 27 , 31 ] ) has focused on easier algorithms that are not used in practice . We can leverage these methods to prove consistency of LSF as well ( Theorem 1 ) ; and as a byproduct we also prove consistency of QRFs under the CART-splitting algorithm ( Theorem 3 ) . In order to have a consistency assurance , we require mild assumptions on the data generating process , and some intuitive assumptions on the point forecasting algorithm A . Assumption 1 . 1 . Assumption on the data generating process : ( a ) There is a uniformly equicontinuous family of functions py ( x ) ( as y varies ) so that py ( x ) integrates to P ( Y y|X ) for all y . Further assume that E ( Y |X ) is bounded . ( b ) For every `` > 0 there is a > 0 so that for every x in the image of X the probability that X lies in the ball of radius `` centered at x is at least . ( c ) There exists a function m ( x ) that integrates to E ( Y |X ) so that 8 '' > 0 9 > 0 such that for all x , x0 2 Rd and 0 y 1 : |m ( x ) m ( x0 ) | < implies |py ( x ) py ( x0 ) | < `` . 2 . Assumptions on the point estimator A : Let Wn : = { f ( X1 ) , ... , f ( Xn ) } , with k : = |Wn| . Assume that A has a choice ( for every n ) of hyperparameters that would satisfy : ( a ) Collisions among the training data f ( Xi ) = f ( Xj ) are rare in the precise sense that there exists a positive number C such that : P ✓ C < k n ◆ ! 1 . ( b ) For X an independent variable following the distribution of the Xi ’ s the value f ( X ) does not tend to be extremal among the f ( Xi ) ’ s . To be precise , for any sequence dn ! 0 of positive numbers we have that P ✓ | { v 2Wn|f ( X ) < v } | k > dn ◆ ! 1 , P ✓ | { v 2Wn|v < f ( X ) } | k > dn ◆ ! 1 . ( c ) The image of f is “ dense in probability ” . To be precise , ( ln ( n ) ) 2 max i=2 , ... , k 1 Vol ( Vi ) ! 0 in probability , where the Vi ’ s are the Voronoi sets from Equation 1 . ( d ) There ’ s a constant c so that 8j , 8x 2 f 1 ( Vj ) 8 '' > 0 the set f 1 ( Vj ) \ B '' ( x ) a.s. contains a ball of radius c · `` . ( e ) The base algorithm A is a mean square consistent estimator of the conditional mean : E ( |f ( X ) E ( Y |X ) |2 ) ! 0 . Theorem 1 . Under Assumption 1 , letting min_bin_size= ( ln ( n ) ) 2 , LSF is mean square consistent . That is , for any value y 2 R , we have E ( |⌘̂LSF ( X ) P ( Y y|X ) |2 ) ! 0 , ( 3 ) where the convergence is uniform in y . As a byproduct of proving this result , we also give the first consistency result ( Theorem 3 ) for QRFs grown under the CART-splitting algorithm under an additive regression model assumption . Remark 2 . Assumptions 1a and 1b are much weaker than the additive regression assumptions in [ 32 ] . Assumption 1c is what allows the base algorithm to be informative about the conditional pdfs . In light of the empirical success of LSF ( see Section 6 ) , it appears that Assumption 1c holds quite generally in real-world data . We expect that Assumptions 2a , 2b , 2c , and 2d , which simply ensure non-degenerate behavior , hold for any reasonable choice of A , but the precise statement is left as a future work . ( If f is locally constant on hyperrectangles then Assumption 2d holds with c = 1p d+1 . )

Summary: The authors propose a general method to derive probabilistic / conditional distribution predictions for any given trained base prediction model f(x) - i.e., CDF of y|x, by using an approach somewhat related to quantile regression forests (QRFs).  In this approach prediction function f(x_i) outputs are grouped with nearby values, so that groups have a minimum size from the training data.  Each group's true values {y_i} define a sample for that group that can be used to compute any CDF value - and, for instance provide a prediction interval via a set of quantiles.  A prediction for a new sample x is transformed into the probabilistic prediction by finding the nearest group to f(x), and using its samples to derive the distribution statistics.  The authors prove consistency under fairly reasonable assumptions of the prediction algorithm, as well as proving consistency for QRFs with CART splitting.  The authors also perform extensive experiments showing the proposed approach is effective at estimating distribution statistics / prediction intervals for a variety of datasets, compared to other approaches to deriving distributions from point predictions (conformal approach) as well as other probabilistic approaches (QRF, quantile regression approaches, DeepAR), showing it is both efficient and competitive in terms of accuracy, even out-performing other methods.  These include experiments with forecasting datasets showing how their approach applied to the point forecasting winner of M5 made it competitive for probabilistic forecasting with the top 5 probabilistic forecasting methods from the competition.