# A Bayesian Nonparametric Learning Approach to Ensemble Models Using the Proper Bayesian Bootstrap  
[PERSON]  
1 Department of Mathematics, University of Pavia, 27100 Pavia, Italy; [EMAIL_ADDRESS] (M.G.); [EMAIL_ADDRESS] (C.B.)  
[PERSON]  
1 Department of Mathematics, University of Pavia, 27100 Pavia, Italy; [EMAIL_ADDRESS] (M.G.); [EMAIL_ADDRESS] (C.B.)  
[PERSON]  
2 Department of Political and Social Sciences, University of Pavia, 27100 Pavia, Italy; [EMAIL_ADDRESS] 21 Department of Political and Social Sciences, University of Pavia, 27100 Pavia, Italy; pietro.  
computational cost and provide a Bayesian competitive prediction result with respect to the Random Forests algorithm.  
More recently, Bayesian nonparametric learning has been recognised as a good approach to solve predictive problems, overcoming the main weaknesses of parametric Bayesian models which assume fixed-dimensional probabilistic distributions ([12]). In [13], a Bayesian nonparametric procedure based on the [PERSON]'s bootstrap technique ([14]) is proposed to obtain a new class of algorithms called by the authors Bayesian Forests, under the idea that ensemble tree models can be represented as a sample from a posterior distribution over trees. [PERSON]'s bootstrap has been discussed as bagging procedure for different prediction models also in [15; 16], proving that Bayesian bootstrap lead to more stable prediction results in particular with small sample sized datasets.  
We remark that ensemble tree models based on [PERSON]'s and [PERSON]'s bootstraps ([4; 14]) are both non informative procedures since they do not take into account apriori knowledge (i.e., they build the prediction model just considering only observed values included in the data at hand and giving zero probability to values not available in the sample data). In addition these two procedures are proved to be asymptotically equivalent and first order equivalent from the predictive point of view ([17; 18]).  
The element of novelty of this paper is to introduce Proper Bayesian Bootstrap proposed by [19] in classification and regression ensemble trees models and compare the results obtained on simulated and real datasets with respect to classical bootstrap approaches available in the literature as [PERSON]'s and [PERSON]'s bootstraps. More precisely, Proper Bayesian bootstrap is used to sample the posterior distribution over trees, introducing prior distributions on the covariates and the target variable.  
The main aim of this paper is to employ Proper Bayesian bootstrap method in the data generative process introducing an ensemble approach based on decision tree models. In this work, bootstrap resampling techniques are applied to approximate the posterior distribution of a statistical functional \(\phi(F)\), where \(F\) is a random distribution function as defined in the Proper Bayesian bootstrap ([19]) and \(\phi(F)\) is a decision tree. Note that our methodological proposal inherits the main advantages of Bayesian nonparametric learning such as the flexibility and the computational strength, considering also prior opinions and thus overcoming the main drawbacks of [PERSON]'s and [PERSON]'s bootstrap procedures. On the basis of the results achieved in simulated and real datasets, our approach provides a reliable gain in predictive performance regarding the stability of the model, coupled with a competitiveness in terms of prediction accuracy.  
The paper is structured as follows: Section 2 describes the Bayesian nonparametric learning framework, Section 3 shows the extension of bootstrap techniques in the Bayesian nonparametric framework, Section 4 introduces our methodological proposal involving Proper Bayesian bootstrap procedure, Section 5 presents empirical evaluations of our method with respect to other ensemble learning models. Finally, conclusions and further ideas of research are reported in Section 6.