# Application of Machine Learning to Express Measurement Uncertainty
## 2 Methods
### DART Algorithm  
Ensemble training algorithms achieve high accuracy for numerous machine learning tasks. For ensembles to achieve better accuracy than the individual hypotheses from which they are composed, the hypotheses should be with uncorrelated errors. While some classes of ensemble training algorithms such as random forests independently add each hypothesis to the set, algorithms using boosting such as _AdaBoost_ and Multiple Additive Regression Trees (MART) [38] incrementally add hypotheses. This technique increases the ensemble's accuracy by reducing the sensitivity to specific features or cases that may exist in individual hypotheses. Boosting algorithms add hypotheses focused on improving the current set of hypotheses, which is achieved by modifying the training problem between iterations. Although this guarantees that the added hypothesis differs from the set, the new hypotheses usually focus on a small subset of the problem and thus have no significant prediction of the original problem. This increases the risk of adding hypotheses that fit too well to specific cases. It is a well-known problem in boost training and in the MART algorithm that uses ensemble training by boosting regression decision trees. In a given algorithm, trees added in later iterations tend to affect the prediction of only a few hypotheses and make a negligible contribution to the prediction of all remaining hypotheses. In turn, this can negatively affect the algorithm's performance on new data by increasing the capacity of the set without significantly improving its training error. Additionally, this makes the set hypersensitive to the contributions of a few initially added trees. This issue of posterior trees affecting the prediction of only a tiny fraction of hypotheses is referred to as over-specialization. The most common approach to combat over-specialization in the MART algorithm is reduction. Here, the contribution of each new tree is reduced by a constant value called the reduction factor. Downscaling helps to reduce the influence of the first trees; however, as the size of the set increases, the problem of over-specialization reappears. In the Dropout Additive Regression Trees (DART) algorithm [39], which is based on the MART algorithm, a different approach is proposed to solve the issue of over-specialization. This approach involves dropout, a technique proposed in the context of regularizing deep neural networks. In neural networks, dropout is used to silence a random portion of the neural connections during the training process. This method has significantly contributed to the success of deep neural networks for many tasks, including, for example, object classification in images. The DART algorithm introduces a new way of using dropout for a set of decision trees: complete tree silencing instead of feature silencing. In the available literature, evaluation results of the DART algorithm on three different tasks, ranking, regression, and classification, using publicly available large-scale datasets, show that DART outperforms MART and random forest in each task by significant margins [39]. One of the reasons for the improved performance of the DART algorithm is that it addresses the issue of over-specialization and results in a more balanced contribution of all trees in the hypothesis set. Another place where DART differs from MART is the addition of a new tree to the set where DART performs the normalization step. To ensure that the impact of the omitted trees and the newly added tree remains the same as the impact of the omitted trees before adding the new tree to the set, the newly added tree is first scaled by \(1/k\) and then together with the omitted trees by \(k/k+1\). The number \(k\) represents the number of omitted trees in a given iteration step of the algorithm. The DART algorithm reduces the problem of over-specialization. Therefore, it can be seen as regularization, where the number of omitted trees controls the degree of regularization. If no tree is left out, DART is no different from the MART algorithm.