Adatree 2 : Boosting to build decision trees or Improving Adatree with soft splitting rules

We extend the framework of Adaboost so that it builds a smooth ed decision tree rather than a neural network. The proposed method, “Adatree 2”, is derived from the assumption of a probabilistic observ ation model. It avoids the problem of over-fitting that appears in other tr e-growing methods by reweighing the training examples, rather than sp litting the training dataset at each node. It differs from Adaboost by st eering the input data towards weak classifiers that are “tuned” to the co nditional probability determined by the output of previously evaluat ed classifiers. As a consequence, Adatree 2 enjoys a lower computation cost t han Adaboost. After defining this method, we present some early exp erimental results and identify some issues left open for future resear ch. 1 Boosting and decision trees Boosting [1, 2] refers to methods to build weighed combinati ons of “weak classifiers”. At each training round, a weak classifier is added to the existin g combination. These methods reduce the classification error on a training dataset at a geometric rate and have been shown to have good properties [3]. They are known to often not overfit and generalize well even when training is continued way beyond perfect classific ation of the training dataset. However, the computation cost of a boosted classifier increa ses linearly with the number of training rounds, so that, in computation-sensitive situat ons, some adaptations [4, 5, 6, 7, 8] are done to benefit from the good properties of boosting at a re asonable computation cost. It has been noted [9] that decision tree-growing can also be v iewed as a boosting mechanism, in the sense that the training error can be made to decre ase geometrically. However, decision trees are notorious for over-fitting [10, Ch. 3.7] a nd tree “pruning”, “shrinking” [11, cited by [12]] and “smoothing” [11, 13] are techniques d eveloped to address this problem. Even then, over-fitting remains and decision trees have been combined by boosting to improve the generalization ability. The over-fitting of decision trees can be attributed to the fa ct that each node is determined by only the examples that reach it during training. Conseque ntly, deeper nodes are built from dwindling training sets that are less and less represen tative of the general population. This is a very slightly modified version of an article submitt ed to NIPS 2004. Table 1: Training and classification processes, of Adaboost , decision trees and Adatree 2. Classification Training Adaboost • All classifiers are evaluated. • Output is linear combination of all classifiers. • Each classifier is trained with all examples, appropriately weighed. Adatree 2 • Only weak classifiers on path of input are evaluated. • Output is linear combination of evaluated classifiers. • Each classifier is trained with all examples, appropriately weighed. Decision Tree • Only weak classifiers on path of input are evaluated. • Output is that of last classifier or (smoothed trees) a linear combination of evaluated classifiers. • Each classifier is trained with the fraction of examples that reaches it. The proposed method, “Adatree 2” differs in that respect by d iminishing the training weight of examples rather than discarding them altogether. This re-weighing scheme is derived by assuming a probabilis tic observation model that will be introduced in Section 3. Given the observation model , we derive Adatree 2 by minimizing the expected classification error on the trainin g dataset, in a similar manner to the minimization of the classification error in Adaboost. In Section 4, we will show that different assumptions on the o bservation model yield variants of Adatree 2 that are equivalent to Adaboost or to our pre vious method [14], which we shall call “Adatree 1” in the sequel. Table 1 compares the tra ining and classification stages of Adaboost, decision trees and Adatree 2. This section also states the main problem that we leave open for future study, namely the choice of an observ ation model. Section 5 presents some preliminary experimental results t hat suggest that Adatree 2 has a practical potential : its generalization ability appear l ess good than that of Adaboost while improving over Adatree 1. Also, Adatree 2 improves ove r Adaboost in terms of computation cost. Finally, some concluding notes are given in Section 6.