Generalized Convolutional Forest Networks for Domain Generalization and Visual Recognition

1 INTRODUCTION . Random forests have been applied to various problems ranging from object classification ( Bosch et al. , 2007 ) , object detection ( Gall & Lempitsky , 2013 ) , image segmentation ( Schroff et al. , 2008 ; Shotton et al. , 2008 ) , pedestrian detection ( Marin et al. , 2013 ; Tang et al. , 2012 ) to semantic hashing ( Qiu et al. , 2018 ) . In addition , random forests have been applied to the head pose ( Fanelli et al. , 2011 ) , landmark ( Cootes et al. , 2012 ) and age ( Shen et al. , 2018 ) estimation as a regressor and the sparse feature matching problems ( Lepetit & Fua , 2006 ; Ozuysal et al. , 2010 ) . The main reason for the robust performance of random forests is the decision tree ensembles . While each decision tree may achieve mediocre performance , the aggregated random forest performs significantly better and less correlated if the decision trees are heterogeneous . The overall accuracy of the trees can be considered as strength , and the heterogeneity of the trees can be measured by correlation . In ( Breiman , 2001 ) , the upper bound of the generalization error of random forests is expressed in terms of strength and correlation . High strength and low correlation are important properties to minimize the generalization error of a random forest . However , these two conflicting factors make it is difficult to improve strength and lower correlation simultaneously . If the individual decision trees in a random forest are strengthened independently , it is likely for the trees to resemble the strongest tree in the forest , and consequently the correlation of the forest becomes high . To reduce correlation , the decision trees must be in different shapes , and the strength of individual trees would not be as high as the best decision tree . In this work , we introduce generalized learning for random forests with convolutional neural networks to address these issues . The proposed method iteratively improves the generalized ability of random forests for higher strength and lowers correlation by probabilistic triplet sampling . For a triplet of an anchor , a positive , and a negative sample , the loss function is designed to pull the anchor and the positive closer and to push the anchor and the negative apart . The positive is sampled among the ∗Corresponding author . data with the same label with the anchor but in different leaf nodes of the decision trees , and the negative is from the data in the same leaf nodes with the anchor . The former contributes to improving the classification accuracy by positive sampling , whereas the latter discourages the algorithm from constructing similar decision trees by negative sampling . Note that both data points of the same label and of different labels with respect to the anchor can be in the negative training sample set . To directly improve the strength of the random forest only , we may design a method where negative examples are sampled among the data in the same leaf nodes and with different labels from the anchors . However , in practice , it suffers from early saturation and local minima . We describe the details of the proposed learning algorithm and experimental results in the following sections . The main contribution of the proposed work are summarized as follows : • We propose a generalization algorithm of random forests with convolutional neural networks . The proposed method minimizes the triplet loss function which 1 ) encourages to have samelabeled data in different leaf nodes move close and 2 ) pushes away data points that frequently fall in the same leaf nodes regardless of their class labels . • We consider both strength and correlation of random forest simultaneously . These two conflicting properties are handled by triple sampling . We demonstrate that the proposed method increases strength while maintaining a correlation in the experiments . • We show that the proposed algorithm performs well on domain generalization and image recognition against the baseline random forest methods . Furthermore , we show the proposed method performs favorably against state-of-the-art methods for the same tasks . 2 PRELIMINARIES . 2.1 RANDOM FOREST . Since the introduction of random forests ( Breiman , 2001 ) , numerous methods have been developed to increase the performance . We discuss recent and relevant methods for vision tasks in this section . In ( Gall & Lempitsky , 2013 ) , the Hough transform is used to tally probabilistic voting of detection hypotheses of parts from a random forest for object detection . ( Bosch et al. , 2007 ) use local shape and image appearance together to enhance the discriminative strength of the random forest . Visual bag-of-words model and gradient feature are utilized for encoding the shape and local information . In ( Zhang & Suganthan , 2014 ) , a method that uses linear discriminant analysis to increase the strength of a decision split is proposed . However , in most cases , increasing the strength does not maximize the performance of the random forest because the correlation also increases . The conflicting factors of strength and correlation in designing decision trees have been studied in recent methods ( Rodriguez-Galiano et al. , 2012 ) to analyze the performance of random forest . ( Yao et al. , 2011 ) propose to use SVM and image patches to enhance strength while maintaining the correlation of random forest . In this method , the SVM is used as a split function of each node at a decision tree to increase strength and use randomly selected image patches to decrease correlation . While this method performs well on fine-grained classification , it is not clear how this approach can be generalized to other tasks . In contrast , we propose a generalized learning algorithm of random forests that can be applied to various tasks using deep neural networks while considering both strength and correlation . 2.2 GENERALIZED ERROR BOUND OF RANDOM FORESTS . ( Breiman , 2001 ) shows that the generalized error PE∗ of a random forest is bounded by : PE∗ ≤ ρ̄ ( 1− s 2 ) s2 , ( 1 ) where ρ̄ is the correlation and s is the strength of the random forest . The strength s is the expectation of the margin function mr ( · ) with respect to a feature X and its label Y : s = EX , Y [ mr ( X , Y ) ] , and fmr ( X , Y ) = PΘ ( h ( X , Θ ) =Y ) −max j 6=Y PΘ ( h ( X , Θ ) =j ) , where h ( · ) is a classifier , and Θ is a random vector used to generate decision trees . In addition , PTheta ( · ) is the probability described the parameter Θ for a classifier h. The strength represents the expected margin of probability that the random forest makes a correct classification than a wrong classification . To define the correlation , the raw margin function is used : frmg ( Θ , X , Y ) =I ( h ( X , Θ ) =Y ) − I ( h ( X , Θ ) =ĵ ( X , Y ) ) , where ĵ ( X , Y ) = arg maxj 6=Y PΘ ( h ( X , Θ ) = j ) , and I ( · ) is an indicator function . When ρ ( Θ , Θ′ ) is the correlation between rmg ( Θ , X , Y ) and rmg ( Θ′ , X , Y ) , the correlation of a random forest ρ̄ is the mean value of the correlations over Θ . More details on the generalized error bound of random forests can be found in ( Breiman , 2001 ) . From Eq . 1 , it is straightforward to see that high strength of random forest reduces the upper bound of the generalization error if the correlation is suppressed ( Breiman , 2001 ) . However , these two conditions can not be simultaneously satisfied in practice . 2.3 TRIPLET LOSS . In general , the loss function of the triplet is defined as : ∑ ( a , p , n ) ∈S max ( 0 , ‖ f ( a ) − f ( p ) ‖22 − ‖ f ( a ) − f ( n ) ‖ 2 2 + b ) , where f ( a ) , f ( p ) and f ( n ) are the feature vectors of the anchor , positive , and negative data for a triplet ( a , p , n ) in the training set S , and b is a user-specified margin . The goal of the training process is to find the best feature f∗ that minimizes the loss function . In optimization process , the feature positions of the anchor and the positive sample are pulled closer and those of the anchor and the negative sample are pushed away . The triplet loss function has been used in numerous applications including face recognition ( Parkhi et al. , 2015 ; Schroff et al. , 2015 ) , image retrieval ( Zhao et al. , 2015 ) , person re-identification ( Cheng et al. , 2016 ; Zhang et al. , 2016 ) , and metric learning ( Norouzi et al. , 2012 ; Wang et al. , 2014 ) , to name a few . The triplet loss function uses both positive and negative samples at the same time , thus achieves improved performance . In the existing methods , positive samples are the data of the same labels with the anchor , and the negative samples are the data with different labels . For face recognition , person identities are the labels , and for person re-identification , the tracklet determines the positive and negative sample sets . Minimizing the triplet loss then gathers the same-labeled data together and separates differently-labeled data apart on the learned feature space , so that a classifier can easily partition the data . Compared to the above approaches , our focus on how to sample the effective triplets to improve both classification ability and heterogeneity of random forests . We show that in random forest training simply clustering data according to the labels does not bear the best result . Since it increases both the strength and correlation of the random forest , and we present the supportive experimental results . Algorithm 1 Learning algorithm for generalized convolutional forest networks . Input : Training Image I , Class label Y Output : Generalized Neural Networks N∗ 1 : for i← 1 to maximum iterations do 2 : F ′ ← Ni ( I ) 3 : Construct decision trees from F ′ and Y 4 : Construct P p and Pn from split results by the decision trees 5 : S ← Sample triplets by P p and Pn 6 : Ni+1 ← Update by minimizing triplet loss on S 2.4 DOMAIN GENERALIZATION . Generalizing models learned from one domain to another is an important topic in machine learning and computer vision . Learning deeply connected neural networks with a large-scale dataset helps improve the generalized ability such that CNN features trained on the ImageNet dataset are used as the generic representation for various visual domains ( Sharif Razavian et al. , 2014 ) . However , it is still difficult to adapt a model to different domains when large domain gaps exist . In addition , it is even more challenging if we do not have any data from the target domain , or if we have a mixture of source and target domains data . Hence , numerous domain generalization methods have recently been developed to tackle this problem . Several approaches have been developed including regularization with meta-learning ( Balaji et al. , 2018 ) , domain-invariant conditional learning ( Li et al. , 2018b ) , adversarial back-propagation ( Li et al. , 2018a ) , and episodic training algorithm ( Li et al. , 2019 ) .

Summary: The paper aims to improve the random forest performance by iterative constructing & feeding more powerful features that are to be used by the random forest learning processing where a random subset of features are chosen from the current feature pool when making growing/stopping decision at the current split node. The idea is new and interesting, and its usefulness has been empirically shown. On the other hand, it is not clear how this additional procedure would do with the good properties of RFs such as less subjective to overfitting and bias. It would be very helpful if the paper could shred some lights in this regard.