Title: COLLABORATIVE TRAINING OF BALANCED RANDOM FORESTS FOR OPEN SET DOMAIN ADAPTATION Anonymous

Abstract: In this paper, we introduce a collaborative training algorithm of balanced random forests for domain adaptation tasks which can avoid the overfitting problem. In real scenarios, most domain adaptation algorithms face the challenges from noisy, insufficient training data. Moreover in open set categorization, unknown or misaligned source and target categories adds difficulty. In such cases, conventional methods suffer from overfitting and fail to successfully transfer the knowledge of the source to the target domain. To address these issues, the following two techniques are proposed. First, we introduce the optimized decision tree construction method, in which the data at each node are split into equal sizes while maximizing the information gain. Compared to the conventional random forests, it generates larger and more balanced decision trees due to the even-split constraint, which contributes to enhanced discrimination power and reduced overfitting. Second, to tackle the domain misalignment problem, we propose the domain alignment loss which penalizes uneven splits of the source and target domain data. By collaboratively optimizing the information gain of the labeled source data as well as the entropy of unlabeled target data distributions, the proposed CoBRF algorithm achieves significantly better performance than the state-of-the-art methods. The proposed algorithm is extensively evaluated in various experimental setups in challenging domain adaptation tasks with noisy and small training data as well as open set domain adaptation problems, for two backbone networks of AlexNet and ResNet-50.

1. INTRODUCTION:

In recent years, domain adaptation has been researched as it can help to solve major difficulties in the real world. Due to the huge overhead in labeling large-scale training data, it is desirable if an existing network can be adapted to different target domains. More importantly, it is common that the training dataset for adaptation is noisy and small, or the labels in the target domain do not match with the source or even unknown. These are inherent challenges in the domain adaptation problem as in real world it is common for the data to contain such class bias, noise and unlabeled data.

However, in practice, since the adapted networks are often overfitted to the provided source data or the data distribution of the target domain is frequently quite different from the source, they do not perform well to the target domain. To properly deal with these real-world conditions with insufficient information, it is critical to learn the shared data distribution that is effective both in the source and target domain. To this end, we propose the collaborative training algorithm of balanced random forest (CoBRF) to mitigate the challenging problems such as noisy labels, lack of training data, and misaligned or unknown categories (open set categorization).

In random forests, multiple decision trees are learned by optimizing the information gain for the randomly selected subset features at each node split. Since random forests ensemble the internal decision trees, they are more robust to noise and overfitting problem than single decision trees. To improve the robustness of the random forests, we take one step further by balancing the decision trees, i.e., maximizing the number of leaf nodes for the same tree depth. Our method builds more balanced decision trees by enforcing the sizes of the data in the left and right child nodes to be equal. While this split strategy is not locally optimal in terms of information gain, the resulting decision trees have far more leaf nodes, and it endows more expressive power which can be helpful in dealing with noise and unseen data or classes. It also helps to avoid overfitting as it prevents a node committing too early for a specific pattern, or in other words, it postpones the decision as late as possible so that various discriminant information in the training data can be fully considered.

To enforce even splits while maintaining the discriminability, the CoBRF uses the hyperplanes estimated by the linear support vector machine (SVM). First, it randomly assigns the classes in the nodes to binary pseudo labels and equalizes the sizes of two pseudo classes by randomly removing data in the larger class. Then a linear classifier is found by SVM, and its hyperplane is translated until the data sizes on both sides are equal. In a sense, it finds the even split of the data projected onto the normal direction of the hyperplane and places the hyperplane there. The node split by the translated hyperplane is simple yet effective. The ablation study in Sec. 4.2 confirms that the CoBRF boosts the performance compared to the baseline random forests.

Since the above training process only considers maximizing the information gain of labeled data in the source domain, which is referred to 'class information gain', it does not resolve the domain misalignment problem between the source and target domain. Because the target labels are not available during training, we try to keep the overall distribution of the target data as close to that of the source data as possible. Since the source data are evenly split, we guide the algorithm to minimize the information gain between the source and target domain, which encourages even split of the target data also. The CoBRF combines the ideas, minimizing the 'domain information gain' between source and target data for the domain alignment while keeping the class information gain to be maximized. Note that the domain alignment term is the same as the negative information gain of the binary domain labels (source/target). Thus, the CoBRF can be seen as an example of adversarial learning, as it considers the domain information gain in an adversarial manner compared to the conventional objective function of the random forest.

We summarize the main contributions as three-fold.

• We introduce the collaborative training algorithm based balanced random forest (CoBRF) using the discriminative and even node split function. Linear SVM with binary pseudo labeling is used to find the discriminative hyperplane and the even split ensures the decision tree to be balanced.

• We also adopt the adversarial learning of domain information gain to align the source and target data distribution. To align two domains, the information gain between the source and target data is minimized, which learns the common data distribution of both the (unlabeled) target domain and the source domain data.

• We perform an extensive evaluation of the domain adaptation to show the performance of the proposed method according to various challenging evaluation protocols. Specifically, it is compared to the baseline and state-of-the-art methods using noisy and small training data, and with open-set domain adaptation protocols. In both cases we observe significant performance improvements. The hyperplanes estimated by binary pseudo labels followed by translation for even split. Dotted line is the hyperplane estimated using linear SVM. The data are evenly split by the hyperplane shift (solid lines). Among these hyperplanes, the one with maximum information gain is chosen: yellow hyperplane in (a). (c) In CoBRF, both the source information gain and target entropy is considered. The yellow is better in source information gain, the target data split is biased, while the blue splits the source and target evenly well.

2. RELATED WORK:



3. PROPOSED METHOD:

In this section, we first explain the limitation of the conventional random forests for the domain adaptation task, and then we introduce the even node split function in Sec.3.1 and the domain information gain for selecting the domain-aligned split function in Sec. 3.2.

3.1. EVEN CONSTRAINED RANDOM FOREST LEARNING:

A random forest consists of multiple random decision trees, whose nodes learn a binary classifier for the randomly-selected subset of features to maximize the information gain (IG). We abuse the term node for the training data in the node interchangeably. The entropy of a node n is defined as

EQUATION

where C(n) represents the set of classes of the data in n, and p c (n) is the probability of class c in n (i.e., the data count of class c divided by |n|). Then the information gain for a node n with the left and right child nodes is defined as

EQUATION

Conventionally, the simple split functions that compares only a couple of feature values are used, but recently more elaborate split functions using the linear classifiers are used Yao et al. (2011) ; Ristin et al. (2015b) . The hyperplane split function for a node n is written as

EQUATION

where ψ n (•) is the sub-feature selection function and w n and k n are the hyperplane parameters either randomly set or learned by a linear support vector machine Cortes & Vapnik (1995) . The hyperplane with the largest information gain is the most discriminative classifier at the given node, but for the entire decision tree and the random forest it may not be the best option, because it causes the learned trees to be skewed and not well balanced (Fig. 1a ).

We propose to add a hard constraint of equal-size in splitting the node to get more balanced trees.

The detailed learning process is as follows. For the SVM to build a binary classifier, the classes in the node are randomly assigned to binary pseudo labels, and the training data for each class are assigned to the corresponding pseudo label. As the data sizes of the pseudo labels will be different, we randomly erase the data in the larger pseudo class to match the sizes. Then the base hyperplane (w n and k n ) is computed to classify the binary pseudo labels.

Still, the split of the training data by the hyperplane is not equal-sized; thus we update the bias k n of the hyperplane so that the data size on each side is equal or differs at most by one (||n lef t |-|n right || ≤ 1). Geometrically this process is moving the hyperplane along the normal vector w n , so that it is placed at the even split of the data projected onto the normal direction (Fig. 2a, 2b ). Among the estimated hyperplanes from randomly selected sub-features, the one that maximizes the information gain IG C (n) is chosen as the node split function. To build a decision tree, like the conventional random forest, the node split is repeatedly applied until the maximum depth is reached or too few data are left in the node (Fig. 1b ).

Inherently the proposed split method creates balanced trees, and for the same depth, the number of nodes is much larger than that of the conventional random forest. We argue that having more (leaf) nodes in the decision tree has advantages in domain adaptation tasks. The conventional split function is locally optimal, but because of that, it is more susceptible to overfitting by committing too early, and eventually, it decreases the discriminative power of the entire random forest. In the balanced trees, the data sizes in the leaf nodes are almost the same; thus, they represent local data distribution more faithfully. The even-size constraint can be thought of as a regularization in learning decision trees. The experimental results of the ablation study in Sec. 4.2 supports this argument.

3.2. COLLABORIVE LEARNING OF RANDOM FORESTS:

Balanced data distribution is a big advantage in domain adaptation. However, as it does not use the unlabeled target data for learning, it still does not correctly align the data distribution of the source and target domain. In other words, the distribution of the target data also needs to be considered in building a random forest. We propose a new collaborative measure for selecting the split function that considers both the conventional IG and the domain distribution of the source and target data together. The collaborative information gain (co-IG) is defined as

EQUATION

)

where λ is a user parameter, n s is the labeled training data (in the source domain), and D is the binary domain label {source, target} representing the domain that the data belongs to. More specifically, IG D (n) is the information gain on the domain distribution, when the data labels are either source or target, disregarding the classes in the source domain. The CoBRF chooses the hyperplane that maximizes co-IG when splitting the nodes.

Note that the IG on the domain distribution, IG D (n), is subtracted in Eq. 4, to ensure that we prefer even distribution of the source and target data in the child nodes. IG D (n) is minimized when both source and target data are evenly split into the children, as it maximizes the entropy of the children (Eq. 2). Thus IG D (n) in CoBRF collaboratively enforce the even split of target data also.

Fig. 2c illustrates the effect of co-IG compared to conventional IG in split function selection. The yellow line has higher IG as it segments the source data (colored) better, but co-IG also considers the split of target data (gray). Although the blue line has lower IG than the blue, it separates the target data more evenly; thus, the blue line is chosen as the split function. The resulting decision trees by CoBRF are shown in Fig. 3 .

The co-IG is closely related to the adversarial learning of the network backpropagation Long et al. (2015) ; Ganin et al. (2016) . In this framework, IG C and IG D can be thought of as the classification and adversarial domain alignment, respectively. Thanks to the domain alignment term (co-IG), the CoBRF learns the robust models even with very noisy or small training data without overfitting. We validate the proposed methods from the moderate challenging condition such as 40% noise data to the very severe condition such as 80% noise or merely 10% training data. Further, we evaluate the open set domain adaptation, which has received attention in recent years, in the following section.

4. EXPERIMENTAL RESULTS:



4.1. EXPERIMENTAL SETTING:

We use three domain adaptation datasets such as Office-31 Saenko et al. ( 2010), ImageCLEF-DA1 and Office-Home Venkateswara et al. (2017) . We evaluate algorithms using three challenging protocols: noisy, small training data, and weakly supervised open set domain adaptation. To train the CoBRF, we use 100 trees with a maximum depth of 8. When there is no training data fallen on a node, we prune the tree at that node. The number of randomly selected feature dimension for the SVM training is set to 250. The input feature of SVM is normalized for the stable learning of the hyperplane. We repeat the SVM training 15 times to select the optimal split in each node. Due to the space limitation, only representative results are shown in this section. Refer to the appendix for detailed information of the datasets, metric, and full experimental results.

4.2. ABLATION STUDY:

We evaluate the effect of components in the CoBRF proposed in this paper. The CoBRF uses the balanced pseudo labeling (mid_pseudo), and the hyperplane shift (h_shift) for even data split. As binary labeling is necessary for hyperplane computation, the pseudo method uses randomly-assigned binary pseudo labels without removing the data to make label sizes equal. Therefore the four 1 and 8 of appendix, both balancing the pseudo labels and enforcing even splits by translating hyperplanes improve the performance.

Table 2 shows the effect of the parameter λ in co-IG formulation (Eq. 4). It confirms that optimizing for the cobalanced distribution helps the alignment of domain distributions. 

4.3. NOISY DATA:

In this experiment, the training labels of the specified portion of the source domain are randomly changed for the noise condition, which is also referred to as the label corruption in Shu et al. (2019) .

Corruption levels are set to 40, 60 and 80% of the source domain (refer to the appendix for full experimental results).

We conduct noisy conditions for the Office-31, ImageCLEF-DA and Office-Home datasets in Table 4 and 5. We test DAN Long et al. (2015) , JAN Long et al. (2017), and CDAN+E Long et al. (2018) 4 shows the more severe the noise is, the larger the performance improvement gets.

4.4. SMALL TRAINING DATA:

In this experiment, we use only 10% of training samples to evaluate the performance against overfitting. We perform the experiments on Office-31, Office-Home, and ImageCLEF-DA datasets with ResNet-50. The result of Table 4 shows the CoBRF achieves favorable performance compared to the other algorithms. Full experimental results are presented in the appendix. 

5. CONCLUSION:

We propose a novel cobalanced random forest (CoBRF) algorithm for challenging conditions and open set protocols. The CoBRF enhances the discriminative ability of the random forest by building balanced decision trees by the even split. The proposed CoBRF algorithm also employs the adversarial learning for domain alignment and benefits the effectiveness against the overfitting to the labeled source data. We extensively evaluate the proposed algorithms using challenging experimental protocols and demonstrate its superior performance over the baseline and state-of-the-art methods.

:

https://www.imageclef.org/2014/adaptation

DAN, JAN: https://github.com/thuml/Xlearn , CDAN+E: https://github.com/thuml/CDAN

Figures and Tables:

Figure 1: Split examples in decision tree according to the split functions: (a) the conventional method chooses the split that maximizes the information gain. (b) In contrast, the proposed method additionally enforces the size of child nodes to be equal, resulting in a random balanced tree. Note that CoBRF has far mode nodes which improves the generalization ability for domain adaptation.

Figure 3: Visualization of trees learned by the proposed methods. The white and gray circles at the leaf nodes represent the source and target data fallen into the node, respectively. As a tree without domain alignment only considers the labeled source data, the target data distributions in leaf nodes are not even, whereas that with domain alignment generates more uniform splits. Refer to Sec. 3.2 and Fig. 2c.

OPEN SET EXPERIMENTSWe perform two open set evaluation protocols proposed inSaito et al. (2018);Tan et al. (2019).OpenSet1:The first open set protocolSaito et al. (2018) uses 11 classes (10 known and 1 unknown) of the Office-31 dataset. The labels from 1 to 10 of both source and target domains are marked as the known class, and all data with label 21∼31 in the target domain are used as one unknown class. According toSaito et al. (2018) the unknown class of the source data is not used in training, and the unknown class of the target data is classified by thresholding the class probability. The thresholding value is set to 0.3. Table6shows the result of CoBRF as well as the state-of-the-art methods. The CoBRF achieves the best performance among all algorithms on Office-31. It also demonstrates the effectiveness of the proposed method under the challenging adaptation condition.OpenSet2:Recently, another open set protocol is proposed inTan et al. (2019), which uses partially overlapping known classes between the source and target domain. Each domain has 5 known-andcommon classes, 5 known-but-different classes, and 1 unknown class for all other training data, thus in total there are 15 known and 1 unknown classes. First, according toTan et al. (2019), 3 samples per class per domain and 9 samples in the unknown class per domain are used in training. Hence the total number of training samples is (3 samples × 10 classes/domain + 9 samples_in_unknown) × 2 domains = 78. All other algorithms and CoBRF results in

CoGAN proposed byLiu & Tuzel (2016) learns the joint distribution of multiple domains without corresponding image pairs.Sankaranarayanan et al. (2018)  propose the combined adversarial and discriminative learning method using the generator and discriminator of GAN.2.2 EVALUATION PROTOCOLSIN DOMAIN ADAPTATION Recently, many challenging protocols are introduced to evaluate the domain adaptation in realistic settings. Regarding domain generalization on deep neural networks Li et al. (2018); Balaji et al. (2018), they divide multiple domain data into training and test set, then use the leave-one-domain-out scheme for evaluation. The domain adaptation on the partially overlapping source and target domains is presented in Zhang et al. (2018a); Cao et al. (2018). Multiple sources and target domains are mixed into the source or target domains in Zhao et al. (2018); Mancini et al. (2018); Hoffman et al. (2018).The adaptable model is aimed to be learned using the distribution to the multiple domains of the mixed set.

Table content: <table><tr><td>many domain adaptation methods Liu &amp; Tuzel (2016); Sankaranarayanan et al. (2018); Volpi et al.</td></tr><tr><td>2.1 DOMAIN ADAPTATION Recently adversarial learning has been one of the dominant approaches in domain adaptation with deep neural networks. The gradient reversal layer Ganin &amp; Lempitsky (2015) is introduced to train (2018). There have been many recent works to improve the performance of random forests: Dheenadayalan</td></tr><tr><td>the networks so that the discrimination of source and target domains is penalized. It improves the et al. (2016) proposes pruning nodes for efficient learning, Ristin et al. (2015a) presents incremental</td></tr><tr><td>classification performance compared to the networks learned only with the source data. Tzeng et al. modeling for large scale recognition, and Probst &amp; Boulesteix (2017) investigates how to tune the</td></tr><tr><td>(2017) suggest the domain adaptation framework based on the discriminative network learning, which number of trees. SVM Yao et al. (2011); Ristin et al. (2015b) or random projection Bosch et al.</td></tr><tr><td>assigns individual weights to the source and target domains. In training the networks, they also (2007); Bossard et al. (2014) is often used as the binary classifier for better node split. Training</td></tr><tr><td>consider the adversarial weight update to align the domains. Several other domain adaptation papers balanced decision trees has been also an important topic Bosch et al. (2007); Bossard et al. (2014);</td></tr><tr><td>in adversarial learning using conditional learning Long et al. (2018), domain-symmetric Zhang et al. Yao et al. (2011); Lei et al. (2014); Ristin et al. (2015b). They split a node into child nodes by the</td></tr><tr><td>(2019), and collaborative Zhang et al. (2018b) methods have been introduced. Also, in Tzeng et al. binary classifier, which is trained by evenly-divided training data in the node. We argue that training</td></tr><tr><td>(2014); Long et al. (2015; 2016), maximum mean discrepancy (MMD)-based methods have been balanced random forests helps to alleviate the overfitting problem since balancing random decision</td></tr><tr><td>studied. Tzeng et al. (2014) propose the domain confusion loss to improve domain distribution</td></tr><tr><td>alignment. Long et al. (2015) introduce the task-specific embedding and multiple kernel approach</td></tr><tr><td>along with MMD to decrease the domain discrepancy. The residual transfer module presented in Long</td></tr><tr><td>et al. (2016) associates the classification ability of the source and target domain. MMD is further</td></tr><tr><td>extended to multiple domain alignment in the joint adaptation networks (JAN) Long et al. (2017)</td></tr><tr><td>using adversarial learning. The generative adversarial networks Radford et al. (2015) are adopted in</td></tr></table>

Ablation study of components for the split function of random forests without the domain alignment. The experiment is performed on Amazon (A), Webcam (W) and DSLR (D) domains of Office-31 with ResNet-50.

Table content: <table><tr><td colspan="5">Hyperplane estimation pseudo mid_pseudo pseudo mid_pseudo</td></tr><tr><td>h_shift</td><td>X</td><td>X</td><td>O</td><td>O</td></tr><tr><td>Accuracy</td><td>70.6</td><td>72.1</td><td>74.3</td><td>74.6</td></tr></table>

The effect of λ for domain alignment in the CoBRF. The experiment is performed on Office-31 with ResNet-50.

Table content: <table><tr><td>λ</td><td>0</td><td>0.001 0.01 0.1</td><td>0.5</td><td>1.0</td></tr><tr><td colspan="5">Accuracy 70.3 71.4 72.3 74.0 74.6 74.5</td></tr></table>

The effect of the tree numbers and maximum depth in training the CoBRF. The experiment is performed on 40 % noisy condition of the Office-31 (W->A) with ResNet-50.

Table content: <table><tr><td>Depth</td><td colspan="4">The number of trees (T)</td></tr><tr><td/><td>5</td><td>10</td><td>50</td><td>100</td></tr><tr><td>6</td><td colspan="4">62.2 65.4 67.5 67.7</td></tr><tr><td>7</td><td colspan="4">60.2 63.5 67.1 67.8</td></tr><tr><td>8</td><td colspan="4">58.9 63.6 67.5 68.0</td></tr><tr><td>9</td><td colspan="4">55.6 60.0 66.2 67.6</td></tr></table>

Performance comparison of the 60 and 80% Noisy and 10% Small training data protocol on Office-31, ImageCLEF-DA and Office-Home dataset with ResNet-50.

Table content: <table><tr><td/><td colspan="2">Office-31</td><td colspan="2">ImageCLEF-DA</td><td colspan="2">Office-Home</td></tr><tr><td>Method</td><td>Noisy 60% 80%</td><td>Small</td><td>Noisy 60% 80%</td><td>Small</td><td>Noisy 60% 80%</td><td>Small</td></tr><tr><td>DAN</td><td>37.6 19.8</td><td>66.8</td><td>36.3 19.2</td><td>74.4</td><td>32.1 18.4</td><td>43.8</td></tr><tr><td>JAN</td><td>48.7 24.6</td><td>69.7</td><td>42.4 19.7</td><td>76.6</td><td>35.6 21.6</td><td>45.3</td></tr><tr><td colspan="2">CDAN+E 49.8 22.0</td><td>67.5</td><td>54.5 25.0</td><td>79.6</td><td>34.0 15.1</td><td>44.2</td></tr><tr><td>CoBRF</td><td>65.6 44.3</td><td>74.6</td><td>67.8 32.6</td><td>79.8</td><td>56.8 46.5</td><td>51.8</td></tr><tr><td colspan="7">combinations of (pseudo, mid_pseudo)×h_shift are tested with Office-31. The baseline is (pseudo +</td></tr><tr><td colspan="2">no_h_shift). As shown in Table</td><td/><td/><td/><td/><td/></tr></table>

Table 3 also represents the effect of hyperparameters in training CoBRF. The accuracy of the maximum depth 8 with 100 decision trees shows the best result.

Table content: <table/>

Performance comparison of the 40% Noisy protocol on Office-31 with ResNet-50.

Table content: <table><tr><td>Method</td><td/><td/><td colspan="3">Domain adaptation</td><td/><td/></tr><tr><td/><td colspan="7">A→D A→W D→A D→W W→A W→D Average</td></tr><tr><td>RTN Long et al. (2016)</td><td>76.1</td><td>64.6</td><td>49.0</td><td>71.7</td><td>56.2</td><td>82.7</td><td>66.7</td></tr><tr><td>ADDA Tzeng et al. (2017)</td><td>61.2</td><td>61.5</td><td>45.5</td><td>65.1</td><td>49.2</td><td>74.7</td><td>59.5</td></tr><tr><td colspan="2">MentorNet Jiang et al. (2018) 75.0</td><td>74.4</td><td>43.2</td><td>70.6</td><td>54.2</td><td>85.9</td><td>67.2</td></tr><tr><td>TCL Shu et al. (2019)</td><td>83.3</td><td>82.0</td><td>60.5</td><td>77.2</td><td>65.7</td><td>90.8</td><td>76.6</td></tr><tr><td>CoBRF</td><td>81.9</td><td>82.1</td><td>65.4</td><td>81.1</td><td>68.0</td><td>92.8</td><td>78.5</td></tr></table>

Performance comparison of the OpenSet1 protocol on the Office-31 dataset with AlexNet Krizhevsky et al. (2012).

Table content: <table><tr><td>Method</td><td colspan="7">A→D A→W D→A D→W W→A W→D Average</td></tr><tr><td>OSVM</td><td>59.6</td><td>57.1</td><td>14.3</td><td>44.1</td><td>13.0</td><td>62.5</td><td>40.6</td></tr><tr><td>ATI-λ + OSVM</td><td>72.0</td><td>65.3</td><td>66.4</td><td>82.2</td><td>71.6</td><td>92.7</td><td>75.0</td></tr><tr><td colspan="2">Saito et al. (2018) 76.6</td><td>74.9</td><td>62.5</td><td>94.4</td><td>81.4</td><td>96.8</td><td>81.1</td></tr><tr><td>CoBRF</td><td>86.0</td><td>80.5</td><td>73.0</td><td>94.5</td><td>69.4</td><td>94.6</td><td>83.0</td></tr></table>

Performance comparison of the OpenSet2 protocol on the Office-31 dataset with ResNet-50. Results of CoBRF* are from a more challenging setup. Refer to Sec. 4.5. the same noisy condition for comparison. Table5shows the result of 40% noisy training data for Office-31 with ResNet-50He et al. (2016). The proposed CoBRF outperforms all other algorithms in average accuracy. The result confirms that the CoBRF improves the performance in most settings, and interestingly, Table

Table content: <table><tr><td>Method</td><td colspan="4">A ↔ D A ↔ W D ↔ W Average</td></tr><tr><td>JAN Long et al. (2017)</td><td>65.5</td><td>63.8</td><td>74.7</td><td>68.0</td></tr><tr><td>ATI-semi Panareda Busto &amp; Gall (2017)</td><td>72.0</td><td>73.4</td><td>77.8</td><td>74.7</td></tr><tr><td>CDA Tan et al. (2019)</td><td>75.2</td><td>77.1</td><td>88.1</td><td>80.1</td></tr><tr><td>CoBRF</td><td>82.3</td><td>83.1</td><td>92.9</td><td>86.1</td></tr><tr><td>CoBRF*</td><td>82.0</td><td>81.2</td><td>89.7</td><td>84.3</td></tr><tr><td>algorithms 2 on</td><td/><td/><td/><td/></tr></table>

Table7are using this protocol. Additionally, we evaluate more challenging setup, where the training data are sampled regardless of the domain, i.e., the data in common classes (including unknown) are merged before being sampled. In this case, 3 samples × 15 classes + 9 samples_in_unknown = 54 in total are used. The results in CoBRF* rows are acquired in this setup. We confirm that CoBRF works well compared to state-of-the-art methods under the OpenSet2 and more challenging condition.

Table content: <table/>

Peer review (iclr2020):
Summary: This paper introduces a method for domain adaptation, where each domain has noisy examples. Their method is based on a decision tree in which the data at each node are split into equal sizes while maximizing the information gain. They also proposed a way to reduce domain alignment. Their method is tested on several noisy domain adaptation settings and performs better than other baseline methods. Pros: Their idea to utilize a decision tree for domain adaptation sounds novel. Experiments indicate the effectiveness of their method. Cons: This paper is not well-written and has many unclear parts. 1, The presentation of the problem set is unclear throughout this paper. In the abstract, they mentioned that they tackle the situation where both source and target domains contain noisy examples. However, they did not define the exact problem setting in any section. I could not understand what kind of problem setting motivated their method, which makes it hard to understand their method. 2, How they actually optimized the model is also unclear. From Eq 1~4, it is hard to grasp how they trained the model. 3, In open-set domain adaptation, simply minimizing domain-distance can harm the performance. How does the method avoid this issue? It was also unclear. 4, Experimental setting seems to be wrong and unclear. In Openset1, they say that *The labels from 1 to 10 of both source and target domains are marked as the known class, and all data with label 11?20 in the source domain and label 21?31 in the target domain are used as one unknown class*. However, Saito et al. (2018) used 21-31 classes in the target domain as one unknown class. In addition, *According to Saito et al. (2018) the target data of the unknown class is not used in training, *, they used the 21-31 classes for training in an unsupervised way. How is this method used to detect unknown class? Is there any threshold value set for it? 5, The experimental setting is unclear. In 4.4, *, we use only 10% of training samples*, does it mean 10 % training source examples or target examples? This setting is also unclear. From the cons written above, this paper has too many unclear parts in the experiments and method section. I cannot say the result is reproducible given the content of the paper and the result is a reliable one. They need to present more carefully designed experiments.