Title: Semi-Supervised Domain Generalization with Known and Unknown Classes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Domain Generalization

Semi-Supervised Learning

Out-of-Distribution Detection

Deep Learning
Abstract: Semi-Supervised Domain Generalization (SSDG) aims to learn a model that is generalizable to an unseen target domain with only a few labels, and most existing SSDG methods assume that unlabeled training and testing samples are all known classes. However, a more realistic scenario is that known classes may be mixed with some unknown classes in unlabeled training and testing data. To deal with such a scenario, we propose the Class-Wise Adaptive Exploration and Exploitation (CWAEE) method. In particular, we explore unlabeled training data by using one-vs-rest classifiers and class-wise adaptive thresholds to detect known and unknown classes, and exploit them by adopting consistency regularization on augmented samples based on Fourier Transformation to improve the unseen domain generalization. The experiments conducted on real-world datasets verify the effectiveness and superiority of our method.

Semi-Supervised Domain Generalization with Known and Unknown Classes

Abstract

Semi-Supervised Domain Generalization (SSDG) aims to learn a model that is generalizable to an unseen target domain with only a few labels, and most existing SSDG methods assume that unlabeled training and testing samples are all known classes. However, a more realistic scenario is that known classes may be mixed with some unknown classes in unlabeled training and testing data. To deal with such a scenario, we propose the Class-Wise Adaptive Exploration and Exploitation (CWAEE) method. In particular, we explore unlabeled training data by using one-vs-rest classifiers and class-wise adaptive thresholds to detect known and unknown classes, and exploit them by adopting consistency regularization on augmented samples based on Fourier Transformation to improve the unseen domain generalization. The experiments conducted on real-world datasets verify the effectiveness and superiority of our method.

1 Introduction

The machine learning community has witnessed the great progress of deep learning models and their applications, e.g., computer vision [1, 2], natural language processing [3, 4]. Such a huge success is mostly based on the i.i.d. assumption, i.e., the training data and testing data are identically and independently distributed. However, when these popular models are evaluated on new testing data, whose distribution is slightly different from the training data, a significant drop in performance will be observed [5, 6].

In order to improve the generalization of the model under distribution shifts, Domain Adaptation (DA) [7, 8, 9] and Domain Generalization (DG) [10, 11, 12] have been widely studied. The goal of DA is to transfer the knowledge learned

Photo

Art

Cartoon

Sketch

known classes

known classes

seen unknown classes

known classes

seen unknown classes

unseen unknown classes

L3

from label-rich source domains to the unlabeled or partially labeled target domain. However, the accessibility of the target domain may not always be satisfied in some applications, e.g., autonomous driving and medical diagnosis, since we are not able to anticipate the domain that deployed systems will encounter. To this end, DG is introduced to develop domain-generalizable models on unseen target domain by using multiple different and labeled source domains.

To reduce the expensive cost of the labeling process [13], Semi-Supervised Domain Generalization (SSDG) [14, 15, 16, 17] is proposed recently. In SSDG, each source domain consists of a few labeled samples and a large number of unlabeled samples. The goal is to learn a domain-generalizable model from the partially labeled samples. However, in a more realistic scenario, known classes are probably mixed with some unknown classes in unlabeled training and testing data (Figure 1). In such a scenario, when deployed on an unseen target domain, the learned model is not only required to classify known classes , but also to recognize unknown classes . In this paper, we propose the Class-Wise Adaptive Exploration and Exploitation (CWAEE) method for semi-supervised domain generalization when known and unknown classes are mixed in unlabeled data. The intuition is to first explore unlabeled training data by detecting known and unknown classes , and then exploit them in different ways to improve the generalization of the model. In particular, we use one-vs-rest classifiers and class-wise adaptive thresholds to detect known and unknown classes in unlabeled training data, and adopt consistency regularization on augmented samples based on Fourier Transformation to improve the unseen domain generalization. The experiments on real-world datasets verify that our CWAEE method can achieve better performance than compared methods.

2 Related Work

Domain Generalization (DG) aims to help the model generalize to an unseen target domain by utilizing one or several related domains. Most existing DG methods can be classified into three different categories: domain-invariant representation learning, meta-learning and data augmentation. Motivated by the learning theory of domain adaptation [18], domain-invariant representation learning methods attempt to learn a representation space where the discrepancy between different source domains is as small as possible. For example, Muandet et al. [11] proposed a kernel-based optimization algorithm for Support Vector Machine that learns an invariant transformation by minimizing the distribution discrepancy across domains. After the domain-adversarial neural network was proposed for domain adaptation [8], domain-adversarial training is widely used for DG [19, 20]. Following Model-Agnostic Meta-Learning (MAML) [21], Li et al. [22] applied meta-learning strategy to DG through dividing the training data from multiple source domains into meta-train and meta-test sets to simulate domain shifts. The meta-optimization objective was designed to minimize the loss not only on meta-train domains but also on the meta-test domain. Shu et al. [23] conducted meta-learning over augmented domains to learn open-domain generalizable representations for Open Domain Generalization (OpenDG) problem. Compared to domain-invariant representation learning and metalearning, data augmentation methods have shown better performance on various DG benchmarks recently. Zhou et al. [24] mixed feature statistics extracted by bottom layers of Convolutional Neural Networks (CNNs) between instances of different domains to synthesize instances in novel domains for training. Xu et al. [25] applied MixUp [26] in amplitude spectrums of two images to augment training data while preserving high-level semantics of the original images in phase spectrums. To reduce the labeling cost of DG, Semi-Supervised Domain Generalization (SSDG) [14, 15, 16, 17] receives much attention recently. Sharifi-Noghabi et al. [14] proposed the first SSDG method which combines the pseudo-label method and meta-learning strategy. Zhou et al. [16] adopted stochastic classifier and multi-view consistency to improve the quality of pseudo-labels and generalization of model respectively. However, they assumed that both unlabeled training and testing data have the same label space with labeled training data while we consider a more realistic scenario that the label space of labeled data is a subset of unlabeled training and testing data, i.e., both unlabeled training and testing data are a mixture of known and unknown classes .

Out-of-Distribution (OOD) detection aims to detect test samples whose labels are not contained in the label space of training data [27]. Hendrycks and Gimpel [28] proposed the first effective baseline that uses the maximum softmax probability to identify OOD samples. Post-hoc methods attract much attention because they can be implemented without modifying the training procedure and objective. For example, Liang et al. [29] used temperature scaling and input perturbations to make In-Distribution (ID) and OOD samples more separable. Liu et al. [30] proposed to replace softmax

confidence scores with energy scores to detect OOD samples, and considered samples with higher energy to be OOD samples. Hendrycks et al. [31] maximized the entropy of the model's predictions on enormous unlabeled OOD samples to improve OOD detection. Yang et al. [32] proposed the Unsupervised Dual Grouping (UDG) method that adopts two classification heads to dynamically separate ID and OOD samples in the unlabeled data and optimizes the model with different learning objectives. Since the commonly-used OOD detection benchmarks [28, 29, 31] are based on dataset splitting setup, i.e., one dataset as ID and all the others as OOD, the existing OOD detection methods overfit to low-level dataset statistics instead of learning semantics, which leads to that even the same class from another dataset will be detected as OOD samples [32].

Semi-Supervised Learning (SSL) is an active research area in the last decade, whose goal is to leverage both labeled and unlabeled data to improve the performance of models and reduce the label cost. The most popular methods include consistency regularization [33, 34] and entropy minimization [35, 36]. The basic idea behind consistency regularization is that the model's outputs of similar samples should be similar. Sajjadi et al. [33] proposed the Π model that minimizes consistency loss over two random augmentations of one sample to regularize the model. Tarvainen et al. [34] proposed the Mean Teacher method that averages Student model weights using Exponential Moving Average (EMA) over training steps as Teacher model to provide more stable targets for the consistency loss optimization of the Student model. Entropy minimization methods encourage models to make low entropy predictions on unlabeled data, so that the learned decision boundary passes through a low-density region rather than a high-density area. Lee [36] proposed the Pseudo-label method that sets the maximum confidence prediction of unlabeled samples as pseudo-labels, and then trains the model in a supervised way with labeled and pseudo-labeled data. Sohn et al. [37] proposed the FixMatch method that combines consistency regularization and entropy minimization. It generates highly confident pseudo-labels on weakly-augmented unlabeled samples, and trains the model to predict the pseudo-labels for the strongly-augmented version of the same samples. OpenMatch [38] extended FixMatch with OVANet [39] and open-set soft-consistency regularization to detect outliers in unlabeled data for Open-set Semi-Supervised Learning (OSSL).

3 Method

We have multiple source domains D 1 , D 2 , . . . , D s available for training, and each D k ∈{ 1 , 2 ,...,s } has one labeled set D l k = { ( x ( k ) i , y ( k ) i ) } n l k i =1 and one unlabeled set D u k = { ( u ( k ) i ) } n u k i =1 , i.e., D k = D l k ∪D u k . Generally, the size of D u k may be much larger than that of D l k , i.e., n u k ≫ n l k . The total size of labeled dataset is denoted as n l and the total size of unlabeled dataset is denoted as n u , i.e., n l = ∑ s k =1 n l k , and n u = ∑ s k =1 n u k . The model is evaluated on an unseen target domain D t = { ( x ( t ) i , y ( t ) i ) } n t i =1 . C l is the label set of labeled data D l = D l 1 ∪ D l 2 ∪ · · · ∪ D l s , C u is the label set of unlabeled data D u = D u 1 ∪ D u 2 ∪ · · · ∪ D u s , and C t is the label set of target domain D t . Due to the existence of unknown classes in unlabeled data and target domain, we have C l ⊂ C u ⊂ C t . The classes in C l are called known classes , the classes in C u \C l are called seen unknown classes since they are seen in unlabeled data during the training process and the classes in C t \C u are called unseen unknown classes since they are unseen during the training process. The goal is to learn the model f ∗ that generalizes well on target domain D t for both known and unknown classes , i.e., for each ( x ( t ) i , y ( t ) i ) ∈ D t , if y ( t ) i ∈ C l , then f ∗ ( x ( t ) i ) = y ( t ) i , else f ∗ ( x ( t ) i ) = unknown .

To effectively utilize unlabeled data to improve the generalization and robustness of the model, we should carefully exploit known and unknown classes in them. If known classes are treated as unknown ones, the generalization of the model may decrease significantly; if unknown classes are treated as known ones, the robustness of the model may degrade.

3.1 Detecting Known and Unknown Classes

In order to explore unlabeled data and detect known and unknown classes in them, we rely on the outputs of the deep neural networks and the intuition is that known classes will have higher scores than unknown classes [28]. The deep neural network model f Θ = h ω ◦ g θ contains two parts, a feature extractor g θ and a |C l | -way linear classifier h ω = [ h 1 ω ; . . . ; h |C l | ω ] . The model maps each sample x i into a |C l | -dimensional logits z i = h ω ◦ g θ ( x i ) , and then feeds z i into a softmax function

Procedure 1 Calculating Thresholds

Input : Score queues q 1: |C l | , feature extractor g θ , linear classifiers h 1: |C l | ω , validation dataset D v , number of known classes |C l | .

Output : Temperatures τ 1: |C l | , thresholds δ 1: |C l | knw and δ 1: |C l | unk .

1: for c = 1 to |C l | do

2: Calculate temperature τ c to calibrate g θ , h c ω on D v with Eq.1;

4: Calculate known and unknown classes thresholds δ c knw = α c 1 α c 1 + β c 1 , δ c unk = α c 2 α c 2 + β c 2 .

3: Fit a two-component beta mixture model α c 1:2 , β c 1:2 on queue q c with EM algorithm;

5: end for

̸

S to produce the estimated posterior probability ˆ p i ( a.k.a. score) of each class, i.e., ˆ p i = S ( f Θ ( x i )) . The training is to minimize the cross-entropy loss L CE (Θ) between ˆ p i and one-hot label y i over labeled data, i.e., Θ ∗ = argmin Θ L CE (Θ) = argmin Θ -1 n l ∑ n l i =1 ∑ |C l | c =1 y c i log ˆ p c i , where n l is the size of labeled data. The gradient of cross-entropy loss w.r.t. linear classifier h ω corresponding to class c is ∂ L CE (Θ) ∂h c ω = 1 n l ∑ n l i =1 (ˆ p c i -y c i ) g θ ( x i ) . Due to the relative normalization of the softmax function, the score of x i on class c , i.e., ˆ p c i , not only depends on the logit z c i , but also depends on other logits z j i , j = c . Thus, the update of h c ω is affected by the predictions of other classifiers h j ω , j = c . When the logit of x i on target class y i is large, the scores of x i on non-target class, i.e., ˆ p c i , c = y i , will be small. Since the ˆ p c i -y c i is small, the gradient of the non-target classifiers may converge to zero. The parameters of the neural network are updated with the stochastic gradient descent (SGD) algorithm, so, the logits of samples on non-target classifiers induced by h j ω , j = c will not decrease.

̸

̸

In order to update the logits on non-target classifiers induced by h j ω , j = c , we replace the softmax function S with the sigmoid function σ . In this way, we obtain an individual score on each way to indicate the probability that one sample belongs to each class, i.e., ˆ p c i = σ ( z c i ) , c ∈ C l , and we get |C l | one-vs-rest classifiers. With these one-vs-rest classifiers, we can detect known and unknown classes. A straightforward way is to set a fixed threshold δ = 0 . 5 [39, 38]. Let j = argmax c (ˆ p c i ) , if ˆ p j i ≥ δ , it means that it belongs to class j ; if ˆ p j i < δ , it means that it is an unknown classes sample. Due to the overconfidence of modern neural networks [40], it is not proper to use such a fixed threshold of 0 . 5 for all classes. A more reasonable way is to use the class-wise adaptive threshold [41, 42] for each class. To get well-calibrated scores, we first calibrate the classifiers on validation data D v with temperature scaling [40]:

̸

where I ( · ) is an indicator function and τ c is the temperature for class c ∈ C l . Note that the validation dataset D v only contains samples of known classes from source domains. And then, we use a two-component beta mixture model to model the score distributions of known classes and unknown classes in an unsupervised way, since it is a more flexible approximator than the Gaussian mixture model [43]. The adaptive thresholds for known classes δ j knw (the large one) and unknown classes δ j unk (the small one) can be set as the mean values of two fitted beta distributions. Thus, for an unlabeled sample u i ∈ D u , let ˜ p j i = σ ( z j i /τ j ) be the scaled score, if ˜ p j i > δ j knw , we consider it belongs to known class j , i.e. ˆ y i = j ; if ˜ p j i < δ j unk , we consider it belongs to unknown classes , i.e. ˆ y i = unknown ; if δ j unk ≤ ˜ p j i ≤ δ j knw , it is difficult to determine its label, hence we set its prediction as null , i.e. ˆ y i = null . Since the model is updated in a mini-batch way, the scores predicted by the model many steps ago are less useful for modeling the score distributions. Hence, we utilize the most recent scores predicted by the model for the fitting of the beta mixture model. In particular, for each class j we maintain a queue q j with fixed length to record the most recent scaled maximum score of unlabeled sample u i if its maximum score belongs to class j , i.e., argmax c (˜ p c i ) = j . The overall procedure is summarized in Procedure 1.

̸

̸

3.2 Improving Target Domain Generalization

For labeled training data, we use the following supervised loss,

̸

For unlabeled training data, we should exploit them carefully to improve the unseen target domain generalization after having detected known and unknown classes. We construct the weakly and strongly augmented version of each sample following FixMatch [37], and calculate the model's scaled confidence score on its weakly augmented version, i.e., ˜ p c i,weak = σ ( f Θ ( T weak ( u i )) /τ c ) , c ∈ C l . Then, we assign pseudo-label ˆ y i to it according to Section 3.1, and force the model's prediction on the strongly augmented version ˆ p c i,strong = σ ( f Θ ( T strong ( u i ))) , c ∈ C l to match the pseudo-label. For unlabeled training data predicted as known classes (i.e., ˆ y i ∈ C l ), the loss is defined as

̸

where n u knw is the number of samples predicted as known classes . For unlabeled training data predicted as unknown classes (i.e., ˆ y i = unknown ), the loss is defined as

where n u unk is the number of samples predicted as unknown classes .

Besides known and unknown classes , there are some unlabeled samples that are predicted as null . In order to fully exploit these data and improve the generalization of the model on the unseen target domain, we adopt Fourier Transformation to disentangle the semantics and styles of the sample [25, 44], and then MixUp [26] the styles of two randomly sampled samples to augment training data for regularizing the model. Specifically, for each x , its Fourier Transformation F ( x ) is formulated as:

where H,W are the height and width of the sample, and J 2 = -1 . The amplitude component A ( x ) and phase component P ( x ) are then respectively expressed as

where R ( x ) and I ( x ) represent the real and imaginary part of F ( x ) . It is well-known that, the phase component of the Fourier spectrum preserves high-level semantics of the original signal, while the amplitude component contains low-level statistics. Therefore, we could mix up amplitude components of two samples of the same batch while keeping the phase components to generate the augmented samples ˜ x [25, 44], i.e.,

where F -1 ( x ) is the inverse Fourier Transformation, λ ∼ U (0 , 1) , and U is the uniform distribution.

To avoid overfitting on domain-related low-level statistics, we minimize the consistency regularization loss between the original unlabeled samples and the augmented ones to push the model to pay attention to the high-level semantics of the samples, defined as

where n u is the number of unlabeled data. The overall loss of the training process is formulated as:

where λ 1 , λ 2 and λ 3 are hyper-parameters to balance each loss. During the warm-up process, the model is trained only with labeled data, i.e., λ 1 = λ 2 = λ 3 = 0 . The whole process is summarized in Algorithm 1 (the framework figure can be found in Appendix A of the supplementary material), where the domain index is omitted for brevity.

Algorithm 1 Class-Wise Adaptive Exploration and Exploitation (CWAEE)

Input : Labeled dataset D l , unlabeled dataset D u , validation dataset D v , training epoch T , number of known classes |C l | . Output : Feature extractor g θ , linear classifiers h 1: |C l | ω . 1: Initialize g θ , h 1: |C l | ω , score queues q 1: |C l | , temperatures τ 1: |C l | , thresholds δ 1: |C l | knw , δ 1: |C l | unk ; 2: for t = 1 to T do 3: for i = 1 to max _ iteration do 4: Draw a batch of labeled samples B l and unlabeled samples B u from D l and D u ; 5: Calculate loss L l on B l with Eq.2; 6: Predict on B u with τ 1: |C l | to get scaled confidence scores ˜ P u ; 7: Split B u into known classes B u knw , unknown classes B u unk and null B u null with ˜ P u and δ 1: |C l | knw , δ 1: |C l | unk ; 8: Calculate loss L u knw and L u unk on B u knw and B u unk with Eqs.3 and 4 respectively; 9: Conduct data augmentation within B u to get ˜ B u with Eqs.5-7; 10: Calculate loss L u con on B u and ˜ B u with Eq.8 and get total loss L with Eq.9; 11: Backward loss L and update g θ , h 1: |C l | ω with SGD; 12: Update q 1: |C l | with ˜ P u ; 13: Update τ 1: |C l | , δ 1: |C l | knw and δ 1: |C l | unk with Procedure 1. 14: end for 15: end for

4 Experiment

4.1 Datasets

We use PACS [45], Office-Home [46] and miniDomainNet [47] datasets in the experiments. PACS [45] consists of four domains corresponding to four different image styles, including Photo (P), Art painting (A), Cartoon (C) and Sketch (S), and the four domains have the same label set of 7 classes, and contain 9,991 images in total; OfficeHome [46] consists of images from four different domains: Artistic (A), Clipart (C), Product (P) and Real-World (R), and has a large domain gap and around 15,500 images of 65 classes; miniDomainNet [47] is a subset of DomainNet [48] and has four domains including 18,703 images of Clipart (C), 31,202 images of Painting (P), 65,609 images of Real (R) and 24,492 images of Sketch (S), and it has 126 classes and maintains the complexity of the original DomainNet.

We adopt the common leave-one-domain-out protocol [12, 16]: three domains are used as the source domains and the remaining one as the target domain. Similar to [23, 38] we split the classes into known and unknown classes , specifically we split the original label set into 3:2:2, 25:20:20 and 42:42:42 ( known classes , seen unknown classes and unseen unknown classes ) in PACS [45], OfficeHome [46] and miniDomainNet [47] respectively in alphabetical order of the class name. On each source domain, 10 labeled samples of each known class are randomly sampled to construct the labeled data, and the remaining samples of known classes and seen unknown classes construct the unlabeled data. All samples on the target domain are used for evaluation.

4.2 Compared Methods

We compare our method with various DG methods, OOD detection methods and SSL methods. DeepAll naively puts labeled data from all source domains together, and trains the model with Empirical Risk Minimization (ERM); DAML [23] conducts meta-learning over augmented domains to learn open-domain generalizable representations for OpenDG problem; UDG [32] adopts two classification heads to dynamically separate ID and OOD samples in the unlabeled data and optimizes the model with different learning objectives; FixMatch [37] generates highly confident pseudo-labels on weakly-augmented unlabeled samples, and trains the model to predict the pseudo-labels for the strongly-augmented version of the same samples; OpenMatch [38] extends FixMatch with OVANet [39] and open-set soft-consistency regularization to detect outliers in unlabeled data for OSSL;

StyleMatch [16] extends FixMatch with stochastic classifier and multi-view consistency to improve the quality of pseudo-labels and generalization of model respectively for SSDG.

4.3 Setup

Most hyper-parameters follow the setting in [16] for a fair comparison. The ImageNet-pretrained ResNet18 [2] is used as the CNN backbone, and the linear classifiers is implemented with stochastic classifiers. The initial learning rate of SGD optimizer is set to 0.003 for the pretrained backbone and 0.01 for the randomly initialized stochastic classifier, both decaying following the cosine annealing rule. The running epochs are 40, 20 and 20 for PACS, OfficeHome and miniDomainNet respectively. For each mini-batch, we

randomly sample 16 labeled samples and 16 unlabeled samples from each source domain. We set λ 1 = 1 . 0 on PACS, λ 1 = 0 . 4 on OfficeHome and λ 1 = 0 . 1 on miniDomainNet. We set λ 2 = 0 . 4 and λ 3 = 1 . 0 for all three datasets. We evaluate the accuracy on known classes and AUROC on unknown classes of the methods with 3 different random seeds, and report the average results. More implementation details can be found in Appendix B of the supplementary material.

4.4 Results

The results are summarized in Table 1 (the results with standard deviations can be found in Tables 6 and 7 in Appendix C of the supplementary material). From the results, it can be found that our method consistently outperforms the compared methods on all datasets. Although the UDG and DAML methods are developed for unknown class detection on the unseen target domain, their performance is much worse than ours. FixMatch has poor classification performance when evaluated on an unseen target domain, due to not considering the domain shifts problem. The performance of our method is much better than that of OpenMatch in most cases since it depends on the fixed thresholds to detect unknown classes in unlabeled data. StyleMatch has a worse performance than ours since it treats all unlabeled samples as one of the known classes. In order to evaluate the detection performance of methods on seen and unseen unknown classes respectively, we also report the AUROC of seen and unseen unknown classes on the target domain in Table 2 (the results with standard deviations can be found in Table 8 in Appendix C of the supplementary material). It can be found that our method has better detection performance on both seen and unseen unknown classes .

4.5 Ablation Study

In order to provide additional insight into what makes our method successful, we conduct ablation experiments on OfficeHome.

4.5.1 Effectiveness of Used Modules

Our method contains several modules, including supervised loss (Eq.2), unsupervised loss on known classes (Eq.3), unsupervised loss on unknown classes (Eq.4), class-wise adaptive thresholds (Procedure 1) and consistency regularization loss (Eq.8). The results with standard deviations are presented in Table 3. With the supervised loss, our

method trains the model only with labeled samples. After detecting known and unknown classes with the fixed thresholds (i.e., δ 1: |C l | knw = δ 1: |C l | unk = 0 . 5 ) and trained with the unsupervised loss on known and unknown classes , the performance can be improved with these detected samples. By replacing the fixed thresholds with class-wise adaptive thresholds, the performance can be further boosted. With all the used modules, we can get the best performance.

To better show the effectiveness of the classwise adaptive thresholds, we compare the quality of pseudo-labels when trained with fixed thresholds and the class-wise adaptive thresholds. The results are depicted in Figure 2. It can be found that the accuracy of pseudo-labels assigned by our method is higher than that of compared method on known classes and unknown classes in all 4 domains, which means that our method has better performance on known classes classification and unknown classes detection.

4.5.2 Sensitivity to Number of Seen Unknown Classes

In order to study the sensitivity of our method to the number of seen unknown classes , we compare our method with existing SSL methods under different numbers of seen unknown classes ( |C u \C l | varies from 0 to 30), and the results are depicted in Figure 3. It can be found that our

0

5

10

15

20

Epoch

75

80

85

90

95

Fixed (A)

Fixed (C)

Fixed (P)

Fixed (R)

Ours (A)

Ours (C)

Ours (P)

Ours (R)

(a) Known classes

0

5

10

15

20

Epoch

20

40

60

80

Fixed (A)

Fixed (C)

Fixed (P)

Fixed (R)

Ours (A)

Ours (C)

Ours (P)

Ours (R)

(b) Unknown classes

0

5

10

15

20

25

30

Number of seen unknown classes

60

64

68

72

76

DeepAll

FixMatch

OpenMatch

StyleMatch

Ours

(a) Accuracy

0

5

10

15

20

25

30

Number of seen unknown classes

60

64

68

72

76

AUROC

DeepAll

FixMatch

OpenMatch

StyleMatch

Ours

(b) AUROC

method outperforms the compared methods. With the increase in the number of seen unknown classes , the AUROC on unknown classes obtained by our method increases while the accuracy on known classes is maintained.

4.5.3 Sensitivity to Number of Labeled Samples

In order to study the sensitivity of our method to the number of labeled samples, we compare our method with existing methods under different numbers of labeled samples (5, 10 and 20 labels of each known class on each domain) and the results are summarized in Table 4 (the results with standard deviations can be found in Table 9 in Appendix C of the supplementary material). It can be found that our method outperforms compared methods under different numbers of labeled samples. The performance of

our method under extremely few labels (i.e., 5 labels of each class on each domain) is promising, since it can exploit known and unknown classes in unlabeled data in a more reasonable way.

4.5.4 Histograms of Confidence Scores

The empirical probability density function of confidence scores of unlabeled data in different queues and different epochs are depicted in Figure 4. Here, we report class 0 and class 1 in different epochs as examples, i.e., Figure 4a vs 4b, Figure 4c vs 4d, which shows that the confidence scores have different distributions in the different training epochs. It can be found that our method CWAEE can find the adaptive thresholds for known and unknown classes .

5 Conclusion

In this paper, we focus on a more realistic semi-supervised domain generalization scenario, where known classes and seen unknown classes are mixed in unlabeled training data while known classes , seen unknown classes and unseen unknown classes are mixed in testing data. In order to utilize unlabeled training data, we detect known and unknown classes in them with the class-wise adaptive thresholds based on one-vs-rest classifiers. We adopt consistency regularization on augmented training samples based on Fourier Transformation to improve the generalization on the unseen target domain. The experimental results conducted on different real-world datasets show that our method outperforms the existing state-of-the-art methods.

Broader Impacts

Our work provides an effective method for semi-supervised domain generalization with known and unknown classes . We believe our work will be beneficial for domain generalization projects with few labeled data and many unlabeled data, and does not have negative societal impacts.

0.0

0.2

0.4

0.6

0.8

1.0

Score

0

4

8

12

16

Empirical p.d.f

Unknown classes threshold

Known classes threshold

Unknown classes

Known classes

(a) Class 0 in epoch 1

0.0

0.2

0.4

0.6

0.8

1.0

Score

0

4

8

12

16

Empirical p.d.f

Unknown classes threshold

Known classes threshold

Unknown classes

Known classes

(b) Class 0 in epoch 20

0.0

0.2

0.4

0.6

0.8

1.0

Score

0

4

8

12

16

Empirical p.d.f

Unknown classes threshold

Known classes threshold

Unknown classes

Known classes

(c) Class 1 in epoch 1

0.0

0.2

0.4

0.6

0.8

1.0

Score

0

4

8

12

16

Empirical p.d.f

Unknown classes threshold

Known classes threshold

Unknown classes

Known classes

(d) Class 1 in epoch 20

Acknowledgments

This work is supported by the National Science Foundation of China (62276125, 61921006), the Fundamental Research Funds for the Central Universities (022114380013), and the Collaborative Innovation Center of Novel Software Technology and Industrialization.

References

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25 (NIPS'12) , pages 1106-1114, 2012.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'16) , pages 770-778, 2016.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 , 2019.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33 (NeurIPS'20) , pages 1877-1901, 2020.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet Classifiers Generalize to ImageNet? In Proceedings of the 36th International Conference on Machine Learning (ICML'19) , pages 5389-5400, 2019.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural Adversarial Examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'21) , pages 15262-15271, 2021.

Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting Visual Category Models to New Domains. In Proceedings of the 11th European Conference on Computer Vision (ECCV'10) , pages 213-226, 2010.

Yaroslav Ganin and Victor Lempitsky. Unsupervised Domain Adaptation by Backpropagation. In Proceedings of the 32nd International Conference on Machine Learning (ICML'15) , pages 1180-1189, 2015.

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning Transferable Features with Deep Adaptation Networks. In Proceedings of the 32nd International Conference on Machine Learning (ICML'15) , pages 97-105, 2015.

Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from Several Related Classification Tasks to a New Unlabeled Sample. In Advances in Neural Information Processing Systems 24 (NIPS'11) , pages 2178-2186, 2011.

Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain Generalization via Invariant Feature Representation. In Proceedings of the 30th International Conference on Machine Learning (ICML'13) , pages 10-18, 2013.

Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to Unseen Domains: A Survey on Domain Generalization. IEEE Transactions on Knowledge and Data Engineering , 2022.

Wei Wang and Zhi-Hua Zhou. On Multi-View Active Learning and the Combination with Semi-Supervised Learning. In Proceedings of the 25th International Conference on Machine Learning (ICML'08) , pages 1152-1159, 2008.

Hossein Sharifi-Noghabi, Hossein Asghari, Nazanin Mehrasa, and Martin Ester. Domain Generalization via Semi-supervised Meta Learning. arXiv:2009.12658 , 2020.

Xiao Liu, Spyridon Thermos, Alison O'Neil, and Sotirios A. Tsaftaris. Semi-supervised Meta-learning with Disentanglement for Domain-Generalised Medical Image Segmentation. In Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI'21) , pages 307-317, 2021.

Kaiyang Zhou, Chen Change Loy, and Ziwei Liu. Semi-Supervised Domain Generalization with Stochastic StyleMatch. In NeurIPS Workshop on Distribution Shifts Connecting Methods and Applications , 2021.

Huifeng Yao, Xiaowei Hu, and Xiaomeng Li. Enhancing Pseudo Label Quality for Semisupervised Domain-Generalized Medical Image Segmentation. In Proceedings of the 36th AAAI Conference on Artificial Intelligence (AAAI'22) , pages 3099-3107, 2022.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A Theory of Learning from Different Domains. Machine Learning , 79(1):151-175, 2010.

Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain Generalization with Adversarial Feature Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'18) , pages 5400-5409, 2018.

Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep Domain Generalization via Conditional Invariant Adversarial Networks. In Proceedings of the 15th European Conference on Computer Vision (ECCV'18) , pages 647-663, 2018.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML'17) , pages 1126-1135, 2017.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to Generalize: MetaLearning for Domain Generalization. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI'18) , pages 3490-3497, 2018.

Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open Domain Generalization with Domain-Augmented Meta-Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'21) , pages 9624-9633, 2021.

Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain Generalization with MixStyle. In Proceedings of the 9th International Conference on Learning Representations (ICLR'21) , 2021.

Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A Fourier-Based Framework for Domain Generalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'21) , pages 14383-14392, 2021.

Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond Empirical Risk Minimization. In Proceedings of the 6th International Conference on Learning Representations (ICLR'18) , 2018.

Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized Out-of-Distribution Detection: A Survey. arXiv:2110.11334 , 2021.

Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Out-ofDistribution Examples in Neural Networks. In Proceedings of the 5th International Conference on Learning Representations (ICLR'17) , 2017.

Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. In Proceedings of the 6th International Conference on Learning Representations (ICLR'18) , 2018.

Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based Out-of-distribution Detection. In Advances in Neural Information Processing Systems 33 (NeurIPS'20) , pages 21464-21475, 2020.

Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep Anomaly Detection with Outlier Exposure. In Proceedings of the 7th International Conference on Learning Representations (ICLR'19) , 2019.

Jingkang Yang, Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically Coherent Out-of-Distribution Detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'21) , pages 8301-8309, 2021.

Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning. In Advances in Neural Information Processing Systems 29 (NIPS'16) , 2016.

Antti Tarvainen and Harri Valpola. Mean Teachers are Better Role Models: Weight-averaged Consistency Targets Improve Semi-supervised Deep Learning Results. In Advances in Neural Information Processing Systems 30 (NIPS'17) , pages 1195-1204, 2017.

Yves Grandvalet and Yoshua Bengio. Semi-supervised Learning by Entropy Minimization. In Advances in Neural Information Processing Systems 17 (NIPS'04) , pages 529-536, 2004.

Dong-Hyun Lee. Pseudo-label: The Simple and Efficient Semi-supervised Learning Method for Deep Neural Networks. In Workshop on challenges in representation learning of ICML , 2013.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying SemiSupervised Learning with Consistency and Confidence. In Advances in Neural Information Processing Systems 33(NeurIPS'20) , 2020.

Kuniaki Saito, Donghyun Kim, and Kate Saenko. OpenMatch: Open-Set Semi-supervised Learning with Open-set Consistency Regularization. In Advances in Neural Information Processing Systems 34 (NeurIPS'21) , pages 25956-25967, 2021.

Kuniaki Saito and Kate Saenko. OVANet: One-vs-All Network for Universal Domain Adaptation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'21) , pages 9000-9009, 2021.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML'17) , pages 1321-1330, 2017.

Xian-Jin Gui, Wei Wang, and Zhang-Hao Tian. Towards Understanding Deep Learning from Noisy Labels with Small-Loss Criterion. In Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI'21) , pages 2469-2475, 2021.

Bowen Zhang, Yidong Wang, Wenxin Hou, HAO WU, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling. In Advances in Neural Information Processing Systems 34 (NeurIPS'21) , pages 18408-18419, 2021.

Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Unsupervised Label Noise Modeling and Loss Correction. In Proceedings of the 36th International Conference on Machine Learning (ICML'19) , pages 312-321, 2019.

Yanchao Yang and Stefano Soatto. FDA: Fourier Domain Adaptation for Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'20) , 2020.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, Broader and Artier Domain Generalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'17) , pages 5543-5551, 2017.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep Hashing Network for Unsupervised Domain Adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'17) , pages 5385-5394, 2017.

Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain Adaptive Ensemble Learning. IEEE Transactions on Image Processing , 30:8008-8018, 2021.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment Matching for Multi-Source Domain Adaptation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'19) , 2019.

Peer reviews:
comments: SUMMARY:
This paper introduces a new methodology for Semi-Supervised Domain Generalization (SSDG). This paper proposes the Class-Wise Adaptive Exploration and Exploitation (CWAEE) method, which contains one-vs-rest classifiers, class-wise adaptive thresholds, and consistency regularization based on Fourier Transformation. This algorithm shows the improvement in their SSDG settings.

STRENGTHS:
Pros.
- This paper is well-written, motivated, and easy to follow.
- This paper proposes several methods in which class-wise adaptive thresholds and consistency regularization based on Fourier Transformation are very interesting works. (If the overview image for consistency regularization in the appendix is added in the main paper, it would be much easy to understand the concept. It is very novel and interesting. ) 
- This paper shows extensive experiments and shows the effectiveness of this method.

WEAKNESSES:
Cons.
- (Major) This paper didn’t follow the previous training pipeline for the baseline. In FixMatch, it trains the model 2^20 iterations. For Cifar10/Cifar100, it is about 10k epochs. With only 80 epochs, I would say it is not the original other baseline numbers. 
- (Major) This paper uses Imagenet pre-trained model. It means that unlabeled data could be trained in the pre-training phase.
- (Major) This paper evaluates their algorithm in only their setting. I think it can be evaluated in the same OOD setting in OpenMatch. (known/unknown in cifar10/cifar100 or Imagenet) If this paper could show the improvement in this setting, it would be more solid results. 
- (Minor) The one-vs-rest classifier is very similar to One-vs-All Outlier Detector in OpenMatch. (Novelty)

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
Please see the weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
This paper focuses on the realistic scenario and proposes a semi-supervised domain generalization method. The method first explores unlabeled data by detecting known and unknown classes, and then exploits the data by adopting consistency regularization based on Fourier Transformation. The experiments show the effectiveness and superiority of the proposed method.

STRENGTHS:
This paper is well-written and organized. It is easy to follow.
This paper focuses on the realistic scenario where the known classes are mixed with unknown classes in the data, and proposes a method which outperforms the state-of-the-art semi-supervised domain generalization methods.
This paper uses one-vs-rest classifiers and class-wise adaptive thresholds, which is helpful to detecting unknown classes.

WEAKNESSES:
The method needs to train one-vs-rest classifiers. When the number of classes is large, the computation cost is high.
The calibration needs the validation data.

QUESTIONS:
The method uses the class-wise adaptive thresholds to detect unknown classes. Why do these thresholds work intuitively?
There are hyper-parameters in Eq(9), is the performance sensitive to the hyper-parameters?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper considers the setting of semi-supervised domain generalization (SSDG) when both unlabeled source and target domains can contain unknown classes, i.e. not seen as labeled instances in source domains. The goal here is to learn a classifier which will be able to (i) reliably distinguish seen classes from unknown classes and (ii) effectively utilize unlabeled source data to learn domain generalizable representations. To approach the aforementioned goals authors suggest using one-vs-rest classifiers and class adaptive thresholds to distinguish between known and unknown classes. And then they apply different optimization objectives for known/unknown classes to improve learning domain generalizable representations. The proposed methodology leads to improvements over considered baselines on the standard benchmarks.\n\nSTRENGTHS:\n-  The proposed problem formulation is important\n\nWEAKNESSES:\n-  Though the proposed setting is indeed realistic, the experimental setup is limited. It would be helpful to consider WILDS [1, 2] or other competitive benchmarks to understand the applicability of the proposed methodology.\n\n- The proposed methodology looks to me like combination of many well-known techniques without thorough motivation and deep analysis. \n\n[1] Pang Wei Koh et al' WILDS: A Benchmark of in-the-Wild Distribution Shifts. ICML 2021\n\n[2] Shiori Sagawa et al' Extending the WILDS Benchmark for Unsupervised Adaptation. https://wilds.stanford.edu\n\nQUESTIONS:\n- How ImageNet pretraining affects the performance of the proposed method? What would be the performance if the method is trained from scratch?\n\nLIMITATIONS:\nAuthors do not discuss limitations. One possible limitation that I see is that the proposed setting decides not to distinguish between different unknown classes. It would be interesting to push the limits of semi-supervised learning and study the ultimate setting when both domain/class shift are presented and the classifier is able to distinguish between different unknown classes. Some related papers include but not limited to [1, 2]\n\n[1] Kaidi Cao, Maria Brbic, Jure Lescovec. Open-World Semi-Supervised Learning. ICLR 2022\n\n[2] Sagar Vaze, Kai Han, Andrea Vedaldi, Andrew Zisserman. Generalized Category Discovery. CVPR 2022", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper considers the realistic semi-supervised domain generalization setting where known classes are mixed with some unknown classes in the unlabeled training and testing data, and proposes the Class-Wise Adaptive Exploration and Exploitation (CWAEE) method. The experiments conducted on the datasets show the advantages over the previous baselines.

STRENGTHS:
1. Most previous semi-supervised domain generalization methods assumed that there is no unknown class in the unlabeled training data and testing data. The paper relaxes this assumption and considers a more challenging setting. The setting is new and important. 

2. The paper proposes a two-step method. It first uses one-vs-rest classifiers and class-wise thresholds to detect the known and unknown classes, and then uses Fourier Transform and data augmentation to improve the generalization on target domain and unknown classes. 

3. The method performs well on the datasets and the ablation study is sufficient to support the results.

WEAKNESSES:
1. Fourier Transform based on data augmentation may not work for non-image tasks. 

2. Some techniques are similar to the well-known method FixMatch.

QUESTIONS:
My concerns for this paper are as follows:
1. Why does the method use the two-component beta mixture model to calculate the thresholds? 

2. Why do the known classes have higher scores than the unknown classes? I think the authors should discuss more about this. 

3. Will the number of unknown classes influence the performance? For example, the number of unknown classes is far larger than that of known classes in the unlabeled training data.

LIMITATIONS:
no negative societal impacts
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06