Semi-Supervised Domain Generalization with Known and Unknown Classes
Authors: Lei Zhang
Ji-Fu Li
Wei Wang
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Domain Generalization
Semi-Supervised Learning
Out-of-Distribution Detection
Deep Learning

Abstract: Semi-Supervised Domain Generalization (SSDG) aims to learn a model that is generalizable to an unseen target domain with only a few labels, and most existing SSDG methods assume that unlabeled training and testing samples are all known classes. However, a more realistic scenario is that known classes may be mixed with some unknown classes in unlabeled training and testing data. To deal with such a scenario, we propose the Class-Wise Adaptive Exploration and Exploitation (CWAEE) method. In particular, we explore unlabeled training data by using one-vs-rest classifiers and class-wise adaptive thresholds to detect known and unknown classes, and exploit them by adopting consistency regularization on augmented samples based on Fourier Transformation to improve the unseen domain generalization. The experiments conducted on real-world datasets verify the effectiveness and superiority of our method.

Semi-Supervised Domain Generalization with Known and Unknown Classes
Lei Zhang, Ji-Fu Li, Wei Wang"
National Key Laboratory for Novel Software Technology. Nanjing University, China School Artificial Intelligence, Nanjing University, China {zhang], lliff, wangw) (aangw@@lamdil @lamda. nju. edu. nju.edu.cc
Abstract
Semi-Supervised Domain Generalization (SSDG) aims to learn a model that is generalizable to unseen target domain with only few labels, and most ex- isting SSDG methods assume that unlabeled training and testing samples are all known classes. However, more realistic scenario is that known classes may be mixed with some unknown classes in unlabeled training and testing data. To deal with such scenario, we propose the Class-Wise Adaptive Exploration and Exploitation (CWAEE) method. In particular, we explore unlabeled training data by using one-vs-rest classifiers and class-wise adaptive thresholds detect known and unknown classes, and exploit them by adopting consistency regularization on augmented samples based on Fourier Transformation to improve the unseen domain generalization. The experiments conducted on real-world datasets verify the effectiveness and superiority of our method.
Introduction
The machine learning community has witnessed the great progress of deep learning models and their applications, e.g., computer vision [1.21, natural language processing 3. Such huge success mostly based on the i.i.d. assumption, i.c., the training data and testing data are identically and independently dis- tributed. However, when these pop- ular models are evaluated on new testing data, whose distribution slightly different from the training data, a significant drop in perfor- mance will be observed 6].
Sourve Domains
Target Domain
Art
Carizon
Photo
Sketch
ENows closses
Ensrn clarnes
Labeled Samples
anbloon classes
undinciee ckapses
Unlaboled Samples
Testing Samples
Figure 1: Semi-supervised domain generalization with known and unknown classes. Labeled samples source domains (Art, Cartoon and Photo) are known classes (green box). Known classes (green box) and seen wiknown classes (blue box) are mixed in unlabeled samples on source domains, while known classes (green box), seen unknown classes (blue box) and unseen unknown classes (red box) are mixed in testing samples on target domain (Sketch).
In order improve the general- ization of the model under distri- bution shifts, Domain Adaptation (DA) 7 8 and Domain General- ization (DG) [1 have been widely studied. The goal of DA is to transfer the knowledge learned
Cooresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

from label-rich source domains to the unlabeled or partially labeled target domain. However, the accessibility of the target domain may not always be satisfied in some applications, e.g., autonomous driving and medical diagnosis, since we are not able to anticipate the domain that deployed systems will encounter. To this end, DG is introduced to develop nomain-generalizabee models on unseen target domain by using multiple different and labeled source domains.
To reduce the expensive cost of the labeling process [3] Semi-Supervised Domain Generalization (SSDG) 14 [5 16, 17 is proposed recently. In SSDG, each source domain consists of a few labeled samples and a large number of unlabeled samples. The goal is to learn a cammain-generalizee model from the partially labeled samples. However, in more realistic scenario, known classes are probably mixed with some unknown classes in unlabeled training and testing data (Figure []] In such a scenario, when deployed on an unseen target domain, the learned model is not only required to classify known classes, but also to recognize unknown classes. In this paper, we propose the Class-Wiss Wise Adaptive Exploration and Exploitation (C (CWAEE) method for semi-supervised domain generalization when known and unknown classes are mixed in unlabeled data. The intuition is first explore unlabeled training data by detecting known and unknown classes, and then exploit them in different ways to improve the generalization of the model. In particular, we use one-vs-rest classifiers and class-wise adaptive thresholds to detect known and unknown classes in unlabeled training data, and adopt consistency regularization on augmented samples based on Fourier Transformation to improve the unseen domain generalization. The experiments on real-world datasets verify that our CWAEE method can achieve better performance than compared methods.
2 Related Work
Domain Generalization (DG) aims to help the model generalize to an unseen target domain by utilizing one or several related domains. Most existing DG methods can be classified into three different categories: domain-invvrian representation learning, meta-learning and data augmentation. Motivated by the learning theory of domain adaptation [8], domain-invariant representation learning methods attempt to learn a representation space where the discrepancy between different source domains is as small as possible. For example, Muandet et al. (IT) proposed a kernel-based optimization algorithm for Support Vector Machine that Iearns invariant transformation by minimizing the distribution discrepancy across domains. After the domain-adversaria neural network was proposed for domain adaptation (8) aommain-adveraarii training is widely used for DG 19 20) Following Model-Agnostte Meta-Learning (MAML) (21). Li et al. [22] applied meta-learning strategy to DG through dividing the training data from multiple source domains into meta-train and meta-test sets to simulate domain shifts. The meta-optimization objective was designed to minimize the loss not only on meta-train domains but also on the meta-test domain. Shu el al. 23 conducted meta-learning over augmented domains to learn open-domain generalizable representations for Open Domain Generalization (OpenDG) problem. Compared to
representation learning and meta- learning, data augmentation methods have shown better performance on various DG benchmarks recently. Zhou et al. (24) mixed feature statistics extracted by bottom layers of Convolutional Neural Networks (CNNs) between instances of different domains to synthesize instances in novel domains for training. Xu er al.
25 applied MixUp 26 in amplitude spectrums of two images to augment training data while preserving high-level semantics of the original images in phase spectrums. To reduce the labeling cost of DG, Semi- Semi-Supervised Domain Generalization (SSDG) [14 [5] 16 I7 receives much attention recently. combines the pseudo-label method and meta-learning strategy. Zhou el al. [16] adopted stochastic classifier and multi-view consistency to improve the quality of pseudo-labels and generalization of model respectively. However, they assumed that both unlabeled training and testing data have the same label space with labeled training data while we consider more realistic scenario that the label space of labeled data is a subset of unlabeled training and testing data, i.c., both unlabeled training and testing data are a mixture of known and wnknown classes.
el al. [14] proposed the first SSDG method which
Out-of -iit----iitiittiinnn (OOD) detection aims to detect test samples whose labels are not contained in the label space of training data (27). Hendrycks and Gimpel (28) proposed the first effective baseline that uses the maximum softmax probability to identify OOD samples. Post-hoe methods attract much attention because they can be implemented without modifying the training procedure and objective. For example, Liang et al. [29] used temperature scaling and input perturbations to make In- In-Distribution (ID) and OOD samples more separable. Liu et al. [30] proposed to replace softmax
2

---

confidence scores with energy scores to detect OOD samples, and considered samples with higher energy to be OOD samples. Hendrycks et al. [I maximized the entropy of the model's predictions on enormous unlabeled OOD samples to improve OOD detection. Yang et al. [32 proposed the Unsupervised Dual Grouping (UDG) method that adopts two classification heads to dynamically separate ID and OOD samples in the unlabeled data and optimizes the model with different learning objectives. Since the commonly-usee splitting setup, i.e., one dataset as ID and all the others OOD, the existing OOD detection methods overfit to low-level dataset statistics instead of learning semantics, which leads to that even the same
OOD detection benchmarks [28 22 31 are based on dataset
class from another dataset will be detected OOD samples [32]
Semi-Supervised Learning (SSL) is an active research area in the last decade, whose goal is to leverage both labeled and unlabeled data to improve the performance of models and reduce the label cost. The most popular methods include consistency regularization [33 34 and entropy minimization 35 36].
The basic idea behind consistency regularization is that the model's outputs of similar
samples should be similar. Sajjadi et al. [33 proposed the IT model that minimizes consistency loss over two random augmentations of one sample to regularize the model. Tarvainen et al. [34] proposed the Mean Teacher method that averages Student model weights using Exponential Moving Average (EMA) over training steps as Teacher model to provide more stable targets for the consistency loss optimization of the Student model. Entropy minimization methods encourage models to make low entropy predictions a low-density region rather than a high-density area. Lee (36] proposed the Pseudo-label method that sets the maximum confidence prediction of unlabeled samples as pseudo-labels, and then trains the model in a supervised way with labeled and pseudo-labeled data. Sohn et al. (37 proposed the FixMatch method that combines consistency regularization and entropy minimization. It generates highly confident pseudo-labels on weakly-augmentte unlabeled samples, and trains the model to predict the pseudo-labels for the strongly- troogly-augmented version of the same samples. OpenMatch (38) extended FixMatch with OVANet [39] and open-set soft-consisenncy regularization to detect outliers in unlabeled data for Open-set Semi-Supervised Learning (OSSL).
unlabeled data, that the learned decision boundary passes through
3 Method
We have multiple source domains D1, D DD available for training, and each D has one labeled set D' = {xx{1) . w, ) and one unlabeled set D = {(u, ) i.e., Dr Dj. UDD. Generally, the size of D2 may be much larger than that of Dj-- i.e., n >> n².. The total size of labeled dataset is denoted as n and the total size of unlabeled dataset is denoted as n² i.c., n' = =Ei= n/e
and n = Ex=1 n. The model evaluated on an unseen target domain D. = {(x!", (0)) ) C¹ is the label set of labeled data D' Dí U D U U...UU ... C" is the label set of unlabeled data D" DJ U Dz U... ..... UDD and C is the label set of target domain D.. Due to the existence of unknown classes in unlabeled data and target domain, we have C¹ C C Ct. The classes in C' are called known classes, the classes in Ca)C²' are called seen unknown classes since they are seen in unlabeled data during the training process and the classes in COCC" are called unseen unknown classes since they are unseen during the training process. The goal is to learn the model f that generalizes well on target domain D, for both known and unknown classes, i.e., for each (x (1) (t) ) E Dr, if
vt E C², then (*(((") = yi else (x(()) (0) = unknown.
To effectively utilize unlabeled data to improve the generalization and robustness of the model, we should carefully exploit known and unknown classes in them. If known classes are treated as unknown ones, the generalization of the model may decrease significantly; if unknown classes are treated as known ones, the robustness of the model may degrade.
3.1 Detecting Known and Unknown Classes
In order to explore unlabeled data and detect known and unknown classes in them, we rely on the outputs of the deep neural networks and the intuition is that known classes will have higher scores than unknown classes [28]. The deep neural network model fe h go contains two parts, a feature extractor go and C'I-way linear classifier ha [h];... ... ALS" J.. The model maps each sample x; into (C' |ddimensional logits z; = hw ge (xx), and then feeds z; into softmax function
3

---

Procedure Calculating Thresholds
Score queues q gl:|e"| feature extractor go, linear classifiers h 1:|c"| validation dataset DD number of known classes [C
Input:
Output: Temperatures y1:|0"), thresholds 6 31:Cc1 knaw and 6 31:1c1| unk
1: for c= to |CI| do
2: Calculate temperature 7" calibrate ge. hy on D with Eq
3: Fit a two-component beta mixture model 0f.2 Pi:2 on queue of with EM algorithm;
4: Calculate known and unknown classes thresholds &nww aitDi"
8
unk
a5
a5+
5:
end for
S to produce the estimated posterior probability p: (akaa. score) of each class, i.e., p: S (x,)). The training is to minimize the cross- cross-entropy loss (Cx(O) between P: and one-hot label y, over labeled data, i.e., O argming (Cx(O) argming nf Σ E y log where n' the size of labeled data. The gradient of cross-entropy loss w.r.t. linear classifier has corresponding to class is @Cor(O) ne' (pf yi)ge (x+). Due to the relative normalization of the softmax 0hg
function, the score of x, on class i... pf. not only depends on the logit zf, but also depends other logits zi, o c. Thus, the update of hg, is affected by the predictions of other classifiers h j o c. When the logit of x; on target class is large, the scores of x; on non-target class, i.e., pf, o yi- will be small. Since the D - y is small, the gradient of the non-target classifiers may converge to zero. The parameters of the neural network are updated with the stochastic gradient descent (SGD) algorithm, so, the logits of samples on non-target classifiers induced by hig # will not decrease.
In order to update the logits on non-target classifiers induced by h A c, we replace the softmax function S with the sigmoid function σ. In this way, obtain individual score each way to indicate the probability that one sample belongs to each class, i.e., D ozl), c E C and we get |C|| one-vs-rest classifiers. With these one-vs-rest classifiers, we can detect known and unknown classes. A straightforward way is to set fixed threshold 6 0.5 [39 38 Let j argmax, (pf), if p 6, means that it belongs to class j: if p 6, means that is unknown classes sample. Due to the overconfidence of modern neural networks 40) it is not proper to use such a fixed threshold of 0.5 for all classes. A more reasonable way to use the class- wase adaptive threshold 41 [42 for each class. To get well-calibrated scores, we first calibrate the classifiers on validation data 2d with temperature scaling (40):
7 arrgmin
Σ (N...)DD
[y lo (ollgo(((///))) + (ys + co -00)/o(()/))) (1)
where I(-) is an indicator function and 7 is the temperature for class E C². Note that the validation dataset D only contains samples of known classes from source domains. And then, we use two-component beta mixture model to model the score distributions of known classes and unknown classes an unsupervised way, since it a more flexible approximator than the Gaussian mixture model (33. The adaptive thresholds for known classes 6% (the large one) and unknown classes Book (the small one) can be set as the mean values of two fitted beta distributions. Thus, for an unlabeled sample u, D let pi-
(T5) be the scaled score, if p > 8kmm. we consider it belongs to known class j. i.e. 14 = if p < 82 unk" we consider belongs to unknown classes, i.e. 0: unknown; if 52 unk < P < 5% kknu. it is difficult to determine its label, hence we set its prediction as null, i.e. ÿ = nail.. Since the model is updated in a mini-batch way, the scores predicted by the model many steps ago are less useful for modeling the score distributions. Hence, we utilize the most recent scores predicted by the model for the fitting of the beta mixture model. In particular, for each class j we maintain a queue & with fixed length to record the most recent scaled maximum score of unlabeled sample u, if its maximum score belongs to class j, i.e., argmax, (pf) = j. The overall procedure is summarized in Procedure !

---

3.2 Improving Target Domain Generalization
For labeled training data, we use the following supervised loss,
n' |e I = Im(p:) + c I(u + In(11 - p.).
C' For unlabeled training data, we should exploit them carefully to improve the unseen target domain generalization after having detected known and unknown classes. We construct the weakly and strongly augmented version of each sample following FixMatch [37] and calculate the model's 's scaled confidence score on its weakly augmented version, i.e., P Piweek
(2)
(Je (Tweak (u.) /Tr) E C
Then, we assign pseudo-label ii to it according to Section 3.1 and force the model's prediction on the strongly augmented version Pf, Pi,etrong a (fe (Istrong u, ()))) C² match the pseudo-label. For unlabeled training data predicted as known classes (i.e., D: E C¹), the loss is defined as
Cknw
I0 c) In(ff, +
Mi + c) In (itrongg))
12 Kemow
|e
where neww is the number of samples predicted as known classes. For unlabeled training data predicted unknown classes (i.e., Di unknown), the loss is defined
|e'1
Cunk
In(1 - P(.trong):
(4)
where W the number of samples predicted unknown classes.
Besides known and unknown classes, there are some unlabeled samples that are predicted as null. In order fully exploit these data and improve the generalization of the model the unseen target domain, we adopt Fourier Transformation to disentangle the semantics and styles of the sample (25 44 and then MixUp (6) the styles of two randomly sampled samples augment training data
for regularizing the model. Specifically, for each x, its Fourier Transformation F(x) is formulated as:
H-1W-
F(x)(u, h=0 ==
Σ Σ
Σ
x(h, w)e 122( u+
(5)
where H, W are the height and width of the sample, and 2 = 1. The amplitude component A(xx)
and phase component P(x) are then respectively expressed as
uu)) [ v) + (u) w)] 1/2 P(x)(u, v) arctan ((x)(u, u) R(x)
(6)
where R(x) and [(x) represent the real and imaginary part of F(x). is well-known that, the phase component of the Fourier spectrum preserves high-level semantics of the original signal, while the amplitude component contains low-level statistics. Therefore, we could mix up amplitude components of two samples of the same batch while keeping the phase components to generate the augmented
samples x (25 44), i.e.,
Á(x,) (1 A)A(x)) + XA(x;) ), =F (A(x.) * =J--p())))
where (x) is the inverse Fourier Transformation, X UO, 1), and U is the uniform distribution.
To avoid overfitting on domain-related low-level statistics, we minimize the consistency regularization loss between the original unlabeled samples and the augmented ones to push the model to pay attention the high-level semantics of the samples, defined
Ic
C
(xx
(x:))
(8)
T2
where n the number of unlabeled data. The overall loss of the training process formulated as: L L' Ai + >e + Aa
(9)
where År A2 and A are hyper-parametern to balance each loss. During the warm-up process, the model is trained only with labeled data, i.e., ò X2 X3 = O. The whole process is summarized in Algorithm (the framework figure can be found in Appendix A of the supplementary material), where the domain index is omitted for brevity.
5

---

Algorithm Class-Wa Wise Adaptive Exploration and Exploitation (CWABE)
Input: Labeled dataset D', unlabeled dataset D", validation dataset D", training epoch T. number of known classes |C*|.
1:|01
Output: Feature extractor go, linear classifiers ltu
1: Initialize go, I/w 1:10%) score queues q=:|C", temperatures ,1:|6), thresholds &lilc)) cl:|C%),
2: for /. = 1 to T do
3: for
to mar mar_mmaaeaaaeo iteration do
4: Draw a batch of labeled samples B¹ and unlabeled samples B from D' and D
5: Calculate loss L on B with 2q
6: Predict on 8" with 11:| to get scaled confidence scores Pu;
7: Split 8" into known classes B unknown classes unk and mill Buau with pa and c1:10"| Knnw 81:16%),
8:
Calculate loss Cin and Cumn on kno and Wok with Eqs and respectively;
9:
Conduct data augmentation within B to get 8 with Eqs
Calculate loss Loom on B and 8dd with Eq and get total loss L with Eq
Backward loss C and update ge. fw 1:|C'l with SGD;
Update a 1|(||| with pm:
12:
Update 7l:|6",, 5l:1c"1 and g1:|c"| with Procedure
14:
end for
15:
end for
4 Experiment
4.1 Datasets
We use PACS [45), Office-Home [46] and miniDomainNet 44 datasets in the experiments. PACS 45 consists of four domains corresponding to four different image styles, including Photo (P), Art painting (A), Cartoon (C) and Sketch (S), and the four domains have the same label set classes, and contain 9,991 images in total; OfficeHome [46] consists of images from four different domains: Artistic (A), Clipart (C), Product (P) and Real- World (R), and has large domain gap and around 15,500 images of 65 classes; miniDomainNet [47] is a subset of DomainNet [48] and has four domains including 18,703 images of Clipart (C), 31 ,020 images of Painting (P), 65,609 images of Real (R) and 24,492 images of Sketch (S), and it has 126 classes and maintains the complexity of the original DomainNet.
We adopt the common leave-one-domain-on protocol 12 16] three domains are used as the source domains and the remaining one as the target domain. Similar to [23 38) we split the classes into known and unknown classes, specifically we split the original label set into 3:2:2, 25:20:20 and 42:42:42 (known classes, seen unknown classes and unseen unknown classes) in PACS [45 Office- Home [46] and miniDomainNe [47] respectively in alphabetical order of the class name. On each source domain, 10 labeled samples of each known class are randomly sampled to construct the labeled data, and the remaining samples of known classes and seen unknown classes construct the unlabeled data. All samples on the target domain are used for evaluation.
4.2 Compared Methods
We compare our method with various DG methods, OOD detection methods and SSL methods. DeepAll naively puts labeled data from all source domains together, and trains the model with Empirical Risk Minimization (ERM); DAML (33 conducts meta-learning over augmented domains learn open-domain generalizable representations for OpenDG problem; UDG [32 adopts two classification heads to dynamically separate ID and OOD samples in the unlabeled data and optimizes the model with different learning objectives; FixMatch 37 generates highly confident pseudo-labels on weakly-augmented unlabeled samples, and trains the model to predict the pseudo-labels for the arongll-augmentedd version the same samples; Open 38) extends FixMatch with OVANet 39] and open-set soft-consistency regularization to detect outliers in unlabeled data for OSSL;

---

Table 1: Leave-one-domain-e results of known classes accuracy (left of the slash) and unknown classes AUROC (right the slash) on PACS, OfficeHome and meniDomainNet.
PACS
Target Domain
Art
Cartoon
Photo
Sketch
Average
DeepAll
62.96/ 60.06
53.41 / 15
79.17 / 71.26
48.60 50.16
61.03 59.91
UDG 332
42.98 / 49.83
46.92 / 48,52
58.75 57.28
38.82 / 45.21
46.87 / 50.21
DAML
23
42.07 / 50.27
57.74 / 54.80
42.87 / 54.00
45.29 /
46.99 51.57
FixMatch
3
81.32 68.67
61.85 / 56.34
85.63 / 64.87
76.39 48.001
76.30/ 59.47
OpenMatch
38
83.28 68.97
75.39 66.60
91.45 / 68.37
58.05 47..2
77.04 62.84
StyleMatch
16
82.66 63.35
71.95 56.86
90.81 / 67.40
77.34 / 43.33
80.69 57.73
CWAFE
87.08 81.21
76.65
72.88
93.19
80.30
79.87 82.46
84.20 79.21
Officellmme Clipart
Target Domain
Art
Product
Real-World
Average
DeepAll
61.95 / 69.97
50.80 / 60.96
75.23 / 71.38
84.55 / 76.63
68.13 69.73
UDG
32
52.25 60.71
41.97
63.64 / 64.74
72.24 / 65.90
57.52/ 61.73
DAML
23
45.73 62.96
33.98/55.44
58.50 / 67.09
64.46 1..46/67..0 67.75
53.17 63.31
FixMatch
37
65.25 / 67.60
59.32 62.18
73.31 / 66.72
82.35 73.16
70.06 / 67.67
OpenMatch
338
64.95 69.27
55.82 61.60
75.20 / 72.93
11.76/75.77
69.43 / 69.90
StyleMatch
16
67.83/
67.40
63.02 / 60.15
75.46 / 69.16
84.79 74.44
72.77 67.79
CWAEE
70.55 / 75.85
64.00 / 66.57
76.22 76.56
86.60 / 81.22
74.34 75.20
miniDomainNet Painting
Target Domain
Clipart
Real
Sketch
Average
DeepAll
$2.58 / 66.31
52.13 / 62.96
66.10 73.17
44.15 / 64.90
53.74 / 66.83
UDG 32
56.30/ 68.49
49.51 / 61.47
61.70/ 70.21
36.99 / 57.25
51.12 / 64.36
DAML
23
56.16 67.16
50.32/65.66
57.23 / 69.14
/
52.55 / 66.77
FixMatch 37
59.71 62.83
/9.71/
65.63 / 63.58
64.90 64.78
62.01 63.42
OpenMatch
38
64.53 72.70
61.55 69.80
70.61 74.87
61.40/71.30
64.52 72.17
StyleMatch
16
62.42/ 63.63
61.23 62.21
66.02 / 62.58
65.44 / 63.46
63.77 62.97
CWAEE
66.68 73.38
65.65 73.07
69.86 / 75.98
66.36 / 74.96
67.14 / 74.35
StyleMatch [66 extends FixMatch with stochastic classifier and multi-view consistency to improve the quality of pseudo-labels and generalization of model respectively for SSDG.
4.3 Setup
Most hyper-parameters follow the setting in [16] for a fair comparison. The SaaageNet-pretrainng 18 (2 is used as the CNN backbone, and the linear classifiers is imple- mented with stochastic classifiers. The initial learning rate of SGD op- timizer is set to 0.003 for the pre- trained backbone and 0.01 for the randomly initialized stochastic clas- sifier, both decaying following the cosine annealing rule. The running epochs are 40, 20 and 20 for PACS, OfficeHome and miniDomainNet re- spectively. For each mini-batch, we randomly sample 16 labeled samples and 16 unlabeled samples from each source domain. We set
Table 2: Leave-one-domall out average AUROC of seen (left of the slash) and unseen unknown classes (right of the slash) on PACS, OfficeHome and miniDomainNet.
ResNet-
Dataset
PACS
OfficeHoome
miniDomainNet
DeepAll UDG DAML
58.28 / 60.89
68.48 /7.04
69.21 67.89
50.73 49.08
63.35 60.07
64.26 64.33
50.45 / 52.94
62.21
64.46
67.20 / 66.36
FixMatch
53.84/ / 67.87
64.05 771.39
58.56 / 67.45
OpenMatch
55.61 / 73.73
68.12/71.77
72.86 71.55
StyleMatch
49.77 / 68.91
63.46 72.27
56.57 68.31
CWAEE
84.09 74.55
74.57 75.87 76.81 / 72.31
A = on PACS, A 0.4 on OfficeHome and A = 0.1 on miniDomainNet. We set X 0.4 and À3-- = 1.0 for all three datasets. We evaluate the accuracy on known classes and AUROC on unknown classes of the methods with different random seeds, and report the average results. More implementation details can be found in Appendix B of the supplementary material.
7

---

4.4 Results
The results are summarized in Table (the results with standard deviations can be found in Tables and 7 in Appendix C of the supplementary material), From the results, it can be found that our method consistently outperforms the compared methods on all datasets. Although the UDG and DAML methods are developed for wnknown class detection on the unseen target domain, their performance much worse than ours. Fix.Match has poor classification performance when evaluated on an unseen target domain, due to not considering the domain shifts problem. The performance of our method is much better than that of OpenMatch in most cases since it depends on the fixed thresholds to detect unknown classes in unlabeled data. StyleMatch has a worse performance than ours since it treats all unlabeled samples as one of the known classes. In order to evaluate the detection performance of methods seen and unseen unknown classes respectively, unseen unknown classes on the target domain in Table 2 (the results with standard deviations can be found in Table 8 in Appendix C of the supplementary material). It can be found that our method has better detection performance on both seen and unseen unknown classes.
also report the AUROC of seen and
4.5
Ablation Study
In order to provide additional insight into what makes method successful, we conduct ablation experiments on OfficeHome.
4.5.1 Effectiveness of Used Modules
Table Ablation study on the used modules.
Our method contains several mod- ules, including supervised loss [ unsupervised loss on known
Ablation
Accuracy / AUROC
Supervised loss
68. 134.88 / 69.73+1.01
classes 3. unsupervised loss on unknown classes (Eq class-wise
+ Unsupervised loss on known classes
72.30a0.18 / 64.47
with fixed thresholds
adaptive thresholds (Procedure [ and consistency regularization loss
+ Unsupervised loss on unknown classes
72.38a0.16 / 65. 1620.3
with fixed thresholds
Class wise adaptive thresholds
73.70+0.39 / 74.44
The results with standard deviations are presented in Table
(Eq
Consistency regularization loss
74.34+0.35 75.220
3 With the supervised loss, our
method trains the model only with labeled samples. After detecting known and unknown classes with
the fixed thresholds (i.e., gls[c1| = gl:[6"| 0.5) and trained with the unsupervised loss known and unknown classes, the performance can be improved with these detected samples. By replacing the fixed thresholds with class-wise adaptive thresholds, the performance be further boosted. With all the used modules, we can get the best performance.
To better show the effectiveness of the class- wise adaptive thresholds, we compare the quality of pseudo-labels when trained with fixed thresh- olds and the class-wise adaptive thresholds. The results depicted in Figure 2 It can be found that the accuracy of pseudo-labed labels assigned by our method is higher than that of compared method on known classes and unknown classes in all 4 domains, which means that method has better performance on known classes classi- fication and unknown classes detection.
AccumCy
80
40
Feat
feed
75
10 Spoch
15
20
20
10
15
20
Epoch
(a) Known classes
(b) Unknown classes
Figure 2: The accuracy of pseudo-labels on unla- beled samples.
4.5.2 Sensitivity to Number of Seen Unknown Classes
76
AUROC
In order to study the sensitivity of our method to the number of seen unknown classes, we com- pare our method with existing SSL methods un- der different numbers of seen unknown classes (IC"),C varies from 0 to 30), and the results are depicted in Figure It can be found that our
Filtacc
Cuts
DownMaich
10
10
15
unkcown
(a) Accuracy
(b) AUROC
Figure 3: The results with various numbers of seen unknown classes.
8

---

method outperforms the compared methods. With the increase in the number of seen unknown classes, the AUROC on unknown classes obtained by our method increases while the accuracy on known classes is maintained.
4.5.3 Sensitivity Number of Labeled Samples
In order to study the sensitivity of our method to the number of labeled sam- ples, we compare our method with ex- isting methods under different num- bers of labeled samples (5, 10 and 20 labels of each known class on each domain) and the results are summa- rized in (the results with stan- dard deviations can be found in Table 9 in Appendix C of the supplemen- tary material). It can be found that method outperforms compared meth- ods under different numbers of la- beled samples. The performance of method under extremely few labels (i.e., labels of each class each domain) is promising, since it can exploit known and unknown classes in unlabeled data a more reasonable way.
Table 4: Average accuracy / AUROC on OfficeHome with different numbers of labeled samples.
Labels
DeepAll UDG
62.87 67.77
68.13/ 69.73
72.17 70.83
57.52/ 61.73
65.31 /
DAML
52.67 / 64.15
53.17 63.31
50.95 63.07
FixMatch
66.90 64.29
70.06 / 67.67
71.92 70.21
Open
66.24 69.81
69.43 69.90
70.14 69.81
StyleMatch
70.16/ 63.57
72.77 67.79
75.36 70.45
CWAEE
71.24/ 73.23
74.34 75.20
76.09 73.96
4.5.4 Histograms of Confidence Scores
Empliicll o.
Letccum desers
The empirical probability density function of confidence scores of unlabeled data in different queues and different epochs are de- picted in Figure 4 Here, report class O and class in different epochs as examples, i.e. Figure 44| vs 4b] Figure (4c 4d] which shows that the confidence scores have dif- ferent distributions in the different training epochs. It can be found that our method CWAEE can find the adaptive thresholds for known and unknown classes.
8
(a) Class 0 in epoch
(b) Class 0 in epoch 20
Empirial 2.41
Emptrcal
Come doeess
0.2
0.8
0.0
0.6
1.0
Score
Score
(c) Class epach
(d) Class 1 epoch
Figure 4: The empirical p.d.f. of the confidence scores of unlabeled data.
5 Conclusion
In this paper, we focus on a more realistic semi-supervised domain generalization scenario, where known classes and seen unknown classes are mixed in unlabeled training data while known classes, seen unknown classes and unseen unknown classes are mixed in testing data. In order to utilize unlabeled training data, detect known and unknown classes in them with the class- wise adaptive thresholds based on one-vs. one-vs-rest classifiers. We adopt consistency regularization on augmented training samples based on Fourier Transformation to improve the generalization the unseen target domain. The experimental results conducted on different real-world datasets show that our method outperforms the existing state-of-the-art methods.
Broader Impacts
Our work provides an effective method for semi-supervised domain generalization with known and unknown classes. We believe our work will be beneficial for domain generalization projects with few labeled data and many unlabeled data, and does not have negative societal impacts.

---

Acknowledgments
This work is supported by the National Science Foundation of China (62276125, 61921006), the Fundamental Research Funds for the Central Universities (0221143800) 3), and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
References
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25 12), pages 1106-1114, 2012.
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'16) 16), pages 770-778, 2016.
[3] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. /81.0408 2019.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33 (NeurIPS') 20), pages 1877-1901, 2020.
[5] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet Classifiers Generalize to ImageNet? In Proceedings of the 36th International Conference on Machine Learning (ICML'19) pages 5389-5400, 2019.
[6] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural Adversarial Examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'21), 21), pages 15262-15271, 2021.
[7] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting Visual Category Models to New Domains. In Proceedings of the 11th European Conference on Computer Vision (0) pages 213-226, 2010.
[8] Yaroslav Ganin and Victor Lempitsky. Unsupervised Domain Adaptation by Backpropagation. In Proceedings of the 32nd International Conference on Machine Learning (ICML'I5 15), pages
1 1180-11899 1809, 2015.
[9] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning Transferable Features with Deep Adaptation Networks. In Proceedings of the 32nd International Conference on Machine Learning (ICML' 15), pages 97-105, 2015.
[10] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from Several Related Clas- sification Tasks to a New Unlabeled Sample. In Advances in Neural Information Processing Systems 24 (NIPS'1I), pages 2178-2186, 2011.
[11] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain Generalization via Invariant Feature Representation. In Proceedings of the 30th International Conference on Machine Learning (ICML'13) pages 10-18, 2013.
[12] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to Unseen Domains: A Survey Domain General- ization. IEEE Transactions on Knowledge and Data Engineering, 2022.
[13] Wei Wang and Zhi-Hua Zhou. On Multi-Visn Active Learning and the Combination with Semi-Supervised Learning. Proceedings of the 25th International Conference on Machine Learning (ICML '08), pages 1152-1159, 2008
10

---

[14] Hossein Sharifi- Mhgrifi-Noghab,, Hossein Asghari, Nazanin Mehrasa, and Martin Ester. Domain Generalization via semi-supereised Meta Learning. arXiv:2009. 12658, 2020.
[15] Xiao Liu, Spyridon Thermos, Alison O'Neil, and Sotirios A. Tsaftaris. Semi-supervised Meta-learning with Disentanglement for Generai-eGee Medical Image Segmentation. In Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI'2)), pages 307-317, 2021.
[16] Kaiyang Zhou, Chen Change Loy, and Ziwei Liu. Semi- Supeiviseee Domain Generalization with Stochastic StyleMatch. In NeurlPS Workshop on Distribution Shifts Connecting Methods and Applications, 2021.
[17) Huifeng Yao, Xiaowei Hu, and Xiaomeng Li. Enhancing Pseudo Label Quality for Semi- supervised Nomain-Generallizee Medical Image Segmentation. In Proceedings of the 36th AAAI Conference on Artificial Intelligence (AAAI'22), pages 3099-3107, 2022.
[18) Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A Theory of Learning from Different Domains. Machine Learning. 79(1):151-175, 2010.
[19] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain Generalization with Adversarial Feature Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR' 18), pages 5400-5409 2018
[20] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep Domain Generalization via Conditional Invariant Adversarial Networks. In Proceedings of the 15th European Conference Computer Vision (ECCV' pages 647-663, 2018.
[21] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML' (7), pages 122--1135, 2017.
[22] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to Generalize: Meta- Learning for Domain Generalization. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI'18), pages 3490-3497, 2018.
[23] Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang. and Mingsheng Long. Open Do- main Generalization with Domain-Augmenter Meta-Learning, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'21) pages 9624-9633, 2021.
[24] Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain Generalization with MixStyle. In Proceedings of the 9th International Conference on Learning Representations (ICLR'21) 2021.
[25] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. Fourier-Based Frame- work for Domain Generalization. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR'21) pages 14383-14392, 2021.
[26) Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond Empirical Risk Minimization. In Proceedings of the 6th International Conference on Learning Representations (ICIR' 2018.
[27] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized Out-of-Distribution Detection: A Survey. 10. 11334, 2021.
[28] Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Out-of- Distribution Examples in Neural Networks. In Proceedings of the 5th International Conference on Learning Representations (ICLR' 2017.
[29] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing The Reliability of Out-of-distribenn Image Detection in Neural Networks. In Proceedings of the 6th International Conference on Learning Representations ([CLR'18), 2018
11

---

[30] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based Detection. In Advances in Neural Information Processing Systems 33 (NeurIPS'20) pages 21464-21475, 2020.
[31] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep Anomaly Detection with Outlier Exposure. In Proceedings of the 7th International Conference on Learning Represenlations (ICLR'19), 2019.
[32] Jingkang Yang, Haoqi Wang. Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically Coherent Out-of-Distribution Detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV"2),, pages 8301-8309, 2021.
[33] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with Stochastic Trans- formations and Perturbations for Deep ceni-Supervised Learning. In Advances in Neural Information Processing Systems 29 (NIPS' 16), 2016.
[34] Antti Tarvainen and Harri Valpola. Mean Teachers are Better Role Models: Weight-averaged Consistency Targets Improve Semi-supervised Deep Learning Results. In Advances in Neural Information Processing Systems 30 (NIPS' 17), pages 1195-1204, 2017.
[35] Yves Grandvalet and Yoshua Bengio. Semi-supervised Learning by Entropy Minimization. In Advances in Neural Information Processing Systems 17 (NIPS 044), pages 529-536, 2004.
[36] Dong-Hyun Lee. Pseudo-label: The Simple and Efficient Semi-supervised Learning Method for Deep Neural Networks. In Workshop on challenges in representation learning of ICML, 2013.
[37] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying Semi- Supervised Learning with Consistency and Confidence. In Advances in Neural Information
Processing Systems 33(NeurIPS' 20), 2020.
[38] Kuniaki Saito, Donghyun Kim, and Kate Saenko. OpenMatch: Open-Set Semi-supervised Learning with Open-set Consistency Regularization. In Advances in Neural Information Processing Systems 34 2I),, pages 25956-25967, 2021.
[39] Kuniaki Saito and Kate Saenko. OVANet: One-vs-All Network for Universal Domain Adap- tation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'21), pages 9009, 2021.
[40] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML'17), pages 1321-1330, 2017.
[41] Xian-Jin Gui, Wei Wang. and Zhang- -aa Tian. Towards Understanding Deep Learning from Noisy Labels with Small-Loss Criterion. In Proceedings of the 30th International Joint Confer- ence on Artificial Intelligence (IJCAI'2)), pages -465, 2021.
[42] Bowen Zhang, Yidong Wang, Wenxin Hou, HAO WU, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. FlexMatch: Boosting Semi-Supervisee Learning with Curriculum Pseudo Labeling. In Advances in Neural Information Processing Systems 34 (NeurIPS 2/)), 21), pages 18419, 2021.
[43] Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsuper- vised Label Noise Modeling and Loss Correction. In Proceedings of the 36th International Conference on Machine Learning (ICML'19) pages 312-321, 2019.
[44] Yanchao Yang and Stefano Soatto. FDA: Fourier Domain Adaptation for Semantic Segmentation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR 2020.
[45] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, Broader and Artier Domain Generalization. In Proceedings of the IEEE International Conference on Computer Vision (7), pages 2544,,555,,011
12

---

[46] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep Hashing Network for Unsupervised Domain Adaptation. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR'I7) pages 5385-5394, 2017.
[47] Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain Adaptive Ensemble Learning. IEEE Transactions on Image Processing, 30:8008 8018, 2021.
[48] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment Matching for Multi-Source Domain Adaptation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'1), 2019.
13


Reviews:
[{"review_id": "8RVTi2J3WE", "reviewer": "Reviewer_mXoF", "summary": "The paper considers the realistic semi-supervised domain generalization setting where known classes are mixed with some unknown classes in the unlabeled training and testing data, and proposes the Class-Wise Adaptive Exploration and Exploitation (CWAEE) method. The experiments conducted on the datasets show the advantages over the previous baselines.", "questions": "My concerns for this paper are as follows:\n1. Why does the method use the two-component beta mixture model to calculate the thresholds? \n\n2. Why do the known classes have higher scores than the unknown classes? I think the authors should discuss more about this. \n\n3. Will the number of unknown classes influence the performance? For example, the number of unknown classes is far larger than that of known classes in the unlabeled training data.\n", "limitations": "no negative societal impacts", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Most previous semi-supervised domain generalization methods assumed that there is no unknown class in the unlabeled training data and testing data. The paper relaxes this assumption and considers a more challenging setting. The setting is new and important. \n\n2. The paper proposes a two-step method. It first uses one-vs-rest classifiers and class-wise thresholds to detect the known and unknown classes, and then uses Fourier Transform and data augmentation to improve the generalization on target domain and unknown classes. \n\n3. The method performs well on the datasets and the ablation study is sufficient to support the results.", "weaknesses": "1. Fourier Transform based on data augmentation may not work for non-image tasks. \n\n2. Some techniques are similar to the well-known method FixMatch.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Why does the method use the two-component beta mixture model to calculate the thresholds?\n\nQ2: Why do the known classes have higher scores than the unknown classes? I think the authors should discuss more about this.\n\nQ3: Will the number of unknown classes influences the performance? For example, the number of unknown classes is far larger than that of known classes in the unlabeled training data.\n\nA: We train one-vs-rest classifiers for each *known class*, and the output scores of each one-vs-rest classifier indicate whether the sample belongs to the class or not. The samples of *unknown classes* do not belong to *known classes*, so the scores of *unknown classes* are usually lower than that of *known classes* on the corresponding one-vs-rest classifier. The score distributions of *known class* and *unknown classes* are usually different, as shown in Figure 4 in our paper. Thus, we use a two-component beta mixture model to fit the score distributions and set the thresholds with the means of two beta distribution components on each one-vs-rest classifier.\n\nSince we train the one-vs-rest classifier for each *known classes*, the number of *unknown classes* in unlabeled data will not influence the performance of our method on *known classes*, though the performance of our method on *unknown classes* may be influenced not significantly. We conduct more experiments on OfficeHome to verify the effectiveness of our method. We split the original label set into 5:25:20 and 5:50:10 (*known classes*, *seen unknown classes* and *unseen unknown classes*) to make the number of *unknown classes* far larger than that of known classes, and the results are summarized in Table 1. It can be found that the number of *unknown classes* does not influence the *known classes* accuracy of our method significantly, though the *unknown classes* AUROC drops slightly.\n\n\nTable 1: Leave-one-domain-out average results of *known classes* accuracy (left of the slash) and *unknown classes* AUROC (right of the slash) on  OfficeHome.\n|              |      5:25:20      |      5:50:10      |\n|--------------|:-----------------:|:-----------------:|\n| DeepAll      |   86.72 / 81.39   |   86.72 / 81.39   |\n| UDG          |   80.70 / 70.77   |   82.13 / 73.65   |\n| DAML         |   60.14 / 60.24   |   62.13 / 61.16   |\n| FixMatch     |   78.59 / 69.84   |   74.69 / 59.51   |\n| OpenMatch    |   88.20 / 82.31   |   86.88 / 82.37   |\n| StyleMatch   |   82.55 / 63.59   |   77.36 / 54.67   |\n| CWAEE (ours) | **89.40 / 86.69** | **88.44 / 83.80** |"}, {"review_id": "hdEfeXEgZj", "reviewer": "Reviewer_8khn", "summary": "Paper considers the setting of semi-supervised domain generalization (SSDG) when both unlabeled source and target domains can contain unknown classes, i.e. not seen as labeled instances in source domains. The goal here is to learn a classifier which will be able to (i) reliably distinguish seen classes from unknown classes and (ii) effectively utilize unlabeled source data to learn domain generalizable representations. To approach the aforementioned goals authors suggest using one-vs-rest classifiers and class adaptive thresholds to distinguish between known and unknown classes. And then they apply different optimization objectives for known/unknown classes to improve learning domain generalizable representations. The proposed methodology leads to improvements over considered baselines on the standard benchmarks.", "questions": "- How ImageNet pretraining affects the performance of the proposed method? What would be the performance if the method is trained from scratch?", "limitations": "Authors do not discuss limitations. One possible limitation that I see is that the proposed setting decides not to distinguish between different unknown classes. It would be interesting to push the limits of semi-supervised learning and study the ultimate setting when both domain/class shift are presented and the classifier is able to distinguish between different unknown classes. Some related papers include but not limited to [1, 2]\n\n[1] Kaidi Cao, Maria Brbic, Jure Lescovec. Open-World Semi-Supervised Learning. ICLR 2022\n\n[2] Sagar Vaze, Kai Han, Andrea Vedaldi, Andrew Zisserman. Generalized Category Discovery. CVPR 2022", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-  The proposed problem formulation is important", "weaknesses": "-  Though the proposed setting is indeed realistic, the experimental setup is limited. It would be helpful to consider WILDS [1, 2] or other competitive benchmarks to understand the applicability of the proposed methodology.\n\n- The proposed methodology looks to me like combination of many well-known techniques without thorough motivation and deep analysis. \n\n[1] Pang Wei Koh et al' WILDS: A Benchmark of in-the-Wild Distribution Shifts. ICML 2021\n\n[2] Shiori Sagawa et al' Extending the WILDS Benchmark for Unsupervised Adaptation. https://wilds.stanford.edu", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: Though the proposed setting is indeed realistic, the experimental setup is limited. It would be helpful to consider WILDS or other competitive benchmarks to understand the applicability of the proposed methodology.\n\nA: Thanks for your kind suggestions. The experiments of our paper are conducted on the datasets which are usually used within this research community [1, 2, 3], and the results verify the effectiveness of our method. We will conduct more experiments on the mentioned benchmarks.\n\n[1] Shu et al, Open Domain Generalization with Domain-Augmented Meta-Learning, CVPR'21.\n\n[2] Zhou et al, Domain Generalization with MixStyle, ICLR'21.\n\n[3] Zhang et al, Towards Unsupervised Domain Generalization, CVPR'22.\n\nW2: The proposed methodology looks to me like combination of many well-known techniques without thorough motivation and deep analysis.\n\nA: Semi-Supervised Domain Generalization aims to use unlabeled data to improve the generalization of the model on *unseen* domains. Unfortunately, in a more realistic scenario, the unlabeled data usually contains *unknown classes* which may significantly degrade the generalization of the model, and the existing methods are not applicable to this scenario. The main contribution of our paper is that we propose a feasible solution with popularly used basic components, e.g. one-vs-all classifiers [4], class-wise adaptive thresholds [5] and consistency regularization [6], for this realistic scenario. The experiments conducted on realistic datasets verify the effectiveness and superiority of our method.\n\n[4] Rifkin et al., In Defense of One-vs-all Classification, JMLR'04.\n\n[5] Gui et al., Towards Understanding Deep Learning from Noisy Labels with Small-loss Criterion, IJCAI'21. \n\n[6] Sajjadi et al., Regularization with Stochastic Transformations and Perturbations for Deep Semi-supervised Learning, NIPS'16. \n\nQ2: How ImageNet pretraining affects the performance of the proposed method? What would be the performance if the method is trained from scratch?\n\nA: Thanks for your kind suggestions. In order to verify the effectiveness of our method, we train the model from scratch with the compared methods and ours. The results on Officehome with 25:20:20 (*known classes*, *seen unknown classes* and *unseen unknown classes*) are summarized in Table 2. It can be found that our method outperforms other compared methods.\n\nTable 2: Leave-one-domain-out average results of *known classes* accuracy and *unknown classes* AUROC on OfficeHome.\n|              |      Accuracy     |      AUROC      |\n|--------------|:-----------------:|:-----------------:|\n| DeepAll      |   21.75   |   54.07   |\n| UDG          |   19.91   |   55.95   |\n| DAML         |   29.35   |   55.55   |\n| FixMatch     |   40.68   |   57.25   |\n| OpenMatch    |   28.76   |  56.22   |\n| StyleMatch   |   42.57   |   56.72   |\n| CWAEE (ours) | **43.66** | **61.29** |"}, {"review_id": "IXoYOFjMiw", "reviewer": "Reviewer_enwD", "summary": "This paper focuses on the realistic scenario and proposes a semi-supervised domain generalization method. The method first explores unlabeled data by detecting known and unknown classes, and then exploits the data by adopting consistency regularization based on Fourier Transformation. The experiments show the effectiveness and superiority of the proposed method.", "questions": "The method uses the class-wise adaptive thresholds to detect unknown classes. Why do these thresholds work intuitively?\nThere are hyper-parameters in Eq(9), is the performance sensitive to the hyper-parameters?\n", "limitations": "None", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and organized. It is easy to follow.\nThis paper focuses on the realistic scenario where the known classes are mixed with unknown classes in the data, and proposes a method which outperforms the state-of-the-art semi-supervised domain generalization methods.\nThis paper uses one-vs-rest classifiers and class-wise adaptive thresholds, which is helpful to detecting unknown classes.\n", "weaknesses": "The method needs to train one-vs-rest classifiers. When the number of classes is large, the computation cost is high.\nThe calibration needs the validation data.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: The method needs to train one-vs-rest classifiers. When the number of classes is large, the computation cost is high.\n\nA: When training $|\\mathcal{C}^l|$ one-vs-rest classifiers for $|\\mathcal{C}^l|$ *known classes*, we keep the architecture and parameters of the network unchanged and only replace the softmax function after $|\\mathcal{C}^l|$-way linear classifier $h_\\omega$ with $|\\mathcal{C}^l|$ sigmoid function. These $|\\mathcal{C}^l|$ one-vs-rest classifiers share the same backbone network $g_\\theta$ and have their own classification heads $h_\\omega^c$. Thus, the computation cost of $|\\mathcal{C}^l|$ one-vs-rest classifiers in our method is actually equal to that of the softmax classifier. \n\nW2: The calibration needs the validation data.\n\nA: Score calibration, which uses the validation dataset, is a popularly used method for *unknown class* detection, *e.g.* [7, 8, 9, 10].\n\n[7] Guo et al., On Calibration of Modern Neural Networks, ICML'17. \n\n[8] Liang et al, Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks, ICLR'18.\n\n[9] Minderer et al, Revisiting the Calibration of Modern Neural Networks, NeurIPS'21.\n\n[10] Wang et al, Rethinking Calibration of Deep Neural Networks: Do Not Be Afraid of Overconfidence, NeurIPS'21.\n\n\nQ1: The method uses the class-wise adaptive thresholds to detect unknown classes. Why do these thresholds work intuitively?\n\nA: We train one-vs-rest classifiers for each *known class*, and the output scores of each one-vs-rest classifier indicate whether the sample belongs to the class or not. The samples of *unknown classes* do not belong to *known classes*, so the scores of *unknown classes* are usually lower than that of *known classes* on the corresponding one-vs-rest classifier. It is required to set proper thresholds for one-vs-rest classifiers to detect *known class* and *unknown classes*. However, due to the differences between the classes, it is difficult to manually choose the optimal threshold for each classifier. The score distributions of *known class* and *unknown classes* are usually different, as shown in Figure 4 in our paper. Thus, we use a two-component beta mixture model to fit the score distributions and set the thresholds with the means of two beta distribution components on each one-vs-rest classifier. The experiments conducted on realistic datasets also verify the effectiveness of our method.\n\nQ2: There are hyper-parameters in Eq(9), is the performance sensitive to the hyper-parameters?\n\nA: The performance of our method is not sensitive to the hyper-parameters. The results of the ablation study of hyper-parameters are in Appendix C of the submitted supplementary files."}, {"review_id": "lv3atcBgLN", "reviewer": "Reviewer_kd1R", "summary": "This paper introduces a new methodology for Semi-Supervised Domain Generalization (SSDG). This paper proposes the Class-Wise Adaptive Exploration and Exploitation (CWAEE) method, which contains one-vs-rest classifiers, class-wise adaptive thresholds, and consistency regularization based on Fourier Transformation. This algorithm shows the improvement in their SSDG settings.", "questions": "Please see the weaknesses.", "limitations": "Please see the weaknesses.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pros.\n- This paper is well-written, motivated, and easy to follow.\n- This paper proposes several methods in which class-wise adaptive thresholds and consistency regularization based on Fourier Transformation are very interesting works. (If the overview image for consistency regularization in the appendix is added in the main paper, it would be much easy to understand the concept. It is very novel and interesting. ) \n- This paper shows extensive experiments and shows the effectiveness of this method.", "weaknesses": "Cons.\n- (Major) This paper didn’t follow the previous training pipeline for the baseline. In FixMatch, it trains the model 2^20 iterations. For Cifar10/Cifar100, it is about 10k epochs. With only 80 epochs, I would say it is not the original other baseline numbers. \n- (Major) This paper uses Imagenet pre-trained model. It means that unlabeled data could be trained in the pre-training phase.\n- (Major) This paper evaluates their algorithm in only their setting. I think it can be evaluated in the same OOD setting in OpenMatch. (known/unknown in cifar10/cifar100 or Imagenet) If this paper could show the improvement in this setting, it would be more solid results. \n- (Minor) The one-vs-rest classifier is very similar to One-vs-All Outlier Detector in OpenMatch. (Novelty)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: (Major) This paper didn’t follow the previous training pipeline for the baseline. In FixMatch, it trains the model 2\\^20 iterations. For Cifar10/Cifar100, it is about 10k epochs. With only 80 epochs, I would say it is not the original other baseline numbers.\n\nA: Our paper focuses on the Semi-Supervised Domain Generalization (SSDG), thus we follow the training setting of the seminal work [11] of SSDG for a fair comparison.  \n\n[11] Zhou et al, Semi-Supervised Domain Generalization with Stochastic StyleMatch, NeurIPS 2021 Workshop (extension version published in IJCV'23).\n\nW2: (Major) This paper uses Imagenet pre-trained model. It means that unlabeled data could be trained in the pre-training phase.\n\nA: We follow the training setting of the seminal work [11] to use the ImageNet pre-trained model.\n\nThe results of the proposed method trained from scratch on Officehome with 25:20:20 (*known classes*, *seen unknown classes* and *unseen unknown classes*) are summarized in Table 2. It can be found that our method outperforms other compared methods when the model is trained from scratch.\n\nTable 2: Leave-one-domain-out average results of *known classes* accuracy and *unknown classes* AUROC on OfficeHome.\n|              |      Accuracy     |      AUROC      |\n|--------------|:-----------------:|:-----------------:|\n| DeepAll      |   21.75   |   54.07   |\n| UDG          |   19.91   |   55.95   |\n| DAML         |   29.35   |   55.55   |\n| FixMatch     |   40.68   |   57.25   |\n| OpenMatch    |   28.76   |  56.22   |\n| StyleMatch   |   42.57   |   56.72   |\n| CWAEE (ours) | **43.66** | **61.29** |\n\n\nW3: (Major) This paper evaluates their algorithm in only their setting. I think it can be evaluated in the same OOD setting in OpenMatch. (known/unknown in cifar10/cifar100 or Imagenet) If this paper could show the improvement in this setting, it would be more solid results.\n\nA: We focus on a more realistic but harder scenario where the learned model is not only required to classify *known classes* but also to recognize *unknown classes* on an *unseen* target domain. In order to carefully exploit unlabeled data from multiple source domains, we assign pseudo-labels to unlabeled samples whose scores are higher than the  *known classes* thresholds   \n $\\delta^{1:|C^l|}\\_{knw}$ or lower than the *unknown classes* thresholds $\\delta^{1:|\\mathcal{C}^l|}\\_{unk}$. The unlabeled samples whose scores are between $\\delta^{1:|\\mathcal{C}^l|}\\_{knw}$ and $\\delta^{1:|\\mathcal{C}^l|}\\_{unk}$ are only utilized through consistency regularization loss $\\mathcal{L}\\_{con}^u$. In a degenerate situation where the model is trained and tested on the same domain, like the OOD setting in OpenMatch, OpenMatch applies entropy minimization on **all** unlabeled data which may be slightly more effective since it is an easier scenario to separate and utilize **all** unlabeled samples of *known* and *unknown classes*. However, in our setting, OpenMatch is less effective than ours as the experimental results shown in Table 1 in our paper.\n\nW4: (Minor) The one-vs-rest classifier is very similar to One-vs-All Outlier Detector in OpenMatch. (Novelty)\n\nA: Semi-Supervised Domain Generalization aims to use unlabeled data to improve the generalization of the model on *unseen* domains. Unfortunately, in a more realistic scenario, the unlabeled data usually contains *unknown classes* which may significantly degrade the generalization of the model, and the existing methods are not applicable to this scenario. The main contribution of our paper is that we propose a feasible solution with popularly used basic components, e.g. one-vs-all classifiers [4], class-wise adaptive thresholds [5] and consistency regularization [6], for this realistic scenario. The experiments conducted on realistic datasets verify the effectiveness and superiority of our method.\n\n[4] Rifkin et al., In Defense of One-vs-all Classification, JMLR'04.\n\n[5] Gui et al., Towards Understanding Deep Learning from Noisy Labels with Small-loss Criterion, IJCAI'21. \n\n[6] Sajjadi et al., Regularization with Stochastic Transformations and Perturbations for Deep Semi-supervised Learning, NIPS'16. "}]

Author rebuttal:

Meta-review: The paper studies an extension of semi-supervised domain generalization problem in which known classes can be mixed with unknown classes in unlabeled training and testing data.  In the proposed setting, the model needs to classify known classes and also recognize unknown classes. The authors propose a method for tackling this setting that is based on one-vs-rest classifiers and class-wise adaptive thresholds to detect known and unknown classes. Additionally, the authors propose consistency regularization on augmented training samples based on Fourier Transformation to improve generalization on an unseen domain.

Reviewers found that the problem setting tackled in this work is important and realistic, and the proposed method is effective. The paper  is well-written, well-motivated and easy to follow. The experimental results are convincing.

Some reviewers expressed concerns about the experimental design (limitedness of the benchmark datasets, model pretraining). During the rebuttal, the authors included additional results with training the model from scratch and promised to include more challenging benchmarks in the final version. Authors are strongly advised to include these experiments promised during the rebuttal.

Eventually all reviewers agree that the paper should be accepted. I therefore recommend acceptance.
Decision: Accept (spotlight)