CrossMatch: Cross-Classifier Consistency Regularization for Open-Set Single Domain Generalization
Authors: Ronghang Zhu
Sheng Li
Venue: ICLR 2022 Poster (2022)
Keywords: Single Domain Generalization
Open-Set Recognition

Abstract: Single domain generalization (SDG) is a challenging scenario of domain generalization, where only one source domain is available to train the model. Typical SDG methods are based on the adversarial data augmentation strategy, which complements the diversity of source domain to learn a robust model. Existing SDG methods require the source and target domains to have the same label space. However, as target domains may contain novel categories unseen in source label space, this assumption is not practical in many real-world applications. In this paper, we propose a challenging and untouched problem: \textit{Open-Set Single Domain Generalization} (OS-SDG), where target domains include unseen categories out of source label space. The goal of OS-SDG is to learn a model, with only one source domain, to classify a target sample with correct class if it belongs to source label space, or assign it to unknown classes. We design a \textit{CrossMatch} approach to improve the performance of SDG methods on identifying unknown classes by leveraging a multi-binary classifier. CrossMatch generates auxiliary samples out of source label space by using an adversarial data augmentation strategy. We also adopt a consistency regularization on generated auxiliary samples between multi-binary classifiers and the model trained by SDG methods, to improve the model’s capability on unknown class identification. Experimental results on benchmark datasets prove the effectiveness of CrossMatch on enhancing the performance of SDG methods in the OS-SDG setting.

Published as a conference paper at ICLR 2022
CROSSMATCH: CROSS-CLASSIFIER CONSISTENCY REGULARIZATION FOR OPEN-SET SINGLE DOMAIN GENERALIZATION
Ronghang Zhu, Sheng University of Georgia (ronghangzhu, sheng. 11 Lillaeegeeeei
ABSTRACT
Single domain generalization (SDG) is a challenging scenario of domain gener- alization, where only one source domain is available to train the model. Typical SDG methods are based on the adversarial data augmentation strategy, which complements the diversity of source domain learn robust model. Existing SDG methods require the source and target domains to have the same label space. However, target domains may contain novel categories unseen in source label space, this assumption is not practical in many real-world applications. In this paper, we propose challenging and untouched problem: Open-Ser Single Domain Generalization (OS-SDG), where target domains include unseen categories out of source label space. The goal of OS-SDG is to learn model, with only one source domain, to classify a target sample with correct class if it belongs to source label space, or assign it to unknown classes. We design CrossMalch approach to improve the performance of SDG methods on identifying unknown classes by leveraging a multi-binary classifier. CrossMatch generates auxiliary samples out of source label space by using an adversarial data augmentation strategy. We also adopt a consistency regularization on generated auxiliary samples between multi- binary classifiers and the model trained by SDG methods, improve the model's capability on unknown class identification. Experimental results on benchmark datasets prove the effectiveness of CrossMatch enhancing the performance of SDG methods in the OS-SDG setting.
INTRODUCTION
Deep neural networks have obtained remarkable success in many classification tasks (Sze et al., 2017), as they can learn discriminative feature representations from data. These achievements are strongly relied the independent and identically distributed (i.i.) assumption (Vapnik 1992). i.c., the training and testing samples are from the same distribution. However, in real-world application, the i.i.d assumption is often violated due to the dataset shift problem. As result, the performance of deep neural networks degrades notably (Hendrycks & Dietterich, 2019 Recht et al. 2019). To tackle this challenge, domain adaptation (DA) (Long al. [2018] Peng al. 2020 Zhu et al. 2021b is proposed, which aims to mitigate the discrepancy between the source and target domains. DA assumes that both the source and target domains are accessible during model training stage. Even though DA has demonstrated promising performance on dealing with the dataset shift problem, it still fails in some practical scenarios when the target domains are unaccessible at the model training stage.
Domain generalization (DG) (Zhao et al. 2020b; Zhou et al. 2021b) is introduced to deal with the dataset shift problem and the absence of target domains. The goal of DG is train model from multiple source domains comprising the same label space, such that the learned model can generalize well to any unseen target domains. Despite of the success of DG in many real-world applications (Zhou et al. a , it yet fails to worst-case domain generalization scenario, i.e., single domain generalization (SDG). SDG, only one source domain is available to train the model while the learned model need to generalize well many unseen target domains. Several recent studies have made significant progress on this challenging yet practical scenario (Volpi et al. 2018
Zhao et al. 2020a Qiao al. 2020 Fan et al. 2021

---

Published as a conference paper at ICLR 2022
Domain Adaptation
Domain Generalizatin
Single Domain Generalization
Open
Open-Set SDG
Trarnig Slage
Traniin Stage
Traiving Stage
Training Siage
Training Stage
Teosing Stage
Tooking Stage
Tealing Slage
Teating
Teating Stage
Source Damain
Space
Target Domain Space
Figure I: Illustration of label space of domain adaptation (DA), domain generalization (DG), single domain generalization (SDG), open domain generalization (ODG), and open-set single domain generalization (OS-SDG). Different colors represent different domains. As there is no prior know ledge about the target domains, existing DG and SDG methods cannot handle unknown classes (e.g., camera and truck) in target domain. ODG provides multiple source domains to learn a model. However, our proposed OS-SDG only supplies a single source domain, which is more challenging than ODG.
Table 1: Illustration of difference between the proposed Open-Set Single Domain Generalization (OS-SDG) and other related tasks. D, and D, denote source and target domains, respectively, C, and C, denote the label space of source and target domains, respectively.
Task
More than one D. for training
Need C. = C+? Access to D.?
Domain Adaptation
Domain Generalization
X
Single Domain Generalization
Open-Set Domain Adaptation
X
Open Domain Generalization
X
X
The Proposed OS-SDG
X
X
One assumption in DG and SDG is that the label spaces of source and target domains are uniform. However, in some more realistic scenarios such as autonomous driving and biomedical applications, it is possible that the target domains contain novel categories that are unseen in the source label space. As result, the learned model cannot tackle target samples from novel categories, which dramatically degrades the robustness of existing DG and SDG methods. Until recently, this interesting yet tough problem is investigated by /Shu et al. (2021), which proposes a task called open domain generalization (ODG). ODG assumes that both source and target domains have different label spaces, and multiple source domains are accessible for training. An ODG model either classifies target sample with the correct class if it belongs to the multiple source label spaces, or marks as unknown class.
In this paper, we consider worst-case open-set problem (Geng et al. 2020 in SDG: only one source domain is accessible for training, and multiple target domains include samples that do not belong to any class in the source label space. We name this novel and challenging problem Open-Set Single Domain Generalization (OS- (DG). Figure illussraate the relationship between source and target label spaces in DA, DG, SDG, ODG, and SDG. We also summarize how problem is different from several related problems in Table 1 The goal of OS-SDG is to learn a model from a single source domain, which can generalize to unseen target domains with samples from unknown classes.
Inspired by universal domain adaptation Youu et al. 2019) Saito et al., 2020 Zhu & Li 2021), an "unknown" target sample tends to have larger entropy of the learned model's 's output than samples from "known" classes. We can draw a boundary between "known" and "unknown" classes by using the entropy of the learned model's outputs. We evaluate the performance of representative SDG methods (Volpi et al.) 2018; Zhao et al. 2020a) in the proposed OS-SDG task. Results (refer to Figure Section 4) show that the existing SDG methods have very limited capability on identifying unknown classes in target domains, although they obtain promising results in terms of per-class mean accuracy. These SDG methods only focus on synthesizing samples from diverse distributions to enrich source domain, and thus they cannot properly identify samples from unknown classes. The research challenge of modeling and identifying unknown classes in open-set single domain generalization is still untouched.
We propose a novel approach named CrossMatch to address the OS-SDG problem. First, CrossMatch induces unknown classes information by generating auxiliary samples that are potentially out of the source label space. These auxiliary samples may not belong to the actual unknown classes in the target domains, as we do not have access the target label space during training stage. However,
2

---

Published as a conference paper at ICLR 2022
encourage the auxiliary samples to be far away from the known classes in the source domain, and thus they could still assist in identifying whether a sample belongs to known classes or not. In particular, adopt a multi-binary classifier (Liu et al. 2019; Saito & Saenko 202I that consists of multiple one-vs-all binary classifiers. For a given sample, if all the binary classifiers mark it as negative, believe this sample has a high probability of belonging to unknown class. Therefore, this characteristic allows us to design new loss function to generate auxiliary samples for unknown classes. Second, with the generated auxiliary samples for unknown classes, we minimize their entropy of multi-binany classifier's 's output to help model learn discriminative feature distributions between known and unknown classes, and improve the capability of multi- binary classifier on unknown class identification. Last, we design novel consistency regularization to further improve the capability of model on unknown class identification by propagating unknown class information, learned by multi-binary classifier, to the model.
2 RELATED WORK AND BACKGROUND
Domain Generalization (DG) aims to learn model from multiple source domains and expects it to generalize over unseen target domains. MMandet al. (2013) proposed a sernel-based method to learn domain invariant feature representations by minimizing the dissimilarity across source domains. Shankar et al. (2018) augment source domain by using adversarial gradients obtained from a domain discriminator. Zhou et al. (2020) synthesize samples by mapping source samples to seeudo-novel domains with an optimal transport-baas based distance measure, Recently, meta-learning has involved in DG problem (Li et al., 2018 Dou et al. 2019; Du et al. 2020). The main idea lies in learning the model under a domain shift which is induced by meta-train and meta-test from source domains.
Single Domain Generalization (SDG) is a more challenging and realistic problem than DG. It assumes the access to a labeled source domain for training, and many unseen target domains for test. The source and target domains are sampled from different distributions. The goal of SDG is to learn a model from the source domain, which can generalize well on many unseen target domains. Adversarial gradient-based augmentation is usually used in SDG for generating new samples to enrich the diversity of source domain. For instance, Adversarial Data Augmentation (ADA) (Volpi al. 2018 adopts sign-flpped gradients back-propagated from label classifier to generate new samples. Maximum -nnrrop Adversarial Data Augmentation (MEADA) (Zhao et al.] (2020a) further improves ADA by generating more "hard" samples. Qiao al. (2020) use an auxiliary Wasserstein autoencoder to help ADA generate more "challenging" samples. Fan et al. (2021) design adaptive normalization scheme with ADA to enhance the generalization of learned model.
Open-Set Recognition (OSR) aims to identify unknown classes samples that are completely unseen during the training stage GGng et al. 2020). Bendale & Boult (0016) propose a model layer, i.c., OpenMax, to estimate the probability of a test sample belonging to the unknown class. Padhy et al. (200)) apply the one-vs-all classifier to identify unknown class samples. Although many OSR methods have obtained good performance under strong assumption that the source and target domains have similar distributions, they still fail to address the proposed SS-SDG problem, where the source and target domains are selected from different distributions.
Open-Set Domain daataation (OSDA) is proposed by Panareda Busto & Gall (211)) where both source and target domains have private label spaces, respectively, and the common label space is known. Recently, Saito et al. (2018) adjust the OSDA setting by claiming no source private label space, which means target label space contains source label space. Recent OSDA (Liu al. 2019) Bucci et al. 2020) methods focus on this challenging setting. However, OSDA faces two limitations. First, it relies on an assumption that the target domain is accessible during training stage. Second, OSDA methods focus on the scenario that there are only one source domain and one target domain.
Open Domain Generalization (ODA) is proposed by (Shu et al. 2021 J. In this problem, target domain and multiple source domains have different label spaces. The goal is to learn model from multiple source domains to correctly classify each target sample into either a known class in the source label space or an unknown class. They utilize meta-learning framework to learn generalizable representations across domains augmented on both feature-Ievel and label-level. Our OS- OS-SDG problem is more challenging than the ODA problem, as OS-SDG has only one source domain for training but multiple target domains for test, while ODA has multiple source domains for training and only one target domain for test.
3

---

Published as a conference paper at ICLR 2022
Worst-Case Problem is proposed by (Sinha al. 2018). In single domain generalization, consider to learn the model with source domain by solving the following objective function:
min sup (Rl[ (0; D:) D(D,, D.) s e)).
D2
where D represents distance metric that measures the similarity between source and target domains. indicates the largest transportation cost of moving source domain D, to target domain De. 8 denotes model parameters optimized by cross-entropy loss function Lee-
Adversarial Data Augmentation (Volpi et al. 2018) aims to enrich the diversity of source domain by generating domain D to approximate the unseen target domains Dr. It reformulates worst-case problem (in Eq. T into Lagrangian optimization problem with fixed penalty parameter 7 2 0:
min sup [El (0: D)] - "D(D,
D
Here D denotes the Wasserstein metric (ollii et al. 2018) used to preserve the semantics of the generated samples. The overall loss function is formulated as:
Lada Loe (0;
(,; - (0g D, D.),,
where Lconst (Bg: D, |G(x)- G(x,)) + 00 1 + / yo) and G is the feature extractor, The
whole training process is an iterative procedure where two stages are alternated, i.e., maximization stage and minimization stage. At maximization stage, the new domain D is generated from D, by maximizing Lado with learning rate vy:
1++1 f 2z + 1/ V Lada (0; 2z).
(4)
After the maximization stage, the generated domain D is appended D,. At the minimization stage, 9 is optimized by minimizing the loss Lee with D..
Multi-Binas Classifier has proved its capability unknown class identification in domain adapta- tion related problems (Liu et al. 2019: Saito & Saenko 2021). multi multi-binary classier Fb with
classes is defined as F2 = (Fb....... .......... ................................ F.). where F5 is the i-th one-vs-all binary classifier with output
pb. F) (G())) R We use (t 0/æ² and (t 1|zz), where Pt 0|æ) + pt 1|x) 1, to indicate how likely the sample belongs to the i-th class and other classes, respectively. Recently,
Saito & Saenko (2021) proposes a hard negative binary classifier sampling strategy help F; learn
better boundary to identify unknown classes by minimizing the following loss function:
Lova (2, v) log(p)(t 0|z)) min log(1 pé 11x)).
(5)
ify
Here the objective is to improve the robustness of multi multi-binary classifier on unknown class identifica- tion by optimizing each binary-classifier with the corresponding hard negative samples. Furthermore, for unlabeled samples, minimizing their multi-binary entropy loss Cb
& Σ pt 0/z) log 0|æ) + pt = 1|x) lo 1|x).
Cen (x)
(6)
They will either aligned to source samples or be kept as unknown, and help improve the confidence of model unknown class identification.
3
METHODOLOGY
In this section, we formally introduce the Open-Set Single Domain Generalization (OS-SDG) task, and then describe the proposed CrossMatch approach in detail.
3.1 OPEN-SET SINGLE DOMAIN GENERALIZATION
Given labeled source domain D, { v.) Ma with label space C,, and multiple unseen target domains D, {D) ..... D where Dj {<x] D with label space C, C. The source and target domains are sampled from different distributions. As target domains have novel classes that do not belong to C., denote those novel classes unknown class space in target
C,
4

---

Published as a conference paper at ICLR 2022
Classifior
6.
0.3
Leaa
Po. Sopt-ta Lonk
X
X
[oA)
G
La
Xx*
G
Multicltess Choodter
X5
Murl-Chwes Clanadie
9,
0.
P
Lad
[C]
Minimization Stage
Maximization Stage
Figure 2: Overview of CrossMatch in SDG methods. represents source samples while 2y. and Ta denote generated known class and auxiliary unknown class samples, respectively. is the feature extractor where
G(v.). The multi-binary classifier is Fb = ( P.C.., P. where P = (g))). F' is the multi-class classifier where = Fgg)). (a) At minimization stage, we adopt Lova (Eq. 5) and Lin. (Eq. 6) optimize while train / by cross-entropy loss Lee. We also propose novel loss Leer to improve the capability of F on unknown class identification. (b) At maximization stage, SDG methods maximize their defined losses, e.g.. Lada (Eq. (3). to generate new samples to enrich the diversity of source domain in C.. while we propose auxiliary sample generation loss Lunk to generate samples that are far way from C.. After maximization stage, the generated Zy is appended to source domain D.,
domains, C. C. The goal of OS-SDG is train model with D, classify target samples into [c_] + classes, where the novel classes in target domains are treated as one unknown class. Figure Iillstraaess the relationship of label space between source and target domains OS-SDG. The research challenges in OS-SDG are twofold. First, how we generate unknown classes samples that could complement sample diversity in the source domain? Second, what is an effective strategy to improve the model's capability on unknown class identification?
3.2 CROSSMATCH
The proposed CrossMatch approach generates auxiliary samples for unknown classes out of C, (Sec. (.2.1 and then exploits those samples improve the capability of existing SDG methods on unknown class identification (Sec. 3.2.2). An overview of the proposed CrossMatch approach is shown in Figure
Inspired by the worst-case problem defined in SDG, we extend it to OS-SDG setting based on the CrossMatch approach, follows:
[E[[x(0,0), D.) Lu(0,,) Da) (Dn) : D(D,, D.) 3 P. D.) 2 p}. 8.0% D= (D..D.,
min
sup
where 0 (fg. 0,} includes the parameters of feature extractor G and multi-class classifier F. B,, is the parameters of muult--inany classifier Fj. D {Dx. D.) denotes the augmented domain consists of known classes domain Dr and auxiliary unknown classes domain D.,. L. = {Lor + Lova) represents those loss functions who use D. to optimize model, i.e., cross-entropy loss (0 O xxx) and one-vs-all loss Lova (0g, f,, 2x) (in 5.. L. - taLcer) includes two loss functions that optimize model with D.,. i.c. multi- binti-binary entropy loss (0, 1/. ; (Eq. 6 and the proposed cross-classifier consistency regularization loss Leer (0,, Oz, 1..00: =.) that improve the capability of F on unknown class identification. Lunk (D.) (Eq. 13 a newly proposed loss function for generating auxiliary samples. D is the Wasserstein metric and p is the largest domain discrepancy between two domains.
At minimization stage, the overall loss function is defined follows:
Lmin Loe + Lova + C + aLoers
where a is the hyper- paramee.r
At maximization stage, aim to generate two kinds of samples: known class samples 2ą Dj and auxiliary samples x, e D, which are far away from the known classes in the source label space. For known class samples, consider the following Lagrangian relaxation derived from Eq. with
5

---

Published as a conference paper at ICLR 2022
penalty parameter y
sup -DD( Dz)) DE
(9)
where
DDD, D.) = Leonst (0g: Dr: D&) is defined in Eq. 3 To solve this penalty problem in
Eq. 9 we adopt the same strategy in (Volpi et al. 2018) with the suitable conditions (Boyd al.] 2004) by performing stochastic gradient descent procedures. We define the robust surrogate loss @(0; Zz. Uo) SuPayED, (ee 10; Uo) - Yoonst (0 Ix. T.)} and have Vo0(O; 2 Me) = V Lze (8; 2g Vs), where z = arg maxx1 ED, ( (fee Fy. Vs) -eonst (g: Fk. hz}} is an adversar- ial perturbation of I, at the current model 9. We adopt few iterations to generate known class samples with learning rate 17 by:
(10) The generation procedure in Eq. 10 is consistent with the single domain generation method ADA (Volpi et al. 2018)) which can be further extended other single domain generation methods, e.g., MEADA (Zhao et al. 2020a),. by adding more loss objectives.
2& ++1 2/ + 1/ (C:c Us) (0gi
Similar to the way of generating known class samples, we also convert Eq. 7 into a Lagrangian relaxation to generate auxiliary samples with penalty parameter y:
sup [E[Cunk (Dw)] + (D( D.))
(11)
Da
By adopting the same strategy known class sample generation process, can generate auxiliary samples with the following rule:
+ at (Lunk(z()) + Coonet (0gi x Nz)].
(12)
where 7y is the learning rate of stochastic gradient ascent. The details of Lunk and Foor will be described in Section 3 and Section 3.2.2 respectively.
Remarks. CrossMatch aims to tackle the challenges in OS-SDG from two aspects: (i) generation of auxiliary samples for unknown classes, and (ii) improvement of unknown class identification. The key innovation of the proposed approach lies in jointly encouraging the model to generate auxiliary samples for unknown classes by maximizing the newly designed loss Lunk at the maximization stage, and improving the capability of multi-clas classifier F on unknown class identification. In particular, optimize the proposed cross-classifie consistency regularization loss Leer with generated auxiliary samples for unknown classes at the minimization stage.
3.2.1 AUXILIARY SAMPLE GENERATION FOR UNKNOWN CLASSES
The key idea of adversarial data augmentation is sign-flipped gradients back- aack-propagated from a label classifier or other related terms (Volpi et al. 2018 Qiao et al.) 2020) to generate diversified samples. Inspired by the structure of milti-binary classifier Fb. if given sample classified as "other class" by all the binary classifiers in Ft. it has high probability to be associated with unknown classes. For example, C, contains four known classes (e-g.. "phone", "airplane", "radio" and "car") and C. consists of two unknown classes "truck" and "camera". as illustrated in Figure [ We assume that F3 has been well trained in the source domain. If a target sample belongs to the class "phone", the corresponding binary classifier should output high probability to "phone". while the other binary classifiers will classify it as "other class". But, if a target sample belongs to "trcck", all the binary classifiers would classify it as "other class".
With this insight, we design a novel auxiliary sample generalization loss Lank to generate samples that possibly out of source label space. Given source sample x, with label ys, multi-binary classifier Fú and feature extractor G, we denote Lunk as:
Lunne(Ess Us) log(r); 0(r,)) + Σ log 1/xxs,
(13)
where ph; (t = 0/2,) and pt: = 1|x,) represent the probabilitics of being ye-th class and other classes from the yt binary classifier, respectively. By maximizing Lunk at the maximization stage, it encourages the model to generate new samples that all binary classifiers in Fè predict it as "other class". Thus, the new generated sample could belong to unknown classes. As entropy loss function diversifying the adversarial generated samples (Zhao et al. (020a),, to
has proved its effectiveness further enrich the diversity of auxiliary samples, we also incorporate the multi multi-binryy loss D and one-vs-all loss Lave to Lunk-

---

Published as a conference paper at ICLR 2022
3.2.2 BROSS-CLASSIFIES CONSISTENCY REGULARIZATION
With the generated auxiliary samples for unknown classes, we minimize the multi-binary entropy loss Lent to enhance the capability of multi-binary classifier F. on unknown class identification, and help feature extractor learn discriminative feature representations, which benefit the multi-class classifier F on unknown class identification. Each auxiliary sample 2u will be aligned to source samples if belongs to C.. or kept as unknown. To make full use of unknown classes information learned by Fb. we propose the cross-classifier consistency regularization (CCR) loss to propagate those information learned by F to F. The key idea lies in encouraging F& and to produce the similar output distribution, in form of one-vs-all, for each generated auxiliary sample from D.:
k
Loer (Zw) = E Ilp.... - Pér,uli 111
(14)
where pi.,. E R² represents the i-th binary classifier's output. piy, indicates the one-vs-all version of multi-class classifier F's output Pa for i-th class, i.e., Pby [p.. pi]. Pt, is the probability of belongs i-th class. Different from previous consistency regularization (Laine & Aila [017 [Sajjadi] et al. 2016). which encourages the classifier to produce similar outputs for sample under different augmentations, our CCR encourages different classifiers to generate similar output distributions for given sample.
X
3.3 INFERENCE
Figure 3 illustrates the inference procedure where the learned model M consists of a feature extractor G and a multi-class classifier F. The target sample is marked as unknown class its entropy is larger than the threshold μ. Otherwise, it will be assigned to a class in the source label space C..
M
prediction
0.1
0.2
0.3
No
entropy
|c,|
Yes
unknown
EXPERIMENTS
Figure 3: Inference procedure.
4.1 EXPERIMENTAL SETUP
We briefly introduce the experimental setup in this section. More details about the experimental settings and experimental results are provided the Appendix due space limit.
Datasets. (1) Digits comprises of five digits datasets: MNIST (LeCun et al., 1989). SVHN (etzer et al. 201 USPS (Hull, 1994), MNIST-M and SYN (Ganin & Lempitsky 2015). MNIST is the source domain, and its label space includes numbers from 0 to 4. The other datasets are treated as target domains, and their target unknown label space C,¹ contains numbers from to 9. (2) Office31 (Saenko al.) (010)) contains 31 classes collected from three visually distinct domains: Amazon, DSLR and Webcam. The 10 classes shared by Office-31 and Caltech-256 (Gong et al.) 2012) form the source label space C.. In alphabetical order, the last 11 classes unknown class space C," Due to DSLR and Webcam are relatively small, we only use Amazon as the source domain. (3) Office-Home (Venkateswara al. 2017) consists of four domains: Artistic, Clip Art, Product, and Borl-World. It comprises of 65 categories from four dissimilar domains. In alphabetic order, we select the first 15 classes as the source label space C.. The remaining 50 classes are viewed as target unknown label space C₁ Each domain serves as source domain, and the rest are target domains. (4) PACS (Li et al.) 2017) consists four domains: Art Paint, Cartoon, Sketch, and Photo. It has 9,991 images from seven object categories.
used target
Implementation Details. For Digits, we adopt the ConvNet (LeCun et al. 1989) with architecture resize all images to 32x32, and follow the settings in Volpi et al. (2018): Zhao et al. 2020a). For Office3l, Office-Home and PACS, we use an ImageNet-pretraing ResNet18 (He et al. 2016) as the base network.
con--pool-cony-pot fo---oo-f--ooofooieee
Baselines and Metrics. We adopt the Empirical Risk Minimization (ERM) (Kotchinskii 2011) simple baseline without any consideration of SDG. We compare with two ofate-of-the- the SDG methods, Adversarial Data Augmentation (ADA) (Volpi et al. 2018) and Maximum Entrrop Adversarial Data Augmentation (MEADA) (Zhao et al. 2020a). We also adopt Open Set Domain
7

---

Published as a conference paper at ICLR 2022
Table 2: Results (%) on Digits (ConvNet) and Office31 (ResNet18).
Dataset
Metric
OSDAP
OpenMax
ERM
+CM
ADA
+CM
MEADA
+CM
acc
41.42
42.38
49.17
49.07
50.22
49.71
52.98
51.27
Digits
aOCu
70.60
83.81
13.04
53.52
15.11
52.07
29.83
46.11
acck
35.59
34.40
56.40
48.67
57.24
49.24
57.61
52.30
hs
40.46
40.67
17.97
40.15
20.14
39.93
30.37
38.70
76.51
18.19
79.82
78.30
80.13
78.61
80.26
78.98
Office31
00cu
84.28
100.0
27.04
37.60
25.24
34.51
25.09
41.08
acck
75.77
10.01
85.10
82.37
85.62
83.02
85.78
82.77
hs
77.68
16.74
40.69
51.14
38.65
48.50
38.55
54.69
Table 3: Results (%) on Office Home (ResNet18).
Artistic
Clip Art
Product
Real-World
Average
Method
ace
hs
acc
hs
acc
acc
hs
acc
hs
OSDAP
45.61
52.35
52.78
58.82
41,45
47.95
53.51
58.40
48.34
54.38
OpenMax
22.42
30.64
22.67
29.51
15.10
16.65
25.54
33.07
21.43
27.47
ERM
65.00
31.07
64.12
35.78
60.53
36.33
66.59
33.92
64.06
34,28
ERM+CM
65.49
52.85
63.37
50.51
58.03
47.25
67.75
52.60
63.66
50.80
ADA
68.29
32.94
65.10
42.09
60.52
34.72
67.04
34.86
65.24
36.15
ADA+CM
66.30
46.68
62.64
49.31
58.72
47.47
66.82
50.47
63.62
48.48
MEADA 68.31
33.29
65.25
42.05
60.43
35.68
67.04
34.65
65.01
36.42
MEADA+CM
65,85
53.22
62.90
48.87
58.36
45.34
67.10
50.77
63.55
49.55
representative method in open-set domain adaptation and OpenMax (Bendale Boult (016) typical open-set recognition method as baselines. Furthermore, we extend ERM, ADA, and MEADA using the proposed CrossMatch (CM) approach (denoted by "+CM"). We adopt the overall mean accuracy (acc), known class accuracy (accz) unknown class accuracy (ace,) and h-score (hs) (Fu et al. 2020) evaluation metrics.
Adaptation by Backpropagation (OSDAP) (Saito et al. 2018)
4.2 RESULTS AND DISCUSSIONS
Digits and Officel1. Table shows the results of our approach and baselines on the Digits and Office31 datasets. Our proposed method is superior to ADA by more than 19% and 9% with respect to on Digits and Office31, respectively. Compared with the state-of-the-art method MEADA, which obtains the best acc among all the methods, our method still significantly outperforms it with more than 8% and 16% in terms of hs on Digits and Office31, respectively. Meanwhile, our method achieves comparable mean accuracy (acc) than baselines. The strong baseline method OSDAP, uses source and target domains to learn the model, gets better results with regard to acc, in Digits and Office31 datasets. While method leams the model without target domain but gets similar performance in terms of hs in Digits, which proves the effectiveness of our method. Compared with representative open-set recognition method, OpenMax, our method obtains similar results in terms of hs and better performances with regard to acc and acck in Digits dataset. OpenMax, without considering the domain shift problem, gets high performance terms of acc, by wrongly marking most of testing samples as unknown class. When domain shift is larger in Office31 dataset, we find that OpenMax gets worse performance terms of acc, acck, and hs.
Office-Home. Table 3 shows the acc and hs on Office-Home Each time, one domain serves as the source domain and the rest three ones are target domains. Office- Home is more challenging than the previous two datasets because it contains more target unknown classes than them. Although the performance of our method on ace is slightly lower than baselines, we observe remarkable improvement of 16.52%. 12.22%, and 13.13% compared with ERM, ADA, and MEADA in terms of hs, respectively. Such improvements over basclines demonstrate the effectiveness of our method. Furthermore, compared with strong baseline OSDAP, our method gets better performance in terms of acc and similar results on hs, demonstrates the efficiency of our method. OpenMax fails to correctly identify known classes due significant domain shift existed in Office-Home dataset.
PACS. We further evaluate the effectiveness our method on PACS dataset. As shown in Table 4 our method outperforms all the baselines in terms of ace and hs. In particular, our method significantly
8

---

Published as a conference paper at ICLR 2022
Table 4: Results (%) on PACS (ResNer18).
Art Paint
Cartoon
Sketch
Photo
Average
Method
acc
hs
acc
acc
acc
hs
acc
hs
OSDAP
53.30
46.58
43.73
38.81
42.05
41.03
30.81
32.89
42.47
39.83
OpenMax
52.59
53.60
31.71
25.23
29.85
19.87
27.60
19.47
35.44
29.54
ERM
62.24
38.90
55.34
40.96
39.19
28.89
38.32
35.74
48.77
36.12
ERM+CM
63.52
44.9
57.6
48.31
38.53
30.43
42.52
41.6
50.54
41.31
ADA
62,48
39.02
56.43
41.55
39.03
26.93
40.28
38.13
49.56
36,41
ADA+CM
64.26
42.4
60.41
51.81
42.48
35.18
43.97
42.76
52.78
43.04
MEADA
62,43
38.85
56.1
41.34
38,89
26.43
39.88
38.24
49.33
36,22
MEADA+CM
62.63
41.88
60.03
51.36
41.51
35.76
43.5
41.6
51.92
42.65
improves the capability of baselines, ie., ERM, ADAN and MEADA, unknown class identification while slightly enhances the averaged accuracy over all classes. For other two baselines, i.e., OSDAP and OpenMax, we observe the similar phenomenons as Office-Home when large domain gaps existed among source and target domains.
Table 5: Ablation study for auxiliary sample generation on Digits dataset.
4.3 ANALYSIS
On the Effect of Lank and Leer. We con- duct ablation studies to examine the effec- tiveness of the proposed bunn (Eq. [3] and
LenitLova Lunk
Leer
acc
0CCa
hs
52.98
29.83
30.37
feer (Eq. 14) and show the results Ta- ble[5 The top row denotes the performance of baseline, i... MEADA. Firstly, the mid- dle demonstrates that simply maximiz- ing the combination C.nt and Love to generate "auxiliary" samples for unknown class would slightly decrease the performance of baseline. With the proposed Lunk, we observe that the performance surpasses the baseline by more than 12% and 6% in terms of acc,, and hs, which proves the effectiveness of proposed Lunkk- Moreover, observe that incorporating Leer can consistently improve the performance of both models in the middle row, which validate the effectiivnness of Leor.
51.86
29.01
29.36
51.08
42.55
36.94
52.16
30.47
30.55
51.27
46.11
38.70
Varying Size of known classes. To verify the robustness of our method under different sizes known classes, we conduct experiments on Office-Hone where Real- Harl-World is selected as the source domain. The size of known classes varies from to 60. As shown in Figure 4 our method continuously outperforms MEADA with significant improvements in terms of hs while obtains similar results as MEADA with respect to acc. It indicates that our method can consistently improve the capability of SDG methods unknown class identification.
89
MEIDA Vetru
Percent
60
52
known classes
Figure 4: Varying the size of known classes.
CONCLUSION
In this paper, we propose a new domain generalization task, i..., open-set single domain generalization (OS-SDG), which aims to generalize the model from single source domain to unseen target domains and simultancously deal with unknown classes. We further propose a novel CrossMatch approach to tackle the OS-SDG task, which generates auxiliary samples for unknown classes and improves the capability of unknown class identification with a novel consistency regularization. Extensive experiments demonstrate that the proposed CrossMatch method largely facilitates the capability of existing single domain generalization methods on unknown class identification.
ACKNOWLEDGEMENT
This research is supported by the U.S. Army Research Office Award (N911NF-2)---01009)
9

---

Published as a conference paper at ICLR 2022
REFERENCES
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1563-1572, 2016.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Silvia Bucci, Mohammad Reza Loghmann, and Tatiana Tommasi. On the effectiveness of image rotation for open set domain adaptation. In European Conference Computer Vision, pp. 422-438, 2020.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostie learning of semantic features. In Advances in Neural Information Processing Systems, volume 32, pp. 6450 1001,4611222 2019.
Yingjun Du Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snock, and Ling Shao. Learning learn with variational information bottleneck for domain generalization. In European Conference on Computer Vision, pp. 200-216, 2020.
Xinjie Fan, Qifei Wang, Junjie Ke, Feng Yang, Boqing Gong, and Mingyuan Zhou. Adversari- ally adaptive normalization for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9208-8210 2021.
Bo Fu. Zhangjie Cao. Mingsheng Long, and Jianmin Wang. Learning to detect open classes for universal domain adaptation. In European Conference on Computer Vision, pp. 567-583, 2020.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, pp. 1 1180-11 189, 2015.
Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman, Geodesic flow kernel for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2066-2073. IEEE, 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations, 2019.
Jonathan J. Hull. A database for handwritten text recognition research. IEFE Transactions on Pattern Analysis and Machine Intelligence, 16(5):550-53 1994.
Vladimir Koltchinskii. Oracle Inequalities in Empirical Risk Mininization and Sparse Recovery Problems: Ecole d'Elé de Probabilités de Saint-Flour XXXVIII-0008, volume 2033. Springer Science & Business Media, 2011.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference Learning Representations, 2017.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-555 1989.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5542-5550, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. Proceedings of the AAAI Conference on Artificial Intelli- gence, 2018.
10

---

Published as a conference paper at ICLR 2022
Hong Liu, Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Qiang Yang. Separate to adapt: Open set domain adaptation via progressive separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2927-2936, 2019.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1647-1657, 2018.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18, 2013.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Shreyas Padhy, Zachary Nado, Jie Ren, Jeremiah Liu, Jasper Snoek, and Balaji Lakshminarayanan. Revisiting one-vs-all classifiers for predictive uncertainty and out-of-distribution detection in neural networks. arXiv preprint arXiv:2007.05 2020.
Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In Proceedings of the IEEE International Conference Computer Vision, pp. 754-763, 2017.
Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adaptation. In International Conference on Learning Representations, 2020.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In 12556-12565, 2020.
Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400, 2019.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision, pp. 213-226, 2010.
Kuniaki Saito and Kate Saenko. Ovanet: One-vs-all network for universal domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, 2021.
Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In European Conference on Computer Vision, pp. 153-168, 2018.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. Advances in Neural Information Processing Systems, 33, 2020.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems, volume 29, pp. 1163-1171, 2016.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross- coos--gra training. In International Conference on Learning Representations, 2018.
Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open domain generalization with domain-augmento meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9624-9633, 2021.
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. In International Conference Learning Representations, 2018.
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, (0((12):2295-2222) 2017.
Vladimir Vapnik. Principles of risk minimization for learning theory. In Advances in Neural Information Processing Systems, pp. 831-838, 1992.

---

Published as a conference paper at ICLR 2022
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. Proceedings of the IEEE/CVE Conference Computer Viston and Pattern Recognition, pp. 5018-5027, 2017.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmenttioo. In Advances in Neural Information Processing Systems, 2018.
Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Universal domain adaptation. In Proceedings of the IEEC/CVF Conference Computer Vision and Pattern Recognition, pp. 2720-2729, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference. British Machine Vision Association, 2016.
Long Zhao, Ting Liu, Xi Peng, and Dimitris Metaxas. Maximum-entropp adversarial data augmenta- tion for improved generalization and robustness. In Advances in Neural Information Processing Systems, 2020a.
Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain generalization via entropy regularization. In Advances in Neural Information Processing Systems, volume 33, 2020b.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In European Conference Computer Vision, pp. 561-578, 2020.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arXiv preprint arXiv:2103.02500 2021a.
Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference Learning Representations, 2021b.
Ronghang Zhu and Sheng Li. Self-supervised universal domain adaptation with adaptive memory separation. In International Conference on Data Mining, pp. 1547-1552, 2021.
Ronghang Zhu, Xiaodong Jiang, Jiasen Lu, and Sheng Li. Cross-domain graph convolutions for adversarial unsupervised domain adaptation. IEEE Transactions on Neural Networks and Learning Systems, 2021a.
Ronghang Zhu, Xiaodong Jiang, Jiasen Lu, and Sheng Li. Transferable feature learning graphs across visual domains. In International Conference on Multimedia and Expo, pp. 1-6, 2021b.
12

---

Published as a conference paper at ICLR 2022
A
APPENDIX
The appendix provides descriptions of algorithm, details for datasets and experimental settings, and additional experimental results.
A.1
ALGORITIM
Algorithm Dilluttrate details of our proposed method.
Algorithm 1 CrossMatch with Adversarial Data Augmentation based Single Domain Generalization methods.
Input: Source domain D. = {(ar), Ri and generated auxiliary unknown class domain D., = ()- Initialized parameters 0g. f and f for feature extractor G, multi-class classifier F and multi- binary classifier Fi-
Output: Learned feature extractor G and multi-class classifier F:
1: for k = 1 to K do
> Run the maximizing procedure K times Run the mininization stage TMIN times
2:
for = to TMIN do
3:
Sample (Ix, uniformly from D.
4:
(0g, (2) - (0g. 0y) (000) 3V (0,.0; fee (0,, 0f Vo)
5:
(0., 02.) + (0,, Op.) - VV (V....0,,) )ovo (0,, Oz Ba Uo)
6:
D, is not empty then Sample zu uniformly from D'
7:
8:
(0g, f,,)) + (0g,0,) 3V (1 C (on 0g:0
9:
(0g. of, (2,0%, + (0g. 0,) - (V 0, foor 0p ((
10:
end if
end for
12:
for all
(Iz, E "D do 2k <- F, and In s- Js for f = 1 to TA do
14:
>> Run the maximization stage TMAX times
15:
3g +- 2x + 1 nn {Ledg - Tooons (22,
Ta f- Ju + V zu fuunk + fove + +h + Eooonst(E +2)
Append (#k, ys) to D2 and xa D.,
18:
end for end for
20: end for
21: while not reach maximum steps 7 do
22:
Sample (I., Ms) uniformly from Dz
23:
Sample Ia uniformly from D.,
24:
(0g, 01) (0g. 0y) -V, Lee ( 0f;
25:
(0g. (%) + (8y, 0/.) -3V (V,,, Lova (0 gg // Je
26:
(0g, f,,) - (0,, 0%,, - (V (0,,0, c (ty, Rn)
27:
(0g, B, ,r.) + (0 0,,) - BV 10 0y focr (f O, f,: zw)
28: end while
A.2 DATASETS
Digits consists of 5 different datasets: MNIST (LeCun et al. 1989), SVHN (Netzer et al.) 2011 1).
USPS (Hull 1994.. SYN and MNIST-M (Ganin Lempitsky 2015). Following the setting deñned by (Volpi et al. 20188] Zhao al.) 2020a), select 10,000 images from MNIST the source domain and the rest datasets are viewed as target domains. The source label space C, contains numbers from 0 to 4 and target unknow class space C; includes numbers from 5 to 9. We resize all the images to 32x32 and duplicate their channels to convert all the grayscale images to RGB.
Office31 (Sennoo et al. 2010) includes 4,652 images in 31 categories collected from three domains: Amazon, Webcam and DSLR. We follow the protocol proposed by (Panareda Busto & Gall 2017). We use the 10 classes (back pack, bike, calculator, headphones, keyboard, laptop, monitor, mouse,
13

---

Published as a conference paper at ICLR 2022
mug and projector) shared by Office-31 and Caltech-255 256 (Gong et al. 2012) the source domain label space C.. Then in alphabetical order, the last 11 classes (ruler, punchers, stapler, scissors, trash can, tape dispenser, pen, phone, printer, ring binder and speaker) used target unknown class space C,M Due to the number of images in DSLR and Webcam are relatively small, we only use Amazon the source domain.
Office-Home /Venkateswara et al. 2017) contains 15,500 images from four domains: Artistic, Clip Art, Product, and Real-World. As comprises of 65 categories from four dissimilar domains, it's more challenging than Digits and Office-31. In alphabetic order, we select the first 15 classes (alarm clock, backpack, battery, bed, bike, bottle, bucket, calculator, calendar, candles, chair, clipboards, computer, couch and curtains) as the source label space C, The remaining 50 classes are viewed as target unknown label space Ct Each time, one domain is selected as source domain and the rest are viewed as target domain.
PACS (Li et al. 2017) is recent challenging benchmark for domain generalization that includes four domains: Art Paint, Cartoon, Sketch, and Photo. It consists of 9,991 images and 7 object categories shared across these domains. alphabetic order, we choose the first four classes (dog, elephant, giraffe, and guitar.) as the source label space C.. The last three classes (horse, house, and person.) are selected as target unknown label space C.¹. We train the model by one domain and evaluate the learned model by the rest domains
A.3 EVALUATION METRICS.
We adopt the evaluation protocol in Open-Set Set Domain Adaptation (Panareda Busto & Gall 2017). where all the target classes accuracy averaged over all classes in |C,| + 1 (i.e., acc) and the unknown class accuracy (i.e., acc.) across all target domains. To better evaluate the capability of methods on unknown class identification, we also adopt the ht-ccore (hs) (Fu et al. 2020) metric, hs 2 acc, acc./ (acc, + acc,). It is the harmonic mean of average per-class accuracy C2 (i.e., accz.) and acc.. The h-score value is high only when both acc; and acc, high.
Cg are regarded as one unified unknown class. We compute the mean
A.4 IMPLEMENTATION DETAILS
We set the same parameters and optimization functions to baselines and ours.
Digits. Following the setup of (Volpi et al. 2018 Zhao al., 2020a). We use ConvNet (LeCun the base model. The batch size is 32. We adopt Adam with learning rate 3 0.0001 for minimization stage and SGD with learning rate 1) 1.0 for maximization stage. We set T 10000, TMIN 100, TMAX = 15, K 3, 1.0 and a 1. The reported results are the averaged results based on 10 random experiments.
et al. 1989) ccom--pol-coom-polll1 -oe--ooofffff--fffmma-
Office31, Office-Home and PACS. Referring to the setup of (Zhao et al. 2020a) PACS, we adopt ImageNett-retrainedd ResNet18 as base network and set the batch size to 32. We use SGD with learning rate 8 0.001 which adjusted by cosine annealing schedule (Zagoruyko & Komodakis 2016), the momentum of 0.9 and the weight decay of 0.00005 for minimization stage. We adopt the SGD with learning 7 = 1.0 for maximization stage. We set T 10000, TMIN 100, TMAX 15, K = 1, = 1.0 and a = 1. The reported results are the averaged results based on random experiments.
A.5 CONVERGENCE ANALYSIS
We testify the convergence performance of our method under acc and acc, by comparing it against ADA and MEAD on the Digits and Office31 datasets, respectively. (5g and (b) illustrate the capability of our method on improving the unknown class identification performance. We observe that the value of acc, is relatively high at the early training stage. This phenomenon is induced by the model that is not well-trained and prefers output high entropy values for testing samples. Even though the performance acc, drops during the training stage, our method still outperforms ADA and MEADA in terms of ace, while achieves similar mean accuracy acc.
14

---

Published as a conference paper at ICLR 2022
100
100
80
80
Percent
80
Percent
60
AGAteccl ADAIDOCJ
MEASAGATTT
MEADALaCC)
ADAr-Miacct
MEADA--MMaCH
2000
4000
scon
RC00
10000
2000
4000
6000
RO0D
10600
Iteration
Iteration
(8)
(b)
Figure 5: (a) acc and occ, convergence of ADA and ours on Digits. (b) acc and ace convergence of MEADA and ours on Office31.
Table 6: Results (%) on Digits (ConvNet) under different values of a.
a
0.4
1.0
1.4
acc
49.65
50.6
49.56
49.71
49.18
50.18
48.64
ADA+CM
acCa 49.69
51.56
47.53
52.07
52.36
48.14
56.62
hs 38.44
41.04
38.54
39.93
39.6
39.13
40.81
acc 50.65
52.3
51.6
51.27
51.57
50.47
51.01
MEADA+CM
aCCu 46.66
37.78
42.53
46.11
42.29
51.53
48.02
hs 39.14
36.34
37.07
38.70
37.36
41.67
40.08
A.6 PARAMETER SENSITIVITY
We analyze the sensitivity of a in Eq. on Digits datasets by varying it from 0.4 to 1.6. performance our method in terms acc, acc,, and hs. Compared ADA and MEADA performance on Digits datasets (in Table 2). our method significantly outperforms them with respect to acc, and hs while obtains comparable results on acc, which proves our method robust to different choices of We also evaluate the sensitivity of μ in Figure 3 on Digits. As shown in Table With small value of μs, the inference framework would mark many known classes samples as unknown class. With large value of jr, the inference framework will reject many unknown class samples to known classes.
A.7 PERFORMANCE UNKNOWN CLASS IDENTIFICATION AND KNOWN CLASSES CLASSIFICATION
We report the performance of our method and baselines terms of unknown class accuracy ace, and known classes accuracy accş on -ffice-Home and PACS datasets. More details can be found in Table g and 9
A.8 EXPERIMENTAL RESULTS WITH STANDARD DEVIATION (STD)
We provide STD value for our method and baselines, i.c. ERM, ADA, and MEADA four datasets, i.c., Digits, Office31, Office-Home, and PACS in Table 10 and I2
15

---

Published as a conference paper at ICLR 2022
Table 7: Results (%) Digits (ConvNet) under different values of μ.
μ=0.2
μ=0.6
μ=0.800
1=1.0
4=1.2
Metric
ERM
ERM+CM
ERM
ERM+CM
ERM
ERM+CM
ERM
ERM+CM
ERM
ERM+CM
48.54
42.34
54.07
48.36
56,40
48.67
58.13
53.68
58.98
56.35
acta
47.25
69.61
25.21
53.91
13.04
53.52
5.91
38.17
1.66
29.36
acc
48.37
46.88
49.26
49.28
49.17
49.07
49.42
51.09
48.02
51.85
hs
40.82
42.68
29.69
40.28
17.97
40.15
9.77
33.88
28.98
Table 8: Known classes accuracy (%) and unknown class accuracy (%) on Office-
Office-Home Home (ResNet18)
Artistic
Clip Art
Product
Woal-World
Average
Method
0CCk
acc,
aCCk
acc,
OCCk
acG,
acck
acc,
acck
OCC₁
OSDAP
44.13
67.84
51.69
69.26
40.00
63.47
52.48
66.92
47.07
66.87
OpenMax
17.38
98.08
17.72
97.04
9.53
98.59
20.78
97.01
16.35
97.68
ERM
68.54
20.53
66.75
24.65
62,81
26.26
69.48
23,18
66.90
23.66
ERM+CM
66.48
48.57
64.80
41.95
59.17
40.94
69.36
43.69
64.95
43.79
ADA
71.36
22.05
67.37
31.19
62.91
24.55
69.92
23.88
67.89
25,42
ADA+CM
67.53
39.59
64.10
40.67
59.92
40.72
68.53
40.79
65.02
40.44
MEADA
71.37
22.36
66.45
31.27
62.75
25.60
69.92
23.71
67.62
25.74
MEADA+CM
66.63
45.28
64.43
37.84
59.74
37.71
68.82
41.28
64.90
40.53
Table Known classes accuracy (%) and unknown class accuracy (%) on PACS (ResNet18).
Paint
Cartoon
Sketch
Photo
Average
Method
acck
@CC₁
acck
acc,
aCC2
OCC,
OSDAP
54.17
49.84
41.36
51.68
38.84
54.92
28.09
41.62
40.62
49.51
OpenMax
42.87
91.48
15.27
97.44
13.16
96.61
11.96
90.22
20.82
93.94
ERM
68.80
24.57
59.46
33.08
43.34
20.27
37.54
30.03
52,29
26,99
ERM+CM
68.66
44.56
62.25
43.18
41.01
33.16
39.91
54.21
52.96
44.53
ADA
70.95
28.80
62.08
33.83
43.18
22.41
40.65
38.77
54.22
30,93
ADA+CM
72.93
40.12
64.39
49.06
44.98
40.85
43.27
52.53
56.40
45.64
MEADA
70.90
28.65
62.09
33.55
43.42
22.90
39.78
40.31
54.05
31.35
MEADA+CM
70.45
33.36
63.76
53.74
40.25
48.79
42.89
50.57
54.34
46.61
Table 10: Results (%) with STD Value on Digits (ConvNet) and Office31 (ResNet18).
ERM
ERM+CM
ADA
ADA+CM
MEADA
MEADA+CM
acc
49.17-10.2
49.07-0.2
50.22:10.1
49.71
52.9811.2
51.27-0.1
Digits
acc,,
13.04.0..1
53.52-0.11
15.11
52.07-10.3
29.83.1.22
46.11
aces
56.40.10.2
48.67-0.2
57.24.10.3
49.24
57.6110.3
52.30
hs
17.97.0.2
40.15
20.1410.2
39.9310.2
30.37.0.2
38.70
acc
79.82-4033
78.30-0.3
80.1340.2
78.61+0.2
80.2640.2
78.98=0.2
Office31
acc.
27. 27.
37.60=0.2
25.
34.51+0.3
25.09+0..3
41.08 +0.2
acc₁
85.10
82.37=0.2
85.62+003
83.02=0.3
85.78+00.3
82.77=0.3
hs
40.69=0.4
51.14=0.2
38.65+0.3
48.50-0.3
38.55+0.2
54.69-0.2
16

---

Published as a conference paper at ICLR 2022
Table II: Results (%) with STD value on Office-Hlome (ResNet18).
Mechod
Artistic
Clip Art
Product
Real-Worid
Average
BCC
arC
Ar
act
oce
FRM
65.00:20.3
31007:06
64.12:20.3
61.53:04
36.33:005
66
33.92=0.5
14.78
65.49:2033
63.37
50.51:00.6
58.0320.3
47.25:20.7
67.75-006
52.60=0.4
ADA
68.29:00.6
32.9420.4
65.10:004
42.09:0.3
60.5240.2
34.72-0.33
67.0440.3
34.86=0.4
65.34
36.15
66.30:20.4
62.64::0.5
58.72::0.3
47.47:00.3
50.47=0.3
MEADA
68.31:10.3
33.29:10.4
65.25:10.4
42.05:10.5
60.43::.33
35.68:10.2
67.01.0.3
34,55004
65.01
36.42
65.85:2044
62.90
48.87:0022
58.3620.3
45.34:00.3
67.10+022
50.77=0.3
Table 12: Results (%) on PACS (ResNet18) 88.
Meloo
Art Paint
Cartoca
Sketch
Photo
Average
arc
dec
ERM
62.24::0.5
38.00+0.44
$5.34:00.6
40.96-20.5
39.19:04
28 28.89-003
38 38.32-0.22
35.74=0.4
48.77
36.12
$7.612:.4
48.31:0.5
30.43:20.6
41.6:20.4
ADA
62.48:10.6
39.02:10.4
56.43:50.4
41.55:00.5
39.03::022
26.93:1.44
40.25:10.6
38.13=0.5
49.56
36.41
ADAICM
64.26:005
42.44:06
60.41:044
51.81:04
42.8810.3
35.18:0.1
43.97:0044
42.76=0.7
MEADA
62.43:10:4
38.85:1033
$6.11:0.6
41.34104
38.89::0.7
26 26.43:10.5
39.88-10.6
38.21-0.4
69.33
36.22
51.36:0.66
35.76:20.4
17


Reviews:
[{"review_id": "fekzYLMZqPf", "reviewer": "Reviewer_46jP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n1. this paper proposes to study a new and interesting problem, open-set domain generalization\n\n2. experiments show that CrossMatch improves previous domain generalization methods like ADA (NeurIPS, 2018) and MEADA (NeurIPS, 2020) and obtains impressive results on three object recognition datasets\n\nweaknesses:\n1. novelty of the proposed method sounds somewhat incremental compared with prior works in DG (Volpi et al., 2018; Zhao et al., 2020a) and UDA (Liu et al., 2019; Saito & Saenko, 2021)\n\n2. the intuition behind L_{ccr} in Eq.(14) is hard to understand, how does matching the outputs of the one-vs-all classifier and the multi-class classifier benefit the generalization ability? The results of the variant \"L_unk^'+L_unk\" are missing in Table 4.\n\n3. Also, the results of open-set domain adaptation are missing in the experiments, which are vital and could be considered as the upper bound of this new studied problem setting.\n\n\nTypos:\n1. around Eq.(12), L_{consis} should be L_{ccr}", "summary_of_the_paper": "This paper proposes a new method called CrossMatch for open-set single domain generalization where only one source domain is available to train the model. \nThis problem studied here is interesting and sounds reasonable, which is also closely related to open-set domain adaptation and single domain generalization.\nIn particular, CrossMatch designs a new strategy to generate auxiliary samples for unknown classes and develops a novel consistency regularization to help identify samples from unknown classes in target domains.\nResults on several datasets verify that the proposed method achieves impressive results on three widely-used datasets.", "summary_of_the_review": "This paper proposes to study a new and interesting transfer learning setting called open-set domain generalization and develops a new method (CrossMatch) for this problem. However, CrossMatch is mainly built on several previous methods, e.g., the multi-binary classifier (Liu et al., 2019; Saito & Saenko, 2021), adversarial data augmentation (Volpi et al., 2018; Zhao et al., 2020a), making the overall novelty incremental for ICLR. Thus, I tend to give a \"borderline reject\" score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a-hT077n3x", "reviewer": "Reviewer_N4tv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n+ If there are unknown classes in the target domain, existing DG methods cannot handle this situation. As a result, existing methods will cause significantly prediction errors on such unknown-class data points in the target domain, making the prediction of a network unreliable. In this paper, they propose a new problem setting and a new method to handle this challenging problem.\n\n+ Introducing multi-binary classifier to the OS-SDG problem seems very interesting, since it may identify the unknown-class region well.\n\n+ This paper is easy to follow. Experiments can partially support the claims made in this paper. A plus should be that the authors also design some baselines to their problem setting, which provides solid baselines to see if the gains obtained by CM are significant.\n\nCons:\n\n- The presentation should be improved. There are many modules introduced in this paper, however, they are not well-motivated. It is better to explain their intuitions why they can help improve the performance.\n\n- I am not sure if it is necessary to list the contributions in the introduction. Such contributions have been described clearly in intro and abs. It seems that you do not need to restate them.\n\n- Key related works are missing. In the literature, open-set domain adaptation and open-set learning are very relevant topics to your proposed problem setting. They should be carefully reviewed and discussed with your method/setting. In some open-set learning papers, they also consider to generate unknown-class data, which is also a key module of your method.\n\n- In the theory of DG, researchers need to assume the relations between source domains and target domain to ensure that DG can be solved. However, this paper does try any theoretical analysis to their problem setting, which raises my concerns regarding the performance of CM on other datasets. It would be great if some theoretical analysis can be concluded or analysed. I would not like to make this problem can only be addressed by some heuristic methods. \n \n- How many times do you repeat your experiments? I did not see error bar/STD values of your methods. This should be provided to verify that the experimental results are stable.\n\n- One key experiment is missing. Openness experiments should be done to show the performance of your method when unknown classes change.\n\n- Although hs is a new criterion for open-set DA, it is better to include the known acc as well. Besides, I did not see acc_u in Table 3, which should be provided.\n\n- I did not see how the threshold affects the performance of your method. The threshold \\mu is a very important hyperparameter. How do you choose \\mu in the DG problem? \n\n", "summary_of_the_paper": "In domain generalization (DG), label set of target domain is that of source domains. However, we might meet the unknown classes in the target domain, which will cause significantly prediction error on such unknown-class data points in the target domain. To avoid this issue, this paper formulates a new problem setting: open-set DG, where the label set of target domain contains the label set of source domains. This paper actually considers a more challenging problem: open-set single DG (OS-SDG) that extends the problem setting of DG to a more general situation.\n\nTo address this very challenging problem, this paper designs a CrossMatch approach to improve the performance of SDG methods on identifying unknown classes by leveraging a multi-binary classifier. CrossMatch generates auxiliary samples out of source label space by using an adversarial data augmentation strategy. This paper also adopts a consistency regularization on generated auxiliary samples between multibinary classifiers and the model trained by SDG methods, to improve the model’s capability on unknown class identification. Experimental results on benchmark datasets prove the effectiveness of CrossMatch on enhancing the performance of SDG methods in the OS-SDG setting.\n\nIn general, this paper contributes a novel problem setting and a validate solution to this setting. Although some motivations are unclear, the contributions of this paper are enough.\n", "summary_of_the_review": "In general, considering the significance of the researched problem, this paper might be accepted by the ICLR2022. However, some points should be clarified and strengthened in the revision.\n\n\n-----POST REBUTTAL-----\n\nThe authors have addressed my concerns. Thus, I increase my score from 6 to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZT1iBJZ3rzm", "reviewer": "Reviewer_zAkR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n+ OS-SDG is an interesting and realistic problem, and the paper clearly described its difference from existing methods.\n+ The proposed method makes sense and is technically sound.\n+ The experimental results show the effectiveness of the proposed methods.\n\nWeakness\n- One concern is that the author maybe could analyze which parts of their approach decrease the known class's accuracy. This could be interesting to know since in some settings like Table 3, adding CM decreases the accuracy for known classes by about 3%. \n- Another thing is that some results in the appendix could move forward to replace the converging figures in Figure 4. \n", "summary_of_the_paper": "The paper presents a new task: open-set single domain generalization, where only one source domain is available and unknown classes and unseen target domains increase the difficulty of the task. A new method CrossMatch is proposed to solve this new problem. Firstly, auxiliary examples are generated for unknown classes out of the source classes. Then multi-binary classifiers are used to deal with unknown class identification in domain adaptation. Then, the paper proposes a cross-classifier consistency regularization that minimizes the multi-binary classifier's output and one-vs-all multi-class classifier's output. The experiments show the proposed method could largely improve the accuracy for unknown classes in the target domain.", "summary_of_the_review": "The paper is well-written and is technically sound and the proposed problem makes sense.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HDvAAhsAzII", "reviewer": "Reviewer_3bup", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n- Open-set single domain generalization is a novel, challenging, but practically important problem setting. Figure 1 is a good summarization on relationship between this problem setting and related ones.\n\n- Experimental results with several benchmark datasets show that the proposed scheme makes it possible to extend existing domain generalization methods to detect unknown-class data in the inference phase.\n\n\nWeakness\n\n- I have several concerns regarding the design of CrossMatch.\n\n  - Do we need to assume that all target domains share the same label space?\n\n  - In Eq. (7), since D(D_u, D_s) > \\rho is the only constraint on D_u, it seems that D_u can freely go far away from the source data distribution, if we take supreme with respect to D_u.\n\n  - Although the objective function for a whole training process is defined in Eq. (7), the loss functions in each stage do not follow this definition.\n\n    - Eq. (8) adopts an additional hyperparameter \\alpha.\n\n    - Eq. (9) and (10) adopt totally different loss functions, which are L_sdg and L_unk.\n\n  - Considering the motivation of cross-classifier consistency regularization, it would be better to stop gradient for F_b. Is there any reason to minimize L_ccr with respect to F_b?\n\n- Although open-set single domain generalization can be seen as SDG + open-set classification task, the authors only use SDG methods as baselines in the experiments. Naive open-set classification (or out-of-distribution detection) methods should perform good for unknown classes but not for known classes due to domain shift, which highlights the advantage of the proposed method more clearly. I could not judge the significancy of the proposed method from the current experimental results.\n\n- How did the authors tune \\mu in the experiments? And is it also used for baseline methods (ERM, ADA, and MEADA)? The accuracy on unknown classes should heavily depend on the setting of \\mu.\n\n- Several notations are confusing. I list some in the following.\n\n  - In general, \"domain\" means a pair of representation and distribution of data [R1], not a dataset. \n\n    [R1] \"A Survey on Transfer Learning,\" IEEE Trans. on KDE, 2009.\n\n  - The arguments of the loss functions sometimes change in the manuscript.\n\n  - In Eq. (7), it should be better to explicitly describe which data distribution is used to take expectation.\n\n  - In Eq. (13), p_b^i (t=1|x) should be equal to p_b^i.\n\nMinor concerns that do not affect my score\n\n- In Table 4, L_unk seems to have more impact on performance than L_ccr, though classifier consistency regularization is only included in the title.\n\n", "summary_of_the_paper": "This study tackles a novel domain generalization task, called open-set single domain generalization, in which a model is to be trained with single source domain but needs to generalize well at unseen target domains that may include unseen classes. To solve this problem, the authors extend single domain generalization methods to be able to learn detection of unseen classes by adopting adversarial data augmentation. Experimental results show that the proposed scheme can facilitates the capability of existing methods on detecting unknown-class data.", "summary_of_the_review": "Open-set single domain generalization is an interesting and important problem setting. However, the proposed method is not approapriately designed for this problem setting. In addition, the experiments and discussion lack an important related work, which is open-set classification (or out-of-distribution detection). I vote for \"weak reject.\"\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new problem: open-set single domain generalization, where only one source domain is available and unknown classes and unseen target domains increase the difficulty of the task. To tackle this challenging problem, this paper designs a CrossMatch approach to improve the performance of SDG methods on identifying unknown classes by leveraging a multi-binary classifier. CrossMatch generates auxiliary samples out of source label space by using an adversarial data augmentation strategy. Then, the paper proposes a cross-classifier consistency regularization that minimizes the multi-binary classifier's output and one-vs-all multi-class classifier's output. 

The proposed OS-SDG is an interesting and realistic problem. However, since it is way more challenging, the optimal solution to it remains elusive. Some reviewers think the method might be heuristic and lack theoretical guarantees. Nevertheless, the results are promising and the paper makes a first step toward the challenging OS-SDG problem. Another concern is that the CCR loss needs more ablation studies to further analyze its role. Though the authors have added more explanation of this part, I suggest the authors put more ablation studies in the final supplementary document. 

Overall, the paper is novel and interesting.  I would recommend acceptance of this paper given its novelty and impressive performance, but I highly suggest the authors add more ablation studies in the final supplementary, as suggested by the reviewers.
Decision: Accept (Poster)