Unknown-Aware Domain Adversarial Learning for Open-Set Domain Adaptation
Authors: JoonHo Jang
Byeonghu Na
Dong Hyeok Shin
Mingi Ji
Kyungwoo Song
Il-chul Moon
Venue: NeurIPS 2022 Accept (2022)
Keywords: domain adaptation
open-set domain adaptation
domain adversarial learning
representation learning
open set recognition

Abstract: Open-Set Domain Adaptation (OSDA) assumes that a target domain contains unknown classes, which are not discovered in a source domain. Existing domain adversarial learning methods are not suitable for OSDA because distribution matching with $\textit{unknown}$ classes leads to negative transfer. Previous OSDA methods have focused on matching the source and the target distribution by only utilizing $\textit{known}$ classes. However, this $\textit{known}$-only matching may fail to learn the target-$\textit{unknown}$ feature space. Therefore, we propose Unknown-Aware Domain Adversarial Learning (UADAL), which $\textit{aligns}$ the source and the target-$\textit{known}$ distribution while simultaneously $\textit{segregating}$ the target-$\textit{unknown}$ distribution in the feature alignment procedure. We provide theoretical analyses on the optimized state of the proposed $\textit{unknown-aware}$ feature alignment, so we can guarantee both $\textit{alignment}$ and $\textit{segregation}$ theoretically. Empirically, we evaluate UADAL on the benchmark datasets, which shows that UADAL outperforms other methods with better feature alignments by reporting state-of-the-art performances.

Unknown-Aware Domain Adversarial Learning for Open-Set Domain Adaptation
JoonHo Jang KAIST
Byeonghu Na KAIST
DongHyeok Shin KAIST
adkto80930kaist.cm ac.kr
wpo30520kaist.ac..k ac.k
clsebdgurOBkaist..oo
Mingi Ji
Kyungwoo Song University of Scoul
II-Chul Moon Moon
KAIST
KAIST, Summary.Al icmoon@kaist.o ac kr
quertggfarbbkkiist.ac.u
kyungwoo. songuuos ac kr
Abstract
Open-Set Domain Adaptation (OSDA) assumes that target domain contains un- known classes, which are not discovered in source domain. Existing domain adversarial learning methods are not suitable for OSDA because distribution match- ing with unknown classes leads to negative transfer. Previous OSDA methods have focused on matching the source and the target distribution by only utilizing known classes. However, this known-only matching may fail to learn the target-unknown feature space. Therefore, we propose Unknown-Awe Aware Domain Adversarial Learn- ing (UADAL), which aligns the source and the target-known distribution while simultaneously segregating the target-unknown distribution the feature align- ment procedure. We provide theoretical analyses on the optimized state of the proposed unknown- feature alignment, so we can guarantee both aligument and segregation theoretically. Empirically, we evaluate UADAL on the benchmark datasets, which shows that UADAL outperforms other methods with better feature alignments by reporting state-of-the-ar performanced
1 Introduction
Unsupervised Domain Adaptation (UDA) means leveraging knowledge from labeled source domain
unlabeled target domain [4 3 2 14 3 This adaptation implicitly assumes the source and the target data distributions, where each distribution is likely to be drawn from different distributions, i.e., domain shift (see Figure Tap. Researchers have approached the modeling of distributions by statistical matching (20 27 [6]. or domain ad adversarial learning [7] 28 18). Among them, domain adversarial learning has been successful matching between the source and the target distributions via feature alignment, so the model can accomplish the Nomain-invarian representations.
There is another dynamic aspect of the source and the target distribution. In realistic scenario, these distributions may expand a class set, which is called unknown classes. This expansion creates a field of Open-Set Domain Adaptation (OSDA) 23 13. Existing adversarial learning methods of UDA have limitations to solve OSDA because matching the source and the target distribution with the unknown classes may lead the negative transfer (6) due the class set mismatch (see Figure [b).
Previous OSDA methods focused on matching between the source and the target domain only embedded in the known class set via domain adversarial learning [23 13]. However, this known- only feature alignment may fail to learn the target-unknown feature space because of no alignment
now Google (mingiji @ google.com)
t The code will be publicly available on wtttps///ggthbbbccm/ooobooooeeee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Benle A Tagae Toggel Milarnee
Siase d Segeritioo @ tergeri dinee 6-
Scare A Teggeitoce Decides De aranday Seare Hariees
Target Linsoonn de Soero @ Deckies Sa ellyyy Liireons
(a) Domain Shift
(b) DANN g
(c) STA 13
(d) UADAL (ours)
The feature distributions from the source and the target domain, with the decision boundaries. The blue/orange/gray arrows represent the alignment signal on the source/target Suureffagge-Knnow//urg wwwwn//arge-unkkntneee unknown domain.
Figure I:
signal from the target-unknown instances. Therefore, a classifier is not able to learn a clear decision boundary for unknown classes because the target-unknown instances are not segregated enough in the aligned feature space (see Figure Ic]. On the other hand, some OSDA methods propose to learn intrinsic target structures by utilizing self-supervised learning without distribution matching [24] [2]. However, this weakens the model performance under large domain shifts, Therefore, in order to robustly solve OSDA, distribution matching via the domain adversarial learning is required, and the class set mismatch should be resolved to prevent the negative transfer, simultaneously.
This paper proposes new domain adversarial learning for OSDA, called Unknown- Domain Adversarial Learning (UADAL). Specifically, we aim at enforcing the target-unknown features to move apart from the source and the target-known features, while aligning the source and the target-known features. We call these alignments as the segregation of the target-unknown features (gray-colored arrow in Figure Id). Although this distribution segregation is an essential part of OSDA, the segregation has been only implicitly modeled because of its identification on the target-unknown instances. UADAL is the first explicit mechanism to simultancously align and segregate the three sets (source, sarget-known and target-unknown instances) via the domain adversarial learning. Therefore, the proposed unknown-aware feature alignment enables classifier to learn clear decision boundary on both known and unknown class in the feature space.
UADAL consists of three novel mechanisms. First, we propose a new domain discrimination loss to engage the target-unknown information. Second, we formulate a sequential optimization to enable the unknown-aware feature alignment, which is suitable for OSDA. We demonstrate that the optimized state of the proposed alignment is theoretically guaranteed. Third, we also suggest a posterior inference to effectively recognize target-knowwnluuknww information without any thresholding.
2
Preliminary
2.1
Open-Set Domain Adaptation
This section provides a formal definition of OSDA. The fundamental properties are two folds: 1) the different data distributions of a source domain, [2, y), and target domain, pe(a,, y); and 2) the ad- ditional classes of the target domain, which were not observed in the source domain. Specifically, we define the source and the target datasets as Xx = {(*), ) and X = { vt 3= respectively. V is not available training under the UDA setting. Additionally,
designate C, to be the source class set (a.k.a. a shared-known class), and C, to be the target class set, i.e., y E C. OSDA dictates C. Ct (23). C. \ C. is called unknown classes. In spite that there can be multiple unknown classes, we consolidate C," C. as Yunk to be a single unknown class, due to no knowledge on C. \ C..
The learning objectives for OSDA become both 1) the optimal class classification in Xt if a target instance belongs to C., and 2) the optimal unknown classification if a target instance belongs to C2 \ C,. This objective is formulated as follows, with a classifier f and the cross-entropy loss function Cee.
min (x.)) [. Fon (f(,). v)) + 1 Iy,e(c,,,)) (f Bunk)) )].
(1)
2.2 Adversarial Domain Adaptation
We first step back from OSDA to Closed-set DA (CDA) with C, \ C. = Φ. where the domain adversarial learning is widely used. Domain-Adversane Neural Network (DANN) 7 proposes,
min max max(1Ep, (x. f.G D
(C ((GG..) vo)l - La}.
(2)

---

Meneee laknewn
Feature Extractor
Discriminator D
Source
E
Posterior
X
Estrary
Ressentet
Target-Known
X;
G(x,)
X
G(X))
c
9t
Target-Urknown
Food Forword
Input
Feature
p(Anoww))))
Domaln Lass
-- Gradicnt
p(known]x,)
Feed-Forrwad
Minimiza
Maximize > No Gradient Signal
(a) The overall structure of UADAL
(b) The optimization procedure of UADAL
Figure 2: Overview of the proposed Unknown-Aware Domain Adversarial Learning (UADAL.) approach.
This objective assumes Gzx) to be a feature extractor that learns the domain-invariant features, enabled by the minimax game with domain discriminator D, with respect to Ld, as below.
L = (x) [- log (G(())) + pp log D,(G())))
(3)
The adversarial framework adapts (iz) toward the indistinguishable feature distributions between the source and the target domain by the minimax game D(G(z)) [D, (G(((z)) Here, D is set up to have a two-dimensional output to indicate either source or target domain, denoted as D2 and Dt, respectively. Given these binarized outputs of D, this formulation is appropriate to differentiate the target-know and the target-unknown features in OSDA. DANN enforces to include the aarget-unknown features in the distribution matching, which leads to performance degradation by negative transfer [6]. Figure lb
represents this undesired alignment of unknown classes.
Separate To Adapt (STA) 13 proposes weighted domain adversarial learning, to resolve the negative transfer by utilizing a weighting scheme. Therefore, STA modifies La as follows,
CSTA
E px)) [- log (.(G(=))) + pp) w. log (G((()))
(4)
where wwr represents a probability of an instance x belonging to the shared-known classes. L enables the domain adversarial learning align the features from the source and the most likely target-known instances with the high weights. However, the feature extractor, G, is not able to move apart the target-unknown instances due to lower weights, i.e., alignment signals (see Figure Tep. STA modeled this known-only feature alignment by assuming the unknown feature would be implicitly learned through the estimation w,. Besides learning Wix, the target-unknown instances cannot contribute to the training of STA because their loss contribution will be limited by lower Wa. in the second term of Eq. 4.. Therefore, STA does not take information from the target-unknowns, and later Section 4.3 empirically shows that the target-unknown segregation of STA is limited.
2.3 Comparison to Recent OSDA Research
In terms of the domain adversarial learning, in addition to STA, OSBP (23 utilizes classifier to predict the target instances to the pre-determined threshold, and trains the feature extractor to deceive the classifier for aligning to known classes rejecting as unknown class. However, their recognition on unknown class only depends on the threshold value, without considering the data instances. PGL [19] introduces progressive graph-based learning to regularize the class-specific manifold, while jointly optimizing the feature alignment via domain adversarial learning. However, their adversarial loss includes all instances of the target domain, which is critically weakened by the negative transfer.
On the other hand, self-supervised learning approaches have been proposed recently, in order to exploit the intrinsic structure of the target domain [5] 24 [2]. However, these approaches do not have any feature alignments between the source and the target domain, which leads to performance degradation under significant domain shifts. There is also a notable work, OSLPP [3] optimizing projection matrix toward a common subspace to class-wisely align the source and the target domain. However, the optimization requires pair-wise distance calculation, which results in a growing complexity of Onn²) by the n data instances. It could be limited to the large-scale domain. We provide detailed comparisons and comprchensive literature reviews in Appendix A
3

---

3 Methodology
3.1 Overview of UADAL
Figure(2al 2a illluutttttss the neural network compositions and the information flow. Our model consists of four networks: 1) feature extractor, G; 2) domain discriminator, D; 3) open-set recognizer, E; and 4) a class classifier network, C. The networks G, D, E, and C are parameterized by 0g. Od. fe. and d., respectively. First, we propose new domain discrimination loss, and we formulate the sequential optimization problem, followed by theoretical analyses. In order to recognize target- unknown information, we introduce a posterior inference method, followed by open-set classification.
3.2 Sequential Optimization Problem for Unknown-Aware Feature Alignment
Model Structure We assume that we identify three domain types, which are 1) the source (s). 2) the target-known (tk), and 3) and target-unknown (tu). As the previous works such as STA and DANN utilize a two-way domain discriminator with (Da. Dz), they can treat only s and rk (or t). However, a domain discriminator in OSDA should be able to handle tu information for segregation. Therefore, we designed the discriminator dimension to be three (Da, Dek. Dru). as follows,
D(G(=)) (Dx((())), Dua(G(z)) (G=))
(5)
Given these three dimensions of D, we propose new domain discrimination losses, Cg and C3
(0 0a) (x) |- log D,
(6)
(0, 02) = [x)) u log Du<(G(z)) - (1 wx)) log Dru
(7)
where Whg == ((Koown]aa (r), named as open-sel recognition, is the probability of a target instance, ar belonging to a shared-known class. We introduce the estimation of this probability, Wzz, in the section 3.4 later. From L32 in Eq. (7). our modified D) becomes to be able to discriminate ik and tu, explicitly.
Our new unknown awareness of D allows to segregate the target-unknown features via the domain adversarial learning framework. Firstly, we decompose (02, , 0a) into C (0g, 0a) and C (0,, Oa).
E2 0a) = =t' (0,, 0a) + C (0,, 0a), C (0,. 0s) (ex | log Des 10po<G((S (G(x))],
(8)
(9)
L/ (0g 0d) >tuTEp log D.. (G()))
(10)
where Pek (x) Pe(x: known), Ptu (I) == pr(2)) (nkknow)) duk = p pkknown), and Xeu pl ((unknown). The derivation of Eq. 8 is based on (x AesPPe (x) + )tuPru(z)) which comes from the law of total probability (Details in Appendix B. Therefore, this decomposition of 2 into C and Ctk enables the different treatments on tk and ru feasible. The three-way discriminator and its utilization in Eq. (0) becomes the unique contribution from UADAL.
Optimization Our goal of this unknown-awand domain discrimination is to align the features from the source and the target-known instances while simultaneously segregating the target-unknown features. To achieve this goal, we propose new sequential optimization problem w.r.t. G and D. Based on the losses, (0 Oa), (0,, 0a). and L (0,, 0a), we formulate the optimization problem as, min Ln(0g, 0z) = (0 0a) + ct (0g. 0a) + C. (0,, 0a),
(11)
0a
Lo(0g, 0a) = C0 0a) + C (0,, 0a) - C.a (0,. 0a)-
max 8,
(12)
This alternating objective by Eq. and Eq. 12 equivalent the adversarial domain adaptation models [7] [3] to learn the domain- invauiinnniiint features. Unlike the previous works, however, Eq. (IT) represents that we train the domain discriminator, D. to classify an instance as either source, target-known, or target-unknown domain. In terms of the feature extractor, G, we propose Eq. (12) to maximize the domain discrimination loss for the source and the target-known domain while minimizing the loss for the target-unknown domain. From the different signals on Ltk and 2 we only treat the adversarial effects on L and Lt Mcanwhile, minimizing L provides for the network G to learn the discriminative features on Iu. Eventually, G and D align the source and target-known features and segregate the target-unknown features from the source and the target-known features (the optimization details in Figure 2b). We provide the theoretic analysis of the optimized state of the proposed feature alignment in the next subsection, which is unexplored in the OSDA field.

---

3.3 Theoretic Analysis of Sequential Optimization
This section provides theoretic discussion on the proposed feature alignment by our sequential optimization, w.r.t. Eq. III ( Since our ultimate goal is training G to learn unknown-awand feature alignments, we treat G' as leader and D as a follower in the game theory. We first optimize Eq. (11) to find D" with fixed G. The optimal D" given the fixed G results in the below output
(2) P=(2) >esPoe(z) )euPtn(2) 2Pavg (2)
(13)
2pac9 (2)
where Pavg(2) (P+(=) (x) AeuPtn (2)) /2 (Details in Appendix 1.1.2).
Here, 2 stands for the feature space from G. i.e., Pa(z) = (G(:; gg)lc ~ pa(x)) where d is 8, tk, or tu. Given D with the optimal parameter 02 we optimize G in Eq. (12). Here, we show that the optimization w.r.t. G is equivalent to the weighted summation of KL Divergences, by Theorem 3.1
Theorem 3.1. (Proof in Appendix B. [BI.3) Let f7 be the optimal parameter of by optimizing Eq.
[Ty. Then, -La 82) can be expressed as, with constant Co-
-Lo 03) = DrL. (Psl(PalPaa))) + XeR DR k (Poog) - Xeu Dr... (P (Paog)))Pg))))))))))))) ) Co-
(14)
We note that Ps: Pek, and Pru are the feature distribution of each domain, mapped by G, respectively. Therefore, we need to minimize Eq. (14) to find the optimal G". Having observed Theorem 3.1 Eq. (14) requires Ptu to have a higher deviation from the average distribution, Poogs while matching towards ps RS Pang and Pek R Paug- From this optimization procedure, we obtain the domain-invariant features over the source and the target-known domain, while segregating the target-unknowns, which is formalized P& R Pik and Ptu <> {Pt. P&). We show that minimizing Eq. 14) does not lead to the negative infinity by the third term, regardless of the other KL Divergences, by Proposition 3.2
Proposition 3.2. (Proof in Appendix 1.1.4 D Dr.. (prm (otgg) is bounded log 2 log Atu-
This boundness guarantees that the first two KL Divergence terms maintain their influence on optimizing the parameter 0g of G stably while segregating the target-unknown features. Furthermore, we show Proposition 3.3 to characterize the feature alignment between and prk. under Eq. (14).
Proposition 3.3. (Proof in Appendix B. 1.5) Assume that supp(p.) n supp(pea) 0 and supp(pek)
(ppp(p = 0. where supp(p): := {a E 2(p(2) > 0} is the support set of Then, the minimization w.r.l. G, by Eq. 444. is equivalent to the minimization of the summation on two f-divergences:
D (P& llpe)) + Axx (Peelle)))
where Ji(1) = -u .olog (I-a)uto and f2(u) wloog cut((1-j. with minimum of Eq. (4 is achieved if and only if ps PPkk
Therefore, the "+Acx"
Auk
Therefore, under the assumption, Proposition 3.3 theoretically guarantees that the source and the aurget-known feature distributions are aligned, which represents Pa Ptk. From these analyses, can theoretically guarantee both alignment and segregation from the proposed optimization.
3.4 Open-Set Recognition via Posterior Inference
This section starts from estimating w, = p(known|zz) for a target instance, x. First, we utilize the labeled source dataset, Xs, to train the feature extractor G and the open-set recognizer F,, as below,
(0 02) =
Σ
(BG((((()))
(15)
(Hx.NA)EEx
where E(G())) E cc.. contains the softmax activation. We note that the open-set recognizer, E, is classifier to learn the decision boundary over C, by the source domain.
Given the decision boundary by E, our assumption for open-set set recognition includes two statements: for the target instances, 1) the higher entropy is caused by an uncertain classification case, and 2) unknown class may induce the higher entropy because there are no prior training instances the unknown classes. Based on two assumptions, the open-set recognizer, E, will provide higher entropy for the target-unknown instances than the target-known instances. Therefore, we consider the entropy value the open-set recognition indicator. We verified our assumptions empirically in Figure Sa
5

---

While we notice the entropy value from a single instance as the indicator between known and unknown, the open-set recognition should be holistically modeled over the target domain. Therefore, we model the mixture of two Beta distributions on the normalized target entropy values as below,
p(t)) = (e + >euP(
jamknown).
(16)
where the definition of dek and Aru. is the same as in the previous subsection. l. is the entropy value for the target instance, æ,, i.., lx H (E (E)()))) with the entropy function H. The Beta distribution is a perfect fit for this case of the closed interval support of [o, 1], which is the normalized range of the entropy. Thus, introduce the estimator uy of wz by Posterior Inference, from fitting the Beta (EM) algorithm (Details in Appendix B.2.T).
mixture model through the Expectation-Maxtnn
Xepp(2 (kowm)
xx := p(known|e
(17)
(known))))) + >eaP(Ea unknown)
where the denominator from Eq. (T6). Since the fitting process incorporates target information, the entire dataset is utilized to set more informative weighting without any thresholds. Here, A and Atn are also explicitly estimated by the EM algorithm since they are not given (see Eq.
(19 in
Appendix B.2.Tp whereas threshold hyperparameter is used by the previous researches [19 31
3.5 Open-Set Classification
Given the proposed feature alignments, we train a classifier, C, correctly classify the target instance over C, or to reject as unknown. Note that the classifier, C, is the extended classifier with the dimension of [C.l + 1 to include the unknown class. Firstly, we construct the source classification loss on Xs- Secondly. for the target domain, we need to learn the decision boundary for the unknown class. Based on the proposed open-set recognition, Wz, we train the classifier, C, as follows:
Les (os(0g00e)
1
Σ L) +
1 -1-w (1. TH! XxEx
Yunk), (18)
This (xu:U.)Ex,
where n2' is normalizing constant, and Yunk is the one-hot vector for the unknown class. Further- more, we incorporate entropy minimization loss for enforcing the unlabeled target instances to be recognized effectively 99 13 31. with the entropy loss function, LH,
Le (0g. Oz)
Σ Ln(C(G(x)))
(19)
Th x.EX:
3.6 Conditional UADAL
Domain adversarial learning may fail capture a class-wise discriminative pattern in the features when the data distributions are complex [17]. Therefore, we also experimented with the conditional UADAL (cUADAL) with the discriminative information between classes. Inspired by 17), we replace the input information of the domain discriminator, D, from G(xz) to [G(x), C(x) in Eq. (6)- 77 cUADAL has the conditions on the prediction of the extended classifier, C, which is including the unknown dimension. Therefore, cUADAL provides more discriminative information than UADAL.
3.7 Training Procedure
We introduce how update the networks, E, G, D, and C, based on the proposed losses. Before we start the training, we briefly fit the parameters of the posterior inference to obtain ú.. This starting stage trains and G' on the source domain with few iterations to catch the two modalities of the target entropy values, inspired by the previous work [. as below,
(de, 0,) = argming, C.).
(20)
After the initial setup of u.. we train D. C, E, and G, alternatively. This alternation is composed of two phases: (i) learning D; and (ii) learning G. C. and E networks. The first phase of the alternative iterations starts from learning the domain discriminator, D. by the optimization on LD in Eq. [I]
0₂ argming, Oa),
(21)

---

Barkhone (WV
Office-t
Offrr-Home
Model
A.W
A-13
13WW
WAD
D-A
W-A
Agg.
F3R
14:
P-A
A-F
A-6
A-C
RcA
R-F
C-
C-A
CPP
Avg
DruseNot-1 121 (7.00) EmmdeenNee 200 (5.3M)
DANN
65.2
12.7
92.6
14.8
63.7
57.3
14:1-113
K5.7
16.5
18.3
14.1
46.3
22.9
40.7
47.5
25.2
12.4
7.5
13:4
27
CDAN
65.5
73.6
92.4
94.6
64.5
57.9
74
57.9
18.1
20.4
35.6
47.0
24.6
44.1
49.8
56.1
13.5
8.9
15.0
28 28.8+:06
STA
58.3
62.2
81.6
79.6
69.5
67.4
09:8:112 1.2
99.4
43.6
51.9
53.5
60.6
49.5
59.8
53.5
49.9
53,4
09.5
49.4
50.54:02
OSaP
82.9
87.0
35.8
96-7
27:3
69.9
06.35 2.11
65.0
46.0
58.6
64.3
71.0
54.0
58.3
62.5
50.3
63.7
50.1
50.6
1808
69.2
80.1
99.6
79.0
59.3
79.4-0.
06.9
44.9
51.7
60.5
50.0
62.0
53.0
61.3
90.5
DANCE
6R.1
68.8
91.3
83.0
68.5
74.24
17.2
47.5
7.2
19.6
16.6
0.2
10:9
6.4
4.3
19.0
18.14:2
DCC
69.1
89.4
70-9-29
41.0
56.5
66.4
52.8
604
45.1
60.8
UATAL
74.1
12.9
5257
65.3
CUADAL
86.5
97.3
72.5
50.5
71.4
65.3
DANN
71.9
72.0
0013
R3.5
75.5
22.9
71.8-1.5
65..6
$5.4
48.7
63.6
71.9
45.3
62.6
66.7
45.9
62.3
630
CDAN
69.5
69.8
36.8
84.5
73.8
72.5
76.2-01
65.9
39.2
51.9
62.6
71.8
47.1
63.5
68.0
48.7
62.8
49.3
15.2
57.41.03
STA
77.0
68.6
86.0
77.2
76.5
75.1
16.411
63.5
46.1
59.4
58.8
64.3
50.4
62.5
58.6
51.1
61.0
560
50.9
57, 12.0
OS3P
$1.9
3:.0
88.9
96.6
73.1
74.9
83
1::2
71.9
46.0
50.3
67-1
72.3
54.5
85.9
71.1
53.7
96.8
16
1008
67.0
67.8
97.4
99.4
77-1
1
73.0
49.8
59.3
67.8
75.5
50.8
66.4
74.8
54.3
848
519
538
DANCE
69.9
71.9
73.62:03
51.8
59.7
58.2
43.4
41.3
586
1006
5
79.7
1.
75.1
58.0
56.5
71,3
550
62-3
UADAL
96.7
75.7
101
57.3
1030
1011
CUADAL
35.1
77.5
25.0
61.7
52.8
29.1
09.6
M.3103
RosNeL-30 SO (25.5M)
DANN
CDAN
64.9
63.1
71.3
66.8
36..7
34.3
80.5
12:5
72.7
1337
71.0
12.5
13.4
15.92115
69.7
47.2
44.5
59.5
36.3
65.1
03:2
76.7
71.0
51.2
52.9
65.4
66.0
67.5
14.4
52.7
50.9
67.1
6601
51.6
59.2
61.7
(AL 001220
SEAV
75.9
75.0
69.8
75.2
73.2
66.1
09.5
53.2
51.9
54.0
68.3
55.8
67.1
64.5
54.5
66.8
53.4
00.4
61. 61.11:03
OSRP*
62.7
67.3
75.1
7
73.9
61.3
72.9
55.1
66.7
70.6
64.3
64-7
PGLL-
74.6
76.5
72.2
60.5
70.
43.6
47.3
20.3
11.4
45.6
26.8
35.2
ROS"
82.1
96.0
17.9
77.3
0::0
60.6
60.1
68.8
68.6
58.9
60-2
66:2:003
DANCE
66.9
65.5
54.3
294
53.1
27.5
302
44.22:066
87.1
91.
65.5
64.0
59.5
674
90.6
52.9
56.0
02.7
67.0
89.8
66.6
64.2
OSI P9
07X
GUADAL
982
33
66.5
Table 1: HOS score (%) on Office-31 & Office-Home using EfficientNet-BO (5.3M) DenseNet-121 (7.9M) and Re (25. (5) as backbone network, where (ff) represents the number of the parameters. A-W in column means that A is the source domain and W is the target domain. Avg. is the averaged HOS over all tasks each dataset. (bold: best performer, underline: sccond-best performer, " officially reported performances.) The detailed experimental results including the other metrics such as OS* and UNK are in Appendix C.2.9
Based on the updated 0a. the parameter, 0g. is associated with both optimization loss and classification loss. Therefore, for the second phase, the parameters, 0g and 0z, are optimized below,
(0g. 0.) = argming,,0, Cole(0g, 0c) + Co 0e) - Lo 08))
(22)
We also keep updating 0g by Eq. (20). which leads to a better fitting of the posterior inference on the aligned features (empirically shown in Figure (9). Algorithm [ in Appendix [.3.I [ enumerates the detailed training procedures of UADAL. Computational Complexity is asymptotically the same as existing domain adversarial learning methods, which linearly grow by the number of instances ((n). The additional complexity may come from the posterior inference requiring O(nk); & is a constant number of EM iterations (the detailed analysis the computations in Appendix BB3.2 and C.2.8).
4 Experiments
4.1 Experimental Settings
Datasets We utilized several benchmark datasets. Office-31 (22 consists of three domains: Amazon
(A), Webcam (W), and DSLR (D) with 31 classes. Office-Home 30 is a more challenging dataset with four different domains: Artistic (A), Clipart (C), Product (P), and Real-World (R), containing 65
classes. VisDA 21 is a large- dataset from synthetic images to real one, with 12 classes. In terms of the class settings, we follow the experimental protocols by 23
Baselines We compared UADAL and cUADAL with several baselines by choosing the recent works in CDA, OSDA, and Universal DA (UniDAA). For CDA, we choose DANN 77 and CDAN 17 to show that the existing domain adversarial learning for CDA not suitable for OSDA. For OSDA, we compare UADAL to OSBP (23). STA [3]. PGL 19], ROS (5), and OSLPP [3I] For UniDA, we conduct the OSDA setting in DANCE
2 and DCC 122
Implementation We commonly utilize three alternative pre-trained backbones: 1) ResNet-50 10, 2) DenseNet-121 [II), and 3) EfficientNet-BO [26] to show the robustness on backbone network choices, while using VGGNet (25) on VisDA for a fair comparison. (Details in Appendix C.I).
Metric We utilize HOS, which is a harmonic mean of OS*, and UNK 55 OS* is class-wise averaged accuracy over known classes, and UNK an accuracy only for the unknown class. HOS suitable for OSDA because it is higher when performing well in both known and unknown classifications.
7

---

Method
O8*
ElbcientNet-B BR
UNK
HOS
OS
De-
UNK
HOS
OS*
ResNet-50
UNK
HOS
OS*
VGGNet
UNK
HOS
Weight
Othee:31 with ResNet-S0
Core
HOS Avg.
STA
49.3
56.4
52.5
53.1
76.7
62.7
36.9
75.8
65.0
63.9"
84.2"
72.7
STA
72.5#0.
OSBP
PGL
56.9
70.4
26.1
35.8
48.7
70.3
77.2
33.1
15.3
$2.8*
85.1"
68.1"
69.8"
74.7"
STA
UADAL
85,5+09
DANCE
ROS
36.7
58.9
72.0
63.2
48.7
48.1
42.7
59.8
81.1
67.3
62.3
55.9
61.3
45.8
64.8
72.9
53.7
66.5
ROS
UADAL
ROS
85,920.2
86.7+0.2
DCC
54.2
44.8
16.7
27.2
88.0
23.3
68.0"
73.6"
70.7"
UADAL
UADAL
88.1+0.2
UADAL
47.0
76.8
58.3
56.2
81.5
66.5
58.0
86.2
69.4
63.1
93.3
75.3
Table 3:
Ablation Studies for Different Weighting / Core
eUADAL
47.2
76.5
58.4
57.9
84,1
68.6
58.5
87.6
70.1
64.3
92.6
75.9
Table 2: Results on VisDA dataset. (bold/underlinef": Refer to the Table
Schemes on STA and ROS.
4.2
Experimental Results
Quantitative Analysis Table I reports the quantitative performances of UADAL,, cUADAL., and baselines applied to Office-3T and Office-Home, with three backbone networks. UADAL and cUADAL show statistically significant improvements compared to all existing baselines in both Office-31 and Office- Moreover, there is a large improvement by UADAL from baselines in EfficientNet-BO and DenseNet-121. even with fewer parameters. In terms of the table on ResNet- 50, for fair comparison, UADAL outperforms the most competitive model, OSLPP (HOS=67.0), Compared to the domain adversarial learning methods, such as STA and OSBP, UADAL significantly outperforms in all cases. UADAL also performs better than other sulf-supervised learning methods such as ROS and DCC. These performance increments are commonly observed in all three backbone structures and in two benchmark datasets, which represents the effectiveness of the proposed model.
Table 2 shows the experimental results on VisDA, which is known to have a large domain shift from synthetic to real domain. It demonstrates that UADAL and cUADAL outperform the other baselines
3 Jnder the significant domain shift. Moreover, the baselines with the feature alignments by the domain adversarial learning, such as STA, OSBP, and PGL, outperform the self-supervised learning models, ROS, DANCE, and DCC, in most cases. Therefore, these results support that OSDA essentially needs the feature alignments when there are significant domain shifts. The detailed discussions about the low accuracies of some baselines are in Appendix C.2.2, In addition, we observe that cUADAL mostly performs better than UADAL, which shows that cUADAL provides more discriminative information both on known and unknown by the prediction of the classifier C.
STA
ROS
HOS
UADAL
(a) DANN (b) STA
(c) cUADAL
Proxy A-Distance (PAD)
0.0
0.6
1.2
1.8
10 of Known Classes
Figure 3: t-SNE Visualization on D W of 31fice-31. Figure 4: PAD value on (Blue/Red/Grreen: Kongee-KnowwnnUUnnnwo//SS
Figure 5: Openness on
ik/tu feature distributions.
Office-31 31 (ResNet-50).
Qualitative Analysis Figure represents the t-SNE visualizations 22 of the learned features by ResNet-50 (the details in Appendix C.2233. We observe that the target-unknown (red) features from the baselines are not segregated from the source (green) and the target- (blue) features. On the contrary, UADAL aligns the features from the source and the target-known instances accurately with clear segregation of the target-unknown features. In order to investigate the learned feature distributions quantitatively, Figure provides Proxy A-Distance (PAD) between the feature distri- butions from STA, ROS, and UADAL. PAD is an empirical measure of distance between domain distributions S (Details in Appendix C.2.4). and a higher PAD means clear discrimination between two distributions. Thus, we calculate PAD between the target-known (tk) and the target-unknown (tu) feature distributions from the feature extractor, G. We found that PAD from UADAL is much higher than STA and ROS. It shows that UADAL explicitly segregates the unknown features.
4
In order to provide more insights and explanations compared to STA, we provide an analysis of the correlation between the evaluation metric and the distance measure of the feature distributions. We utilize UNK, and HOS as evaluation metrics, and PAD between th and tu as the distance measure. Figure the scatter plots of (UNK& & PAD) and (HOS& & PAD). The grey arrows mean the corresponding tasks by UADAL and STA. As shown in the figure, HOS and UNK have a positive correlation with the distance between the target-unknown (fu) and the target-known (tk). In simple
FOSLPP is infeasible to large dataset, such as VisDA dataset (Detailed discussions in Appendix C.2.1 1.2.1
8

---

UADAL
STA
W D
100
UADAL
55
STA
1-W
UNK( (1))
D W+A
HOS(%))
10
A-D
D-A
A-+ D-W
D
D-
103 100
LB5 130
PAD between tk and tu Figure 6: Correlation analysis between the PAD value and each of UNK (left) and HOS (right), in Office-31 dataset (with ResNet-50). The arrow represents the same task, such A- for UADAL and STA.
PAD between tk and tu
words, better segregation enables better HOS and UNK. Specifically, from STA to UADAL on the same task, the distances (PAD on tk/tu) are increased, and the corresponding performances, UNK and HOS, are also increased. It means that UADAL effectively segregates the target feature distributions, and then leads to better performance for OSDA. In other words, the proposed unknown-awand feature alignment is important to solve the OSDA problem. From this analysis, our explicit segregation of tu is a key difference from the previous methods. Therefore, we claimed that this explicit segregation of UADAL is the essential part of OSDA, and leads better performances.
Robust on Openness We conduct the openness experiment to show the robustness of the varying number of the known classes. Segure/rrpresenttt that UADAL performs better than the baselines over all cases. Our open-set recognition does not need to set any thresholds since we utilize the loss information of the target domain, holistically. Therefore, UADAL is robust to the degree of openness.
4.3 Ablation Studies
Effectiveness of Unknown-Aware Feature Alignment We conducted ablation studies in- vestigate whether the unknown-aware feature alignment is essential for the OSDA problem.
repreents the experimental results where the feature alignment procedure of STA or ROS is replaced by UADAL, maintaining their weighting schemes, in Office-31 with ResNet-50. terms of each weight- ing, the UADAL core leads to better performances in both STA and ROS cases. Surprisingly, when we replace STA's known-only matching with the unknown-aware feature alignment of UADAL, the performance in- creases dramatically. means that the implicit separation of STA at the classification level is limited to segregating the harget-unknowe Therefore, explicit segregation by UADAL is an essential part to solve the OSDA problem. Moreover, Figure 7 shows the empirical convergence of the proposed sequential optimization as PAD value between Ps and Pek and between ps and Ptu over training, i.e., Ps R3 Ptk and ptu ++ {pek. Ps}.
A-W in Office-31 (ResNet)
(s. m/
PAD 1ETr
Training Iterations
Figure 7: Convergence of Sequential Optimization.
features.
by Posterior
Inference
Auti 201
Known
Binany Finany" Accuracy
HOS Score
Frequency
Fresunce
Unkkoown
Posteriorr Entropy
Lereay inttal /pdes
Normalized Entropy
Normalized Entropy
0. 03
0. Entropy Thresholding
(a) Early stage of training
(b) End stage of training
(a) Binary Ace. (b) Ablation Studies
Figure 8: (a) The histogram of the source and target entropy with the fitted posterior inference model, at the Early/ / End stage of training, where the subfigure in (a) shows the histogram of vù at this stage.
Figure 9: (a) Binary accuracy on tk/tu by thresholding dh. (b) Ablations for posterior inference with thresholding.
Superiorities of Open-set Recognition: (Effective Recognition) Figure represents the histogram
of the source/target entropy values from E after the early stage of training by Eq. (20) (sensitivity analysis on the varying number of iterations for the early stage Appendix C.2.5) Therefore, demonstrates that our assumption two modalities of the entropy values holds, robustly. Further- more, the subfigure Figure shows the histogram of the ú. by the posterior inference. It shows that our open-set recognition, Wz, provides more clear separation for the target-known (bluc) and

---

the unketw (red) instances than the entropy values. Figure represents the histogram of the entropy the end of the training, which is further split than Figure Sal This clear separation implicitly supports that the proposed feature alignment accomplishes both alignment and segregation.
(Informativeness) Furthermore, we conduct binary classification task by thresholding the weights, Le. whether it predicts th or tu correctly. Tigure(alrepresent the accuracies over the threshold values when applying the posterior inference or the normalized entropy values themselves, in Office-31. It shows that the posterior inference discriminates tk and tu more correctly than using the entropy value itself. Additionally, the posterior inference has similar accuracies over the threshold values (0.2, 0.5, and 0,8), which also indicates a clear separation. Therefore, we conclude that the posterior inference provides a more informative weighting procedure than the entropy value itself.
(Comparison to Thresholding) Figure 9b shows that UADAL with the posterior inference outper- forms all cases of thresholding the entropy values. It means that the posterior inference is free to set thresholds. Moreover, the results demonstrate that UADAL with the posterior inference outperforms the case of using the normalized entropy value, and its frequent updating (solid line) leads to better performance than only initial updating (dashed line). It is consistent with the analysis from Figure Sb
350
3200
72
Dataset
Ent.
OS
UNK
HOS
Wall-Clock Time (s)
t:
HOS (96)
Wall-Clock Time
300
HOS (%)
Office-31
X
61.9
84.8
85.4
90.0
82.1
75.2
88.1
67.6
87.5
600
Office-Home
62.6
78.0
68.7
50-
300
Table 4: Ablation on Entropy Minimiza-
HOS
tion (Ent.) Loss from Eq. [19] the
Target Sampling Ratio (%)
10
50
100
Target Sampling Ratio (%)
0.1
1
10
50
100
Office-31 and Office-Home. The values
in the table are the averaged score over the tasks in each dataset. The detailed
Figure 10: Ablation study of applying sampling on the target domain with Office-31 (left) and Office-Home (right) w.r.t. the averaged scores.
results are in Table inn Appendix C.2.7
(Efficiency) In terms of computation, the posterior inference increases the complexity because need to the mixture model, where the computational complexity is ((nk) with the number of samples, n, and the number of fitting iterations, & (n >>>>>>>>>>> k). As an alternative, model only by sampling the target instance. Figure 10 represents the wall-clock time increased by the fitting process and the performances over the sampling ratio (%) for the target domain. Since the computational complexity is linearly increased by the number of samples, the wall-clock time is also linearly increased by increasing the sampling ratio. Interestingly, when the sampling ratio is small, i.e. 10%, the performances do not decrease, on both Office-31 and Office-Homn datasets. The qualitative analysis of this sampling is provided in Appendix C.2.8
fit the mixture
observed that
Entropy Minimization Loss order to show the effect of the entropy minimization by Eq. (19). conduct the ablation study the Office-31 and Office-Homn Home. Table shows that the entropy minimization leads to better performance. However, UADAL without the loss still performs better than the other basclines (compared with Table It represents that UADAL learns the feature space appropriately as we intended to suit Open-Set Set Domain Adaptation. The properly learned feature space leads to effectively classifying the target instances without entropy minimization.
Conclusion
We propose Unknown-Aware Domain Adversarial Learning (UADAL), which is the first approach to explicitly design the segregation of the unknown features the domain adversarial learning for OSDA. We design a new domain discrimination loss and formulate the sequential optimization for the unknown-awar feature alignment. Empirically, UADAL outperforms baselines on all benchmark datasets with various backbone networks, by reporting state-of-the-an art performances.
Acknowledgments and Disclosure of Funding
This research was supported by Research and Development on Key Supply Network Identification, Threat Analyses, Alternative Recommendation from Natural Language Data (NRF) funded by the Ministry of Education (202
10

---

References
[1] E. Arazo. D. Ortego, P. Albert, N. O'Connor, and K. Meguinness. Unsupervised label noise modeling and loss correction. In International Conference on Machine Learning, pages 312-321, 2019.
[2] M. Baktashmotlagh,, M. T. Harandi, B. C. Lovell, and M. Salzmann, Domain adaptation on the statistical manifold. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2481 2481-2488, 2014.
[3] S. Ben-David, J. Blitzer. K. Crammer, A. Kulesza, Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine learning, 19(1):55-177, 2010.
[4] K. M. Borgwardt, A. Gretton, M. Rasch, H.-P. Kriegel, B. Schölkopf, and A. J. Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformaties, 22(14):c4--557, 2006.
[5] S. Bucci, M. R. Loghmani, and T. Tommasi. On the effectiveness of image rotation for open set domain adaptation. In European Conference on Computer Vision, pages 422-438. Springer, 2020.
[6] Z. Fang. J. Lu, F. Liu, J. Xuan, and G. Zhang. Open set domain adaptation: Theoretical bound and algorithm. IEEE Transactions on Neural Networks and Learning Systems, pages 1-14, 2020. doi: 11.9TTNTS..0000.00000
[7] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference machine learning, pages 1180- 1189. PMLR, 2015.
[8] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096- 2030, 2016.
[9] Y. Grandvalet and Y. Bengio. Semi-supervised learning by entropy minimization. In Advances in neural information processing systems, pages 529-536, 2005.
[10] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[11] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[12] G. Li, G. Kang, Y. Zhu, Y. Wei, and Y. Yang. Domain consensus clustering for universal domain adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9757-9766, 2021.
[13] H. Liu, Z. Cao, M. Long, J. Wang, and Q. Yang. Separate to adapt: Open set domain adaptation via progressive separation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 2927-2936, 2019.
[14] M. Long, Y. Cao. J. Wang. and M. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, pages 97-105, 2015.
[15] M. Long, Cao. J. Wang. and M. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML115, page 97-105. IMIR.org, 2015.
[16] M. Long, H. Zhu, J. Wang, and M. I. Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in neural information processing systems, pages 136- 144,144,2 2016.
[17] M. Long, Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. arXiv preprint arXiv: 1705. 10667, 2017.
[18] M. Long, H. Zhu, J. Wang, and M. Jordan. Deep transfer learning with joint adaptation networks. In International conference machine learning, pages 2208-2217. PMLR, 2017.
[19] Y. Luo, Z. Wang, Z. Huang, and M. Baktashmotlagh. Progressive graph lcarning for open-set domain adaptation. In International Conference on Machine Learning, pages 6468-6477 PMLR, 2020.
[20] S. J. Pan. I. W. Tsang, T. Kwok, and Q. Yang. Domain adaptation via transfer component analysis. JEEE transactions neural networks, 22(2):199 210, 2010.

---

[21] X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko. Visda: The visual domain adaptation challenge, 2017.
[22] K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In Proceedings the 11th European Conference Computer Vision: Part /V, ECCV" 10, page 213-226, Berlin, Heidelberg, 2010. Springer-Verte
ISBN 364215560X.
[23] K. Saito, S. Yamamoto, Y. Ushiku, and T. Harada. Open set domain adaptation by backpropagation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 153-168, 2018.
[24] K. Saito, D. Kim, Sclaroff. and K. Saenko. Universal domain adaptation through self supervision. Advances in Neural Information Processing Systems, 33, 2020.
[25] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint /409 1556, 2014.
[26] M. Tan and Q. Le. Efficientmet: Rethinking model scaling for convolutional neural networks. In Interna- tional Conference on Machine Learning, pages 6105 61 14, PMLR, 2019.
[27] E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: | 1412.3474 2014.
[28] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 7167-7176, 2017.
[29] L. Van der Maaten and G. Hinton. Visualizing data using t-snc. Journal of machine learning research, 9 (11), 2008
[30] H. Venkateswara, J. Fusebio, S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[31] Q. Wang, F. Meng, and T. P. Breckon. Progressively select and reject pseudo-labelled samples for open-set domain adaptation. arXiv preprint arXiv:2110. 12..555,, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Details in Appendix D
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Details in
Appendix D
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] We provide code
in supplemental material, and URL of data and instructions are provided in Appendix
C.155
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The details in Appendix CC..I
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
12

---

(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes]
See Appendix [CI.44 and C.I.5
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] We mentioned that we use the public benchmark dataset, in
Section 4.
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] We use image object recognition datasets, so there
personaly identifiable information offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "LEchD_smT7C", "reviewer": "Reviewer_whZG", "summary": "This paper proposed an unknown-aware domain adversarial learning framework for open-set domain adaptation (OSDA), where the target domain has a larger label set than that of the source domain. This also means that the model needs to see novel class samples during the adaptation, which brings extra difficulty for domain adversarial learning. This work tackles this problem by introducing a three-dim discriminator for aligning the source (Ds) and target known (Tk) while separating the target unknown (Tu). The authors provide both theoretical and experimental analysis to support this type of domain adversarial learning and their methods are compared with state-of-the-art OSDA works.", "questions": "1.Regarding weakness 1, how can UADAL handle situations where: (1) Entropy(Tu) ≈ Entropy(Tk), under the case that there is a clear domain shift between the source and target domains; (2) Entropy(Tu) < Entropy(Tk), where there might exist target-unknown samples that yield lower entropy than the target-known ones and cause possibly negative effect when models are optimized via Eq. (11) and Eq. (12).\n\n2.It is better to provide not just HOS but also OS, OS* and UNK for the Office-31 and Office-Home benchmarks (i.e., Table 1), as some recent and previous methods did not include HOS as the metric when reporting results.\n\n3.Comparisons with some recent OSDA works are missing, including JPOT (IJCAI’20) on Office-31 and Office-Home and LGU (WACV’22) on VisDA.\n\nReferences:\n\nJPOT: Xu, et al. “Joint Partial Optimal Transport for Open Set Domain Adaptation,” IJCAI, 2020.\n\nLGU: Baktashmotlagh, et al. “Learning to Generate the Unknowns as a Remedy to the Open-Set Domain Shift,” WACV, 2022.\n\n\n###################### After rebuttal ###################\n\nI would like to thank the authors' detailed comments which well addressed my concerns. I will keep my score unchanged.\n\n", "limitations": "The limitations stated in the supplementary materials, i.e., suppL281 – 289, are the limitations for the general DA problem, but not specifically for their own methods. It is the current paradigm of DA that cannot tackle streamlined data gathering and needs sufficient source domain categories, not just for this paper. More concrete limitations of the proposed methods in this paper are recommended included.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.This work proposed an unknown-aware domain adversarial learning (UADAL) framework to handle the OSDA problem and the idea of aligning Ds and Tk while separating Tu via explicit adversarial learning is new.\n\n2.This work provides theoretic analysis of the optimization process for the proposed UADAL.\n\n3.The experimental results show that the proposed UADAL can achieve better performance than the state-of-the-art OSDA methods.\n\nWeaknesses:\n\n1.The framework is based on the assumption that (as in L192) Tu will have higher entropy than Tk, which might be too strong to hold.\n\n2.Domain adversarial learning is known to be unstable during optimization and lacks interpretability.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "618cXtfKxUs", "reviewer": "Reviewer_aB36", "summary": "This paper addresses open-set domain adaptation (ODA) problem, where target domain contains some unknown categories not seen in the source domain. Starting from the widely used adversarial training, the authors propose to introduce the known class recognition into it. Furthermore, the authors propose to deploy posterior inference to estimate the probability of belonging to unknown class. Thorough quantitative and analysis experiments prove the efficacy of the proposed method.", "questions": "1. Did the authors try using only one classifier head $C$?\n\nAs the first $\\mathcal{C}_s$ dimension of $C$ can be directly utilized in the same way as $E$ did. Having only one unified classifier may improve the performance since two classifier heads may have disagreement.\n\n2. Ablation study of entropy minimization in Eq. 19.\n\nMy feeling about this entropy minimization is that it may improve performance a lot. It will be interesting to see the effect of this term.", "limitations": "Maybe one limitation is that the posterior inference takes much time if with large dataset.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. Motivation of the paper is clear, that is, the unknown separation should be explicitly considered into the current adversarial training (more specifically STA [1]), which is the first contribution of the paper.\n2. The probability of belonging to unknown is estimated by posterior inference, which is conducted on the whole target data. This avoid introducing extra hyperparameter to reject unknown categories as most existing ODA methods did. This is the second contribution.\n3. The paper proposes to use $|\\mathcal{C}_s| + 1$ -way classifier, which could directly lead to a decision boundary for unknown category. The deployed entropy minimization on it is expected to further improve the performance.\n4. The experiments including both quantitative ones and other analysis, such as thresholding analysis, open-set recognition ability compared to other mechanism in Fig.6 and 7.\n\n\n### Weakness\n\nThe method seems to be a little heavy, as the posterior inference demands access to all target data, and also the existing of two classifier heads ($E$ and $C$). Details in Questions below.\n\n\n### Reference\n[1]. Liu, Hong, et al. \"Separate to adapt: Open set domain adaptation via progressive separation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2YfIWlZ69LQ", "reviewer": "Reviewer_31hV", "summary": "This paper proposes to deal with the open-set domain adaptation problem by not only considering the known classes matching but also the target-unknown distribution segregating. Both the theoretical analysis and corresponding algorithm for dealing with the problem are provided. in general, the paper is well-motivated, and the experiments are solid.", "questions": "+ what are the key differences in the insights between the proposed method and previous methods?\n+ Why do some baseline methods only achieve very low performance? the reliability of the experimental comparisons are questionable.\n\n\n###################### After rebuttal ###################\n\nThe authors have addressed my main concerns. I will keep my original score as borderline accept.", "limitations": "No", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The proposed method has some novel ideas that separate the target unknown classes from the shared classes in the feature space. \n+ The estimation of the open-set recognition is achieved by posterior inference, which is interesting.\n+ Theoretical analysis is provided.\n+ The experiments are extensive and good results are achieved on several benchmark datasets.\n\nWeaknesses\n\n- The paper a little bit over claim their main observation and insights. The baseline methods, such as STA also claimed the unknown classes should be separated from the known ones. Though the proposed method is very different from STA as compared and discussed by the authors, the key insights are not brand new.\n\n- Though the difference between the proposed method and STA is compared, it is still unclear why the proposed method by separating the unknown classes from the known ones in the feature space with an adversarial learning framework is superior to STA with a classifier. More insights, explanations, and analyses would be better.\n\n- In the experiments, the implementation of other baseline methods is questionable. For example, the DANCE method only achieves very low performance on the Office-Home dataset with the EfficientNet backbone. It is unclear why the very low implementation results are achieved, leading to unreliable comparisons.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept