Unified Optimal Transport Framework for Universal Domain Adaptation
Authors: Wanxing Chang
Ye Shi
Hoang Duong Tuan
Jingya Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Universal Domain Adaptation
Optimal Transport

Abstract: Universal Domain Adaptation (UniDA) aims to transfer knowledge from a source domain to a target domain without any constraints on label sets. Since both domains may hold private classes, identifying target common samples for domain alignment is an essential issue in UniDA. Most existing methods require manually specified or hand-tuned threshold values to detect common samples thus they are hard to extend to more realistic UniDA because of the diverse ratios of common classes. Moreover, they cannot recognize different categories among target-private samples as these private samples are treated as a whole. In this paper, we propose to use Optimal Transport (OT) to handle these issues under a unified framework, namely UniOT. First, an OT-based partial alignment with adaptive filling is designed to detect common classes without any predefined threshold values for realistic UniDA. It can automatically discover the intrinsic difference between common and private classes based on the statistical information of the assignment matrix obtained from OT. Second, we propose an OT-based target representation learning that encourages both global discrimination and local consistency of samples to avoid the over-reliance on the source. Notably, UniOT is the first method with the capability to automatically discover and recognize private categories in the target domain for UniDA. Accordingly, we introduce a new metric H^3-score to evaluate the performance in terms of both accuracy of common samples and clustering performance of private ones. Extensive experiments clearly demonstrate the advantages of UniOT over a wide range of state-of-the-art methods in UniDA. 

Unified Optimal Transport Framework for Universal Domain Adaptation
Wanxing Chang!
Ye Shi
Hoang Duong Tuan?
Jingya Wang 1.3
University of Technology Sydney SShanghai Engineering Research Center of Intelligent Vision and Imaging
ShanghaiiTec University
{changwx, shiye, wangj Sahagyaa@hanngaittehhh edu. cn tuan. hoang@uts. edu.au au
Abstract
Universal Domain Adaptation (UniDA) aims to transfer knowledge from a source domain a target domain without any constraints label sets. Since both domains may hold private classes, identifying target common samples for domain alignment is an essential issue in UniDA. Most existing methods require manually specified or hand-tumed threshold values to detect common samples thus they are hard to extend more realistic UniDA because the diverse ratios of common classes. Moreover, they cannot recognize different categories among prgge--private samples as these private samples are treated whole. this paper, we propose to use Optimal Transport (OT) to handle these issues under unified framework, namely UniOT. First, an OT-based partial alignment with adaptive filling is designed detect common classes without any predefined threshold values for realistic UniDA. can automatically discover the intrinsic difference between common and private classes based on the statistical information of the assignment matrix obtained from OT. Second, we propose an OT-based target representation learning that encourages both global discrimination and local consistency of samples avoid the over-reliance on the source. Notably, UniOT is the first method with the capability automatically discover and recognize private categories in the target domain for UniDA. Accordingly, we introduce a new metric H -core to evaluate the performance in terms of both accuracy of common samples and clustering performance of private ones. Extensive experiments clearly demonstrate the advantages of UniOT wide range of state-of-the-ar methods in UniDA.
Introduction
Deep neural networks have boosted performance in extensive computer vision tasks but still struggle to generalize well in cross-domain tasks that source and target domain data drawn from the different data distributions. Unsupervised Domain Adaptation (UDA) (28) aims to transfer knowledge from fully labeled source unlabeled target domain by minimizing the domain gap between source and target. However, existing UDA methods tackle the domain gap under a strong closed-set real-world
assumption that two domains share identical label sets, limiting their applications scenarios. Recently, Partial Domain Adaptation (PDA) 4 and Open Set Domain Adaptation (OSDA) [2] are proposed relax the closed-set assumption, allowing the existence of private classes the source and target domain respectively. However, all the above-mentioned settings heavily rely on
"Corresooding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

UniDA w/o PCD
Source-private classes set
Common samples
Target- private classes set
UniDA PCD
Target-private samples
Common classes
(a)
(b)
Figure I: (a) Realistic UniDA scenarios with diverse ratios of common classes. (b) Target representa- tion of UniDA with or without considering the Private Class Discovery (PCD).
prior knowledge where the common classes lie in the target domain. It may be not reasonable and realistic for UDA since the target domain unsupervised.
To address the above problem, a generalized setting, termed as Universal Domain Adaptation (UniDA) [3, was proposed to allow both domains to own private classes but without knowing prior information, e.g. matched common classes and classes numbers in the target domain. Detecting common and target-private samples in the target domain an essential issue in UniDA.
detect common and sanget-private samples by using some manually specified or hand-tuned threshold values. Therefore, these methods are not applicable to more realistic UniDA due to the diverse ratios of common categories as shown in Fig. [a). Moreover, most existing UniDA methods treat all target-private samples as a single class and cannot recognize different categories among them, as shown in Fig. [b). This paper is the first to emphasize that UniDA methods should have the capability automatically discover and recognize private categories in the target domain.
Existing UniDA methods [39
32
Essentially, the common class detection and private class discovery problems can be both viewed as distribution transportation problems. Therefore, these issues can be formulated within the framework of Optimal Transport (OT), which is promising optimization problem to seek efficient solution for transporting one distribution to another. Even though many OT-based methods [I] 24 38
I3
have been proposed for Unsupervised Domain Adaptation, most of them consider cross- rross-domain sample-to-sample mapping under the closed-set condition and are not specialized in UniDA problem with unaligned label sets. Specifically, OT encourages a global mapping to mine domain statistics property for discovering intrinsic differences among common and target private samples. Additionally, the OT constraints can also avoid degenerate solutions in clustering representation problem (. Inspired by this, OT can be a proper formulation with respect to inter-domain common class detection and intra-domain private class discovery.
In this paper, we propose a unified optimal transport framework, namely UniOT, to solve Universal Domain Adaptation from the perspectives of partial alignment for common class detection and target representation learning for private class discovery. With regards to common class detection, propose OT-based partial alignment detect common samples in the target domain and develop an adaptive filling method to handle the diverse ratios of common categories. Without any predefined threshold values, and private classes based on the global statistical information of the assignment matrix obtained from OT. For private class discovery, we propose an OT-based target representation learning that encourages both global discrimination and local consistency of samples to avoid the over-reliance on source supervision. In addition, UniOT has the capability automatically discover and recognize categories in the target domain, benefiting from the representation learning with OT. We believe competitive UniDA method should achieve high classificatinn accuracy for the common class and also learn a discriminative representation for the target private class. However, existing methods do not provide any quantitative metric to evaluate the target representation performance for those unknown samples. For target-private class discovery purpose, we introduce new evaluation metric, H3 -3-score, considering not only common class accuracy but also the clustering performance of private classes.
automatically discover the intrinsic difference between common

---

Our contributions are summarized as follows: (1) We propose UniOT to handle two essential issues in UniDA, including common class detection and private class discovery. To our best knowledge, this is the first attempt to jointly consider common class detection and private class discovery in a unified framework by optimal transport. (2) We propose an OT-based partial alignment with adaptive filling for common class detection without any predefined threshold values. It can automatically adapt to more realistic UniDA scenarios, where the ratios of common classes are diverse as shown in Fig. Ia). (3) We design OT-based representation learning technique for private class discovery, considering both global discrimination of clusters and local consistency of samples. Unlike most existing methods that treat all target-private samples as whole, our UniOT can automatically discover and recognize private categories in the target domain.
2 Related Work
39 is more challenging Universal Domain Adaptation As more generalized UDA setting, UniDA
and realistic since the prior information of categories is unknown. UAN 39 CMU 17 and TNT R designed sample-level uncertainty criteria to measure domain transferability. Samples with lower uncertainty are encouraged for adversarial adaptation with higher weights. Most UniDA methods detect common samples with the sample-level criteria, which requires some manually specified and hand-tumed threshold values. Moreover, over-reliance on source supervision under category neglects discriminative representation in the target domain. DANCE (32 proposed neighborhood clustering as a self--upervised technique to learn features useful for discriminating "unknown" categories. DCC [3 enumerated cluster numbers of the target domain to obtain optimal cross- domain consensus clusters as common classes, but the consensus clusters are not robust enough due to the hard assignment of K-Means (26.. MATHS 77 detected the existence of the target private class by Hartigan's 's dip test in the first stage, then trained hybrid prototypes by learning from K-Means assignment where fixed hyper-parameter for K-Means was adopted. In this paper, we use OT to handle both common sample detection and target representation learning under a unified framework. It is worth noting that our method is adaptive for various unbalanced compositions of common and private classes. which does not require any predefined threshold. In addition, our method encourages both the global discrimination and the local consistency of samples for target representation learning.
Optimal Transport in Domain Adaptation Optimal transport (OT) was first proposed by Kan- torovich 22 to obtain an efficient solution for moving one distribution of mass another. Interior point methods can handle OT problem with computational complexity at least (dd"log())) Cuturi [10 first proposed to use Sinkomn's 's algorithm [44 to compute an approximate transport coupling with an entropic regularization. This method is lightspeed and can handle large-scale problems efficiently. Later, DeepJDOT (11) used OT to address the domain adaptation problems to pair sam- ples in the source and target domain. Accordingly, a new classifier is trained on the transported source data representation. Since classical OT does not allow partial displacement in transport plan, unbalanced optimal transport (UOT) is proposed to relax equality constraint by replacing the hard marginal constraints of OT with soft penalties using Kullback-Leible (KL) divergence
[16), which
can also effectively be solved by generalized Sinkhorn's 's algorithm 99. JUMBOT [13 proposed a minibatch strategy coupled with unbalanced optimal transport, which can yield more robust behavior for handling minibatch UDA and PDA problem. Also, TS-POT (27 addressed the partial mapping problem in UDA and PDA with partial OT [4. However, these methods consider cross cross-domann sample-to-sample mapping under the closed-set condition or label-set prior, and are not suitable for the unknown category gap problem in UniDA. Besides, a sample-to-sample mapping neglects the global consensus of two domains and intrinsic differences among common and target private samples. In this paper, we propose an inter-domain partial alignment based on UOT for common class detection and intra-domani representation learning based on OT for private class discovery, mapping is designed to encourage a more global-awar assignment.
where a
Clustering for Representation Learning Deep clustering has raised increasing attention in deep learning community due to its capability of representation learning for discriminative clusters. DeepCluster (3) proposed an end-to-ent framework that iteratively obtains clustering assignment by K-Means (26) and updates feature representation by the assignment, while PICA (21) simultaneously learns both feature representation and clustering assignment. A two-step approach was proposed by SCAN (36), which firstly learns better initial representation, then implements an end-to-end clustering learning in the next stage. These deep clustering methods assume the number of clusters is
3

---

Common Class Detection
Prototypes
LOT
Adeptive Filling
z
Similarity Matris S
UOT Couplings
2
Memory Quene
Coop
Pezada Labellig.
Private Class Discovery
NN
LPCD
NN
Z
Targel Sample
Nearest Neighbor (NN)
Target Prototype
Target Prototypes
Similarity
Couplings
Figure 2: An overview of the proposed method. The UniOT mainly contains two branches: (I) Common Class Detection: We consider partial mapping problem between target samples and source prototypes based on UOT. Highly confident target samples selected by statistics mean are pesudo-labeled as common classes for partial alignment. Adaptive Filling is developed for handling the diverse ratios of common categories. (2) Private Class Discovery: We consider a full mapping problem between target samples and target prototypes based on for representation learning. Lgiobul encourages global discrimination of clusters and Lioent encourages local consistency of samples.
known beforehand. Self-labelling [1] and SwAV (6) formulated the clustering assignment problem into an OT problem for better feature representation learning. Notably, the number of clusters does not rely on the ground-truth categories, benefiting from the equality constraints in OT. Therefore, OT-based representation learning is more suitable for UniDA, where the number of private class unknown. However, existing OT-based representation learning methods [L] 6 consider global discrimination of clusters but ignore local consistency samples, limiting direct applications in UniDA. In this paper, we develop an OT-based target representation learning technique for both better global and local structures in UniDA.
Methodology
In Universal Domain Adaptation (UniDA), there is labeled source domain D* = {x] [xxxy' y. Y and an unlabeled target domain D' = \x Jim We use C, and C2 to denote the label set of source and target domain respectively. Denote C C2 C, as the common label set shared by both domains. Let Cet = C, \C and C, = C, \C represent label sets source private and target private, respectively. UniDA aims to classify target samples into |C] + classes, where target-private samples are treated as unknown class uniformly.
UniDA aims to transfer knowledge from source to target under domain gap and category gap (i.c., different label sets among source and target domain), which makes it challenging align common samples and reject private samples. Moreover, over-reliance on common knowledge extracted from the source domain neglects the target intrinsic structure. Therefore, this paper, we propose unified optimal transport framework to address UniDA problem from two perspectives, i.e., common class detection and private class discovery, as shown Fig.
3.1 Preliminary
Optimal Transport is constrained optimization problem that seeks an efficient solution of trans- porting one distribution of mass another. We now briefly recall the well-known optimal transport formulation.

---

Let Σ, : f f = 1} denote the probability simplex, where 1, is a d-dimensional vector of ones. Given two simplex vectors a E E, and , E Σ., we can write the transport polytope of a and [3 as follows:
(1) The transport polytope U(a (3) can also be interpreted as a set of all possible joint probabilities of (X, where X and Y are two d-dimensional random variables with marginal distribution a and g. respectively. Following SwAV [6], given similarity matrix M E Rexe instead of distance matrix in [10]. the coupling matrix (or joint probability) Q" mapping a to B can be quantified by optimizing the following maximization problem:
U(a,9)= (Q E M R QL. a, Q QT, 93.
OT (M, az a) 3)= argmax Tr(Q M) + +)) (=U(r.
(2)
where E > O and H(Q) - - Eij Qay log Qıy is the entropic regularization term. The optimal Q' has been shown to be unique with the form Q = Diag(u) exp (M/e) Diag(v), where n and v can be solved by Sinkhorn's Algorithm [10].
Since Q" satisfies the mass constraint U(a, 33) strictly, the objective OT (M, a, B) is not suitable for partial displacement problem. In light of this, Unbalanced OT is proposed to relax the conservation of marginal constraints by allowing the system to use soft penalties [9. The unbalanced OT formulated as
UOTE- (OT a. 3) = argmax Tr(QUMe + +H(()) <<Daa + (Q (, (3)
QER+
where DKL is Kullback-Leibler Divergence. Then the optimization problem can be solved by generalized Sinkhorn's 's algorithm 9
3.2 Inter-domain Partial Alignment for Common Class Detection
To extract common knowledge, we propose an Unbalanced OT-based Common Class Detection (CCD) method, which considers partial mapping problem between target samples and source prototypes from the perspective of cross domain. Note that our method uses prototypes to encourage a more global-awarg alignment, which discovers intrinsic differences among common and target- privvate samples by exploiting the statistical information of the assigmment matrix obtained from the optimal transport. Since UniDA allows private classes in both source and target domains, partial alignment should be considered to avoid misalignment between arget-private samples and source classes. Hence we formulate the problem mapping target features to the source prototypes C, [c] ... 9ic, with unbalanced OT objective to obtain the optimal assignment matrix Q*t, i.e.
1 1B, [C.] lc,,))
1
nd (S
"
where = C. refers to cosine similarity between target samples and source prototypes, and the mini-batch target noromalllde features Z [=i,... , 2/. T with batch-size B are extracted by feature extractor f, i.e. z = f(x))/ (x())) Here we use the same prototype-based classifier in [32 to learn source prototypes. Note that the private samples will be assigned with relatively low weights in Qat. Based on this observation, our CCD selects target samples with top confidence as common classes.
Firstly, the assignment matrix is normalized as Q = Q /E Then we discover statistical property from normalized Q and generate scores from the statistical perspectives of both target samples and source prototypes. For the i-th row in Qxt, can be seen as a vanilla prediction probability vector and confidence score ii with the maximum value of the i-th row in Q i.e.
get pseudo-label i by argmax operation. And we assign target samples
wf = max Q Q
Q.
(5)
A higher score / means z is relatively closer to a source prototype than any other samples and is more likely from common class. Meanwhile, to select target common samples with top confidence, we also evaluate source prototypes confidence score w with the sum of the j-th column, i.e.
B
w
Q
(6)
1=1
5

---

Analogously, higher score wj means cj is more likely a common source prototype, which is assigned to target samples more frequently. Then the common samples are detected by statistics mean, i.e.
w and w E
82
|c.1
0,
otherwise
where 6, indicates the sample x is detected common sample with top confidence, which can be assigned with pseudo label 1/ by argmax operation. We can use pseudo labels of selected target samples to compute inter-domain common class detection loss by standard cross-entropy loss, i.e.
LCCD EES 8; (cE(Z).))
8
E 5;
To ensure that statistics mean can discover the intrinsic difference between common and private classes, we need guarantee that the input features for partial mapping should be sampled from the target domain distribution with enough sampling density. Therefore, we implement a FIFO memory queue [19] to save previous target features for filling mini-batch features, which avoids the limitation led by the lack of statistical property for target samples.
Adaptive filling for unbalanced proportion of positive and negative. Unfortunately, using statistics mean as a threshold in Eq. (7) may misclassify some common samples as private class in some extremely unbalanced cases, such as there are few private samples indeed. Essentially, the assignment weight is determined by the relative distance between samples and prototypes, and private samples with low similarity design an adaptive filling mechanism for positive and negative unbalanced proportions. Our strategy is to fill the gap of unbalanced proportion to be balanced like Fig. deppcted. Firstly, samples with high similarity larger than a rough boundary are defined as positive and the others are negative. Then we implement adaptive filling to be balanced when the proportion of positive or negative samples exceeds 50%. For negative filling, we synthesize fake private feature zł by mixing up target feature zł and its farthest source prototypes evenly, i.c. 2. (z) + argmine; (z]C² )).
negative examples for the others. To automatically detect target samples,
For positive filling, we first do the CCD without adaptive filling by Eq.(7 to obtain confident features and then reuse these filtered confident features for positive filling. Note that the filling samples are randomly sampled from mix-up negatives or CCD positives and the filling size is the gap between unfilled positive and negative. After adaptive filling for balanced proportion, our CCD adaptive for detecting confident common samples.
Adaptive update for marginal probability vector. The marginal probability vector B in Objective (4) can be interpreted as a weight budget for source prototypes to be mapped with target samples. However, the source domain may own source private class, which is unreasonable set the weight budget equally for each source prototype, i.e. B = |c,l Ic,)- Therefore, we use the Q obtained in the last iteration to update /3r with the moving average for adapting to the real source distribution, i.c.
g(++1) = (3t(t) + (1 - (1) and (00)
1 C.
lc..
where 3(0) is the sum of column in Qat solved in the (09) case.
3.3 Intra-domain Representation Learning for Private Class Discovery
To exploit target global and local structure and feature representation, we propose an OT-based Private Class Discovery (PCD) clustering technique that considers full mapping problem between target sample features and target prototypes. Especially, our PCD avoids reliance on the number of target classes and encourages global discrimination of clusters as well as local consistency of samples.
To solve this, we pre-define K learnable target prototypes C, and initialize them randomly, where larger K can be determined by the larger number of target samples. Then we assign target features Z' with the prototypes C, by solving the optimal transport optimization problem:
1 1 KK
Q" nd OT (Stt.
1x, 12n),
(10)
2B

---

= C refers to cosine similarity between target samples and target prototypes, and Z' the concatenation of the mini-batch target features Z' and their nearest neighbor features. The marginal constraint of uniform distribution enforces that on average each prototype is selected at least 2B/K K times in the batch.
where =
The solution Q" satisfies the constraint strictly, i.c. the sum of each row equals to 1/K. To obtain prototype assignment for each target sample, the soft pseudo-label matrix Q = Q x B ensures the i-th row qf is a probability vector. To encourage global discrimination clusters, the learnable prototypes C and the feature extractor f are optimized by minimizing Lloobats i.c.
Eglobal
1 B
Σ
2,),
(11)
where the cross-entropy loss E(q: 7;) is presented as
k Σ k=1
exp (z Cx/T) exp (z: /T)
f(qi, 2i) =
Pitk log
where
PiA
(12)
To encourage local consistency of samples, we swap prediction between anchor feature z and its nearest neighbor feature ž retrieved from the memory queue. Note that instead of swapped prediction between the different views of anchor image in SwAV [6] our swapped Liocal is based on anchor-neighbor swapping, i.e.
Liocal
1
Σ 1=1
E(a 2/) + E(q 2))
(13)
2B
Eventually, to encourage both global discrimination of clusters and local consistency of samples, our CPOD is presented as
LPCD 1 (Cglobat + Llocat).
(14)
Similar to 3.2 since OT encourages more deliberative mapping under marginal constraint, we also need to fill input features with the previous target features in memory queue before solv solving the optimization problem (T).
3.4 A Unified Framework for UniDA by Optimal Transport
In UniDA, the partial alignment for common class detection and representation learning for private class discovery are based on UOT and OT models, respectively. Therefore, we can summarize the above two models in a unified framework, i.e. UniOT. Considering cross-entropy loss Cets on source samples, Common Class Discovery loss LOOD and Private Class Discovery loss LPCD, our overall objective can be expressed as
Coverall = Cets + X(LOOD + LPCD).
(15)
In the testing phase, only feature extractor f and prototype-baat based classifier C, preserved. We solve the optimization problem (4) for all the concatenated target features and adaptive filled features. Then compute w in Eq. 5 for each sample. For those samples who satisfy wf 2 assigned with the nearest source class, where is the sum of the total size of target samples ni, and adaptive filling size. Otherwise, the samples are marked unknown.
Experiments
4.1 Experimental Settings
Dataset. Our method will be validated popular datasets Domain Adaptation. Office contains 31 categories and about 4K images in 3 domains: Amazon(A), DSLR(D) and Wehcam (W). Office-Home (37] contains 65 categories and about 15K images in 4 domains: Artistic images(Ar). Clip-Art (ml), Product images(Pr) and Rerl-World (mages(Rw). VisDA 30 is a large dataset
7

---

which constains 12 categories in source domain with 15K synthetic images and target domain with 5K real- world images. DomainNet (29] the largest domain adaptation dataset which contains 345 categories and about 0.6 million in 6 domains but we only use 3 domain: Painting (P), Real (R), and Sketch (S) like [77. Note that we use A2W to denote that transfer task from Amazon to DSLR. We follow the dataset split in [7] to conduct experiments
Evaluation metric. For UniDA, the trade-off between the accuracy of common and private classes is important in evaluating performance. Thus, we use the H-score [17 to calculate the harmonic mean of the instance accuracy on common class ag and accuracy on the single private class ac,- However, H-score treats all private samples a single class. For surget-private class discovery purposes, we further introduce -ossc as the harmonic mean of accuracy on common class, accuracy on unknown class and Normalized Mutual Information (NMI) for target-private clusters, i.e.
3
H score- 1/ac- + 1/ac, + 1/NMI
(16)
where NMI is well-known measure for the quality of clustering, and NMI is obtained by K-Means with the ground truth of the number of private classes in the inference stage.
4.2 Experimental details
Our implementation of optimal transport solver is based on POT [.5. We employ pretrained Res- Net-50 (00 on ImageNet [12] as our initial backbone. The feature extractor consists of backbone and projection layers which are the same as [6] but BatchNorm layer is removed. We adopt the same optimizer details as [23]. The batch size is set 36 for both source and target domains. For all
experiments, the initial learning rate is set to 1 x 10-2 for all new layers and 1 x 10-3 for pretrained backbone. The total training steps are set to be 10K for all datasets. For the pre-defined number of target prototypes, a larger size of target domain indicates larger K. Therefore, we empirically set K = 50 for Office, K 150 for Office- K 500 for VisDA, K 1000 for DomainNet. We default y = 0.7, μ = 0.7, = 0.1, & 0.01, 0.5 and X = 0.1 for all datasets. We set the size of memory queue 2K for Office and officeHHome 10K for VisDA and DomainNet.
All experiments are implemented on a GPU of NVIDIA TITAN V with 12GB. Each experiment takes http:/awwww..///aaaa/////toooooo
about 2.5 hours. Our code is available at:
4.3 Results
Comparison with state-of-the-arts. Tab. andd [show the H-score results for Office, Office-Home, VisDA and DomainNet. Our UniOT achieves the best performance on all benchmarks. H-score non-UniDA methods are reported from [7), and UniOT methods are reported from original papers accordingly, except for DANCE (32 since they did not report H-score and reproduce it with their released code for fair comparison, marked as $ In particular, with respect to H -cose in Tab. 3 our UniOT surpasses other methods by 7% and 9% in Office and Office-Home datasets respectively, which demonstrates that our UniOT achieves balanced performance for both common class detection and private class discovery.
Office
DomainNet
A2D
A2W
D2A
D2W
W2A
W2D
Avg
P2R
R2P
Avg
ResNet
20
49.78
47.9
48.48
54.94
48.
55.60
50.94
30.06
28.34
26.95
26.90
26.89
29.74
28.15
DANN
50.18
48.82
47.69
52.73
49.33
54.87
50.60
31.18
29.33
27.84
27.84
27.77
30.84
29.13
KIN
50.18
50.21
47.65
54.68
49.28
55.24
51.21
32.27
30.29
28.71
28.71
28.63
31.90
30.08
IWAN 40
50.64
50.13
49.65
54.06
49.79
55.44
51.62
35.38
33,02
31.15
31.15
31.06
34.94
32.78
PADA
50.00
49.65
42.87
52.62
49.17
55.60
28.92
27.32
26.03
26.03
25.97
27.15
ATL 3
50.48
48.58
48.48
55.01
48.98
55.45
51.16
32.59
30.57
28.96
28.96
28.89
32.21
30.36
OSBP
51.14
50.23
49.75
55.53
50.16
57.20
52.34
33.60
33.03
30.55
30.53
30.61
33.65
32.00
DAN 39
59.68
58.61
60.11
70.62
60.34
71.42
63.46
41.85
43,59
39.06
38.95
38.73
43.69
40.98
CMU 17
68.11
67.33
71.42
79.32
72.23
80.42
73.14
50.78
52.16
45.12
44.82
45.64
50.97
48.25
DANCE 32
72.64
62.43
63.27
76.29
57.37
82.79
62.62
DCC
88.50
78.54
70.18
79.29
75.87
88.58
80.16
56.90
50.25
43.66
44,92
43.31
$6.15
49.20
INT
85.70
80.40
83.80
92.00
79.
91.20
85.37
UniOT
86.97
88.48
88.35
98.83
87.60
96.57
91.13
59,30
47.79
51.79
46.81
48.32
58.25
52.04
Table 1: H-score(()) on Office and DomainNet.
8

---

Offic-Homme
VisDA
A12C1
Ar2P:
Ar2Rw
C12Ar
C12P
C12Rw
Pr2Ar
Pr2C1
Pr2Rw
RwZAr
Rw2C1
Rw21t
Avg
ResNet 20
44.65
48.04
50.13
46.64
46.91
48.96
47.47
43.17
50.23
48.45
44.76
48.43
47.32
25.44
DANN I
42.36
48.02
48.87
45.48
46.47
48.37
45.75
42.55
48.70
47.61
42.67
47.40
19.19
25
RTN 23
38.41
44.65
45.70
42.64
44.06
45.48
42.56
36.79
45.50
44.36
39.79
44.53
42.89
26.02
TWAN 40
40.54
46.96
47.78
44.97
45.06
47.59
45.81
41.43
47.55
46.29
42.49
46.54
45.25
27.64
PADAA
34.13
41.89
44.08
40.56
41.52
43.96
57.04
35.84
23.05
A
39.88
45.77
46.63
44.13
44,39
46.63
44.73
41.20
46.59
45.05
41.78
45.45
44.35
26.34
OSBP 33
39.59
45.09
46.17
45.70
45.24
46.75
45.26
40.54
45.75
45.08
41.64
46.90
44.48
27.31
51.64
51.70
54,30
61.74
57.63
47.62
52.61
65.19
30.47
CMU 17
56.02
56.93
59.15
66.95
64,27
67.82
$4.72
$1.09
66.39
68.24
57.89
69.73
61.60
34.64
DANCES 32
26.67
11.27
18.03
33.17
12.50
14.35
41,56
39.92
33.34
16.31
27.12
25.86
25.01
DCC
57.97
58.01
74.64
70.62
77.52
73.60
74.94
80.96
75,12
80.38
70.18
TNT
61.90
74,60
80.20
73.50
71:40
79.60
74,20
69. 50
82 70
77.30
70.10
81,20
74.70
55.30
UniOr
67.27
80.54
86.03
77.33
84.28
75.54
63.33
85.99
65.37
81.92
76.57
57.32
Table 2: H-score((6) on Office-Home and VisDA.
Hone-Houe
A120
A2W
D2A
D2W
W2A
W3D
20%
AV2CI
A/2P:
Ar2R>
C.2.M
C.2
C12EW
P:N
P:2C
PV2RW
Rw2AMr
Ry2CC
Avg
Reeheel 200
46.81
41.42
50.35
40.35
45:57
41:40
50.54
46.18
DANCE 13
73.19
65.51
6161
40.95
52.63
08.25
22.76
48.50
58.39
40.97
DCC 2u
81.47
74.80
63.54
37
69
71.45
75
55
76.31
18.15
64
37.77
59
65.17
53.81
05 10
63.17
13.18
$0.09
$1.66
83.00
71.46
7843
90.54
52.34
60.11
76.72
69.62
79.54
Table 3: H2 (6) on Office and Office-Home
Evaluation of the effectiveness of the proposed CCD and PCD. To evaluate the contribution of LOCD. giobbal and Lisent, we train the model with different combination of each component. As shown in Tab the combination of Gginbal and Liseat, i.e. LPCD. brings significant contribution since target representation is crucial for distinguishing common and private samples. Especially, Loglobal contributes more than Lioeat, which demonstrates the benefit of global discrimination of clusters for representation learning. Besides, we also conduct experiments for CCD without adaptive filling and denote the loss as LCCD which verifies the effectiveness of adaptive filling design.
Office
OfficeTlomme
Office
Office-Home
Loop
Loco
Cysesl
Ang taxis)
C12Rw
taiss)
Ang
74.24
73.53
71:40
72.61
79:74
74.24
75.55
89.19
89.86
75.10
79.14
72.65
70.75
74.02
85.48
85.35
91.13
30.54
84.28
76.57
71,46
82.34
78.72
76.83
70,84
Table 4: Evaluation of the effectiveness of the proposed CCD and PCD.
Robustness in realistic UniDA. We compare the performance of our UniOT with DCC, DANCE and UAN under different categories split. In this analysis, we perform A2W of Office with fixed common label set C and surget-private label set C, but the different size of source- private label set C., We set 10 categories for C, 11 categories for C. and vary the number of C., Fig. 3 a) shows that UniOT always outperforms other methods. Besides, UniOT is not sensitive the variation of source-private classes while other methods are not stable enough. Additionally, we also analyze the behavior of UniOT under the different size of target-private label set C, We also perform on A2W of Office with fixed common label set for 10 categories and source-private label set C. for 10 categories. Fig. (b) shows that our UniOT still performs better, while the other methods present sensitivity to the number of target-private classes. Therefore, UniOT is more robust to the variation of target-private classes.
Feature visualization for comparison with existing UniDA methods. We use t-SNE to visualize the learned target features for Rw2Pr of Office-Home. As shown in Fig. the common samples are colored in black and the private samples are colored with other non-black colors by their ground-truth classes. Fig. b)) shows that UAN cannot recognize different categories among target-private samples since they treat all target-private samples as a single class. Especially, our UniOT discovers the global and local structure of target-private samples. Fig. d) validates that UniOT learns better target representation with global discrimination and local consistency of samples compared to DANCE shown in Fig. c)..

---

Varying number of source private categories
Varying number of target private categories
Hiscore
Hiscare
26-
D1C
28-
BCG
10001
10th2
2051:
10011
121211
10021
15025
13835
10307
12/0011
Class split
Class split
(b)
Figure 3: Robustness in realistic UniDA with diverse ratios of common classes. Different class splits C/C,/C, reveal diverse ratios of common classes.
(a) Source-only
(b) UAN
(c) DANCE
(d) UniOT
Figure 4: Feature visualization target domain on Rw2Pr of Office-Home
5
Conclusion
In this paper, we have proposed to use Optimal Transport to handle common class detection and private class discovery for UniDA under unified framework, namely UniOT. More precisely, an OT-based cross- cross-domain partial alignment was designed to detect common class with an adaptive filling strategy handle the diverse UniDA settings. addition, we have proposed an OT-based target representation learning for private class discovery, which encourages both global discrimination and local consistency of samples. Experimental results on four benchmark datasets have validated the superiority of our UniOT over wide range of state-of-the-an methods. In the future, we will modify memory model for more efficient features filling in OT.
Acknowledgement
This work supported by the Shanghai Sailing Program (21YF1429400, 22YF1428800), Shanghai Local College Capacity Building Program (23010503100), and Shanghai Frontiers Science Center of Artificial Intelligence (ShangHAI).
10

---

References
[1] YM Asano, C &uppreccht, and A Vedaldi. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations, 2019.
[2] Silvia Bucci, Mohammad Reza Loghmann, and Tatiana Tommasi. On the effectiveness of image rotation for open set domain adaptation. In Eurepean Conference on Computer Vision, pages 422-438. Springer, 2020.
[3] Pau Panareda Busto, Ahsan Iqbal, and Juergen Gall. Open set domain adaptation for image and action recognition. IEFE transactions on pattern analysis and machine intelligence, 42(2):413-429, 2018.
[4] Zhangjie Cao, Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. In P Proceedings of the European Conference Computer Vision (ECCV), pages 135-150, 2018.
[5] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Mathhis Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pages 132-149, 2018.
[6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual fcatures by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912 1299924, 2020.
[7] Liang Chen, Qianjin Du Yihang Lou, Jianzhong He, Tao Bai, and Minghua Deng. Mutual nearest neighbor contrast and hybrid prototype trainin for universal domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence. pages 6248 6257, 2022.
[8] Liang Chen, Yihang Lou, Jianzhong Hc, Tao Bai, and Minghua Deng, Evidential neighborhood contrastive learning for universal domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6258-6267, 2022.
[9] Lenaic Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xavee Xavier Vialard. Scaling algorithms for unhalanced optimal transport problems. Mathematics of Computation, 87(314):2533-261) 2018.
[10] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292-2300, 2013.
[11] Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty, Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 447-463, 2018.
[12] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
Icce, 2009.
[13] Kilian Fatras, Thibault Séjourné, Rémi Flamary, and Nicolas Courty. Unbalanced minibatch optimal transport; applications to domain adaptation. In International Conference on Machine Learning, pages 3186-3197. PMLR, 2021.
[14] Alessio Figalli. The optimal partial transport problem. Archive for rational mechanics and analysis, 195(2):533-560. 2010.
[15] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud. Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021.
[16] Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya, and Tomaso A Poggio. Learning with wasserstein loss. Advances in neural information processing systems, 28, 2015.
[17] Bo Fu, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Learning to detect open classes for universal domain adaptation. In Fumpean Conference Computer Vision, pages 567-583. Springer, 2020.
[18] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, adverssaril training of neural networks. The journal of
Mario Marchand, and Victor Lempitsky. machine learning research, 17(1):2096-208 2030, 2016.
[19] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9732 2020.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] Jiabo Huang, Shaogang Gong, and Xiatian Zhu. Deep semantic clustering by partition confidence maximisation. In P Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8849 8858, 2020.
11

---

[22] Leonid Kantorovitch. On the translocation of masses. Munagement science, 5(1):1-4, 1958.
[23] Guangrui Li, Guoliang Kang, Yi Zhu, Yunchao Wei, and Yi Yang. Domain consensus clustering for universal domain adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 9757- 2021.
[24] Mengxue Li, Yi-Ming Zhai, You-Wer Luo, Peng-Fei Ge, and Chuan Ren. Enhanced transport distance for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13936-13944, 2020.
[25] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. Advances in neural information processing systems, 29, 2016.
[26] MacQueen. Classification and analysis of multivariate observations. In 5th Berkeley Symp. Math. Statist. Probability, pages 281-297, 1967.
(27] Khai Nguyen, Dang Nguyen, Tung Pham, Nhat Ho, et al. Improving mini-batch optimal transport via partial transportation. In International Conference on Machine Learning, pages (6656-16699 PMLR, 2022.
[28] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering. 22 13100)1335--339 2009.
[29] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision. pages 1406-1415, 2019.
[30] Xingchao Peng. Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda; The visual domain adaptation challenge. arXiv preprint arXiv:1710.0692. 2017.
[31] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models new domains. Earopean conference on computer vision, pages 213-226. Springer, 2010.
[32] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Sacnko. Universal domain adaptation through self supervision. Advances in neural information processing systems, 33:16282-16292, 2020.
[33] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 153-168, 2018.
[34] Richard Sinkhorn. Diagonal equivalence to matrices with prescrihed row and column sums. The American Mathematical Monthly, 74(4):402-405, 1967.
[35] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal of machine learning research, 9(11), 2008
[36] Wouter Van Gansheke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In European Conference on Computer Vision, pages 268-285. Springer. 2020.
[37] Hemanth Venkateswara. Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 0018-507,, 2017.
[38] Renjun Xu, Pelen Liu, Liyan Wang, Chao Chen, and Jindong Wang. Reliable weighted optimal transport for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4394-4403, 2020.
[39] Kaichao You, Mingsheng Long, Zhangjic Cao,, Jianmin Wang, and Michael Jordan, Universal domain adaptation. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 2720-2729, 2019.
[40] Jing Zhang, Zewei Ding, Wanqing Li, and Philip Ogunbona. Importance weighted adversarial nets for partial domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8156 8164, 2018
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See abstract and introduction.
(b) Did you describe the limitations of your work? [Yes] See Conclusion.
(c) Did you discuss any potential negative societal impacts of your work? [No] Since our topic is Universal Domain Adaptation, there is no direct potential negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all thcoretical results? [No] There is no theoretical result in our work.
(b) Did you include complete proofs of all theoretical results? [No] There is no theoretical result our work.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] See Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] See Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [No] The license of assets will be mentioned in the supplemental material.
(c) Did you include any assets either in the supplemental material URL? [No] There is no curating/releasngg new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [No] There no curating/releasing new assets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] The dataset used is about image classi-
fication, which did not con contain personally identifiable information or offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [No] We did not use any crowdsourcing conducted research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [No] We did not use any crowdsourcing or conducted research with human subjects.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [No] We did not use any crowdsourcing or conducted research with human subjects.
13


Reviews:
[{"review_id": "umLbOuXaCID", "reviewer": "Reviewer_4ds8", "summary": "In this work, the authors proposed a unified domain adaptation framework supported by optimal transport in order to transfer among those domains having different categories. Specifically, the authors detect those common classes by predicting the statistics of an unbalanced OT objective. Next, the private class discovery is achieved by doing local alignments in the target domain.  ", "questions": "Please see my comments in the last box. I am willing to improve my score if there are more interpretations and qualitative illustrations. ", "limitations": "No negative societal impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- Overall, this paper is well written. The authors provided detailed information about how they designed the training process step by step.\n- The authors have provided enough quantitative results in order to demonstrate the effectiveness of their method. \n\nCons:\n - The proposed method is relatively ad-hoc. The proposed method serves as discriminative losses for supervised learning problems and improves performance in the target domain. \n - Missing discussion on Batch optimal transport. The authors mainly implemented their framework using batch samples. However, It is known that minibatch optimal transport leads to nonoptimal coupling so there should be some discussions on how batch optimal transport affects this method.\n - Qualitative illustrations will be very helpful in illustrating this method's performance. For example, the authors did a great job in creating figure 2 to give a high-level illustration of the framework. It would be great if the authors could give an example of a batch of data, not only the OT couplings but also the corresponding source and target samples/polytopes. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZRKbY0QN2QE", "reviewer": "Reviewer_V2fC", "summary": "This paper proposes a novel framework, named UniOT, which jointly considers common class detection and private class discovery in Universal Domain Adaptation (UniDA). UniOT leverages Unbalanced Optimal Transport (UOT) to detect common classes in the target domain without any predefined threshold values. In addition, UniOT learns an OT-based representation learning for private class discovery by considering both global discrimination of clusters and local consistency of samples. Furthermore, the authors introduce a new evaluation metric for the UniDA problem. Finally, experiments show that UniOT outperforms several baselines on UniDA.\n", "questions": "I have the following questions:\n\n1) **Related work.** For the application of OT in (P)DA, partial OT instead of UOT can also be used to allow partial displacement [r1].  \n[r1] Khai Nguyen, Dang Nguyen, Tung Pham, and Nhat Ho. Improving mini-batch optimal transport via partial transportation. In Proceedings of the 39th International Conference on Machine Learning, 2022.\n2) **Figure 2.** The authors should add more description in Figure 2 or its description because it is quite difficult to understand the whole framework at the beginning of the paper. Even after reading the Methodology, it is still time-consuming to figure out different components and their functionalities.\n3) **Formulation of OT.** What is the definition of **M** in Equations 2 and 3? Why do the authors convert a minimization problem into a maximization one?\n4) **Common class detection.** The authors claimed that this method does not use some manually specified or hand-tuned threshold values to detect common and target-private samples. Though the threshold in Equation (7) is not completely random, users have to tune $\\epsilon, \\kappa$ for UOT instead. Therefore, it seems like this method only parametrizes the threshold value into other hyper-parameters.\n5) **The number of target prototypes.** How is K chosen? Why is it much larger than the actual number of classes?\n6) **Equation 13.** Should different notations be used for the prediction of $z_i^t$ and $\\tilde{z}_i^t$? Speaking of which, does the prediction of the nearest neighbor of $z_i^t$ lie in the $(B+i)$-th row of $\\tilde{Q}^{tt}$ based on the description in L232-233?\n7) **Target-private samples classification.**  When is the private class in the target domain classified? L259-260 state that they are all classified as unknown. As far as I understood, UniOT still treats all target-private samples as a single class as in prior works. \n8) **Experimental settings.** The authors use a lot of formulas and settings from other papers. It is better to describe them again in the Appendix to improve the readability and make the paper more self-contained. \n9) **Baselines' results.** Why are the performance of DANCE in Table 2 and Table 3 worse than weak baselines such as ResNet or DANN? In addition, the authors should add a symbol to indicate which results are taken directly from the original papers and which are the reproduced results.\n\n**Minor:**\n* There is a typo in L270. It is either A2D or Amazon to Webcam.\n* In column D2A of Table 4, the best H-score is not in the last row so it should not be in bold. In addition, it seems to be unnecessary to additionally report two random tasks for each dataset. It is alright to just report the average performance over all the tasks.\n\nI am happy to increase my score if the above questions are adequately addressed.\n", "limitations": "The authors specified one limitation of their method in Section 5.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In this paper, the authors raise two interesting problems of UniDA that previous works do not consider carefully. Their method to tackle the aforementioned issues is novel to the best of my knowledge. Experimental results back the efficiency of the proposal in UniDA. The authors also conducted adequate ablation studies to verify the importance of different components in their framework. Overall, this paper is well-structured and easy to follow. The main concerns include the clarity of the paper in both methodology and experiment parts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "P-3Yzt8gB8O", "reviewer": "Reviewer_xveW", "summary": "This paper proposes an optimal transport framework to jointly consider common class detection and private class discovery for UniDA. The key point is automatically discovering the intrinsic difference between common and private classes based on the statistical information of the assignment matrix.", "questions": "How do you tune the hyperparameters in the method, such as Equation 15? Only sensitivity to one hyper-parameter is stated in the supplementary material.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\nThis paper addresses Universal Domain Adaptation. The partial alignment for common class detection and representation learning for private class discovery based on UOT and OT models have achieved relatively high classification accuracy. The method of experimental comparison is comprehensive.\n\n[Weaknesses]\nThe proposition of this method, to automatically discover private specific classes, lacks meaning in practice. Since we do not have the labels of these classes, we cannot use these data, but it brings more complex computing resource consumption. The fundamental goal of UniDA is to improve the precision and recall rate of common classes. Identifying private classes is only a process, so it can be regarded as a class.\nThe new metric H^3-score is the same problem.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FpZPojnP6FP", "reviewer": "Reviewer_1NtB", "summary": "This work is progressing towards solving the Universal Domain Adaptation (UniDA) problem, where we don’t assume any prior knowledge about the target classes. The proposed algorithm, named UniOT, utilizes Optimal Transport (OT) to handle two main issues: i) common class detection and ii) private class discovery.\n\nThe common class detection is run by establishing partial alignment between the target features and the source prototypes, while the private class discovery is conducted through aligning the target features with the learnable target prototypes. Both are achieved by solving the OT optimization problem.\n\nFrom the empirical evaluations on the Office, Office-Home, VisDa, and DomainNet, the proposed algorithm UniOT performs significantly better than the existing UniDA methods indicated by the H-scores. Under varying ratios of common classes, UniOT shows a certain level of robustness compared to DANCE and UAN.", "questions": "Still related to the previous point, I'd like to see in more detail about the time complexity of the training and inference.", "limitations": "It is implicitly stated in the conclusion that the current approach requires relatively high memory usage and will find a more memory-efficient way.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The OT optimization approach to addressing the UniDA problem by encouraging common class detection and private class discovery is novel and technically sound.\n- The performance gain over the existing methods is substantial along the line of UniDA.\n- The manuscript is well-written.\n\nWeakness:\n- Would be great if information of both training and inference elapsed time is provided, especially at the inference stage as it requires to run an additional OT optimization.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept