Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

Joint Partial Optimal Transport for Open Set Domain Adaptation
Renjun Xu1∗ , Pelen Liu1 , Yin Zhang1∗ , Fang Cai2 , Jindong Wang3 , Shuoying Liang1 , Heting
Ying4 , Jianwei Yin1
1
Zhejiang University
2
Stanford University
3
Microsoft Research
4
China Jiliang University
Abstract
Domain adaptation (DA) has achieved a resounding success to learn a good classifier by leveraging
labeled data from a source domain to adapt to an
unlabeled target domain. However, in a general setting when the target domain contains classes thatare never observed in the source domain, namely
in Open Set Domain Adaptation (OSDA), existing
DA methods failed to work because of the interference of the extra unknown classes. This is amuch more challenging problem, since it can easily result in negative transfer due to the mismatch
between the unknown and known classes. Existing researches are susceptible to misclassification
when target domain unknown samples in the feature space distributed near the decision boundary
learned from the labeled source domain. To overcome this, we propose Joint Partial Optimal Transport (JPOT), fully utilizing information of not only
the labeled source domain but also the discriminative representation of unknown class in the target
domain. The proposed joint discriminative prototypical compactness loss can not only achieve
intra-class compactness and inter-class separability, but also estimate the mean and variance of
the unknown class through backpropagation, which
remains intractable for previous methods due tothe blindness about the structure of the unknownclasses. To our best knowledge, this is the first optimal transport model for OSDA. Extensive experiments demonstrate that our proposed model cansignificantly boost the performance of open set domain adaptation on standard DA datasets.

1

Introduction

Deep learning recently has improved the progress of diverse
computer vision tasks such as image classification, object
detection and semantic segmentation with the help of largescale labeled datasets which is time and labor consuming tocollect. However, it is often highly difficult to achieve satisfying performance when transferring knowledge from label∗

Corresponding authors: {rux, zhangyin98}@zju.edu.cn

2540

rich domain (i.e., source domain) to label-scarce domain(i.e., target domain), especially when there are unknown samples in the target domain. The characteristics of large scale
unlabeled data in the target domain can be different from thelabeled source domain, resulted in misclassification.
An underlying assumption in domain adaptation is thatsamples in the target domain necessarily belong to the classes
observed in the source domain, from which we train the classifier. Since the target classes are often larger than trainingclasses in real-world settings, considering the open set domain problem could lead to more practical results and applications. Under this regime, the target domain consists ofsubset of classes of the source domain and along with extra
unknown classes. The main task is seeking to classify dataof known classes correctly, and categorize the extra classes as“unknown”. Lack of openness adaptation is a major critiqueagainst OSPB [Saito et al., 2018].
We solve the problems above by a strategy based on the
optimal transport framework. Optimal transport has been
applied to closed set domain adaptation recently [S. and S.,
2016; Luo et al., 2017; Damodaran et al., 2018]. Damodaran
et al. recognized the strength of Convolutional Neural Network (CNN) to obtain a scalable solution by learning jointly
the feature embedding between the two domains and the classifier in a single CNN framework. All previous OT based
methods align the whole source and target domains and perform badly for OSDA since data of unknown classes in thetarget domain can make performance of domain adaptation
model even inferior to a model without adaptation. Such phenomenon is known as negative transfer [Pan and Yang, 2010].
Unknown samples in the target domain need to be excludedin the transfer to avoid negative transfer. To this end, we
propose a more flexible partial optimal transport frameworkthat allows for transportation in only well-matched pairs ofsamples. We formulate the problem as transporting a fraction of the “well-matched” mass of source domain onto thetarget domain minimizing an overall displacement cost. We
eliminate the far-fetched pairings that cause negative transfer in the global optimal coupling to obtain a partial optimal
coupling. Besides the regular displacement loss, the overall
displacement loss includes the loss for displacing to the unknown class applied to the fraction of mass not transferred.
As a bonus, our method naturally adapts to different levels of
openness of the target domain, because fraction size selectedProceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

from global coupling is data-adaptive.
Furthermore, we want to choose the feature representationthat leads to better generalization and robust performance.
For this goal, we include dispersion penalty for each classso that samples are concentrated around the means of theirclasses and therefore lie far from the decision boundary; we
add to the objective function the joint discriminative prototypical compactness loss, which provides a way to estimate
the mean and manipulate (shrink) the covariance of each class(including the unknown class). We firstly estimate the mean
and variance for the unknown class from the residuals of ourpartial optimal transport which was a challenge for previous
researchers as we are blind about the structure of unknownclasses. Experimental results show that JPOT works stably invarious datasets and outperforms existing methods.

2

Related Work

Closed Set Domain Adaptation (CSDA). Currently closed
set domain adaptation methods [Tzeng et al., 2014; Long etal., 2015; Long et al., 2016] minimized the feature distribution discrepancy to alleviate performance degradation in both
source and target domains. Zellinger et al. [Zellinger et al.,2017] achieved domain adaption by Central Moment Discrepancy (CMD). Contrastive Adaptation Network (CAN)[Kang et al., 2019] optimizes a new metric that explicitly
models the intra-class domain discrepancy and the inter-classdomain discrepancy. Transferrable Prototypical Networks(TPN) [Pan et al., 2019] exploits the prototypical distance[Snell et al., 2017] to avoid misclassification.
Open Set Domain Adaptation. OSDA differs from CSDA
by extra unknown classes in the target domain but not in thesource domain. Recently, researchers are concentrating on
open set recognition, intending to recognize “unknown” samples in the target domain during testing [Ganin and Lempitsky, 2015; Saito et al., 2018]. Separate to Adapt (STA) network was proposed by [Liu et al., 2019] to adapt different
openness between source and target classes with coarse-tofine weighting mechanism. Universal Adaptation Network[You et al., 2019] was proposed to quantify sample-leveltransferability and recognize the “unknown” samples. However, all of them failed to solve the problem that target domain
samples were distributed near the decision boundary learned
from the source domain, leading to degraded performance.
Optimal Transport on Domain Adaptation. Optimaltransport has been applied in domain adaptation to align the
representations between the source domain and target domain [Courty et al., 2016; Yan et al., 2018]. As mentioned
above, the coupling γ [Courty et al., 2017] was applied totransport the source samples to the target domain by an estimated mapping. Deep Joint Optimal Transport (DeepJDOT)[Damodaran et al., 2018] applied the coupling matrix γ totransport the source samples to the target domain in discriminative feature space, achieving high accuracy on many transfer tasks. However, the discriminative representation of “unknown” samples in the target domain has not been consideredin previous approaches that causes mismatching.

2541

3

Methodology

Following the settings of OSDA [Liu et al., 2019], we defineswith ns labeled sama source domain Ds = {(xsi , yis )}ni=1tt ntples and a target domain D = {(xi )}i=1 with nt unlabeled
samples. The label space in the source domain is defined
as C s , which is a subspace of target-domain label space C t .
t
The target domain further contains additional classes Cunk,
which are all referred as “unknown” since these classes aretabsent in the source domain, i.e. C t = C s ∪ Cunk
. Data fromsource and target domains can have different probability distributions p and q respectively (p 6= q). In OSDA, we further
observe that p 6= qC s , where qC s represents the probability
distribution of the target data belonging to shared classes C s .
Following [Saito et al., 2018; Chen et al., 2019], we adopt
the two-stream CNN architecture with shared weights. We
propose joint partial optimal transport (JPOT) for open set
domain adaptation, which is an end-to-end method that learnsa feature generator Gf (·) and a classifier Gy (·) to separate
known and unknown classes in the target domain, as illustrated in Figure 2.

3.1

Partial Optimal Transport

Optimal transport for domain adaptation performs the alignment of the sample representations between the source domain and target domain [Courty et al., 2016]. However, existing optimal transport strategies fail to address the problem caused by the pair-wise transport error in pairing unknown target samples to known source samples, resulting in
negative transfer of unknown classes in the target domain.
Despite similarities with the formulation [Damodaran et al.,2018], we propose partial optimal transport strategy exploiting global coupling matrix γ given by joint probability distribution, which aims to identify the most correlated features ofshared classes between the source and target domains. Moreover, we utilize global optimal matrix to capture distinct features of unknown class samples in the target domain and torealize inter-class separability of unknown vs known classes.
The main procedure of JPOT is shown in the Figure 1.
The optimization of partial optimal transport is based onpartial Kantorovich problem [Angenent et al., 2003] seeking
for a general coupling γ ∈ X (Ds , Dt ) between Ds and Dt :Z
γ ∗ = arg min
M(xs , xt ) dγ(xs , xt ),(1)γ∈X (D s ,D t )D s ×D t

where X (Ds , Dt ) denotes the probability distribution between Ds and Dt . The cost function matrix M(xs , xt ) =||Gf (xs ) − Gf (xt )||k denotes the cost to move probability
mass from xs to xt , where k = 2 [Damodaran et al., 2018].
The discrete formulation can be expressed asγ ∗ = arg min hγ, MiF ,γ∈X (D s ,D t )(2)where γ ∗ ∈ RN ×N is the ideal coupling matrix between thesource and target domains, representing as a joint probability measure with source data xs and target data xt . N is thebatchsize. The elements of each row of γ ∗ are all zeros except
one equals 1. h·, ·iF is the Frobenius dot product. To separate data of unknown class from known classes in the target

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

5
7

1

2

3

4

5

6

7

8

1
2

1

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

3

0

0

1

0

0

0

0

0

4

0

0

0

1

0

0

0

0

5

0

0

0

0

1

0

0

0

Target

6

0

0

0

0

0

1

0

0

Unknown
෢∗
𝜸∗
𝜸

7

0

0

0

0

0

0

1

0

8

0

0

0

0

0

0

0

1

1

3
2

6

4

1

8

2
4

7
8

6

5

3

𝜸∗

Source

1

0

𝜌

෢∗
𝜸

ℎ ℳ 𝒙𝒔𝒊 , 𝒙𝒕𝒋 , 𝜌

∗
𝒔 𝒕
∗
𝛾෢
𝑖,𝑗 = 𝛾𝑖,𝑗 × ℎ ℳ 𝒙𝒊 , 𝒙𝒋 , 𝜌

1

2

3

4

5

6

7

8

1
2

1

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

3

0

0

1

0

0

0

0

0

4

0

0

0

1

0

0

0

0

5

0

0

0

0

0

0

0

0

6

0

0

0

0

0

0

0

0

7

0

0

0

0

0

0

0

0

8

0

0

0

0

0

0

0

0

Figure 1: The necessity of the proposed method partial optimal transport for open set domain adaptation. Three colors denote three sets ofdata. Red: Source samples; Deep Blue: Target common samples. Light Blue: Target unknown samples. The γ ∗ denotes the optimal transport
solution on the global perception which indicates that the unknown samples will be negatively transported to the source domain. Adopting
partial optimal transport strategy, exploiting the mean cost ρ of the coupling matching. JDOT chooses the most reliable matching γˆ∗ forpartial domain alignment, eliminating negative transfer caused by unknown class. Best viewed in color.
𝐺𝑦

𝐺𝑓
SourceTarget

ℒcls
Known

Tied

Tied

Tied

Bottleneck
Layer

ℒ𝑝

𝛾∗, 𝛾̂ ∗, ∆𝛾∗

which means that the best matched pairs should be pulledeach other closely. Meanwhile, with samples “labeled as”
unknown class by partial coupling matrix, they should leaveaway from known classes to avoid negative transferℒ𝑠=ℒ𝑐𝑙𝑠+𝛼ℒ𝑝+𝛽ℒ𝑑

ℒ𝑑

Lunk
=
o

Unknown


1X
∗
∆γi,j
log 1 + exp(−η(||Gf (xsi ) − Gf (xtj )||2 ) ,
η i,j
(6)

Figure 2: The proposed model JPOT for open set domain adaptation. The partial optimal transport loss Lp is adopted to exploit
the unknown target samples then avoid negative transfer. The joint
discriminative prototypical compactness loss Ld utilize the statistics information on the target domain to make features more discriminative. Note that the Gf denotes the CNN network and thefully-connected layers to extract features in the bottleneck layer. Gy
denotes the output classification layers. Best viewed in color.

domain, we utilize the mean cost of optimal transport matrix
ρ to measure the similarity between each target sample andeach source sample
ρ=

N N
N
1 X
1 XX ∗
γi,j M(xsi , xtj ) :=ρj ,N j=1 i=1
N j=1γˆ∗ i,j (||Gf (xsi ) − Gf (xtj )||2 ),3.2

Joint Discriminative Prototypical Compactness

Σsz =

(5)

i,j

N
1 X
Gf (xsi )δ(yis = z),|Nz | i=1

NX
1
(Gf (xsi ) − µsz )(Gf (xsi ) − µsz )T δ(yis = z),|Nz − 1| i=1(8)

where z = {1, 2, · · · , k} denotes the corresponding shared
known classes, k denotes the number of classes in Ds , δ(x) =1 when x is True, and δ(x) = 0 when x is False. The variancevanishes when there is only one sample in the batch. Due tothe lack of reliable labels in the target domain, it is difficult
to measure the unknown samples in the target domain distribution Pt . The most difficult task is to find the mean andvariance of unknown samples. According to the partial optimal transport, we provide unknown index Uj to measure theprobability of sample xtj to be in the Unknown class.
Uj =

N
X

∗
∆γi,j
,

(9)

i=1

The number of the unknown class samples can be given as
t
Nunk
=

N
X
j=1

2542

(7)

Inspired by previous work [Herath et al., 2019], exploiting
discriminative statistics information on two domains will enhance the transfer performance. When the feature distribution, Ps of domain Ds , is parameterized with the mean µsz
and the covariance Σsz(3)∗γˆ∗ i,j = γi,j× h(M(xsi , xtj ), ρ),(4)1h(M(xsi , xtj ), ρ) = 1 − (1 + sgn(M(xsi , xtj ) − ρ)),2where γˆ∗ represents partial coupling matrix for known classesof both domains. sgn(x) = 1 when x > 0, and equals−1 otherwise. Thus, we rank the similarity ρj for all target
samples and select samples with highest/lowest similarity toalign/separate. The loss for known classes in the source andtarget domains can be defined as

Lkno
=
o

Lp = Lkno+ Lunk
.
o
o

µsz =

where N denotes the number of source samples which equals
that of target samples. If the coupling pair distance is bigger
than the mean cost ρ, it indicates that this coupling pair is
nontransferable. The partial optimal transport for open set
domain alignment can be defined asX∗∗
where ∆γi,j= γi,j− γˆ∗ i,j denotes the residual set of thepartial optimal transport γˆ∗ i,j . η denotes as a constant andequals to 0.1 in the experiment. Therefore the partial optimaltransport loss Lp is given as

Uj .

(10)

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

The mean µunk and the covariance Σunk can be described asµtunk =

Σtunk =

1

N
X

t
Nunk
j=1

Gf (xtj ) × Uj ,(11)

1
Uj ×(Gf (xtj )−µtunk )(Gf (xtj )−µtunk )T .
t
|Nunk
− 1| j=1(12)

To quantify spatial prototypical information in both domains, we define µA = {µs1 , µs2 , · · · , µsk , µtunk } and ΣA ={Σs1 , Σs2 , · · · , Σsk , Σtunk } for computation. Therefore µA ∈R(k+1)×d and ΣA ∈ R(k+1)×d×d . d is the number of output neurons in the bottleneck layer. The spatial prototypicalmatrix D ∈ R(k+1)×N is defined astA

e−d(Gf (xj ),µz )D(z, j) = Pk+1 −d(G (xt ),µA ) ,
f
m
j
m=1 e

(13)

where d(Gf (xtj ), µAz ) is the distance between the target sample Gf (xtj ) and z-th class center µA
z . Therefore, the Mahalanobis distance formulation of d(Gf (xtj ), µAz ) can be defined asqT
A −1 (G (xt ) − µA ).
(Gf (xtj ) − µAfz ) (Σz )zj(14)

The joint discriminative prototypical compactness loss Ld isdefined as
Ld = Ldc + Ldp ,(15)

The Ldc denotes the centroid clustering loss in the sourceclass domain
Ldc =

N
X

Gf (xsi ) − µsyis

i=1

2

.

(16)

2

The Ldp denotes the prototypical compactness loss for thetarget samples
Ldp =

k+1
N
XX

D(z, j)F (Sof tmax(Gy (Gf (xtj )), z)),(17)

z=1 j=1

where F (·, ·) denotes the cross-entropy loss. Furthermore,
we introduce an additional loss Lcls to minimize the probability of source samples to be misclassified as unknown class.
Lcls =NX

F (Sof tmax(Gy (Gf (xsi ), k + 1)).

(18)

i=1

In conclusion, the total loss of our JPOT model is
L = Lcls + αLp + βLd ,(19)where the α and β denote the trade-off parameters.
To calculate the statistic information, we adopt the incremental learning strategy as follows:A(t−1)A(t)µA(t),
accu. = m × µaccu. + (1 − m) × µ
A(t−1)A(t)ΣA(t),
accu. = m × Σaccu. + (1 − m) × Σ

A(t)

(20)

A(t)

where µaccu. and Σaccu. denote the accumulation results ofthe mean and variance, 0 ≤ m ≤ 1 is the momentum hyperparameter for the accumulation.

2543

Experiments

Now we evaluate our method with state-of-the-art domain
adaptation approaches on several benchmark datasets.

4.1

N
X

d(Gf (xtj ), µA
z ) =

4

Setup

Digits contains three standard digit classification datasets:
MNIST [Lecun et al., 1998], USPS [Hull, 2002], and SVHN
[Netzer et al., 2011]. Each dataset consists of 10 classes ofdigits, ranging from 0 to 9. Following the same evaluation
protocol of [Saito et al., 2018], we construct three open set
domain adaptation tasks and report adaptation results on thetest sets: SVHN→MNIST (one task) and MNIST↔USPS(two tasks).
Office-31 [Saenko et al., 2010] is a standard dataset in computer vision for domain adaptation which contains 4652 images from 31 categories with three domains: Amazon (A), Webcam (W) and DSLR (D). According to previous work [Liuet al., 2019], we adopt the same set of known classes and unknown classes in the target domain and evaluate all methods
on following four challenging settings: A↔W and A↔D.
Office-Home [Venkateswara et al., 2017] is a more challenging domain adaptation dataset which consists of around15500 images from 65 object classes in 4 distinct domains:
Aetistic (Ar), Clipart (Cl), Product (Pr), and Real-Word(Rw). We select (in alphabetic order) the first 25 classes asknown classes shared by the source and target domains [Saito
et al., 2018]. The 26-65 classes are regarded as the unknownclass. We report performances of all the 12 adaptation tasks
to enable thorough evaluations: Ar↔Cl, Ar↔Pr, Ar↔Rw,
Cl↔Pr, Cl↔Rw, and Pr↔Rw.
Compared Approaches. We mainly compare our proposal
with several open set recognition, domain adaptation andopen set domain adaptation methods as previous work [Liu etal., 2019]: Open Set SVM(OSVM) [Jain et al., 2014], DANN
[Ganin and Lempitsky, 2015], RTN [Long et al., 2016],
OpenMAX [Bendale and Boult, 2016], MMD+OSVM,
DANN+OSVM, OSBP [Saito et al., 2018] and STA [Liu etal., 2019]. MMD+OSVM and DANN+OSVM are two variants of OSVM that incorporate Maximum Mean Discrepancy[Tzeng et al., 2014] and domain adversarial network [Ganinand Lempitsky, 2015] in OSVM. In our experiments, we compare the average accuracy of each method on five random experiments.
Evaluation Metrics. Following previous works [Saito etal., 2018], we employ four evaluation metrics for open set
domain adaptation: OS: the accuracy averaged over all theclasses including the unknown class as one class; OS*: theaccuracy averaged only on known classes; ALL: the accuracyof all samples; UNK: the accuracy of unknown instances.

4.2

Implementation Details

For the experiments on Office-31 and Office-Home, our proposal and the compared approaches are both trained by finetuning with ResNet-50 [He et al., 2016] pre-trained on ImageNet. And we adopt LeNet [Lecun et al., 1998] to investigate the efficacy of our framework for the experiments on
Digits datasets. We use all labeled source examples and unlabeled examples for training.

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

A→W

MethodResNet
RTN
DANN
OpenMax
OSBP
STA
DeepJDOT
JPOT

A→D

D→W

W→D

D→A

W→A

Avg

OS

OS*

OS

OS*

OS

OS*

OS

OS*

OS

OS*

OS

OS*

OS

OS*

82.5±1.285.6±1.285.3±0.787.4±0.5
86.5±2.0
89.5±0.686.1±0.592.8±0.682.7±0.988.1±1.087.7±1.187.5±0.387.6±2.1
92.1±0.588.7±0.992.2±0.485.2±0.3
89.5±1.486.5±0.687.1±0.988.6±1.493.7±1.586.9±0.795.2±0.985.5±0.990.1±1.687.7±0.688.4±0.989.2±1.3
96.1±0.4
89.0±0.5
96.0±0.6

94.1±0.3
94.8±0.3
97.5±0.2
96.1±0.4
97.0±1.0
97.5±0.2
96.9±0.2
98.1±0.3

94.3±0.796.2±0.7
98.3±0.596.2±0.3
96.5±0.4
96.5±0.595.6±0.896.2±0.4

96.6±0.2
97.1±0.2
99.5±0.1
98.4±0.3
97.9±0.999.5±0.2
96.1±0.1
99.5±0.1

97.0±0.4
98.7±0.9
100.0±.0
98.5±0.3
98.7±0.6
99.6±0.1
98.1±0.5
98.6±0.271.6±1.0
72.3±0.975.7±1.6
83.4±1.088.9±2.589.1±0.585.6±1.2
93.0±0.771.5±1.172.8±1.576.2±0.982.1±0.6
90.6±2.393.5±0.888.4±1.0
94.1±0.475.5±1.073.5±0.674.9±1.282.8±0.985.8±2.587.9±0.981.5±1.3
88.9±1.075.2±1.673.9±1.475.6±0.882.8±0.684.9±1.387.4±0.6
83.1±0.988.4±0.4

84.2
85.4
86.6
89.0
90.8
92.9
88.9
94.6

84.4
86.8
87.6
89.3
91.3
94.1
90.5
94.3

Table 1: Classification accuracy (%) on Office-31 for open set domain adaptation (ResNet-50).
Method

Ar→Cl

Pr→Cl

Rw→Cl

Ar→Pr

Cl→Pr

Rw→Pr

Cl→Ar

Pr→Ar

Rw→Ar

Ar→Rw

Cl→Rw

Pr→Rw

Avg

ResNetDANN
OpenMax
OSBP
STA
DeepJDOT
JPOT

53.4±0.454.6±0.756.5±0.456.7±1.958.1±0.656.7±0.859.6±0.552.7±0.649.7±1.652.9±0.751.5±2.1
53.1±0.950.4±1.154.2±0.751.9±0.551.9±1.453.7±0.449.2±2.454.4±1.053.7±1.254.6±0.969.3±0.769.5±1.1
69.1±0.367.5±1.571.6±1.2
67.1±1.472.3±1.1

61.8±0.563.5±1.0
64.8±0.465.5±1.569.3±1.0
64.4±0.870.1±0.674.1±0.472.9±0.874.5±0.6
74.0±1.581.9±0.576.2±0.782.1±0.961.4±0.661.9±1.2
64.1±0.962.5±2.0
63.4±0.562.5±0.662.9±0.7

64.0±0.3
63.3±1.0
64.0±0.864.8±1.1
65.2±0.864.9±1.268.3±0.8

70.0±0.371.3±1.071.2±0.869.3±1.174.9±1.0
72.5±1.0
75.1±1.1

78.7±0.680.2±0.880.3±0.880.6±0.9
85.0±0.2
82.1±0.784.8±0.4

71.0±0.671.7±0.4
73.0±0.574.7±2.275.8±0.4
74.0±0.577.4±0.574.9±0.974.2±0.476.9±0.371.5±1.980.8±0.377.1±0.681.2±0.4

65.3
65.4
66.7
65.7
69.5
66.8
70.2

Table 2: Classification accuracy OS (%) on Office-Home for open set domain adaptation (ResNet-50).

OSVM
MMD+OSVM
DANN+OSVM
OSBPSTADeepJDOTJPOTUSPS→MNISTSVHN→MNIST

MethodMNIST→USPS

Avg

OS

OS*

ALL

UNK

OS

OS*

ALL

UNK

OS

OS*

ALL

UNK

OS

OS*

ALL

UNK

54.3
55.9
62.9
63.0
76.9
61.2
79.2

63.1
64.7
75.3
59.1
75.4
71.6
75.3

37.4
39.1
39.2
71.0
80.0
41.3
80.8

10.5
12.2
0.70
82.3
84.4
9.21
86.7

43.1
62.8
84.4
92.3
92.2
85.6
92.4

32.3
58.9
92.4
91.2
91.3
89.4
91.2

63.5
69.5
72.9
94.4
93.9
79.4
94.4

97.5
82.1
0.90
97.6
96.5
66.7
98.4

79.8
80.0
33.8
92.1
93.0
83.9
92.9

77.9
79.8
40.5
94.9
94.9
85.8
92.1

84.2
81.3
21.4
88.1
90.3
87.9
93.9

89.0
81.0
44.3
78.0
83.5
74.4
96.9

59.1
68.0
60.4
82.4
87.3
76.9
88.2

57.7
68.8
69.4
81.7
87.2
82.3
85.4

61.7
66.3
44.5
84.5
88.1
69.5
89.7

65.7
58.4
15.3
85.9
88.1
50.1
94.0

Table 3: Classification accuracy (%) on Digits for open set domain adaptation (LeNet).

All the mentioned deep learning methods are trained with
Adam optimizer. And the model is trained on 256-sizedbatches totally with N = 128 samples from each domain.
We use mini-batch SGD with momentum to 0.9 and the samelearning rate strategy in [Saito et al., 2018]. Following previous work [Herath et al., 2019], m is set as 0.4 in the wholeexperiment. Note that the parameter η is set as 0.1. As for
tradeoff hyper-parameters α and β, we select α = 0.02 andβ = 0.05 for all transfer tasks.

4.3

Result and Discussion

The classification accuracy results on the Office-31 dataset for
open set domain adaptation based on ResNet-50 are shownin Table 1. For fair comparison, all results of comparedapproaches are directly reported from their original papers.
The JPOT exceeds the performance of all previous methods
on most transfer tasks. It is worth noting that our proposal
improves the classification accuracies substantially on hardtransfer tasks, e.g. A→W and W→A, and achieves comparable classification performance on easy transfer tasks, e.g.
D→W and W→D. In particular, JPOT produces higher OS
than OS*, verifying that JPOT is robust to open set domain
adaptation scenarios. Besides, some close set domain adaptation methods perform worse than ResNet on a few tasks
since these methods suffer from negative transfer caused bymismatching unknown classes in the target domain to known

2544

classes in the source domain. JPOT aligns the discriminative
representations of shared class centers in both source and target domains by center-based optimal transport strategy, andachieves intra-class compactness and inter-class separability.
Results on the twelve tasks of Office-Home dataset are
shown in Table 2. Due to the large domain gap between thesource and target domains, we observe that previous open set
domain adaptation approaches obtain poor performance onsome tasks. The JPOT approach outperforms the comparison methods on most transfer tasks. The encouraging results
suggest that our proposal alleviates the negative transfer issue brought by the unknown class in the target domain, andalso highlight the importance of center-based optimal transport strategy in open set domain adaptation.
We further compare JPOT with previous approaches on the
Digits dataset, as reported in Table 3. In contrast to Office-31
dataset, Digits dataset has a much larger domain size. JPOT
overpasses all comparison methods with different metrics onmost transfer tasks. Note that, SVHN dataset contains significant variations (in scale, slanting, blurring and rotation) andconfounding information from background that makes large
domain gap on task SVHN→MNIST. JPOT achieves better
performance than STA on task SVHN→MNIST. In particular, JPOT almost achieves the state-of-the-art performancewith the UNK accuracy in all transfer tasks, which suggests

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)(a) OSBP(b) DeepJDOT(c) JPOT-P(d) JPOT(e) OSBP(f) DeepJDOT(g) JPOT-P(h) JPOT

Figure 3: Visualization of features on MNIST→USPS task. Figure (a-d) represents category alignment (Each color denotes a class. The Red
denotes the Unknown class). Figure (e-h) represents domain alignment (Blue: Source domain; Red: Target domain). Best viewed in color.

MNIST→USPS

MethodJPOT-P
JPOT-T
JPOT-S
JPOT

OS

OS*

ALL

UNK

92.0
91.4
92.4
92.9

91.1
91.1
91.7
92.1

89.2
87.9
92.5
93.9

90.4
87.4
93.1
96.9

Table 4: Ablation study: classification accuracy (%) on Digits.

that JPOT is robust to large domain gap and able to learn moretransferable representations for open set domain adaptation.

4.4

Ablation Study

To tooth apart the separate contributions of center-based optimal transport strategy and discriminative domain alignment,
we compare JPOT with OSBP [Saito et al., 2018], DeepJDOT [Damodaran et al., 2018] and a variant of JPOT using
the t-SNE embeddings [Donahue et al., 2014] of the lastlayer features on transfer task MNIST→USPS in Figures3(a)-3(h). (1) In Figures 3(a) and 3(e), we observe that in
the OSBP model, scattered samples of unknown and knownclasses in the target domain are close or even mixed together
and the domains are not well aligned, which may cause negative transfer. Therefore the OSBP output classification isof low confidence. (2) In Figures 3(b) and 3(f), although theoriginal DeepJDOT computes the global optimal transport solution, the target unknown samples still be mistaken transported to the corresponding source domain. Apparently the
unknown samples mixes up with the known samples whichleads to terrible prediction results. (3) As shown in Figures3(c) and 3(g), compared with OSBP and DeepJDOT, JPOTP (Lcls + αLp ) with partial optimal transport strategy andwithout joint discriminative prototypical compactness loss,
achieves better domain alignment with high classification performance meanwhile avoid negative transfer. Due to the factthat partial optimal transport exploits the most likely to be theunknown samples, the unknown target samples will not fully
mix up with the known samples. (4) As shown in Figures 3(d)and 3(h), JPOT achieves intra-class compactness and interclass separability, and also improves with a large room over
JPOT-P, validating the complement of discriminative feature

2545

learning with the corporation of partial optimal transport.
In Table 4, further ablation tests have been carried out
on variations of JPOTs including JPOT-T (Lcls + αLknoo ),JPOT-S (Lcls +αLp +βLdc ), and JPOT-P. (1) While JPOTT still works well on detecting unknown instances (87.4%
of accuracy), the performance falls behind other JPOT variations. Without the additional loss Lunk
, JPOT-T lacks drivo
ing force to push unknown samples away from the known
samples in the feature space, resulted in inferior performance.
(2) JPOT-S outperforms JPOT-P on all metrics: this indicates that joint discriminative learning especially the compactifying effect due to Ldc to make source samples moreclose to their class means could help to reduce the misclassification error caused by those source samples scattered near thethe decision boundary. (3) JPOT outperforms all variations.
This demonstrates that the prototypical compactness loss Ldp
to compactify target samples and to be more discriminative isa good strategy to boost the efficacy significantly.

5

Conclusion

This paper presents Joint Partial Optimal Transport (JPOT)
for open set domain adaptation, addressing the critical challenge of negative transfer brought by unknown class samples
in the target domain. We propose partial optimal transport
based on global optimal transport with coupling matching
matrix to exploit the most likely unknown target samples.
Meanwhile, the joint discriminative feature learning helps to
not only match shared class samples in both domains, but also
separate samples of unknown and known classes in the target
domain. This well improves the domain adaptation performance. Comprehensive experiments show that our method
outperforms state-of-the-art results on various domain adaptation datasets.

Acknowledgements
This research was supported by National Key Research andDevelopment Program of China (No. 2017YFB1400603),
National Natural Science Foundation of China under grants(No. 61825205, No. 61772459, No. 61402403), and MoE
Engineering Research Center of Digital Library.

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20)

References[Angenent et al., 2003] S. Angenent, S. Haker, and A. Tannenbaum. Minimizing flows for the monge–kantorovich
problem.
SIAM journal on mathematical analysis,
35(1):61–97, 2003.
[Bendale and Boult, 2016] A. Bendale and T.E. Boult. Towards open set deep networks. In Proceedings CVPR,
pages 1563–1572, 2016.
[Chen et al., 2019] C. Chen, Z. Chen, B. Jiang, and X. Jin.
Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation. In Proceedings AAAI, volume 33, pages 3296–3303, 2019.
[Courty et al., 2016] N. Courty, R. Flamary, D. Tuia, and
A. Rakotomamonjy. Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine
intelligence, 39(9):1853–1865, 2016.
[Courty et al., 2017] N. Courty, R. Flamary, A. Habrard, and
A. Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In Advances in Neural Information Processing Systems, pages 3730–3739, 2017.
[Damodaran et al., 2018] B.B. Damodaran, B. Kellenberger,
R. Flamary, D. Tuia, and N. Courty. Deepjdot: Deepjoint distribution optimal transport for unsupervised domain adaptation. In ECCV, pages 467–483, 2018.
[Donahue et al., 2014] J.
Donahue,
Yangqing
Jia,
O. Vinyals, J. Hoffman, Ning Zhang, E. Tzeng, and
T. Darrell. Decaf: A deep convolutional activation featurefor generic visual recognition. In International conferenceon machine learning, pages 647–655, 2014.
[Ganin and Lempitsky, 2015] Y. Ganin and V. Lempitsky.
Unsupervised domain adaptation by backpropagation. In
ICML, pages 1180–1189, 2015.
[He et al., 2016] K. He, X. Zhang, S. Ren, and J. Sun. Deep
residual learning for image recognition. In CVPR, 2016.
[Herath et al., 2019] S. Herath, M. Harandi, B. Fernando,
and R. Nock. Min-max statistical alignment for transfer
learning. In Proceedings CVPR, pages 9288–9297, 2019.
[Hull, 2002] J. J. Hull. A database for handwritten text
recognition research. IEEE Transactions on Pattern Analysis & Machine Intelligence, 16(5):550–554, 2002.
[Jain et al., 2014] L.P. Jain, W.J. Scheirer, and T.E. Boult.
Multi-class open set recognition using probability of inclusion. In ECCV, pages 393–409. Springer, 2014.
[Kang et al., 2019] Guoliang Kang, Lu Jiang, Yi Yang, andAlexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings
CVPR, pages 4893–4902, 2019.
[Lecun et al., 1998] Y. L. Lecun, L. Bottou, Y. Bengio, and
P. Haffner. Gradient-based learning applied to document
recognition. Proceedings IEEE, 86(11):2278–2324, 1998.
[Liu et al., 2019] H. Liu, Z. Cao, M. Long, J. Wang, andQ. Yang. Separate to adapt: Open set domain adaptation
via progressive separation. In Proceedings CVPR, pages
2927–2936, 2019.

2546[Long et al., 2015] Mingsheng Long, Yue Cao, Jianmin
Wang, and Michael Jordan. Learning transferable featureswith deep adaptation networks. In International Conference on Machine Learning, pages 97–105, 2015.
[Long et al., 2016] Mingsheng Long, Han Zhu, Jianmin
Wang, and Michael I Jordan. Unsupervised domain adaptation with residual transfer networks. In NeurIPS, pages
136–144, 2016.
[Luo et al., 2017] Z. Luo, Y. Zou, J. Hoffman, and F. Li. Label efficient learning of transferable representations acrossdomains and tasks. In Advances in Neural Information
Processing Systems, pages 165–177, 2017.
[Netzer et al., 2011] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural
images with unsupervised feature learning. Nips Workshop on Deep Learning & Unsupervised Feature Learning, 2011.
[Pan and Yang, 2010] Sinno Jialin Pan and Qiang Yang. A
survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2010.
[Pan et al., 2019] Yingwei Pan, Ting Yao, Yehao Li,
Yu Wang, Chong-Wah Ngo, and Tao Mei. Transferrable
prototypical networks for unsupervised domain adaptation. In Proceedings CVPR, pages 2239–2247, 2019.
[S. and S., 2016] Baochen S. and Kate S. Deep coral: Correlation alignment for deep domain adaptation. In ECCV
Workshops, 2016.
[Saenko et al., 2010] K. Saenko, B. Kulis, M. Fritz, and
T. Darrell. Adapting visual category models to new domains. In ECCV, 2010.
[Saito et al., 2018] K. Saito, S. Yamamoto, Y. Ushiku, andT. Harada. Open set domain adaptation by backpropagation. In Proceedings ECCV, pages 153–168, 2018.
[Snell et al., 2017] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In NeurIPS,
pages 4077–4087, 2017.
[Tzeng et al., 2014] E. Tzeng, J. Hoffman, N. Zhang,
K. Saenko, and T. Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv:1412.3474, 2014.
[Venkateswara et al., 2017] H. Venkateswara, J. Eusebio,
S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings
CVPR, pages 5018–5027, 2017.
[Yan et al., 2018] Y. Yan, W. Li, H. Wu, H. Min, M. Tan,and Q. Wu. Semi-supervised optimal transport for heterogeneous domain adaptation. In IJCAI, pages 2969–2975,
2018.
[You et al., 2019] K. You, M. Long, Z. Cao, J. Wang, and
M. I Jordan. Universal domain adaptation. In Proceedings
CVPR, pages 2720–2729, 2019.
[Zellinger et al., 2017] W. Zellinger,
T. Grubinger,
E. Lughofer, T. Natschläger, and S. Saminger-Platz.
Central moment discrepancy (cmd) for domain-invariant
representation learning. arXiv:1702.08811, 2017.