Keypoint-Guided Optimal Transport with Applications in Heterogeneous Domain Adaptation
Authors: Xiang Gu
Yucheng Yang
Wei Zeng
Jian Sun
Zongben Xu
Venue: NeurIPS 2022 Accept (2022)
Keywords: optimal transport
keypoint-guided model
relation preservation

Abstract: Existing Optimal Transport (OT) methods mainly derive the optimal transport plan/matching under the criterion of transport cost/distance minimization, which may cause incorrect matching in some cases. In many applications, annotating a few matched keypoints across domains is reasonable or even effortless in annotation burden. It is valuable to investigate how to leverage the annotated keypoints to guide the correct matching in OT. In this paper, we propose a novel KeyPoint-Guided model by ReLation preservation (KPG-RL) that searches for the matching guided by the keypoints in OT. To impose the keypoints in OT, first, we propose a mask-based constraint of the transport plan that preserves the matching of keypoint pairs. Second, we propose to preserve the relation of each data point to the keypoints to guide the matching. The proposed KPG-RL model can be solved by the Sinkhorn's algorithm and is applicable even when distributions are supported in different spaces. We further utilize the relation preservation constraint in the Kantorovich Problem and Gromov-Wasserstein model to impose the guidance of keypoints in them. Meanwhile, the proposed KPG-RL model is extended to partial OT setting. As an application, we apply the proposed KPG-RL model to the heterogeneous domain adaptation. Experiments verified the effectiveness of the KPG-RL model.

Keypoint-Guided Optimal Transport with Applications in Heterogeneous Domain Adaptation
Xiang Gu', Yucheng Yang', Wei Zeng', .Jin Sun (Xx)
Zongben Xu
School of Mathematics and Statistics, Xi' an Jiaotong University, Xi Xi'an, China
Pazhou Laboratory (Huangpu), Guangzhou, China
3 Peng Cheng Laboratory, Shenzhen, China
xjju. aettuued... edu. jizzajimmun,,zae waaaaannnnxuu]aeaa Exjjt edu.
Abstract
Existing Optimal Transport (OT) methods mainly derive the optimal transport plan/matching under the criterion of transport cost/distance minimization, which may cause incorrect matching in some cases. In many applications, annotating few matched keypoints across domains is reasonable or even effortless annotation burden. is valuable investigate how leverage the annotated keypoints to guide the correct matching in OT. In this paper, we propose a novel KeyPoint-Guider model by ReLation preservation (KPG-RL) that searches for the matching guided by the keypoints in OT. To impose the keypoints in OT, first, we propose a mask- based constraint of the transport plan that preserves the matching of keypoint pairs. Second, we propose to preserve the relation of each data point to the keypoints to guide the matching. The proposed KPG-RL. model can solved by the Sinkhorn's algorithm and is applicable even when distributions are supported in different spaces. We further utilize the relation preservation constraint the Kantorovich Problem and Gromov- them. Meanwhile, the proposed KPG-RL model is extended to partial OT setting. As an application, we apply the proposed KPG-RL model to the heterogeneous domain adaptation. Experiments verified the effectiveness of the KPG-RL model. Code is available at https: attps://githut /gtghtt........///g//g -UUTUUU-UUKGG--RRRR
model to impose the guidance of keypoints in
1 Introduction
As mathematical tool for distribution aligmment, mass transport, etc., Optimal Transport (OT) [1] has gained increasing attention in the machine learning community. OT aims derive transport map or plan between a source and a target distribution, such that the transport cost is minimized. Due to its capacity exploit the geometric property of data, OT has been employed in many applications, e.g., computer vision [2, 3], natural language processing [4, 5, 6, 7], generative adversarial network [8], domain adaptation [9]. clustering [10], anomaly detection [11], etc. OT has typical formulations, i.e., the Monge's formulation [12] and the Kantorovich's formulation [13]. The Kantorovich's 's formulation [13] relaxes the Monge's 's formulation and attracts broader studies in applications. Unless otherwise stated, by OT, we refer to the Kantorovich 's formulation in this paper. The original OT model, i.e.. the Kantorovich Problem [13] (KP), linear program that computationally expensive. The entropy-regun regulay-regllriized OT [14] introduces the entropy of the transport plan as regularization to the OT model which is solved by the computationally cheaper Sinkhorn -noop algorithm [15] allowing to use automatic differentiation [16].
KP needs to transport all the mass of source distribution exactly match the mass of target distribution. But in some cases, only partial mass of source and target distributions should be matched, e.g., the source or target samples contain outliers. To overcome this limitation, the partial OT [17, 18, 19, 20],
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Samples
(b) KP
(c) GW
(d) XPG-RL (ours)
Figure 1: (a) Positive (cross) and negative (circle) samples of source (in blue) and target (in green) distributions. (b) KP distorts the data structure, leading to mismatch of some positive and negative samples. (c) GW model better preserves the data structure, but completely mismatches the samples. (d) Our proposed KPG-RL model utilizes some keypoints (red pairs) to guide the transport and produce correct matching.
unbalanced OT [21, 22], and robust OT [23, 24, 25] models are presented, that allow to transport only partial mass of distribution. Another extension of KP is the Gromov- Wasserstein (GW) model [26, 27] that computes the distance between metries defined within each domain rather than between samples across domains as in KP.
In most of the above models, the main criterion for the optimal transport plan/matching is by the minimization of total transport distance or distortion over all samples. Though having achieved promising results in many applications, without any additional guidance, the OT models may lead to incorrect matching of samples. Figures (bb) and I(c) illustrate examples of incorrect matching produced by KP and GW models. In many applications, it is reasonable to annotate some paired keypoints across domains for guiding the matching in OT. For instance, in non-rigid point set or image registration [28, 29], few keypoint pairs that should be matched
annotated in two point sets/images. In semi-supervised domain adaptation [30] and heterogeneous domain adaptation [31], there are a few labeled target domain data and large amount of labeled source domain data available, that could be directly taken as keypoints. Therefore, it is valuable and important to investigate how to take advantage of those keypoints to guide the correct matching in OT. Figure ((d) shows an example that with the guidance of a few keypoints, the correctness of matching can be improved.
In this paper, we propose novel KeyPoint-Guice Guided model by ReLation preservation (KPG-RL) for leveraging the annotated keypoints to guide the matching in OT. In KPG-RL, we first preserve the matching of keypoint pairs in OT using mask-based constraint of the transport plan. We then propose to preserve the relation of each data point to the keypoints in transport, enforcing the matching the data points near paired keypoints across domains. The proposed KPG-RL model is applicable even when distributions lie in different spaces, and can be solved by Sinkhorn's 's algorithm. We further enforce the relation preservation constraint in KP and GW to impose the guidance of keypoints in them. To tackle the problems that only partial mass should be transported, we extend the KPG-RL model partial OT setting, forming partial- partial-KPG-RL KPG-RL model.
As an application, we apply the KPG-RL model to the heterogeneous domain adaptation (HDA) [31]. HDA is transfer learning task that aims to transfer the knowledge of large-scale labeled source domain data to the target domain where a few labeled and larger amounts of unlabeled data are available for training. The "heterogeneity" implies that the source and target domain data are in heterogeneous feature spaces, e-g., generated by different deep networks. This heterogeneity poses a major obstacle in adapting the source trained model to the target domain. We take the labeled target domain data and source class centers as keypoints and transport source domain data to target domain by our KPG-RL model. Upon the transported source domain data and the labeled target domain data, a classification model is trained that is transferable to the target domain. Experiments show that the KPG-RL model is effective for HDA.
In the following sections, we discuss the related works Sect. 2, and introduce the background of OT in Sect. 3. In Sect. 4, discuss the details of the proposed KPG-RL model. In Sect. 5, we apply the KPG-RL model to HDA. Section concludes this paper.
Notations. Σn (p E IT E, P; 1} is the probability simplex. (+))) is the Frobenius dot product of two matrices. @ stands for the Hadamard product. 1 1m denotes m-diimensional all-one vector. Ta. and Tij are respectively the i-th row and j-th column of matrix π.

---

2 Related Works
We review below the most related OT models and HDA methods to our work.
transport plan such that the distance between transported points in target domain is the same as the distance between the original points in source domain. Our KPG-RL model aims to use keypoint pairs to guide the matching in OT by preserving the relation of each point to the keypoints. Our "relation-preserving" scheme preserves the relation of data w.r.t. the given keypoints, different from the pairwise distance-preserving constraint in GW. We experimentally verified the effectiveness of relation-preservinn guidance of keypoints in OT. From the computational point of view. GW is a non-convex quadratic program, while KPG-RL is a linear program. Lin et al. [32] use the anchors to encourage clustering of data and to impose rank constraints on the transport plan to improve its robustness to outliers. The "anchors" in [32] are intermediate points in computation for improving robustness, different from the "keypoints" in this paper, which are the annotated paired data for guiding the matching in OT. Hierarchical OT [7, 33, 34] transports points by dividing them into some subgroups and then derives the transportation of these subgroups using OT. Different in goal and methodology from Hierarchical OT, we impose the guidance of keypoints for pursuing correct matching in OT by preserving the relation to the keypoints. We do not explicitly divide the points into subgroups, and there is no hierarchy in our method. TLBB [27, 35] is a lower bound of GW that can be computed faster. TLB takes the ordered distance of each point to all the points in the same domain as features, and then performs the Kantorovich formulation of OT using such features. Differently, our method uses a carefully designed relation of each point to the keypoints to impose the guidance of keypoints to the other points. Courty et al. [9] constrain the cost function to encourage the matching of labeled data across source and target domains that share the same class labels for domain adaptation. They use the Laplacian regularization to preserve the data structure. Differently, we explicitly model the guidance of keypoints matching to the other data points in our OT formulation. The matching of paired keypoints is enforced by our mask-based constraint on the transport plan. Zhang et al. [36] propose Masked OT model regularization term to preserve the local feature invariances between fine- tuned and pretrained graph neural network (GNN) for the fine-tuning of GNNs. Though the Masked OT [36] shares similar spirits to the mask-based modeling in our method, our main contribution is the relation preservation for imposing the guidance of keypoints, different from [36] in methodology. For our mask-based modeling,
OT models. The GW model [27] seeks a ttistance-preserviii
scheme for introducing the
utilized to impose the matching of keypoints which is theoretically guaranteed. While the mask in [36] aims to preserve the local information of finetuned network from pretrained models. The motivation and design of our mask are different from those in [36].
HDA methods. HDA methods could be roughly categorized into cross- doos-n mapping and common subspace learning methods. The cross- dooss-domain mapping approaches [31, 37, 38, 39, 40] learn transform to map the source features or model parameters to target domain to achieve adaptation. The common subspace learning approaches [41, 42, 43, 44, 45, 46] learn domain-specific projections to map source and target domain data into common subspace such that their distributions are aligned. Our method for HDA belongs to the first category, and could be mostly related to [37]. The method
in [37] transports source samples to target domain using GW model regularized by the distance between the center of transported source samples and the center of labeled target samples having the
same class labels. Different from [37], we take each labeled target domain data and its corresponding source domain class center as a paired keypoint and preserve the relation of each data to the set of keypoints in each domain when conducting OT.
3 Background on Optimal Transport
Kantorovich Problem (KP). We consider two sets of data points, i.e., the source data X = (2;) Im and the target data Y = { of which the empirical distributions are p = Σm Pr and q = Σブ= & With a slight abuse of notations, we may also denote p = (P1, PP..PP..... .Pm) E E,m and q = (q1, 42. ... TT,)T E Dn as the mass supported on X and Y respectively. We define the cost matrix between X and Y as C' = (Ca,s) E IR m with Cr.j = c( vr). where is cost function, which is set to the squared Lz-distance of x; and yj in our experiments. OT aims to optimally transport p towards q at the smallest cost, formulated as the following Kantorovich Problem (KP):
min (x) 2 (I,, C)F, s.t. 11(p q) = { E m T In = x 1m = g).
(1)
xEll(p.g)
3

---

Ps
3
p
11
0
P
π
M
π
Figure 2: Example of modeling the matching of keypoints (red) using mask. The left part illustrates the matching of data points with keypoint pairs K {(3, 2), (6, 5) To preserve the matching of keypoint index pair (i. e.g., (3, 2). the i-th row and j-th column of transport plan тे must be zeros except T.... Therefore, keypoints, and I is to be optimized.
model by M n, where M is the mask determined by the
When taken as distance metric (aka. ground metric), the minimum value of objective function in Eq. (1) is a distance between and q, named Wasserstein distance.
Partial OT model. The KP in Eq. (1) takes the mass preserving assumption that all the mass of is transported to exactly match the mass of In many applications, only partial mass should be transported. The partial OT model [18, 19] seeks the minimal cost of transporting only unit mass from to q, where & & min
| [1), formulated as
min (T), s.t. II (p, q) { T TEllp(p.g)
|r,, & P.
1m
9,
1 x1, 11, s}.
Note that in partial OT, the total mass ||p| and ||q]] are not necessarily equal. In the definition of 11(p, q) and [p, q), m and n are respectively the length of p and q.
Gromov-Wasse Wasserstein (GW) model. If the data points and My lie in different spaces, the distance between 24 and yy may not be computed. The GW [27] model then minimizes the distortion when transporting the whole set of points from one space to another. The GW model relies on the intra-
domain distance of source domain as C (C E mR and target domain as C (C),)
Rax* where C is the distance between source domain data F; and Tk. and C₁2, is the distance between target domain data yj and yı- The GW model is given by
min xEI((p.g)
Lgn (TT A
Σ
Tis C/.l..
1,=1,1111
4 Keypoint-Guided Optimal Transport
This section details our proposed keypoint- guiidd model that leverages the keypoints to guide the matching in OT. The guidance is imposed by preserving the matching of keypoint pairs and the relation of each data point to the keypoints. We first discuss the preservation of matching of keypoints, then introduce the modeling of relation, and finally present the keypoint-guided guided OT models.
Preservation of matching of keypoints in transport. We denote the set of keypoint index pairs
K. (( fu) b4 with U denoting the number of paired keypoints. We respectively denote } u=1 and J [iu u=1 as the sets of source and target keypoint indexes. For the example (3, 2), (6, 5)),
I= I illustrated Fig. 2.
(3, 6). and J 52. To impose the guidance of these keypoints with indexes in I. in deriving the matching in OT, we first guarantee the matching of the keypoint pairs. As illustrated in Fig. we preserve the matching of keypoints in transport using mask-based constraint of the transport plan, which is motivated by the following observation. If the matched, the optimal transport plan क satisfies that the i-th row and j-th column of स must be zeros except fi.j. which means that the all mass of source keypoint 2; must be transported to target keypoint Yj and yj can only receive the mass from Z;. For the example in Fig. 2, the 3-th row and 2-th column are zeros except that π..2 > O. This sparsity of â motivates us to model it as the Hadamard product of mask matrix M = (Mi,) ) E mR and matrix E IR with positive entries, i.e., = M 3 T. with म.. Mi.
paired keypoints (i, E

---

With Eq. (4), we define the admissible solution set for our soyyponnt-guided OT model as
Π(p, q: M)) = { E R (M @ π 1 = (M 3 π) 1 q)-
(5)
Note that the π in Eq. (5) is not necessarily coupling. The entry Ma of the mask matrix M is set to O if #z.j needs to be O, otherwise Mr, is set to 1. Figure 2 illustrates the mask- modeling of Eq. (4). M is constructed in Proposition 1.
Proposition 1. Suppose that the mask matrix M satisfies that
1,
(i, E.
0,
if I and (i,j) $ K, if J and (1 $ K.
Mas
(6)
0,
1.
otherwise (i.e., i $ I and $ J).
and p: = qz. for (i.j) E K. Then, the transport plan M @ with æ E 11(p, q: M) preserves the matching of paired keypoints with index pairs in K.
More explanations of the construction of M and the proof of Proposition are provided in Ap- pendix A.I. Proposition indicates that if p. qj. for (i, E K. the matching of keypoint pairs is preserved by the mask-based constraint. This mask-based modeling in Eq. (4) is also applicable even for the case that there exist some (i,j) E KC such that p; * qj. For this case, we shall use different mask matrices. Please refer to Appendix A. for the details.
Modeling the relation to keypoints. To use the keypoints to guide the matching in transport, we propose to preserve the relation of each point to the of keypoints in transport. Figure illustrates the relation within points of each distribution. For the data point ry. E X, its relation score the keypoint zi, for in E I (illustrated by red circle in the left of Fig. 3) is defined as
E- -C.i/a/
R Eu= -h. Via EI, (7)
Ri
Ri
k
Rf.
where CR.s is the l2-distance between Fg. and Tia, and is temperature set max,, (C,, J in experiments. Similarly, for y E Y. its relation score to the keypoint yj. for Ju I is defined as
R
Ri
Source
Target
e Gisut Ri T
Vju J,
(8)
Illustration of relation of each point to the keypoints (red). In the keypoint- syypoint-guided OT model. the relation of each point keypoints should be preserved after transport.
Figure 3:
where Cf is the distance between y and y
r' is temperature set p * maxy, The setting of 7 and r' implies that the distances are divided by their maximum value, and normalized to [0, 1), which increases the robustness of the relation score the scale of the distances. P is tunable parameter and set 0.1 in our experiments, since 0.1 is commonly used temperature in the softmax function [47, 48]. We denote RE = -Ri Ri .... Ri and Ri (Ri R2 Ri data points F4 and y the keypoints in source and target domains respectively. From the definition of the relation, the cross domain points near to a paired keypoints share similar relation to the keypoints in corresponding domain. For instance, in Fig. 3, 25 and y near to the paired keypoints with indexes (i2..2) have similar relation Rg and Rt. Meanwhile, if 2g is distant from all the keypoints, R2 is close to a uniform probability vector.
Ri,, that represent the relation of
Keypoint-guided model. Based on the relation given above, we define the guiding matrix = (Gx,) E _m* by Gt d(Ri, R2)) where d measures the dissimilarity of R2 and Rt. Then, Gp. is with small and even near to 0 if Rt and R² are similar. d is taken as the Gensen-Shannon divergence in this paper. We will study the effect of d in Appendix B.5. By using the mask-based constraint of transport plan, the KeyPoint- Poidd--GGuuded model by ReLation preservation (KPG-RL) is defined as
min (.Il(p...M)
Lipg 2 (M (MO.G))
(9)
5

---

By the KPG-RL model in Eq. (9), first, the matching of keypoint pairs is enforced by the mask-based constraint of the transport plan. Second, the minimization of the objective function enforces that the optimal transport plan has larger entries in the locations where the entries of G are smaller. Hence the cross-domain points corresponding to these locations (e.g., A and shown in Fig. 3) that are near to a paired keypoints tend to be matched. Based on the softmax-based formulations in Eqs. (7) and (8), R2)) d(Rt
is mainly determined by the relation score to the closest keypoint(s), since relation scores to the distant keypoints are small close to O. This implies that the points are mainly guided by the closest keypoints in our KPG-RL model in Eq. (9). For the points distant from all the keypoints, their corresponding entries of G are close to zeros according to the definition of G. Therefore, the guidance of keypoints to these points is limited. To achieve correct matching of these points, additional information, e-g., point-wise cost or more keypoints, is needed.
Equation (9) is a linear program and can be solved by linear programming algorithms, e-g., the Simplex algorithm. We give the details for reformulating Eq. (9) as the standard form of linear programming in Appendix A.2. Since Sinkhorn's 's algorithm offers a lightspeed computation of the entropy-regularizzed OT [14], a natural question is that, can Sinkhorn's 's algorithm be applied to the KPG-RL model with entropy regularization? We give the positive answer, and the details of the deduction are given in Appendix A.3. The iterative formulas are
u (1+1
Kv v (+1)
(10)
KJ (((11))
where K = -M @ e e-G/e is the coefficient of entropy regularization. The division operator used above is entry-wise. After iteration, the optimal transport plan is diag (u) Kdiag(o).
Note that in this KPG-RL model in Eq. (9), the points in X and Y do not necessarily lie in the same space because we only need to compute the distance within each one of X and Y. Therefore the proposed KPG-RL model in Eq. (9) is applicable even when and q are supported in different spaces. As mentioned in Sect. I, the GW model is applicable for transport across different spaces. However, GW is non-convex quadratic program and is often solved by the Frank- Walfe algorithm in which KP-like problem needs to be solved at each iteration by Sinkhorn's 's algorithm or linear programming. Surprisingly, our KPG-RL model can be directly solved using Sinkhorn's 's algorithm or linear programming without additional iteration, as discussed above.
Imposing keypoint guidance in KP/GW. We can impose the keypoint guidance in KP by adding Lkpg (T) as regularization term KP, obtaining the following KPG-RL-KP model:
{aL (M @ xx) + (1 a) T (T) (M @ n, aC + (1 a))) (0, (11) =)
min
The KPG-RL-KP can be solved by Sinkhorn's algorithm or linear programming, the same as the solution of KPG-RL model. Similarly, we define the KPG-RL-GW model in GW by
min algg (M 2 + (1- =EIlip.g:M)
E 10.
(12)
The KPG-RL-GW model is solved using the Frank-Walt Walfe algorithm of which the details are given in Appendix A.4. The KPG-RL-KPIKPG-RL-CN models take both advantages of KP/GW and KPG-RL models, and could be helpful especially when the number of paired keypoints is small. We show in Appendix A.5 that the KPG-RL-KP and KPG-RL-GW provde a proper metric and divergence under mild conditions. a is simply set to 0.5 in our experiments.
Extension to partial OT. We extend the above KPG-RL model to the more practical partial OT setting that only partial mass is transported. To do this, we first apply the above mask-based constraint of transport plan to the partial OT model in Eq. (2). We then add more constraints to enforce that all the mass of keypoints is transported to preserve the matching of keypoints. Finally, we define the Partial-KPG-RL model
min (P..
(Tepg (M OM
(13)
where (1' q: M) = { E REX* T) < (M o 1m 8; (M @ 1)., = pr, vi E I; @ T).j = gj, Vj E J]. Chapel et al. [20] propose compelling method to solve the original partial OT problem in Eq. (2) by transforming the partial OT
1m & q, (T 3 1) =
problem into an OT-like problem. Inspired by Chapel al. [20], to solve our partial-KPGG KPG-RL model

---

Matchlngg accuracy: 41.7%
Matching accuracy: 7%
Matching accuracy: 96.7%
(a) KP
(b) KPG-RL-KP (w/ keypoints)
KPG-RL-KP (w/ keypoints)
Matching accuracy: 6.7%
Matching accuracy: 96.7%
Matching accuracy: 96.1%cy:9967.. 96.7%
(d) GW
(c) KPG-RL-GW (w/ keypoints)
(f) KPG-RL-GW (w/ 3 keypoints)
Figure 4: Matching produced by (a) KP, (b) KPG-RL-KKP model given 2 keypoint pairs, (c) KPG-RL- KP model given keypoint pairs, (d) GW model, (e) KPG-RL-GW model given keypoint pairs, and (I) KPG-RL-GW model given 3 keypoint pairs.
Eq. (13), we add dummy point to each the marginal distributions transform Eq. (13) into KPG-RL-like problem. To do this, we extend p, q, G, M by
p 119 -
9 11
G
G {1
ELn 2 +
M
M
a 1
b
where A >> 0,5 > 0, R.m b RR.. The element a, of a is if I, otherwise 1. The element b, of b is 0 if E J. and I otherwise. For more motivations this extension, please refer to Appendix A.6. By the following theorem, solving the optimal transport plan of problem (13) boils down to solving the problem min (M @ т. G)F.
Theorem 1. Suppose A > 0, <> 0, Eier Pi < and EjeJ 9j <. 8, then the optimal transport plan M O n² of problem (13) is the m-by-n block in the upper left corner of the optimal transport plan M n" of problem min (M n. G)..
Please refer to Appendix A.7 for the proof of Theorem 1.
5
Experiments
We evaluate the keypoint-guided models on toy data, and apply them to HDA and open-set HDA.
5.1 Toy Data
As illustrated in Fig. 4, in this toy data experiment, each of the source (blue) and target (green) distributions is Gaussian mixture composed of three distinct Gaussian components indicated by different shapes where the same shapes indicate points of the same class. In Fig, 4, have the following observations. In Fig. 4(a), in the KP model, the points in each component of target distribution are mismatched to points different classes of source distributions, and only small fraction of target points are correctly matched to source points belonging to the same class. In Fig. 4(b), given keypoint pairs (from distinct classes), the KPG-RL-KP model apparently improves the correctness of matching. In Fig. 4(c), the KPG-RL-KP model mainly matches the source points to the target points belonging to the same class, thanks the given 3 keypoint pairs. This implies that proposed keypoint-guides model does help to produce correct matching in OT by leveraging the
7

---

Table 1: Accuracy on Office-31 for HDA. "A", "W", and "D" are respectively the domains of amazon, webcam, and dslr. " - *" denotes a heterogeneous adaptation task where and are respectively source domain using DeCAFa feature and target domain using ResNet-50 feature.
Method
A-A
A
A-W
D-
D-D
D
W
W
-A
W-D
W-W
Avg
Labeled-target-only
45.3
69.2
67.3
45.3
69.2
67.3
45.3
69.2
67.3
60.6
STN [50]
58.7
84.8
80.0
51.2
91.0
83.8
52.8
95.2
87.4
76.1
SSAN [51]
56.8
89.7
87.1
54.2
82.6
90.3
50.0
85.8
85.8
75.8
DDACL [52]
44.2
63.2
64.2
43,9
77.4
70.6
39.8
64.5
73.2
60.1
CDSPP [45]
55.5
79.7
76.5
43.2
80.6
84.2
47.4
78.7
84.5
70.0
SGW [37]
49.7
77.7
73.6
49.4
78.4
73.6
48.7
80.3
74.5
67.3
GW [27]
33.6
41.6
35.5
39.7
40.0
31.7
34.8
34.5
29.0
35.6
GW (w/ mask)
41.3
71.6
69.7
41.9
71.2
69.8
40.3
71.6
69.7
60.8
HOT [33]
39.0
44.8
40.0
31.3
52.6
44.8
29.7
60.0
56.5
44.3
HOT (w/ mask)
45.2
60.3
57.4
48.9
63.5
59.2
40.3
67.1
61.4
55.9
TLB [27]
29.4
36.5
43.2
24.5
31.3
51.0
23.6
31.9
49.7
35.7
TLB (w/ mask)
42.5
66.3
64.7
38.5
68.5
65.9
43.1
68.2
67.3
58.3
KPG (w/ dist)
55.2
60.7
71.6
51.3
71.9
77.1
48.7
70.0
77.7
64.9
KPG-RL-GW
58.7
92.9
84.2
57.4
95.5
87.1
55.5
95.5
90.0
79.6
KPG-RL (LP)
56.5
93.6
83.2
58.1
94.5
86.8
55.8
95.2
89.7
79.3
KPG-RL (SH)
60.0
91.6
83.6
57.4
95.8
87.7
59.1
95.2
88.4
79.9
given a few keypoint pairs. In Figs. 4(d), 4(e) and 4(f), the proposed KPG-RL-GW model improves the correctness of matching of GW model by leveraging the guidance of given keypoints pairs. Please refer Appendix B. for the toy experiment for evaluating the partial-KPG-RL model.
5.2 Heterogeneous Domain Adaptation
In HDA, we given large amount of labeled source domain data {124 (4) )m 1=11 few labeled target domain data {uz, Fy 1 and large number of unlabeled target domain data {3/ fj=n++1> where and y; are features, and f; and f, are respectively the class labels of and yj-
m >> ny, >> The heterogeneity means that x; and Yj are from different spaces/modalities. The goal of HDA to train classification model using the given data to predict the label of unlabeled target domain data, leveraging the knowledge of large-scale labeled source domain data. The main challenge that the domain gap between heterogeneous source and target distributions supported in distinct spaces hinders the direct employment of the source domain trained model in the target domain.
We tackle the problem of HDA using our proposed KPG-RL model as follows. We first transport the source domain data using our KPG-RL model to the target domain. More concretely, in KPG-RL. each labeled target domain data and the source domain class center of the same class are taken as matched keypoint pair. The KPG-RL model then performed between empirical distributions of the source domain data along with class centers and the target domain data (including labeled and unlabeled data). Based on the produced optimal transport plan, we transport the source domain data using the barycentric mapping [49] to the target domain. Finally, we train the classification model (taken as a kernel SVM) on the transported source domain data (using their class label before transport) and the labeled target domain data, which is applied to the target domain test data. is set to 0.005. More details of the barycentric mapping, the kernel SVM, etc, are given in Appendix B.4.
We compare our method with the following baseline methods, including 1) "Labeled-targee-onny that trains the kernel SVM using the labeled target data; 2) the OT methods of "GW" "HOT", and "TLB" that transport source domain data using barycentric mapping induced by the transport plan of GW [27], Hierarchical OT [33], and TLB [27]. and then train the kernel SVM on the transported data and labeled target domain data; 3) the typical HDA methods of "SGW" [37] and "STN" [50], and the recent HDA methods of "SSAN" [51], "DDACL" [52] CCDPPD [45]. We conduct experiments on Office-31 [53] dataset. On Office-31, we use the DeCAF, [54] features and the features extracted
by ResNet-50 [55] pretrained on ImageNet [56] to respectively build source and target domains for
constructing heterogeneous adaptation tasks. In each task, one labeled data (1-shot) for each class is

---

Table 2: Results of methods of -aabeleed-trret-onii and KPG given different shots (1, 2, and 3) of labeled target domain data (keypoints) per-class under five distinct samplings (S1,52,53, S4, and S5).
I-shot
2 shot
3-shot
Method
S1
S2
S3
S4
S5
Avg
S3
S4
Avg
S1
S2
S4
S5 Avg
Laheled4arget-oolyy6122
58.4 64.1 60.6
60.6
77.8
76.3 78.3 75.0 74.2
76.3
180.7
76.6
81.0
82.7 80.9
KPG-RI. (SH)
(77.6 76.2 82.1 82.9
80.7
79.9 86.5 86.5 86.8 84.4 82.6 85.4
86.5 88.5
87.0
88.3 88.2 87.7
given in target domain. Note all the methods use the same training data (including labeled source and target domain data, and unlabeled target domain data) and test data.
Table I reports the results on Offee-31 dataset. In Table 1, "KPG-RL (SH)" and "KPG-RL (LP)" denote our KPG-RL model solved using Sinkhorn's 's algorithm and linear programming respectively. KPG-RL (SH) achieves marginal better average accuracy than KPG-RL (LP), which could be because the barycentric mapping based on the more dense transport plan optimized by Sinkhorn's 's algorithm is more robust to wrong matching, We also observe that KPG-RL-GW achieves slightly lower average accuracy than KPG-RL (SH), and KPG-RL (SH) is more computationally efficient as discussed in Sect. 4. "KPG (w/ dist)" is the approach that imposes the guidance of keypoints using distance preservation in our framework, i.e., Ri, and Rí in Eqs. (7) and (8) are taken as Cg, and Ci,,, respectively. KPG-RL (SH) improves the accuracy of KPG (w/ dist) by 15%, implying that our relation-preserving scheme is more effective for imposing the guidance of keypoints than the distance-pree parenccigreriingggge.o. scheme. Table 1 shows that the results of GW, HOT, and TLB are inferior to that of tareeldd-rrgttonly... indicating that GW, HOT, and TLB cause negative transfer. This is reasonable because GW, HOT, and TLB do not take advantage of the guidance of keypoints and can cause wrong matching. We also use our mask-based constraint on the transport plan in GW, HOT, and TLB, of which the corresponding approach are denoted as GW (w/ mask), HOT (w/ mask), TLB (w/ mask). We can observe that with the mask-based constraint, the performances of GW, HOT, and TLB improved but worse than KPG-RL (SH) and KPG-RL (LP), verifying the importance of the relation preservation scheme in KPG-RL. SGW [37] aligns the class centers of labeled target domain data and transported source domain data in the GW model, and realizes positive transfer. Our proposed KPG-RL (SH) outperforms SGW by 12.2%, confirming that our KPG-RL model preserving relation is more effective than SGW preserving class centers for HDA. Compared with the other HDA methods, KPG-RL (SH) achieves the best average accuracy.
The keypoints are important to KPG-RL, since they used to guide the matching. We show the results with varying keypoints by giving different numbers and sampling of labeled target domain data, in Table 2. It can be observed in Table 2 that under different numbers and samplings of labeled target domain data, our approach KPG-RL (SH) consistently outperforms the baseline Labeled-target--nn achieving positive transfer. We also find that as the number of labeled target domain data increases, the margin between the accuracy of Labeled-target-onlly and KPG-RL (SH) decreases. This indicates that given more labeled target domain data, the necessity of knowledge transferred from source domain becomes smaller. Please refer to Appendix B.5 for the effect of source keypoints.
5.3 Open-Set Heterogeneous Domain Adaptation
We conduct open-set HDA experiments evaluate our proposed partial-KPG-RL model. The problem setting of open-set HDA is the same as HDA, except that the unlabeled target domain data contain samples of unknown classes absent in the categories of labeled data. The goal of open-set HDA is to correctly classify the common class samples and to detect the unknown class samples. We consider the case that the proportion 17 of unknown class samples in the unlabeled target domain data PrrtialKKPG-RR to open-set HDA, the keypoint pairs are constructed using labeled target domain data and their corresponding source class centers, same as KPG-RL for HDA in Sect. 5.2. We use the Partial-KPG-R model in Eq. (13) to transport all the labeled source domain data to partially match the target domain data. The unmatched target domain data are detected as unknown class. We then train a kernel SVM on the transported source domain data and labeled target domain data to classify the common class samples detected by partial-KPG-RR The natural Baseline for open-set HDA is the approach that rejects the p-pproootionn unlabeled samples with largest distance to labeled target domain data as unknown, and trains kernel SVM on the labeled target domain data to classify common class data. We also evaluate the HDA methods STN [50] and SSAN [51] in the
is given. For applying

---

Table 3:
Accuracy of common class sample (OS*), Accuracy of unknown class sample (UNK), and their harmonic mean (HOS) on Office-31 for open-set HDA (1) = 0.67).
A
A->D
A
D-+A
D->
Method
OS*
UNK
HOS
OS"
UNK
HOS
OS
UNK
HOS
OS'
UNK
HOS
OS
UNK
HOS
Baseline
40.0
71.0
51.2
37.3
85.3
51.9
29.1
82.7
43.0
40.0
71.0
51.2
37.3
85.3
51.9
STN [50]
48.2
80.6
60.3
67.3
86.6
75.7
54.6
78.4
64.3
46.3
76.6
57.8
63.6
84.4
72.6
SSAN [51]
25.4
66.7
36.8
29.1
68.4
40.8
64.5
83.1
72.7
34.6
70.6
46.4
22.7
64.1
33.6
Partial-KPG-RI.
41.8
77.1
54.2
48.2
74.9
58.6
38.2
73.2
50.2
52.7
82.3
64.3
80.0
92.6
85.9
D-W
W-+A
W--D
W-+W
Avg
Method
OS
UNK
HOS
OS"
UNK
HOS
OS"
UNK
HOS
OS
UNK
HOS
OS"
UNK
HOS
Baseline
29.1
82.7
43.0
40.0
71.0
51.2
37.3
85.3
51.9
29.1
82.7
43.0
35.5
79.7
48.7
STN [50]
54.6
79.2
64.6
49.9
78.4
60.4
60.0
82.6
69.5
55.5
79.2
65.2
55.6
80.7
65.6
SSAN [51]
29.1
66.4
40.4
31.8
68.4
43.3
19.1
64.5
29.4
26.4
67.9
37.9
31.4
68.9
42.4
Partial-KPG-RI
73.6
89.2
80.7
52.7
82.7
64.4
78.2
90.9
84.1
71.8
88.3
79.2
59.7
83.5
69.1
open-set HDA task, For STN [50] and SSAN [51]. we reject the 17-rrpportion unlabeled samples with lowest prediction confidence unknown class. Table 3 reports the results. We use the open-set evaluation metrics [57] including aceuracy of common class sample (OS*), accuracy of unknown class sample (UNK), and their harmonic mean HOS = OS x UNK We can observe in Table 3 that OS + UUN
our proposed partial-KPGG KPG-RL achieves the best results terms average OS*, UNK, and HOS, indicating that the partial-KPCC KPG-RL is effective for both classifying common class data and identifying unknown class data. Compared with the Baseline, partial-KPG-RU outperforms by 20.4% in terms of average HOS, confirming the positive transfer achieved by our method. More implementation details and the solution for Open-Set HDA with unknown v are given in Appendix B.2.
Due to space limit, we include the experiment of our approach for deep unsupervised domain adaptation in Appendix B.3. More empirical analysis and ablation studies, e.g., sensitivity hyper- parameters, the effect of d, the discussion on how to define the keypoints in more general practical applications, the time and memory cost, etc., given in Appendix B.5.
Conclusion
In this paper, we propose novel KPG-RL model that leverages keypoints guide the correct matching in OT. We devise mask-based constraint to preserve the matching of keypoints pairs, and propose to preserve the relation of each point to the keypoints impose the guidance of these keypoints in OT. The effectiveness of the proposed KPG-RL model is verified in the application of HDA. The keypoint- OT model could be possibly applied more applications, e.g., cross-domain data generation/translatii [58, 59], and point-set image registration.
Acknowledgements. This work was supported by National Key R&D Program 2021 O02IYFA0003000 and NSFC (12125104, U1811461 1461, U20B2075, 11971373, 12026605).
References
[1] Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
[2] Nicolas Bonneel, Michiel Van De Panne, Sylvain Paris, and Wolfgang Heidrich. Displacement interpolation using lagrangian mass transport. In SIGGRAPII Asia, 2011.
[3] Justin Solomon, Fernando De Goes, Gabriel Peyré, Marco Cuturi, Adrian Butscher, Andy Nguyen, Tao Du, and Leonidas Guibas. Convolutional wasserstein distances: Efficient optimal transportation on geometric domains. ACM ToG, 34(4):1-11, 2015.
[4] Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In ICML, 2015.
10

---

[5] Gao Huang, Chuan Guo, Matt J Kusner, Yu Sun, Fei Sha, and Kilian Q Weinberger. Supervised word mover's distance. NeurIPS, 2016.
[6] David Alvarez-Melis and Tommi S Jaakkola. Gromov-wasserstein alignment of word embedding spaces. In EMNLP, 2018.
[7] Mikhail Yurochkin, Sebastian Claici, Edward Chien, Farzaneh Mirzazadeh, and Justin M Solomon. Hierarchical optimal transport for document representation. In NeurlPS, 2019.
[8] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML,
[9] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE Trans. PAMI, 1953)1883311853 2017.
[10] Nhat Ho, Xuanlong Nguyen, Mikhail Yurochkin, Hung Hai Bui, Viet Huynh, and Dinh Phung. Multilevel clustering via wasserstein means. In ICML, 2017.
[11] Alexander Tong, Guy Wolf, and Smita Krishnaswamy, Fixing bias in reconstruction-bel bbase anomaly detection with lipschitz discriminators. JSPS, 94(2):29-2433, 2022.
[12] Gaspard Monge. Mémoire sur la théorie des déblais et des remblais. Histoire de I' l'Académie Royale des Sciences de Paris, pages 666-704, 1781.
[13] Leonid V Kantorovich. On the translocation of masses. In Dokl. Akad. Nauk. USSR (NS), 1942.
[14] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NeurIPS, 2013.
[15] Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. PJM, 21(2):343-348, 1967.
[16] Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn divergences. In AISTATS, 2018.
[17] Kevin Guittet. Extended Kantorovich norms: tool for optimizaation. PhD thesis, INRIA, 2002.
[18] Alessio Figalli. The optimal partial transport problem. Arch Ration Mech Anal, 195(2):533-560, 2010.
[19] Luis A Caffarelli and Robert J McCann. Free boundaries in optimal transport and monge-ampere obstacle problems. Ann Math, 171:673-730, 2010.
[20] Laetitia Chapel, Mokhtar Z Alaya, and Gilles Gasso. Partial optimal transport with applications on pusitive-umlabeled learning. In NeurlPS, 2020.
[21] Lenaic Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xavier Vialard. An interpolat- ing distance between optimal transport and fisher-rao metrics. FoCM, 18(1):1-44, 2018.
[22] Matthias Liero, Alexander Mielke, and Giuseppe Savaré. Optimal entropy-transport problems and new hellinger-kantorovich distance between positive measures. Invent Math, 21 131: 1117, 2018.
[23] Yogesh Balaji. Rama Chellappa, and Soheil Feizi. Robust optimal transport with applications in generative modeling and domain adaptation. In NeurIPS, 2020.
[24] Debarghya Mukherjee, Aritra Guha, Justin M Solomon, Yuckai Sun, and Mikhail Yurochkin. Outlier-robust optimal transport. In JCML, 2021.
(25) Khang Le, Huy Nguyen, Quang M Nguyen, Tung Pham, Hung Bui, and Nhat Ho. On robust optimal transport: Computational complexity and barycenter computation. In NewrlPS, 2021.
[26] Khrl-Theodor Sturm. On the geometry of metric measure spaces. Acta Math, 196(1):65-131 2006.

---

[27] Facundo Mémoli. Gromov-wassersten distances and the metric approach to object matching. FoCM, 1144)4477-477, 2011.
[28] Andriy Myronenko and Xubo Song. Point set registration: Coherent point drift. IEEE Trans. PAMI, 32(12):226-22275 2010.
[29] William R Crum, Thomas Hartkens, and DLG Hill. Non-rigid image registration: theory and practice. Br J Radiol, 77(5umpp_2::144--1555 2004.
[30] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In ICCV, 2019.
[31] Yao-Hung Hubert Tsai, Yi-Ren Yeh, and Yu-Chiang Frank Wang. Learning cross-domain landmarks for heterogeneous domain adaptation. In CVPR, 2016.
[32] Chi-Heng Lin, Mehdi Azabou, and Eva L Dyer. Making transport more robust and interpretable by moving data through a small number of anchor points. In ICML, 2021.
[33] John Lee, Max Dabagia, Eva Dyer, and Christopher Rozell. Hierarchical optimal transport for multimodal distribution alignment. In NeurlPS, 2019.
[34] Hongteng Xu, Dixin Luo, Ricardo Henao, Svati Shah, and Lawrence Carin, Learning autoen- coders with relational regularization. In ICML, 2020.
[35] Ryoma Sato, Marco Cuturi, Makoto Yamada, and Hisashi Kashima. Fast and robust comparison of probability measures in heterogeneous spaces. arXiv preprint arXiv:2002.01615, 2020.
[36] Jiying Zhang, Xi Xiao, Long-Kai Kai Huang, Yu Rong, and Yatao Bian. Fine-tuning graph neural networks via graph topology induced optimal transport. IJCAI, 2022.
[37] Yuguang Yan, Wen Li, Hanrui Wu, Huaqing Min, Mingkui Tan, and Qingyao Wu. Semi- supervised optimal transport for heterogeneous domain adaptation. In I/CAI, 2018.
[38] Chen Shen and Yuhong Guo. Unsupervised heterogeneous domain adaptation with sparse feature transformation. In ACML,,
[39] Azadeh Sadat Mozafari and Mansour Jamzad. A svm-based model-transferring method for heterogeneous domain adaptation. PR, 56: 15881, 2016.
[40] Joey Tianyi Zhou, Ivor W Tsang, Sinno Jialin Pan, and Mingkui Tan. Multi-class heterogeneous domain adaptation. JMLR, 2019.
[41] Chang Wang and Sridhar Mahadevan. Heterogeneous domain adaptation using manifold alignment. In IJCAI, 2011.
[42] Hanrui Wu, Hong Zhu, Yuguang Yan, Jiaju Wu, Yifan Zhang, and Michael K Ng. Heterogeneous domain adaptation by information capturing and distribution matching. IEEE Trans. IP, 30:6364- 6376, 2021.
[43] Yuguang Yan, Wen Li, Michael KP Ng. Mingkui Tan, Hanrui Wu, Huaqing Min, and Qingyao Wu. Learning discriminative correlation subspace for heterogeneous domain adaptation. In IJCAI, 2017.
[44] Joey Tianyi Zhou, Sinno Jialin Pan, and Ivor W Tsang. A deep learning framework for hybrid heterogeneous transfer learning. AI, 275:370-11-2 108328, 2019.
[45] Qian Wang and Toby P Breckon. Cross-domain structure preserving projection for heteroge- neous domain adaptation. PR, 123:108362, 2022.
[46] Zhen Fang, Jie Lu, Feng Liu, and Guangquan Zhang. Semi-supervised heterogeneous domain adaptation: Theory and algorithms. IEEÉ Trans. PAMI, In press, 2022.
[47] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607, 2020.
12

---

[48] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. NeurIPS, 2020.
[49] Sebastian Reich. nonparametric ensemble transform method for bayesian inference. SIAM JSC, 35(4):22013-A2224, 2013.
[50] Yuan Yao, Yu Zhang. Xutao Li, and Yunming Ye. Heterogeneous domain adaptation via soft transfer network. In ACM MM, 2019.
[51] Shuang Li, Binhui Xie, Jiashu Wu, Ying Zhao, Chi Harold Liu, and Zhengming Ding. Simulta- neous semantic alignment network for heterogeneous domain adaptation. In ACM MM, pages 3866-3874, 2020.
[52] Yuan Yao, Yu Zhang, Xutao Li, and Yunming Ye. Discriminative distribution alignment: A unified framework for heterogeneous domain adaptation. PR. 101:107165, 2020.
[53] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, 2010.
[54] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML., 2014.
[55] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[56] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy. Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-255 2015.
[57] Silvia Bucci, Mohammad Reza Loghmani, and Tatiana Tommasi. On the effectiveness of image rotation for open set domain adaptation. In ECCV, pages
2020.
[58] Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
[59] Heran Yang, Jian Sun, Aaron Carass, Can Zhao, Junghoon Lee, Jerry Prince, and Zongben Xu. Unsupervised mr-to-ct synthesis using structure-constrrinne cyclegan. IEEE Trans. MI, 49112):4449-4261, 2020.
[60] Morgan A Schmitz, Matthicu Heitz, Nicolas Bonneel, Fred Ngole, David Coeurjolly, Marco Cuturi, Gabriel Peyré, and Jean-Luc Starck. Wasserstein dictionary learning: Optimal transport- based unsupervised nonlinear dictionary learning. SIAM JIS, 11(1):643-678, 2018
[61] Vayer Titouan, Nicolas Courty, Romain Tavenard, and Rémi Flamary. Optimal transport for structured data with application graphs. In ICML, 2019.
[62] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 11(5-6)3355607, 2019.
[63] Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. MI. 109(4):71--76,, 2020.
[64] Daniel Zeiberg, Shantanu Jain, and Predrag Radivojac. Fast nonparametric estimation of class proportions in the positive-unlabeled classification setting. In AAAI, 2020.
[65] Khai Nguyen, Dang Nguyen, Tung Pham, Nhat Ho, et al. Improving mini-batch optimal transport via partial transportation. ICML, 2022.
[66] Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In ECCV, 2018
[67] Hemanth Venkateswara, Jose Fusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
13

---

[68] Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsupervised domain adaptation. In ICML, 2018.
[69] Kilian Fatras, Thibault Séjourné, Rémi Flamary, and Nicolas Courty. Unbalanced minibatch optimal transport; applications to domain adaptation. In ICML, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Abstract and Introduction.
(b) Did you describe the limitations of your work? [Yes] See the end of the first paragraph on page 6.
(c) Did you discuss any potential negative socictal impacts of your work? [N/A] This work does not present any significant societal consequence.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Propositions and A-1, and Theorems I, A-1, A-2, A-3. (b) Did you include complete proof's of all theoretical results? [Yes] See Appendix A.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Sect. 5.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sect. 5 and Appendix B.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] We have reported the results for different samples Table 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] The experiments are conducted on CPU.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sect. 5.
(b) Did you mention the license of the assets? [N/A] The datasets are public.
URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you'l using/curating? [N/A]
(c) Did you include any assets either in the supplemental material
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "vvrcEPf0Bs", "reviewer": "Reviewer_QaNa", "summary": "Paper proposes to guide the optimal transport plan by imposing a priori some couplings between \"keypoints\" into the transport matrix. The relation to each keypoint is also used to define a \"guiding\" matrix that is considered as the cost matrix of the transport problem. Several variants are proposed, such as a GW- or a partial formulation of the problem. The experimental setup considers the Heteregeneous Domain Adaptation (HDA) scenario, in which it is assumed that some points (of all the classes) of the target distribution are labelled. ", "questions": "Could you clarify \n- why one should use your method than hierarchical OT or TLB?\n- could it be extended to the case where the number of keypoints in the source and target distributions are different?", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Paper lies in a line of works that aims at constraining the OT problem to encode some extra information or to fasten the computation. Regarding the use of key points, it is quite common to use few anchor points to supervise some of the matching between points or distributions. From a computational point of view, the problem is rewritten by introducing a mask matrix and an algorithm is proposed to solve the problem. To my knowledge, introducing labeled key points to constrain the matching is original. In the context of HDA, experiments show that it is not enough to beat SotA methods. To improve the classification performances, a dedicated cost matrix is constructed. It is based on a function of the distance of each point to each key points within each domain. \nThe methodology shares similarities with some OT variants, notably with hierarchical OT. In the latter, instead favoring matchings between some keypoints, a matching between subgroups (e.g. clusters) is sought. It also shares some similarities with TLB (« third lower bound » of Gromov-Wasserstein), in which each point of each domain is described as a set of distances to all the points of the same domain, and a « wasserstein of wasserstein » of these distances is computed. The originality and advantages of this « relation preservation » guiding matrix w.r.t. those alternatives is not discussed. No comparison of the performances is neither performed. This constitutes the main weakness of the paper as this guiding matrix seems to be the main ingredient to improve the performances. \nThe experimental validation of the method is performed on a heterogeneous domain adaptation context. In this context, all the labels from the source distribution are known and only few of the target ones are provided. It is shown that adding only the keypoints to drive the matching is not enough to beat the SotA (10 point behind the best competitor) but that adding the guiding matrix improves the results. Additional experiment shows that performances increase when the number of keypoints. It is unclear if the competitors also use some labeled points to drive the learning, which makes the results with the proposed method and SotA difficult to compare. \n\n\nStrengths of the paper:\n- the paper is really well written and easy to follow; figures illustrate clearly the proposed method.\n- the methodology is sound and performs favorably in the considered HDA scenario.\n- extensions with entropic regularization or partial OT is provided.\n\nWeaknesses:\n- insufficient comparison with similar OT variants such as hierarchical OT and TLB\n- experimental setup that does not fit exactly the proposed method: while the hypothesis that we have access to labels of some of the target distribution makes sense, it is unclear how the method is sensitive to the choice of the keypoints within the source domain, or if it could be possible to use all the labeled information of this source domain.  \n- it is unclear how the setting can be extended to deep method, where mini batches are used. \n\n\nMinor comments :\n- «  Fuzzed OT » should read « fused OT »\n- « Frank-Walfe » algorithm\n- «  Offce 31 »\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "li4O7btuyS", "reviewer": "Reviewer_BBx7", "summary": "This paper presents the KeyPoint-Guided model by ReLation preservation (KPG-RL), which allows to combine annotation information into many popular optimal transport (OT) models. The authors show the very competitive performance when integrating KPG-RL into the classical OT models, for the heterogeneous domain adaptation task.", "questions": "- I find the idea of introducing $G$ interesting and smart. So I am curious to see how would KPG-RL perform without $G$?, i.e. what if we only use $L_{kpg}(\\pi) = \\langle M, \\pi \\rangle_F$?\n\n- Each entry in the matrix $G$ is the Jensen-Shannon (JS) divergence between two probability vectors. How does the choice of measure of similarity impact the performance of KPG-RL? Why JS divergence but not the KL divergence, or $L_2, L_1$ distances, or even Wasserstein distance (which is not computationally expensive in this case).\n\n- Do the authors have the motivation to prefer the KPG-RL to the fused GW model above?\n\n- What is the range of $\\alpha$ in the equations $11$ and $12$? Is it $[0,1]$? Is it tuned during the training?\n\n- The alignment in the Fig 4 is not very visible.\n\n- In the experiment 5.2, how is the target data transformed for the GW model?\n\n- In the Appendix, please provide more details on the barycentric mapping in $[48]$.", "limitations": "No, the authors do not discuss these points.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and the authors have well illustrated the advantage and motivation of KPG-RL over the traditional GW and KP models.\n\nHowever, I would say that the main contribution is mostly on the methodology, so more theoritical understand would be desirable. Apart from a few theoritical results which help validating the intuition and motivation, I think it is also necessary to convince that KPG-RL is a reliable divergence/metric (even though the empirical evidence suggests that it is). More precisely, given prior correct matching of keypoint pairs,\n\n- In the setting of OT distance, where points lie on the same ground space, do we have that $L_{kpg} = 0$ (or $\\text{KPG-RL-KP} = 0$) is equivalent to the equality of two reference measures $p$ and $q$?\n\n- When KPG-RL is incorporated in the GW model (i.e. KPG-RL-GW), does it still preserve the isomorphism?, i.e. when the two graphs are isomorphic (in the usual sense of GW distance), do we have $\\text{KPG-RL-GW} = 0$, and vice versa?\n\n- Can we show that KPG-RL or any of its variations (GW and LP) defines a proper metric?\n\nMoreover, while the usual GW model performs very poorly because it can not incorporate prior information, the fused GW model $[35]$ can and should be considered as competing method. For example, one can consider the fused GW model of the form: $\\min_{\\pi \\in \\Pi(p,q)}\\alpha L_{gw}(\\pi) + (1 - \\alpha) L_w(\\pi)$, where either $L_w(\\pi) = \\langle M, \\pi \\rangle$, or $L_w(\\pi) = \\langle M \\odot G, \\pi \\rangle$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "eOO2QON-zu6", "reviewer": "Reviewer_n57r", "summary": "This paper utilizes annotated keypoints to tackle incorrect matchings of optimal transport models. First, the authors impose a mask-based constraint on the optimal transport problem to preserve the matching of keypoint pairs. Secondly, they propose to preserve the relation of each data to the keypoints by defining a new cost matrix. Their method, named KeyPoint-Guided model by ReLation preservation (KPG-RL), can be solved by the Sinkhorn’s algorithm and even supported in incomparable spaces. Furthermore, they extend KPG-RL to the partial OT setting. Finally, they verify the effectiveness of their proposal in the heterogeneous domain adaptation application, both close-set and open-set.", "questions": "I have the following questions:\n1. **Related work**\n* The OT problem in “Preservation of matching of keypoints in transport” is similar to the Masked Optimal Transport (Masked OT) [r1]. Please clarify the difference and contribution of this paper compared to the “previous” work.\n* In the experimental part, a comparison with Masked OT and Masked GW [r1] can be conducted to illustrate the advantages of the proposed methods. For instance, KPG (w/dist) in Section 5.2 is a good candidate.  \n[r1] Zhang, Jiying, et al. \"Fine-tuning graph neural networks via graph topology induced optimal transport.\" arXiv preprint arXiv:2203.10453 (2022).\n\n2. **Relation score.** What is the explanation/intuition/inspiration for defining the relation score in Equation 7? Because there are several forms that can satisfy the properties in L170-172. The authors should discuss the choice of the relation score in more detail because, in my opinion, it is one of the major contributions compared to Masked OT.\n\n3. **Cost function.** An ablation study or an explanation for the choice of d, which is set to Jensen-Shannon in this paper, is recommended. \n\n4. **Reproducibility.** A lot of experimental settings are missing and there is no code for reproduction?\n* The details of the kernel SVM are missing? \n* What is the value of $\\alpha$ for each KPG-based method in Section 5.1?\n* What is the value of hyper-parameters in Section 5.2?\n\n5. **Experiments.** \n* *Sensitivity to $\\epsilon$.* The range of $\\epsilon$ in Appendix B.2 should be wider, e.g. $\\epsilon = 0.001, 0.01, 0.1, 1, \\ldots$.\n* *Open-set HDA.* Does the set of common labels of the target domain contain or equal the label set of the source domain? If it is the case, it is reasonable to transport more than 1 - $\\eta$-proportion (even all) of labeled source domain data. This could be a plausible explanation for the finding in L227-228 in Appendix B.3.\n* *Sensitivity to hyper-parameters.* From Tables A-2, A-3, and especially A-5, it is difficult to conclude that KPG-RL is stable to different choices of hyper-parameters. \n\n**Minors**\n* In Figure 2, the entry in row 3 and column 6 is 0, which is missing. \n* The line numbers in Checklist are outdated.\n\nI am happy to increase my score if the above questions are adequately addressed.\n\n", "limitations": "The authors stated that the limitation of their method is in L184-185 but I think what they meant was L179-182. Other than that, one additional limitation of KPG-RL is the introduction of new hyper-parameters, which is the temperature $\\tau$ and the dissimilarity function $d$. Together with $\\epsilon$ in the entropic regularization, users have to tune a lot to find the best set of hyper-parameters. \n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Though the mask-based constraint version of OT in this paper is not completely new (more details in the next section), the proposed solution, KPG-RL, in this paper is novel and interesting. In terms of the theoretical results, they are simply the extensions of previous works. Moving onto the experimental parts, KPG-RL shows a noticeable improvement over baseline methods in heterogeneous domain adaptation. In addition, the authors also conducted an ablation study to show the stability of their proposal for different choices of hyper-parameters. Overall, this paper is easy to follow with some good illustrations which are really helpful in explaining the concepts. Two major concerns include the lack of references to related works and experimental settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "57aGvEl2wyP", "reviewer": "Reviewer_uAn8", "summary": "The paper introduces a semi-supervised OT formulation and its solutions and applied it to solving heterogeneous domain adaptation. The weak supervision comes from limited labeling of a set of key points in source and target domains. The authors constructed a key point-guided model by the doc product of the transport plan masked by the binary key point connection and a relation matrix between source and target. They then appended the model to the OT formulation as a weighted regularization term. Finally, they extended their model to partial OT problems. Experiments on Office-31 showed that the proposed model achieved better results overall than several existing methods.", "questions": "---\nWhat's the difference between the idea of key point-guided OT and the idea of semi-surprised OT in OTDA by Courty et al. [9]?\n\n---\nLine 138: \"If the paired key points (I,j) are matched, ... all mass of point $x_i$ must be transported to $y_j$ and $y_j$ can only receive the mass from $x_i$.\"\n\nI miss the motivation of this idea. Since we're solving Kantorovich OT, how do we expect the connection between key points and non-key points are one-to-one?\n\n---\nLine 161: Why do we set it to $0.1 \\times \\max_{I,k}\\{C_{I,k}^{s}\\}$? Is $\\tau$ tunable and thus cross-validated?\n\nWhere do $R_{k, i_u}^{s}$ and $G_{k,l}=d(R_k^s, R_l^t)$ come from? Why do we choose these discrepancies to construct the relation?\n\nPlus, $G$ would take quite a lot of memory if $m$ and $n$ are large. What is the peak memory consumption during the experiments with Office-31?", "limitations": "No negative societal impact found. See above for limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The formulations (11-13) are novel to the best of my knowledge.\n+ Theorem 1 looks good. The authors solved their model on partial OT problems by adding dummy components to the formulation and later proved that solution to the original problem can be easily derived from the solution to the reformulated problem and the reformulated problem is solvable.\n\n- The new formulation is quite expansive in memory (perhaps in time as well) because of additional pair-wise discrepancies and the authors didn't discuss that at all.\n- The authors didn't discuss the difference between their key point-guided formulation and semi-supervised OT. I feel that $L_{kpg}$ can be directly incorporated into $L_{OT}$ by designing a more informative mask $M$ that directly encodes the \"relation\". It looks redundant to me that the authors had to design a binary mask but then treat it as a regularizer not a constraint. Perhaps it's a dead end but I feel in general the authors didn't clearly explain their motivation and ideas.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept