Feature-Robust Optimal Transport for High-Dimensional Data
Authors: Mathis Petrovich
Chao Liang
Ryoma Sato
Yanbin Liu
Yao-Hung Hubert Tsai
Linchao Zhu
Yi Yang
Ruslan Salakhutdinov
Makoto Yamada
Venue:  (2021)
Keywords: Optimal Transport
feature selection
semantic correspondence

Abstract: Optimal transport is a machine learning problem with applications including distribution comparison, feature selection, and generative adversarial networks. In this paper, we propose feature-robust optimal transport (FROT) for high-dimensional data, which solves high-dimensional OT problems using feature selection to avoid the curse of dimensionality. Specifically, we find a transport plan with discriminative features. To this end, we formulate the FROT problem as a min--max optimization problem. We then propose a convex formulation of the FROT problem and solve it using a Frank--Wolfe-based optimization algorithm, whereby the subproblem can be efficiently solved using the Sinkhorn algorithm. Since FROT finds the transport plan from selected features, it is robust to noise features. To show the effectiveness of FROT, we propose using the FROT algorithm for the layer selection problem in deep neural networks for semantic correspondence. By conducting synthetic and benchmark experiments, we demonstrate that the proposed method can find a strong correspondence by determining important layers. We show that the FROT algorithm achieves state-of-the-art performance in real-world semantic correspondence datasets.

Under review as a conference paper at ICLR 2021
FEATURE-ROBUST OPTIMAL TRANSPORT FOR HIGH-DIMENSIONAL DATA
Anonymous authors
Paper under double-blind review
ABSTRACT
Optimal transport is machine learning problem with applications including distribution comparison, feature selection, and generative adversarial networks. In this paper, we propose feature-robust optimal transport (FROT) for high- dimensional data, which solves high-dimensional OT problems using feature se- lection to avoid the curse of dimensionality. Specifically, we find transport plan with discriminative features. To this end, we formulate the FROT problem as a min-max optimization problem. We then propose convex formulation of the FROT problem and solve it using a Frank-Woll Moae--Woll--aaeeee optimization algorithm, whereby the subproblem can be efficiently solved using the Sinkhorn algorithm. Since FROT finds the transport plan from selected features, it is robust to noise fea- tures. To show the effectiveness of FROT, we propose using the FROT algorithm for the layer selection problem in deep neural networks for semantic correspon- dence. By conducting synthetic and benchmark experiments, we demonstrate that the proposed method can find a strong correspondence by determining important layers. We show that the FROT algorithm achieves state-of-the-a performance in real-world semantic correspondence datasets.
INTRODUCTION
Optimal transport (OT) is machine learning problem with several applications in the computer vision and natural language processing communities. The applications include Wasserstein distance estimation (Peyré et al. 2019). domain adaptation (Yan et al. 2018), multitask learning (Janati et al.) 2019) barycenter estimation (Cuturi & Doucet 2014), semantic correspondence (Liu et al. 2020), feature matching (Sarlin et al.) 2019), and photo album summarization (Liu et al.) 2019). The OT problem extensively studied in the computer vision community as the earth mover's ' distance (EMD) (Rubner et al. 2000). However, the computational cost of EMD is cubic and highly expen- sive. Recently, the entropic regularized EMD problem was proposed: this problem can solved using the Sinkhorn algorithm with quadratic cost (Cuturi 2013). Owing to the development of the Sinkhorn algorithm, researchers have replaced the EMD computation with its regularized counter- parts. However, the optimal transport problem for high-dimensional data has remained unsolved for many years.
Recently, a robust variant of the OT was proposed for high-dimensional OT problems and used for divergence estimation (Paty & Cuturi 2019 2020). In the robust OT framework, the transport plan computed with the discriminative subspace of the data matrices X Ræ and Y R& The subspace can be obtained using dimensionality reduction. An advantage of the subspace robust approach that it does not require prior information about the subspace. However, given prior information such as feature groups, we can consider computationally efficient formulation. The computation of the subspace can be expensive the dimensionality of data is high, for example, 1
One of the most common prior information items is a feature group. The use of group features is popular in feature selection problems in the biomedical domain and has been extensively studied in Group Lasso (Yuan & Lin 2006). The key idea of Group Lasso is to prespecify the group variables and select the set of group variables using the group norm (also known as the sum of & norms). For example, if we use a pretrained neural network as a feature extractor and compute OT using the features, then we require careful selection of important layers compute OT. Specifically, each

---

Under review as a conference paper at ICLR 2021
Soua samples
Sounte sump
Source samples
-10
10
-10
16
-10
(a) OT on clean data.
(b) OT on noisy data.
(c) FROT on noisy data (1) = 1).
Figure transport plans between synthetic distributions with 10-dimensional vectors Ø (x _², = (y By where two-dimensional vectors ~ ( Eτ) and ~ NN (My En) are true features; and Zs ~ N (Os,, Is) and žy (Os, 18) are noisy features. (a OT between
distribution x: and y is reference. (b) OT between distribution ã and y. (c) FROT transport plan
between distribution E and y where true features and noisy features are grouped, respectively.
layer output is regarded as a grouped input. Therefore, using a feature group as prior information is natural setup and is important for considering OT for deep neural networks (DNNs).
In this paper, we propose high-diimensional optimal transport method by utilizing prior information in the form of grouped features. Specifically, propose feature-robust optimal transport (FROT) problem, for which we select distinct group feature sets to estimate a transport plan instead of deter- mining its distinct subsets, proposed in (Paty & Cuturi 2019 2020). To this end, we formulate the FROT problem as min max optimization problem and transform it into a convex optimization problem, which can be accurately solved using the Frank-Wan Wolfe algorithm (Frank Wolfe [956] Jaggi 2013). The FROT's subproblem can be efficiently solved using the Sinkhorn algorithm (Cu-) turi 2013). An advantage of FROT that it can yield a transport plan from high-dimensional data using feature selection, using which the significance of the features is obtained without any addi- tional cost. Therefore, the FROT formulation is highly suited for high-dimensional OT problems. Through synthetic experiments, we initially demonstrate that the proposed FROT is robust noise dimensions (See Figure|] I.. Furthermore, we apply FROT to a semantic correspondence problem (Liu et al. 2020) and show that the proposed algorithm achieves SOTA performance.
Contribution:
We propose feature robust optimal transport (FROT) problem and derive a simple and effi cient Frank-Wolft Wolfe based algorithm. Furthermore, we propose a feature-robust Wasserstein distance (FRWD).
We apply FROT high-dimensional feature selection problem and show that FROT is consistent with the Wasserstein distance-based feature selection algorithm with less com- putational cost than the original algorithm.
We used FROT for the layer selection problem in semantic correspondence problem and showed that the proposed algorithm outperforms existing baseline algorithms.
2 BACKGROUND
In this section, we briefly introduce the OT problem.
Optimal transport (OT): The following are given: independent and identically distributed (i.i.d.) samples X )R E Rdxx from d-dimensional distribution p. and i.i.d. samples Y {u,} d from the d-dimensional distribution q. In the Kantorovich relaxation of OT, admissible couplings are defined by the set of the transport plan:
V(n,v) = (II E RuXm III = a. II =b,
where II R2x= is called the transport plan, 1, is the n-dimensional vector whose elements ones, and a (a), ..,.... a,)) PT and b (bi, b2, ... bm) T R are the weights. The OT problem between two discrete measures μ Σn a;dm, and ΣØ= byōn; determines the
2

---

consider each feature Under review as a conference paper at ICLR 2021
optimal transport plan of the following problem:
min (pU()
(I)
where c(x, y) is a cost function. For example, the squared Euclidean distance is used, that is, c(x, y) x 2112. To solve the OT problem, Eq. ① (also known as the earth mover's dis- tance) using linear programming requires O(n²), (n m) computation, which computationally expensive. To address this, entropic-regularizzd optimal transport used (Cuturi 2013).
min (EU
Tydla y,) + eII
where c 2 0 is the regularization parameter, and HIII) entropic regularization. If c = 0, then the regularized OT problem reduces to the EMD problem. Owing entropic regularization. the entropic regularized OT problem can be accurately solved using Sinkhorn iteration (Cuturi 2013) with (mm) computational cost (See Algorithm
j==1 Fij (T)) - 1) is the
Wasserstein distance: the cost function defined as c(x, y) = d(x, y) with d(x; y) as a dis- tance function and p 2 1. then define the p- p-Wasserstein distance of two discrete measures Eiz a; and v = Σド= bydy, as
1/P
m Tyd y
Wp(p,v)=
min nEU((M))
Recently, robust variant of the Wasserstein distance, called the subspace robust Wasserstein dis- lance (SRW), was proposed (Paty & Cuturi 2019). The SRW computes the OT problem in the discriminative subspace. This can be determined by solving dimensionality Owing to the robustness, it can compute the Wasserstein from noisy data. The SRW is given as
problems.
SRW SRW(w, v)
min NεU()))) UERTXE UU-.
Tíy
U
U
(2)
where U the projection matrix with & S d, and Is R identity matrix. The SRW its relaxed problem can be efficiently estimated using either eigenvalue decomposition or the Frank-Worr algorithm.
3 PROPOSED METHOD
This paper proposes FROT. We assume that the vectors are grouped as (x(!)T
(L)) TyT
and y (y(1) ..... (L) )T Here, (0) Re and (()) IRde are the de dimensional vectors, where Σt= de d. This setting is useful if we know the explicit group structure for the feature vectors a priori. In an application in L-layer neural networks, we consider (!) and (+()) as outputs of the fth layer of the network. If we do not have priori information, independently (i.e., dj d2 = _... dr = 1 and L d). All proofs in this section are provided in the Appendix.
FEATURE- TOBUSR OPTIMAL TRANSPORT (FROT)
The FROT formulation is given by
m
acc( (1)
min NEU(),)
max DE
Tij
(3)
==1
3

---

Under review as a conference paper at ICLR 2021
where E' = fa R4: : a 1} is the probability simplex. The underlying concept of FROT is to estimate the transport plan II using distinct groups with large distances between {2) (0) J and (y) (0) pm 1j=1- We note that determining the transport plan in nondistinct groups is difficult because the data samples in {ar (o Jiz and (y) J-=1 ) overlap. By contrast, in distinet groups, {α) (a(4) )==1
and ) are different, and this aids in determining an optimal transport plan. This is an intrinsically similar idea to the subspace robust Wasserstein distance (Paty & Cuturi, 2019), which estimates the transport plan in the discriminative subspace, while our approach selects important groups. Therefore, FROT can be regarded as feature selection variant of the vanilla OT problem in Eq. II. whereas the subspace robust version uses ceeensionnalty-rrduttiinn counterparts.
Algorithm Sinkhorn algorithm.
Algorithm 2 FROT with the Frank-Wolfe.
Input: a, b. C, tiar
1: Input: {a; ( {8y 1=1 77 and e.
2: Initialize K = e 0/c = 1,:
2: Initialize II, compute (Ca)
1m: = 0
3: for = 0... T...T'do do
3: while 1 5 Iman and not converge do
4: II argmin nEU(())) (II, Mn +
4: u = a/K))
eH [I(++1) (1 II +사I
5: v b/(K 2)
6: / t= 1+1
6: with y 2+7'
2
7: end while
7: end for
8:
return II diag(w) (u) K ())
return II(T)
Using FROT, can define p-feature robust Wasserstein distance (p-FRWD).
Proposition 1 For the distance function d(ar, y),
1/P
(R,
min TEU()))) aEE
max
T
and(a (0) ,3,
11,00p
j=1
a distance for p 1.
Note that can show that 2-FRWD is special case of SRW with d(x, y) || د|اa (See Appendix). The key difference between SRW and FRWD is that FRWD can use any distance, while SRW can only use d(x; y) | y/|2.
3.2 FROT OPTIMIZATION
Here, we propose two FROT algorithms based on the Prank-Wolfh algorithm and linear program- ming.
We propose a continuous variant of the FROT algorithm using the Frank-Wolfe algorithm, which can be fully differentiable. To this end, we introduce entropic regularization for a and rewrite the FROT function of II. Therefore, we solve the following problem for a::
Frank-Wolfe:
min max H)U a
(I, with (II,
,, (H)
1=1
where 17 0 is the regularization parameter, and ((a)) Σ ar (log(ae)) 1) is the entropic regularization for a. An advantage of entropic regularization is that the nonnegative constraint is naturally satisfied, and the entropic regularizer is a strong convex function.
Lemma The optimal solution of the optimization problem
a² argmax a
J,(II, a), with (II, a)
Σ
arde n1)
8=1

---

Under review as a conference paper at ICLR 2021
with fixed admissible transport plan II U (H, v), is given by
exp (40z)
a; Ee=1 exp (100))
with
= log
Σ exp
[0 71
+ 17.
Using
Lemma (or Lemma 4 Nesterov (2005) ) together with the setting Φe
E 25=1 Tize( v Vi (8) (II, Ct), [Cilss - c(x; .Y (
), the global problem is equivalent to
Σ exp ( ( kTT.C.))
min NEU(a.)
G, with (II = 17 log
(5)
Note that this known as a smoothed max-operator (Nesterov) 2005; Blondel al. 2018). cally, regularization parameter controls the "smoothness" of the maximum.
Specifi-
Proposition? 3 Gy (II) is convex function relative to II.
The derived optimization problem of FROT is convex. Therefore, we can determine globally optimal solutions. Note that the SRW optimization problem is not not jointly convex (Paty & Cuturi 2019) for the projection matrix and the transport plan. In this study, we employ the Frank- algorithm (Frank & Wolfe 1956 Jaggi 2013), using which we approximate (TI) with linear functions at (I and move II toward the optimal solution in the convex set (See Algorithm 2).
The derivative of the loss function G, (T(()) given by
8G,,
L
exp (1 (1 ,.)
0
C = Mrro with of
OT
E'= exp ( (TI( Cc)
T=-O
Then, we update the transport plan by solving the EMD problem:
(1++1) (1-7) + +I with ñ argmin NeU(w))
(II, [IIM,ro MI
where > = 2/(2+ + k)). Note that Mi is given by the weighted sum of the cost matrices. Thus, we utilize multiple features to estimate the transport plan II for the relaxed problem in Eq. (
Using the Frofk-Wolfe algorithm, we can obtain the optimal solution. However, solving the EMD problem requires cubic computational cost that be expensive if and large. To address this, we can solve the regularized OT problem, which requires (nm).. We denote the Frank- Wolfe algorithm with EMD FW-EMD and the Frank-Wolfe algorithm with Sinkhorn as FW-Sinkhorn.
Computational complexity: The proposed method depends on the Sinkhorn algorithm, which requires (mm)) operation. The computation of the cost matrix in each subproblem needs an O (Lnm) operation, where L is the number of groups Therefore, the entire complexity is OT where T is the number of Trank-oolfe iterations (in general, T = 10 sufficient).
Proposition 4 For each t 2 1, the iteration (t) of Algorithm satisfies
-G,T)) 40maz ( TT))
17(1 2) (1+
where maa ($ TT) is the largest eigenvalue of the matrix T and + (vec(C) C1), vec(C2), vee(Cr)) and Z is the accuracy to which internal linear subproblems solved.
Based on Proposition 4 the number of iterations depends n,, and the number of groups. If we set small 17. convergence requires more time. In addition, if we use entropic regularization with a large the 5 in Proposition can be large. Finally, if the matrix i can be larger. Note that the constant term the upper bound is large; however, the Frank-Wolfe algorithm converges quickly in practice.
groups, the largest eigenvalue of
5

---

Under review as a conference paper at ICLR 2021
Linear Programming: Because +0++ G2 maxee(1...... Ei= E Tize(e; .0, ,0,)), the FROT problem also be written
mUm (EU()))) (1.2.2....))
Tyclax (2) .
(
Because the objective is the max linear functions, it is convex with respect to II. We can solve the problem via linear programming:
min (pU()
1, S.L. (1,CH) -1,8= T.
This optimization can be easily solved using an off-the-shelf LP package. However, the computa- tional cost of this LP problem is high in general (i.e., O ) m).
3.3 APPLICATION: SEMANTIC CORRESPONDENCE
We applied our proposed FROT algorithm to semantic correspondence. The semantic correspon- dence is problem that determines the matching of objects in two images. That is, given input image pairs (A, B), with common objects, we formulated the semantic correspondence problem to estimate the transport plan from the key points in A those in B; this framework was proposed in (Liu et al. 2020). In Figure 2 we show an overview of our proposed framework.
Cost matrix computation C:: In our framework, we employed pretrained convolutional neural network to extract dense feature maps for each convolutional layer. The dense feature map of the fth layer output of the sth image is given by
((.,)) (r E ER q 1,2, hg = 1,2, W.,, 6, 11,
where w. and h, are the width and height of the sth image, respectively, and de is the dimension of the fth layer's feature map. Note that because the dimension of the dense feature map different for each layer, we sample feature maps to the size of the 1st layer's feature map size (i.e., h2 x w.).
The éth layer's cost matrix for images s and s' is given by
[Celi [| (l.e") 112
1,2,.... W. hss j = ..
potential problem with FROT is that the estimation depends significantly on the magnitude of the cost of each layer (also known as a group). Hence, normalizing each cost matrix important. Therefore, we normalized each feature vector by f p!C..) & f( g(e,*) /Ils!! I|2. Consequently, the cost matrix is given by (Celyj = 2 2f((.)) (((,e') tances such as the L11 distance.
Source image
Feature Rebust Optimal Transport (FROT)
C2
min
lng
Σ
exp
m
y (2)
L.L.
We
dis-
Targel image
Computation of a and with Figure Semantic correspondence framework based on FROT. staircase re- weightitt; For semantic correspondence, setting a E hg,, and b E hg is im- portant because semantic correspondence can be affected by background clutter. Therefore, we gen- erated the class activation maps (Zhou et al. 2016) for the source and target images and used them as a and b, respectively. For CAM, we chose the class with the highest classification probability and normalized it to the range [0, 1].
4 RELATED WORK
OT algorithms: The Wasserstein distance can be determined by solving the OT problem. An ad- vantage of the Wasserstein distance is its robustness noise; moreover, obtain the transport

---

Under review as a conference paper at ICLR 2021
LP
FW- EMD
0.175
FW-Sinkhon
PW-EMD FW Sinkharn
FWY Sinchar
Objectve score
0.150
13-
0:125
1455
MSE
0.100
11
1-
0.075
0.050-
10
0.2
2
6
10
n1
9-3
(a) Objective score.
(b) MSE (1)).
(c) MSE (e).
Figure 3: (a) Objective scores for LP, FW-EMD, and FW-Sinkhorn. (b) MSE between transport plan of LP and FW-EMD and that with LP and FW-Sinkhorn with different v). (c) MSE between transport plan of LP and FW-Sinkhorn with different e.
plan, which is useful for many machine learning applications. To reduce the computation cost for
the Wasserstein distance, the sliced Wasserstein distance is useful (Kolouri et al. 2016). Recently, tree variant of the Wasserstein distance was proposed (Evans & Matsen 2012; Le et al. 2019 Sato ct al. ; the sliced Wasserstein distance is a special case of this algorithm.
The approach most closely related to FROT is robust variant of the Wasserstein distance, called the subspace robust Wasserstein distance (SRW) (Paty & Cuturi 2019). SRW computes the OT problem in discriminative subspace; this is possible by solving to the robustness, SRW can successfully compute the Wasserstein distance from noisy data, The max- max-sliced Wasserstein distance (Deshpande et al. 2019) and its generalized counterpart (Kolouri eet al. 2019) can also be regarded as subspace-robus Wasserstein methods. Note that SRW (Paty & Cuturi (2019) is min-max based approach, while the max-sliced Wasserstein distances (Deshpande itt al. 2019; Kolouri et al. 2019) are max-min approaches. The FROT is a feature selection variant of the Wasserstein distance, whereas the subspace approaches are used for dimensionality reduction.
paaensionnaltyyrrduttii problems. Owing
As parallel work, a general minimax optimal transport problem called the robust Kantorovich prob- lem (RKP) was recently proposed (Dhouib al., 2020). RKP involves using cutting-set method for a general minmax optimal transport problem that includes the FROT problem as special case. The approaches are technically similar; however, our problem and that of Dhouib et al.) (2020) are intrinsically different. Specifically, we aim to solve nigh-dimensional OT problem using feature selection and apply it to semantic correspondence problems, while the RKP approach focuses on providing general framework and uses it for color transformation problems. As technical dif- ference, the cutting-set method may not converge to an optimal solution if we use the regularized OT (Dhouib et al.) 2020). By contrast, because we use Frank-Wolfe algorithm, verges to a true objective function with regularized OT solvers. The multiobjective optimal transport (MOT) is approach (Scetbon et al. (2200) parallel ours. The key difference between FROT and MOT is that MOT tries to use the weighted sum of cost functions, while FROT considers the worst case. Moreover, as applications, we focus on the cost matrices computed from subsets of features, while MOT considers cost matrices with different distance functions.
algorithm con-
EXPERIMENTS
5.1 SYNTHETIC DATA
We compare FROT with standard OT using synthetic datasets. In these experiments, initially generate two-dimensional vectors x ~ N Σx) and y ~ N(u,, Ey). Here, we set fzy = (-5,0) T) My = (5. T)T. Σ, = Σg = ((5, 1) T). Then, we concatenate Ze ~ N(O,, Ig) and zy ~ N (0g, Is) to æ and y. respectively, to give ã (x _
(y ,x,,)
For FROT, set 1.0 and the number of iterations of the Frank-Wolfe algorithm as T = 10. The regularization parameter is set to c = 0.02 for all methods. To show the proof-of-concept, we set the true features group and the remaining noise features another group.
Fig. la shows the correspondence from æ and y with the vanilla OT algorithm. Figs. 16 and Ic show the correspondence of FROT and OT with and y. respectively. Although FROT can identify
7

---

Under review as a conference paper at ICLR 2021
Table 1: Per-class PCK (athoz 0.1) results using SPair-7Ik. All models use ResNet101. The
numbers in the bracket of SRW are the input layer indicies.
Meihib
Bike
bel
60c
6x
606
d0g
Tame
peros
plan
shoag
JBI
SPk
Trettsed
.1NNe
CSNC
so Roson A011
23.4
22.6
18.5
16.7
41.9
42.0
40.2
16.4
14.3
15.1
36.4
37.3
38.1
276
30.8
27.
26.0
39.2
26.5
12.7
35.6
13.3
123
21.4
25.8
23.6
24.3
22.
22.6
14.1
15.0
13.3
21.6
2855
22.5
22.1
11.3
20.5
17.5
13.9
13.5
19.3
353
31.4
35.1
36.3
36.3
22.3
20.9
20.5
Stsa-Fik
valalasion
FROTIO
12.6
35.1
20.3
18.9
62.5
121
50.5
21.1
30.7
42.9
1631
43.0
227
26.1
271
21.2
20.6
191
61.6
63.5
19.7
15.8
48.3
39.7
41.7
37.9
113
25.2
29,8
20.6
3F4
28.5
29.9
31.8
25.0
28.2
24.3
15.6
21.3
46-3
51.1
39.5
17.0
155
1477
35.9
2015
17.3
3500
313
22.1
21.0
25.3
withous
FROV 9.3, 100
56.9
21.9
56.6
13.4
4003
22.
22.0
62.0
96.2
21.3
28.1
10.1
26.9
12.9
18.5
157
S1
PROESO 2.5. m35
56.1
18.8
56.9
19.0
400
25.6
19.2
61.9
11A
35.7
36.5
25.6
26.9
27.2
26-3
22.1
503
18.6
12.8
valatation
FROT mm 2.b 100
56.
57.0
19.9
400
25.
19.2
36.5
15.6
26.4
38.1
32.7
FR 0.7.
100
35.3
19.5
56.6
19.9
39.5
26.0
1911
62.
36.0
96.5
25.0
21.
26.5
25.6
21.6
16.9
12.7
SRW Cayyen 32-34]1
79
16.0
437
15.6
31.
210
17.6
48.0
135
11.3
265
198
15.
17.6
16.7
153
17.
76.5
SRW Cager 51-34
183
46.
15.
36.
213
128
316
711
100
181
16.9
110
26.8
FROO (eyurru 17-5 22-5
45.1
164
167
224
305
311
119
161
13.0
147
15.1
36.1
FROT Dkgers 31-5455
15
16.
45%
305
130
250
216
461
114
718
364
71.4
163
15.0
16.5
17.1
403
71.
76.6
FROT Sayers
151
48.
365
275
250
513
903
34.
33.1
19.5
18.1
76.0
78.8
a suitable matching, the OT fails to obtain a significant correspondence. We observed that the a parameter corresponding to true group is 01 0.9999. Moreover, we compared the objective scores of the FROT with LP, FW-EMD, and FW-Sinkhom = 0.1). Figure shows the objective scores of FROTs with the different solvers, and both FW-EMD and rW--inkhorn can achieve almost the same objective score with a relatively small 1/. Moreover, F shows the mean squared error between the LP method and the FW counterparts. Similar to the objective score cases, it can yield similar transport plan with a relatively small 17. Finally, we evaluated the FW-Sinkhorn by changing the regularization parameter 17. In this experiment, set r) 1 and varied the values. The result shows that we can obtain an accurate transport plan with a relatively small c.
5.2 SEMANTIC CORRESPONDENCE
We evaluated our FROT algorithm for semantic correspondence. In this study, we used the SPair- 7lk (Min et al.] 2019b). The SPair-7Ik dataset consists of 70, 958 image pairs with variations in viewpoint and scale. For evaluation, we employed a percentage of accurate key points (PCK), which counts the number of accurately predicted key points given a fixed threshold (Min et al. (2019)) All
semantic correspondence experiments were run on a Linux server with NVIDIA P100.
For the optimal transport based frameworks, we employed ResNet101 He et al., 2016) pretrained on ImageNet (Deng et al., 2009) for feature and activation map extraction. The ResNet101 consists of 34 convolutional layers and the entire number of features is d = 32, 576. Note that we did not fine-tune the network. We compared the proposed method with several baselines (Min et al. 2019b) and the SRw(T Owing to the computational cost and the required memory size for SRW, we used the first and the last few convolutional layers of ResNet101 as the input of SRW. In our experiments, we 0.1 for FROT and SRW, respectively. For SRW, of latent dimension as A = 50 for all experiments. HPF (Min et al., 2019a) and OT-HPF (Liu et al.) 2020) are state-of-the-an methods for semantic correspondence. HPF and OT-HPF required the validation dataset to select important layers, whereas SRW and FROT did not require the validation dataset. OT is a simple optimal transport-basee method that does not select layers.
empirically set T = 3 and
set the number
Table lists the per-class PCK results obtained using the SPair-7Ik dataset. FROT (17 = 0.3) out- performs most existing baselines, including HPF and OT. Moreover, FROT (17 = 0.3) consistent with OT-HPF (Liu et al., 2020), which requires the validation dataset to select important layers. In this experiment, setting 7 results favorable performance (See Table in the Appendix). The computational costs of FROT is 0.29, while SRWs are 8.73, 11.73, 15.76, respectively. Surprisingly, FROT outperformed SRWs. However, this is mainly due to the used input layers. Therefore, scaling up SRW would be an interesting future work.
We further evaluated FROT by tuning hyperparameters 77 and e using validation sets, where the max- imum search ranges for 7 and are set to 0.2 to 2.0 and 0.1 to 0.6 with intervals of 0. respectively. Figure in Appendix shows the average PCK scores for (17, <) pairs on the validation split of SPair- 71k. By using hyperparameter search, we selected (17 0.2, 0.4) as an optimal parameter. The FROT with optimal parameters outperforms the state-of-the-art method (Liu et al. 2020).
8

---

Under review as a conference paper at ICLR 2021
Lieal ConoRiie
06.2 A.2.Talaes
Beat
Bbock Radtune
Bed
(a)
Colon dataset.
(b) Leukemia dataset.
Prostate-ge dataset.
(d) GLLS5 dataset.
Figure 4: Feature selection results. We average over 50 runs of accuracy (on test set) of SVM trained with top k features selected by several methods.
5.3 FEATURE SELECTION EXPERIMENTS
Since FROT finds the transport plan and discriminative features between X and Y. FROT as feature-selection method. We considered X E IRdxn and Y E Rdxm as sets of samples from classes 1 and 2, respectively. The optimal important feature is given by
exp (+ (TT,C))
Σ exp (esp(H...c))) (H
d
āj =
Et= exp ( with îi argmin NEU(WN)
17 1,
where [Cilis (x/²) yj )0 Finally, we selected the top K features by the ranking â. Hence, a changes to a one-hot vector a small 1) and to og R 1 for a large 1).
Here, we compared FROT with several baseline algorithms in terms of solving feature-selection problems. In this study, we employed high-dimensional and a few sample datasets with two class classification tasks (see Table 2). All feature selection experiments were run on a Linux server with an Intel Xeon CPU E7-8890 with 2.20 GHz and 2 TB RAM.
In our experiments, we initially randomly split the data into two sets (75% for training and 25% for testing) and used the training set for feature selection and building classifier. Note that we standardized each feature using the training set. Then, used the remaining set for the test. The trial was repeated 50 times, and we considered the averaged classification accuracy for all trials. Considered baseline methods, we computed the Wasserstein distance, maximum ancy (MMD) (Gretton et al., 2007), and linear correlation for each dimension and sorted them in descending order. Note that the Wasserstein distance computed via sorting, which computation- ally more efficient than the Sinkhorn algorithm when d 1. Then, we selected the top K features as important features. For FROT, we computed the feature importance and selected the features that had significant importance scores. In our experiments, we set r7 = 1.0 and T = 10. Then, we trained two-class SVM with the selected features.
discrep-
Fig. shows the average classification accuracy relative to the number of selected features. From Figure FROT consistent with the Wasserstein distance-based feature selection and outperforms the linear correlation method and the MMD for two datasets. Table 2 shows the computational time(s) of the methods. FROT is about two orders magnitude faster than the Wasserstein distance and is also faster than MMD. Note that although MMD is as fast as the proposed method, it cannot determine the correspondence between samples.
CONCLUSION
In this paper, proposed FROT for high-dimensional data. This approach jointly solves feature selection and OT problems. An advantage of FROT is that it is a convex optimization problem and can determine an accurate globally optimal solution using the Frank-Wolfe algorithm. We used FROT for high-dimensional feature selection and semantic correspondence problems. Through extensive experiments, we demonstrated that the proposed algorithm consistent with state-of-the- art algorithms in both feature selection and semantic correspondence.
html
http:///ww..a.oeoooooooo..oo///a s.m..v.CCC mm..vC... html

---

Under review as a conference paper at ICLR 2021
REFERENCES
David Alvarez-Melis, Tommi Jaakkola, and Stefanie Jegelka. AIS7A7S, 2018.
Structured optimal transport. In
David Alvarez-Melis, Youssef Mroueh, and Tommi S Jaakkola. Unsupervised hierarchy matching with optimal transport over hyperbolic spaces. AISTATS, 2020.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICMI., 2017.
Mathieu Blondel, Vivien Seguy, and Antoine Rolet. Smooth and sparse optimal transport. In AIS- TATS, 2018
Charlotte Bunne, David Alvarez-Melis, Andreas Krause, and Stefanie Jegelka, Learning generative models across incomparable spaces. ICML, 2019.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NIPS, 2013.
Marco Cuturi and Arnaud Doucet. Fast computation of wasserstein barycenters. ICMI,, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Ishan Deshpande, Yuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander Schwing. Max-sliced Wasserstein distance and its use for GANs. In CVPR, 2019.
Sofien Dhouib, levgen Redko, Tanguy Kerdoncuff, Rémi Emonet, and Mare Sebban. A swiss army knife for minimax optimal transport. In ICML, 2020.
Steven N Evans and Frederick A Matsen. The phylogenetic kantorovich-rubinsteenn metric for en- vironmental sequence samples. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 74(3):569- 29955669599,000 2012.
Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval research logistics quarterly, 3(1-))95--10, 1956.
Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00800 2017.
Arthur. Gretton, Kenji. Fukumizu, C. Hui. Teo, Le. Song, Bernhard. Schölkopf, and Alex Smola. A kernel statistical test of independence. In NIPS, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Paul Hongsuck Seo, Jongmin Lee, Deunsol Jung. Bohyung Han, and Minsu Cho. Attentive semantic alignment with offset-aware correlation kernels. In ECCV, 2018.
Martin Jaggi. Revisiting wolfe: Projection-free sparse convex optimization. In ICML, 2013.
Hicham Janati, Marco Cuturi, and Alexandre Gramfort. Wasserstein regularization for sparse multi- task regression. In AISTA7S, 2019.
Soheil Kolouri, Yang Zou, and Gustavo K Rohde. Sliced wasserstein kernels for probability distri- butions. In CVPR, 2016.
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badcau, and Gustavo Rohde. Generalized sliced wasserstein distances. In NeurlPS, 2019.
Tam Le, Makoto Yamada, Kenji Fukumizu, and Marco Cuturi. Tree-sliced approximation of wasser- stein distances. NeurIPS, 2019.
10

---

Under review as a conference paper at ICLR 2021
Yanbin Liu, Makoto Yamada, Yao-Hung Hubert Tsai, Tam Le, Ruslan Salakhutdinov, and Yi Yang. Lsmi-sinkhorm: -upi-supee squared-loo loss mutual information estimation with optimal trans- port. arXiv preprint arXiv:1909.023733 2019.
Yanbin Liu, Linchao Zhu, Makoto Yamada, and Yi Yang. Semantic correspondence an optimal transport problem. In CVPR, 2020.
Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Hyperpixel flow: Semantic correspondence with multi-layer neural features. ICCV, 2019a.
Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair- 11k: A large-scale benchmark for semantic correspondence. arXiv preprint arXiv: 10543, 2019b.
Yu Nesterov. Smooth minimization of non-smooth functions. Mathematical programming, 103(1): 127-152, 2005.
François-Pierre Paty and Marco Cuturi. Subspace robust wasserstein distances. In JCML, 2019.
François-Pierre Paty and Marco Cuturi. Regularized optimal transport is ground cost adversarial. ICML, 2020.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Foundations and Trends in Machine Learning, 1165-6):355-607, 2019.
Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. Convolutional neural network architecture for geometric matching. In CVPR, 2017.
Ignacio Rocco, Relja Arandjelović, and Josef Sivie. End-to-end weakly-supervised semantic align- ment. CVPR, 2018a.
Ignacio Rocco, Mircea Cimpoi, Relja Arandjelović, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Neighbourhood consensus networks. In NeurlPS, 2018b.
Yossi Rubner, Carlo Tomasi, and Leonidas Guibas. The earth mover's distance as metric for image retrieval. International journal of computer vision, 40(2):99-121, 2000.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. arXiv preprint ArXiv: 2019.
Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Fast unbalanced optimal transport on tree. In NeurlPS, 2020.
Meyer Scetbon, Laurent Meunier, Jamal Atif, and Marco Cuturi. Handling multiple costs optimal transport: Strong duality and efficient computation. arXiv preprint arXiv:2006.07266 2020.
Hongteng Xu, Dixin Luo, and Lawrence Carin. Scalable gromov-wasserstenn learning for graph partitioning and matching. arXiv preprint arXiv:1905.07666 2019a.
Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. Gromov-wassersteinn learning for graph matching and node embedding. In ICML, 2019b.
Yuguang Yan, Wen Li, Hanrui Wu, Huaqing Min, Mingkui Tan, and Qingyao Wu. Semi-supervised optimal transport for heterogeneous domain adaptation. In IJCAI, 2018.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), (:49) 67, 2006.
Mikhail Yurochkin, Sebastian Claici, Edward Chien, Farzaneh Mirzazadeh, and Justin M Solomon. Hierarchical optimal transport for document representation. In NewrIPS, 2019.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
11

---

Under review as a conference paper at ICLR 2021
APPENDIX
RELATED WORK
In addition to accelerating the computation, structured optimal transport incorporates structural in- formation directly into OT problems (Alvarez-Melis et al. 2018). Specifically, they formulate the submodular optimal transport problem and solve the problem using a saddle-point mirror prox algo- rithm. Recently, more complex structured information the hierarchical structure (Alvarez-Mae Melis et al.] 2020 Yurochkin et al. 2019). These approaches successfully incorporate structured information into OT problems with respect to data samples. By contrast, FROT incorporates the structured information into features.
introduced in the OT problem, including
OT applications: OT has received significant attention for use in several computer vision tasks. Applications include Wasserstein distance estimation (Peyré et al. 2019). domain adaptation (Yan ct al., 2018). multitask learning (Janati al. 2019). barycenter estimation (Cuturi & Doucet 2014
semantic correspondence (Liu et al. (2020), feature matching (Sarlin et al. 2019). photo a album
summarization (Liu et al. 2019). generative model (Arjovsky et al.] [017] Bunne al. 2019 graph matching (Xu et al.
2019a b and the semantic correspondence (Liu et al. 2020).
PROOF OF PROPOSITION 1
For the distance function d(æ,, y). we prove that
1/D
n
I.
FRWD,(u,
min NEUG,))
max
Ttj
and( (2)
j=1 t=1
distance for p 2 1.
The symmetry can be read directly from the definition as we used distances that are symmetric. For the identity of indiscernibles, when (H, ists l such that Oe > 0 (as a is in the simplex set). As there is a max in the definition and
0 with the optimal a and II, there
Eij Mejard(z (E) yj y 0, this means that Vl, Eig Te)d(x (E) (E) 0 and Vl, p(() v(() Therefore, have μ when FRWD,, = 0.
yj
When μ v, this means that x; = yi- = b,, V i, and n = m, and we have xx yy) = 0 for i i= j. Thus, for any ae O, the optimal transport plan is m: for d(æ; yj) 0 and Taj 0 for d(æ;, yj) > 0. Therefore, when v, we have FRWD, v) = O.
TRIANGLE INEQUALITY
Let = >i aidny 1 = -i byō >= =hen Cro2, and a E ΣJ, we prove that
FRWD,(A,)) FRWD,(H, FRWD,
To simplify the notations in this proof, we define Dj the distance "matrix" such that [Delas d(x;)) (8) / is the ith-row and jth-columm element of the matrix D,, [Del5k d(y) ( = ) and
[Dr]ec d(xx =²² Moreover, note that D/ is the "matrix," where each element is an element of De raised to the power p.
Consider that P U(,, v) is the optimal transport plan of FRWD, (H,, v), and Q E (v, is the optimal transport plan of FRWD, 7), where = >x Gz2z, is discrete measure. Similar to the proof for the Wasserstein distance in (Peyré et al. 2019 let S = Pdiag(1/b)Q with b be vector such that by by if by > O, and by otherwise. We show that S E 7p,
12

---

Under review as a conference paper at ICLR 2021
L ar (R, Di))
)²
1 Σ
min REU(NN)
Σ E=1
Σ
(r D2))
=
=
CKE
Sae
Delia
1=1
ik
1
1
Σ
at
Del
ik
Σ
Pijlikk by
=
Σ az
>
iik
Pejlike
bi
[=1
ik
8=1
19k
1
Σ
(Deles + [Delaa) )2 Pit
ai
bj
1=1
ijk
By letting Pikke [Dele (yepe of this inequality can be rewritten as
/// and highe ( the right-hand side
(Dels (Delaa) Peo )
Σ at 1=1
>( 8jk
Σ Σ
Cgiske + haskeep
by
6=1 s5k
Σ
Fikkt
Σ
Σ She
1=1 sjk
e=1 Egke
Σ a8 Dilks Pijliik by
Σ
ae >
Delpe
Piglik
jk
by
igk
by the Minkovski incquality.
#
L
L
L
min (EU( E=1
Σ
or (R, RR D?)
Σ
at
Dulipiss
Σ
Oikk bi
+
Σ
ae
Σ
15k
Σ
Pij bj
1=1
iJ
sk
1
L
L Σ
Σ at
Dulypis
+
ae
Σ
Didd
1=1
ij
1=1
sk
1
L
L
max
ae
Di
+
max AEEL 8=1
Σ
at
Σ
EiDeBe
Y
sk
FRWD,(,, + FRWD,(U,7)
This inequality is valid for all a. Therefore, we have FRWD,(M, v) S FRWD,(H,,) + FR,D,(U,,) 7)
FROT WITH LINEAR PROGRAMMING
Linear Programming: The FROT is a convex piecewise- linear minimization because the objective the max of linear functions. Thus, we can solve the FROT problem via linear programming:
min t, s.t. (II, Ct) < t, = 1,2,.... L. HEU())))
13

---

Under review as a conference paper at ICLR 2021
This optimization can be easily solved using off-the-shelf LP package. However, the computa- tional cost of this LP problem is high in general (i.e., O(n"), n m).
The FROT problem can be written as
min
max 2E(1.2.......
(II. CI),
a, III a, II 1, 11
This problem can be transformed to equivalent linear program by first forming epigraph prob- lem:
min 6 11,t
s.t. (E(1.2.....L)
(II, Ct) St
II a. II 1, b, 1I 21 0.
Thus, the linear programming for FROT is given as
min
s.t.
S1,6 1,,_,,,.... I1 a, 11 b, 0.
Next, we transform this linear programming problem into the canonical form. For matrix II (TT IT2 ... nT)) Iaxm and πτ, R we can vectorize the matrix using the following linewise operator:
vec ( n2 ) ER"..
Using this vectorization operator, can write (II, Ct) s t as
vec(C,)
-1
(C,)
-1
vec(IT) t
) S O0,
vec(CL)
where 02 E L is vector whose elements are zero.
For the constraints II1, m and II 1, = b. we can define vectors q....., In. Rum and Th Tm mm such that q, vec II) a; and r vec by in this way:
q (1 T 0
92 T
= (0
T 0
1 1
m m.
and
Ti = (1, m
1, m ,)
r2 = 1, 0
1, m
Tm (0 10-0111:
1) 1)
We can collect these vectors to obtain the vectorized constraints:
9
vee(II)
r2
vec(II) t
t
a,
q
m
14

---

Under review as a conference paper at ICLR 2021
Thus, rewrite the linear programming
min e TT u
12
s.t. (AT 1r.Ju S Or., (Q" Or.uu (R' Om)u b, u 2 0,
where u = (vec T t)T E Rnm++ e (0) 1)T E RRm++ is the unit vector whose nm + 1- th element is 1, A (vec( (I),...... ...., vec((C)) CL)) mm R = (Ti,.... Pm) E Ramxm Q = (In In ...In)
RamxL, (9.,..., qn) E Roaxn and
PROOF OF LEMMA
We optimize the function with respect ac
max J(a)
a
s.t. a lk = 1, a,, 20,
where
J(x) = agto 11
as oz - 1).
(8)
t=1
Because the entropic regularization is strong convex function and its negative counterpart is strong concave function, the maximization problem concave optimization problem.
We consider the following objective function with the Lagrange multiplier c:
I.
J(aa) = Σ agge 1j
Σ ar (log as 1) + 11 1)
Note that owing to the entropic regularization, the nonnegative constraint is automatically satisfied. Taking the derivative with respect to az, have
aJ(a)
= 0. 17 logar 1 as 1
Ooe
0F
Thus, the optimal ay has the form
1 th
ag = exp
exp
1
ag satisfies the sum to one constraint.
1
exp
El= exp (3
Hence, the optimal ag is given by
exp (d00
ag
=1 exp (1,00))
Substituting this into Eq. 8), we have
(2/0)
exp (3
J(a')= = Σ exp
cxp
Or
7)
log
-
(100))
Σ 1= exp (Horr)
Σl exp (ttp))
=1= exp
log
Σ exp
(0)0))
15

---

Under review as a conference paper at ICLR 2021
Therefore, the final objective function is given by
)
J(a') = log Σ cxp
17 0e
+
PROOF OF PROPOSITION
Proof:
For 0 < and 1) > have
exp - (IT, + (1 - 12, D:)
Σ
exp
-
[I,,
D2) + (1-0)
(122 Dz)
t=1
7
1-8
[I Dz)
_
IL,. De)
exp
exp
17
77
1-8
(I,, Di)
Σ exp (12, Di)
))
exp
Here, we use Hölder's inequality with p 1/0, q = 1/(1 0), and 1/p + 1/9 = 1.
Applying a logarithm both sides of the equation and then premultiplying 17. we have
L
( + (1 - f)I1, Di) < On log
L
1 III Dz)
17 log
exp
exp
t=1
P=1
(H Dil)
(1
0)n log
Σ
exp
8=1
PROOF OF PROPOSITION
Theorem Jaggi 2013) For each t 2 1, the iterates (1 of Algorithms 1, 2, 3, and 4 in 2013 satisfy
Jaggi
2C,
f(I f(I')
(1 8),
+
where II D is optimal solution problem
II" argmin fIII..
NED
Cj is the curvature constant defined as
2 (f f(I) (II' - II,
C = sup I., 72
s.t. I [0, II, II 7 II)
5> 2 0 is the accuracy with which internal linear subproblems are solved.
Lemma 6 Jaggi 2013 Let be convex and differentiable function with its gradient V being Lipschitz contimmus w.t.t some norm || over the domain with Lipschitz-consten 0. Then,
C & diam |-l 2
16

---

Under review as a conference paper at ICLR 2021
Definition 7
The softmax function is given by
exp(A21) exp(de2)
1
(2))
Ee exp(Aze)
exp(AzL)
where À > 0 is referred to as the inverse temperature constant.
Lemma Gao & Pavel 2017) T= X, that is for all 2, E IR"
The softmax function a()) is L--iccccht with respect to || ||2 with
[o(2)) - a22)ll2 S A/Iz - 22
where À is the inverse lemperature constant.
The derivative of G [II) is given
1 (II.
dG g
cxp
C₁ Mn.
an
III, Cti
(=1
P== exp
Thus, we have
vec(Mrr) 4pm.
where
Φ (vec(Ci), vec( C2),,... E RuuLL
exp (+ ( (II, (I1
exp ( (11
T
ER
Pr1
Σk=n exp (II
Ek exp exp (1
Here, pm is the softmax function (z) with 24 (II, Ct).
We have
(IT) "GG,(TI' = ppn ppm ||
< oplppp Pr l1a.
[blppp
Φ vee(II) + vec([IT)]l (Lemma with A
17
c |p + |lop vIC(II) vIc(III)
where |lop is the operator norm. We have || | || $ op-
|| + 1lop
lop
|| 1 and
|ve II] ( < 22. Therefore, the Lipschitz constant for the gradient is L mmax Φ ④). and the curvature constant bounded above by C2 2L, where Tmaa
|| ||| op
(Φ Φ the
largest eigenvalue of the matrix T By plugging Cý in Theorem 5 we have
40maz T +b)
(I((I (IT+) <
(1 + 5).
ntt 2)
MAX/MIN FORMULATION
We define the max formulation of the FROT as
L
m
( ( (0)))
max OEEL
a
min
t=1
where L. fa E R a 1 1) is the probability simplex, the set of probability vectors in
R4
17

---

Under review as a conference paper at ICLR 2021
This problem can be solved by computing the group that maximizes the optimal transport distance k* = argmax, & Wi (alt), _l and then by considering a* = Ôµ-- one-hot vector.
The result of this formulation provides an intuitive idea (the same as for the robust Wasserstein method). Hence, we maximize the group (instead of the subspace) that provides the optimal result. However, the formulation requires solving the OT problem times. This approach may not be suitable if we have a large L. Moreover, the argmax function is generally not differentiable.
Relation the max-sliced Wasserstein distance: The max-sliced Wasserstein-2 distance can be defined as (Deshpande al. 2019)
max-W2(u,
min west IEUU())
Tay (w
0z w y;)
where C. Rd is the of all possible directions on the unit sphere.
The max-sliced Wasserstein is max-min approach. That is, for each w. it requires solving the OT problem. The max-min approach is suited for simply measuring the divergence between two distributions. However, it is difficult to interpret features using the max-sliced Wasserstein, where it is the key motivation of FROT.
Relation to Subspace Robust Wasserstein (Paty & Cuturi, 2019): Here, we show that 2-FRWD with d(x
yll special case SRW. us define U = E Idddd where e, IRd is the one-hot vector whose Łth element is 1 and a 1 = 1,a 0. Then, the objective function of SRW can be written
(Vates, v..e....... vadaea
Tiy U
x;
U y9l12
(x; - yy) TUTT (x: 1 yÿ)
j=1
Tig (x: - y;) dddag(()( yz)
T(j ax(: (6) - - 10),2.
1=
t=1
Therefore, SRW and 2-FRWD are equivalent if set (Voiel, /o2e2.... vodea) and d(ær, y) || 2/12.
18

---

Under review as a conference paper at ICLR 2021
Table 2: Computational time comparison (s) for feature selection from biological datasets.
Data
Wasserstein (Sort)
Linear
MMD
FROT
Colon
2000
62
12.37 (# 3.27)
0.00 0.00)
1.36 (= 0.15)
0.41 (# 0.07)
Leukemia
7070
72
46.76 (# 19.47)
0.01 (# 0.00)
5.03 (= 0.79)
1.13 (# 0.14)
Prostate .GE
5966
102
51.99 (# 16.37)
0.02 (#: 0.00)
6.01 (= 1.17)
1.04 (# 0.11)
GL.L_55
22283
85
142.1 (+ 21.65)
0.04 (+ 0.00)
23.6 (= 1.21)
3,44 (+ 0.36)
0.10
0-0.5
0=0.6
0.08
0.06
0
0.9
0.04
0.00
15
20
30
(a) FROT (17 0.3).
(b) Feature importance of FROT.
Figure One-to-one matching results of FROT (1) 0.3) and feature importance of FROT.
ADDITIONAL SEMANTIC CORRESPONDENCE EXPERIMENTS
Figure 5a shows example of key points matched using the FROT algorithm. Fig shows the corresponding feature importance. The lower the 7 value, the smaller the number of layers used. The interesting finding here is that the selected important layer in this case the third layer from the last. More qualitative results are presented in the Figure
19

---

Under review as a conference paper at ICLR 2021
02
33
44
8g
20
34
0.1
32.4
33.7
33.7
33.3
32.7
32.1
31.5
31.0
30.5
30.0
25.8
30
0.3
34.1
33.8
32.8
31.9
31.0
30.3
29.4
28.6
27.9
23.4
28
X
26
0.4
-2
0.6
33.8
32.3
30.6
29.0
27.5
26.4
25.4
24.5
23.8
19.2
-00
Figure 6: Average PCK scores the validation split of SPair-71k with different pairs of hyperpa- rameters.
Table 3: Per-class PCK (abboz 0.1) results using the SPair-7Ik. All models use ResNet101 as the backbone.
Authen
(riztal
718
11.1
37.4
346
116
116
37.6
18.4
965
146
12.6
11.
16.5
35.5
13.3
11.3
11.
30.0
14.9
11.5
18.9
31.6
21.1
30.1
No.No
34.
196
98.
4Pae-716
234
18.7
143
35.4
2022
123
122
212
223
13.7
253
21.0
273
10.3
333
34.
358
WuskAlgs
223
15.1
22.3
31.6
12.3
26.6
18.3
28.0
17.5
22.9
NC
125
13.3
11.7
2800
365
33
196
15.
54.
35.1
01:41Y
207
42.0
62.4
15.7
28.6
33.9
311
16:
31.1
1911
1111
11:1
11117
171
201
FOOT
85.8
36.9
56.3
23.4
40.7
37.2
21.9
62.0
17.5
15.3
363
27.8
36.0
30.4
36.9
23.
29.7
30.8
13.7
34.0
18.7
2011
65.6
13.2
26.6
10.4
Widsoe
1900 14.00
338
15.
329
385
19.3
63.
137
260
36.1
50.1
761
54%
635
26.5
F3OT
325
15.1
565
15.
2611
19.5
61.
17.
37.0
345
45.2
375
F3OT 2.00
30.0
11.5
152
16.
183
535
16.
13.8
343
25,1
15.
21.1
21.5
36.0
25.5
131
347
15.
15.
1R00 01 5.00
365
11M
18.
134
30.
163
454
1901
31
16A
31
14.
174
156
111
363
319
139
162
580 19
05.7
24.
112
20.
132
255
16
33
115
113
162
130
11.1
12.
128
143
5K0 10
0..7
23.3
120
25.
145
350
11
11.
125
13.
133
m n3.7
lyver
111
08W 10 -m m1.
lnsee
261
11.
113
110
15%
115
10.
185
115
114
16
163
31
165
110
114
116
165
155
um -m
253
310
114
11
156
165
380 12 -.....-.... 85.7
Jsso 12-341)
234
140
433
156
135
110
178
450
110
153
293
193
155
11.6
153
151
111
203
343
12 01.7
142
159
184
403
135
340
157
112
215
252
20

---

Under review as a conference paper at ICLR 2021
SOUTHERS COMFORT
SOUTHERS
COMFORT
WILD TURKEY
WILD
TURKEY
1575
(a) FROT
(b) SRW
Figure 7: Qualitative examples sampled from SPair-71k. For each image. the red points represent the keypoints to be matched while the green points denote the prediction and the blue ones denote the ground truth.
21


Reviews:
[{"review_id": "WoDWNNRyTm-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes variants of robust OT/p-wasserstein-dist (3)/(4), where the ground cost is in some sense the maximum over costs with (prefixed) groups of features. The motivation is similar to that for feature selection: where perhaps only few of these groups of features are critical/sufficient for OT purposes. So it can also be understood as joint feature-group selection with OT. The resulting convex problem is proposed to be solved using FW, whose details are presented (including convergence).\n\nPros:\n1. Though similar in spirit to SRW, the proposed formulation has few advantages: a) allows any cost, b) convex c) FW leads to scalable solver etc.\n2. Overall, the paper is very well-written, with nice organization and sufficient details.\n\nCons:\n1. Pre-fixed groups, more importantly, non-overlapping groups seems restrictive, especially because feature selection with overlapping groups is well-studied. (e.g., https://hal.inria.fr/inria-00628498/document , https://papers.nips.cc/paper/4275-efficient-methods-for-overlapping-group-lasso.pdf ) among others.\n\nMajor Comments:\n1. Given SRW and other robust/min-max OT works, and multitude of feature-selection/group-lasso works, the novelty seems restricted. Even in terms of optimization, it seems a straight-forward application of FW. This seems to restrict the technical contribution. \n2. In section 5.2, I am assuming for FROT, all layers were used as input; whereas for SRW, only few are used. Is this the case? If so, perhaps a case of FROT which uses exactly same input as SRW must be included for a fair comparison (along with the FROT with all layers). The authors do seem to agree that the improvement is more because of this skew in inputs. It is will nice to clarify this.\n3. Why is that T is set in an adhoc manner? for example T=10 in synthetic and T=3 in real-world? why not fix or validate ? Also, convergence plots showing obj vs T as well as accuracy vs T might be insightful when included.\n4. It may also be insightful to visually see some critical examples of image/pairs that highlight why FROT may work better than SRW etc. (more like fig5 in appendix)", "extra_scores": {}, "extra_text": {}}, {"review_id": "WddFG3GiRK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*:\nThe authors try to solve a special kind of high-dimensional optimal transport problem. Specifically, they consider the cases when features are grouped and the grouping is known a-priori. The authors formulate the problem into the feature-robust optimal transport (FROT) problem.\nThe authors propose two solving algorithms, one based on the Frank-Wolfe method, and one based on linear programming.\n\n*Pros*:\nThe connection to the feature group sounds interesting to me, as it has a natural connection to the structure of deep learning models.\nThe presentation (other than the introduction) is easy to follow.\n\n*Cons*:\nNote that the first point is the main contributing factor for my rating.\n\n1. Section 3.1 is very confusing, and it seems to me that the authors fail to establish the correct convergence guarantee.\nAs in page 4, the target is $min_{\\pi} max_{\\alpha} J(\\Pi, \\alpha)$ \nIf we fix $\\pi$, we can solve for the optimal $\\alpha$. Plug this optimal $\\alpha$ back in and we obtain $G(\\Pi)$.\nIntuitively one may choose to solve for $\\alpha$ and $\\pi$ alternatingly.\nHowever the convergence of $G(\\Pi)$ says nothing more than, in a fixed iteration, one can solve exactly for the optimal $\\alpha$ and up to $\\epsilon$ accuracy for $\\Pi$.\nWe still don't know if the solution of the algorithm indeed minimizes the said loss.\nI checked the proof of proposition 4. It just invokes the standard FW-convergence analysis from Jaggi 2013, and argue nothing about the alternative part. Note that, even though the two subproblems (for $\\alpha$ and for $\\pi$) can be solved almost exactly, it could be non-trivial to set up the convergence of the entire alternating algorithm.\nAlternatively, maybe the authors want to argue that solving $min_{\\pi} max_{\\alpha} J(\\Pi, \\alpha)$ is equivalent to solving $\\max_{\\pi} G(\\Pi)$. However, this is also not obviously true for me.\n\n2.  What are the other potential applications of FROT? While Semantic Correpondance is an interesting application, I find it hard to convince myself that FROT is better than Liu's 2020-CVPR work (requiring validation dataset is not a big problem - you can always to train-val split). With its similarity to group lasso, FROT might have more interesting applications.\n\n3. Presentation of the introduction can be improved. I find it hard to parse the introduction until I almost finished reading the entire paper. Putting figure 1 to page 2 only creates more questions in my head instead of offering intuitions. Also, it would be helpful if the author can list their contributions in a more organized way.\n\n4. I didn't quite get the high dimensional part. While 'high-dimensional' appears in the abstract, introduction, and conclusion section, I didn't find the correspondence in the main text.\n\n5. I didn't get the robust part, other than the empirical performance in the evaluation section.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "85VCwtROxIG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed framework FROT - feature-robust optimal transport - seeks to select feature groups to both speed up OT computation for high-dimensional data and make it more robust to noise. The exposition is generally clear. My main concerns are limited novelty and lack of extensive experiments.\n\nThe paper draws the contrast between prior work SRW that yields a discriminative subspace via dimensionality reduction, and offers a dual perspective to use feature selection instead. A thorough discussion on the pros and cons of feature selection vs feature dimensionality reduction would add insight. \n\nTraditional entropy-regularized OT regularizes using the entropy of the transport plan $\\Pi$, whereas FROT regularizes using the probability distribution $\\alpha$. One expects a discussion on the effect of this choice.\n \nCurrently, the optimization for the group selection is done independently of optimization that produces the features, for instance by the choice of a pretrained network in the semantic correspondence application. It's worthwhile to explore joint optimization of feature generation and selection for downstream tasks.\n\nFor the claim of robustness to noise, experiments on data of dimension higher than 10 would be desirable.\n\nThere should be more extensive experiments applying FROT to more tasks and compared with additional baselines. Figure 3 compares the objective scores of FW-EMD, FW-Sinkhorn with that of exact OT, there are many other OT algorithms, such as tree-based methods, as referenced in the paper, that can be compared with. These plots should also include variations of the metric across trials. Similarly for the semantic correspondence results in Table 1. \n\nThank you authors for your response.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Motivated by (1) the problem of scaling up optimal transport to high-dimensional problems and (2) being able to tolerate noisy features, this paper introduces a new optimization problem that they call feature-robust optimal transport where they find a transport plan with discriminative features. They show that the min-max optimization problem admits a convex formulation and solve it using a Frank-Wolfe method. Finally they apply it to the layer selection problem and show that it achieves state-of-the-art performance for semantic correspondence datasets. 

The reviews were mixed for this paper. The main negative, which was brought up in all the reviews, is the lack of novelty compared to earlier methods like SRW which already combine dimensionality reduction and optimal transport. The new method in this paper still does have value since it can scale up to larger dimensional problems. It would have been nice to have a wider range of experiments, which would present a more compelling case for its applicability. Another reviewer brought up a correctness issue, however it is not clear if this is actually a bug or merely a misunderstanding about how the pieces in the overall proof fit together. In any case, the reviewers pointed out various places where the writing could be improved. 
Decision: Reject