Spherical Sliced-Wasserstein
Authors: Clément Bonet
Paul Berg
Nicolas Courty
François Septier
Lucas Drumetz
Minh-Tan Pham
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Optimal Transport
Sliced-Wasserstein
Sphere

Abstract: Many variants of the Wasserstein distance have been introduced to reduce its original computational burden. In particular the Sliced-Wasserstein distance (SW), which leverages one-dimensional projections for which a closed-form solution of the Wasserstein distance is available, has received a lot of interest. Yet, it is restricted to data living in Euclidean spaces, while the Wasserstein distance has been studied and used recently on manifolds. We focus more specifically on the sphere, for which we define a novel SW discrepancy, which we call spherical Sliced-Wasserstein, making a first step towards defining SW discrepancies on manifolds. Our construction is notably based on closed-form solutions of the Wasserstein distance on the circle, together with a new spherical Radon transform. Along with efficient algorithms and the corresponding implementations, we illustrate its properties in several machine learning use cases where spherical representations of data are at stake: density estimation on the sphere, variational inference or hyperspherical auto-encoders.

Optimal transport (OT) [101] has received lot of attention in machine learning in the past few years. As allows to compare distributions with metrics, it has been used for different tasks such as domain adaptation [24] or generative models [8], to name few. The most classical distance used in OT is Spherical Wasserseenn
Anonymous Author(s)
Affiliation Address
email
Abstract
Many variants of the Wasserstein distance have been introduced to reduce its original computational burden. In particular the Sliced Wasserstein distance (SW), which leverages one-dimensional projections for which closed-form solution of the Wasserstein distance is available, has received lot of interest, Yet. is restricted data living in Euclidean spaces, while the Wasserstein distance has been studied and used recently manifolds. We focus more specifically the sphere, for which we define novel SW discrepancy, which we call spherical Sliced- Wasserstein, making a first step towards defining SW discrepancies on manifolds. Our construction notably based on closed-form solutions of the Wasserstein distance on the circle, together with a new spherical Radon transform. Along with efficient algorithms and the corresponding implementations, we illustrate its properties in several machine learning use cases where spherical representations of data at stake: density estimation the sphere, variational inference hyperspherical auto-encoders.
2
3
10
11
Introduction
15
16
17
18
19 the Wasserstein distance. However, calculating can be computationally expensive. Hence, several variants were proposed alleviate the computational burden, such as the entropic regularization
20
[26, 97], minibatch OT [35] the Wasserstenn distance (SW) for distributions supported Euclidean spaces [90].
23 Although embedded in larger dimensional Euclidean spaces, data generally lie practice on manifolds
24 [36]. A simple manifold, but with lots of practical applications, is the hypersphere Sd-1 Several
25 types of data are by essence spherical: a good example is found in directional data [71, 87] for
26 which dedicated machine learning solutions being developed [98], but other applications concern
for instance geophysical data [32], meteorology [11], cosmology [86] extreme value theory
28 for the estimation of spectral measures [44]. Remarkably, in a more abstract setting, considering
29 hyperspherical latent representations of data is becoming more and more common (e.g. [28, 70, 1 110]).
30 For example, in the context of variational autoencoders [58], using priors on the sphere has been
31 demonstrated to be beneficial [28]. Also, in the context of self-supervised learning (SSL), where
32 one wants learn discriminative representations in unsupervised way, the hypersphere usually considered for the latent representation [20, 21, 43, 104, 108). It is thus of primary importance to develop machine learning tools that accommodate well with this specific geometry.
Submitted to 36th Conference Neural Information Processing Systems (NeurIPS 2022). Do distribute.

---

35
The OT theory manifolds is well developed [38, 73, 101] and several works started to use OT maps. For example, Cohen et al. 37 [23], Rezende and Racanière [91] approximate the OT map to define normalizing flows on Riemannian of much interest.
36 in practice, with a focus mainly on the approximation
38 manifolds, Cui et al. [25], Hamfeldr and Turnquist [45, 46] derive algorithms to approximate the OT [5], Hoyos-Idrobo [51] learn the transport map on hyperbolic
39 map on the sphere. Alvarez-Melis spaces. However, the computational bottleneck to compute the Wasserstein distance such spaces remains, and, as underlined in the conclusion of [74], defining SW distances manifolds would be
43 Contributions. Therefore, by leveraging properties of the Wasserstein distance on the circle [89], 44 we define the first, to the best of our knowledge, SW discrepancy on a non trivial manifold, namely the sphere 5d-1, and hence we make first step towards defining SW distances on Riemannian manifolds.
45
46
We make connections with a new spherical Radon transform and analyze some of its properties. We discuss the underlying algorithmic procedure, and notably provide an efficient implementation when computing the discrepancy against uniform distribution. Then, we show that discrepancy on different tasks such as density estimation, variational inference generative modeling.
use this
49
Background
The aim of this paper is to define a Sliced-Wasserstein discrepancy on the hypersphere Sd = for E 51
52 Rd, ||ı||2 = 1}. Therefore, in this section, we introduce the Wasserstein distance on manifolds and 53 the classical SW distance on Rdd.
54
2.1
Wasserstein distance
Since interested in defining SW discrepancy the sphere, start by introducing the Wasserstein distance on a Riemannian manifold M endowed with the Riemannian distance d. We refer to [38, 101] for more details.
57
58 Let 2 1 and μu, / e Pp(M) = {u E P(M). SM (z mo) du(z) < oo for some xo e M). Then,
59 the p- Wasserstein distance between fs and v defined as
WF(mvv) = inf gEll(aa)) [ SAXX dz(z,v) dr(z.v),
(1)
60 where I), = fy E P(M x M), AA C M, (M x A) = v(A) and 7A x M) p(A))
61 denotes the set of couplings.
62 For discrete probability measures, the Wasserstein distance can be computed using linear programs
63 [88]. However, these algorithms have O(n log n)) complexity w.r.t. the number of samples which is computationally intensive. Therefore, whole literature consists of defining alternative discrepancies which are cheaper to compute. On Euclidean spaces, one of them is the Sliced- Wasserstein distance.
66
87
2.2
distance
On A = R with d(x, 3) = || - ullp. a more attractive distance is the Sliced- Wasserstein (SW) 69 distance. This distance relies on the appealing fact that for one dimensional measures μ. E P(R).
66
70 we have the following closed-form [88, Remark 2.30]
WP v) / (w) (") du,
F7
71 where F-1 (resp. F the quantile function of μ (resp. v). From this property, Bonnotte [16], Rabin et [90] defined the SW distance
72
MU,UE (W /0-1 WP(P #14 P2U) dA(0).
(3)

---

where (0 = (x, 0), A is the uniform distribution on S and for any Borel set A E B(RR),
73
74
75
This distance can be approximated efficiently by using a Monte-Carlo approximation [75], and amounts complexity of OL log n) where L denotes the number of projections used for the Monte-Carke approximation and the number of samples.
76
SW can also be written through the Radon transform [15]. Let f E (RR)), then the Radon transform
78
R: L' (R*) L' x 5d-1) is defined as [48]
79
AA S VR Rf(t,0) [ ((n)
(4)
80 Its dual R : Co(R 1h Co(IR)) (also known as back-projection operator), where Co denotes the set of continuous functions that vanish at infinity, satisfies for all f.9. (Rf, ( Rx--11 (S. It g) and can be defined as [13, 15]
Vg E Co x 1, Vx
Vo E Rd, R R'g(z)) ge- 0(x,, 0), d8.
(5)
89 Therefore, by duality, we can define the Radon transform of a measure E MM(R)) as the measure
84 Rp E M(R x 1d-1) such that for all E (R x 5t-1), (R1,, g) = (j,, R g) Since Ry
85 a measure on the product space R x 5d-1, we can disintegrate it w.r.. X, the uniform measure K with probability kernel on Sd-1 B(R). i.e. for all 0 e s ((, :) is probability R, for any Borel set A E B(R), K(,, A) is measurable, and
86 on Sd-1 [6]. as Ru
JEXSHI olt (((R)))()))( = SSg [ ((t,))((o,dtd)N)))
Vő C(R x 1)
(6)
every By Proposition 6 in [15], have that for X-almost every 0 E 1tt (Rp))² PIg where denote K(0,-) (Rp)), Therefore, we have
89
Yu,uEPPPH SWD(KN)= = ga-1
( a)(0)-
90 Variants of SW have been defined in recent works, either by integrating w.r.t. different distributions
91 [31, 80, 81]. by projecting on R using different projections [78, 79] or Radon transforms [22, 60], or by projecting subspaces of higher dimensions [52, 66, 67, 85].
3
A Sliced- Wassersein discrepancy the sphere
93
Our goal here is to define sliced- Wasserstein distance the sphere 5 To that aim, proceed analogously to the classical Euclidean space. We first rely on the nice properties of the Wasserstein distance on the circle [89] and then propose to project distributions lying on the sphere to great circles. Hence, circles play the role of the real line for the hypersphere. In this section, we first describe the problem on the circle, then we define a sliced- Wassersten discrepancy on the sphere and discuss some of its properties. Notably, we derive new spherical Radon transform which is linked to our newly defined spherical SW. We refer to Appendix A for the proofs.
96
97
98
99
100
3.1
Optimal transport the circle
102
On the circle 5 = R/Z equipped with the geodesic distance dsi, appealing formulation of the Wasserstein distance is available [30]. First, let us parametrize 5 by [0, 1[. then the geodesic distance
103
104
can be written as [89], for all x,, E [0, 16, dsi (I, y) = min - y|. - |x ví). Then, for the cost function c(x,8) 8) = h(ds (x, 3)) with R + R+ an increasing convex function, the Wasserstein
105
distance between μ E P(S)) and v E P(S!) can be written as
106
/ ((F (t) - (Fu - -a)- dt,
We(n, v) inf
(8)
aeR
3

---

107
where F [0, [0, 1] denotes the cumulative distribution function (cdf) of μ4, F2r its quantile function and a is a shift parameter. The optimization problem over the shifted cdf F, - a can be seen as looking for the best "cut" (or origin) of the circle into the real line because of the 1-periodicity. Indeed, the proof of this result for discrete distributions in [89] consists in cutting the circle at the optimal point and wrapping it around the real line, for which the optimal transport map the increasing rearrangement F2 Fp which can be obtained for discrete distributions by sorting the points [88].
108
109
110
111
114
Rabin et al. [89] showed that the minimization problem is convex and coercive the shift parameter and Delon et al. [30] derived binary search algorithm to find it. For the particular case of h = Id, can further be shown [19, 106] that
115
116
W Wi(u, = inf L. (F,(()- al dt.
- F+(()
aER
117 In this case, we know exactly the minimum which is attained at the level median [53]. For
118 f: [[, R,
(((o)-aaar] S. ('1r0) - alar) -inf=iinf inf A 8(1x E (0, f(x) 5 2 2
1
LevMed(J) = min argmin
aeR
(10)
where 3 is the Lebesgue measure. Therefore, we also have 119
[[1F.(t (F.(() (t) LevMed(F,, Fu)l dt.
W1 (H. v)
(11)
120
Since we know the minimum, we do not need the binary search and we can approximate the integral very efficiently as we only need to sort the samples to compute the level median and the edfs.
121
Another interesting setting in practice is compute W2, i.e. with h(z) x², w.r.f. a uniform distribution on the circle. We derive here the optimal shift for the Wasserstein distance between l an arbitrary distribution on S¹ and v. We also provide closed-form when μ is discrete distribution.
122
123
Proposition 1. Let μ E P2(S and y Unif(8"). Then,
àl2 dt with à à= dp(x)
1
W2(,,v) J
1F- -t àl
(12)
126
In particular, if <... In and p
n Ei=1 8zp then
1 t=1
1 n
1 (13)
W2 v)
nn + 1 21)a; +
12"
127
This proposition offers an intuitive interpretation: the optimal cut point between an empirical and uniform distributions is the antipodal point of the circular mean of the discrete samples. Moreover, very efficient algorithm be derived from this property, as it solely requires sorting operation to compute the order statistics of the samples.
128
131
3.2
Definition of SW on the sphere
132
On the hypersphere, the counterpart of straight lines are the great cireles, which correspond to the geodesics. Moreover, we can compute the Wasserstein distance on the circle fairly efficiently, Hence, define a sliced- Wasserstein discrepancy on this manifold, we propose, analogously to the classical SW distance, project measures on great circles. The most natural way project points from Sd-1 to great circle C' is to use the geodesic projection [40, 55] defined as
133
134
135
Vo E 5d-1,
pC(ar)) argmin dga-1 (x, v):
(14)
иЕС
where dga-i (z. y) arccos((2, 8)) is the geodesic distance. See Figure 1 for illustration of the geodesic projection on great circle. Note that the projection is unique for almost every x (see

---

[9, Proposition 4.2] and Appendix B.1) and hence the pushforward P f of absolutely continuous measures w.r.t the Lebesgue measure μ (S is well defined.
139
140
Great circles can be obtained by intersecting S with a 2-dimensional plane [56]. Therefore, to average over all great circles, we propose to integrate over the Grassmann manifold Ga.2 = E C RR, dim(E) (E) = 2} [2, 10] and then to project the distribution onto the intersection with the hypersphere. Since the Grassmannian not very practical, we consider the identification using the set of rank projectors:
141
142
143
144
145
Ga.2 = {P Rdxd p = P Tr(P) = 2] = (UU?, U E Va2h.
(15)
where V, = (U E Rdx2, U = I2] } is the Stiefel manifold [10].
146
147
Finally, we can define the Spherical Sliced-Wassersee Wasserstein distance (SSW) for p 2 1 between locally absolutely continuous measures w.r.t the Lebesgue measure [9] μ.v v Pr.ac (S as
148
SSWT(,.V) = J. WP ub P v) da(U),
(16)
where a is the uniform distribution over the Stiefel manifold V pM is the geodesic projection the great circle generated by U and then projected on S i.e.
YU e Va,2, Vz E 5d-1, p (x) U
dga (I, y) argmin dga (2z, U2) VEspan(UU?>nS=
argmin
aEst
(17)
151
and the Wasserstein distance is defined with the geodesic distance dga.
152 Moreover, we can derive a closed form expression which
153 will be very useful in practice:
Lemma 1. Let U E V. then for a.e. x E Sd-1,
154
UFx
pu (z)
(18)
1
155
Hence, we notice from this expression of the projection
156
that we recover almost the same formula as Lin et al. [66] but with an additional e normalization which projects the data the circle. As in [66], we could project a higher dimensional subsphere by integrating over Va,; with & 2 2. However, we would lose the computational
159
160
161
efficiency provided by the properties of the Wasserstein distance on the circle.
162
Figure 1: Illustration of the geodesic projections on a great circle (in black). In red, random points sampled the sphere. In green the projections and in blue the trajectories.
163
3.3
A Spherical Radon Transform
164
As for the classical SW distance, we can derive second
185
formulation using Radon transform. Let f E (Sdd-1)),
we define a spherical Radon transform
R: (Sd-1) (S x Va.2) as
186
e S' V Va2, Rf(2,))= (o)) dz.
(19)
187
This is basically the same formulation as the classical Radon transform [48, 77] where we replaced the real line coordinate by the coordinate the circle and the projection is the geodesic one which is well suited to the sphere. This transform is actually new since we integrate over different sets compared to existing works on spherical Radon transforms.
169
170
171
Then, analogously to the classical Radon transform, we can define the back phok-prooection operator R Co Va.2) + (sddd-1), (ddd- being the space of continuous bounded functions, for
172
g E (S' x Va.) for a.e. ! / s(P"((x) (x),U) (U).
R ((g(x)=
(20)
Jvaa
5

---

Proposition 2. R is the dual operator of R, i.e. for all f e (' Cols' x Va2).
174
(R. (T.gg))) six (J, R² ga
(21)
175
Now that we have dual operator, we can also define the Radon transform of an absolutely continuous measure je E Mao(Su-1) by duality [13, 15] as the measure Rl satisfying
176
Vg Cols x Va.2), / x
g(z, U) d( = 5a- -Sa.. R g()
(22)
Since Rp is measure on the product space S Va,2, Rp can be disintegrated (6, Theorem 5.3.1]
177
178 w.r.f. a as Ry & K where K is probability kernel on Va x S with S the Borel a-field of
179 S We will denote for a-almost every U E Va.2, (Ru)) = ((U,)) :) the conditional probability.
Proposition 3. Lel| ja E ( then a-almost every U E Va, (Rp)) = P
180
Finally, we can write SSW (16) using this Radon transform:
Vu, E Pp.ac ( SSWT = J.L (T((R))"
( (Rv))) du)
(23)
Va.2
182
Note that a natural way to define SW distances can be through already known Radon transforms using the formulation (23). It is for example what was done in [60] using generalized Radon transforms [34, 50] to define generalized SW distances, or in [22] with the spatial Radon transform. However, for known spherical Radon transforms [1. 7] such as the Minkowski- Funk transform [27] or more generally the geodesic Radon transform [95], there is no natural way that we know of to integrate over some product space and allowing to define a SW distance using disintegration.
183
184
185
186
188
As observed by Kolouri et al. [60] for the generalized SW distances (GSW), studying the injectivity of the related Radon transforms allows to study the set on which SW is actually a distance. While the classical Radon transform integrates over hyperplanes of Rd the generalized Radon transform
189
190
191
over hypersurfaces [60] and the Minkowski-Funk transform over "big circles", i.e. the intersection between hyperplane and 5d-1 [96], the set of integration here is half of a big circle. Hence, R is related to the hemispherical transform [94] on Sd-2, We refer to Appendix A.6 for more details on the links with the hemispherical transform. Using these connections, we can derive the kernel of R as the set of even measures which are null over all hyperplanes intersected with 5dl11
192
194
195
Proposition 4. ker = {u E Meven (Sd-1), AH E Gad-1. p(H sd-1)= = 0} where μ E
196
Meven for all f C(Sd-1), f) = (v. with f= = (f(x) f(-x))/2 all x.
197
198
We leave for future works checking whether this set is null or not. Hence, we conclude here that SSW is a pseudo-distance, but a distance on the sets of injectivity of i [4].
199
Proposition 5. Let p 2 1, SSW, is pseudo-distance P (S
200
4 Implementation
201
202
In practice, we approximate the distributions with empirical approximations and, as for the classical SW distance, we rely on the Monte-Carle approximation of the integral on Va.2. We first need to sample from the uniform distribution a E P(V..2). This can be done by first constructing E RRx2 by drawing each of its component from the standard normal distribution A 1) and then applying the QR decomposition [67]. Once we have (Ut)&z a, we project the samples on the circle S by applying Lemma 1 and we compute the coordinates on the circle using the atan2 function. Finally, we can compute the Wasserstein distance on the circle by either applying the binary search algorithm of [30] or the level median formulation (1J) for SSW, In the particular case in which we want to compute 5SW, between a measure ja and the uniform measure on the sphere V = ( we
203
204
205
208
209
210
211
can use the appealing fact that the projection of y on the circle is uniform, i.e. PV = US"
212
(particular case of Theorem 3.1 in [55], see Appendix B.3). Hence, we can use the Proposition
213 I to compute W2, which allows a very efficient implementation either by the closed-form (13) or
approximation by rectangle method of (12). This will be of particular interest for applications in
Section 5 such autoencoders. We sum up the procedure in Algorithm 1.

---

Algorithm SSW
Input: (x:)__1 ~ Me (3,) ~ L the number of projections, the order
for = 1 to I. do
Draw random matrix 7 E Rdx2 with for all i, Zi,j 10)
U = (2) ~0
Project on S the points: j. of
VT= a.la" 03 = wyll
V² y;
Compute the coordinates on the circle S vi.j. J, o ( atan2 ( 21,2 -2.1)))) of
(Tt atan2( -,,,2:-,,,,1 (2,1))((2T)
Compute W Di=1 654 m Dド= dgp) by binary search (11) for p p=
end for
Return SSWF(u, ~ E W Ei 8gg. 23 675)
216
Complexity. Let us note n (resp. m) the number of samples of μ (resp. 1/), and L the number of projections. First, we need compute the QR factorization of L matrices of size 2. This can be done in O(Ld) by using e.g. Householder reflections [42, Chapter 5.2] or the Scharwz-Rutishauser algorithm [41]. Projecting the points S by Lemma 1 is in O((n + m)dL) since need to compute L(n + m) products between U? E R2xd and E Rd. For the binary search or particular case formula (11) and (13), we need first to sort the points. But the binary search also adds cost of (((n + m)) log ())) to approximate the solution with precision [30] and the computation of the level median requires to sort (n + m) points. Hence, for the general SSW,, the complexity OLL(n + m) + log( )) + Ln log n + Lm log m) versus O(L(n + m) + log(n + m))) for SSW1 with the level median and O O( (d + log n)) for SSW2 against a uniform with the particular advantage that we do not need uniform samples in this case.
217
218
221
222
223
224
225
227
Runtime Comparison. We perform here some runtime comparisons. Using Pytorch [83], we imple- mented the binary search algorithm of [30] and used it with c = 10-6 We also implemented SSW, using the level median formula (11) and SSW2 against a uniform measure (12). All experiments are conducted on GPU.
228
229
230
On Figure 2, we compare the runtime between two distributions on S between SSW, the 232
SSW; L= 200
Wassersteln Sinkhorn
SSW,, BS, L 200
233
Wasserstein distance and the entropic approx-
SSW, Unif. L = 200
234
imation using the Sinkhorn algorithm [26] with the geodesic distance as cost function. The distributions were approximated using {102, 10 104, 10", 109} samples of each distribution and we report the mean over 20 computations. We use the Python Optimal Transport (POT) library [39] to compute the Wasserstein distance and the entropic approx-
10
235
236
Seconds
10a
10
10
240
241
242
imation. For large enough batches, we observe that SSW much faster than its Wasserstein counterpart, and also scales better term of memory because of the need store the n cost matrix. For small batches, the computation of SSW actually takes longer because of the computation of the QR factorizations and of the projections. For bigger batches, it is bounded
Number of samples in each distribution
243
Figure 2: Runtime comparison in log-log scale between W. Sinkhorn with the geodesic distance, SSW, with the binary search (BS) and uniform distribution (12) and SSW, with formula (11) be- tween two distributions on S The time includes the calculation of the distance matrices.
244
247
248
249
250
by the sorting operation and we recover the quasi-liness slope. Furthermore, as expected, the fastest algorithms are SSW, with the level median and SSW2 against a uniform as they have a quasilinear complexity. We report in Appendix C.2 other runtimes experiments w.z.t to egg. the number of projections or the dimension.
251
7

---

Sliced- Wasserstein variational inference on the sphere. Another setting of interest is when we have access to some target distribution only up constant. For example in Bayesian inference, we want to sample from a posterior distribution p(- for which the normalizing constant is costly (a) Target
(b) KDE estimate
(a) Target distribution
(b) Density learned
Figure 3: Minimization of SSW with respect to a mixture of vMF.
Figure 4: Amortized SSWVI with a normaliz- ing flow w.t. a mixture of vMF.
Applications
254
255
In this section, we first illustrate the ability to approximate different distributions by minimizing SSW w.r.f. some target distributions on S We first use distributions from which we can draw samples. Then,
256
target distributions from which know the density only up constant. Finally, apply SSW for generative modeling tasks using the framework of Sliced- Weseeersesn autoencoder and we show that we obtain competitive results with other Wasserstein autoencoder based methods using a prior on the hypersphere. We also add in Appendix C.6 some experiments where we use SSW in order to enforce uniformity in a contrastive self-supervised learning context.
280
261
262
SSW as a loss 5.1
We verify on the two first experiments that we can learn some target distribution E P(S by minimizing SSW, i.e. we consider the minimization problem argmin, SSW (p,
263
284
Gradient flow. First, we suppose that we have access to the target distribution through samples, Lec through 2m = - -= - Gy, where (u;)}= are i.i.d samples of v. We choose as target distribution mixture of 6 well separated von Mises- Fishsr distributions [72]. This is a fairly challenging distribu- tion since there are 6 modes which are not connected. We show on Figure 3 the Mollweide projection of the density approximated by a kernel density estimator for a distribution with 500 particles. To
267
268
269
270
optimize directly over particles, we can either perform a Riemannian gradient descent on the sphere
271
[3] or a projected gradient descent. We report in Appendix C.3 additional details and experiments.
272
274
275 to compute, i.e. we can only evaluate some function such that p(-[2) α т. Popular methods to solve these types of problems are MCMCs [93] or variational inference [12, 54].
276
277
Variational inference aims at approximating the target by a distribution q in some family of distribu- tions Q. The classical way of doing it is to minimize the Bellback-Leiblee (KL) divergence. However, the KL divergence suffers from some drawbacks such as under estimating the target distribution and not being distance. Recently, Yi and Liu [111] proposed to use the SW distance instead. The method is called Wasserstenn Variation Inference (SWVI) and relies on running at each iteration few
280
281
282
MCMC steps and then performing gradient descent to learn the variational distribution. We refer to Appendix C.4 and Algorithm for further details on the method.
283
In the following, we replace SW by SSW in SWVI, which denote SSWVI, and we perform amortized variational inference on the sphere by using exponential map normalizing flows (see [92] and Appendix B.4) to learn the distribution and the Geodesic Langevin algorithm [105] as MCMC method. We use the same target as Rezende et al. [92] and we report on Figure the Mollweide projection of the learned density. Since we learn to sample from noise distribution, here the uniform distribution on the sphere, we do not have directly access to the density and we report kernel density estimate with Gaussian kernel. On Figure 5, we plot the evolution of the effective samples size
286
287
288
289
290
(ESS) [33, 69] through the iterations. This indicates how well the flow matches the target. We observe that using SSW gives slightly better results, or at least comparable, than SWVI with SW.
8

---

00 53
5.WVI
(a) SWAE
(b) SSWAE
Figure 5: Comparison of the ESS between SWVI et SSWVI with the mixture target (mean and 95% confidence interval over 10 runs).
Figure 6: Latent space of SWAE and SSWAE for a uniform prior S
293
5.2
SSW autoencoders
Table I: FID (Lower is better).
294
In this section, we use SSW to learn the latent space of autoencoders (AE). We rely on the SWAE framework in-
Method Dataset
MNIST
Fashion
295
SSWAE
1491
0.32
43,94
0.81
298
troduced by Kolouri al. [59]. Let f be some encoder and
SWAE
15.18
0.32
44.78
1.07
g be some decoder, denote pz prior distribution, then the
WAE MMD IMQ
18.12 0.62
68.51
2.76
WAE-MMD RAF
20.09
1.42
70.58
1.75
loss minimized in SWAE is
SAE
19.39
0.56
56.75 + 1.7
Circular GSWAE
0.26
g(f.g) - o(
ASW?(f#H4
(24)
299
where p the distribution of the data for which have access samples. One advantage of this framework
classical VAEs [58] is that parametrization trick is needed here and therefore the choice of the prior is more free.
302
In several concomitant works, was shown that using a prior on the hypersphere can improve the results [28, 110]. Hence, we propose in the same fashion as [59, 60, 84] to replace SW by SSW, which we denote SSWAE, and enforce a prior on the sphere. In the following, we use the MNIST [64] and MaSHT [109] datasets, and we put f2 normalization the output of the encoder. the uniform distribution 510 and compare in Table the Fréchet Inception Distance (FID) [49], for 10000 samples and averaged over 5 trainings, obtained with the Wasserstein Autoencoder (WAE) [99], the classical SWAE [59], the Sinkhorn Autoencoder (SAE) [84] and circular GSWAE [60]. We observe that we obtain fairly competitive results. We add on Figure 6 the
303
304
305
As prior,
308
309
latent space obtained with a uniform prior on S We observe a better separation between classes for
310
311
SSWAE. We refer to appendix C.5 for more details and additional experiments.
Conclusion and discussion
312
313
In this work, we derive a new sliced-Wasserseee Wasserstein discrepancy on the hypersphere, that comes with practical advantages when computing optimal transport distances on hyperspherical data. We notably showed that is competitive or even sometimes better than other metrics defined directly on Rd on variety of machine learning tasks, including density estimation, variational inference or generative models. Our work is, up our knowledge, the first to adapt the sliced Wasserstein framework to non-trioe non-trivial manifolds. The three main ingredients are: closed-form for Wasserstein the circle, ii) closed-form solution to the projection onto great circles, and iii) novel Radon transform on the Sphere. An immediate extension of this work would be to consider sliced- Wasserstein discrepancy in hyperbolic spaces, where geodesics are circular ares as the Poincaré disk. Beyond the generalization to other, possibly well behaved, manifolds, statistical aspects need to examined, such as sample complexity or dependence to the hypersphere dimension. While we postulate that
314
315
316
317
320
321
322
323
324
results comparable the Euclidean case might be reached, the fact that the manifold is closed might
bring interesting differences and justify further use of this type of discrepancies rather than their Euclidean counterparts.

---

327 References
[1] Ahmed Abouelaz and Radouan Daher. Sur la transformation de radon de la sphère S Bulletin de la Société Mathématique de France, 121(3):353-382, 1993. (Cited p. 6)
330
[2] P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Riemannian geometry of grassmann manifolds with view on algorithmic computation. Acta Applicandae Mathematica, 80(2): 199-220, 2004. (Cited p. 5)
331
332
338
[3] P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. In Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2009.
334
335
(Cited on p. 8, 24, 27, 29, 30)
[4] Mark I. Agranovskyt and Eric Todd Quintott. Injectivity of the spherical mean operator and related problems. Complex analysis, harmonic analysis and applications, 347:12, 1996. (Cited
337
338
on p. 6)
[5] David Alvarez-Melis, Youssef Mrouch, and Tommi Jaakkola. Unsupervised hierarchy match- ing with optimal transport over hyperbolic spaces. In International Conference Artificial Intelligence and Statistics, pages 1606-1617. PMLR, 2020. (Cited on p. 2)
342
[6] Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2005. (Cited p.
343
6)
345
[7]
Yuri A Antipov, Ricardo Estrada, and Boris Rubin. Inversion formulas for the spherical means in constant curvature spaces. arXiv preprint arXiv: 107.5999 2011. (Cited on p. 6)
346
347
[8] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
348
349
(Cited on p. 1)
[9] Eleonora Bardelli and Andrea Carlo Giuseppe Mennucci. Probability measures on infinite- dimensional stiefel manifolds. Journal of Geometric Mechanics, 9(3):291, 2017. (Cited on p.
352
5, 18. 23)
353
[10] Thomas Bendokat, Ralf Zimmermann, and P-A Absil. A grassmann manifold handbook: Basic geometry and computational aspects, arXiv preprint arXiv:2011.13699 2020. (Cited on
354
p. 5)
356
[11] Camille Besombes, Olivier Pannekoucke, Corentin Lapeyre, Benjamin Sanderson, and Olivier
357
Thual. Producing realistic climate data with generative adversarial networks. Nonlinear Processes in Geophysics, 28(3):347-370, 2021. (Cited on p. 1)
358
359
[12] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 1 112(518):859-88) 2017. (Cited
360
361
on p. 8, 28)
[13] Jan Boman and Filip Lindskog. Support theorems for the radon transform and cramér-wold theorems. Journal of theoretical probability, 22(3):683-710, 2009. (Cited on 3, 6)
383
364
[14]
Silvere Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2117-2229, 2013. (Cited on p. 24)
385
386
[15]
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51(1): 22 45, 2015. (Cited on p. 3, 6)
367
368
[16] Nicolas Bonnotte. Unidimensional and evolution methods for optimal transportation. PhD thesis, Paris 11, 2013. (Cited on 2)
370
371
[17] Nicolas Boumal. An introduction to optimization on smooth manifolds. To appear with Cambridge University Press, Apr 2022. URL http::/www.we http:///wwwww.olssooo.o... bookbook.
(Cited on p. 24, 27)
10

---

374
[18] Simon Byrne and Mark Girolami. Geodesic monte carlo on embedded manifolds. Scandinavian Journal of Statistics, 40(4))825-845, 845,, 2013. (Cited on p. 29)
375
376
[19]
Carlos A Cabrelli and Ursula M Molter. The kantorovich metric for probability measures on the circle. Journal of Computational and Applied Mathematics, 57(3):345 361, 1995. (Cited on 4)
377
378
379
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Ar- mand Joulin. Unsupervised learning of visual features by contrasting cluster assign- ments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 9912-9924. Curran Associates, Inc., 2020. URL. 70feb62b69f 1wt.a900066000888111222222222282 [21]
[00]
380
381
382
383
neurips...... wc/ppaper//02///// pdf. (Cited on p. 1, 32)
384
385
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 18 Jul 2020. URL mlr press / v 19/ chenn0jj .ttml. (Cited on p. 1, 32, 33, 34)
386
387
380
[22] Xiongjie Chen, Yongxin Yang, and Yunpeng Li. Augmented sliced wasserstein distances. arXiv preprint arXiv:2006.08881 2020. (Cited on p. 3, 6)
382
[23] Samuel Cohen, Brandon Amos, and Yaron Lipman. Riemannian convex potential maps. In International Conference on Machine Learning, pages 2028-2033 PMLR, 2021. (Cited on p.
383
2, 25)
395
[24] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 39()): 1853-1865, 2016. (Cited on p. 1)
398
[25] Li Cui, Xin Qi, Chengfeng Wen, Na Lei, Xinyuan Li, Min Zhang, and Xianfeng Gu. Spherical
399
optimal transportation. Computer-Aided Design, 1 115:181-193, 2019. (Cited on p. 2)
400
[26]
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013. (Cited on p. 7)
401
402
[27]
Susanna Dann. On the minkowski- transform. arXiv preprint arXiv: 1003.5565, 2010.
403
(Cited on p. 6)
404
[28]
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomczak. Hyperspherical variational auto-encoders. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pages 856 865. AUAI Press, 2018. URL http: http://aaaa...oo..o/.o///
405
408
httpss//appp///aaeee /009 pdf. (Cited on 1, 25)
409
[29]
Nicola De Cao and Wilker Aziz. The power spherical distribution. arXiv preprint Sttiv:2006.044331 2020. (Cited on p. 30)
411
[30] Julie Delon, Julien Salomon, and Andrei Sobolevski. Fast transport optimization for monge costs on the circle. SIAM Journal on Applied Mathematics, 70(7):2239-2258, 2010. (Cited on
412
p. 3, 4, 6,
414
[3]] Ishan Deshpande, Yuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander G Schwing. Max-sliced wasserstein distance and its use for gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10648-10656, 2019. (Cited on 3)
415
417
418
[32] Marco Di Marzio, Agnese Panzera, and Charles C Taylor. Nonparametric regression for spherical data. Journal of the American Statistical Association, 109(506);748-76,, 2014. (Cited on on p. 1)
420
[33] Amnad Doucet, Nando De Freitas, Neil James Gordon, et al. Sequential Monte Carlo methods in practice, volume 1. Springer, 2001. (Cited on p. 8, 31)
11

---

423
[34]
Leon Ehrenpreis. The universality of the Radon transform. OUP Oxford, 2003. (Cited on p. 6) Kilian Fatras, Younes Zine, Rémi Flamary, Remi Gribonval, and Nicolas Courty. Learning with minibatch wasserstein asymptotic and gradient properties. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 2131-2141. PMLR, 26-28 Aug 2020. URL https:// //wpc////noccce....o mlr. press/v108/ fatras20a. html. (Cited on p. 1)
[35]
427
428
429
430
[36] Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4)98331049, 2016. (Cited on p. 1)
431
[37]
Jean Feydy, Thibault Séjoumé, François-Xaviee Vialard. Shun-ichi Amari, Alain Trouve, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2681- -
433
434
435
2690, 2019. (Cited on p. 31)
[38]
Alessio Figalli and Cédric Villani. Optimal transport and curvature. In Nonlinear PDE's and Applications, pages 171-217. Springer, 2011. (Cited on p. 2)
437
438
[39]
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexan- der Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021. URL http: / jmlrr.rrr.o/. pwggpapers/v/2//222 20211//2222411111 html.. (Cited on p. 31)
440
441
442
443
444
445
[40] P Thomas Fletcher, Conglin Lu, Stephen M Pizer, and Sarang Joshi. Principal geodesic analysis for the study of nonlinear statistics of shape. IEEE transactions on medical imaging, 23(8):995-1005, 2004. (Cited on p. 4)
446
447
448
[41] Walter Gander. Algorithms for the qr decomposition. Res. Rep, 80(02):1251-1268. 1980.
449
(Cited on p. 7)
[42] Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013, (Cited on p.
451
7)
452
[43] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent- a new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21271-21284. Curran Associates, Inc., 2020. URL https:///// neteeedddnnnnnnnnnne..ee. neurips. httppppprr/2///11eeeee http:////0////000000000000001110 1)
454
455
456
457
458
Paper. pdf. (Cited on p.
459
460
[44] Armelle Guillou, Philippe Naveau, and Alexandre You. A folding methodology for multi- variate extremes: estimation of the spectral probability measure and actuarial applications. Scandinavian Actuarial Journal, 2015(7):549-577) 572, 2015. (Cited on p. 1)
461
462
463
[45] Brittany Froese Hamfeldt and Axel GR Turnquist. A convergence framework for optimal
464
transport on the sphere. arXiv preprint arXiv:2103.05739, 2021. (Cited on p. 2)
465
[46] Brittany Froese Hamfeldt and Axel GR Tumquist. A convergent finite difference method for 2tt.::2005.33330 2021. (Cited on p. 2)
466
optimal transport on the sphere. arXiv preprint
[47] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 20/6 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. (Cited on p. 33)
468
489
[48] Sigurdur Helgason et al. Integral geometry and Radon transforms. Springer, 2011. (Cited on p. 5)
12

---

472
[49]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. (Cited on p. 9)
473
474
475
[50]
Andrew Homan and Hanming Zhou. Injectivity and stability for a generie class of generalized radon transforms. The Journal of Geometric Analysis, 27(2):115-1129, 2017. (Cited on p. 6)
476
477
[51] Andrés Hoyos-ldrobo. Aligning hyperbolic representations: an optimal transport-based approach. arXiv preprint arXiv:2012.01089, 2020. (Cited on p. 2)
478
479
[52]
Minhui Huang, Shiqian Ma, and Lifeng Lai. riemannian block coordinate descent method for computing the projection robust wasserstein distance. In International Conference on Machine Learning, pages 1446-4455 PMLR, 2021. (Cited on p. p)
480
482
[53]
Shayan Hundrieser, Marcel Klatt, and Axel Munk. The statisties of circular optimal transport. arXiv preprint arXiv:2 103. 1.526, 2021. (Cited on p. 4)
484
[54]
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37(2)118-233,, 1999. (Cited on p. 8, 28)
485
486
487
[55]
Sungkyu Jung, Geodesic projection of the von mises-fisher distribution for projection pursuit of directional data. Electronic Journal of Statistics, 15(1):984-1033 1033, 2021. (Cited on p. 4, 6,
488
489
24)
490
[56] Sungkyu Jung, Ian I. Dryden, and James Stephen Marron. Analysis of principal nested spheres. Biometrika, 99(3):551-568 2012. (Cited on p. 5, 20)
481
492
[57] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14126699, 2014. (Cited on p. 27, 30, 33)
483
484
[58] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
arXiv:1312.6114, 2013. (Cited on 1,
[59] Soheil Kolouri, Phillip E Pope. Charles E Martin, and Gustavo K Rohde. Sliced wasserstein auto-encoders. In International Conference on Learning Representations, 2018. (Cited on p.
497
498
9, 31)
499
[60] Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. General- ized sliced wasserstein distances. Advances in Neural Information Processing Systems, 32, 2019. (Cited on p. 3, 6, 9, 19, 31)
500
501
502
[61] Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009. (Cited on p.
503
33)
504
622]
Gerhard Kurz and Uwe D Hanebeck. Stochastic sampling of the hyperspherical von mises- fisher distribution without rejection methods. In 2015 Sensor Data Fusion: Trends, Solutions, Applications (SDF), pages 1-6. IEEE, 2015. (Cited p. 24)
505
506
[63] Shiwei Lan, Bo Zhou, and Babak Shahbaba. Spherical hamiltonian monte carlo for constrained target distributions. In International Conference on Machine Learning, pages 629-637. PMLR, 2014. (Cited on p. 29)
509
510
[64]
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: / //yann. lecun.
511
wwweeedddmmsseeeeeeeeeeeeeeeeeee (Cited on p. 9)
512
[65]
Mufan Bill Li and Murat A Erdogdu. Riemannian langevin algorithm for solving semidefinite programs. arXiv preprint 2rXiv:20 2020. (Cited on p. 29)
513
514
[66] Tianyi Lin, Chenyou Fan, Nhat Ho, Marco Cuturi, and Michael Jordan. Projection robust wasserstein distance and riemannian optimization. Advances in neural information processing systems, 33:9383-993333 9397, 2020. (Cited on p. 3)5)
515
516
517
[67] Tianyi Lin, Zeyu Zheng, Elynn Chen, Marco Cuturi, and Michael Jordan. On projection robust optimal transport: Sample complexity and model misspecification. In International
518
Conference on Artificial Intelligence and Statistics, pages 262-270. PMLR, 2021. (Cited on p.
3. 6)
13

---

521
[68]
Chang Liu, Jun Zhu, and Yang Song. Stochastic gradient geodesic meme methods. Advances in neural information processing systems, 29, 2016. (Cited on p. 29, 30)
522
[69]
Jun S Liu and Rong Chen. Blind deconvolution via sequential imputations. Journal of the american statistical association, 90(430):567-5577 -576, 1995. (Cited on p. 8, 31)
524
525
[70]
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. (Cited on p. 1)
527
528
[71] Kanti V Mardia, Peter E Jupp, and KV Mardia. Directional statistics, volume 2. Wiley Online Library, 2000. (Cited on p. 24)
530
[72] Kantilal Varichand Mardia. Statisties of directional data, Journal of the Royal Statistical
531
Society: Series B (Methodological), 3493)33 1975. (Cited on p. 8)
632
[73] Robert J McCann. Polar factorization of maps on riemannian manifolds. Geometric & Functional Analysis GAFA, 11(3):589-608 608, 2001. (Cited on p. 2)
533
534
[74]
Kimia Nadjahi. Wasseesessr distance for large-scale machine learning: theory, method- ology and extensions. PhD thesis, Institut polytechnique de Paris, 2021. (Cited on p. 2)
535
536
[75]
Kimia Nadjahi, Alain Durmus, Umut Simsekli, and Roland Badeau. Asymptotic guarantees for learning generative models with the -iise-snsnn distance. Advances in Neural Information Processing Systems, 32, 2019. (Cited on 3)
537
538
539
[76] Kimia Nadjahi, Alain Durmus, Lénaic Chizat, Soheil Kolouri, Shahin Shahrampour, and Umut Simsekli. Statistical and topological properties of sliced probability divergences. Advances in Neural Information Processing Systems, 13:20802-20812, 2020. (Cited on p. 26)
540
541
[77] Frank Natterer. The mathematics of computerized tomography. SIAM, 2001. (Cited on p. 5)
543
[78] Khai Nguyen and Nhat Ho. Amortized projection optimization for sliced wasserstein generative models. arXiv preprint arXiv:2203. /3417, 2022. (Cited p. 3)
545
[79] Khai Nguyen and Nhat Ho. Revisiting sliced wasserstein on images: From vectorization to convolution. arXiv preprint arXiv:2204.011 2888 2022. (Cited on p.
548
547
[80] Khai Nguyen, Son Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Improving relational regularized autoencoders with spherical sliced fused gromov wasserstein. arXiv preprint arXiv:2010.01787, 2020. (Cited on p. 3)
548
549
550
[81] Khai Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Distributional disser-wasee and applications to generative modeling. In 9th International Conference on Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL seee//oorumidddyjo0000o
551
552
553
https://openreval
(Cited on p. 3)
554
[82] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1-64, 2021. (Cited on p. 25)
555
556
557
[83] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d''lché-Bue E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019. (Cited on p. 7)
558
559
560
562
583
[84] Giorgio Patrini, Rianne van den Berg, Patrick Forre. Marcello Carioni, Samarth Bhargav, Max Welling, Tim Genewein, and Frank Nielsen. Sinkhorn autoencoders. In Uncertainty in Artificial Intelligence, pages 333-743. PMLR, 2020. (Cited on p. 9, 31)
565
586
[85] François- François-Pierre
Pierre Paty and Marco Cuturi. Subspace robust wasserstein distances. In International conference machine learning, pages 5072-5081. PMLR, 2019. (Cited on p.
14

---

589
[86] Nathanaël Perraudin, Michaël Defferrard, Tomasz Kaeprzak, and Raphael Sgier. Deepsphere:
570
Efficient spherical convolutional neural network with healpix sampling for cosmological applications. Astronomy and Computing, 27:130-146, 2019. (Cited on p. 1)
571
572
[87] Arthur Pewsey and Eduardo Garcia-Portugués. Recent advances in directional statistics. Test, 30(1):1-58, 2021. (Cited on p. 1)
573
574
[88] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data
575
science. Foundations and Trends@ in Machine Learning, 1165-6):355-607, 2019. (Cited on p.
576
2, 4)
577
[89]
Julien Rabin, Julie Delon, and Yann Gousseau. Transportation distances on the circle. Journal of Mathematical Imaging and Vision, 41(1):147-161) 2011. (Cited on p. 2, 4, 17)
578
579
[90] Julien Rabin, Gabriel Peyré, Julie Delon, and Mare Bernot. Wasserstein barycenter and its application to texture mixing. In International Conference on Scale Space and Variational Methods in Computer Vision, pages 435 446. Springer, 2011. (Cited on p. 1,
580
581
582
11]]
Danilo J Rezende and Sébastien Racanière. Implicit riemannian concave potential maps. arXiv preprint arXiv:2 10.0/288, 2021. (Cited on p. 2, 25)
583
584
[92] Danilo Jimenez Rezende, George Papamakarios. Sébastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. In International Conference on Machine Learning, pages 0183-8092.10 PMLR, 2020. (Cited on p. 8, 25, 30,31 31)
5e5
586
587
588
[93] Christian P Robert, George Casella, and George Casella. Monte Carlo statistical methods,
volume 2. Springer, 1999. (Cited on p. 8)
590
194]
Boris Rubin. Inversion and characterization of the hemispherical transform. Journal d' d'Analyse Muthématique, 77(1):105-12) 1999. (Cited on p. 6, 21)
592
[95]
Boris Rubin. Inversion formulas for the spherical radon transform and the generalized cosine transform. Advances in Applied Mathematics, 29(3):471 1-497, 2002. (Cited on p. 6)
588
594
[96]
Boris Rubin. Notes on radon transforms in integral geometry. Fractional Calculus and Applied
585
Analysis, 6(1):25-72, 2003. (Cited on p. 6, 19, 20)
596
[97] Meyer Scetbon, Marco Cuturi, and Gabriel Peyré. Low-rank sinkhorn factorization. In International Conference on Machine Learning, pages 9344-9354. PMLR, 2021. (Cited on p.
597
1)
589
[98] Suvrit Sra. Directional statistics in machine learning: a brief review. Applied Directional Statisties: Modern Methods and Case Studies, 225:6, 2018. (Cited on p. 1)
601
[99] Ilya O. Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. Wasserstein auto- encoders. In 6th International Conference on Learning Representations, ICIR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: / http:///ppeeeeiieeeeee n / forum? 7dd Hidd--kLn----00 -bb. (Cited on p. 31)
602
604
[100] Gary Ulrich. Computer generation of distributions on the m-sphere. Journal of the Royal Statistical Society: Series C (Applied Statistics), 33(2):158-1)) 1984. (Cited on p. 24)
606
[101] Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009. (Cited on p. 1,
608
2, 23)
[102] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Exgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261 -772, 2020. doi: 10. 10.1088/441522-01-000 0666--... (Cited on p. 30)
611
612
613
614
615
15

---

618
[103] Dilin Wang and Qiang Liu. Learning to draw samples: With application to amortized mle for generative adversarial learning. arXiv preprint arXiv: 1,0/722, 2016. (Cited on p. 29)
619
[104] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-99999999 PMLR, 2020. (Cited on p. 1, 32, 33, 34)
622
623
[105]
Xiao Wang, Qi Lei, and Toannis Panageas. Fast convergence of langevin dynamics on manifold: Geodesics meet log-sobolev. Advances in Neural Information Processing Systems, 33:18894
624
625
18904, 2020. (Cited on p. 8, 29)
626
[106] Michael Werman, Shmuel Peleg, and Azriel Rosenfeld. A distance metric for multidimensional histograms. Computer Vision, Gruphics, and Image Processing, 32())328-336, 1985. (Cited
627
628
on p.
629
[107] Andrew TA Wood. Simulation of the von mises fisher distribution. Communications in and computation, 23(1):157-164, 1994. (Cited on p. 24)
630
631
[108] Zhirong Wu, Yuanjun Xiong. Stella X. Yu, and Dahua Lin. Unsupervised feature learning via com-paraettric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. (Cited on p. 1, 32)
632
633
634
[109] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion- miiis:: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 1xXiv:7708.070 07744, 2017. (Cited p.
635
637
[110]
Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii, editors, Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018. pages 4503-4513, Association for Computational Linguisties, 2018. (Cited on p. 1, 9, 25)
640
641
642
[111] Mingxuan Yi and Song Liu. Sliced wasserstein variational inference. In Fourth Symposium on Advances in Approximate Bayesian Inference, 2021. (Cited on p. 8, 29)
643
16

---

644
A
Proofs
645
A.1 Proof of Proposition 1
Optimal Let J E P2(5"), Uni Since v is the uniform distribution on S its cdf is 647 the identity on [o, 1] (where we identified 5 and [0, 1). We can extend the edf F on the real line as
648 in [89] with the convention Fly + 1) F(y) + 1. Therefore, F2 = Id on R. Moreover, we know that for all x E S (Fu (x) = FF (x + a) =x + a and
649
/ (t) (Fi -a) dt. aeR
w2(,v)) inf
(25)
For all a E R, let f(a) = 1o (F7 (Fu- a) (11)² 't. Then, we have:
650
Va R, [(a) // (FT) (1) - a)² dl
(1) - -)²² dt + - 2a // (1) -t) dt
(26)
L. (F7 (1) - 1)² di + 20 (%/ x ddu(t) - 1 2
where we used that , Uni 1]) = μ.
Hence, - + a= fo du(x)) - 2.
652
653
foomed-form for empirical distributions. Let (z:)i_m E [0, 1[" such that æ1 <...< < nn and let
n Em= 8z, a discrete distribution.
654
To compute the closed-form of W2 between Nn, and v Uni we first have that the optimal
656
is an Ei= x3- Z Moreover, we also have:
W2 (Mn v) = (/' (F) - (t + atn))) dt
(27)
["d Fin (1) dt 2 f( (F) 26, [' Fr +
1 + ân + oZg
3
657
Then, by noticing that F() = x, for all l E [F(::), F(x:+1)( we have
L 1P=(())d Σ
tx
x2 1),
(28)
2n
658
[ Fy
(t)
Σ
Fr
(t)dt = 1 n
Σ
Zb:
(29)
11
1=1
659
and we also have:
12
1
1
2
1
1
12
2
ān 2
+
34
(30)
n
2
n
71
71
Then, by plugging these results into (27), we obtain
2
2
W3 (M v)
x
(2
1).: - 2
23
+
+
+
x3
n
71
3
4
2m
1ml
im1
x2
1
12
n
1
F3
+
(n +
20)x: +
72
72
n
12"
(31)
17

---

A.2 Proof of Equation (17)
Let U E a... Then the great circle generated by U E Va.2 is defined as the intersection between 683 span(UU and g And we have the following characterization:
span(U""))nst
By E Rd UUT y and | =
By E Rd, UUT y and |UUT y " u10
3 E S U2.
684
And we deduce that
YU E V2, E 8d-1, p0(x)))))))
(x) = argmin dga-1 U2). 2E81
(32)
A.3 Proof of Lemma
Let U Va and e Sd-1 such that U + O. Denote U (uj u2), i.e. the 2-plane E is E span(UUT) span(u1, u2) and (u1, u2) is an orthonormal basis of E. Then, for all
688
E Sd-1, the projection on E is pp(x))))))))))))()) = (ui,z)u (12, UU
Now, let us compute the geodesic distance between x E gd-1 and Tp Ens
(x)
669
p (xx)
[ =
p (x)
dga-1
x
= arccos (1.
(33)
p (x)]l2/
670 using that x = p (x) + 5 (x)
Let En Sdd-1 another point the great circle. By the
inequality, we have
(z. 9) (x))))))))))))))))))))) 5 (x) =I1p=(x)1 (x)
(34)
Therefore, using that arccos is decreasing (-1, 1),
p²(x)
dga-1 (x. v) arccos((2: (z. y)) 2 arccos((lp"(<)))) dga-1 (z) [1
(35)
673 Moreover, we have equality if and only y = (x). And since E 5dd-1 |a| = [>) Using
5 (m)
again that arecos is decreasing, we deduce that the minimum well attained in y T
875 2
Finally, using that |UUTx|12 = =
| 2/12. deduce that
UTT |UTx|1/2
pd (x)
(36)
677 Finally, by noticing that the projection is unique if and only if U and using [9, Proposition
4.2] which states that there is unique projection for a.e. deduce that {a Srt-1 U x 0} of measure null and hence, for a.c. 5d-1, we have the result.
18

---

680 A.4 Proof of Proposition 2
881 Let f E ('((tt))), Cols x Va,2), then by Fubini's theorem,
(Rf.g) x / Vap Ss Rf(2,U)g(2, U) dzdo(U)
S I S Sa-- (=)) drdzdo(U)
Va.
Igin f(x)) I / 8(2, ()= (y) dzdo(U)dr
Vaz
(37)
Sgm. f(z) / (()"(x), do((d
ge-1
Vez
f(x)R g()) dz
ga-1
(f, R. sa
682 A.5 Proof of Proposition 3
Let E Co(S' x Va.2).
689
S0 / Ss U(2,U) (Rt) (da)
g(2,U) d(Rt)(2,UU)
SixVa
[ gd-1
It g(x) dp(x)
( (x),) do(U))p(z)
ga-1 Jva
(38)
g(P U) du(z)do(UU)
Va.2 U(, d(pl) p) (U)
Hence, for a-almost every U V4 (Ru)U pti μ.
684
685
A.6
Study of the Spherical Radon transform R
In this Section, we first discuss the set of integration of the spherical Radon transform R (19). We further show that it is related the hemispherical Radon transform and we derive its kernel.
686
687
Set of integration. While the classical Radon transform integrates over hyperplanes Rd and the generalized Radon transform integrates over hypersurfaces [60], the set of integration of the spherical Radon transform (19) is half of "big circle", i.e. half of the intersection between hyperplane and 680
688
689
Sd-1 [96]. We illustrate this on S in Figure 7. On S², the intersection between hyperplane and 5 great circle.
19

---

Figure 7: Set of integration of the spherical Radon transform (19). The great circle black and the set of integration in blue. The point Uz E span(UUT)) n Sd in blue.
Proposition Let U Va,2 S The set of integration of (19) is
693
S p (x) = 2) fo E F (x,U2) 0},
su-1,
(39)
where span(UUT)) @ (Ua))
694
Proof. Let U E Va,2, 2 E 8t. Denote E = SUan(UUT)) the 2-plane generating the great circle, and E its orthogonal complementary. Hence, E E Rd and (i = Now, let F E+ span(Uz) (Uz). Since Uz UU E E, have that dim(F) d 1. Hence, F is hyperplane and F 5dd is "big circle" [96], i.e. (d 2)-dimensional subsphere of 5
695
696
689 Now, for the first inclusion, let E fx E Sd-1, pV (x) = }}. First, we show that x E Fnst-1, By 1
Lemma and hypothesis, know that PU (I) 100 = By denoting by D the projection
701 E, we have:
(Fx)=UU' x U(IU x1|22) -IUT x1|2U22 E span(U2)..
(40)
Hence, x' = (x) Ig- = ||UTT||BU> + Øg E F. Moreover, as
702
(x,U2) |0 (12, 111 0,
(41)
703 we deduce that x E {F s (z. U2) > 0}.
For the other inclusion, x {F Sd-1 (z, U2) 0}. Since x F, we have x ZF + AUz,
704
A R. Hence, using Lemma
UTx x À z = (Agg(
PU (1) =
(42)
1 | ||
706 But, we also have (a. U2) 21 > Therefore, sign(A) and PU (x) 2. Finally, we conclude that {ar E sd-1, P (x) =)) (x F (z, Uz) > 0}.
Link with Hemispherical transform. Since the intersection between hyperplane and S is isometric to 5dd2 [56], we can relate 1t to the hemispherical transform H [96] on Sd-2, First, the hemispherical transform of a function f E T.- (5d-11 is defined as
[
Yr sdd-1 Hf(())
So f(y)
(43)
From Proposition 6, write the spherical Radon transform (19) hemispherical transform on Sd-2
20

---

Proposition 7. Let E (s U E da and 2 E S¹ then
713
=f SSs 2
Uf = =
dz Hf((2), H((((Z)
(44)
where for all x E Sd-2, f(x) = I(O" with O the rotation matrix such that for all a E F,
714
Ta-1 and
715 Or E span(ei..... (6-11) where (e1...., ea) denotes the canonical basis, and = 0 01,d-1
Ú = J R(d-1)x2.
Proof. Let f E (S E SES, Va,2, then by Proposition 6,
717
Rf(2,U)= JS-INE - ((a)))d..
(45)
F a hyperplane. Let O Rdxd the rotation such that for all F, Oz E span(ee,..... Cd-1)
718
F where (EI: (el;...,e Ed) is the canonical basis. By applying the change of variable Ox y, and since
O O², det O = 1, we obtain
Rf(z,U)= JOLFNSE.,, f(0" (( f
Pris- f(0" y) (1x.0U)))00) (46)
Now, have that OU Va since (OU)"(OU) 12, and since Us F, OUr F. For all y F, we have (y, ea) = Ya = 0. Let J Id-1 (ddxid-1), then for all y S
(1,d-1))
y Jÿ where 5d-2 is composed of the d - first coordinates of y-
724 Let's s define, for all j E s f(ū) f(o" JØ). Ü J'OU.
725 Then, since À 5 sd-2, we can write:
JSa- f(T) (100))0 Hf((2).
Rf(2,U) ga-2
(47)
727
Kernel of R. By exploiting the expression using the hemispherical transform in Proposition 7, we can derive its kernel in Appendix A.7.
728
729
A.7
Proof of Proposition
First, we recall Lemma 2.3 of [94] on Sd-2.
730
Lemma 2 (Lemma 2.3 [94]). ker(H)) {n Meven (Sa-2), u(Sd=2) 0} where Meven is
the set of even measures, i.e. measures such that for all f C(Sd-2), (v, = (-, where (x) = ((=x) all E gd-2.
Let 4 Mao(S First, we notice that the density of Ry w.r.t A for all 2 E St, U Va.2.
734
735
(Rz)(z, f 1 (1p3())) Im (((.))))))d((z).
(48)
736
Indeed, using Proposition 2, and Proposition 6, we have for all E Co(S" x V42),
RRu, g1 xVa2 = (M, R"'9)a--=
R R(g(x)du(())
sd-1
ga I ( g(2, (t
(49)
- IMsaass o(z,U) o ga-1 ((==)()))du()) dzdo(U) (
=/
Jvaxxsi s(2,U) /
Frisa ((ouss)))ppu(()) dzdo(U).
21

---

Hence, using Proposition 7, we can write (Rp)(2, = (HP))U2) where j = ==
737
Now, let j ker(ft), then for all 2 8 U Va, Ru(Z,U) Hi(Uz) 0 and hence ker(H) {jz Meven (S ((( =
738
0}.
First, let's show that E Moven (S Let f E C(S4-1) and U E Va,,, then, by using the same
740
741
notation as Propositions and 7, we have
(1, f)- = SSn. f(x)du(x)= = Ssa-1 J11 (=))1(==))()) dz du(x))
J
Sa-1 (ao
I
rrss (())
by Prop.
J JsL / (1
(50)
[ (Hij, Mijifissas dz
gt
[ (ū, H.f)ss dz
si
[ (ú, (Hlf) ssa- de since jı E Meven
J f (x)du(()) = (
using for the last line all the opposite transformations. Therefore, μ Meven (Stt-1). Now, we need to find on which set the measure is null. We have
743
Vz E S,, U E Va2. 3) =0
(51)
Vz E Va m(0
Hence, we deduce that
ker(Pt) {n E (S VU E V2 F span(UU*))+ (span(Uz),
(52)
1 ((F =0..
Moreover, have that Fu,, Sd
( = c Rd, dim(H) = = d- 1}.
Indeed, on the one hand, let an hyperplane, x E H sd-1 U E Va., and note = (x) Then,
746
F gd-1 by Proposition and H 5 Uv, Fu...
747
On the other hand, let U E V4 E S1, F is hyperplane since dim(F) - 1 and therefore F n Sd-1 c (H, dim(H) d (d-13 1}.
750
Finally, we deduce that
ker( (2) = (u (St-n( AH E Ga.-1, ((
(53)
A.8
Proof of Proposition
Let p 1. First, it is straightforward to that for all p.v Pp(S SSW,u,,)) SSW,(u,)) and that we have the triangular
SSW, (H,p) (s,,
22

---

754
inequality since
(n.. WPIPH P do(())) 3
Vu,Vaa E Pp(S ()
(W W + W (p( P do(l)) $
Van
Wp(P do(l)))
WH a, v) do(U))²
= ss, a) + SSW,
(54)
755
using the triangular inequality for W, and the Minkowski inequality. Therefore, it is least
pseudo-distance.
757
To be distance, we also need SSW, (M, v) = - = v. Suppose that SSW v) = 0. Since, for all U E Va.2, ( # μ. P v) 2 0, SSWP(u, = implies that for o-ae U E V4,2.
pl/
758
P or (Rz)) ( for o-ae U E Va, since W, is distance on the circle. Therefore, it is a distance on the sets of injectivity of R.
WT(P% ,i, P v) and hence PV
759
780
761
A.9
Convergence Properties
Proposition 8. Let (u&), 1 (p((S-11) such that P* 14, then
k-+00
SSWp (b)
O.
(55)
1-+00
783
Proof.
Since the Wasserstein distance metrizes the weak convergence (Corollary 6.11 [101]), we have pf (by continuity) + W Mbs # ()) O and hence by the dominated
pl
784
k-vx0
785
convergence theorem, SSW (1),
0.
k-000
786
B
Background on the Sphere
767
B.1
Uniqueness of the Projection
Here, we discuss the uniqueness of the projection PU for almost every x. For that, we recall some results of [9].
768
769
Let M be closed subset of a complete fnite-dimensional Riemannian manifold N. Let d be the Riemannian distance on N. Then, the distance from the set M is defined as
(x(x) inf d(x, y). yEM
(56)
772
The infimum is a minimum since M is closed and N locally compact, but the minimum might not be unique. When it is unique, let's denote the point which attains the minimum as <()) i.e. d(x, (x)) dzr(x.).
773
774
775
Proposition 9 (Proposition 4.2 in 19]). Let M be a closed set in a complete m-dimensional Rieman- nian manifold N. Then, for almost every x, there exists a unique point π ((x) E M that realizes the minimum of the distance from x.
778
777
778
From this Proposition, they further deduce that the measure u is well defined on M with y locally absolutely continuous measure w.r.l. the Lebesgue measure.
779
In our setting, for all Va.2, we want to project a measure μ E P(S on the great circle sUa" S Hence, we have N 5d-1 which is complete finite-dimensional Riemannian manifold and M span(UUT) n s closed set in N. Therefore, and the fowwad measures well defined for absolutely continuous measures.
780
apply Proposition
23

---

784
B.2
Optimization on the Sphere
Let : 8d-1 - R be some functional on the sphere. Then, we can perform a gradient descent on Riemannian manifold by following the geodesics, which are the counterpart straight lines in IR Hence, the gradient descent algorithm [3. 14] reads as
785
786
787
vk 2 0, Fk+1 expx; ( (-rgrad
(57)
where for all æ E S exp, T, + 5dh1 is a map from the tangent space Tz = for e Rd, o)) = 0} to 5 such that for all T,S expa (v) = (1) with yo the unique geodesic
788
789
790
starting from x with speed v, i.e. 7(0) and 70(0) U.
For S6-1, the exponential map is known and is Vx E s Vu T (x) cos(||)))) + sin([[u|l2) |ull
(58)
Moreover, the Riemannian gradient on Sd is known as [3, Eq. 3.37] (x) = Proja (V)(x)) Vf(x) (VI()), Projg denoting the orthogonal projection on T,, Sd-
792
(59)
794
For more details, we refer to [3, 17].
795
B.3
Von Mises-Fisher Distribution
The von Mises-Fisher (yMF) distribution is a distribution 5 characterized by concentration parameter x > O and a location parameter μa E 5d-1 through the density
797
pd/211
vO E sd-1, fuMr(0; H.) (2x) 11/2211)))1 exp(xx"00)
(60)
798
where (x) lo exp(1 cos( cos(v0)d0 is the modified Bessel function of the first kind.
799
Several algorithms allow to sample from it, see e.g. [100, 107] for algorithms using rejection sampling
800
or [62] without rejection sampling-
For d 1, the vMF coincides with the von Mises (vM) distribution, which has for density
Y0 E |-,, fvm(0; Io(k) exp(x cos(0 -m)),
(61)
with j E [0,2x[ the mean direction and ε > 0 its concentration parameter. We refer to [71, Scction 803 3.5 and Chapter 9] for more details on these distributions.
802
In particular, for 0, the vMF (resp. vM) distribution coincides with the uniform distribution on the sphere (resp. the circle).
805
806
Jung [55] studied the law of the projection of a vMF on a great circle. In particular, they showed that, while the vMF plays the role of the normal distributions for directional data, the projection actually
807
808
does not follow a von Mises distribution. More precisely, they showed the following theorem:
Theorem 1 (Theorem 3.1 in (55]). Lei d 2 3, X ~ vMF( k) E Sd-1 1, U E V and T = pU (X)
809
810
the projection on the great circle generated by U. Then, the density function Tf is
Vt E |- f(t)= = S. fn(r)fva 0, x cos(6)r) dr,
(62)
where 5 the deviation of the great circle (geodesic) from f and the mixing density is a11
2
Vr 170,111 fR(r) = (x) -1
(63)
812 with (d - 2)/2 and [² = (1) 0. (0) (000))=1/((e)) (v 1). 813 Hence, as noticed by Jung [55], in the particular case a 0, iee X Unif((d-1), then
f(1) = ['¹ fn(r) (1: dr SVM(C000)) S: fn(()drr = fvu(t;
(64)
and hence T ~ Uni
24

---

815
B.4
Normalizing Flows on the Sphere
816
Normalizing flows [82] are invertible transformations. There has been recent interest defining
817
such transformations on manifolds, and in particular on the sphere [23, 91, 92].
Here, we implemented the Exponential map normalizing flows introduced in [92]. The transformation T is
Vr E sd-1 = T(x) = exp2 (Proj.
(65)
where ((x) = E & (3)); and (μt); are the learnable parameters.
ea 1) a; 2 0. Σ, a; S 1, μr E s and 3 > 0 for all i. (as)s.
820
822
The density can be obtained as
pz(2) (x(x) det
(66)
where J is the Jacobian in the embedded space and ((x) it the matrix whose columns form an orthonormal basis of Sd-d
824
825
The common way training normalizing flows to use either the reverse or forward KL divergence. Here, we use them with different loss, namely SSW.
828
C
Additional Experiments
827
828
C.1
Evolution of SSW between von Mises-Fisher distributions
829
The KL divergence between the von Mises-Fisher distribution and the uniform distribution has been
830
derived analytically in [28, 110] as
La/2(k)
d
-1) log log((2)) loog (k))-1(K))
Ta/2-1(k)
d
d
+ 2
log ग + llogg log I'
2
(67)
831
We plot on Figure the evolution of KI. and SSW w.r.t k for different dimensions. We observe a different trend. SSW seems to get lower with the dimension contrary to KL.
832
Ssw?
50
d=3
25
100
0.04
d=3
20
25
d= 100
50
100
130
200
230
50
100
150
200
250
(a) (TLFVV ()([MPF(
(b) (D>SWW (3
Figure Evolution w.r.f K. between vMK(u, k) and v.MF(.,0). For SW, we used 100 projections (for memory reasons for d 100), and computed it for & E (1,5, 10, 20, 30, 40, 50, 75, 100, 150, 200, 250), times by dimension and 6, and with 500 samples of both distributions.
833 As a sanity check, we compare on Figure the evolution of SSW between vMF distributions 834 where we fix vMF(uo, 10) and we rotate the first vMF along a great circle. More precisely, plot SW2 (vMF((1, 0, O...., 10), *MF((cos(()) sin( O..... 10)) for 0 k ( As expected, we obtain a bell shape which is maximal when the second vMF distribution has for location
25

---

837 parameter -po-
We observe a similar behavior between SSW,, SSW1 and SW2 with different scales.
838
SW?
55W
0.150
d 3
0.45
0.8
0.125
25
0.30
d= 25
0.25
0 50
0.6
d=3
0.75
0.20
0.15
0.4
25
d0
0.30
0.2
0.25
0.05
Figure Evolution of SW between vMF samples in Sd (mean over 100 batch).
839 Figure 10, we plot the evolution of SSW w.s. the number of projections different dimensions.
840 We observe that for around 100 projections, the variance seems be low enough.
50
d 1000
== 10
== 100
100
d = 2000
200
500
0.06
SSW?
SSW?
10
0.02
0.00
10
10a
10a
Number projections
Number projections
Figure 10: Influence of the number of projections. We compute (VV(((( 0)) 20 times, for n = 500 samples in dimension d = 3.
841 Nadjahi et al. [76] proved that, contrary to the Wasserstein distance, the classical sliced- Wasserstein 842 distance has sample complexity independent of the dimension d. We show empirically on Figure 11
843 that we expect to have similar results for SSW by plotting SSW and the Wasserstein distance (with geodesic distance) between samples of the uniform distribution the sphere w.E.t. the number of samples. We observe indeed that the convergence rate of SSW is independent of the dimension.
26

---

SSW3, d=3
10
Wg.
SSW3. d-50
W3. d-50
10
10
104
Number of Samples
and Wasserstein distance (with geodesic distance) between samples of the uniform distribution on the sphere. Results are averaged over 20 runs and the shaded correponds the standard deviation.
Figure 11: Spherical
846
C.2
Runtime Comparisons
847
We study here the evolution of the runtime w.E.f different parameters. On Figure 12, we plot for several dimensions the runtime compute SSW, w.t. the number of projections and the number of samples. We observe the linearity w.r.f. the number of projections and the quasi-linearity w.r.f. the number of samples.
850
1.0
d=3
0.7
d=3
100
Seconds
Computational
0.5
500
0.6
d=10000
d 1000
0.2
250
500
1250 1500 Number projections
750
1000
1750
2000
500
1000
1500
2000
2500
30.00
Number samples
Figure 12: Computation time w.t. the number of projections samples, taken for 10 and 71 500 samples for the left figure, and к = 10 and 200 projections for the right figure, and for 20
times.
851
C.3
Gradient Flows
Mixture of vMF distributions. For the experiment in Section 5.1, we as target distribution of mixture of 6 vMF distributions from which have access samples. We refer to Appendix B.3 for background on vMF distributions.
855
The 6 vMF distributions have weights 1/6, concentration parameter K = 10 and location parameters 1,
856
11 = (1, 0, 0), /12 (0, 1, 1), /3 = (0, 0, 1), /4 (-1, 0). M4 (0,-1, 10,-1,00 0) and μ4g = (0, 0, -1).
We two different approximation of the distribution. First, approximate it using the empirical distribution, i.e. ja Σi dz, and we optimize over the particles (r:)) To optimize over particles, we can either use projected gradient descent:
859
(((+1)
=((k) V aa a SSW3 (111
(68)
(((+1)
[2
860 or a Riemannian gradient descent on the sphere [3] (see Appendix B.2 for more details). Note that
861 the projected gradient descent is Riemannian gradient descent with retraction [17].
882 We also use neural networks such as multilayer perceptron (MLP). We used MLP composed
of 5 layers of 100 units with leaky relu activation functions. The output of the MLP is normalized on
the sphere using L normalization. We perform gradient descent using Adam [57] the optimizer
27

---

885
with learning rate of 10 10 for 2000 epochs. We approximate SSW with L = 1000 projections and
886 a batch size of 500. The base distribution is choose as the uniform distribution on the sphere.
867 We report on Figure 13 comparison of the approximations where the density is estimated with Gaussian kernel density estimator.
(a) Target: Mixture of vMF
(b) KDE estimate of the MLP
(c) KDE estimate of 500 particles
Figure 13: Minimization of SSW with respect mixture of vMF.
vMF distribution. A simpler experiment, we choose simple vMF distribution with x = 10. We 870 report on Figure 14 the evolution of the density approximated using a KDE, and on Figure 15 the
871 evolution of particles.
(a) Target distribution
(b) Initial samples
(c) Approximated dis- tribution at the epoch 100
(d) Approximated dis- tribution at the epoch 1000
Figure 14: Gradient Flows on SW with a vMF target and Mollweide projections. The distributions approximated using KDE.
Target
Target i
Target
i
i
10
10
0.5
0.5
0.5
0.
-1.0
000
0
10 a5 00
10 0.5 0.0
10 05 0.0
-.5.10
10
y
-.5
10
-05
10
y
(a) Initial samples
(b) Approximated distribution at the epoch 100
(c) Approximated distribution the epoch 1000
Figure 15: Gradient Flows on SW with vMF target and Mollweide projections.
872
C.4
Sliced- Wasserstein Variational Inference
873
C.4.1
Variational Inference
In variational inference (VI) [12, 54], we have some observed data (x,)"= and some latent data (z)) The goal of variational inference is to approximate the posterior distribution p(: (æ) by some distribution g Q where Q is family of probabilities. The usual way of doing that is to minimize
28

---

Algorithm 2 SWVI [111]
Input: V a potential, K the number of iterations of SWVI, N the batch size, & the number of
MCMC steps
Initialization: Choose do a sampler
for k = to K do
Sample (=))) qo
Run é MCMC steps starting from (=)))=1 to get (z))
// Denote AA0 = N شز= 82p and fig = N ΣN 84g
Compute J SW2 (Jo: ite)
Backpropagate through J w.r.t. 0 Perform a gradient step
end for
a77 the Kullback-Leibler divergence among this family, i.e.
.(2)
min (T(lll(-11)) = F Dlg qEQ
(69)
(((Z(zr
878
But the KL divergence suffers from some drawbacks, as it is only divergence (i.e. it does not satisfy the triangular inequality, and it is non symmetric), but also suffers from under estimating the target distribution (or over estimating it for the reverse KL).
879
880
Yi and Liu [111] propose to which gives the iasserseen variational inference method. Basically, given some unnormalized probability p(-[2) that we want to approximate with some variational distribution 4g. we can first apply a MCMC algorithm and then learn using a gradient descent on SW with the target being the empirical distributions of the samples given by the MCMC. But running long MCMC chain is time consuming and it might be difficult to diagnose burn-in period. Therefore, they propose to only run each iteration some number of steps & of MCMC chain, and then learn by gradient descent the variational distribution. Therefore, the variational distribution is guided at each step by the MCMC samples toward the stationary distribution which is the target. This is called an amortized sampler (see Problem 1 in [103]). We sum up the procedure in Algorithm 2.
optimal transport distance instead, namely the SW distance
883
884
885
886
887
890
891
We propose here to substitute SW by SSW in order to perform SSWVI on the sphere. To do that, we first need a MCMC method on the sphere.
882
883
C.4.2
MCMC on the Sphere
894
Several MCMC methods on the sphere have been proposed. For example, Hamiltonian Monte-Carlo (HMC) methods were proposed in [18, 63, 68), and Riemannian Langevin algorithms were proposed in [65, 105].
897
In our experiments, we use the Geodesic Langevin algorithm (GLA) introduced by Wang et al. [105]. This algorithm is a natural generalization the Unadjusted Langevin Algorithm (ULA) and consists at simply following the geodesics of the regular ULA step, i.e.
896
899
Vk O, Ja+1 = exPrx (Projz, (-XVV(a)) + /2xZ)), Z ~(0
(70)
900
where for the sphere,
Vo S Yo E T., exp = x cos([lull) - Tel sin((l
(71)
Projr is the projection on the tangent space Tz = fo E Rd, (x, = 0} (which is the orthogonal space) and is defined as
902
Proja(2) = U)..
(72)
For more details, we refer to [3].
29

---

904
We use GLA here for simplicity and as proof of concept. But note that GLA, as ULA, is biased and therefore the distribution learned will not be the exact true stationary distribution. However, a Satropolis-Hasting step at each iteration could be used to enforce the reversibility w.zt. the target distribution or we could use other MCMC with more appealing convergence properties (see egg. [68]).
905
906
907
908
C.4.3
Applications
Target: Power spherical distribution. First, simple example S the power spherical distribution introduced by De Cao and Aziz [29]. This distribution has the advantage over the vMF distribution to allow for the direct use of the reparameterization trick since it does not require rejection sampling. The pdf is obtained as,
912
Vx E S Px (x(H,) (1+ x))"
(11x011111
(73)
with le S and K O. We can sample from drawing first Beta( ( + K, -
913
~ U then constructing T = 2Z - 1 and Y = [T" vz T-j" Finally, apply Householder reflection about μ to Y. All the operations are well differentiable and allow to apply the reparametrization trick. For the algorithm, see Algorithm in [29]. Hence, in this case, if we denote ge the map which takes samples from a uniform distribution on S and from Beta distribution as input and outputs samples of power spherical distribution with parameters 0 (K5, pi), the sampler. We test the algorithm with target being a power spherical distribution of parameter (0, 1) and = 10, starting from μ = (1, 1) and K 0.1. Performing 2000 optimization steps with a gradient descent (Riemannian gradient descent on μ to stay on the sphere), and 20 steps 922 of the GLA algorithm, we are getting close enough to the true distribution as we can see on Figure 16.
914
915
916
917
918
921
923
For the hyperparameters, we used step size 10- for GLA, 1000 projections approximate SSW, Riemannian gradient descent the sphere [3] to learn the location parameter ju with a learning rate of 2, and learning of 200 for K. We performed K 2000 steps and used N 500 particles.
O.0
O
(a) Target distribution
(b) Initial sampler
(c) Approximated dis- tribution the epoch 1000
(d) Approximated dis- tribution at the epoch 2000
Figure 16: SWVI on Power Spherical Distributions with Mollweide projections.
928 Target: mixture of vMFs. In Section 5.1, we perform amortized variational inference with
927 mixture of vMF distributions as target. For this, we train exponential map normalizing flows (see
928 [92] and Appendix B.4). Moreover, we use the same target as Rezende et al. [92], i.e. the target (0.7, 1.5), (-1, 1), μs = (0.6, 0.5) and f4 = --0.7,44 4). These are spherical coordinates which are be converted to euclidean using (0 (0, 6) (sin @ cos sin øssnn 0, cos Ø).
has a density p(z) x Et=1 10z""T, sc(()) with
929
930
931
932
The exponential map normalizing flow is composed of N 6 blocks with K 5 components. We the algorithm for 10000 iterations, with at each iteration 20 steps of GLA with ? 10-1 as learning rate, and one step of backpropagation through SSW using the Adam [57] optimizer with learning rate of 10-3
934
935
996
We report on Figure 4 the Mollweide projection of the learned density. Since we learn samples from noise distribution, here the uniform distribution on the sphere, we do have directly access the density and we report kernel density estimate with Gaussian kernel using the implementation of Scipy [102].
937
30

---

940 We also report in Figure 5 the effective sample size (ESS) [33, 69] over the iterations. The ESS is
941 estimated by [92]
Varunis(E (e-Nns(((BuX)))
P)(X))
Σs=1
Wis
ESS
(74)
Varg
w2
where w, =,, The ESS is reported percentage of the sample size. Higher ESS indicates that the flow matches the target better [92].
944
C.5
Sliced- Wasserstein Autoencoder
945 We recall that in the WAE framework, we want to minimize
((f.g) = o().
+ xD(FHH.pz)
(75)
946
where f is an encoder, 9 decoder, pz a prior distribution, some cost function and D is a divergence
947 the latent space. Several were proposed. For example, Tolstikhin et al. [99] proposed to use
948 the MMD, Kolouri al. [59] used the SW distance, Patrini et [84] used the Sinkhorn divergence,
949
Kolouri al. [60] used the generalized SW distance. Here, we use SSW?
Architecture and procedure. For the encoder f and the decoder g, we use the same architecture
as Kolouri et al. [59].
952
For both the encoder and the decoder architecture, we use fully convolutional architectures with 3x3 convolutional filters. More preciscly, the architecture of the encoder
958
28x22 + Conv2d16 + LeakyReLUn 0.2
+ Conv2dje + LeakyReLU + AvgPool2
+ Conv2d32 + LeakyReLUp>
Conv2d32 + LeakyRel.Up.3 + A AvgPoolz
Conv2das Leaky
Conv2de1 + LeakyReLUp2 AvgPool2
Flatten - FC12s + ReLU
FC4z + L² normalization
954
where dz the dimension of the latent space (cither 11 for 110 or 3 for S2).
The architecture of the decoder is
Rd
FC128 + FC1024 ReLU
Reshape(64x4xx) + Upsample, + Conve4 + LeakyReLU
+ Conves + LeakyReLUp
- Upsample, + Conv64 + LeakyReL.Up,
Conv LeakyReLUup.
Upsample, - Conv32 + LeakyReLUp
Conv Sigmoid
956
To compare the different autoencoders, we used the reconstruction loss the binary cross entropy. = 10, Adam [57] optimizer with learning rate of 10-3 and Pytorch's default momentum parameters for 800 epochs with batch of size n 500. Moreover, when using SW type of distance,
959 we upproximated it with L = 1000 projections.
960
report in Table the FID obtained using 10000 samples and we report the mean over trainings.
For SSW, used the formulation using the uniform distribution (12). To compute SW, we used the POT library [39]. To compute the Sinkhorn divergence, used the GeomLoss package [37].
31

---

983
Additional experiments.
We report on Figure 17 samples obtained with SSW for uniform prior
S
984
on
2
4
4
R
7
9
7
1
3
4
9
4
4
6
2
2
3
&
9
3
4
7
4
3
3
1
4
0
7
S
6
2
3
8
3
8
6
0
3
9
9
0
5
2
/
1
9
9
6
7 3
7
8
q
A
3
4
5
7
6
9
5
3
4
7 2
4 4 8 8 O 6
4
9
2
8
8
0
5
7
3
1
7
6
7 1 0 7 4
4
Y
4
9
9
6
3
9
8
7
-
6
S
8
7
0
3
3 4
5
0
3
0
7
3
0
3 6 /
2
9
3
f
/
5
9
3
d
9
6
0
6
8
5
4
8
0
6
2
7
1
/
3
T
8
4
8 7
8 8
/
7
4
9
5
0 Q
4
7
8
4
0
/
6
8 7
7
Q 3
9
3
6
4
(a) SSWAE
(b) SWAE
(c) SAE
Figure 17: Samples generated with Sliced-Wassersee Wasserstein Autoencoders with a uniform prior on $10
On Figure 18, we add the evolution over epochs of the Wasserstein distance between generated images and samples from the test set.
966
logio (W (u))
logio(W3lgefau pI)
1.6
1.65
SSWAE SWAE
1.5
SSWAE
SWAE
1.60
1.4
1.3
1.55
1.2
1.1
1.45
1.0
0.9
1.40
600
Epochs
Epochs
(a) With uniform prior on S
(b) With prior on S10
Figure 18: Comparison of evolution of the Wasserstein distance over epochs between SWAE and SSWAE on MNIST (averaged over 5 trainings).
C.6
Self-supervised learning
Table 2: Linear evaluation CIFAR1O. The features are taken either on the encoder output or directly on the sphere $2
988
We conduct experiments using SSW to prevent collapsing representations in con-
969
970
trastive self-supervised learning (SSL) models. Such contrastive losses on the hy- persphere have exhibited great representa- tive capacity [20, 21, 108] unlabelled datasets by learning robust image represen- tations invariantly to augmentations. As proposed in [104], the contrastive objec- tive can be decomposed into an alignment loss which forces positive representations
Method
Encoder output
S
971
Supervised
82.26
81.43
972
Chen al. [21]
66.55
59.09
Wang and Isola [104]
60.53
55.86
975
SW-SSL, 1, 10
62.65
57.77
976
SW-SSL, = 1, 1, 3
62.46
57.64
977
SSW-SSL, A 20, = 10
64.89
58.91
978
SSW-SSI, À L
63.75
59.75
979
coming from the same image to be similar
and uniformity loss which preserves maximal information of the feature distribution and hence avoids collapsing representations. Without the uniformity loss, the representations tend to converge
32

---

towards constant representation which yields the best alignment loss possible but also contains no information about original images. Wang and Isola [104] propose to enforce uniformicity by leveraging the Gaussian potential kernel which is bound to the uniform distribution on the sphere. This formulation is also related to the denominator of the contrastive loss as specified in Chen et al. [21]. We propose replace the Gaussian kernel uniformity loss with SSW for which the complexity 986
982
988
984
985
987
is more linear w.e.f. the number of batch samples. A simple choice of the alignment loss is to minmmize the mean squared euclidean distance between pairs of different augmented versions of the same image. A self-superviser learning network is pre-trained using this alignment loss added with an uniformity term. Our overall self-supervised loss can be defined as:
990
X SSW3 SSW
Sssw-sSL-
Σ ==1
1
B 2
(76)
2
Uniformity loss
Alignment loss
where 24, =A,=B E Raxd are the representations from the network projected on the hypersphere of two augmented versions of the same images, V Uni is the uniform distribution on the hypersphere and > is used to balance the two terms.
981
992
983
994
We pretrain a ResNet18 [47] model on the CIFARIO [61] data with projections projected onto the sphere 5². This feature dimension allow us to visualize the entire validation set of CIFAR10 and its distribution on the sphere. The visualization of the projections on S are visible on Figure 19. We then evaluate the performance of each contrastive objective by fitting a linear classifier on top
985
996
987
988
of the output of the layer before the projection on the sphere on the training dataset is common for SSL methods. For comparison, the sphere. As baseline, we also train a predictive supervised encoder by training jointly the linear classifier and the image encoder in supervised manner using cross entropy.
also report the results when the features are taken directly on
1000
1001
1002
We use a ResNet18 [47] encoder which outputs 1024 features that are then projected onto the sphere 22 using last fully connected layer followed by a L² normalization. We pretrain the model for 200 epochs using minibatch stochastic gradient descent (SGD) with a momentum of 0.9, a weight decay of 0.001 and initial learning rate of 0.05. We use batch size of 512 samples. The images are augmented using standard set of random augmentations for SSL: random crops, horizontal flipping.
1003
1004
1005
1006
1007
color jittering and gray scale transformation as done in Wang and Isola [104]. For the trade-off parameter X, we À = 20 for SSW and À = 1 for SW.
1008
1009
To evaluate the performance representations, we use the common linear evaluation protocol where linear classifier is fitted top of the pre-trained representations and the best validation accuracy is reported. The linear classifiers are trained for 100 epochs using the Adam [57] optimizer with learning rate of 0.001 with a decay of 0.2 at epoch 60 and 80. We compare our methods with two other contrastive objectives, Chen et al. [21] with the normalized temperature-scaled cross-entropy Xen-X loss and Wang and Isola [104] which proposes to decompose the objective in two distinct terms Lalign and Luniform- We recall the respective uniformity loss of each method Table 3. As one can see Table 2, our method achieves here comparable performances to two state-of-the-an approaches, yet slightly under- perierr-rformiinggg compared to [21]. We suspect that finer validation of the balancing parameter À is needed. Especially since the representations on Figure 19b are not completely uniformly distributed around the sphere after pre-training compared to other contrastive methods. Nevertheless, these preliminary results show that SSW-SSL is a promising contrastive learning approach without explicit distances between negative samples, especially compared to SW on the sphere. To this end, further works should be devoted to finding a good balance between the alignment and uniformity objectives.
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
33

---

Vianc
brrd
argione
30
477
brind
ATTE
withmaatee
1500 SES GRAA O De 10-130
CDt
doct
dog
125
deet
dog
Flog
frog
Iastuc
Iutrac
sttp
J5
truct
truck
(a) Wang and Isola [104]
(b) SSW-SSL, 6, L = 200
SimCLR
Random inthalization
drglann
lard
autommaile
304
autometie
lrdd
deer
350
77
(c) Chen al. [21]
Random initialization
Supervited predictive
eirplane
30
autommblie
ape aze 1o
flig
horse
buck
(c) Supervised prediction
Figure 19: The CIFARIO validation set on 5 after pre- pre-training.
Table 3: Comparison of contrastive methods and their respective uniformity objective where 21 B E Raxd are representations from augmented versions of the same set of images and Sdd-() is the uniform distribution on the hypersphere. Method
Luniform (2a) + Lyniform (B
Complexity
Chen et al. [21]
D log Ej exp( (2, 2,, 2,) ) cat( ( BB)
O(n'd)
Wang and Isola [104]
Ex 23 log n(a-1) Σi>y exp(-1l|z 21111))
O(n2d) ((Lndd logn))
SSW-SSL (Ours)
(SSW3 A v) S=
34


Reviews:
[{"review_id": "nggcRNFe9Cb", "reviewer": "Reviewer_JCLC", "summary": "The paper proposed a new version of Sliced Wasserstein Distance on the sphere as a first step to deal with manifold data. Section 2 is to recall the definition of Wasserstein distance and Sliced Wasserstein distance. In Section 3, the authors introduced the spherical sliced Wasserstein distance, mainly based on the property that the sphere could be described by spherical coordinate system. They also defined the Radon transform for the SSWD and show some properties of Wasserstein distance on circle. The implementation is shown in Section 4. Section 5 is devoted for applications which include variational inference, generative modeling and density estimation.", "questions": "No", "limitations": "It is fine.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The idea of Spherical Sliced Wasserstein Distance is nice and interesting. The SSWD is well-defined and some of its properties are explored.\n\nWeaknesses: The applications of SSWD shown in the paper are limited. In those experiments, the SSWD does not really outperform the SWD. The authors need to find applications with spherical data, where the advantages of SSWD could be shown clearly. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eFJBUYzZ1m", "reviewer": "Reviewer_ET9y", "summary": "The paper introduces a new formulation to compute the distance between probabilities distribution on the hyper-sphere, namely, the Spherical Sliced-Wasserstein (SSW). The proposed formulation involves a novel spherical Radon transform (to project to great circles) and the computation of the WD on the circle.\nThe method is validated on two tasks:  variational inference (matching a target distribution up to a scale) and autoencoders with a prior distribution on the spherical latent space.", "questions": "I would like the authors to address my previous two concerns:\n- trying to add an intuitive layer to the mathematical formulation (possibly with supporting figures)\n- expanding the analysis on the benefits of using a spherical distribution prior with autoencoder on datasets with different characteristics.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is quite difficult to follow in the mathematical formulation without the proper background. If possible, it would be helpful to add some intuitive explanation or figure to better understand the main steps of the proposed formulation. Nevertheless, the paper tackles an interesting problem, and being able to compare density distributions on a hyperspherical domain could be useful for some machine learning tasks, as shown with the (toy) applications variational inference and autoencoders.\n\nOn this latter point, I found the experimental setup not properly explained. Most of the details are demanded of the supplementary material, which makes it difficult to understand what was done without jumping forth and back. It would also be nice to see more examples and analyses on the spherical latent space of the autoencoders (for instance, the different behaviour between datasets for which the data manifold is known to be spherical (e.g. rotations) and datasets the data manifold is presumed to be more complex or hierarchical). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Rk714vC5SSK", "reviewer": "Reviewer_q6N2", "summary": "The paper introduces sliced Wasserstein discrepancies defined on the hypersphere $S^{d-1}$. \n\nFrom a computational perspective, this corresponds to computing an expected Wasserstein distance on great circles. \n\nFrom a theoretical perspective, the authors introduce a new spherical Radon transform that allows to formally define the proposed discrepancy. Further, this formal treatment allows to show that the proposed discrepancy is at least a pseudo-distance on the set of distributions on $S^{d-1}$ with finite $L_p$-norm.\n\nFinally, the authors conclude with various experiments ranging from simple gradient-flow examples to more sophisticated variational inference or auto encoder experiments.", "questions": "* Was the improved separation of the latent space for SSAWE consistent among seeds? (Figure 6)\n* What is the measure of variability in Figure 5? Standard deviation? Standard error? And how many intervals are visualized? Two times?", "limitations": "In general, the authors do not make exaggerated claims about applicability or potential of the method. In my opinion, only the statement in line 289/290 regarding improvement in the context of VI may be a bit exaggerated/unclear, as it is not indicated which variability measure is visualized in Figure 5 and further the measures overlap. Consequently, the authors could simply say that SSWVI performs just as well as SWVI.\n\nFinally, a common problem of sliced Wasserstein distances is that the required number of projections potentially grow exponentially with dimension (as e.g. argued for in the \"Generalized Sliced Wasserstein Distances\" paper by Kolouri et al.). The authors could mention this shortcoming and further link it to Figure 8b) in the appendix, as the reduction of $SSW_2^2$ distance with increasing dimension may indeed by explained by the aforementioned dimensionality issue of any sliced Wasserstein distance.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is overall well-written and easy to follow. The theoretical claims sound reasonable to me, although I am not a mathematician and not particularly specialized in the theory of Radon transforms. Finally, the experiments are insightful and demonstrate that the discrepancy measure behaves as expected.\n\nWhile the experiments do not show a strong performance increase over \"naively\" using the standard sliced Wasserstein distance, I still believe that the paper makes an important contribution as the introduced discrepancy is clearly more suited to the space $S^{d-1}$. Further, Figure 6 seems to show an improved class separation in the auto encoder experiment.\n\nI see the following points for further improving the paper:\n * The authors could more clearly highlight the reason why their paper is not just a particular instance of the generalized sliced Wasserstein distance (GSW) by Kolouri et al. If I understand correctly, GSW treats the \"projected particles\" as members of the real line, and hence cannot take the particular geometry of a circle into account. Or would it be possible to define the proposed discrepancy as a particular instance of a GSW distance?\n * Following up on the previous point, the authors could additionally compare to GSW in their experiments. If the proposed method can indeed not be defined as a GSW instance, the GSW approach would I guess correspond to an intermediate version between regular SW and SSW, which uses the projection onto the great circles but does not respect the circular nature of the projected space.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M1FZmheaQd", "reviewer": "Reviewer_yBhy", "summary": "The paper proposes spherical sliced Wasserstein discrepancy, a variant of sliced Wasserstein on the sphere. To construct the discrepancy, the authors develop a new variant of Radon Transform which is spherical Radon Transform, and utilize the closed-form solutions of the Wasserstein distance on the circle. In more detail, the spherical sliced Wasserstein is defined as the Wasserstein distance between geodesic projected measures on the great circles measured by the uniform distribution over all the projections or Stiefel manifold. On the application side, the authors apply the new discrepancy to hyperspherical auto-encoders and density estimation on the sphere.", "questions": "* One benefit of sliced Wasserstein is a good sample complexity which escapes the curse of dimensionality of the Wasserstein distance. Is this still hold for spherical sliced Wasserstein?\n* It seems that both spherical sliced Wasserstein and sliced Wasserstein need to use empirical samples when dealing with continuous measures in hyperspherical autoencoder. This can be seen as the usage of mini-batch OT [1,2,3,4,5]. Can we derive some special cases of spherical sliced Wasserstein where the Wasserstein distance between projected continuous measures has closed-form e.g, von Mises Fisher distribution?\n* One paper might be related [6].\n\nI will raise my score if the authors can add additional experiments on real datasets e.g., hyperspherical autoencoder on CIFAR10, CelebA, estimating density on Tiff dataset [7], and so on.\n\n[1] \"Learning with minibatch Wasserstein : asymptotic and gradient properties\" \n[2] \"Minibatch optimal transport distances; analysis and applications\"\n[3] \"Improving Mini-batch Optimal Transport via Partial Transportation\"\n[4] \"On Transportation of Mini-batches: A Hierarchical Approach\"\n[5] \"Unbalanced minibatch Optimal Transport; applications to Domain Adaptation\"\n[6] \"Improving Relational Regularized Autoencoders with Spherical Sliced Fused Gromov Wasserstein\"\n[7] \"https://sedac.ciesin.columbia.edu/data/set/gpw-v4-population-density-adjusted-to-2015-unwpp-country-totals-rev11/data-download#\"", "limitations": "The paper develops a fundamental tool hence there is no foreseen negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n## Originality \n* The paper proposes the first variant of sliced Wasserstein on the hypersphere.\n* Spherical Radon Transform is new.\n\n## Quality\n* The authors derive the metricity of spherical sliced Wasserstein, the connection of spherical Radon Transform and geodesic projection, and the kernel of spherical Radon Transform. \n\n## Clarity\n* The paper is well-written and easy to follow.\n\n## Significance\n* The contribution of the paper is significant for estimating densities and modeling on the hypersphere and for the community of sliced Wasserstein and optimal transport. \n* Spherical sliced Wasserstein is better than sliced Wasserstein on variational inference task on the sphere.\n\n# Weaknesses\n* The injectivity of spherical Radon Transform has not been established yet hence spherical sliced Wasserstein is not a metric.\n* The experiments are not intensive. The only experiment on real data in the main paper is hyper-spherical AE on MNIST. However, the improvement is not very significant compared to the conventional sliced Wasserstein. Other datasets such as CIFAR10 and CelebA should be considered. \n* The experiments on self-supervised learning should be moved to the main text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject