Improving Relational Regularized Autoencoders with Spherical Sliced Fused Gromov Wasserstein
Authors: Khai Nguyen
Son Nguyen
Nhat Ho
Tung Pham
Hung Bui
Venue: ICLR 2021 Poster (2021)
Keywords: Relational regularized autoencoder
deep generative model
sliced fused Gromov Wasserstein
spherical distributions

Abstract: Relational regularized autoencoder (RAE) is a framework to learn the distribution of data by minimizing a reconstruction loss together with a relational regularization on the prior of latent space. A recent attempt to reduce the inner discrepancy between the prior and aggregated posterior distributions is to incorporate sliced fused Gromov-Wasserstein (SFG) between these distributions. That approach has a weakness since it treats every slicing direction similarly, meanwhile several directions are not useful for the discriminative task. To improve the discrepancy and consequently the relational regularization, we propose a new relational discrepancy, named spherical sliced fused Gromov Wasserstein (SSFG), that can find an important area of projections characterized by a von Mises-Fisher distribution. Then, we introduce two variants of SSFG to improve its performance. The first variant, named mixture spherical sliced fused Gromov Wasserstein (MSSFG), replaces the vMF distribution by a mixture of von Mises-Fisher distributions to capture multiple important areas of directions that are far from each other. The second variant, named power spherical sliced fused Gromov Wasserstein (PSSFG), replaces the vMF distribution by a power spherical distribution to improve the sampling time of the vMF distribution in high dimension settings. We then apply the new discrepancies to the RAE framework to achieve its new variants. Finally, we conduct extensive experiments to show that the new autoencoders have favorable performance in learning latent manifold structure, image generation, and reconstruction.

Published as a conference paper at ICLR 2021
IMPROVING RELATIONAL REGULARIZED AUTOENCODERS WITH SPHERICAL SLICED FUSED GROMOV WASSERSTEIN
Khai Nguyen
Son Nguyen VinAI Research, Vietnam V. son3@vinni..is
Nhat Ho'
VinAI Research, Vietnam v. ww..hhhnniinaa..
University of Texas, Austin VinAI Research, Vietnam minhnhat unthhhat@utexas edu
Tung Pham
Hung Bui
VinAI Research, Vietnam v. Saangph4vinnae
VinAl Research, Vietnam ww..unnhhhee @eiraii. io
ABSTRACT
Relational regularized autoencoder (RAE) is a framework to learn the distribution of data by minimizing reconstruction loss together with relational regularization on the latent space. A recent attempt to reduce the inner discrepancy between the prior and aggregated posterior distributions is to incorporate sliced fused Gromov- Wasserstein (SFG) between these distributions. That approach has weakness since it treats every slicing direction similarly, meanwhile several directions are not useful for the discriminative task. To improve the discrepancy and consequently the relational regularization, we propose new relational discrepancy, named spherical sliced fused Gromov Wasserstein (SSFG), that can find an important area of projections characterized by a von Hises-Fisher distribution. Then, we introduce two variants of SSFG to improve its performance. The first variant, named mixture spherical sliced fused Gromov Wasserstein (MSSFG), replaces the vMF distribution by mixture of von Mises-Fisher distributions to capture multiple important areas of directions that are far from each other. The second variant, named power spherical sliced fused Gromov Wasserstein (PSSFG), replaces the vMF distribution by a power spherical distribution to improve the sampling time in high dimension settings. We then apply the new discrepancies to the RAE framework to achieve its new variants. Finally, we conduct extensive experiments show that the new proposed autoencoders have favorable performance learning latent manifold structure, image generation, and reconstruction.
INTRODUCTION
In recent years, autoencoders have been used widely as important frameworks in several machine learning and deep learning models, such as generative models (Kingma & Welling, 2013; Tolstikhin et al., 2018; Kolouri et al., 2018) and representation learning models (Tschannen et al., 2018). Formally, autoencoders consist two components, namely, an encoder and a decoder. The encoder denoted by Eg maps the data, which is presumably in a low dimensional manifold, to latent space. Then the data could be generated by sampling points from the latent space via a prior distribution p. then decoding those points by the decoder Gg. The decoder is formally function from latent space to the data space and it induces distribution PG, on the data space. generative modeling, the major task is to obtain a decoder Gg- such that its induced distribution PG,, and the data distribution are very close under some discrepancies. Two popular instances of autoencoders are the variational autoencoder (VAE) (Kingma & Welling, 2013), which uses KL divergence, and the Wasserstein autoencoder (WAE) (Tolstikhin et al., 2018), which chooses the Wasserstein distance (Villani, 2008) as the discrepancy between the induced distribution and the data distribution.
'The work finished when Nhat Ho worked VinAI Research in the summer of 2020.

---

Published as a conference paper at ICLR 2021
In order to implement the WAE, a relaxed version was introduced by removing the constraint on the prior and the aggregated posterior (latent code distribution). In particular, a chosen discrepancy between these distributions is added to the objective function and plays role as regularization term. With that relaxation approach, the WAE becomes a flexible framework for customized choices of the discrepancies (Patrini al., 2020; Kolouri al., 2018). However, the WAE suffers either from the over-regularization problem when the prior distribution is too simple (Dai & Wipf, 2018; Ghosh et al., 2019), which is usually chosen be isotropic Gaussian, or from the under-regularizzttoo problem when learning an expressive prior distribution jointly with the autoencoder without additional regularization, e.g., structural regularization (Xu al., 2020). In order to eircumvent these issues of WAE, relational regularized autoencoder (RAE) was proposed in (Xu et al., 2020) with two major changes. The first change is to use a mixture of Gaussian distributions as the prior while the second change is to set regularization on the structural difference between the prior and the aggregated posterior distribution, which is called the relational regularization. The statc-of-ther RAE, deterministic relational regularized autoencoder (DRAE), utilizes the sliced fused Gromov Wasserstein (SFG) (Xu et al., 2020) as the relational regularization. Although DRAE performs well in practice and has good computational complexity (Xu et al., 2020), the SFG does not fully exploit the benefits of relational regularization due to its slicing drawbacks. Similar to sliced Wasserstein (SW) (Bonnotte, 2013; Bonneel et al., 2015) and sliced Gromov Wasserstein (SG) (Vayer et al., 2019), SFG uses the uniform distribution over the unit sphere to sample projecting directions. However, that leads to the underestimation of the discrepancy between two target distributions (Deshpande et al., 2019; Kolouri et al., 2019) since many unimportant directions are included in that estimation. A potential solution by using only the best Dirac measure over the unit sphere to sample projecting directions in SFG, which was employed in max-sliced Wasserstein distance Deshpande et al. (2019). However, this approach focuses on the discrepancy of the target probability measures based on only one important direction while other important directions are not considered. As one alternative solution, authors in (Nguyen et al., 2021) proposed the distributional slicing approach which is a general technique to design probabilistic way to select important directions.
version of
Our contributions. To improve the effectiveness of the relational regularization in the autoencoder framework, we propose novel sliced relational discrepancies between the prior and the aggregated posterior. The new sliced discrepancies utilize von Mises-Fisher distribution and its variants instead of the uniform distribution the distributions over slices. An advantage of the vMF distribution and its variants is that they could interpolate between the Dirac measure and uniform measure, thereby improving the quality of the projections sampled from these measures and overcoming the weaknesses of both the SFG and its max version max-aaa--- MFG. In summary, our major contributions are as follows:
1. First, we propose a novel discrepancy, named spherical sliced fused Gromov Wassersetein (SSFG). This discrepancy utilizes vMF distribution as the slicing distribution to focus on the area of directions that can separate the target probability measures on the projected space. Moreover, we show that SSFG is well-defined pseudo-metric on the probability space and does not suffer from the curse of dimensionality for the inference purpose. With favorable theoretical properties of SSFG, we apply it to the RAE framework and obtain variant of RAE, named spherical deterministic RAE (sDDR..).
2. Second, we propose an extension of SSFG to mixture SSFG (MSSFG) where we utilize a mixture of vMF distributions as the slicing distribution (see Appendix C for the details). Comparing the SSFG, the MSSFG is able to simultaneously search for multiple areas of important directions, thereby capturing more important directions that could be far from each other. Based on the MSSFG, we then propose another variant of RAE, named mixture spherical deterministic RAE (RA-D.
3. Third, to improve the sampling time and stability of vMF distribution in high dimension settings, we introduce another variant of SSFG, named power SSFG (PSSFG), which uses power spherical distribution instead of the vMF distribution the slicing distribution. Then, we apply the PSSFG to the RAE framework to obtain the power spherical deterministic RAE (RS-DR
4. Finally, we carry out extensive experiments on standard datasets to show that proposed autoencoders achieve the best generative quality among well- autoencoders, including the state-of-the-as RAE-DRAE. Furthermore, the experiments indicate that the s- D-DRAE, MS-DRAE, and PS-DRAE can learn nice latent manifold structure, a good mixture of
2

---

Published as a conference paper at ICLR 2021
Gaussian prior which can cover well the latent manifold, and provide more stable results in both generation and reconstruction than DRAE.
We note in passing that the proposed sliced-fused Gromov-Wassersee Wasserstein divergences are not just applicable the applications within the RAE framework. In Appendix E.6, we provide initial experiments to show that these divergences can be used to improve over other fliced-baset distances in color transfer applications. We leave a thorough extension of the proposed sliced-fuse fusedd discrepancies to other upplications in the future work.
Organization. The remainder of the paper is organized as follows. In Section 2, we provide backgrounds for DRAE and vMF distribution. In Section 3, we propose the spherical sliced fused Gromov Wasserstein and its extension. We then apply these spherical discrepancies to the relational regularized autoencoder. Extensive experiment results are presented in Section 4 followed by conclusion in Section 5. Proofs of key results and extra materials are in the supplementary material.
Notation: Let gdd1 be the d-dimensional hypersphere and 11(5-1) be the uniform distribution on 8*-1. For a metric space (X', dd],, we denote by P(2)) the space of probability distributions over X with finite moments. We say that d1 is a pseudo-metric in space X' if it is non-negative, symmetric, and satisfies the inequality: di (Ir, 2) I [d (I, y) + di (v, for universal constant C > 0 and for all æ, y, E X. For any distribution μ and v. 11 1/) is the set of all transport plans between and v. For x E IR4, denote 62 to be the Dirae measure at x. For any I E g and any measure μ., 0tu denotes the pushforward measure of through the mapping Rg where Ro(x) = f x for all x.
2 BACKGROUND
In this section, we provide backgrounds for the sliced fused Gromov Wasserstein and the relational regularized autoencoders. Then, we give backgrounds for the von Mises-Fisher distribution.
2.1 DETERMINISTIC RELATIONAL REGULARIZED AUTOENCODER AND SLICED FUSED GROMOV WASSERSTEIN
First, we review the WAE framework (Tolstikhin et al.. 2018), which is used to learn a generative model by minimizing relaxed version of Wasserstein distance (Villani, 2008) between data distribu- tion pa(2) and model distribution po (x) == (2), where p(z) is a noise distribution. The model aims to find the autoencoder which solves the following objective function:
min ( (oo [d(z, (=)) + AD(go())llp()),, 0.0
where d is the ground metric of Wasserstein distance, D is a discrepancy between distributions, and (2(x)))) is distribution for encoder Es X + Z. parameterized by 6. Due to the efficiency in training generative models, several autoencoder models are derived from this framework. For example, WAE uses standard Gaussian distribution for p(2) and chooses D to be either maximum mean discrepancy (MMD) or GAN. Later, by using sliced Wasserstein distance (Bonneel et al., 2015) for D, Kolouri et al. (2018) achieved another type of autoencoder, which is called SWAE
Deterministic relational regularized autoencoder (DRAE); In DRAE, Xu et al. (2020) parametrizes the prior mixture of Gaussians E (2)) and makes it learnable. Addi- tionally, they introduce the sliced fused Gromov Wasserstein as the discrepancy between the posterior and the prior distributions.
Definition 1. (SFG) Let n. P(R2) be two probability distributions, 3 be constant in (0, 11. and d1 IR R4 be a pseudo-metric on R. The sliced fused Gromov Wasserstein (SFG) between and is defined as;
SFG(H, v; B) (o) fg (OiHe 0tu; 8, 8,d1)]
where the fused Gromov Wasserstein D Digoo is given by:
{ {1-33) x d= x, 0T ())
Digo (0t,, 0tv; B, d1)
min TEL(CEH.OFO)
y, gT //)) dr v) ( 11).. (3)
+ 3
(d](0 x, TT x') d1 (0T
(Ruys
3

---

Published as a conference paper at ICLR 2021
Given the definition of SFG, the objective function of the deterministic relational regularized autoen- coder (DRAE) takes the following form:
min 0.0-M1ick Pisk
[d( Go(z))) + ( (2) (=))
(4)
where iN (2) and (x are the empirical distributions of (o(2) and Pania respectively.
Properties of SFG: From equation (2), SFG is linear combination of sliced Wasserstein (SW) and sliced Gromov Wasserstein (SG). In particular, SFG becomes SW and SG when 3 = 0 and respectively. Hence SFG is able take advantages of both of them. If μ and have n supports and uniform weights and (x. y) = (I - y)? SFG has computational complexity of the order Onglogg) log It is because under d1, both SW and SG have dlosed-form expressions (Vayer et al., 2019; Bonnotte, 2013) where the optimal transport map in fiw (Vayer et al., 2018) can be obtained by sorting the projected supports of 4 and
Limitation of SFG: The major limitation of SFG is that the outer expectation with respect to P~ ((($=1) in SFG is generally intractable. In practice, projections from the unit sphere are uniformly sampled and we then apply the Monte Carlo method to obtain approximate of that expectation. However, the difference between two distributions certainly not distributed uniformly, meaning that informative directions mixed up with many non-informative ones. Hence, sampling blindly slices in high dimensional space not only is ineffective but also underestimates the discrepancy between the two distributions. The Mises-Fisher (vMF) distribution provides way to have concentrated weight on the most important directions and assigns less weight to further directions. Therefore, we gain better representation of the discrepancy between probability measures.
2.2 VON MISES-FISHER DISTRIBUTION
Now, we review the definition of the von Mises-Fisher distribution.
Definition 2. The von Mises-Fisher distribution (vMF) is a probability distribution on the unit sphere gd-1 where its density function is given by (Jupp et al., 1979):
[(xle, == Ca(is) exp(neT=x) x)).
(5)
where 2 0 is the concentration parameter, E gd-1 is the location vector, and Ca(R) x4/2-1 (2m) (2m) 166) the normalization constant. Here, Ta/2-1 is the modified Bessel function of the first kind at order d/2 (Temme, 2011).
The vMF concentrates around mode and its density decreases when æ goes away from e. When - vMF converges the uniform distribution, and when - oc, vMF approaches the Dirac distribution centered at e (Sra, 2016). These properties are illustrated by toy example in Figure 3 in Appendix B.I.
SPHERICAL SLICED FUSED GROMOV WASSERSTEIN AND ITS RELATIONAL REGULARIZED AUTOENCODER
In this section, we introduce a novel discrepancy, named spherical sliced fused Gromov Wasserstein (SSFG), that searches for the best vMF distribution which distributes more masses to the most important area projections on the unit sphere $d-1 Then, we discuss an application of SSFG to the relational regularized autoencoder framework.
3.1 SPHERICAL SLICED FUSED GROMOV WASSERSTEIN
We first start with a definition of spherical sliced fused Gromov Wasserstein.
Definition (SSFG) Let μ,v P(Rd) be two probability distributions, K 0, E 10 d1 : R X IR + R., be pseudo-metrte mettoo-meriicooo on R. The spherical sliced fused Gromov Wasserstein (SSFG) between μ and v is defined as follows:
VSFG(u,V, == max Eg--MF(()c,s)) [D (0tgu00p,,1 0tv; , 3, dz)]. cEBA-1
(6)
where the fused Gromov Wasserstein Digu n is defined equation (3).
4

---

Published as a conference paper at ICLR 2021
p(z)
Sampling
Reletional Regulerization
Encoder
q(z)
Pused Wassersian
Japooao
Reconstructin
Figure 1: Visualization of the spherical deterministic relational regularized autoencoder (s- (s-DRAE).
A few comments on the SSFG are in order. The family of vMF distributions is controlled by two parameters e and K, where e the mode of the vMF distribution while controls its concentration. By changing K from 0 to infinity, the vMF family could interpolate from the uniform distribution to any Dirac distribution on the sphere. other words, it allows us to control distributing weight to the most important direction and other directions based on the geodesic distance the sphere. Optimizing over the family of vMF distributions helps us to identify where the best direction is as well as how much weight need to put there in comparison with other less important directions.
Sampling procedure and reparameterization trick with the vMF: To generate samples from vMF, follow the procedure in (Ulrich, 1984), which is described in Algorithm 1 in Appendix B. Note that this procedure does not suffer from the curse of dimensionality. Furthermore, to compute integral with respect to the vMF distribution, we use the reparametrization scheme in (Naesseth et al., 2017), which was extended for vMF in Lemma 2 in (Davidson et al., 2018). Finally, Davidson et al. (2018) proved that samples from Algorithm 1 (cf. Appendix 2) can provide differentiable estimator for the parameters of vMF distribution. More details of this scheme are given in Appendix B.3.
Complexity of computing SSFG: Let μ and be two discrete distributions that have n supports with uniform weights. For the general case of d1, similar to SFG, the complexity of computing SSFG can be expensive (at least the order (nn²) as the fused Gromov Wasserstein D a quadratic programming problem). However, the complexity of SSFG can be greatly improved under specific choices of dj. For example, when dj (z, y) = (x - yy)² with a similar argument to the SFG case, the SSFG has computational complexity of the order (n log n))
Key properties of SSFG: We first prove that SSFG is pseudo-metric on the probability space. Theorem 1. For any 8 [0, 1] and > 0, SSFG(,, B, k)) is pseudo-metric in the space of probability measures, namely, it is non-negative, symmetric, and satisfies the weak triangle inequality.
The proof of Theorem is in Appendix A.I. Our next result establishes relations between SSFG and SEG and the max version of SFG, named as max-SFG.
Theorem 2. For any probability measures μ,, v E ((RT)), the following holds: (a)
lim SSFG(u, v3 k) = SFG( vz B)
K-0
lim USFG(B, a-)= max K000
Dfg (01H, 02u; = max--FFG((L 3: 0E94-1
(b) For any K. > O, we find that exp( (xp(-Aa]Ca(e)SFF(uu.. S USFG(U,U;B,,g (xp(x))a v; SSFG(H, B, S max-SFG(y,,;
The proof of Theorem 2 is in Appendix A.2. Theorem shows that SSFG is an interpolation between SFG and max-SFG, namely, it combines the properties of both SFG and max-SFG. Furthermore, the result of part (b) of Theorem 2 indicates that SSFG is strongly equivalent to SFG.
Our next result shows that SSFG does not suffer from the curse of dimensionality for the inference purpose under certain choices of d, Therefore, it will be a statistically efficient discrepancy to compare the prior distribution to the encoder distribution in the DRAE framework.
5

---

Published as a conference paper at ICLR 2021
Theorem 3. Assume that is probability measure supported compact subset ④ C R Let
X1,.... Nx, be i..dd. data from P and d1 y) = yl" for a positive integer r. We denote by 1
Un " >>== ôx, the empirical measure of the data points XI...., Xn. Then, for any 3 E [0, 1] and K. > 0, there exists constant depending only on r and diameter of Q such that
E SSFG(un M4; 1;; 3,
c
72
Theorem together with the earlier argument about the computational complexity of SSFG suggests that the choice of d1 (I. y) (z y)² is not only convenient for the computation but also statistically efficient. Therefore, we will specifically use this choice of d, in our experiments in Section 4.
Spherical deterministic relational regularized autoencoder: We replace SFG by SSFG in the deterministic relational regularized autoencoder framework in equation (4) to obtain a new variant of DRAE with a stronger relational regularization. The new autoencoder is named as spherical deterministic relational regularized autoencoder (s-DRAE), Intuitive visualization of s-DRAE is presented in Figure 1. The detailed training procedure for s-DRAE is left in Appendix B.5.
3.2 EXTENSIONS AND VARIANTS OF SSFG
We first propose extension of SSFG to its mixture variant.
Definition 4. (MSSFG) Let μr, v E ((R))) be two probability distributions, B E [0, 1] be constant, {os) 1k be given mixture weights, and { (k:) be given mixture concentration parameters where 1. Furthermore, let di : IR x R - R be a pseudo-metric on R. Then, the mixture spherical sliced fused Gromov Wasserstein (MSSFG) between μ and is defined as follows:
MSSFO v2, (Ro (a))))
Mo--MoMPt das)t [D (0tye, 0dv; 3. d.di),,
max 61
where Digw is defined in equation (3) and the mixture of vMF distributions is defined as MovMF(-Jeras (Ri)i (os)k=))
E a.NMF(-le; K.).
Comparison between MSSFG and SSEG: When h: = 1, the MSSFG becomes SSFG. Recall that SSFG tries to search for the best location parameter in the unit sphere gd-1 that maximizes the expected value of the fused Gromov Wasserstein between the projected probability measures. Intuitively, it places a large weight on the best projection and some weights on other important projections. However, if these important projections are far from the best projection, i.e., the center of the best von Mises Fisher distribution, their weights will be very small, which can be undesirable. To account for this issue, the mixture of von Mises-Fiskee Fisher distributions aims to find & best location parameters whose weights are guaranteed to be large enough. Furthermore, when k: is chosen to be sufficiently large, mixture of von Mises-Fisher distributions will give a good coverage of the unit sphere; therefore, the important directions that MSSFG can find will be able to reflect more accurate differences between the target probability distributions than those from SSFG.
Properties of MSSFG and its DRAE version: As SSFG, MSSFG is pseudo-metric in the proba- bility space and does not suffer from the curse of dimensionality. Its computational complexity is of the order (n log r2) when μ and y are discrete measures with n atoms and uniform weights and (z, y) (z y)² The detailed discussion of the properties of MSSFG is in Appendix C. An application of MSSFG to the DRAE framework leads to the mixture spherical DRAE (ms-DRAF).
Improving computational time of (MSSFG: Drawing the samples from the vMF distribution and its mixtures can be slow in high dimension settings, which affects the computation of (M)SSFG. To account for this issue, we propose using power spherical distribution (De Cao Aziz, 2020) instead of vMF and its mixtures as the slicing distribution to improve the computational time of (M)SSFG. It leads to a new discrepancy, named power SSFG (PSSFG), between the probability distributions (see Appendix D for the definition). In Section 4, we show that (M)PSSFG has better computational time than (M)SSFG while its DRAE version, named (mixture) power spherical DRAE ((m)ps-DR DRAE), has comparable performance (m)s-DRAF.

---

Published as a conference paper at ICLR 2021
Table 1: FID scores and reconstruction losses of different autoencoders. (*) denotes the results that taken from (Xu et al., 2020) due to the reproducing failure. The results are taken from different runs.
MNIST
CelebA
Method
FID
Reconstruction
FID
Reconstruction
VAE
71.55 + 26.65
18.59 = 2.22
5*.99(*)
96.36(*)
GMVAE
75.68 111.9
18.19 = 0.14
212.59 18.15
97.77 0.19
Vampprior
138.03 = 34.09
29.98 4.09
PRAE
100.25 +41.72
16.20 +3.14
52.20 (*)
63.21(*)
WAE
80.77
11.53 +0.33
52.07 (*)
63.83(*)
SWAE
80.28 #19.22
86.53 + 2.49
89.71#2.15
DRAE
58.04 20.74
14.07 4431
50.09 +1.33
66.05 + 2.56
m-DRAE (ours)
52.92
13.13 0.33
49.05 0.93
66.30 0.22
s- S-DRAE (ours)
47.97 13.83
11.17
46.63 +0.83
66.62 0.51
ps- Ds-DRAE (ours)
49.15 12.93
11.71 #1.21
41.01
66.31 0.43
0.43
mps- -DAE (ours)
44.67 19.98
11.01 #1.32
46.61 + 1.01
66.23 = 0.56
ms-DRAE (ours)
43.57 10.98
11.12 +0.91
46.01 0.91
65.91 0.4
4 EXPERIMENTS
In this section, we conduct extensive experiments on MNIST (LeCun et al., 1998) and CelebA datasets (Liu et al., 2015) to evaluate the performance of S-DRAF, pS-DRAE and m(p)s-DRAE with various autoencoders, including DRAE (trained by SFG), PRAE (Xu et al., 2020), m-DRAE (trained by max- SFG-see its definition in Theorem 2). VAE (Kingma& & Welling, 2013), WAE (Tolstikhin et al., 2018), SWAE (Kolouri et al., 2018), GMVAE (Dilokthanakul et al., 2016), and the VampPrior (Tomezak & Welling, 2018). We use two standard scores as evaluation metrics: (i) the Frechet Inception distance (FID) score (Heusel et al., 2017) is used measure the generative ability; (ii) the reconstruction score is used to evaluate the reconstruction performance computed on the test set. For the computational details of the FID score, compute the score between 10000 randomly generated samples and all samples from the test set of each dataset. To guarantee the fairness of the comparison, we use the same autoencoder architecture, Adam optimizer with learning rate 0.001, 31 0.5 and (32 0.999; batch size 100; latent size 8 on MNIST and 64 on CelebA; coefficient >=1; fused parameter 0.1. We set the number of components K = 10 for autoencoder with a mixture of Gaussian distribution as the prior. More detailed descriptions of these settings are in Appendix F. Comparing with other autoencoders: We first report the performances of autoencoders on MNIST (LeCun et al., 1998) and CelebA datasets (Liu et al., 2015). Table 1 presents the FID scores and reconstruction losses of trained autoencoders. All results are obtained from five different runs and reported with empirical mean and standard deviation. On the MNIST dataset, ms ms- DRAE achieves the lowest scores in both FID score and reconstruction loss among all the autoencoders. (m)ps-DRAB also have better scores than DRAF. On the CelebA dataset, we cannot reproduce results from VAF, PRAF, and WAE; therefore, we use the results with these autoencoders from DRAE paper (Xu al., 2020). Table suggests that ms ms-DRAE also obtains the lowest mean and standard deviation in FID score than other autoencoders, meanwhile its reconstruction loss is almost the same as other DRAES. The FID scores of S-DRAE and ps-DRAE are also better than those of DRAE. These results suggest that the proposed spherical discrepancies truly improve the performances of the DRAE framework. In these experiments, we set the number of projections = 50 for every dicceddiiccrepacc.. For S-DRAF, PS-DRAE and m(p)s-DRAF (10 vMF components with uniform weights and same concentration parameters), we search for x E (1,5, 10, 50, 100) which gives the best FID score on the validation set of the corresponding dataset. By tuning K. find that the performance of both S-DRAE and ps-DRAE is close to that of DRAE when k E {0.001, 0.01, 0.1, 1}. namely, the reconstruction loss and FID score are nearly equal to the scores of DRAE. On the other extreme, when x = 100, s- s-DRAE and DS-DRAE behave like m-DRAE in both evaluation metrics. Further details are given in Figures 12 and 15 in Appendices E. and E.3 respectively.
addition, s-DRAE and
Detailed results including generated images, reconstruction images and visualizing latent spaces are in Appendix 1. These results indicate that S-DRAE, DS-D,,, and ms-DRAE can learn nice latent
7

---

Published as a conference paper at ICLR 2021
MNIST
DRAE MHRRK
DRAE THORKE CRAE PS DRAE DRAE
95
5 FRREE DS-DRAE
42-
54
MEN-DRAE
TIENDRAE
F1D scote
FID score
64-
Computational time per minibatch (5)
Computational per minibatch
Each dot represents the computational time minibatch and FID score. For DRAE, we vary the number of projections L E {1, 10, 20, 50, 100, 200, 500, 1000]; for s-DRAE set I (1,5, 10, 20, 50, 100); for ps-DRAE set x. 50, L E {1,5, 10, 20, 50, 100); and for m(p)s- DRAE we set = 50, the number of vMF components & {2, 5, 10, 50} (for each k; we find the best E (1,5, 10, 50, 100)).
Figure 2:
10.
structures and mixture Gaussian priors which can cover well these latent spaces. As a consequence, the spherical DRAES can produce good generated images and reconstruct images correctly.
Detailed comparisons among deterministic RAEs: It is well-known that the quality and computa- tional time of diceeediiscrsssniies depend on the number of projections (Kolouri al., 2018; 2019; Deshpande et al., 2019). Therefore, we carry out experiments on MNIST and CelebA datasets to compare Ps-DRAF, mpp)S-DRAE, S-DRAE to DRAE in a wide range the of number of projections. In detail, we set the number of projections L. E {1, 5, 10, 20, 50, 100, 200, 500, 1000} in DRAE; and L E [1,5, 5, 10, 20, 50, 100} for S-DRAE (K = 10), and ps- DS-DRAE (x = 50). We then report the (minibatch) computation time and FID score of these autoencoders in Figure 2. In this figure, we also plot the time and FID scores of m-DRAE and m(p)s-DRAE (using = 50 and the number of vMF components & E {2, 5, 10, 50}. On the MNIST dataset, with only projection, s-DRAE achieves lower FID score than all settings of DRAE; however, S-DRAE requires more time to train due to the sampling of vMF and its optimization problem. Having faster sampling procedure of PS distribution, PS-DRAE has better computational time than S-DRAF while it still has a comparable performance to S-DRAE. With the computational time greater than about 0s21 ps-DRAE always produces lower FID score than DRAE. We also observe the same phenomenon on CelebA dataset, namely, PS-DRAE and S-DRAE have lower FID score than DRAE with any value of L but L Between S-DRAE and PS-DRAE, s-DRAE gives better results but ps- DRAE is faster in training. On both the datasets, m-DRAE has faster speed than (p)s-DRAE but its FID score is higher. In terms of the FID score, ms-DRAE is the best autoencoder though it can be more expensive in training. Moreover, mps-DRAE has more efficient computational time than ms- mS-DRAE while its FID scores are comparable. Finally, observe that increasing the number of components in m(p)s-DRA) DRAE enhance the FID score but also worsen the computational speed.
We provide further comparisons between MSSFG, PSSFG, SSFG and SFG in ex-post density estimation of autoencoder (Ghosh et al., 2019) and GAN (Goodfellow et al., 2014) applications in Appendices E.4 and E.5. In the ex- ex-post density estimation framework, find that MSSFG, PSSFG and SSFG give better FID score than SFG. Like traditional training procedures, MSSFG achieves the best performance in this task. In the GAN application, we use toy example, which is to learn generator to produce 4 Gaussian modes. We observe that MSSFG, PSSFG and SSFG help the model distribution converge faster than SFG does. Finally, we also apply the proposed diceeddiicrepanciess to image color adaption (Rabin et al., 2014; Bonneel et al., 2015; Perrot et al., 2016) in Appendix E.6, where we find that using (MyMF, (M)PS distributions sample projecting directions can improve the performance of the sliced-basee color adaption algorithms (Rabin et al., 2010; Bonneel et al., 2015; Muzellec & Cuturi, 2019).
Additional experiments:
5 CONCLUSION
In the paper, we first introduced new spherical relational discrepancy, named spherical sliced fused Gromov Wasserstein (SFFG), between the probability measures. This discrepancy is obtained by
8

---

Published as a conference paper at ICLR 2021
replacing the uniform distribution over slicing direction in sliced fused Gromov Wasserstein by von Misee-Fisher distribution that can cover the most informative area of directions. To improve the performance and stability of SFFG, then propose two variants of SSFG: (i) the first variant mixture SSFG (MSSFG), obtained by using a mixture vMF distributions instead of a single vMF distribution to capture more informative areas of directions; (ii) the second variant is power SSFG (PSSFG), obtained by replacing the vMF distribution by the power spherical distribution to improve the sampling time of vMF distribution in high dimension settings. An application of these discrepancies to the DRAE framework leads to several new variants of DRAE. Extensive experiments show that these new autoencoders are more stable and achieve better generative performance than the previous autoencoders, including DRAE, in comparable computational time.

---

Published as a conference paper at ICLR 2021
REFERENCES
Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, and Suvrit Sra. Clustering on the unit hypersphere using von Mises-Fisher distributions. Journal of Machine Learning Research, 6(Sep): 13355-382, 2005.
Sergey Bobkov and Michel Ledoux. Ane-dimensional empirical measures, order statistics, and Kantorovich transport distances. Memoirs of the American Mathematical Sociery, 2019.
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and Radon Wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 5 51(1):22 45, 2015.
Nicolas Bonnotte. Unidimensional and evolution methods for optimal transportation. PhD thesis, Paris 11, 2013.
Charlotte Bunne, David Alvarez-Melis, Andreas Krause, and Stefanie Jegelka. Learning generative models across incomparable spaces. In International Conference on Machine Learning. 2019.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. In International Conference Learning Representations, 2018.
Tim R Davidson, Luca Falorsi, Nicola De Cao, and Thomas Kipf Jakub M Tomezak. Hyperspherical variational auto-encoders. Conference on Uncertainty Artificial Intelligence (UAI), 2018.
Nicola De Cao and Wilker Aziz. The power spherical distribution. arXiv preprint arXiv:2006.04437, 2020.
Ishan Deshpande, Yuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander G Schwing. Max-sliced W Wasserstein distance and its use for GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10648-10656,, 2019.
Nat Dilokthanakul, Pedro AM Mediano, Marta Gamelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with Gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.026455 2016.
William Feller. Introduction to Probability Theory and its Applications II. Wiley, New York, 1966.
Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterization gradients. In Advances in Neural Information Processing Systems, pp. 441-452, 2018.
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From variational to deterministic autoencoders. In International Conference on Learning Representations, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2680, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, pp. 6626 067666667,440 2017.
Peter E Jupp, Kanti V Mardia, et al. Maximum likelihood estimators for the matrix von Mises-Fisher and Bingham distributions. The Annals of Statistics, 7(3):599-606, 1979.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 14X2::/4/2.6980 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint 131iv: 14,61114110000 2013.
Soheil Kolouri, Phillip E Pope, Charles Martin, and Gustavo K Rohde. Sliced Wasserstein auto-encoders. In International Conference on Learning Representations, 2018.
10

---

Published as a conference paper at ICLR 2021
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. Generalized sliced Wasserstein distances. Advances in Neural Information Processing Systems, pp. 261-272, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2222 1998.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Boris Muzellec and Marco Cuturi. Subspace detours: Building transport plans that are optimal on subspace projections. In Advances in Neural Information Processing Systems, pp. 6917-6928,
2019.
Christian Naesseth, Francisco Ruiz, Scott Linderman, and David Blei. Reparameterization gradients through acceptance-rejection sampling algorithms. In Arlificial Intelligence and Statistics, pp. 489-498, 2017.
Khai Nguyen, Nhat Ho. Tung Pham, and Hung Bui. Distributional siiced-wasserstein and applications to generative modeling. In International Conference on Learning Representations, 2021. URL httts:://pperrevv//
net fentt/forum?ee Lorum?1d=QX
Giorgio Patrini, Rianne van den Berg, Patrick Forre, Marcello Carioni, Samarth Bhargav, Max Welling, Tim Genewein, and Frank Nielsen. Sinkhorn autoencoders. In Uncertainty in Artificial Intelligence, pp- 733-743. PMLR, 2020.
Michaël Perrot, Nicolas Courty, Rémi Flamary, and Amaury Habrard. Mapping estimation for discrete optimal transport. Advances in Neural Information Processing Systems, 29:4197-4205, 2016.
Julien Rabin, Julie Delon, and Yann Gousseau. Regularization of transportation maps for color and contrast transfer. In 2010 IEEE International Conference on Image Processing, pp. 1933-1936. IEEE, 2010.
Julien Rabin, Sira Ferradans, and Nicolas Papadakis. Adaptive color transfer with relaxed optimal transport. In 2014 IEEE International Conference Image Processing (ICIP), pp. 4852-4856.
IEEE, 2014.
Suvrit Sra. Directional statistics in machine learning: brief review. arXiv preprint arXiv:1605.0031 2016.
Nico M Temme. Special functions: An introduction to the classical functions of mathematical physics. John Wiley & Sons, 2011.
llya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018.
Jakub Tomezak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics, pp. 1214-1223, 2018.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in uutoencoder-based representation learning. arXiv preprint 181200099, 2018.
Gary Ulrich. Computer generation of distributions on the m-sphere. Journal of the Royal Statistical Soccitt: Series C (Applied Statistics), 33(2):158-1633 1984.
Titouan Vayer, Laetitia Chapel, Rémi Flamary, Romain Tavenard, and Nicolas Courty. Optimal transport for structured data with application graphs. arXiv preprint 1rXiv:1805.09114 2018.
Titouan Vayer, Rémi Flamary, Romain Tavenard, Laetitia Chapel, and Nicolas Courty. Sliced Gromov-Wasserster Wasserstein. Advances in Neural Information Processing Systems, 2019.
C. Villani. Optimal transport: Old and New. Springer, 2008
Cédric Villani. Topics in Optimal Transportation. American Mathematical Society, 2003.
Hongteng Xu, Dixin Luo, Ricardo Henao, Svati Shah, and Lawrence Carin. Learning autoencoders with relational regularization. arXiv preprint arXiv:2002.02913, 2020.


Reviews:
[{"review_id": "RImzX-0E-oh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new pseudo-distance called the spherical slices fused Gromov Wasserstein distance (SSFGW). It builds on top of the slices fused Gromov Wasserstein distance (SFGW) that takes a weighted combination of the Wasserstein distance and the Gromov Wasserstein distance on sliced spaces. The paper tackles the problem of solving for the best sampling directions of slicing. Existing approaches either assume uniform sampling strategies or a single direction that maximizes the discrepancy of two measures. This paper uses von Mises-Fisher measures (vMF) as the bridge to combine the advantages of the two. Changing the parameters of vMF is equivalent to (non-linearly) interpolating between a Dirac impulse and a uniform distribution, which converges to uniform-sliced Wasserstein and max-sliced Wasserstein, respectively. The authors provided the proof of its pseudo metric properties and upper bound and extended their SSFGW to mixtures of vFW, creating MSSFGW, and applied these variants to deterministic relational regularized autoencoder (DRAE). The results of comparing the DRAE equipped with proposed distance with other DRAEs demonstrate its superiority in stability and generative capacity.\n\nSome comments and questions:\n\n-- Where does von Mises-Fisher distribution come from? \n\nIf I understand it correctly, von Mises-Fisher is a variant of Gaussian on hyperspheres. The authors can consider citing existing works or proving some arguments of choosing vMF as an interpolation of a uniform distribution and a Dirac.\n\n-- Please refine 2 Background.\n\nDefinition 2: is $I_v$ actually $I_d$ in $C_d(k)$?\n\n\n\"The vMF distribution provides a way to have concentrated weight on the most important directions and assigns less weight to further directions. Therefore, we gain a better representation of the discrepancy between probability measures.\" \n\nSwap SFG and DRAE in the title of 2.1 to consist with the content of 2.1.\n\n\n-- The introduction of power SSFG at the end of Section 3 seems abrupt.\nIf power SSFG performs better than SSFG, then what is the point of introducing vMF? Can we build mixtures of power SSFG? The paper does not mention an mps-DRAE.\nOn page 20 of the Appendix, at the top, it seems power SSFG is superior to SSFG in all desired aspects, but ps-DRAE underperforms s-DRAE, as shown in Table 1. Any explanation?\n\n\n-- Empirical evidence of interpolating between a uniform and a Dirac\nIn Figure 12, abd, the graphs of s-DRAE cross the red baseline with a large momentum when $k$ decreases to 1, which suggests that it could go far away from the baseline when $k \\rightarrow 0$ because when $k=1$, vFM is still very far from a uniform distribution. Please explain that.\n\n-- A Wasserstein paper or a generative modeling paper?\nThe main contribution of the paper, which tackles the problem of generalizing sliced Wasserstein distance and max-sliced Wasserstein distance, is on the new pseudo metric but the paper only argues the contribution from the perspective of its power in generative modeling. I wonder if the authors ever considered evaluating the metric without add-ons like a neural network. Some typical areas of applications of the Wasserstein distances like color transfers, shape interpolation, rigid transformation since it is Gromov Wasserstein, etc? Or is the evaluation without an AE setup not necessary?\n\n---------------------------\n--AFTER REBUTTAL--\n---------------------------\nI appreciate the authors response. Most of my trivial comments and questions have been resolved. I stand by my initial rating. This is a solid paper. The authors clearly introduce the problem and develop a clear story with a straightforward solution. The paper ends with extensive experiments. My main concern remains: the contributions of the paper to the ML community is moderate because the story is very narrow. I think the idea can be substantially extended to solving the fundamental problems in sliced Wasserstein distances but I don't object acceptance of the paper in the current form. I recommend the authors incorporate suggestions from all the reviewers and polish the language especially Section 2 to make the paper more accessible for readers outside the sliced Wasserstein community. Thank you.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KtK7MY8z6Pf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds on the work of Xu and colleagues (2020) on auto encoders (AE) with relational regularization. The core idea is to enforce a notion of structure in the latent space of an AE, by measuring the (composed, structrural) divergence with a target distribution. In order to do so, a quadratic optimal transport problem is used (the fused Gromov-Wasserstein -FGW-), which has a super-cubical complexity. In order to alleviate this cost, Xu et al. defined a sliced version of it (akin to sliced Wasserstein -SW-), which consists in solving several simple 1D versions of the problem, that admit a close form solution, after projecting onto random directions drawn uniformly on the unit hypersphere. This work proposes to replace this uniform distribution over the sphere by a von Mises-Fisher distribution on the sphere, that is alike a ‘Gaussian distribution’ on the sphere, and also a mixture of those distributions. Their parameters are optimized during the training so that it maximises the  FGW divergence. This strategy is similar to recent trends in computing SW, that also replace the uniform distribution by either the replacing the expectation by a max or looking for subspaces that maximizes this the expected SW. \n\nThe paper is clearly written and interesting. It can be seen as incremental with respect to the work of Xu et al., but the formulation of there sliced optimal transport problem with a parametrized von Mises-Fisher distribution is novel and could also been applied to the original sliced Wasserstein, but maybe also to compute the sliced Fused Gromov Wasserstein for other task such as graph classification, as in the original FGW paper (see minor comment). \nThe experimental results are very good and clearly show the benefits of the method. Yet I have some questions, which answers might be critical for the final evaluation of the paper :\n - in the mixture part, how do you train for the parameters of the mixture ? Do you have a kind of EM algorithm, or do you perform a gradient ascent ? \n - in general, directions are drawn randomly for every batches of samples. What is the meaning of fixing in advance the number of projections, as done in Figure 3 and the related experiment ?\n - it seems that the sliced FGW is computed on mini-batches of samples. While I acknowledge there is a common practice to do so, a 1D FGW on a mini batch is not the same as computing the 1D FGW on the full dataset. As such, this mini batch version of FGW is not the same as computing the true FGW. In the end, i) the size of the mini-batch might have an impact on the estimation quality, that should be discussed ii) if computing 1D FGW on mini batches, why not computing and comparing with the mini batch version of the original version of FGW ? From the paper, the batch size is 100, this would be solved very quickly by current FGW solvers.\n\nIn the end, I think that using the von Mises Fisher distribution is an interesting and original idea, which might have a broader impact than the sliced FGW. The paper has also the merit to push a little forward the structural regularized AE which is, in the reviewer’s opinion, a good point. On the negative side, this work can be considered a little bit as incremental, and some questions remain on the experimental part.  I am willing to change my rating depending on the answers to my comments. \n\nMinor remarks\n\nIt is interesting to note that another paper in consideration for ICLR is developing a similar strategy in the case of the simple sliced Wasserstein distance:\nhttps://openreview.net/forum?id=QYjO70ACDK\nIt is also a little bit strange that the original fused Gromov-Wasserstein paper is not credited in the paper:\n[1] Titouan, Vayer, et al. \"Optimal Transport for structured data with application on graphs.\" International Conference on Machine Learning. 2019.\n\n### After author response\nI thank the authors for their detailed response to my comments. I do not agree with the complexity of FGW being solved in $n^4$ which should be more related to $n^3$ for the type of distance considered in the paper (see analysis in [1]), but yet the point is still sensible for considering the minibatch version. My other comments have been adressed, and I am changing my note to the score of 7. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "-1oJtD4ab8a", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper presents a novel method to improve the relational regularized autoencoders. The proposed method is based on the new relational discrepancy which is called the spherical sliced fused Gromov Wasserstein (SSFG). It is seen that the SSFG is an extension of the sliced fused Gromov Wasserstein (SFG) and its max version. Two variants of the SSFG are also presented. Experiments suggest that the proposed autoencoders outperform some existing autoencoders in terms of generative performance in comparable computational time. \n\n\n##########################################################################\n\nReasons for score: \n\nOverall, I have a positive impression about the paper. I think that the proposed relational discrepancy using the von Mises--Fisher distribution is a reasonable extension of the SFG using the uniform distribution. My concern is whether the estimation of the tuning parameters of the proposed relational discrepancies could be computationally expensive in practice (see cons below). Hopefully the authors can address my concern in the rebuttal period. \n\n\n##########################################################################\n\nPros: \n\n(1) Applying the fact the von Mises--Fisher distribution is an extension of the uniform distribution and the Dirac distribution, the authors successfully presented an extended relational discrepancy of the SFG and its max version.\n\n(2) The mixture spherical sliced fused Gromov Wasserstein (MSSFG) achieves an even better flexibility than the SSFG. I reckon that the MSSFG, which adopts a mixture of the von Mises--Fisher distributions, is a reasonable extension of the SSFG. \n\n(3) The extensive experiments suggest that the proposed autoencoders show satisfactory performance in terms of FID scores and reconstruction losses and are not particularly expensive in terms of computational time.\n\n \n##########################################################################\n\nCons:\n\n(1) I wonder whether the estimation of the tuning parameters of the proposed relational discrepancies could be computationally expensive in practice. For example, the MSSFG requires the values of $k$, $\\kappa_1,\\ldots,\\kappa_k, \\alpha_1 , \\ldots, \\alpha_k $. I fear that the estimation of these tuning parameters requires expensive computational cost. In the experiments of the paper, only limited combinations of these tuning parameters are considered. I wonder whether these combinations really cover a sufficient area of the parameter space.\n\n(2) The von Mises--Fisher distribution has the property that as the concentration parameter $\\kappa$ increases, the concentration of the distribution monotonically increases. In particular, the von Mises--Fisher distribution tends to the Dirac distribution as $\\kappa$ goes to infinity. However I am not sure this nice property also holds for the power distribution. If not, I wonder whether the power SSFG and power spherical DRAE have sufficient flexibility compared with the SSFG and spherical DRAE.\n\n\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above.\n\n#########################################################################\n\nTypo: \n\n(1) p.1, abstract, last line: generation ,and reconstruction -> generation, and reconstruction\n\n---\n\n## Updates:\n\nThe authors have carefully responded to my comments. Their response addresses most of my concerns. I will keep my score high. I understand that the choice of the hyperparameters can be computationally heavy, but the authors have given an idea to solve this problem. It is good to find that the Power Spherical distribution also includes the Dirac distribution as a limiting case. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a solid paper that proposes a new slicing approach to the fused Gromov Wasserstein distance using projection on directions sampled from  von-mises fisher direction, the location parameter of the von mises fisher is choosen to be maximally discriminating between the distribution 
$(\max_{\epsilon}\mathbb{E}_{\theta|vMF(\theta|\epsilon,\kappa)}\beta W(\theta\mu,\theta\nu) +(1-\beta)GW(\theta\mu,\theta\nu))$, the new sliced distance is analyzed and extended to mixture of von mises distributions with $k$ locations or directions.  This contribution of the paper is of general interest beyond the application of the paper as mentioned by the reviewers.  Authors applies the new sliced Fused Gromov Wasserstein distance to relational auto encoders and show improvement.

The spherical slicing is original and new and of independent interest and the application is good as it pushes the boundary of relational auto encoders .Reviewers  and AC did not have any concerns with the paper and the rebuttal and revisions addressed all questions raised. Accept


Decision: Accept (Poster)