Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a new pseudo-distance called the spherical slices fused Gromov Wasserstein distance (SSFGW). It builds on top of the slices fused Gromov Wasserstein distance (SFGW) that takes a weighted combination of the Wasserstein distance and the Gromov Wasserstein distance on sliced spaces. The paper tackles the problem of solving for the best sampling directions of slicing. Existing approaches either assume uniform sampling strategies or a single direction that maximizes the discrepancy of two measures. This paper uses von Mises-Fisher measures (vMF) as the bridge to combine the advantages of the two. Changing the parameters of vMF is equivalent to (non-linearly) interpolating between a Dirac impulse and a uniform distribution, which converges to uniform-sliced Wasserstein and max-sliced Wasserstein, respectively. The authors provided the proof of its pseudo metric properties and upper bound and extended their SSFGW to mixtures of vFW, creating MSSFGW, and applied these variants to deterministic relational regularized autoencoder (DRAE). The results of comparing the DRAE equipped with proposed distance with other DRAEs demonstrate its superiority in stability and generative capacity.

Some comments and questions:

-- Where does von Mises-Fisher distribution come from? 

If I understand it correctly, von Mises-Fisher is a variant of Gaussian on hyperspheres. The authors can consider citing existing works or proving some arguments of choosing vMF as an interpolation of a uniform distribution and a Dirac.

-- Please refine 2 Background.

Definition 2: is $I_v$ actually $I_d$ in $C_d(k)$?


"The vMF distribution provides a way to have concentrated weight on the most important directions and assigns less weight to further directions. Therefore, we gain a better representation of the discrepancy between probability measures." 

Swap SFG and DRAE in the title of 2.1 to consist with the content of 2.1.


-- The introduction of power SSFG at the end of Section 3 seems abrupt.
If power SSFG performs better than SSFG, then what is the point of introducing vMF? Can we build mixtures of power SSFG? The paper does not mention an mps-DRAE.
On page 20 of the Appendix, at the top, it seems power SSFG is superior to SSFG in all desired aspects, but ps-DRAE underperforms s-DRAE, as shown in Table 1. Any explanation?


-- Empirical evidence of interpolating between a uniform and a Dirac
In Figure 12, abd, the graphs of s-DRAE cross the red baseline with a large momentum when $k$ decreases to 1, which suggests that it could go far away from the baseline when $k \rightarrow 0$ because when $k=1$, vFM is still very far from a uniform distribution. Please explain that.

-- A Wasserstein paper or a generative modeling paper?
The main contribution of the paper, which tackles the problem of generalizing sliced Wasserstein distance and max-sliced Wasserstein distance, is on the new pseudo metric but the paper only argues the contribution from the perspective of its power in generative modeling. I wonder if the authors ever considered evaluating the metric without add-ons like a neural network. Some typical areas of applications of the Wasserstein distances like color transfers, shape interpolation, rigid transformation since it is Gromov Wasserstein, etc? Or is the evaluation without an AE setup not necessary?

---------------------------
--AFTER REBUTTAL--
---------------------------
I appreciate the authors response. Most of my trivial comments and questions have been resolved. I stand by my initial rating. This is a solid paper. The authors clearly introduce the problem and develop a clear story with a straightforward solution. The paper ends with extensive experiments. My main concern remains: the contributions of the paper to the ML community is moderate because the story is very narrow. I think the idea can be substantially extended to solving the fundamental problems in sliced Wasserstein distances but I don't object acceptance of the paper in the current form. I recommend the authors incorporate suggestions from all the reviewers and polish the language especially Section 2 to make the paper more accessible for readers outside the sliced Wasserstein community. Thank you.
review_title: A moderate Wasserstein paper that fills a gap in recent literature with a straightforward solution.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:

The paper presents a novel method to improve the relational regularized autoencoders. The proposed method is based on the new relational discrepancy which is called the spherical sliced fused Gromov Wasserstein (SSFG). It is seen that the SSFG is an extension of the sliced fused Gromov Wasserstein (SFG) and its max version. Two variants of the SSFG are also presented. Experiments suggest that the proposed autoencoders outperform some existing autoencoders in terms of generative performance in comparable computational time. 


##########################################################################

Reasons for score: 

Overall, I have a positive impression about the paper. I think that the proposed relational discrepancy using the von Mises--Fisher distribution is a reasonable extension of the SFG using the uniform distribution. My concern is whether the estimation of the tuning parameters of the proposed relational discrepancies could be computationally expensive in practice (see cons below). Hopefully the authors can address my concern in the rebuttal period. 


##########################################################################

Pros: 

(1) Applying the fact the von Mises--Fisher distribution is an extension of the uniform distribution and the Dirac distribution, the authors successfully presented an extended relational discrepancy of the SFG and its max version.

(2) The mixture spherical sliced fused Gromov Wasserstein (MSSFG) achieves an even better flexibility than the SSFG. I reckon that the MSSFG, which adopts a mixture of the von Mises--Fisher distributions, is a reasonable extension of the SSFG. 

(3) The extensive experiments suggest that the proposed autoencoders show satisfactory performance in terms of FID scores and reconstruction losses and are not particularly expensive in terms of computational time.

 
##########################################################################

Cons:

(1) I wonder whether the estimation of the tuning parameters of the proposed relational discrepancies could be computationally expensive in practice. For example, the MSSFG requires the values of $k$, $\kappa_1,\ldots,\kappa_k, \alpha_1 , \ldots, \alpha_k $. I fear that the estimation of these tuning parameters requires expensive computational cost. In the experiments of the paper, only limited combinations of these tuning parameters are considered. I wonder whether these combinations really cover a sufficient area of the parameter space.

(2) The von Mises--Fisher distribution has the property that as the concentration parameter $\kappa$ increases, the concentration of the distribution monotonically increases. In particular, the von Mises--Fisher distribution tends to the Dirac distribution as $\kappa$ goes to infinity. However I am not sure this nice property also holds for the power distribution. If not, I wonder whether the power SSFG and power spherical DRAE have sufficient flexibility compared with the SSFG and spherical DRAE.


##########################################################################

Questions during rebuttal period: 

Please address and clarify the cons above.

#########################################################################

Typo: 

(1) p.1, abstract, last line: generation ,and reconstruction -> generation, and reconstruction

---

## Updates:

The authors have carefully responded to my comments. Their response addresses most of my concerns. I will keep my score high. I understand that the choice of the hyperparameters can be computationally heavy, but the authors have given an idea to solve this problem. It is good to find that the Power Spherical distribution also includes the Dirac distribution as a limiting case. 

review_title: A novel method to improve relational regularized autoencoders

final_score: rating: 7: Good paper, accept
review_content: This paper builds on the work of Xu and colleagues (2020) on auto encoders (AE) with relational regularization. The core idea is to enforce a notion of structure in the latent space of an AE, by measuring the (composed, structrural) divergence with a target distribution. In order to do so, a quadratic optimal transport problem is used (the fused Gromov-Wasserstein -FGW-), which has a super-cubical complexity. In order to alleviate this cost, Xu et al. defined a sliced version of it (akin to sliced Wasserstein -SW-), which consists in solving several simple 1D versions of the problem, that admit a close form solution, after projecting onto random directions drawn uniformly on the unit hypersphere. This work proposes to replace this uniform distribution over the sphere by a von Mises-Fisher distribution on the sphere, that is alike a ‘Gaussian distribution’ on the sphere, and also a mixture of those distributions. Their parameters are optimized during the training so that it maximises the  FGW divergence. This strategy is similar to recent trends in computing SW, that also replace the uniform distribution by either the replacing the expectation by a max or looking for subspaces that maximizes this the expected SW. 

The paper is clearly written and interesting. It can be seen as incremental with respect to the work of Xu et al., but the formulation of there sliced optimal transport problem with a parametrized von Mises-Fisher distribution is novel and could also been applied to the original sliced Wasserstein, but maybe also to compute the sliced Fused Gromov Wasserstein for other task such as graph classification, as in the original FGW paper (see minor comment). 
The experimental results are very good and clearly show the benefits of the method. Yet I have some questions, which answers might be critical for the final evaluation of the paper :
 - in the mixture part, how do you train for the parameters of the mixture ? Do you have a kind of EM algorithm, or do you perform a gradient ascent ? 
 - in general, directions are drawn randomly for every batches of samples. What is the meaning of fixing in advance the number of projections, as done in Figure 3 and the related experiment ?
 - it seems that the sliced FGW is computed on mini-batches of samples. While I acknowledge there is a common practice to do so, a 1D FGW on a mini batch is not the same as computing the 1D FGW on the full dataset. As such, this mini batch version of FGW is not the same as computing the true FGW. In the end, i) the size of the mini-batch might have an impact on the estimation quality, that should be discussed ii) if computing 1D FGW on mini batches, why not computing and comparing with the mini batch version of the original version of FGW ? From the paper, the batch size is 100, this would be solved very quickly by current FGW solvers.

In the end, I think that using the von Mises Fisher distribution is an interesting and original idea, which might have a broader impact than the sliced FGW. The paper has also the merit to push a little forward the structural regularized AE which is, in the reviewer’s opinion, a good point. On the negative side, this work can be considered a little bit as incremental, and some questions remain on the experimental part.  I am willing to change my rating depending on the answers to my comments. 

Minor remarks

It is interesting to note that another paper in consideration for ICLR is developing a similar strategy in the case of the simple sliced Wasserstein distance:
https://openreview.net/forum?id=QYjO70ACDK
It is also a little bit strange that the original fused Gromov-Wasserstein paper is not credited in the paper:
[1] Titouan, Vayer, et al. "Optimal Transport for structured data with application on graphs." International Conference on Machine Learning. 2019.

### After author response
I thank the authors for their detailed response to my comments. I do not agree with the complexity of FGW being solved in $n^4$ which should be more related to $n^3$ for the type of distance considered in the paper (see analysis in [1]), but yet the point is still sensible for considering the minibatch version. My other comments have been adressed, and I am changing my note to the score of 7. 
review_title: Original work but maybe a little bit incremental

Meta-review: This is a solid paper that proposes a new slicing approach to the fused Gromov Wasserstein distance using projection on directions sampled from  von-mises fisher direction, the location parameter of the von mises fisher is choosen to be maximally discriminating between the distribution 
$(\max_{\epsilon}\mathbb{E}_{\theta|vMF(\theta|\epsilon,\kappa)}\beta W(\theta\mu,\theta\nu) +(1-\beta)GW(\theta\mu,\theta\nu))$, the new sliced distance is analyzed and extended to mixture of von mises distributions with $k$ locations or directions.  This contribution of the paper is of general interest beyond the application of the paper as mentioned by the reviewers.  Authors applies the new sliced Fused Gromov Wasserstein distance to relational auto encoders and show improvement.

The spherical slicing is original and new and of independent interest and the application is good as it pushes the boundary of relational auto encoders .Reviewers  and AC did not have any concerns with the paper and the rebuttal and revisions addressed all questions raised. Accept


Decision: Accept (Poster)