Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose Gromov-Wasserstein Autoencoders (GWAE). The application of the GW metric allows to match data distributions with a given learnable prior, even when the distributions lie in spaces of different dimensions, thus allowing matching between latent and data space (in equation 4). Further, the GW objective allows rich meta-priors for flexible representation learning. Experiments on disentanglement and clustering using benchmark datasets (CelebA, MNIST and 3D Shapes) demonstrate the capabilities of the proposed approach.
strength_and_weaknesses: The authors present a new perspective of using GW for the construction of flexible variational autoencoders. For the most part the proposed approach is well motivated and properly justified, though requiring three auxiliary and complementary (based on the ablation studies) loss functions. Though mostly presented in the supplementary material, the authors present a thorough overview of the variational autoencoding landscape.

The work has some key weaknesses:

The justification for \rho=1 citing alleviating outlier effects is not necessarily convincing, more so when in (9) they adopt a \zeta=2 reconstruction loss, so why are outliers a concern in (7) but not in (9)?

Though the ablation study is a welcome addition to the experiments as they demonstrate the contribution of L_W, L_D and R_H to (13), it will be important to include results without L_GW.

The results in Figure 2 though impressive at first raise some questions: i) it seems that both VAE and GWAE separate (disentangle) object hues in 2 out of the 16 dimensions considered, and ii) even though it seems that the x-axis in Figure 2c captures the object hue, either it does not preserve the order (see color map) or there is no ordering in the hue, in which case one dimension captures the hue, but one wonders how consistently across runs and model hyperparameters.

In Table 2, it is not clear how biased are the results considering that hyperparameters were selected to optimize DCI-C that is used as performance metric and happens to show the largest difference with existing methods.

From Table 2 it is not clear why the authors could not obtain results for WAE considering the closeness with the proposed approach. Further, the table seems to indicate that FID was taken from the original papers but not PSNR? 

The results for clustering structure are interesting, however, are probably not the best way to showcase the clustering capabilities of the prior used in the proposed approach. In fact, as the authors show, a relatively simple VAE lacking a clustering prior achieves near perfect outlier detection.

The authors do not discuss why for Section 2 they use NP exclusively. It will be interesting to see results for GW estimation and minimization using FNP.

Minor points:

The marginal p_\theta(x) in (8) is not defined.
clarity,_quality,_novelty_and_reproducibility: The paper is relatively easy to follow, though somewhat inconvenient due to the large amount of content in the supplementary material. The approach seems novel and properly justified in light of the existing literature, though innovation is somewhat difficult to assess considering the complexity of the proposed objective (for instance ablation without L_GW is not considered). The authors provide details of the data, model parameter, optimization details and source code that should in principle facilitate reproducibility.
summary_of_the_review: The authors offer an interesting and seemingly flexible approach to optimize variational autoencoders by leveraging the GW metric, which allows direct matching between observed and latent spaces, however, not without the help of complementary objective functions, specifically, marginal matching given the approximate posterior (WAE), critic-based joint distribution matching (KR), and entropy regularization to alleviate collapsing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an integrated VAE-based approach for representation learning by minimizing Gromov-Wasserstein distance between the metric-measure space of the data and latent vectors. Isometric encoding is obtained from the trained prior and their performances in meta-priors (e.g. disentanglement and clustering) are empirically demonstrated without additional penalty terms. In addition, the model has good generation performance although its goal is matching latent and data distributions.

strength_and_weaknesses: ## Strengths 

This paper proposes a method that can perform disentanglement and clustering tasks within a single model. The objective for the proposed model is intuitively derived. The performance of the proposed method is extensively demonstrated with many supporting experiments. The effect of each regularization term is experimentally confirmed.

## Weaknesses

1. Merging Eqs. (8) and (10) into the constraint $p_{\theta}(x,z) = q_{\phi}(x,z)$ needs a justification. The latter implies Eqs. (8) and (10), but the vice versa. I wonder if you can explain further about the advantages of using $\mathcal{L}_D$ in (11) compared to the maximum-mean discrepancy (MMD) between the aggregated posterior and the prior.

2. That the generated CelebA images preserves the details of hair is encouraging, but this may be a consequence of adding high-frequency noise to the generated smooth image. In fact, other parts of the faces exhibit spotty noise.


clarity,_quality,_novelty_and_reproducibility: This paper is well constructed and easy to understand. Experiments and their results are well described, but the proposed method seems to lack details. An autoencoder model that approximately minimizes the Gromov-Wasserstein distance is novel. Its ability of reconstructing the high-frequency components of hair seems novel, although need scrutization. It appears that the structure of the neural sampler of $Z$ is not included.
summary_of_the_review: An autoencoder model that approximately minimizes the Gromov-Wasserstein distance seems novel, although its derivation needs further justification. The proposed model is extensively compared with other models.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: A representation learning scheme is proposed using the Gromov-Wasserstein (GW) distance, a variant on optimal transport distances which allows comparison between distributions in different metric spaces. An auto-encoder based on the GW distance is proposed and evaluated on common image datasets.  Empirical tests on disentangling and clustering are made.

strength_and_weaknesses: [+] Convincing empirical evidence that GW metric is indeed estimated (Figure 1a).

[+] Good evidence that proposed method quantitatively improves disentanglement scores (Table 1).

[+] Comprehensive review of literature.

[-] Unclear empirical advantage of GWAE for sample generation task (Table 1). Prior work 2-stage VAE appears better.

[-] No discussion of computational cost or scalability of GW estimation. Isn't L_GW (Eq. 8) quadratically expensive?

[-] Only (relatively) small scale datasets are considered (CelebA, 3D Shapes, Omniglot). Is this related to the cost of estimating GW? More discussion is desired.


clarity,_quality,_novelty_and_reproducibility: This work is clearly written and appears to me to novel. The results appear reproducible after a quick look at the submitted code.

summary_of_the_review: This work is a natural successor to many OT based generative modeling works existing in the literature. The empirical advantage of the proposed method appears limited with regards to generative quality measures, but does some improvements in disentanglement. I would like to see more discussion of cost of estimating GW. Nevertheless this work appears to me to be a worthwhile contribution and therefore I support acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper is an extension of variational auto-encoders that uses a different type of metric for performing training. The suggested metric directly matches the latent and data distributions using the variational autoencoding scheme; this is based on the Gromov-Wasserstein (GW) metric between the trainable prior and given data distributions.


strength_and_weaknesses: Pros: 
1. The authors perform an extensive ablation study of the method which convinces me that the shown results are not statistical artifacts.
2. The method is based on a solid theory which is motivated well in the manuscript. 
3. The derivations, although straightforward, are correct. 


Cons:
1. The idea is a delta upon the large corpus of VAEs; importantly, it heavily builds upon WAE.
2. Even though the method shows promise in the considered datasets, I think that much more datasets are needed for the empirical evidence to be convincing.
clarity,_quality,_novelty_and_reproducibility: The method is sound, the assumptions are clear, and the empirical results clearly reproducible. The method is only marginally novel. 
summary_of_the_review: The method is interesting and sound, though marginally novel. The experiments show promise and depth, but we need more datasets for these to be convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors propose a new auto-encoder based on Gromov-Wasserstein loss, where the proposed method is based on WAE (Tolstikhin et al. 2018). The proposed approach is interesting and practically works well. Overall, the reviewers agree to accept the paper; I will also vote for acceptance.

However, there exist some concerns in the paper:
1. The discussion with recent work about autoencoder with Gromov-Wasserstein loss is not adequately discussed. (e.g., http://proceedings.mlr.press/v119/xu20e/xu20e.pdf )
2. The proposed method only compares relatively old methods.

I strongly encourage authors to address these issues in the final version.
justification_for_why_not_higher_score: The paper has a score larger than borderline. However, there exist some concerns in the paper:
1. The discussion with recent work about autoencoder with Gromov-Wasserstein loss is not adequately discussed. (e.g., http://proceedings.mlr.press/v119/xu20e/xu20e.pdf )
2. The proposed method only compares relatively old methods.
justification_for_why_not_lower_score: At least, the proposed formulation is new and different from the previous work. Moreover, a reviewer is excited to publish the paper. So, it is good to be in the venue.

However, there exist some concerns in the paper:
1. The discussion with recent work about autoencoder with Gromov-Wasserstein loss is not adequately discussed. (e.g., http://proceedings.mlr.press/v119/xu20e/xu20e.pdf )
2. The proposed method only compares relatively old methods.

So, I am OK with the paper is rejected if there is not enough space for presentation.
Decision: Accept: poster