Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes GEM model to capture the underlying structure of datasets across modalities. It can obtain perceptually consistent interpolations between samples, and can further recover points on the manifold and realize audio/image signals completion. Besides, it can also generates new samples. 
main_review: Strength:
+ The paper focuses on manifold learning and proposes an interesting idea with various interesting applications. 
+ The proposed model GEM with the local isometry loss and the LLE loss outperforms the three baselines in most cases both quantitatively and qualitatively.

Weakness:
- Missing comparisons to other baselines in the literature, such as Learning Manifold Implicitly via Explicit Heat-Kernel Learning. 

Question:
In the ablation study table, it shows that the local isometry loss and the LLE loss are quite effective with GEM structure. Of noting, StyleGANv2 sometimes outperforms the GEM without the local isometry loss and the LLE loss. I am wondering how these losses work with other models, say StyleGANv2? 
How does the visualization in Fig.3 for GEM look like without the local isometry loss and the LLE loss?
limitations_and_societal_impact: I don't see any potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a modality-agnostic implicit generative model. A majority of the existing generative models involve architecture adept in a particular domain (e.g., convolutional layers for images), often requiring careful design and incurring difficulties in dealing with multi-modal data. To tackle this, the authors propose a generative manifold learning (GEM) scheme, which faithfully reflects the manifold hypothesis by utilizing hypernetwork-based implicit modeling. While implicit modeling had been proposed in previous work, unique to GEM is the use of the proposed local isometry and LLE loss functions. Such loss design removes the need of auxiliary networks (e.g., discriminator for GANs), which might require modality-dependent desgin, making the training of the model as well be completely modality-agnostic.
main_review: 
The paper is well organized and reads easily. The considered topic of finding a generic architecture that can be shared across different modalities is interesting and important. However, the proposed method seems to have the following weak point: 

* The model seems to be good at reconstruction, but weak at generation. 

Since the latent space structure/distribution is of a very flexible form, the reconstruction should be good by design. However, the cost from which is difficulty in obtaining samples from such a complicated latent distribution. 

Generating samples by interpolating two points, as suggested in Sec. 4.4, could be too crude: 1) linear approximations of a manifold can have large error, especially when the number of training data is small. 2) samples obtained from such a process does not have a gurantee to converge to the true underlying density in probabilistic sense. As a result, generated samples can be inaccurate.

This might partially explain the results in Table 2 and Figure 10. The proposed model shows a FID score far worse than StyleGAN2 and the generated images are blurry. It however shows better recall, as the interpolation-based sampling scheme seems to be advantageous in covering all the data (almost perfect if reconstruction error is low).

---

All in all, it is understandable that the suggested framework and the loss functions are a solution to the modality-agnostic training; GANs need a discriminator, which usually requires modality-dependant design. However, considering the aforementioned issues, and given that the the loss functions themselves are not directly connected to the modality-independance, there seems to be an alternative method. For example, one could employ the techniques from adversarial auto-encoders (A. Makhzani et al., 2015), which can enforce the latent distribution to be Gaussian, resolving the aforementioned issues, and still be modality-agnostic. 



Minor:

1) (line 278) Isn't the (non-)blurriness indeed an important property we want to measure from the generated samples? 
2) The loss in Eq. (7) seems to be closely related to the Jacobian of the generative map. Is there a particular reason that Eq. (7) is used instead of Jacobian?


limitations_and_societal_impact: The authors properly described the limitations and the societal impact in Sec. 6.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work introduces GEM - a model which learns the low-dimensional latent manifold structure of a dataset in a modality-independent way. GEM’s objective function is designed to encourage the learning of manifolds which exhibit the following properties, namely they: cover the data, are locally metric, and are globally consistent.  The model is trained on four datasets with different modalities (image, audio, 3D shape, and cross-modal image and audio). Qualitative and quantitative evaluations suggest that GEM can be used to do linear interpolation between points on the learned manifold, it can generate reconstructions of corrupted samples as well as generate new samples. 
main_review: ### Strengths

- To the best of my knowledge, this is a novel approach to manifold learning which can easily be applied to data from different modalities.
- The evaluation protocol is extensive and is both qualitative and quantitative. It includes visualizations of interpolation between signals, signal generation, and signal inpainting (for images), as well as hallucinations of one modality given another. 
- The paper is organized and written clearly. 

### Room for Improvement and Questions
1. The presented quantitative results are missing error bars with respect to the number of random seeds used even though the answer to this question on the Checklist is [YES].
2. How were the hidden layer size (of 512) and number of layers in the implicit function $\Phi$ and hypernetwork $\Psi$ chosen? Have the authors tried other values for these hyperparameters and measured their effect on the reported results? 
3.  How was hyperparameter search performed for the benchmarks (StyleGAN2, FDN, ShapeNet)?
4. Since GEM’s loss is a sum of three different components, how are the relative weights of each loss component determined?
5. I noticed that in the Image Hallucination visualization in Fig 8 for an empty spectrogram the bow is very blurry or missing. Do the authors have a guess of what the reason for this might be? 

### Nitpicks
- Line 72: “embeddings $\it{obtained}$”
- Figure 1: “represents $\it{an}$ audio spectrogram”
- Line 102: there is an extra “method”
- Line 106: “coordinate” 
- Line 111: “the same across signal types” (without “for”)
- Line 120: $W^{L_{h}}$ should be $W^{L}_{h}$
- Line 129: there is an extra “in”
- Line 172: “than signals that $\it{are}$ perceptually distinct” 

### Rebuttal Period Update
I've decided to increase my score to 6 after reading the author's response to my questions and concerns. 
limitations_and_societal_impact: Yes, they have.
review_title: null

Meta-review: The paper proposes GEM, an approach for multi-modal manifold learning, which is novel and with interesting results. I found the rebuttal very well done and I think it clarified many open issues with the approach, although not to the degree to have a total agreement of the reviewers. Nevertheless I find the clarifications sufficient and I am inclined to accept the work. 

Decision: Accept (Poster)