Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes to leverage conformal embedding flows for modeling probability distributions on low-dimensional manifolds. Differing from other manifold-based flows, the proposed methods maintain tractable densities for low-dimensional latent space. The unique contribution enables fast sampling, invertibility for inference, and efficient likelihood estimation, which inherits from normalizing flows. 

Existing approaches share the drawback that exact densities cannot be calculated through the dimensionality reducing step. In the paper,
the authors present an approach that offers tractable density estimation and is straightforward to train by constraining this reduction to the class of conformal embeddings. 
main_review: Followed by recent work on manifold flows, the paper poses an attractive research direction for normalizing flows.  Overall, the paper is well-written and easy to follow. The mathematical derivations are rigorous and careful so I did not see any unclear part in the thought process. Through two experiments, the authors demonstrate the idea clearly and also give a detailed discussion about the current limitation and future direction, which would be helpful. I just have few concerns listed as follows:

1. The experiments are not very strong. I think it would be better if the authors could show more synthetic examples, for example, a more challenging one with complex manifolds. For the synthetic example, there are no other baselines for comparison such that we may not get a straightforward insight why the proposed methods are better.  For the image example, could you show more examples, maybe from a simpler one with less computational cost?  To compare the performance of generative models, I know FID is commonly used but recently some other metrics are proposed, for example, density and coverage metrics [1], which may help to evaluate the performance better. 

[1] "Reliable fidelity and diversity metrics for generative models." In International Conference on Machine Learning, pp. 7176-7185. PMLR, 2020.

2. The baseline and ablation study.  The authors discussed two important baselines [3] and [27], is that possible to compare them quantitatively with some experiments, even toy examples?  It might be helpful to highlight the original contribution and unique strength of the proposed flows. As for the size of low-dimensional space, have you investigated the effect of various latent dimensionality? It is interesting to see the performance if you have the results in experiments. 

3. Implementation and details. I expect to see the implementation if the authors would like to provide the code in the supplementary materials. Unfortunately, I can not reproduce the code and see more details. I just worried about the efficiency, scalability, and flow expressivity as the authors mentioned.  For the flow-based architecture, the authors only use the affline coupling layers in RealNVP and Glow, does it get improvement if some advanced flow-based models, like neural spline flows, are used?  Is the easy to incorporate the conformal embedding operator into other flow-based models? If so, that would increase the general applicability of this method. 



limitations_and_societal_impact: No potential negative societal impact
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Many datasets of interest that are modelled with normalizing flows are assumed to lie on a lower-dimensional manifold embedded in a higher-dimensional ambient space. Standard normalizing flows that are restricted to learn a *bijective* transformation are not well suited for modelling such data, as the target transformation between the lower-dimensional manifold space and the ambient space (in which said manifold is embedded) is *injective*.

In theory, an *injective flow* can be defined, where we'd parametrize an injective transformation that would only be invertible *for points on the manifold*. In practice there are two problems with this, however. First is that historically there's been much less work on designing flexible *injective* transformations, hence we don't have the same arsenal of invertible layers as we do for bijective flows. Second, the change-of-variable formula for an injective transformation includes a $\det J_f^T J_f$ term, where $J_f$ is the (rectangular) Jacobian matrix of the injective transformation $f$. This determinant, even if tractable, is typically overly expensive to compute exactly.

Authors of this work make an observation that in the bijective flow literature the game has always been to design parametric transformations $f$ that are restricted to be invertible, but *also* ones that have and easy-to-compute $\det J_f$. In line with this, authors propose a class of transformations that would have an easy-to-compute $\det J_f^T J_f$, specifically *conformal embeddings*. A conformal embedding $f$ is a transformation that preserves *angles*, but more importantly has a property that $J_f^T J_f$ is a scalar multiple of the identity, the determinant of which is simply the product of diagonal terms (i.e. an exponent of said scalar). 

Authors consider a few examples of conformal embeddings (translation, scaling, an orthogonal transformation, zero-padding, etc.), which could be stacked in the same way as invertible layers are in a bijective flow. In addition, authors consider *piecewise* conformal embeddings, which are conformal *almost everywhere* (i.e. for all points except for a subset of measure zero), and can increase the expressivity of the otherwise simple conformal embeddings.

By using the conformal embeddings as injective transformations authors come up with the first injective flow with a tractable and *cheap-to-compute* density on the learned manifold. This allows said density to be used as a part of the training objective, even though an additional regularization term still needs to be included in the objective to prevent degenerate solutions. The method demonstrates excellent results on a synthetic 3D problem where the target density lies on a sphere, as well as the CelebA image dataset, where the visual fidelity of the samples is improved in comparison to the more expressive baseline flow.
main_review: What an interesting idea, and what a well-written paper --- very much enjoyed reading it! It could be one of these papers that kick-starts a new sub-field within the normalizing flow community, one where researchers would be working on designing novel types of expressive conformal embeddings.

Authors motivate the proposed method very well, taking the reader on a journey that arrives at the conformal embeddings as almost the one logical solution, which of course it might only be in retrospect.

The mathematical exposition is excellent: authors manage to keep things precise without drowning the reader in equations.

In my view the weakest part of the paper is the experiments. While what is done is definitely enough to get the reader interested, it's probably not sufficient to get an idea of whether the method is ready for real-world use. A comparison with alternative injective flow implementations would be of great help, as well exploring the hyper-parameters of the method (like the latent dimensionality) and additional datasets. In similar work datasets produced by sampling from a generative model with a lower-dimensional latent space (itself trained on real data) are sometimes used, where  the "real" intrinsic dimensionality of the dataset is known.

Nevertheless, I think the paper will be of great interest to the community, and the theoretical part is sufficiently novel to give authors some leeway in regards to the empirical part. I recommend acceptance, but also encourage authors to focus their efforts on the experiments for future iterations of the paper. 

Minor points:
- Line 164: while it might indeed be trivial that zero-padding is conformal, I suggest including it in Table 1 for completeness (and perhaps writing a sentence summarizing the proof of the transform being conformal). This is the part that allows the flow to change dimensionality, hence it's worth being particularly precise here.
- In two places authors refer to a normalizing flow (I think?) as a "stump": this doesn't seem to be defined anywhere, and I've not seen this terminology before.
- Table 3: it might be useful to have another baseline where the "Sequential Baseline" would be run with $g$ that would have the number of parameters that is similar to $g$ used in CEF, just to make the runs more comparable. 

---

I thank the authors for their response. These additional results and clarifications in the exposition will definitely strengthen the paper. Having read the other reviews/responses, I maintain my recommendation to accept the paper.
limitations_and_societal_impact: Authors carefully discuss both the limitations of the method and its potential societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes classes of conformal building blocks that can be used to parameterize a homomorphic flow network that can map from low dimensional space to a continuous manifold in high dimension space.
main_review: Originality: The paper restricts function g to be a conformal embedding. The k*k convolution is new, the SCT is new to the machine learning field.

Clarity: The paper is clearly written. But I think the novel building blocks like k*k convolution should be detailed discussed.

Quality: The submission is technically sound.

Significance and Concerns:
1. My first concern is more general. The traditional flow models aim to have a tractable likelihood for maximum likelihood estimation and achieves success in a lot of applications. The paper interests in modeling distributions that lie on a low-dimensional manifold. As the author mentioned, the density is no longer well defined (the distribution is not absolutely continuous with respect to Leabage measure), so the maximum likelihood estimation (MLE)  and the KL divergence are not well defined,  so why we need to construct a function with tractable log density in the first place when the density is not defined? Furthermore, the method has to use L2 reconstruction loss as a regularization to alleviate the ill-defined MLE (equation 7), does this objective provide a consistent estimation, and what distance it is minimizing? Why not just directly use a distance which is well defined for the singular distribution like e.g. Wasserstein distance. Although I understand there are several papers that use this setting but the benefit of such a setting is unclear to me which makes me less confident about the motivation of the paper.

2. The paper restricted the function g to be conformal, and propose several building blocks. However, the experiments are limited and cannot support the significance of the proposed methods. The samples of the CelebA experiments are far from good, which maybe because of the limited function power or the training objective with the l2 norm. The reconstruction quality is less meaningful for generative models, it can only verify the invertibility.  I think more experiments and comparisons should be conducted to support the usefulness of the framework. I can understand the  CelebA dataset may not satisfies the model assumption (lies exactly on a low-dimensional manifold ) and the intrinsic dimension is unknown. So I suggest the training data can be generated from a trained GAN model, so the intrinsic dimension is (approximately) equal to the dimension of the latent space and the generated samples will lie on a low dimensional manifold.


limitations_and_societal_impact: Yes, the author mentioned the major limitation is expressive power.
review_title: null

Meta-review: The submission proposes to use conformal mapping to implement injective flow to map into a lower dimensional embedding space. 
The proposed idea is original, well explained, and well motivated. While the experiments were lacking in the initial submission, the authors have properly engaged with the reviewers: they offered the appropriate additional experiments to improve the paper and address reviewers concerns. 

I recommend this paper for acceptance.
Decision: Accept (Poster)