Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses the challenge of learning a normalizing flow, under the assumption that the data probability function is concentrated on a low dimensional manifold. Utilizing the manifold assumption introduces a challenge in computing the volume change term. This paper suggests addressing this computational challenge in two ways: i) If the manifold dimension is low enough, it is reasonable to calculate the map differential exactly based on AD forward mode; ii) alternatively, an unbiased stochastic estimator for the log det can be used.

main_review: The paper addresses the challenge of learning normalizing flows. Previous work has recently suggested incorporating the manifold assumption into the normalizing flow [1].  The loss that follows from the manifold normalizing flows in [1] depends on a volume change term, a term that [1] did not compute directly. The current paper claims that there is an advantage in optimizing directly the loss involving the volume change term, and suggests two options to compute it. Even though there is not a lot of novelty in the techniques related to the suggested computations, I mark this paper as a solid contribution to the ongoing effort of learning normalizing flows and rate it with a positive rating. Detailed comments are provided next. 

Generally, the paper is well written and easy to follow. The introduction and the method presentation are of good quality. I appreciate the author's effort to include all the relevant details to the method, in addition to some issues that were considered in devising the method. This is also reflected in the experiments section, where details about failed experiments are included.

The main contribution of the paper is that in fact the loss suggested in [1] can be computed directly in some cases or stochastically estimated. It seems to me that the stochastic estimation of the logdet is not novel, see [2]. Also, the exact computation is based on the forward mode AD, which is a known technique. 

I am concerned about the inconsistency in the results of figure 1. Do the authors have some thoughts on what can be done? This should be in the main text. 

The experiments show an advantage over [1], except for the OOD experiment. However, why other alternatives have not been considered? In table 3 it would be important to see results from other square flow models (without the manifold assumption), or contractive auto-encoders (that are known to approximate the derivative of the data log probability function [3]). As the current method cannot run on more complex datasets like CIFAR, these additional comparisons can some light if this line of works is worth pursuing. 

[1] : Flows for simultaneous manifold learning and density estimation, Brehmer and  Cranmer.
[2] : Invertible Residual Networks, Behrmann et al.
[3] : What Regularized Auto-Encoders Learn from the Data Generating Distribution, Alain and Bengio,
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper addresses the problem of constructing injective normalizing flows, which connect some low-dimensional space with the data manifold of interest in the high-dimensional space. Typically injective (or rectangular) flows are composed of two square flows with the “upsampling” padding layer between them. Because the volume change term in the case of injective flows is seemingly intractable, current approaches find workarounds by training these two square flows separately step-by-step. Though this decision has certain drawbacks and may result in suboptimal performance. Current work suggests to employ numerical linear algebra methods to make volume change computation tractable. Experiments demonstrate that taking into account volume сhanging term improves the generation results both for synthetic and real-life data. Interestingly, it also improves out of distribution detection score, allowing the model trained on FashionMNItST to assign lower probability for the MNIST data samples.
main_review: - Paper is clearly written, exposition is coherent with all the  preliminaries and motivations for the technical decisions in place. Derivations are plain and comprehensible. 

- While injective flows as a class were described before, previous heuristics for implementing them were oversimplified and in the best case suboptimal. Thus making volume-change tractable for the injective flows seems to be an important step forward. 

- While (as stated in the paper) conjugate gradients were previously applied to the calculation of the volume change for the square flows, authors are the first to adopt them to the rectangular flows and Jacobian-Transpose Jacobian product calculation. Thus the novelty in this sense is moderate. Any comments?

- Experiments on the synthetic and tabular data are convincing and clear demonstrate, that two-step method leads to learning incorrect probability distribution. On the Fashion MNIST FID improvements are quite notable. 

- Though, for me, the fact that increasing the number of steps and thus decreasing variance leads to worse results is a bit puzzling, and thus I would like to see more experiments in this direction, which would clarify this question.  Any comments?

- Out-of-distribution experiments also are promising and hint that low-dimensional manifold is a important inductive bias. 

- Overall the result on the toy data and simple image data are quite convincing, though until it is verified on the real-life level of complexity data the broader significance is under question. Thus I would also l like to see some experiments on CIFAR to get sure.  Any comments?
limitations_and_societal_impact: The authors adequately addressed the limitations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper is motivated by the desire to circumvent a problematic requirement in Normalizing Flows (NFs) on the function $f: \mathbb{R}^d \rightarrow \mathbb{R}^D$ that produces the pushforward measure --- namely, that $f$ must be strictly invertible. This is difficult for generative models, since the dimension of the latent space $d$ is usually much smaller than that of the target data space $D$, i.e. $d << D$. The invertibility requirement centers around the change-of-variables formula, which is crucial for computing the cost function in NFs, and which requires computing the Jacobian of $f^{-1}$. To enable the use of NFs on latent/data spaces where $d << D$, the authors propose "Rectangular" NFs (RNFs). The name comes from the fact that the Jacobian of $f$ is now rectangular in the $d << D$ case. This involves the use of a differential geometric variant of the change-of-variables formula, and is much more computationally taxing, but provides superior density estimation performance to previous work.

main_review: The authors explore a very natural and compelling question that arises from the invertibility requirement (of the aforementioned $f$) in NFs, when applying NFs to generative modeling. 

$f$, which can be thought of as the "decoder" or "generator" network, often maps from some lower dimensional latent space to a higher dimensional data space. If $f$ and the latent space are suitably well behaved, it would be reasonable to assume that, even though $f$ is not invertible, its restriction to the data manifold $\mathcal{M}$ could be. Intuitively, if $f$ is injective, this suggests that there does still exist a tractable way to compute the pushforward density, and one wonders if there is a more general case of the change of variables formula that can be used.

To this end the authors demonstrate that, while this general form does exist, the computation of its terms (namely, the inverse of the Jacobian) is unfortunately quite heavy and numerically unstable, and moreover, stochastic estimations of this Jacobian via random projections is noisy. 

Though these are not strictly positive results, they highlight important obstacles to further progress in an area of very active research. Moreover, the work is presented in a lucid and understandable manner, and the limitations of the current method are clearly delineated. For these reasons, I am inclined to accept the paper.
limitations_and_societal_impact: There is limited experimental validation of the proposed technique. The authors compare their method to only one other work, on three datasets: a synthetic circle dataset, MNIST, and FMNIST (they mention having tried CIFAR10 and SVHN, but were unable to tune their model in a reasonable amount of time). Even with MNIST and FMNIST, it could be informative to train an NF with a 2-dimensional latent space and visualize manifold to see if some structure of the dataset were preserved.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents new methods for estimating densities using rectangular normalizing flows. Rectangular normalizing flows, unlike squared normalizing flows, allow considering injective instead of bijective flows. This property is exploited for estimating the density of data residing on a low (d) dimensional manifold in R^D, for D>d. Existing works using rectangular flows to estimate densities supported on low dimensional manifolds assume that the manifold is known, and therefore are restricted to tori, spheres, etc. In contrast, this work does not make this assumption and shows that density estimation for injective flows based on ML is tractable. Two methods are proposed, which are based on incorporating and estimating the “Jacobian-transpose-Jacobian” in the objective. One is based on an exact derivation, which is computationally demanding. The other is based on stochastic gradient estimates. Experimental results on a simulation and a couple of datasets are shown, demonstrating the advantage of the proposed methods compared to an existing method as well as the tradeoff between memory and variance incurred by the two proposed methods.
main_review: This is a well-written, timely, and interesting paper. It addresses an important problem and presents a solid contribution. 
The work has several notable limitations:
- It can only learn densities supported on manifolds homeomorphic to R^d. 
- It is computationally expensive and feasible only when the manifold dimension is low.
- Partly as a result of the limitation above, the experimental study is not impressive as it includes only relatively simple examples. 
I commend the authors for clearly putting these limitations forward in a nice discussion in Section 6, which also highlights some intriguing results and the gained insights. 
Despite its shortcomings, I believe the paper improves existing methods and advances this line of work, and I would be happy to see it published.

Minor comments:
- A clear description of the two algorithms would help (even if put in the appendix due to length limitations).
- More details on the choice of the number of gradients to estimate K (in the stochastic variant) would help practitioners.
- In Appendix F, failed runs on the simulated data are presented, which raises two questions: is there an explanation for the failures? can the results be quantified (over many runs)?  
- The FID score, which is used to evaluate the results in Sec 5.2, is defined only in the appendix. I believe that some details, or a basic definition, need to appear in the main paper.
- In the experimental study on MNIST (Sec 5.3), the dimension d is set to 20. Why? Was it obtained empirically? Is it based on previous works? Is this a result of the computational limitation? Please elaborate.

limitations_and_societal_impact: The authors very nicely discuss the limitations of their work. 
I do not foresee any potential negative societal impact
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The task of interest is to simultaneously learn a manifold (with a fixed predefined dimensionality) and the density on it, from training data. The proposed method, rectangular flows, is developed based on an existing work (Brehmer and Cranmer [6]) that used a two-step training procedure (for manifold matching and density matching, respectively). The main contributions include: (1) revealing that the two-step training can be improved by joint training; (2) a new technique to enable computing the Jacobian-transpose-Jacobian term of the joint training objective with better efficiency. Experiments on simulated data and MNIST/FMNIST are conducted to demonstrate the proposed method.
main_review: Originality. The presented method in Section 4.2, for calculating the volume-change term, is believed new; it's a novel combination of existing techniques. 

Quality. The presented method is technically sound. Some empirical observations are not discussed/analyzed in detail. For example, the inconsistent results on simulated data, and the unexpected behavior on MNIST/FMNIST.

Clarity. The writing is OK, but can be certainly improved; for example, to highlight the underlying logic, the main contributions over existing methods, algorithms that show the big picture and/or the details (like VJP/JVP), and so on.

Significance. The contributions might be incremental; the presented method enables joint training for existing work, but the corresponding importance is not well demonstrated with serious experiments and/or analysis.

Other comments:

Line 48, the exact method is quite straightforward. I don't think it is a contribution, so no credit for it.

Eq (8). Since beta is just treated as a hyperparameter, the KKT conditions are not met or used. Please clarify.

Lines 164-165. What's the meaning of the statements here?

How to calculate a VJP or a JVP? It's not clear in the manuscript.

Eq (12). Since the (J^T J eps) term of the loss is calculated via VJP/JVP, how do you guarantee the correct gradient wrt theta?

Line 323. Why RNFs can remedy the phenomenon? Also, too many conjectures are made in this paragraph without solid analysis.

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: Congratulations on the acceptance of your paper! Please incorporate changes, edits and additional promised experiments from "Author Discussion" in the final paper/appendix.
Decision: Accept (Poster)