Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present flowification techniques for linear and convolutional layers by proposing to see linear layer weight matrices as parametrized by SVD decomposition, and convolutions as coordinate repetition followed by matrix multiplication. This enables the likelihood contribution of these layers and the computation of their inverse-- i.e., flowify them.
strengths_and_weaknesses: *Strengths*

- The proposed formulation for unifying general architectures and normalizing flows looks mathematically elegant and sound. It would make it possible to realise the benefits of flows for general layers, as the authors experimentally demonstrate as well.
- The paper is clear and well-written-- the intuition provided for the definitions and theorems is helpful to understand them. 

*Weaknesses*

- There are experiments for flowified convolutional architectures, but I couldn't find any experiments for flowified linear layers. Wouldn't it make sense to see them as well?
- It would make it clearer to have a representational diagram for a dimension-increasing flowified linear layer as well to get an overview.
- Although it's interesting to see flowified CNNS, their performance is not at the level of flow-based models, which is recognised by the authors as well. 
questions: - Why are there no experiments on density modelling considered just for flowified linear layers? I am curious as to how they perform as compared to flow-based models.
- How does the flow for the flowified CNNs look visually? It would be interesting to compare the flows from FCONV with other state of the art models like i-ResNet etc. 
- Also, wouldn't it make sense to also consider classification error experiments on MNIST and CIFAR-10 with non flow-based models included as baselines as well?

It is possible that I did not understand the experiment design properly, and am willing to adjust my score accordingly.
limitations: Yes, I think the authors adequately address the limitations and impact of the proposed flowification in terms of its performance and the possible causes for it, and how it enables us to see standard architectures as subsets of normalizing flows. However, it would also be great to see potential negative societal impact be addressed in the paper (if there is anything specific), which I don't see as being addressed in the current version.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a general approach for making commonly used neural network layers (e.g. linear and convolutional) invertible and usable as construction blocks of normalizing flows. Notably, under some assumptions, these “flowified” layers are allowed to shrink or expand dimensionality of the inputs. The proposed building blocks are tested as parts of convolutional neural networks on the density modeling and generation tasks on the MNIST and CIFAR10 benchmarks.
strengths_and_weaknesses: *Strengths*
* The material is well-presented and the paper is clearly written. The main contribution (“flowification” procedure) is described in an accessible way despite its mathematical complexity.
* The paper contains novel theoretical contributions, namely the “flowification” procedure, that would be interesting to a broad audience.

*Weaknesses*
* Empirical evaluation of the proposed method is rather limited (just two datasets). The results are also somewhat underwhelming (far behind the presented baseline methods). But more importantly, as indicated by the authors, the cause of underwhelming empirical performance is unclear. Providing further insight into this through additional experiments would be very valuable.

questions: (See Strengths & Weaknesses)
* The authors propose several hypotheses for why their method performs worse compared to the conceptually similar i-ResNet (Section 5). Can additional experiments be provided in order to support one or more of these hypotheses?
limitations: Limitations of the proposed method are unclear due to the limited set of presented experiments and architectural differences between the authors’ model and the competing methods their model is compared against.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper claims to provide a method to convert any convolutional or fully-connected layer into a normalizing flow. This is primarily performed by converting the representation into an equivalent SVD-representation using a pair of learnable orthonormal matrices(/tensors) in conjunction with a positive, diagonal matrix. This allows any NN composed of conv/linear layers to be converted to a NF and trained via MLE, possibly in conjunction with some other objective.
strengths_and_weaknesses: Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?
The paper is heavily reliant on SURVAEs and does cite the relevant work. However, given the relative importance of the method it would be helpful if the paper explained this idea, possibly in an Appendix instead of just referencing the work. I found much of the paper confusing until I read this preceding paper.
The paper relies heavily on the use of the SVD reparameterization method but does not cite any of the preceding efforts that explore this idea. Some comment on the use of the skew-symmetric -> matrix exponentiation formulation of the orthogonal weights vs the more typical Househoulder reflection method should probably be discussed as well.

Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
The paper claims that their method converts any MLP/CNN into a NF. However, in fact, their method only produces a tractable likelihood estimate when the layer is contractive in dimension (or when the layer is dimension preserving). When the layer is dimension expanding, the resultant "NF" is only capable of providing an ELBO instead of an exact likelihood estimate. "Everything" isn't a NF unless you can (exactly) marginilize out the introduced random variables at the inlet of the dimension expanding layers. Since a lower bound on log-likelihood is an upper bound on negative log-likelihood and, therefore, BPD, it is unsurprising that the experimental results show much higher "BPD" than alternative NF which do provide a likelihood estimate and not a bound.

Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)
Most of the paper is accessible and well written. The discussion of padding and unfolding was somewhat confusing but could be understood on careful read.

Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
The general idea could be very important but I am skeptical that the proposed solution is as effective as presented. The method also claims to work for MLPs and CNNs but the experiments only demonstrate CNNs. The application of the idea to a toy dataset or the standard UCI likelihood datasets (power, gas, miniboone, hepmass, bsds) could be very informative.

General comments:
I really like the goal of the paper but am unconvinced that the paper achieves its claims, largely because of the limiation of dimension expanding operations. Perhaps I have misunderstood this operation and the conversion from a tractable likelihood estimator to a bound. If you can soften your claims or demonstrate my misunderstanding I am happy to increase my score. Similarly, if you can demonstrate your method on non-image datasets (possibly in an Appendix) I will reconsider my score.
questions: Please see above.
limitations: Please see above.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper introduces a method that transforms any multilayer perception into the flow model. The authors investigate various scenarios where they apply linear and convolutional layers as transformations inside the flow, either reducing or extending the dimensionality of the data. Some experiments on MNIST and CIFAR are performed to investigate the quality of the approach. 
strengths_and_weaknesses: STRENGTHS

The defined problem and direction are important for the committee. The idea of searching for novel forms of transformations and connections with existing models seems to be attractive. 

WEAKNESSES

I read the paper several times, and I had a problem identifying the contribution of this work. The title and the abstract suggest rather that we should expect to transfer standard MLPs into Bayesian networks using flows. In my opinion, the proposed solution rather enriches the family of transformations for SurVAE flows: 

Nielsen, D., Jaini, P., Hoogeboom, E., Winther, O., & Welling, M. (2020). Survae flows: Surjections to bridge the gap between vaes and flows. Advances in Neural Information Processing Systems, 33, 12685-12696.

with linear and convolutional transformations. Assuming this, it is not clear if the proposed transformations are surjective for inference or generation. Assuming we consider the surjective inference, how the sampling process is implemented. 

The paper is difficult to follow, the math transformations are difficult to follow, the notation seems to be unexplained. For instance:
- eq. (7) it is not clear what $p_{inv}$ is and why it is not expressed in log domain. What is $p(z)$ here. 
- The choice of $\phi(z|x)$ is also not discussed in the paper. 
- matrix $\Sigma$ should have positive values on diagonal I Guess. 

The application of the linear layer is equivalent to the application of probabilistic PCA, where the likelihood can be easily calculated in the closed-form: $p(x)=\int p(x|z)p(z)dz$ can by calculated analytically. 

There are no clear advantages of provided transformations compared to the proposed in SurVAE. Authors should discuss using the benefits of their approach compared to an existing one. 

The clarity of writing and consistency with previous works regarding notation should be improved. Instead of \emph{Sketch of proof} the exact proof should be provided in the appendix, instead of above $\mathcal{L}$ the reference to the definition should be delivered. Moreover, using $\log \Sigma$ is further explained but practically means another type of operation. 

My primary concern is about the experimental part. For such works, achieving SOTA is not crucial, but the results are far from baselines. There are also no toy examples that show the benefits of the approach for particular assumptions. 

Concluding, the paper seems to be a bit of a work in progress and should be reorganized, and additional experimental evaluation should be included.  




questions: I would like to ask the authors to refer to the remarks from the Weakness section, mainly focusing on:

1. Relation to SurVAE approach, how the surjective for inference or generation are designed, and how sampling for inference is performed.
2. The concerns about notations and the design choices (eq. (7), $\phi(z|x)$)
3. Relation to probabilistic PCA that uses the linear transformation.  
 
limitations: The limitations were included in the conclusion section.
review_title: null

Meta-review: metareview: This paper proposes a straightforward framework to adapt a wide class of DNNs to be amenable to build normalizing flow. The framework which is derived is neat and is mostly supported experimentally. While some reviewers pointed that the links with SURVAEs could be stated more clearly in the initial submission and noted some weaknesses in the experimental part, the authors have done a convincing rebuttal. It seems that the contribution of this work is significant enough to lead to acceptance.
confidence: Certain
Decision: Accept