Flowification: Everything is a normalizing flow
Authors: Bálint Máté
Samuel Klein
Tobias Golling
François Fleuret
Venue: NeurIPS 2022 Accept (2022)
Keywords: Normalizing flows

Abstract: The two key characteristics of a normalizing flow is that it is invertible (in particular, dimension preserving) and that it monitors the amount by which it changes the likelihood of data points as samples are propagated along the network. Recently, multiple generalizations of normalizing flows have been introduced that relax these two conditions \citep{nielsen2020survae,huang2020augmented}. On the other hand, neural networks only perform a forward pass on the input, there is neither a notion of an inverse of a neural network nor is there one of its likelihood contribution. In this paper we argue that certain neural network architectures can be enriched with a stochastic inverse pass and that their likelihood contribution can be monitored in a way that they fall under the generalized notion of a normalizing flow mentioned above. We term this enrichment \emph{flowification}. We prove that neural networks only containing linear and convolutional layers and invertible activations such as LeakyReLU can be flowified and evaluate them in the generative setting on image datasets.

Flowification: Everything is a Normalizing Flow
Bálint Máté' University of Geneva
Samuel Klein" University of Geneva samuel.. www.inlnngee..o ch
balint. www.itt..ateuunig.... ch
Tobias Golling University of Geneva tobias. boooaas.gllingeuiige
François Fleuret University of Geneva francois. fleuret@unige.co
Abstract
The two key characteristics of a normalizing flow is that is invertible (in particular, dimension preserving) and that it monitors the amount by which changes the likelihood of data points as samples are propagated along the network. Recently, multiple generalizations of normalizing flows have been introduced that relax these two conditions [1 2 On the other hand, neural networks only perform forward pass on the input, there neither notion of an inverse of neural network nor is there one of its likelihood contribution. In this paper we argue that certain neural network architectures can be enriched with a stochastic inverse pass and that their likelihood contribution can be monitored in a way that they fall under the generalized notion a normalizing flow mentioned above. We term this enrichment flowification. We prove that neural networks only containing linear and convolutional layers and invertible activations such as LeakyReLU can flowified and evaluate them in the generative setting on image datasets.
Introduction
Density estimation techniques have proven effective on wide variety of downstream tasks such as sample generation and anomaly detection [ Normalizing flows and autoregressive models perform very well at density estimation but do not easily scale to large dimensions [8HI and have to satisfy strict design constraints to ensure efficient computation of their Jacobians and inverses. Ad- vances in other areas of machine learning cannot be utilized flow architectures because they typically seen as being invertible; this restricts the application of highly optimized architectures from many domains to density estimation, and the of the likelihood for diagnosing these architectures.
Methods using standard convolutions and residual layers for density estimation have been developed for architectures with specific properties [1H. These methods do not provide recipe for converting general architectures into flows. There no known correspondence between normalizing flows and the operations defined by linear and convolutional layers.
In this paper we show that large proportion of machine learning models can be trained as normalizing flows. The forward pass of these models remains unchanged apart from the possible addition of uncorrelated noise. To demonstrate our formulation works we apply it fully connected layers, convolutions and residual connections.
The contributions of this paper include:
"Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In I we show that linear layers induce densities as augmented normalizing flows 2 with the multi-scale architecture used in RealNVPs G We also show how these layers can be viewed as funnels [5] to increase their expressivity. We term this process flowification.
In $..2 argue that most ML architectures can be decomposed into simple building blocks that are easy to flowify. As an example, we derive the specifics for two dimensional convolutional layers and residual blocks.
In S we flowify multi-layer perceptrons and convolutional networks and train them as normalizing flows using the likelihood. This demonstrates that models built from standard layers can be used for density estimation directly.
2 Background
Normalizing flows
Given a base probability density {po(z)]= E Z} and a diffeomorphism f Z, the pullback along / induces probability density (r(x)) E X} on X, where the likelihood of any x E X given by p(z) (o(ff)))det(({)) where J is the Jacobian of f evaluated at r. Thus, the log-likelin logllikelihoods of the two densities are related by an additive term, which will be referred to as the likelihood contribution (ax 2) M. Normalizing flows parametrize a family fo of invertible functions from X to Z. The parameters 0 are then optimized to maximize the likelihood of the training data. lot of development has gone into constructing flexible invertible functionn with easy to calculate Jacobians where both the forward and inverse passes are fast to compute [688 16
As the function f must be invertible it required to preserve the dimension of the data. This limits the expressivity of ʃ and makes expensive to model high dimensional data distributions. To reduce these issues several works have studied dimension altering variants of flows
21
Dimension altering generalizations of normalizing flows
Reducing the dimensionality A simple method for altering the dimension of a flow is to take the output of an intermediate layer 2' and partition it into two pieces z' { 2/)) Multi- scale architectures [6] match z/2 directly a base density and apply further transformations z1- Funnels 115 generalize this by allowing z/2 to depend on zz. i.e. they work with the model p(=') = p(f (z)))()))()))))) ((()) 1 where the conditional distribution (==)) train- able. It is useful to think of these factorization schemes as dimension reducing mechanisms from dim(z') to (1)).
Increasing the dimensionality Dimension increasing flow layers improve models flexibility, as demonstrated by augmented normalizing flows (2] To increase the dimensionality, x is embedded into larger dimensional space and data independent noise is added the embedding to obtain a distribution with support of nonzero measure. This noise addition x + (ær, u) is similar to dequantization 6 221, but is orthogonal the distribution of a and increases its dimension from dim æ to dim I + dim u. Under an augmentation the likelihood of x can be estimated using
log p(x) = log / du puz u)
(1)
log /
du plu)p(a, u) p(u)
(2)
du p(u) log p(x,1) u)
(3)
p(u)
Em-p(u) log p(x, u)
(4)
p(u)
Eupp(() log u) log p(u)].
(5)
In practice, we estimate this expectation value by sampling u everytime datapoint passed through the network. This means the integral is estimated with single sample as in sur VAEs M

---

3 Flowification
Suppose A is a network architecture with parameter space ④ Then for any choice of 0 E θ the network with parameters 0 realizes a function Ag R C for some D and C. Similarly, normalizing flow model F a parametric distribution on some Re, where for any choice of y from their parameter space I' they define density function Fy on _ In this work we show that large class of neural network architectures can be thought of as a flow model by constructing a map network architectures
}
- {
flow models
(6)
The embedding of A to its flowification F results in flow model that can realize density functions on the augmented space RD RN for some 2 O, which in turn induces a density on RD by integrating out the component on R.V. The parameter space of F- factorises θ x & where θ is the parameter space of A and also that of the forward pass of FA,, while parametrises the inverse pass of JFA. In the simplest case t 0, i.c. flowification does not require additional parameters. It is in this sense that we claim that a large fraction of machine learning models are normalizing flows.
Terminology In what follows we work with conditional distributions such as p(=(æ) and will be practical to think of them as "stochastic functions" p I + 2, that take an input and produce an output ~ p)z]z). Conversely, we think of function f : + 2 as the Dirac o-distribution (z|æx) 8(2 - f(x)). These definitions allow us to have a unified notation for deterministic and stochastic functions such that we can talk about them in the same language. Consequently, when we say "stochastic function", it will include deterministic functions as a corner case. Depending on whether f and f-1 are deterministic or stochastic, we talk about left, right or two-sided inverses. We will be careful to be precise about this.
Method the following we consider the standard building blocks of machine learning architectures and enrich them by defining (stochastic-Jinvers functions and calculating the likelihood contribution of each layer. Treating each layer separately allows density estimation models to be built through composition [1] The stochastic inverse can the funnel approach, which increases the parameter count, or the multi-scale approach, which does not. For simplicity we will only consider conditional densities in the inverse this is more general, though it not required. We will refer this process as flowification and the enriched layers as slowified;; non-flowified layers will be called standard layers. Flowified layers can then be seen as simultaneously being
Flow layers that are invertible, their likelihood contribution is known and therefore can be used to train the model to maximize the likelihood.
Standard layers that can trained with losses other than the likelihood, but for which the likelihood can be calculated after this training with fixed weights in the forward direction.
3.1 Linear Layers
Let Lw. R + ImM denote the the linear layer of a neural network with parameters defined by a weight matrix W Rmxx and bias b Rm Formally, Lw,s is defined as the affine function
a Lw,( = Wx +6
I ER"
Definition! 1. Let d(2(xr): Ra - Th be stochastic function. funchstiiffunction We say that à is linear in expectation if there exists W E Rmxx and E R such that for any x E R the expected value of ó coincides with the application of Lwb
[=]](]]]]<]]
Lw,(
(8)
Similarly, we say that a stochastic function (2/(x) is convolutional in expectation if the deterministic function x + Bewp(sls)]z]] is convolutional layer.
In this section we flowify linear layers, by which we mean we construct pair of stochastic functions, a forward C(=[x) R² a - Rm and an inverse ->((x))) Rm -> R such that the forward is linear in expectation and is compatible with the inverse in way that will be made precise in the following paragraphs.
3

---

SVD parametrization
To build flowified linear layer, the first step is to parametrize the weight matrices by the singular value decomposition (SV This involves writing W Rmxx as a product W VEU, where U E RTx" is orthogonal, Σ E Rmxx is diagonal and V m is orthogonal. This parametrization particularly useful for our purposes because the orthogonal transformations are easily invertible and do not contribute to the likelihood, and the non-invertible piece of the transformation is localized to Σ
Parametrizing U and V We generate elements of the special orthogonal group SO(d) by applying the matrix-exponential to elements of the Lie algebra so(d) of skew-symmetric matrices. We parametrize so(d) and perform gradient descent there. As the Lic-algebra is a vector space, this is significantly easier than working directly with SO(d). See Appendix G for details.
Parametrizing Σ The matrix Σ of shape m x n containing the singular values on the main diagonal. We ensure maximal rank of Σ, by parameterizing the logarithm of the main diagonal, this way all singular values are greater than
It is important to note that this parametrization is not without loss of generality. In particular, does not include matrices of non-maximal rank nor orientation reversing ones, where either U E O(n) (0)n)) or V E O(m) \ (0mm). This implementation detail does not change the general perspective we provide of linear layers as normalizing flows, but instead simplifies the implementation of flowified layers.
Reducing the dimensionality
Definition 2. We call the tuple (I())x), (=(=)))))(()) dimension decreasing flowified linear layer if is dimension decreasing, linear in expectation and the following conditions are satisfied
(i) The forward is deterministic, given by [(=(x) Lw(a(x),
(ii) The layer is right-invertible, L. h-1 idz,
(iii) The likelihood contribution of can be exactly computed.
To flowify dimension decreasing linear layers, we define the forward function L as a standard linear layer with parameters W and b,
C(2(x) 8(2 Lw.b(z)).
Since is parametrized by the SVD decomposition, W = VEU, we need to invert V, U and separately. As V and U are rotations, they invertible in the usual sense. To construct stochastic inverse to Σ, we think of it as a funnel 15] and use a neural network ((Ux) [Ux) that models the is not required to calculate the likelihood under the model, even a fixed distribution could be used, but introducing some trainable parameters significantly improves the performance of the flow that is defined by the layer. We use Σ-1 to denote this stochastic inverse to Σ. The stochastic inverse function C-1 can then be written as
dropped coordinates
function of the non-dropped coordinates. Again, this
(-)(z) = U - o VT(2 b).
(10)
Since the rotations don't I contribute to the log- liglliicl,hood, the likelihood of data under a dimension decreasing flowified linear layer
log p(z) = Lone ((Ux) (m:) [U+) + log + log p(2),
(11)
where log Σ. denotes the sum of the logarithms of the diagonal elements of Theorem 3. The above choices for L and L-1 define dimension decreasing flowified linear layer.
Sketch of proof. The definition of the forward pass 2 makes the forward pass linear expectation and satisfies 1 by definition. Unpacking the definitions and decomposing W into its SVD form yields right-inertibiliity iii which in turn implies that the likelihood contribution can exactly computed i
When the inverse density is not made to conditional the above ideas can visualized standard multi-scale flow architecture [ as shown in Fig. |

---

2
4
2/
U
2/2
Figure 1: A mutli-scale flow with base density p(z, =2) on the left. A dimension reducing linear layer with activation as multi-scale flow with a base density p(z, 2/2 on the right.
Increasing the dimensionality
Definition 4. We define the Moore-Penrose pseudoinverse LWv of linear layer Lw,6 : IR. as the affine transformation T + I
R
T (=
Rm
(12)
where W+ denotes the Moore-Penrene pseudoinverse of the matrix W.
Definition 5. We call the tuple (C()))) a dimension increasing flowified linear layer if f. is dimension increasing, linear in expectation and the following conditions satisfied
(iv) The inverse L is deterministic,, given by L (xlz) = L.v((2).
(v) The layer is left-invertible, C-1 -= idz,
(vi) The likelihood contribution of can be bounded from below.
To construct dimension increasing flowified linear layers, rely again on the SVD parametrization where the only nontrivial component is Σ. In this case Σ is a dimension increasing operation and we think of as an augmentation step composed with diagonal scaling. To augment, we sample m - n coordinates from distribution p(u) with zero mean and then apply scaling in m dimensions. The likelihood contribution then given by
log ((x) Eurp(u) [lgp(p log [(w] log E,
(13)
logp(æ) log 2 + l(g)
(14)
where log Σ denotes the sum of the logarithms of the scaling parameters. The inverse function C-1 is the composition of the inverse rotations, the inverse scaling and the dropping of the sampled coordinates. This sequence of steps is visualized in Fig. -
U
1
Augmant
Σ
2
V
u
Figure 2: dimension increasing flowified linear layer.
Theorem 6. The above choices for L and L-1 define a dimension increasing flowified linear layer.
Sketch of proof. The augmentation step results in lower bound on the likelihood contribution, implying [vi) Since Eucpp(a) 0, the augmentation does not influence the expected value of z, i.e. the forward pass is linear in expectation. Simple calculations using the SVD decomposition then
imply both (iv) and (v)
Preserving the dimensionality
Dimension preserving layers are corner case of both of the above scenarios, where padding and sampling are not needed in either direction and the layer is non-stochastically invertible. All this implies
(
and are satisfied.

---

3.2 Convolutional layers
Convolutions can be seen as dimension increasing coordinate repetition followed by a matrix multiplication with weight sharing. In the previous section derived the specifics of matrix multiplication. We begin this section with the details of coordinate repetition after which we put the pieces together build flowified convolutional layer. In Appendix H we describe an alternative approach relying on the Fourier transform.
Repeating coordinates In this paragraph focus the N-fold repetition of single scalar coordinate z. This significantly simplifies notation, but the technique generalizes in an obvious way. Intuitively, the idea is to expand the one dimensional volume to N dimensions by first embedding and then increasing the volume of the embedding such that the volume in the N 11 directions complementary the embedding be controlled.
We have seen in that the operation
(x,
(W1, m UN-1);
(15)
has likelihood contribution En log p(u) which maps 0) to J (1,1) to obtain
Now, we can apply any N-dimensional rotation Ry
Rx u) (x + (0 (x,x) + Rx u).
(16)
Note that this rotation does not contribute to the likelihood. Finally, we apply diagonal scaling in N dimensions with factor v N such that
x
(r. x + (O, VNu),
(17)
where the final scaling has likelihood contribution N log( N) = (N/2) log N and x is now repeated N times. The overall contribution to the likelihood of the embedding [7
Vax 2) = Eul Feul-logp(un p(u) (N/2) N.
(18)
By construction, the padding distribution Ry (0, /Nu) is orthogonal the diagonal embedding I + (x. 2) of the data distribution. The inverse function is given by the projection to the diagonal embedding,
(N) N Σ 24
(19)
General architectures Now that the likelihood contribution of arbitrary linear layers and coordinate repetition has been computed it is possible to flowify more general architectures such as convolutions and residual connections. It is important to note that just because an architecture works well for certain tasks, it not clear its flowified version will perform well at density estimation.
Decomposing convolutional layers To flowify convolutional layers, we decompose it as sequence of building blocks that are easy flowify separately. A standard convolutional layer performs the following sequence of steps:
1. Padding of the input image with zeros to increase its size.
2. Unfolding of the padded image into tiles. This step replicates the data according to the kernel size and stride.
3. Applying a linear layer. Finally, we apply the same linear layer to each of the tiles produced in the previous step. The outputs then correspond to the pixels of the output image.
00 and 1 denote the (N - 1i-dimensional vectors (0....0) and (1, 1), respectively, Similarly x denotes the (N dimeensiil vector (z...., x)).

---

Flowification Steps 1 and 3 are already flowified, i.e. their likelihood contribution computed and inverse is constructed, in $3.1 We denote their flowification with Pad and Linear, respectively. Step fits into the discussion of the previous paragraph of repeating coor- dinates, where both its inverse (19) and its likelihood contribution (18) are given. We will denote this opera- tion by Unfold.
SVD
21
2/2 + w
Augment
222
22
SVD
3/3
22
223
Figure 3: A flowified ID-convolution with kernel size 2 applied to vector with features. The component ap- pears in the operation twice, and is first duplicated so that the kernel can be applied con-overlapping tiles.
Definition 7. Let Linear, Unfold and Pad be above and define and C-1 be the following stochastic func- tions
C = Linear o Unfold Pad
(20)
=-1= Unfold o Linear-
(21)
call the resulting layer (C, (C,C-)) a flowified convolu- tional layer.
flowified convolutional layer (C, C-1) then convolutional in expectation (Definition D i.e. there exists a convolutional layer Co with parameters 0 such that
E [=~C(=|=] Co(x).
(22)
The flowification of a convolution without padding can be seen in Fig. 3 The Unfold operation is implemented as coordinate duplication and Linear flowified linear layer parameterized by the SVD.
Activation functions Functions that are surjective onto IR and invertible fit well in our framework as they can be used out of the box without any modifications. In our experiments we use LeakyReI.U and rational-quadranl splines [7] activations functions. non-invertible activations can also be used when equipped with additional densities [
Residual connections Residual connections can be seen as coordinate duplication followed by two separate computational graphs f2) with the outputs recombined in sum. The sum can be inverted by defining density over one of the summands p(fi (a + uf (x + u), f2(x u)) and sampling from this density, which will also define the likelihood contribution. Then, if the likelihood contribution can be calculated for each individual computational graph, the likelihood the total operation can be calculated []]
Experiments
To test the constructions described in the previous section we flowify multilayer perceptrons and convolutional architectures and train them to maximize the likelihhod of different datasets.
Tabular data In this section study selection of UCI datasets e and the BSDS300 collection
of natural images (25) using the preprocessed dataset used by masked autoregressive flows [7 26 We compare the performance with several baselines for comparison in Table D We see that the flowified models have the right order of magnitude for the likelihood but are not competitive.
Table I: Test log likelihood (in nats, higher is better) for UCI datasets and BSDS300, with error bars corresponding to two standard deviations.
MODEL.
POWER
GAS
HEPMASS
MINBOONE
BSDS300
GLOW
0.38 0.01
12.02 + 0.02
-17.22 + 0.02
-10.65 0.45
156.96 + 0.28
NSF
0.63 + 0.01
13.02 0.02
-14.92 + 0.02
-9.58 + 0.48
157.61 + 0.28
FMLP
-0.50 + 0.02
5.35 + 0.02
-19.56 + 0.04
-14.05 0.48
144.22 + 0.28

---

Image Data In this section we use the MNIST 27 and CIFARIO 28 datasets with the standard training and test splits. The data is uniformly dequantized as required to train on image data [29] 30 For both datasets we trained networks consisting only of flowified linear layers (FML.P) and also networks consisting of convolutional layers followed by dense layers (FCONV1). To minimize the number of augmentation steps that occur in each model we define additional architectures with similar numbers of parameters but with non-overlapping kernels in the convolutional layers (FCONV2). The exact architectures can be found in Appendix F.I The flowified layers sample from (00, a) for dimension increasing operations, where a is a per-layer trainable parameter. We use rational quadratic splines 77] with 8 knots and a tail bound of activation functions, where the same function is applied per output node. We also ran experiments with coupling layers using rational quadratic splines 7 mixed into FCONv2, in all other cases the parameters of the model are not data data-dependent operations. The improved performance of these models suggests that flowified layers do not mismodel the density of the data, but they do lack the capacity model it well. Samples from these models shown in Appendix
Table 2:
Test-set bits per dimension (BPD) for MNIST and CIFAR-10 models, lower better. Results from several other works were included for comparison. Flowified models with overlapping kernels FCONV1 1 and non-overlapping kernels FCONV2 are shown, with a similar parameter budget to the neural spline flow The models FCONV1 + NSF and FCONV2 NSF correspond architectures using rational quadratic spline layers in-between the flowified layers of FCONV1 and FCONV2, respectively. Samples from these models can be found in Appendix C
MODEL GLOW
MNIST
CIFAR-10
1.05
3.35
REALNVP
3.49
NSF 7
3.38
I-RESNET
1.06
3.45
I-CONNNET 14
4.61
MAF 17
1.91
4.31
FMLP
4.19
5.45
FCONVI
3.11
4.91
RCONV2
1.41
4.20
FCONV1 + NSF
2.70
3.93
FCONV2 NSF
1.35
3.69
The results of the density modelling can be seen in Table. 2 The images in the left column of Fig. 4 are generated by sampling from a standard gaussian the latent space and taking the expected output of the inverse in every layer. The images in the right column use the same latent samples as the left column but also sample from the distribution defined by the inverse pass of the layers.
As seen in Fig. 2 the FMLP models are outperformed by the FCONV models and the convolutional models with overlappppp kernels achieve better results than the ones with overlapping kernels. This suggests both that the inductive bias of the convolution is useful for modelling distributions of images and that the augmentation step costs more in terms of likelihood than it provides in terms of increased expressivity.
Related Work
Several works have developed methods that allow standard layers be made invertible, but these approaches restrict the space of the models, whereas we consider networks in their full generality. Invertible ResNets [2] require that each residual block has Lipschitz constant less than one, but
even with this restriction they attain competitive results on both classification and density estimation.
The same Lipschitz constraint can also be applied to other networks [I3. these architectures the multi-scale architecture used in RealNVPs [6] is not leveraged, and so no information is discarded by the model. is unclear why this approach outperforms flowified layers, as seen in Table. 2 but it could be due to the preservation of information through the model, the very large number of parameters that are used in these approaches, the restricted subspace of the models, or some combination of these three.
8

---

6
7
1
4
4
7
7
0
5
3 0
3
3
5
6
7
9
3
4
6
4
7
7
7
4
0
5
4
7
6
6
5
6
8
7
6
7
O
7
2
8
6
2
3
5
7
3
3 ]
9
Q
7
6
G
4
/
8
9
6
4 7
7
6
4
7
5
2
6
8
7
7
3 4
3
&
0
0 3
2
7
7
9
3
3
0 0 5 4
7
8
0
3
3
6
8
4
7
7
9
0 5 4
T 6
7
8
9
4
/
6
9
4
0
7
8
6
8
4
6
8
/ 4
4
7
4
0
7
8
5
8
4
8
2
3
O
3
9
9
5
6
7
6
5
8
2 3
4
3
4
9
5
6
7
5
0
6
8
4
4
Q
8
6
O
7
7
1
J
5 6 5
4
4
2
8
0
0
9
7
2
3
0
7
3
9
5
5"
3
6
4
2 8
3
7
7
A
(
9
3
5
4
3
9
6
6
4
Q
5
7
7
6
2 8
3 6
9 8
9
9
5
7
7
8
2 3
9
6
3
5
4
4
6
Q
7
8
A
)
1
5
5
4
Y
Figure 4: Samples from flowified multilayer perceptrons (top two rows) convolutional networks with overlapping (third and fourth rows) and non-overlapping (bottom two rows) kernels trained on MNIST and CIFAR-10. 10. Samples where the mean is used to invert the SVD in the inverse pass (left column). Samples generated by drawing from the inverse density to invert the SVD in the inverse pass (right column) More samples from these models can be found Appendix B
can also be shown that convolutions with the same number of input and output channels can made invertible [14] These layers perform poorly the task of density estimation and are outperformed by flowified layers. This is likely due to the increased expressivity that comes from considering larger space of architectures. There have been several works that develop convolution inspired invertible transformations B3 but these architectures consider restricted transformations to maintain invertibility.
6 Future Work and Conlusion
Our experiments suggest that flowified convolutional networks do match the density estimation performance of similarly sized normalizing flows. A possible explanation is that the dimension reducing steps discard information and more expressive encoding layers are necessary to transform the distributions before reducing the dimensionality. This is supported by the experiments using NSF layers in-between the flowified layers (see App. C.. The addition of NSF layers leads to improved performance both in terms of visual quality and also BPD values. Possibly the main limitation of a network consisting purely flowified layers is the fact that, unlike flow layers, the forward passes of standard línear and convolutional layers are not data-dependent. This is reinforced by the fact that the entanglement capability typically used in flows also appears in attention mechanisms, which have been shown to excel capturing complex statistical structures R4 36

---

With further development such as increased capacity given to the inverse density, or data dependent parameters in the forward pass standard architectures could become competitive density estimators in their own right and allow for general purpose models to be developed. The focus of this work was on employing standard layers for density estimation, variants of standard layers that are more flow-like could improve their performance as classification and regression. The flowification procedure provides a useful means for designing such models, and demonstrates that standard architectures can be considered subset of normalizing flows, a correspondence that has not previously been demonstrated.
is possible that designing data dependent
tasks such
The code for reproducing our experiments is available under http:///ggaabbbboooooollaaattttt
MIT
license at
Acknowledgement
The authors would like to acknowledge funding through the SNSF Sinergia grant called Robust Deep Density Models for high-Energy Particle Physics and Solar Flare Analysis (RODEM) with funding number CRSI15_19371
References
[1] Didrik Nielsen, Priyank Jaini, Emiel Hoogeboom, Ole Winther, and Max Welling. Survae flows: Surjections to bridge the gap between vaes and flows. Advances in Neural Information Processing Systems, 33: 13:1266 1685-12266, 2020.
[2] Chin-Wei Huang, Laurent Dinh, and Aaron Courville. Augmented normalizing flows: Bridging the gap between generative flows and latent variable models, 2020.
[3] Esteban G Tabak and Cristina V Turner A family of nonparametric density estimation algo- rithms. Communications on Pure and Applied Mathematics, 66(2):145-164, 2013.
[4] George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. arXiv preprint UrXiv:170..07000 2017.
[5] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayann Normalizing flows for probabilistic modeling and inference, 2021.
[6] Laurent Dinh, Jascha Dicl-Dicksttin, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803 2016.
[7] Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows, 2019.
[8] Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolu- tions. arXiv preprint arXiv: 2018.
[9] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know?, 2019.
[10] Claudius Krause and David Shih. Caloflow: Fast and accurate generation of calorimeter showers with normalizing flows, 2021.
[11] Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. 2018. doi: 10.48550/ARXI..1811.0099 URL https://arxiv org/abs/1811 00995
[12] Ricky T. Chen, Jens Behrmann, David Duvenaud, and Jöm-Henrik Jacobsen. Residual flows
for invertible generative modeling, 2019, URL https: httpp:////axii.orr//ab/b wttp:bb///0...2200... 02735
[13] Yura Perugachi-Diaz, Jakub Tomezak, and Sandjai Bhulai. Invertible densenets, 2020. URL
https://arxiv org/ abs///as//200... 02125
10

---

[14] Mare Finzi, Pavel Izmailov, Wesley Maddox, Polina Kirichenko, and Andrew Gordon Wilson. Invertible convolutional networks. In Workshop on Invertible Neural Nets and Normalizing Flows, International Conference on Machine Learning. 2019.
[15] Samuel Klein, John A Raine, Sebastian Pina-Otey, Slava Voloshynovskiy, and Tobias Golling. Funnels: Exact maximum likelihood with dimensionality reduction. arXiv preprint arXiv:2112.08069 2021.
[16] Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improving variational inference with inverse autoregressive flow, 2016. URL https
//axxiv.org/ab//1666 04934
[17] George Papamakarios, Theo Pavlakou, and lain Murray, Masked autoregressive flow for density estimation, 2017. URL https://axxxi.. wrgg/bs/////000 07057
[18] Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation, 2020. URL https://arxiv, wwg/abs.////eeoooooo...o.e.
13913
[19]
Edmond Cunningham and Madalina Fiterau. A change of variables method for rectangular matrix-vector products. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of Artificial Intelligence and Statistics, volume 130 of
The 24th International Conference Proceedings of Machine Learning Rexearch, pages 2755-2763, PMLR, 13-15 Apr 2021. URL mlr
https: nttps://proceedingg httpooeeiiiggg....pppe/////////a http://///113///nnnnnnnaa2/222/ html
[20] Anthony L. Caterini, Gabriel Loaiza-Ganem, Geoff Pleiss, and John P. Cunningham. Rectangu- /aarg//bb//1/// 2111/1111101111333
lar flows for manifold learning, 2021. URL
[21] Brendan Leigh Ross and Jesse C. Cresswell. Tractable density estimation on learned manifolds with conformal embedding flows, 2021.
05275
[22] Benigno Uria, lain Murray, and Hugo Larochelle. Rnade: The real-valued neural autoregressive density-estimator. 2013. doi: (0.44550/ARXII.1306..11 URI. httpt://////xxii..tttm /bs/
1306. 0186
[23] Jakub M. Tomezak and Max Welling. Improving variational auto-encoders using householder flow, 2016. URL htttpp///ariv.oorrra/////...... 09630
[24] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. Uttptt:://archhee
ics uci. edu/ml
[25] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int'l Conf. Computer Vision, volume 2, pages 416-423, July 2001.
[26] George Papamakarios, Preprocessed datasets for maf experiments, January 2018. URL https //doi.org/10 5001/ 1161203
[27] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: Syann.lecure com/ cwtm/ex///iist//////////////////
[28) Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[29] Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design, 2019. URL
00275
[30] Lucas Theis, Aäron van den Oord, and Mathias Bethge. A note on the evaluation of generative 01844
models, 2015. URL http::::/aaiiv.oog///aaaaaa
[31] Mahdi Karami, Dale Schuurmans, Jascha Sohl-Dickstein, Laurent Dinh, and Daniel Duckworth. Invertible convolutional flow. Advances in Neural Information Processing Systems, 32, 2019.
[32] Emiel Hoogeboom, Rianne van den Berg, and Max Welling. Emerging convolutions for generative normalizing flows, 2019.
11

---

[33] Emiel Hoogeboom, Victor Garcia Satorras, Jakub M. Tomczak, and Max Welling, The con- volution exponential and generalized sylvester flows, 2020.
2006. 01910
[34] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[35] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerberttVosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.
[36] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020. URL htttttsss//axxii.orrggg.eee wtwpo////aa///ooooa 20.0000..00119999
[37] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. 2017. URL hRtpss::://axiii org wwwpa///....eeee.a 11///9912...0 01703
[38] Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www wandb.com/ Software available from wandb.com.
[39] Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios, nflows: normalizing flows in PyTorch, November 2020. URL
wwt.o//......//.ooo.ooooooo 42962877
[40] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2016. URL
[41] B. Hall. Lie Groups, Lie Algebras, and Representations: An Elementary Introduction. Graduate Texts in Mathematics. Springer International Publishing, 2015.
12


Reviews:
[{"review_id": "i4krDjlxj_9", "reviewer": "Reviewer_LKEV", "summary": "The paper claims to provide a method to convert any convolutional or fully-connected layer into a normalizing flow. This is primarily performed by converting the representation into an equivalent SVD-representation using a pair of learnable orthonormal matrices(/tensors) in conjunction with a positive, diagonal matrix. This allows any NN composed of conv/linear layers to be converted to a NF and trained via MLE, possibly in conjunction with some other objective.", "questions": "Please see above.", "limitations": "Please see above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?\nThe paper is heavily reliant on SURVAEs and does cite the relevant work. However, given the relative importance of the method it would be helpful if the paper explained this idea, possibly in an Appendix instead of just referencing the work. I found much of the paper confusing until I read this preceding paper.\nThe paper relies heavily on the use of the SVD reparameterization method but does not cite any of the preceding efforts that explore this idea. Some comment on the use of the skew-symmetric -> matrix exponentiation formulation of the orthogonal weights vs the more typical Househoulder reflection method should probably be discussed as well.\n\nQuality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\nThe paper claims that their method converts any MLP/CNN into a NF. However, in fact, their method only produces a tractable likelihood estimate when the layer is contractive in dimension (or when the layer is dimension preserving). When the layer is dimension expanding, the resultant \"NF\" is only capable of providing an ELBO instead of an exact likelihood estimate. \"Everything\" isn't a NF unless you can (exactly) marginilize out the introduced random variables at the inlet of the dimension expanding layers. Since a lower bound on log-likelihood is an upper bound on negative log-likelihood and, therefore, BPD, it is unsurprising that the experimental results show much higher \"BPD\" than alternative NF which do provide a likelihood estimate and not a bound.\n\nClarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)\nMost of the paper is accessible and well written. The discussion of padding and unfolding was somewhat confusing but could be understood on careful read.\n\nSignificance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\nThe general idea could be very important but I am skeptical that the proposed solution is as effective as presented. The method also claims to work for MLPs and CNNs but the experiments only demonstrate CNNs. The application of the idea to a toy dataset or the standard UCI likelihood datasets (power, gas, miniboone, hepmass, bsds) could be very informative.\n\nGeneral comments:\nI really like the goal of the paper but am unconvinced that the paper achieves its claims, largely because of the limiation of dimension expanding operations. Perhaps I have misunderstood this operation and the conversion from a tractable likelihood estimator to a bound. If you can soften your claims or demonstrate my misunderstanding I am happy to increase my score. Similarly, if you can demonstrate your method on non-image datasets (possibly in an Appendix) I will reconsider my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HYwufW7zAGc", "reviewer": "Reviewer_PUhx", "summary": "The paper proposes a general approach for making commonly used neural network layers (e.g. linear and convolutional) invertible and usable as construction blocks of normalizing flows. Notably, under some assumptions, these “flowified” layers are allowed to shrink or expand dimensionality of the inputs. The proposed building blocks are tested as parts of convolutional neural networks on the density modeling and generation tasks on the MNIST and CIFAR10 benchmarks.", "questions": "(See Strengths & Weaknesses)\n* The authors propose several hypotheses for why their method performs worse compared to the conceptually similar i-ResNet (Section 5). Can additional experiments be provided in order to support one or more of these hypotheses?", "limitations": "Limitations of the proposed method are unclear due to the limited set of presented experiments and architectural differences between the authors’ model and the competing methods their model is compared against.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n* The material is well-presented and the paper is clearly written. The main contribution (“flowification” procedure) is described in an accessible way despite its mathematical complexity.\n* The paper contains novel theoretical contributions, namely the “flowification” procedure, that would be interesting to a broad audience.\n\n*Weaknesses*\n* Empirical evaluation of the proposed method is rather limited (just two datasets). The results are also somewhat underwhelming (far behind the presented baseline methods). But more importantly, as indicated by the authors, the cause of underwhelming empirical performance is unclear. Providing further insight into this through additional experiments would be very valuable.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BkQQCLzt74", "reviewer": "Reviewer_Kcoo", "summary": "The authors present flowification techniques for linear and convolutional layers by proposing to see linear layer weight matrices as parametrized by SVD decomposition, and convolutions as coordinate repetition followed by matrix multiplication. This enables the likelihood contribution of these layers and the computation of their inverse-- i.e., flowify them.", "questions": "- Why are there no experiments on density modelling considered just for flowified linear layers? I am curious as to how they perform as compared to flow-based models.\n- How does the flow for the flowified CNNs look visually? It would be interesting to compare the flows from FCONV with other state of the art models like i-ResNet etc. \n- Also, wouldn't it make sense to also consider classification error experiments on MNIST and CIFAR-10 with non flow-based models included as baselines as well?\n\nIt is possible that I did not understand the experiment design properly, and am willing to adjust my score accordingly.", "limitations": "Yes, I think the authors adequately address the limitations and impact of the proposed flowification in terms of its performance and the possible causes for it, and how it enables us to see standard architectures as subsets of normalizing flows. However, it would also be great to see potential negative societal impact be addressed in the paper (if there is anything specific), which I don't see as being addressed in the current version.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\n- The proposed formulation for unifying general architectures and normalizing flows looks mathematically elegant and sound. It would make it possible to realise the benefits of flows for general layers, as the authors experimentally demonstrate as well.\n- The paper is clear and well-written-- the intuition provided for the definitions and theorems is helpful to understand them. \n\n*Weaknesses*\n\n- There are experiments for flowified convolutional architectures, but I couldn't find any experiments for flowified linear layers. Wouldn't it make sense to see them as well?\n- It would make it clearer to have a representational diagram for a dimension-increasing flowified linear layer as well to get an overview.\n- Although it's interesting to see flowified CNNS, their performance is not at the level of flow-based models, which is recognised by the authors as well. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7l45Cqv3zsv", "reviewer": "Reviewer_rJok", "summary": "The paper introduces a method that transforms any multilayer perception into the flow model. The authors investigate various scenarios where they apply linear and convolutional layers as transformations inside the flow, either reducing or extending the dimensionality of the data. Some experiments on MNIST and CIFAR are performed to investigate the quality of the approach. ", "questions": "I would like to ask the authors to refer to the remarks from the Weakness section, mainly focusing on:\n\n1. Relation to SurVAE approach, how the surjective for inference or generation are designed, and how sampling for inference is performed.\n2. The concerns about notations and the design choices (eq. (7), $\\phi(z|x)$)\n3. Relation to probabilistic PCA that uses the linear transformation.  \n ", "limitations": "The limitations were included in the conclusion section. ", "rating": 4, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n\nThe defined problem and direction are important for the committee. The idea of searching for novel forms of transformations and connections with existing models seems to be attractive. \n\nWEAKNESSES\n\nI read the paper several times, and I had a problem identifying the contribution of this work. The title and the abstract suggest rather that we should expect to transfer standard MLPs into Bayesian networks using flows. In my opinion, the proposed solution rather enriches the family of transformations for SurVAE flows: \n\nNielsen, D., Jaini, P., Hoogeboom, E., Winther, O., & Welling, M. (2020). Survae flows: Surjections to bridge the gap between vaes and flows. Advances in Neural Information Processing Systems, 33, 12685-12696.\n\nwith linear and convolutional transformations. Assuming this, it is not clear if the proposed transformations are surjective for inference or generation. Assuming we consider the surjective inference, how the sampling process is implemented. \n\nThe paper is difficult to follow, the math transformations are difficult to follow, the notation seems to be unexplained. For instance:\n- eq. (7) it is not clear what $p_{inv}$ is and why it is not expressed in log domain. What is $p(z)$ here. \n- The choice of $\\phi(z|x)$ is also not discussed in the paper. \n- matrix $\\Sigma$ should have positive values on diagonal I Guess. \n\nThe application of the linear layer is equivalent to the application of probabilistic PCA, where the likelihood can be easily calculated in the closed-form: $p(x)=\\int p(x|z)p(z)dz$ can by calculated analytically. \n\nThere are no clear advantages of provided transformations compared to the proposed in SurVAE. Authors should discuss using the benefits of their approach compared to an existing one. \n\nThe clarity of writing and consistency with previous works regarding notation should be improved. Instead of \\emph{Sketch of proof} the exact proof should be provided in the appendix, instead of above $\\mathcal{L}$ the reference to the definition should be delivered. Moreover, using $\\log \\Sigma$ is further explained but practically means another type of operation. \n\nMy primary concern is about the experimental part. For such works, achieving SOTA is not crucial, but the results are far from baselines. There are also no toy examples that show the benefits of the approach for particular assumptions. \n\nConcluding, the paper seems to be a bit of a work in progress and should be reorganized, and additional experimental evaluation should be included.  \n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept