On the Invertibility of Invertible Neural Networks
Authors: Jens Behrmann
Paul Vicol
Kuan-Chieh Wang
Roger B. Grosse
Jörn-Henrik Jacobsen
Venue:  (2020)
Keywords: Invertible Neural Networks
Stability
Normalizing Flows
Generative Models
Evaluation of Generative Models

Abstract: Guarantees in deep learning are hard to achieve due to the interplay of flexible modeling schemes and complex tasks. Invertible neural networks (INNs), however, provide several mathematical guarantees by design, such as the ability to approximate non-linear diffeomorphisms. One less studied advantage of INNs is that they enable the design of bi-Lipschitz functions. This property has been used implicitly by various works to design generative models, memory-saving gradient computation, regularize classifiers, and solve inverse problems. 
In this work, we study Lipschitz constants of invertible architectures in order to investigate guarantees on stability of their inverse and forward mapping. Our analysis reveals that commonly-used INN building blocks can easily become non-invertible, leading to questionable ``exact'' log likelihood computations and training difficulties. We introduce a set of numerical analysis tools to diagnose non-invertibility in practice. Finally, based on our theoretical analysis, we show how to guarantee numerical invertibility for one of the most common INN architectures.

Under review as a conference paper at ICLR 2020
ON THE INVERTIBILITY OF INVERTIBLE NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Guarantees in deep learning are hard to achieve due to the interplay of flexible modeling schemes and complex tasks. Invertible neural networks (INNs), how- ever, provide several mathematical guarantees by design, such as the ability to approximate non-linear diffeomorphisms. One less studied advantage of INNs is that they enable the design of bi-L.ipschiz functions. This property has been used implicitly by various works design generative models, memory-savinne gradient computation, regularize classifiers, and solve inverse problems.
In this work, we study Lipschitz constants of invertible architectures in order investigate guarantees on stability of their inverse and forward mapping. Our analysis reveals that commonly-used INN building blocks can easily become non- invertible, leading to questionable "exact" log likelihood computations and training difficulties. We make use of numerical analysis tools to diagnose non-invertibility in practice. Finally, based on theoretical analysis, we show how to guarantee numerical invertibility for one of the most common INN architectures.
INTRODUCTION
Invertible neural networks (INNs) have become a standard building block in the deep learning toolkit.
Invertibility useful for training generative models with exact likelihoods (Dinh al., 2014 2017]
Kingma & D 2018,) Kingma et al. 2016) Behrmann et al., 2019 [Cen et al. 2011)). increasing posterior flexibility in VAES (Rezende & Mohamed 20155] Tomezak & Welling 2016 Papamakarios
al. 2017). learning transition operators in MCMC samplers (Song al.) 2017 Levy et al. 2017).
computing mnemory-efficieee gradients (Gomez et al., [017] Donahue Simonyan [2019), allowing
for bi-directional training (Grover et al., 2018), solving inverse problems (Ardizzone et al. 2019 and
analysing adversarial robustness (Jacobsen et al. 2019).
The application space of INNs is rapidly growing and many approaches for constructing invertible architectures have been proposed. A common way to construct invertible networks is to use triangular coupling layers (Dinh al. 2014 2017 Kingma & Dhariwal (018). where dimension partitioning is interleaved with ResNet-type computation. Another approach is to use various forms of masked convolutions, generalizing the dimension partitioning approach of coupling layers (Song et al. 2019 Hoogeboom et al., 2019). To avoid dimension partitioning altogether, multiple approaches based on efficiently estimating the negdeetrrmnant of the Jacobian, necessary for applying the change of variable formula, have been proposed to allow for free-form Jacobian structure (Grathwhhl et al. 2019 Behrmann et al. 2019 Chen al. 2019).
From a mathematical perspective, invertible architectures enable several unique guarantees like:
Enabling flexible approximation of non-linear diffcomorphisms (Rezende & Mohamed
2013 Dinh et al. 2017 Kingma & Dhariwal 2018 Chen et al. 2019)
Memory-saving gradient computation (Gomez et al. 2017 Donahue &imonyyn 2019)
Fast analytical invertibility (Dinh et al., 2014)
Guaranteed preservation of mutual information and exact access to invariants of deep
networks (Jacobsen et al. 2018 2019).
Despite the increased interest in invertible neural networks, little attention has been paid to guarantees on their numerical invertibility. Specifically, this means analyzing their ability learn bisciit

---

Under review as a conference paper at ICLR 2020
neural networks, i.. Lipschitz continuous neural networks with bound the Lipschitz constant of the forward and inverse mapping.
While the stability analysis of neural networks has received significant attention e.g. due to adversarial examples (Szegedy et al. 2013). the focus here is only on bounding Lipschitz constants of the forward mapping. However, bounding the Lipschitz constant of the inverse mapping is of major interest, e.g. when reconstructing inputs from noisy or imprecise features. In fact, analytical invertibility as provided by some invertible architectures does necessarily imply numerical invertibility in practice.
In this paper, first discuss the relevance of controlling the bi-Lipschitz bounds of invertible networks. Afterwards we analyze Lipschitz bounds of commonly used invertible neural network building blocks. Our contributions are:
We argue for forward and inverse stability analysis unified viewpoint invertible network (non-) nonn)jinerttiiility. To this end, we derive Lipschitz bounds of commonly-use invertible building blocks for their forward and inverse maps.
We numerically monitor and detect (non-) niveeriivttib for different practical tasks such as classification and generative modeling.
We show how this overlooked issue with non-invertibility can lead questionable claims when computing exact likelihoods with the change-of-variable formula.
Finally, study spectral normalization stabilizer for one of the most commonly-used family INN architectures, namely additive coupling blocks.
BACKGROUND AND MOTIVATION
Invertible neural networks are bijective functions with a parametrized forward mapping Fg Rdd - Rd with Fa } 2, where E RR defines the parameter vector. Additionally, they define inverse mapping F3 Rd + Rd with F3 2 + r. This inverse can be given in closed-form (analytical inverse, e.g. Dinh al. (2017) Kingma Dhariwal (2018)) approximated numerically (numerical inverse, e.g. Behrmann et al. 2019 Song et al. (2019)).
Before we discuss building blocks of invertible networks, we provide some background and motivation for studying forward and inverse stability.
Definition 1 (Lipschitz and bi-Lipschitz continuity). A function F (IRd,, -I-10 II) - (Rd, 11-,,11-10 is called Lipschitz continuous if there exists constant L Lip(F) such that
F F( - F(x2))l S L| æ2l1. Van, 22 E Rd.
an inverse F-1; (IRde, 1|-11 - (Rd || Il) and constant L Lip(F-1) exists such that
| |
- - s L well. Vy,. 2/2 E 12
then F is called bi bi-Lipschitz continuous.
Remark 2. We focus on invertible functions F (Rd, 11 1122 (Rd, I| l12), i.e. functions where the domain and co-domain are of the same dimensionality d and the norm is given by the euclidian norm.
Lemma 3. (Rademacher (Federer 1969 Theorem 3.1.6))
F Rd + IRd is a locally Lipschitz continuous function (i.e. functions whose restriction neighborhood around any point is Lipschitz), then F is differentiable almost everywhere. Moreover, F Lipschitz continuous, then
Lip(F)- suu ||
where Jr(x) is the Jacobian matrix of F at x and [|Jr(x)]l denotes its spectral norm.
Lipschitz bounds on the forward mapping are of crucial importance in several areas, including in adversarial example research (Szegedy al. 2013). avoid exploding gradients, or the training of Wasserstein GANs (Anil et al. 2019). The stability of the inverse, however, can have a similar impact. For instance, having a Lipschitz bound the inverse may avoid vanishing gradients during training.
2

---

Under review as a conference paper at ICLR 2020
Given that deep-learning computations carried out with limited precision, imprecision is always
introduced in both the forward and backward passes, i... 2 = (z) + å and 1 = F (2") Instability in either pass will aggravate this problem, and essentially make the invertible network numerically non-invertible. To summarize, this problem occurs in the following situations:
Numerical reconstruction x, where features 2 are inexact due to limited precision (e.g. when computations are executed in single precision common modern hardware). Reconstruction based imprecise measurements from physical devices (e.g. when using invertible networks for inverse problems (Ardizzone et al. 2019)).
Numerical re-computation of intermediate activations of the neural network to allow for memory-efficiet backpropagation (Gomez et al. 2017).
Furthermore, some computations are performed via numerical approximation, which in turn adds another source of imprecision that might be aggravated via instability. Examples include:
Numerical forward computation, as in Neural ODEs (Chen et al. 2018 (numerical solver is used to approximate dynamic of ODE).
Numerical inverse computation, e-g. via fixed-point iterations in i-ResNets (Behrmann) et al. 2019) or MintNet (Song et al. (2019) or via ODE-solvers for the backward dynamics
as in Neural ODEs (Chen et al.) 2018).
As an example of why bi-Lipschitz continuity is critical for numerical stability in invertible functions, let's consider the simple mappings F1 (x) log(z), FT (z) = exp(z), and (2 x, Pg (=) 2. Though both functions tend to infinity when x + co. Fi much less stable. Consider the introduction of numerical imprecision as 2 (i) + 8 where denotes the introduced imprecision. Then this imprecision magnified in the inverse pass as:
OFT 056
(2) -
(= (2 ~
12 = | exex(==))11
A similar example can be constructed for both the forward and backward passes, which speaks to the importance of bi-Lipschitz continuity. For an additional discussion on the connection of Lipschitz constants and numerical errors, we refer to Appendix
3 STABILITY OF INVERTIBLE NEURAL NETWORKS
3.1 LIPSCHITZ BOUNDS FOR BUILDING BLOCKS OF INVERTIBLE NETWORKS
Research on invertible networks has produced a large variety of architectural building blocks. Yet, the focus of prior work was on obtaining flexible architectures while maintaining invertibility guarantees. Here, we build on the work in (Behrmann et al., 2019), where bi-Lipschitz bounds were proven for invertible ResNets, by deriving Lipsehitz bounds the forward and inverse mapping of common building blocks. Together with an overview of common invertible building blocks, we provide our main results Table We chose these particular model classes in order cover both coupling- -bsed approaches and free-form approaches like Neural ODE (Chen et al. 2018) and i-ResNets (Behrmann et al. 2019). The derivations of the bounds are given Appendix[A] Note that the bounds provide the worst-case stability and serve mainly as a guideline for future designs of invertible building blocks.
3.2 CONTROLLING STABILITY OF BUILDING BLOCKS
As shown in Table [ there are many factors that influence the stability of INNs. Of particular importance are the Lipschitz constants Lip(g) of the sub-network g for i-ResNett (Behrmann et al. 2019) and affine coupling blocks (Dinh et al. 2014), and Lip(s), Lip(!) for additive coupling blocks (Dinh et al. (017). Whereas computing the Lipschitz constants of neural networks is NP-hard Virmaux & Scaman 2018), there is simple data-independent upper bound:
Lip(g)= <
A for g(x) = A ooo AL-1 o...o Ag o A
-=1
3

---

Under review as a conference paper at ICLR 2020
Building Block
Forward Operation
Lipschitz Forward
Lipschitz Inverse
Additive Coupling Block (Dinh ct al. 2014
F(a), 21, F(a), 21, + +(()))
$ +1+ Lip(g)
S 1 Lip(g)
Affine
F(a),,
S max(1,,,)
c2) M
S max(11,y) o4) M
Coupling Block Dinh et al. 2017
F(2)22 = 21, o g(s((1,)) + +((x))
local for x 6 [o,
local for é ja","" (0) S cg
8(-) #0
g(x) S Cg.
Invertible
F(x) - 9(2) Lip(g) <
$ 1 Lip(g)
S 1 Lipg)
Residual Laver
[Behrmann et al. 2019
Neural ODE
da(t) t()(().t)
Chen al. 2018
E [0.T]
Invertible
F(a) Pr P permutation
Downsampling Squeeze (Dinh et al. 2017
Diagonal Scaling
F(x) Dr
max; Dal
min Dii
Dinh ct al. 20144
1) diagonal Da # 0
Norm
Kingma & Dhariwal 2018
Invertible X Convolution
F(z) = PLU + diag(s)) =: W P permutation, L lower-triangular U upper-triangular, e R
S [|
IW
(Kingma & Dhariwal. 2018
Table 1: Lipschitz bounds on building blocks of invertible neural networks. The second column
shows the operations of the forward mapping and the last two columns show bounds on the Lipschitz constant of the forward and inverse mapping. M in the row for the forward mapping of an affine block is defined as M = max([al, [6]) Cgy Lip(s) + Lip(t). Furthermore, M for the inverse of an affine block is MM = max(la"), I. [b*]] '())* Lip(s) + 'gy Lip(s) ct + " Lip(t). Note that
the bounds of the affine blocks hold only locally. Derivations of the bounds are given in Appendix A
where A, linear layers, || ||2 is the spectral norm and á contractive activation function (Lip()) S 1). The above bound was used by (Behrmann et al. 2019) in conjunction with spectral normalization (Miyato et al. 2018 Gouk et al. 2018) ensure contractive residual block g. In particular, this employs a normalization via:
À Ā= Á= A
with 6 0
||ll2 (approx. via power-method),
where >> O is coefficient that sets the approximate upper bound the spectral norm of each linear layer A,. Thus, by setting an appropriate coefficient i depending on the targeted Lipschitz bound of the building block, this approach enables one control both forward and inverse stability. Note that the above discussion can be generalized to other -n-nrmss, see (Chen et al. 2019).
However, this is not sufficient when using affine coupling blocks because their bound the Lipschitz constant holds only locally. In particular, it depends on the regions of the inputs I to the coupling block. While inputs to the first layer are usually bounded by the nature of the data, obtaining bounds for intermediate activations is less straightforward. One interesting avenue for future work could be local regularizers like gradient penalties (Gulrajani al. 2017), where spectral normalization could be used post-hoc to certify stability.
Lastly, we use ActNorm (Kingma & Dhariwal 2018) in several architectures and avoid small diagonal terms which would yield large Lipschitz constants in the inverse (see Table [T by adding positive constant. Further stabilization could be achieved via bounding the scaling.
NUMERICAL EXPERIMENTS
In this section, we study numerical invertibility for several objectives and architecture settings. This section is structured by task:
4

---

Under review as a conference paper at ICLR 2020
1. Classification: show that INN classifiers become non-invertible on CIFAR- 10 and discuss consequences for memory-efficient backpropagation.
2. Density estimation: we analyze the numerical invertibility of SOTA trained density models.
3. Generative modeling: we study the stability of adversarially trained INN generators, discuss the consequences for likelihood evaluation, and stabilize an additive-couplin based INN generator using spectral normalization.
4. Decorrelation: we perform an in-depth study of the effect of different architecture settings on a simple task, where both stable and unstable solutions are possible. Furthermore, we show that spectral normalization is effective stabilizing additive-coupling based flows.
We use the following measures to diagnose the numerical instability of invertible models:
Reconstruction error. We measure the &g-distance between the input x and its reconstruc- tion, i.e. ||xx(0 F (Fo(
Conditioning of the Jacobian and max/min singular values. For forward stability, we are interested in the behavior of the Jacobian Jr(z), while for inverse stability interested in the Jacobian of the inverse mapping JÆ-1 (rr). We compute the singular values of the Jacobians using the SVD, which allows to compute its condition number. !
While the reconstruction error allows us to quantitatively monitor non-invertibility even before reconstruction artifacts are perceptible, the linear approximation Jr(r) and its singular values provide insights into unstable directions of the forward (very large singular values of ./p(z)) and inverse (very small singular values of JF (x)) mapping. Both measures were also used in (Jacobsen et al. 2018), where ill-conditioned inverse was observed.
4.1
CLASSIFICATION WITH INVERTIBLE MODELS
In this section, we show that when training an INN for clas- sification, there exist both stable and unstable solutions. We compare stable model that uses additive coupling and an unstable model that uses affine coupling and ActNorm both inside and between the blocks- for additional ex- perimental details see Appendix C These models achieve similar test accuracies of 90.2% and 90.5%, respectively (Figure 6 Appendix C however, we note that the goal of this experiment is not to achieve SOTA accuracy CIFAR-10. Rather, we aim to show that models trained for classification with reasonable accuracy, can vary greatly with respect to stability. To observe the differences in stability, we plot the reconstruction results in Figure 2
35
Additive
30
Affine
20
Angle (in
15
0
50
100
150
200
Epoch
Figure 1: Impact of stability on memory-efficient training. We mea- sure the angle between the true gra- dient (using stored activations) and the memory-saving gradient (using re- computing activations). For all epochs after the dotted vertical line, the affine model had numerically infinite or nan gradients, while the additive model gives accurate memory eeemory-efficintt gradients.
An important use-case of INNs is to enable memory- efficient training. by re-computing activations in the back- ward pass rather than storing them in memory during the forward pass (Gomez et al. 2017). This approach en- ables e.g. large-scale generative modeling (Donahue & Simonyan 2019) and scaling segmentation networks high-resolution medical images (Brügger et al.) 2019).
Re-computing the activations, however, relies on numerically precise inverse mapping. To better understand the effect of numerical errors, we perform an analysis similar to Gomez. et al. (2017): we track the angle between the true and memory-saving gradients during training (Figure []. As expected from the reconstruction results, we observe that the affine model yiclds gradicnts very different from the true gradient; in fact, after approximately 20 epochs of training, the memory-saving gradients of the affine model contain numerically infinite the affine model successfully using memory-saving gradients.
values. Thus, it would not be possible train
Using the SVD is feasible for CIFAR-10, 10, but becomes prohibitively expensive for ImageNet (with images of size 3 x 256 x 256); for such larger images, one could instead use the Lanczos algorithm (I [lanzzos] 1950) to find the largest and smallest singular values, compute the condition number.
5

---

Under review as a conference paper at ICLR 2020
Original Images
Additive
Recons, Error
10"2
10"3
R
10"4
437
1240
8185
Omaa 40
maa = 50
mas = 215
0k
20k
40k Iteration
60k
80k
min 9e-2
min 5e-2
min 3e-2
Affine + ActNorm
f
Recons, Error
10c
10*
102
R
10
10
c= 33188
c= 1.8e8
9.3e13
mma = 42
0 maz = 143
0 maz = 560
0k
20k
40k
60k
80k
Iteration
0 mann le-3
0 min 7
min 6 12
Training
Figure Measuring stability additive and affine INN classifiers on CIFAR-10. Right: we track the reconstruction errors for fixed minibatch over the course of training. In each iteration (vertical slice) we plot the errors each minibatch element. Orange points in the affine plot denote inf and nan values. denotes the condition number of the Jacobian, mmaz and main denote the max and min singular values. At the top, we show the original images x and in each snapshot, the reconstructions f = F-1 (F(x)) and reconstruction errors
2
4.2 ANALYZING STATE OF- ART DENSITY MODELS
Figure 3: Crafting non-invertible inputs for CIFAR-10 Glow model. For three different images, we show: 1) the original datapoint from which we start running PGD: 2) the crafted input x' that results from PGD; and 3) the reconstruction V' = F (F((')) of the crafted input. In cases, the reconstructions of adversarial inputs are heavily corrupted, indicating that the attacks were successful, and exposing non-invertibility in this Glow model.
In this section, we analyze the invertibility of trained density models. In particular, we expose non-invertibility in models that otherwise appear stable by optimizing input space to find examples that are poorly reconstructed by the model. Here, we take a trained Glow model (Kingma & Dhariwal 2018 and optimize the imput using Projected Gradient Descent (PGD) (Madry et al. 2018). Our goal to find point x' in the domain of the invertible model such that the reconstruction F FF'(F(x') differs from z'. In particular, we start with a datapoint æ and use PGD to find perturbed example x' that has high reconstruction error via Eq. (additional details in Appendix Eb.
arg max || -
||ا - (F
this attack is effective for finding examples that are perceptually identical to test examples, yet induce large reconstruction errors. This attack can be understood as a worst- case invertibility diagnosis; however, note that unsuccessful attacks can be due to algorithmic
As shown in Figure 3
We used the PyTorch implementation from h PyTrchh This
trained model achieves likelihood of 3.39 bits-per-dimension.

---

Under review as a conference paper at ICLR 2020
issues and thus do not necessarily imply stable invertible models. Also, it is not always clear how to get gradients that are able to exploit numerical instabilities, leaving room for improvement via gradient-free free methods (c.g., the boundary attack from Brendel et al. (2018)).
We present Appendix E additional experiments using the PGD attack on the additive Glow model from (Kingma & Dhariwal 2018) trained CelebA, where show that the model becomes non-invertible outside the valid input range of images. Furthermore, we analyze a trained residual flow model (Chen al., 2019)) on CIFAR-IO, where we cannot find such dramatic non-invertible inputs, which is to be expected given the stability bounds of i-ResNet blocks (see Table IT.
4.3 GENERATIVE MODELING WITH INVERTIBLE MODELS
Generative models based on invertible networks have been predominantly trained using maximum likelihood estimation (MLE). Another viable approach is to train them adversarially (ADV), ) as done in Flow-GAN (Danihelka et al. 2017 Grover et al. 2018). Flow-GAN ís appealing as it can result in generator capable of producing high-quality samples (as in GANs). while also giving access to exact density estimates, which GANs lack. Prior work (Danihelka et al. 2017 Grover al. 2018) has compared these two techniques for training flows (MLE vs ADV); the main conclusion of these studies that training with MLE yields good likelihoods but relatively poor samples, while training with a GAN loss yields good samples but likelihoods orders of magnitude worse than MLE training.
MLE Affine
MLE Additive
ADV Additive Unstable
ADV Additive Stable
(Test) BPD FID
1.22
1.38
3.87e12*
709
99.9
95.1
25.3
195
Table 2: Comparison bits- bits-eer-dimensoon (BPD) and sample quality via FID scores for MLE- and ADV-trained models. For reference, the FID of untrained Flow is roughly 1500. Using the stable version with ADV, though improves BPD significantly, might come at a trade-off on FID.
Here, we analyze in depth the effect of Flow-GAN training on numerical stability. We use net- works with repeated additive coupling layers, and ActNorm between blocks. We examine two architectures, both with 3 levels, i.e., 'squeeze' between levels (additional details Appendix D
Stable: having a depth of 4 (i.e., 4 blocks per level) and spectral normalization applied to all the convolution layers, see section 3.2 for details.
Unstable: having depth of 16, ActNorm within coupling layer, and no spectral normaliza- tion applied to the convolutional layers.
Table shows that models trained with MLE objective can achieve good bits-per-dii bitt-per-iimenstion (BPD), but models trained with ADV can achieve better sample quality measured by the Frechet Inception Distance (FID), a common measure of sample quality (Heusel et al., 2017 Lucic et al. (0188). This justifies why considering training INN with objective functions other than MLE is desired.
Broken Flow-GAN. In Figure 4 we show that an INN trained only with adversarial loss can become non-invertible. depending on the architecture. We perform forward and inverse passes repeatedly on the same mini-batch. The unstable model shows visible reconstruction errors quickly. Table shows that the unstable model has BPD orders of magnitude larger than the stable model.
(Stable) Recons
(Stable) Errors
(Unstable] Recons (Unstable) Errors
Recon Errors
100
7
2
O
4
7
2
4
Stable
7
2
0
4
?
Unstable
-0
7
2
0
4
7
2
0
4
:
3
-20
7
2
0
4 (Stable) Samples
Recon Pess
(Unstable) Samples
L
7
9
9
7
4
9
5
2
9
8
2
3
2
6 3
Figure 4: Reconstructions of images through multiple forward/inverse passes. First row in the 'Recons' subfigures real-data, and each row after is one forward/inverse pass.
7

---

Under review as a conference paper at ICLR 2020
Model
(Test) BPD
Max-SV
Min-SV
Cond-Num
Analytic LDJ
ADV (Stable)
709
1.99c3
0.6129
3.253c3
4734
ADV (Unstable) 3.87e12*
3.83e8
0,0309
1.239e10
9874
Table 3: Stability analysis on two architectures trained adversarially (ADV), and with maximum likelihood (MLE). *means this number is meaningless as the network is visibly non-invertible. LDJ denotes the log- bgg-ettemminat of the Jacobian, BPD denotes bits-pett sits-per-dimenionn and SV singular value.
Flow-GAN "Likelihood". Typically likelihood is computed by a change of variables, which assumes invertibility. When network is numerically non-invertible, the assumption breaks, and the computed value becomes some numerical approximation the density. The model used Figure consists of additive coupling blocks, ActNorm, and squeezing operations. All these operations have data independent log -og-determinant Jacobian. Thus, obtaining a numerical value via the change of variable formula is straightforward. However, it is unclear what this numerical value represents, likely cannot be trusted as true likelihood due to the lack of invertibility. In this case, we advocate for ensuring invertibility using the remedies discussed here make sure BPD values In terms of sample quality, however, we currently observe a tradeoff since the stable model yields higher FII) scores than the unstable model (Table 2 includes also MLE-models for comparison). Yet, we believe that proper tuning c.g. of spectral normalization could remove this tradeoff.
trustworthy.
Lastly, one can adjust for the likelihood by adjusting the prior (see Appendix I}. In summary, in this section we point out that Flow-GAN can become non-invertible. in which case the computed likelihood cannot be taken as ground truth likelihood (Grover al., 2018). sum, INN trained with MLE are stable, but the sample quality is worse than those trained with ADV. Yet, training with ADV loss might make INN non-invertible, which defeats the purpose of using INN in the first place. Hence, when training with alternative objectives, numerical stability is a crucial property to consider.
4.4 DECORRELATION TASK
As the last part of our empirical study, we use simple decorrelation task to benchmark the stability of invertible models. particular, we compute the correlation matrix C' E Rdxd via
1 N
(Fa(z()) - -as) (Fo( pi))
C2
(4)
where jiy is the estimated Then, we optimize the parameters 0 to minimize the off-diagonal correlation, i.e.
output samples Fo (a)(()) and Oj the estimated standard deviation.
min ||C* cliag(C)))
(5)
where || || is the Frobenius-noma Decorrelation objectives have been used in (Cogswell et al. 2015) to reduce overfitting and in (Cheung al. 2014) to disentangle hidden activations.
This objective serves as a good task for our purposes for two reasons:
1. Decorrelation is simpler objective than optimizing outputs 2 (z) to follow factorized Gaussian as in Normalizing Flows (Rezende & Mohamed] (2015). Furthermore. it will show that changing the objective to less standard tasks can lead to larger instabilities compared to using INNs on more common tasks such as density estimation.
2. Decorrelation allows multiple solutions using invertible mappings, where both stable and unstable transforms are equally valid for the given objective. See Appendix(Fjoor a motivation based simple 2D toy example.
In summary, the decorrelation objective offers an environment to study which INN components steer the mapping towards stable or unstable solutions, that are equally plausible for the given task.
In our experiments shown in Figure 5 we focus on coupling- based models like Glow (Kingma & Dhariwal 2018). which are analytically invertible and thus allow simpler analysis compared to models relying on numerical inversion like i-ResNets (Behrmann et al., 2019). We evaluate the effects of different architectural choices on numerical stability, including additive vs. affine coupling layers, ActNorm, and architecture depth. For ActNorm we study two settings: 1) between coupling blocks, 2) inside blocks, i.e. as part of the function g in additive blocks or s and in affine blocks. Details on the architectures/ training schemes and extended results are provided in Appendix H
We provide example PyTorch code for the decorrelation objective in Appendix
8

---

Under review as a conference paper at ICLR 2020
Original Images
Affine: ActNorm Inside and Between Blocks
10a
Recons Error
R
10-1
10-3
c= 2.6e4
c= 5e10
== 1.4e12
mar = 6.1
max 8.1
maz 1.7e2
mmin = 2.3e-4
min = 1.7e-10
min = 1.3e-10
Iteration
Affine: No ActNorm
Recons, Error
10
200
R
10a
100
c= 4e2
lc7
3e11
10-3
0maz = 2e-1
mas = 2e-1
man = 2e-1
Mmin Se-4
min 2e-8
miin 8e- 13
Iteration
Additive: ActNorm Inside and Between Blocks
1010
Recons, Error
10r
R
10a
c=
c= 3c10
c=
10"2
mmaz 121
maa 2e5
maz 2e6
Omin 4e-3
0 mmin 9e-6
m 6e-7
Iteration
Additive: No ActNorm
Recons. Error
102
R
10:
10-1
c= 8e4
c= 2c13
c= 3e13
10:2
Omax
maz 9e5
maa 5e7
0k
60K
120k
180k
Omin = 5e-4
mii 5e-8
Jmin 2
Additive: No ActNorm Spectral Norm
Recons. Error
R
16
10-5
Umar = 1.9
maz = 4.6
0 Maaz = 4.4 Gmin 0.2
0k
60k
120k
180k
Umin 0.1
0 Min 0.1
Iteration
Training
Figure 5: Instability in affine and additive models. On the right, we track the reconstruction errors for a fixed minibatch over the course training: each iteration (vertical slice) we plot the errors of each minibatch element. Thus, we can observe the distribution of errors. Orange points in the affine plots denote inf and nan values. с denotes the condition number the Jacobian, o mmaz and min denote the maximum and minimum singular values, respectively. At the top, show the original images x and in each snapshot, the reconstructions & F (F(()) and reconstruction errors R |x: -|- Note that the x- and y-axes differ for different settings, the models become unstable at different points during training.

---

Under review as a conference paper at ICLR 2020
5 RELATED WORK
Invertibility and stability of deep networks. The inversion from activations in standard neural networks to inputs has been studied in various works, e.g. via optimization in input space (Mahendran & Vedaldi 2014). Linking invertibility and inverse stability for relu-networks was e.g. done in Behrmann et al. (2018). However, few works study the stability of INNs: Gomez et al. (2017) study the numerical errors in the gradient computation when using their memory-efhciree bac backpropagation variant. Similarly to our empirical analysis, (Jacobsen et al. 2018) computed the SVD of the Jacobian of trained i- N-RevNet and observed ill-conditioned Jacobian. Lastly, the i-ResNet architecture (Behrmann et al. 2019) yields bi-L.isschiz bounds by design.
On the other hand, the stability of neural networks has been of major interest due the problem of exploding and vanishing gradients, and more recently due to adversarial examples (Szegedy et al.) 2013) and training of Wasserstein GANs (Arjovsky et al. 2017). See e.g. (Anil et al., 2019) for promising approach to learn flexible Lipschitz neural networks.
Invertible building blocks. Besides the invertible building blocks we studied in Table several other approaches were proposed. Most prominently, autogressive models like MAF (Papamakarios ict al., 2017) or IAF (Kingma et al., 2016) provide invertible models that are not studied our analysis. Furthermore, several newer coupling layers that require numerical inversion have been introduced (Jaini et al.) 2019 Durkan et al. 2019). Besides the coupling- -ased approaches, multiple approaches (Chen al. 2018: Behrmann et al. 2019: Chen et al., 2019: [ong et al. (019) use numerical inversion schemes, where the interplay of numerical errors due to stability and errors due to the numerical approximation of the inverse adds another dimension the study of invertibility.
Fixed-Point arithmetic and limited precision. Maclaurin et al. (015): MacKay et al. (2018) implement invertible computation using fixed-point numbers, with seecially-designnn schemes to store information that is "lost" when bits are shifted due to multipiicationddivsioon enabling exact invertibility at the cost of additional memory usage. As Gomez et al. (2017) point out, this approach allows exact numerical inversion when using additive coupling blocks independent of stability issues. However, our stability analysis aims for a broadly applicable methodology beyond the special case of additive coupling. Lastly, there may be connections to deep learning using limited precision, e.g. (upta et al. 2015). which could provide more insights into our observed numerical errors.
6 CONCLUSION
Numerical instability is an important concern for the practical application of invertible models. If for instance analytical invertibility does not carry through to the numerical computation due to instabilities or numerical errors, the consequences can be arbitrarily severe. As shown in our experiments, this can impact memory-efficient backpropagation (Gomez et al. 2017) and thus significantly reduces the usability of invertible networks if not handled appropriately. Flow-GAN illustrates another application where non- -oneinvettibility poses a serious threat, as instabilities can strongly influence or even break ikelihhodd-computation..
In this paper, we shed light on the underlying causes of instability by deriving Lipschitz bounds on many of the atomic building blocks commonly used construet INNs. From practical standpoint, we used diagnostics to measure stability and provided an empirical framework to benchmark stability. Further, have shown how to guarantee stability for one of the most common INN architectures. We hope that this will inspire future work to view numerical stability as a crucial axis in the design of new building blocks and architectures for invertible neural networks.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Cem Anil, James Lucas, and Roger Grosse. Sorting out Lipschitz function approximation. Interna- tional Conference on Machine Learning, 2019.
Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf's S Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. International Conference on Learning Representations, 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, 2017.
U.M. Ascher. Numerical methods for evolutionary differential equations. Computational Science and Engineering. Society for Industrial and Applied Mathematics, 2008.
Jens Behrmann, Sören Dittmer, Pascal Fernsel, and Peter Maaß. Analysis of Invariance and Robust- ness via Invertibility of ReLU-Networks. arXiv preprint arXiv:1806.09) 09730, 2018.
Jens Behrmann, Will Grathwohl, Ricky 'T Q. Chen, David Duvenaud, and Jömn-Henrik Jacobsen. Invertible residual networks. International Conference on Machine Learning, 2019.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-baseee adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Representations, 2018.
Robin Brügger, Christian F. Baumgartner, and Ender Konukoglu. A partially reversible u-net for memory-efficieee volumetric image segmentation. In Medical Image Computing and Computer Assisted Intervention - MICCAI, 2019.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 2018.
Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, and Meen-Kerrik Jacobsen. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 2019.
Brian Cheung, Jesse A Livezey, Arjun K Bansal, and Bruno A Olshausen. Discovering hidden factors of variation deep networks. arXiv preprint arXiv:1412.6688, 2014.
Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfit- ting in deep networks by decorrelating representations. arXiv preprint arXiv: 1511,00068 2015.
Ivo Danihelka, Balaji Lakshminarayanan, Benigno Uria, Daan Wierstra, and Peter Dayan. Com- parison of maximum likelihood and GAN-based training of real NVPs. arXiv preprint arXiv: 2017.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. arXiv preprint arXiv:1410.88 10.85 16, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. Interna- tional Conference on Learning Representations, 2017.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. arXiv preprint arXiv:1907.02544, 2019.
Conor Durkan, Artur Bekasov, Ian Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, 2019.
H. Federer. Geometric measure theory. Grundlehren der mathematischen Wissenschaften. Springer, 1969.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In Advances Neural Information Processing Systems, pp. 2224-2224, 2017.
11

---

Under review as a conference paper at ICLR 2020
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael Cree. Regularisation of neural networks by enforcing Lipschitz continuity. arXiv preprint arXiv: 1844v00804,0436, 2018.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, llya Sutskever, and David Duvenaud. FFJORD: Free-form continuous dynamies for scalable reversible generative models. International Conference on Learning Representations, 2019.
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-GAN: Combining maximum likelihood and adversarial learning in generative models. In Thirly-Second AAAI Conference on Artificial Intelligence, 2018.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, 2017.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning. 2015.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 8626-6637, 2017.
Emiel Hoogeboom. Rianne van den Berg, and Max Welling. Emerging convolutions for generative normalizing flows. International Conference on Machine Learning. 2019.
Jörn-Henrik Jacobsen, Arnold Smeulders, and Edouard Oyallon. i-RevNet: Deep invertible networks. International Conference on Learning Representations, 2018
Jörn-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability. International Conference on Learning Representations, 2019.
Priyank Jaini, Kira Selby, and Yaoliang Yu. Sum-of-squares polynomial flow. In International Conference on Machine Learning, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pp. 4743-4751, 2016.
Comelius Lanczos. An iteration method for the solution of the eigenvalue problem of linear differ- ential and integral operators. Journal of Research of the National Bureau of Standards, 45(4), 1950.
Daniel Levy, Matthew D Hoffman, and Jascha Sohl-Dickstein. Generalizing Hamiltonian Monte Carlo with neural networks, arXiv preprint 1tX.vv:711.092660 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In International Conference Computer Vision, 2015.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? A large-scale study. In Advances in Neural Information Processing Systems, pp. 700-709, 2018.
Matthew MacKay, Paul Vicol, Jimmy Ba, and Roger B Grosse. Reversible recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 9029-9040, 2018.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. tradient-based hyperparameter optimization through reversible learning. In International Conference on Machine Learning, pp. 2113-2122. 2015.
12

---

Under review as a conference paper at ICLR 2020
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. Conference on Computer Vision and Pattern Recognition, 2014.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp. 2338 2338-2347, 2017.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, 1530-1538, 2015.
Jiaming Song, Shengjia Zhao, and Stefano Ermon. A-NICE-MC. Adversarial training for MCMC. In Advances in Neural Information Processing Systems, pp. 5140-5150, 2017.
Yang Song, Chenlin Meng, and Stefano Ermon. MintNet: Building invertible neural networks with masked convolutions. Advances in Neural Information Processing Systems, 2019.
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Jakub M Tomezak and Max Welling. Improving variational auto-encoders using householder flow. arXiv preprint arXiv: 1.09630, 2016.
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: Analysis and efficient estimation. In Advances in Neural Information Processing Systems, 2018.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.077 2016.
13

---

Under review as a conference paper at ICLR 2020
A DERIVATIONS OF LIPSCHITZ BOUNDS
The bounds for invertible ResNets are taken from (Behrmann et al. 2019). For Neural ODEs (Chen al. 2018 one needs consider Lipschitz constant Lip(F) that holds for all [0,7], i.e.
[f(..z)) F [t, z2)]l2 I Lip [[ 12ll2. for all
e [O,
Then, the claimed bound is a standard result, see e.g. (Ascher 2008 Theorem 2.3). Note that the inverse is given by dy(() (3)(t), t), hence the same bound holds.
In the subsequent subsections, we derive the bounds for coupling layers.
A. DERIVATION OF LIPSCHITZ BOUND FOR ADDITIVE COUPLING LAYERS
Consider an additive coupling block defined
F(x),, I1, F(x),, = 2I2 + g(x),),
where 12 is a disjoint partition indices {1...., d} of the same cardinality, i.e. [h] = [12 2 Further, 21, 21, correpsonds to the corresponding dimension of Rd and g R R By Lemma 3 is
Lip(F) = sup [1Jr())] xERd
Thus, in order to obtain bound on the Lipschitz constant, it is helpful to look into the structure of the Jacobian. If the partitions T and correspond to the first and last indices, the Jacobian has lower-block structure with an identity diagonal, i.c.
Jr(x) = (J((z) 9)
By using this structure, we can derive the following upper bound:
Lip(F)
sup
[>)
=cR4
||F(2>>*||
sup sup #ERd -==1
#eld |=]| [[Jr(z)a* )r ), 11 + ((J)()))")
sup sup
sup sup
| + || 1,(02, l
#ERd | 1|
| (1 + 112)
sup sup
RER
sup sup || + + || 11
RERd
sup | |+113~+
=ERd 11
sup sup 1+2 1+20,:|h1117,0000|||--04,,000,,0
=ERd |-]
sup sup wERd 1==1
1++2
(1 + (1)) 122)
sup sup
=ERd | =|
sup (1 + 1z)
=ERd
Lip(FF < Lip(g).
14

---

Under review as a conference paper at ICLR 2020
Furthermore, the inverse of F can be obtained via the simple algebraic transformation (y := F()))
F (y)n, = 91, (1) = Vr, - -(n,)))
Since the only difference to the forward mapping is the minus sign, the Lipschitz bound for the inverse the same as for the forward mapping.
A.2 DERIVATION OF LIPSCHITZ BOUND FOR AFFINE COUPLING LAYERS
Since the structure of the forward and inverse mapping for affine coupling layers has some differences, we split the derivation of the Lipschitz bounds into two sections. First, we start with the forward mapping and then reuse several steps for the bounds on the inverse mapping.
A.2.1 DERIVATION THE FORWARD MAPPING
Consider an affine coupling block defined as
F(z), Fr, F(z),, o g(s(z,)) + t(zr),
where g(-) * for all X1, and 11, as before. The Jacobian for this operation has the structure
I + J(()) Da(s(an))
O
Jr(#)= = (or, (on
where are following diagonal matrices
DI(x1,) diag ((rr)),..., (Etz))(a))
(x1,), = diag (f g((((2)))))))
D9(s(F1,)) = diag
Denote
M(x) = (z,,
+ Je(1).)
By using an analogous derivation in equation (up to the inequality sign), we get
Lip(F)² sup sup
|| 112 + (1D)(s(tr))),,,lee (|)
nERd ||==1|==1
sup max (1, (1, -2 max (Dg(s(z))))) SE[IA]]
=ER4 ie[lA]]
S sup max (1, D2 (s.2, (E[ neRd +E[
max (1, (st, ||M
2
max (1, ()) + [M)())
= sup aERd
Lip(F) < max (1, (s(rr)))) + sup (z)
T
Next, will look into the structure of (x) to derive more precise bound. Since inputs assumed to be bounded as x E (a, b)d it holds
(2)))))) S max([al,16))
Furthermore, let the derivative g' the dementt-wise function be globally bounded by c, i.e. supzeR o' (x) S Cg', Then, it is
(21, Cg'
15

---

Under review as a conference paper at ICLR 2020
In similar manner as in section A.1 the spectral norm of the Jacobian of the scale-function and translation- fintiionn can be bounded by their Lipschitz constant, i.e.
[J,( )l2 s Tip(s)
[ ll S Lip(t).
By using above bounds, we obtain
sup
[|M() S max([a], (6]) c. e-Lip(s) + Lip(t).
wERd
If we further assume, that the elementwise-function g globally upper bounded by oo and insert above bounds, we obtain
Lip(F) max(11, Cg) + max((a), [8]) Cg' Lip(s) + Lip(t).
A.2.2 DERIVATION FOR THE INVERSE MAPPING
For the affine coupling block from section A.2. the inverse is defined as
F-)( = 91,
F- F-)(y) (vı, ((r1)) @ ((s(u,)),
where g(-) * O for all X12, I1, 12 as before and @ denotes elementwise division. The Jacobian for this operation has the structure
O Jr(x) (M()) Dy Dy(o(en)))
where Dí (s(x1,)) denotes a diagonal matrix, as before. Furthermore, M defined as
(v) D2 (r(,))) (P()) (1) ( (wn)
(y(s(wn)))
where D( ((s))(((Tn) also denotes diagonal matrix. Using analogous arguments as in section A.2.1 we obtain the bound
Lip(F-1) < max (1, Dj (s(xr, k)) + sup [x)
wERd
Hence, we need to further bound the spectral norm of M First, assume that J. the derivative and translation t is globally upper bounded by cy. (()) and e respectively. Furthermore consider bounded inputs y E o.]dd Then we obtain the bound
(
sup | |M' (x) < max([a"),16*) "(y)" Lip(s) + Qgy Lip(s) a c₁ Lip(t). #ER4
Hence, we can bound the Lipschitz constant of the inverse of an affine block as
Lip(F-1) < max (1,cg) + max([a"), [b")) 1,,1(())),1"))) Cgy Lip(s) + Cgy Lip(s) a. tc Lip(t).
xE([A:]]
B NUMERICAL ERRORS AND LIPSCHITZ CONSTANTS
In a general setting, connecting numerical errors e.g. due to floating point operations to Lipschitz constants of the underlying mapping in quantitative manner is not straightforward. For example, numerical errors due to limited precision occurs when summing to floating point numbers. As discussed in (Gomez et al.] 2017). this occurs in additive coupling layers and is one source of numerical errors we observe in our experiments.
To formalize the connection the Lipschitz constant, consider the following mappings:
F(x) = 2, (analytical exact computation)
Fs(x) + 5 5=: 26,
(floating point inexact computation)
16

---

Under review as a conference paper at ICLR 2020
In order to bound the error in the reconstruction due to the imprecision in the forward mapping, let Is, = F-1 (28). Now consider
x| ns 2l S Lip(F - 2s 2l2 = Lip(F=))
where the Lipschitz constant of the inverse is used to bound the influence of the numerical error in the forward mapping. However, similarly to the forward mapping. the inverse mapping can also be imprecise. Thus, we introduce
(2s) 26, + 82 882
to formalize the numerical error in the inverse mapping. Hence, we obtain the bound
| (1s, + do)ll2 < || 26, ||2 + ||62|12
Lip(FF zal12 + ||22|2
Lip + ||62|12,
where the numerical errors of the mapping are denoted via 8 (forward) and ó2 (inverse). While obtaining quantitative values for and 82 for a model as complex as deep neural networks is hard, above formalization still provides insights into potential role of the inverse stability when reconstructing inputs.
C ADDITIONAL DETAILS FOR CLASSIFICATION EXPERIMENTS
Both affine and additive coupling-basse based models have the same architecture, that consists of 3 levels, 16 blocks per level, and 128 hidden channels. Each level consists of sequence of residual blocks that operate on the same di- mensionality. Between levels, the input is spatially down-
0.90
Accuracy
sampled by 2x both width and height, while the number of channels is increased by 4x. Each residual block con- sists of a chain of 3 x 3, 1 X 1, 3 x 3 convolutions, with ReLU activations in between. We trained on CIFAR-IO for 200 epochs, using SGD with Nesterov momentum 0.9 and weight decay 5e-4, with initial learning rate 0.01, decayed by factor of 5 at epochs 60, 120, and 160. We found that using a smaller initial learning rate of 0.01 was important for training the INN classifier, as opposed to the standard initial learning rate of 0.1 used for ResNets (Zagoruyko & Komodakis 2016). We used standard data augmentation (random cropping and horizontal flipping).
Additive
Affine
50
100
150
200
Epoch
Figure 6: Test classification accuracy on CIFAR-10.
D EXPERIMENTAL DETAILS FOR FLOW-GAN
For the experiments in Section 4.3 the ADV models are trained with standard binary cross entropy loss and the hyperparameters in Table whereas the MLE models are trained with learning rate le-3.
Variable
Values
batch size
64
learning rate (generator)
le-5
learning rate (discriminator)
le-4
weight decay (both)
optimizer
Adam(0.5,, 0.99)
Table 4: Hyperparameters for ADV models
17

---

Under review as a conference paper at ICLR 2020
E EXTENDED RESULTS FOR CRAFTED, NON-INVERTIBLE INPUTS
PGD Setup. To find non-invertible inputs for Glow and Residual Flows, we used PGD (Fq. 3 with = 0.1 and step size 0.01. For the Glow model in Section (.2) we consistently found inputs with severe reconstruction enrors (as shown in Figure 3 in fewer than 10 PGD iterations. For the Residual Flow model analyzed in this section, we ran 200 iterations PGD. In each iteration, the pixel values of the perturbed image were clipped to the valid input range that the respective model trained on (l0..5, 0.5] for Glow and (0, 1] for the Residual Flow).
Crafted Inputs for Residual Flows. We also applied the PGD attack from Section 4.2 to a Residual Flow (Chen et al. 2019) pre-trained on CIFAR-10 (Figure 7 We find that, while there are visible differences between the crafted input æ' and its reconstruction 4', the reconstruction errors are less severe for the Residual Flow compared to Glow (analyzed in Section 4.2)..
Figure 7: Crafting non-invertible inputs for CIFAR-10 Residual Flow model.
Instability Outside the Range Training Inputs. Here, we applied the PGD attack from Sec- tion 4.2 to Glow model pre-trained Celeb-A (Liu et al. 2015 This model was trained on images normalized to the range (-0.5, 0.5). While the PGD attack was not successful at finding adversarial inputs in [-0.5, 0.5), it succeeded when the range was increased to [-0.7, 0.7] (by using e = 0.2 and not clipping the perturbed inputs to [-0.5, 0.5)). yielding the example shown Figure 8
Thus, we found that invertible models can become numerically non-invertible out-of-distribution data.
Original
Original Recons.
Adversarial
Adversarial Recons.
Figure 8: Glow becomes non-invertible for inputs outside the training distribution. We use SOTA Glow model trained on images normalized to [-0.5, 0.5), and use PGD to find an adversarial input constrained to the larger range [-0.7, 0.7). This attack succeeds in finding examples that induce dramatic reconstruction error.
"We used the pre-trained model from ftt.:ttt:://ggttb...oo//ttgg//// - We used the model from http::://ttthib..cm//ee/aaa////p
18

---

Under review as a conference paper at ICLR 2020
F MOTIVATING THE DECORRELATION TASK
Figure 9; Toy example motivate the decorrelation task. Left: standard normal data, 2. left: scaled by D and rotated data by R, 2. right: scaling and rotation backwards, low correlation but higher condition number, right: rotation backwards, low correlation and low condition number.
To motivate the decorrelation task as simple toy environment for stability invertible models consider the following task (and its visualization Figure 9):
Consider input data that is distributed via standard normal distribution, i.e. x I) (
Assume the data is transformed by a rotation matrix and diagonal matrix D, i.e. y RDr (Figure from left)).
Goal: decorrelate transformed data using an invertible mapping.
Since correlation is independent of scale (scaling by standard deviation of the data), at least the two solutions A₂ = D-1 RR" (Figure 9 (2. from right)) and A₁ = Rº (Figure 9 (right)) are equally valid for the given decorrelation task. However, the conditioning of the mappings A and Aa can be largely different the scaling matrix D has a high condition number. Hence, this task both offers a stable solution, namely Aj, and (potentially) unstable solution A2.
To conclude, decorrelation can allow multiple solutions with different stability. Hence, decorrelation a natural simple task study which solution the INN picks. Furthermore, guiding the network to stable solution is justified strategy for this task and it is not expected to harm performance.
G DECORRELLTION EXAMPLEE CODE
Here we provide an example implementation the decorrelation objective used Section 4.4 that minimizes the norm of the off-diagonal entries in the correlation matrix.
Listing 1: Example Py PyTorch code to implement the decorrelation loss used in our experiments.
2 model (img)
2_flat 2. z.view (z. size
(z.szez. (0), -1)
flfl z_flat 2 2_flat.ment mean (dim=0) Subtract mean
z_flat z_flat / (z_flat. std (dim=0) 1e-8) Standardize
correlation (torch. mm (2 (2_flat. z_flat) / (correlation))
(0)-1))
loss = torch. norm (correlation optimizer.zero_pe
serch.diage
loss backwaack optinizer.step
19

---

Under review as a conference paper at ICLR 2020
H EXTENDED RESULTS FOR DECORRELATION
Decorrelation Experiment Details Here we provide additional details on the model architectures and training schemes used in our numerical experiments.
For all the decorrelation experiments, we used a 3-level model with blocks of depth 16. We used Adam (Kingma & Ba 2013) with fixed learning rate le-4 and no weight decay, and trained of size 64. The CIFAR- 10 images were normalized to the range [-0.5, 0.5], and were dequantized with uniform noise in [0, le-6].
Effeet of Model Depth. Furthermore, we investigated the effect of network depth stability since its an additional influence factor besides the selection of each invertible building block. Starting with 3-level additive model with ActNorm, we vary the depth of the blocks between {4, 16, 32) and train with the decorrelation objective. The quantitative reconstruction errors and condition numbers of the Jacobians are shown in Figure T0 As expected, deeper architectures become unstable faster than shallow ones.
101a
10"
Condition Num
1014
10ae
Recons, Error
10
1014
10t
10a2
10"
1010
10a
Depth
102
10a
10a
Depth 16
10-1
104
Depth
10-1
0k
100k
200k
300k
400k
0k
100k
200k
300k
400k
Iteration
Iteration
(a) Condition numbers
(b) Reconstruction error
Figure 10: Comparing stability of additive flows of different depths. These models all have ActNorm both between and inside the additive blocks.
Loss Plots. Here we show that all the model variants investigated in the decorrelation experiments achieve their objective, i.e., the loss decreased enough for the correlation matrices to be diagonal.
Evolution of Condition Numbers, Max & Min Singular Values for Decorrelation. Here plot the condition numbers, maximum and minimum singular values during training for the decorrelation task. We include plots for all settings discussed in the main paper well the appendix.
20

---

Under review as a conference paper at ICLR 2020
Original Images
Additive: ActNorm Between Blocks
a
10?
Recons Error
10a
16a
R
c= 3e4
4e9
8e12
10-3
0maz = 14
mmaz 9e4
3e5
0k
60k
120k
180k
Omin 5e-4
mmin 3
min 3e-8
Iteratio
Additive: ActNorm Inside Blocks
a
10:9
Recans. Error
10:5
10t
R
107
103
Se4
9e9
2c12
10-1
mmaz = 90
Maz 7e5
maa 6c6
0k
40k
80k
120k
Omin 2c-3
mmin 8
min 3e-6
Iteration
Training
Figure 11: Additional settings for additive coupling, where: 1) ActNorm was applied only between blocks; and 2) ActNorm applied only inside blocks. Both models become unstable and exhibit severe reconstruction artifacts.
Original Images
4
Affine: ActNorm Between Blocks
a
1012
Recons, Error
R
c= 5e2
c= 7e9
c= 7e12
10-3
Omar = 3.1
Omaz 10
Oman 15
Omin = 7e-3
Omin = 2e-9
Omin = 2e- 12
Iteration
Training
Figure 12: Additional setting for affine coupling, where ActNorm applied only between blocks. Similarly to the other affine settings described in Section 4.4 this model becomes highly unstable rapidly during training.
21

---

Under review as a conference paper at ICLR 2020
1100
1100
1100
1000
1000
1000
1000
900
900
800
800
800
700
600
600
600
600
500
500
500
400
400
400
400
Iteration
Iteration
(a) Affine: No ActNorm
(b) Additive:
ActNorm (c) Additive: No ActNorm (d) Additive: No ActNorm
Both Inside and Between Blocks
+ Spectral Norm
Figure 13: Decorrelation loss plots. These plots track the norm of the off-diagonal entries in the correlation matrix while training with the decorrelation objective.
10'
10
Condition Num
1013
Maximum SV
10
10
10th
10a
Minimum
10
10a
10a
10-4
10a
107
10
102
10
10a
10a
10-9
0k
60k
120k
180k
0k 60k
120k
180k
0k
60k
120k
180k
Iteration
Iteration
Iteration
(a) Condition number
(b) Max singular values
(c) Min singular values
Figure 14: Additive: ActNorm Between Blocks
M Whrlep Mhor Mar!
Condition Num
1013
Maximum SV
107
10a
10-4
10-2
10t
10s
Minimmm
10
10a
10s
10-5
107
10
10a
10-7
10a
10a
10-4
0k
40k
80k
120k
0k
40k
80k
120k
0k
40k
80k
120k
Iteration
Iteration
Iteration
(a) Condition number
(b) Max singular values
(c) Min singular values
Figure 15: Additive: ActNorm Both Inside and Between Blocks
1047
1015
Condition Num
10is
Maximum SV
1013
10-2
1gls
Minimum
10:3
10a
10-4
10a
107
10:
107
10a
10a
10a
10-4
0k
40k
80k
120k
0k
40k
80k
120k
40k
80k
120k
Iteration
Iteration
Iteration
(a) Condition number
(b) Max singular values
(c) Min singular values
Figure 16: Additive: ActNorm Inside Blocks
22

---

Under review as a conference paper at ICLR 2020
10ts
10a
10-2
Condition Num
1gls
Maximum SV
10y
10c
10-1
10
10
Minimum
10-1
10a
10
10-4
107
10a
10
102
10a
10a
10-7
10
OK
60k
120k
180K
0k
60k
120k
180k
60k
120k
180k
Iteration
Iteration
Iteration
(a) Condition number
(b) Max singular values
(c) Min singular values
Figure 17: Additive: No ActNorm
10"
10
Condition Num
1013
Maximum SV
Minimum SV
10-5
104
6x10-1
10-7
10a
10-9
107
10a
3x10-
10-11
10-13
10a
2x11-
Ok
2k
4k
6k
8k
10k
OK
2k
4k
8k
10k
0k
2k
4k
6k
8k
10k
Iteration
Iteration
Iteration
(a) Condition number
(b) Max singular values
(c) Min singular values
Figure 18: Affine: No ActNorm
Condition Num
10th
Maximum 5V
10a
wwwa
Minimum SV
10-5
10t1
10
10-7
10a
10a
10-
10
107
10-10
w www My
10a
101
10-11
10-12
1000
2000
0
1000
2000
1000
2000
Iteration
Iteration
Iteration
(a) Condition number
(b) Max singular values
(c) Min singular values
Figure 19: Affine: ActNorm Both Inside and Between Blocks
Condition Num
1012
10t4
Mar Merrnthouss
Maximum SV
Minimum SV
10-3
10-5
10to
10a
10a
10-9
10a
10-11
10c
10-13
0k
2k
4k
6k
0k 1k 2k 3k
4k 5k 6k Iteration
0k
2k
4k
6k
Iteration
Iteration
(a) Condition number
(b) Max singular values
(c) Min singular values
Figure 20: Affine: ActNorm Between Blocks
23

---

Under review as a conference paper at ICLR 2020
I REFITTING PRIOR IN FLOW-GAN
In general, if the model is not optimized with forward KI. DKL OKLa(PaaalP)) Po), as in the case when optimizing with maximum likelihood, we cannot be F(z), standard Normal. Hence, a reasonable strategy, without changing anything in the learned network, is refit the prior parameters. Here we simply optimize for maximum likelihood (as typically done for flow models) while only fitting diagonal variance in prior. This can be interpreted as increasing the entropy of our model. Kingma & Dhariwal (018), they observed the opposite phenomenon that model trained with maximum likelihood generates better samples after decreasing the entropy in the prior. See Figure 21 for samples after refitting the prior.
Paata is best fitted with
ADV (Stable)
ADV (Unstable)
9
9
3
7
6
q
7
2
2
9
7
/
9
Figure 21: The number corresponds to the number of epochs after refitting the prior variance. The unstable model fails to generate samples (i.e., outside of valid pixel values) after the prior refitted, whereas even though the stable model degrades in sample quality, it is able to generate valid images.
24


Reviews:
[{"review_id": "ryTvSq2Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper claims that for invertible neural networks, mathematical guarantees on invertibility is not enough, and we also require numerical invertibility. To this end, the lipschitz constants/condition numbers of Jacobians of both the forward and inverse maps of invertible NNs based on coupling layers are examined mathematically and experimentally. The paper also displays cases that expose non-invertibility in these architectures via gradient-based construction of adversarial inputs, as well as a decorrelation benchmark task, and show that spectral normalization can be a remedy for stabilizing these flows.\n\nI think it’s a good point that we need to monitor the Lipschitz constant/bounds of both directions of these invertible functions. It’s true that the focus for stabilising NNs by bounding Lipschitz constants was always on the forward function, and for invertible functions we should also ensure that the inverse is numerically stable to compute.\n\nThe mathematical contribution of the paper is twofold - 1. deriving bounds on the lipschitz constants of the forward and inverse mapping of additive/affine coupling blocks 2. summarising known lipschitz bounds of forward and inverse mappings of other invertible layers (iResNet, neuralODE, invertible 1x1 convolutions etc). The main contribution lies in 1, and the derivation for the additive coupling block (volume preserving) is neat (although fairly straightforward), but the derivation for the affine coupling layer (NVP) is not useful nor insightful; they are local Lipschitz bounds (so require bounds on all intermediate activations, which is difficult as pointed out by the authors), and the numerical value of this bound was not used at all in relation to the numerical experiments - I imagine the bound is loose. Given that it seems difficult to find a tight global lipschitz bound, I think it would be more insightful to compute a lower bound to the lipschitz constant of the model (with fixed parameter values) by maximising the spectral norm of the Jacobian with respect to the inputs (or outputs if looking at the inverse map) - this will yield a lower bound by Lemma 3. This will be numerical, but more informative since it will give you an indication of where in the input space (or output space if looking at the inverse) there could be numerical instabilities. Also I think the bound on the local lipschitz constant of the inverse for the affine coupling block might be incorrect, because in A.1.1, the inverse map is F^{-1}(y)_I1 = y_I1, F^{-1}(y)_I2 = (y_I2 - t(y_I1))/g(s(y_I1)), so the scale and shift is s’(y_I1) := 1/g(s(y_I1)) and t’(y_I1):=- t(y_I1)/g(s(y_I1)), and hence I think this needs to be taken into account for computing the lipschitz bound of the inverse \n\nI have mixed feelings about the experimental section. In section 4.1, it is interesting to see that we can find inputs where trained flow models can show numerical non-invertibility, evident in the poor reconstructions. It would be a nice addition to investigate whether this is coming from the forward function or its inverse, by examining the norm of the Jacobian of F and F^{-1} at the input x_delta and output F(x_delta) respectively. \n\nHowever, the decorrelation task introduced in section 4.2 is puzzling. I don’t understand why for these invertible models, you are investigating invertibility for parameter values trained to decorrelate, as opposed to parameter values used in the usual task of density estimation with flows (or any other standard application of invertible NNs). The two reasons given in the paper are that 1) decorrelation is a simpler task and 2) it allows both stable and unstable transforms as solutions, but these are not convincing. Point 2) holds for flow-based density estimation as well, and regarding point 1), density estimation is the task we usually care about when using invertible NNs, and this is also computationally plausible/tractable, whereas even if decorrelation is a simpler task, it’s not a task that users of invertible NNs are interested in. It is good to know that these invertible NN architectures CAN admit values that are numerically non-invertible, but I would be much more interested to know whether this actually holds when they have been trained for flow-based density estimation. I’m not sure whether the experimental results on models trained for the decorrelation task are useful, because a model that is stable when trained for the decorrelation task may be unstable when trained for flow-based density estimation and vice versa. The observation that spectral normalization can help address numerical instability is useful, but from the perspective of someone who wants to use these invertible NNs for density estimation, I would like to know what is the sacrifice in expressivity/validation performance (if any) when using spectral normalization in these invertible architectures. Also, the results would be more relevant if the architectures resembled the architectures used for invertible models used in the literature (e.g. GLOW) where we not only have coupling layers but they are interleaved with PLU linear flows.\n\nIn section 5, the result that Flow-GANs can be numerically non-invertible is more relevant, and it is useful to know that spectral normalisation can help resolve this issue, but again it would be useful to quantify whether this comes at the cost of the quality of generated samples (Figure 3 shows several samples, but a more thorough quantitative & qualitative comparison would be welcome). Also regarding the point about likelihood in Section 5, where the authors state “it cannot be trusted as true likelihood due to lack of invertibility”, I think it should be emphasised that this point holds specifically for flow-GANs where for F: z -> x, you need a numerically accurate F^{-1} to compute the density, but for standard flow-based density estimation where F:x -> z, you never need to compute the inverse for computing the likelihood, hence if F has a small lipschitz constant then the likelihood will be accurate, regardless of whether the inverse is numerically stable or not.\n\nOverall I believe the experimental section can be largely improved, and given that the motivation of the paper is nice and the paper is clearly written and nicely presented, it would be a shame to leave the experiment section as it is.\n\nMinor typos/Qs:\np2: this problems <- this problem\np8: and with maximum likelihood (ML) - should this be removed?\np13: t(x_I2) <- t(x_I1)\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "rkxViW62KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyses the numerical invertibility of analytically invertible neural networks (INN). The numerical invertibility depends on the Lipschitz constant of the respective transformation. The paper provides Lipschitz bounds on the components of building blocks for certain INN architectures, which would guarantee numerical stability. Furthermore, this paper shows empirically, that the numerical invertibility can indeed be a problem in practice.\n\nThis work is interesting and could be important to many researchers working with INNs. The worst case analysis and the corresponding table with Lipschitz bounds is useful. \nHowever, I have some concerns regarding the experimental evaluation. \n- Experiments in 4.1. nicely show that there exist non-invertible inputs for a GLOW model trained on CIFAR. But I wish the authors also considered other popular INN models and non-image datasets for this set of experiments (showing if this is also an issue in scenarios other than CIFAR/CELEBA + GLOW). \n- Although the authors spend significant space in the main text and the appendix to motivate the experiments in 4.2, I cannot follow this motivation. For example, “decorrelation is a simpler objective than optimizing outputs z = F(x) to follow a factorized Gaussian as in Normalizing Flows”. Why is this is simpler, and, more importantly, why would this be an argument? Another example is “… this decorrelation objective offers a controlled environment to study which INN components steer the mapping towards stable or unstable solutions, …”. Why is this more controlled? What exactly is controlled here that is not controlled in training a an INN for, e.g., density estimation?  I am not sure if this set of experiments is any useful for determining whether numerical precision is actually problematic for posterior approximation with normalizing flows, density estimation, etc.\n- the experimental sections is somewhat badly structured and makes it difficult to read. It is not clear if this paper is analysis-only or whether the authors propose a remedy. The authors write in the abstract and conclusion that they show how to guarantee invertibility for one of the most common INN architectures. After reading this, I would expect a designated experimental section which shows a fix. I suppose they refer to Additive blocks + Spectral Norm, discussed in 4.2.1. However, that reads more like a post-hoc insight (“it turns out that…” rather than “we show how“). In short, the experiments section could be much better structured. \n- The paper would be greatly improved, if the authors would propose how to tackle these numerical problems. I doubt that additive coupling is “one of the most common INN architectures”. It would be nice if the authors would conduct more extensive experiments and propose solutions for other building blocks. \n- I expect at least a few experiments that quantify numerical instability with multiple different random seeds (for initialization etc.).\n\nFor these reasons I vote for rejection. \nI think it would be advisable to rethink the goals of the experimental evaluation, come up with a better structure, and expand at several places. E.g. (i) expand 4.1 to other architectures and data, (ii) show how this is relevant in practice (e.g. posterior inference with NFs and density estimation) and how it questions published results (currently Sec. 5), and (iii) evaluate proposed solutions. \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1gig7Jf9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper points out invertible neural networks are not necessarily invertible because of bad conditioning. It shows some cases when invertible neural networks fail, including adding adversarial pertubations, solving the decorrelation task, and training without maximum likelihood objective (Flow-GAN). The paper also shows that spectral normalization improves network stability. \n\nI think this is a solid work. The main contribution is it points out a problem that is overlooked before, which can possibly explain some unstable behavior for training neural networks. The paper also has some study on various architectures, which sheds some light on the designing of invertible neural networks. I think this paper can be important for future researchers to design models and algorithms. \n\n===============\n\nUpdate:\n\nAfter reading other reviewer's comment I agree with other reviewers that the experimental section is problematic. It seems to be unrelated with the theoretical results proposed in this paper. I think currently the experiments only make a point that invertible networks can be non-invertible in practice. But the paper has large room to improve if it has\n\n1. A complete discussion on which invertible blocks / modeling tasks are easier to be non-invertible, and why (theoretically, and combine with direct experimental evidence)\n2. A remedy (using additive coupling layer is not an acceptable one since it severely limits the modeling power)\n\nI still think posing the problem itself is important. Thus I will still give it an accept, but lower it to a weaker score.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission analyses the numerical invertibility of analytically invertible neural networks and shows that analytical invertibility does not guarantee numerical invertibility of some invertible networks under certain conditions (e.g. adversarial perturbation).

Strengths:
-The work is interesting and the theoretical analysis is insightful.

Weaknesses:
-The main concern shared by all reviewers was the weakness of the experimental section including (i) insufficient motivation of the decorrelation task; (ii) missing comparisons and experimental settings.
-The paper clarity could be improved.

Both weaknesses were not sufficiently addressed in the rebuttal. All reviewer recommendations were borderline to reject.

Decision: Reject