On the Invertibility of Invertible Neural Networks

1 INTRODUCTION . Invertible neural networks ( INNs ) have become a standard building block in the deep learning toolkit . Invertibility is useful for training generative models with exact likelihoods ( Dinh et al. , 2014 ; 2017 ; Kingma & Dhariwal , 2018 ; Kingma et al. , 2016 ; Behrmann et al. , 2019 ; Chen et al. , 2019 ) , increasing posterior flexibility in VAEs ( Rezende & Mohamed , 2015 ; Tomczak & Welling , 2016 ; Papamakarios et al. , 2017 ) , learning transition operators in MCMC samplers ( Song et al. , 2017 ; Levy et al. , 2017 ) , computing memory-efficient gradients ( Gomez et al. , 2017 ; Donahue & Simonyan , 2019 ) , allowing for bi-directional training ( Grover et al. , 2018 ) , solving inverse problems ( Ardizzone et al. , 2019 ) and analysing adversarial robustness ( Jacobsen et al. , 2019 ) . The application space of INNs is rapidly growing and many approaches for constructing invertible architectures have been proposed . A common way to construct invertible networks is to use triangular coupling layers ( Dinh et al. , 2014 ; 2017 ; Kingma & Dhariwal , 2018 ) , where dimension partitioning is interleaved with ResNet-type computation . Another approach is to use various forms of masked convolutions , generalizing the dimension partitioning approach of coupling layers ( Song et al. , 2019 ; Hoogeboom et al. , 2019 ) . To avoid dimension partitioning altogether , multiple approaches based on efficiently estimating the log-determinant of the Jacobian , necessary for applying the change of variable formula , have been proposed to allow for free-form Jacobian structure ( Grathwohl et al. , 2019 ; Behrmann et al. , 2019 ; Chen et al. , 2019 ) . From a mathematical perspective , invertible architectures enable several unique guarantees like : • Enabling flexible approximation of non-linear diffeomorphisms ( Rezende & Mohamed , 2015 ; Dinh et al. , 2017 ; Kingma & Dhariwal , 2018 ; Chen et al. , 2019 ) • Memory-saving gradient computation ( Gomez et al. , 2017 ; Donahue & Simonyan , 2019 ) • Fast analytical invertibility ( Dinh et al. , 2014 ) • Guaranteed preservation of mutual information and exact access to invariants of deep networks ( Jacobsen et al. , 2018 ; 2019 ) . Despite the increased interest in invertible neural networks , little attention has been paid to guarantees on their numerical invertibility . Specifically , this means analyzing their ability to learn bi-Lipschitz neural networks , i.e . Lipschitz continuous neural networks with a bound on the Lipschitz constant of the forward and inverse mapping . While the stability analysis of neural networks has received significant attention e.g . due to adversarial examples ( Szegedy et al. , 2013 ) , the focus here is only on bounding Lipschitz constants of the forward mapping . However , bounding the Lipschitz constant of the inverse mapping is of major interest , e.g . when reconstructing inputs from noisy or imprecise features . In fact , analytical invertibility as provided by some invertible architectures does not necessarily imply numerical invertibility in practice . In this paper , we first discuss the relevance of controlling the bi-Lipschitz bounds of invertible networks . Afterwards we analyze Lipschitz bounds of commonly used invertible neural network building blocks . Our contributions are : • We argue for forward and inverse stability analysis as a unified viewpoint on invertible network ( non- ) invertibility . To this end , we derive Lipschitz bounds of commonly-used invertible building blocks for their forward and inverse maps . • We numerically monitor and detect ( non- ) invertibility for different practical tasks such as classification and generative modeling . • We show how this overlooked issue with non-invertibility can lead to questionable claims when computing exact likelihoods with the change-of-variable formula . • Finally , we study spectral normalization as a stabilizer for one of the most commonly-used family of INN architectures , namely additive coupling blocks . 2 BACKGROUND AND MOTIVATION . Invertible neural networks are bijective functions with a parametrized forward mappingFθ : Rd → Rd with Fθ : x 7→ z , where θ ∈ Rp defines the parameter vector . Additionally , they define an inverse mapping F−1θ : Rd → Rd with F −1 θ : z 7→ x . This inverse can be given in closed-form ( analytical inverse , e.g . Dinh et al . ( 2017 ) ; Kingma & Dhariwal ( 2018 ) ) or approximated numerically ( numerical inverse , e.g . Behrmann et al . ( 2019 ) ; Song et al . ( 2019 ) ) . Before we discuss building blocks of invertible networks , we provide some background and motivation for studying forward and inverse stability . Definition 1 ( Lipschitz and bi-Lipschitz continuity ) . A function F : ( Rd1 , ‖ · ‖ ) → ( Rd2 , ‖ · ‖ ) is called Lipschitz continuous if there exists a constant L = : Lip ( F ) such that ‖F ( x1 ) − F ( x2 ) ‖ ≤ L‖x1 − x2‖ , ∀x1 , x2 ∈ Rd1 . If an inverse F−1 : ( Rd2 , ‖ · ‖ ) → ( Rd1 , ‖ · ‖ ) and a constant L∗ = : Lip ( F−1 ) exists such that ‖F−1 ( y1 ) − F−1 ( y2 ) ‖ ≤ L∗‖y1 − y2‖ , ∀y1 , y2 ∈ Rd2 , then F is called bi-Lipschitz continuous . Remark 2 . We focus on invertible functions F : ( Rd , ‖ · ‖2 ) → ( Rd , ‖ · ‖2 ) , i.e . functions where the domain and co-domain are of the same dimensionality d and the norm is given by the euclidian norm . Lemma 3 . ( Rademacher ( Federer , 1969 , Theorem 3.1.6 ) ) If F : Rd → Rd is a locally Lipschitz continuous function ( i.e . functions whose restriction to a neighborhood around any point is Lipschitz ) , then F is differentiable almost everywhere . Moreover , if F is Lipschitz continuous , then Lip ( F ) = sup x∈Rd ‖JF ( x ) ‖2 , where JF ( x ) is the Jacobian matrix of F at x and ‖JF ( x ) ‖2 denotes its spectral norm . Lipschitz bounds on the forward mapping are of crucial importance in several areas , including in adversarial example research ( Szegedy et al. , 2013 ) , to avoid exploding gradients , or the training of Wasserstein GANs ( Anil et al. , 2019 ) . The stability of the inverse , however , can have a similar impact . For instance , having a Lipschitz bound on the inverse may avoid vanishing gradients during training . Given that deep-learning computations are carried out with limited precision , imprecision is always introduced in both the forward and backward passes , i.e. , zδ = F ( x ) + δ and x̂δ = F−1 ( zδ ) . Instability in either pass will aggravate this problem , and essentially make the invertible network numerically non-invertible . To summarize , this problem occurs in the following situations : • Numerical reconstruction of x , where features zδ are inexact due to limited precision ( e.g . when computations are executed in single precision as common on modern hardware ) . • Reconstruction based on imprecise measurements from physical devices ( e.g . when using invertible networks for inverse problems ( Ardizzone et al. , 2019 ) ) . • Numerical re-computation of intermediate activations of the neural network to allow for memory-efficient backpropagation ( Gomez et al. , 2017 ) . Furthermore , some computations are performed via numerical approximation , which in turn adds another source of imprecision that might be aggravated via instability . Examples include : • Numerical forward computation , as in Neural ODEs ( Chen et al. , 2018 ) ( numerical solver is used to approximate dynamic of ODE ) . • Numerical inverse computation , e.g . via fixed-point iterations as in i-ResNets ( Behrmann et al. , 2019 ) or MintNet ( Song et al. , 2019 ) or via ODE-solvers for the backward dynamics as in Neural ODEs ( Chen et al. , 2018 ) . As an example of why bi-Lipschitz continuity is critical for numerical stability in invertible functions , let ’ s consider the simple mappings F1 ( x ) = log ( x ) , F−11 ( z ) = exp ( z ) , and F2 ( x ) = x , F −1 2 ( z ) = z . Though both functions tend to infinity when x → ∞ , F1 is much less stable . Consider the introduction of numerical imprecision as zδ = F1 ( x ) + δ where δ denotes the introduced imprecision . Then this imprecision is magnified in the inverse pass as : ||F−11 ( z ) − F −1 1 ( z δ ) ||22 ≈ ||δ ∂F−11 ( z δ ) ∂zδ ||22 = ||δ exp ( zδ ) ||22 . ( 1 ) A similar example can be constructed for both the forward and backward passes , which speaks to the importance of bi-Lipschitz continuity . For an additional discussion on the connection of Lipschitz constants and numerical errors , we refer to Appendix B . 3 STABILITY OF INVERTIBLE NEURAL NETWORKS . 3.1 LIPSCHITZ BOUNDS FOR BUILDING BLOCKS OF INVERTIBLE NETWORKS . Research on invertible networks has produced a large variety of architectural building blocks . Yet , the focus of prior work was on obtaining flexible architectures while maintaining invertibility guarantees . Here , we build on the work in ( Behrmann et al. , 2019 ) , where bi-Lipschitz bounds were proven for invertible ResNets , by deriving Lipschitz bounds on the forward and inverse mapping of common building blocks . Together with an overview of common invertible building blocks , we provide our main results in Table 1 . We chose these particular model classes in order to cover both coupling-based approaches and free-form approaches like Neural ODE ( Chen et al. , 2018 ) and i-ResNets ( Behrmann et al. , 2019 ) . The derivations of the bounds are given in Appendix A . Note that the bounds provide the worst-case stability and serve mainly as a guideline for future designs of invertible building blocks . 3.2 CONTROLLING STABILITY OF BUILDING BLOCKS . As shown in Table 1 , there are many factors that influence the stability of INNs . Of particular importance are the Lipschitz constants Lip ( g ) of the sub-network g for i-ResNets ( Behrmann et al. , 2019 ) and affine coupling blocks ( Dinh et al. , 2014 ) , and Lip ( s ) , Lip ( t ) for additive coupling blocks ( Dinh et al. , 2017 ) . Whereas computing the Lipschitz constants of neural networks is NP-hard ( Virmaux & Scaman , 2018 ) , there is a simple data-independent upper bound : Lip ( g ) ≤ L∏ i=1 ‖Ai‖2 , for g ( x ) = AL ◦ φ ◦AL−1 ◦ · · · ◦A2 ◦ φ ◦A1 , ( 2 ) where Ai are linear layers , ‖ · ‖2 is the spectral norm and φ a contractive activation function ( Lip ( φ ) ≤ 1 ) . The above bound was used by ( Behrmann et al. , 2019 ) in conjunction with spectral normalization ( Miyato et al. , 2018 ; Gouk et al. , 2018 ) to ensure a contractive residual block g. In particular , this employs a normalization via : Ã = κ A σ̂1 , with σ̂1 ≈ σ1 = ‖A‖2 ( approx . via power-method ) , where κ > 0 is a coefficient that sets the approximate upper bound on the spectral norm of each linear layer Ai . Thus , by setting an appropriate coefficient κ depending on the targeted Lipschitz bound of the building block , this approach enables one to control both forward and inverse stability . Note that the above discussion can be generalized to other ` p-norms , see ( Chen et al. , 2019 ) . However , this is not sufficient when using affine coupling blocks because their bound on the Lipschitz constant holds only locally . In particular , it depends on the regions of the inputs x to the coupling block . While inputs to the first layer are usually bounded by the nature of the data , obtaining bounds for intermediate activations is less straightforward . One interesting avenue for future work could be local regularizers like gradient penalties ( Gulrajani et al. , 2017 ) , where spectral normalization could be used post-hoc to certify stability . Lastly , we use ActNorm ( Kingma & Dhariwal , 2018 ) in several architectures and avoid small diagonal terms which would yield large Lipschitz constants in the inverse ( see Table 1 ) by adding a positive constant . Further stabilization could be achieved via bounding the scaling .

Summary: The paper claims that for invertible neural networks, mathematical guarantees on invertibility is not enough, and we also require numerical invertibility. To this end, the lipschitz constants/condition numbers of Jacobians of both the forward and inverse maps of invertible NNs based on coupling layers are examined mathematically and experimentally. The paper also displays cases that expose non-invertibility in these architectures via gradient-based construction of adversarial inputs, as well as a decorrelation benchmark task, and show that spectral normalization can be a remedy for stabilizing these flows.