On the Inversion of Deep Generative Models

1 INTRODUCTION . In the past several years , deep generative models , e.g . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) and Variational Auto-Encoders ( VAEs ) ( Kingma & Welling , 2013 ) , have been greatly developed , leading to networks that can generate images , videos , and speech voices among others , that look and sound authentic to humans . Loosely speaking , these models learn a mapping from a random low-dimensional latent space to the training data distribution , obtained in an unsupervised manner . Interestingly , deep generative models are not used only to generate arbitrary signals . Recent work rely on the inversion of these models to perform visual manipulations , compressed sensing , image interpolation , and more ( Zhu et al. , 2016 ; Bora et al. , 2017 ; Simon & Aberdam , 2020 ) . In this work , we study this inversion task . Formally , denoting the signal to invert by y ∈ Rn , the generative model as G : Rn0 → Rn , and the latent vector as z ∈ Rn0 , we study the following problem : z∗ = argmin z 1 2 ‖G ( z ) − y‖22 , ( 1 ) where G is assumed to be a feed-forward neural network . The first question that comes to mind is whether this model is invertible , or equivalently , does Equation 1 have a unique solution ? In this work , we establish theoretical conditions that guarantee the invertibility of the model G. Notably , the provided theorems are applicable to general non-random generative models , and do not depend on the chosen inversion algorithm . Once the existence of a unique solution is recognized , the next challenge is to provide a recovery algorithm that is guaranteed to obtain the sought solution . A common and simple approach is to draw a random vector z and iteratively update it using gradient descent , opting to minimize Equation 1 ( Zhu et al. , 2016 ; Bora et al. , 2017 ) . Unfortunately , this approach has theoretical guarantees only in limited scenarios ( Hand et al. , 2018 ; Hand & Voroninski , 2019 ) , since the inversion problem is generally non-convex . An alternative approach is to train an encoding neural network that maps images to their latent vectors ( Zhu et al. , 2016 ; Donahue et al. , 2016 ; Bau et al. , 2019 ; Simon & Aberdam , 2020 ) ; however , this method is not accompanied by any theoretical justification . We adopt a third approach in which the generative model is inverted in an analytical fashion . Specifically , we perform the inversion layer-by-layer , similar to Lei et al . ( 2019 ) . Our approach is based on the observation that every hidden layer is an outcome of a weight matrix multiplying a sparse vector , followed by a ReLU activation . By utilizing sparse representation theory , the proposed algorithm ensures perfect recovery in the noiseless case and bounded estimation error in the noisy one . Moreover , we show numerically that our algorithm outperforms gradient descent in several tasks , including reconstruction of noiseless and corrupted images . Main contributions : The contributions of this work are both theoretical and practical . We derive theoretical conditions for the invertiblity of deep generative models by ensuring a unique solution for the inversion problem defined in Equation 1 . In short , these conditions rely on the growth of the non-zero elements of consecutive hidden layers by a factor of 2 for trained networks and by any constant greater than 1 for random models . Then , by leveraging the inherent sparsity of the hidden layers , we introduce a layerwise inversion algorithm with provable guarantees in the noiseless and noisy settings for fully-connected generators . To the best of our knowledge , this is the first work that provides such guarantees for general ( non-random ) models , addressing both the conceptual inversion and provable algorithms for solving Equation 1 . Finally , we provide numerical experiments , demonstrating the superiority of our approach over gradient descent in various scenarios . 1.1 RELATED WORK . Inverting deep generative models : A tempting approach for solving Equation 1 is to use first order methods such as gradient descent . Even though this inversion is generally non-convex , the works in Hand & Voroninski ( 2019 ) ; Hand et al . ( 2018 ) show that if the weights are random then , under additional assumptions , no spurious stationary points exist , and thus gradient descent converges to the optimum . A different analysis , given in Latorre et al . ( 2019 ) , studies the case of strongly smooth generative models that are near isometry . In this work , we study the inversion of general ( non-random and non-smooth ) ReLU activated generative networks , and provide a provable algorithm that empirically outperforms gradient descent . A close but different line of theoretical work analyzes the compressive sensing abilities of trained deep generative networks ( Shah & Hegde , 2018 ; Bora et al. , 2017 ) ; however , these works assume that an ideal inversion algorithm , solving Equation 1 , exists . Different works Bojanowski et al . ( 2017 ) ; Wu et al . ( 2019 ) suggest training procedures that result with generative models that can be easily inverted . Nevertheless , in this work we do not assume anything on the training procedure itself , and only rely on the weights of the trained model . Layered-wise inversion : The closest work to ours , and indeed its source of inspiration , is Lei et al . ( 2019 ) , which proposes a novel scheme for inverting generative models . By assuming that the input signal was corrupted by bounded noise in terms of ` 1 or ` ∞ , they suggest inverting the model using linear programs layer-by-layer . That said , to assure a stable inversion , their analysis is restricted to cases where : ( i ) the network weights are Gaussian i.i.d . variables ; ( ii ) the layers expand such that the number of non-zero elements in each layer is larger than the size of the entire layer preceding it ; and ( iii ) that the last activation function is either ReLU or leaky-ReLU . Unfortunately , as mentioned in their work , these three assumptions often do not hold in practice . In this work , we do not rely on the distribution of the weights nor on the chosen activation function of the last layer . Furthermore , we relax the expansion assumption as to rely only on the expansion of the number of non-zero elements . This relaxation is especially needed in the last hidden layer , which is typically larger than the image size . Neural networks and sparse representation : In the search for a profound theoretical understanding for deep learning , a series of papers suggested a connection between neural networks and sparse coding , by demonstrating that the forward pass of a neural network is in fact a pursuit for a multilayer sparse representation ( Papyan et al. , 2017 ; Sulam et al. , 2018 ; Chun & Fessler , 2019 ; Sulam et al. , 2019 ; Romano et al. , 2019 ; Xin et al. , 2016 ) . In this work , we expand this proposition by showing that the inversion of a generative model is based on sequential sparse coding steps . 2 THE GENERATIVE MODEL . Notations : We use bold uppercase letters to represent matrices , and bold lowercase letters to represent vectors . The vector wj represents the jth column in the matrix W. Similarly , the vector wi , j represents the jth column in the matrix Wi . The activation function ReLU is the entry-wise operator ReLU ( u ) = max { u,0 } . We denote by spark ( W ) the smallest number of columns in W that are linearly-dependent , and by ‖x‖0 the number of non-zero elements in x . The mutual coherence of a matrix W is defined as : µ ( W ) = maxi 6=j |wTi wj| ‖wi‖2‖wj‖2 . Finally , we define xS and WSi as the supported vector and the row-supported matrix according to the set S , and denote by Sc the complementary set of S. Problem Statement : We consider a typical generative scheme G : Rn0 → Rn of the form : x0 = z , xi+1 = ReLU ( Wixi ) , for all i ∈ { 0 , . . . , L− 1 } , G ( z ) = φ ( WLxL ) , ( 2 ) where xi ∈ Rni , { xi } L−1i=1 are the hidden layers , Wi ∈ Rni+1×ni are the weight matrices ( nL+1 = n ) , x0 = z ∈ Rn0 is the latent vector that is usually randomly selected from a normal distribution , z ∼ N ( 0 , σ2In0 ) , and φ is an invertible activation function , e.g . tanh , sigmoid , or piece-wise linear . Given a sample x = G ( z ) , that was created by the generative model above , we aim to recover its latent vector z . Note that each hidden vector in the model is produced by a ReLU activation , leading to hidden layers that are inherently sparse . This observation supports our approach to study this model utilizing sparse representation theory . In what follows , we use this observation to derive theoretical statements on the invertibility and the stability of this problem , and to develop pursuit algorithms that are guaranteed to restore the original latent vector . 3 INVERTIBILITY AND UNIQUENESS . We start by addressing this question : “ Is this generative process invertible ? ” . In other words , when given a signal that was generated by the model , x = G ( z∗ ) , we know that a solution z∗ to the inverse problem exists ; however , can we ensure that this is the only one ? Theorem 1 below ( its proof is given in Appendix A ) provides such guarantees , which are based on the sparsity level of the hidden layers and the spark of the weight matrices ( see Section 2 ) . Importantly , this theorem is not restricted to a specific pursuit algorithm ; it can rather be used for any restoration method ( gradient descent , deep encoder , etc . ) to determine whether the recovered latent vector is the unique solution . Definition 1 ( sub-spark ) . Define the s-sub-spark of a matrix W as the minimal spark of any subset S of rows of cardinality |S| = s , sub-spark ( W , s ) = min|S|=s spark ( WS ) . Definition 2 ( sub-rank ) . Define the s-sub-rank of a matrix W as the minimal rank over any subset S of rows of cardinality |S| = s , sub-rank ( W , s ) = min|S|=s rank ( WS ) . Theorem 1 ( Uniqueness ) . Consider the generative scheme described in Equation 2 and a signal x = G ( z∗ ) with a corresponding set of representations { x∗i } Li=1 that satisfy : ( i ) sL = ‖x∗L‖0 < spark ( WL ) 2 . ( ii ) si = ‖x∗i ‖0 < sub-spark ( Wi , si+1 ) 2 , for all i ∈ { 1 , . . . , L− 1 } . ( iii ) n0 = sub-rank ( W0 , s1 ) ≤ s1 . Then , z∗ is the unique solution to the inverse problem that meets these sparsity conditions . Theorem 1 is the first of its kind to provide uniqueness guarantees for general non-statistical weight matrices . Moreover , it only requires an expansion of the layer cardinalities as opposed to Huang et al . ( 2018 ) ; Hand & Voroninski ( 2019 ) and Lei et al . ( 2019 ) that require dimensionality expansion that often does not hold for the last layer ( typically n < nL ) . A direct corollary of Theorem 1 is in the case of random matrices . In such case , the probability of heaving n linearly dependent columns is essentially zero ( Elad , 2010 , Chapter 2 ) . Hence , the conditions of Theorem 1 become : ( i ) sL < n+ 1 2 . ( ii ) si < si+1 + 1 2 . ( iii ) s1 ≥ n0 . ( 3 ) In fact , since singular square matrices have Lebesgue measure zero , this corollary holds for almost all set of matrices . In practice , to allow for a sufficient increase in the cardinalities of the hidden layers , their dimensions should expand as well , excluding the last layer . For example , if the dimensions of the hidden layers increase by a factor of 2 , as long as the hidden layers preserve a constant percentage of non-zero elements , Theorem 1 holds almost surely . Notably , this is the common practice in various generative architectures , such as DC-GAN Radford et al . ( 2015 ) and PGAN Karras et al . ( 2017 ) . Nevertheless , in the random setting , we can further relax the above conditions by utilizing a theorem by Foucart & Rauhut ( 2013 ) . This theorem considers a typical sparse representation model with a random dictionary and states that a sparse representation is unique as long as its cardinality is smaller than the signal dimension . Therefore , as presented in Theorem 2 , in the random setting the cardinality across the layers need to grow only by a constant , i.e . si < si+1 and sL < n. Theorem 2 ( Uniqueness for Random Weight Matrices ) . Assume that the weight matrices comprise of random independent and identically distributed entries ( say Gaussian ) . If the representations of a signal x = G ( z∗ ) satisfy : ( i ) sL = ‖xL‖0 < n. ( ii ) si = ‖xi‖0 < si+1 , for all i ∈ { 1 , . . . , L− 1 } . ( iii ) s1 = ‖x1‖0 ≥ n0 , then , with probability 1 , the inverse problem has a unique solution that meets these conditions . The above theorem states that to ensure a unique global minimum in the stochastic case , the number of nonzero elements should expand by only a single parameter . The proof of this theorem follows the same protocol as Theorem 1 ’ s proof , while replacing the spark-based uniqueness ( Donoho & Elad , 2003 ) with Foucart & Rauhut ( 2013 ) . As presented in Section 6.1 , these conditions are very effective in predicting whether the generative process is invertible or not , regardless of the recovery algorithm used .

Summary: In this submission, the authors study the inversion of ReLU networks (where the output of the network is subject to an invertible activation function).  This is an important task, for example for inverse problems using generative priors.  The authors introduce spark-based conditions for the invertibility of each layer of the network, leveraging sparsity that is induced by ReLUs.  The authors also introduce a novel layer wise inversion algorithm and provide provable recovery guarantees in both noisy and noiseless settings.  Empirical results demonstrate the superiority of the proposed algorithm relative to baselines for inversion in particular parameter regimes.