PIE: Pseudo-Invertible Encoder

1 INTRODUCTION . We consider the problem of information compression from high dimensional data . Where many studies consider the problem of compression by non-invertible transformations , we emphasize the importance of invertible compression as there are many cases where one can not or will not decide a priori what part of the information is important and what part is not . Compression of images for person ID in a small company requires less resolution then person ID at an airport . To loose part of the information without harm to the future purpose of viewing the picture requires knowing the purpose upfront . Therefore , the fundamental advantage of invertible information compression is that compression can be undone if a future purpose so requires . Recent advances of classification models have demonstrated that deep learning architectures of proper design do not lead to information loss while still being able to achieve state-of-the-art in classification performance . These i-RevNet models Jacobsen et al . ( 2018 ) implement a small but essential modification of the popular RevNet models while achieving invertibility and a performance similar to the standard RevNet Gomez et al . ( 2017 ) . This is of great interest as it contradicts the intuition that information loss is essential to achieve good performance in classification Tishby & Zaslavsky ( 2015 ) . Despite the requirement of the invertibility , flow-based generating models Dinh et al . ( 2014 ; 2016 ) ; Rezende & Mohamed ( 2015 ) ; Kingma & Dhariwal ( 2018 ) demonstrate that the combination of bijective mappings allows one to transform the raw distribution of the input data to any desired distribution and perform the manipulation of the data . On the other hand , Auto-Encoders have provided the ideal mechanism to reduce the data to the bare minimum while retaining all essential information for a specific task , the one implemented in the loss function . Variational Auto Encoders ( VAE ) Kingma & Welling ( 2013 ) and Wasserstein Auto Encoders ( WAE ) Tolstikhin et al . ( 2018 ) are performing best . They provide an approach for stable training of autoencoders , which demonstrate good results at reconstruction and generation . However , both of these methods involve the optimization of the objective defined on the pixel level . We would emphasise the importance of avoiding the separate decoder part and training the model without relying on the reconstuction quality directly . Combining the best of Invertible mappings and Auto-Encoders , we introduce Pseudo Invertible Encoder . Our model combines bijectives with restriction and extension of the mappings to the dependent sub-manifolds Fig . 1 . The main contributions of this paper are the following : • We introduce new class of likelihood-based Auto-Encoders , which we call Pseudo Invertible Encoders . We provide the theoretical explanation of their principles . • We demonstrate the properties of Gaussian Pseudo Invertible Encoder in manifold learning . • We compare our model with WAE and VAE on MNIST , and report that the sharpness of the images , generated by our models is better . 2 RELATED WORK . 2.1 INVERTIBLE MODELS . ResNets He et al . ( 2016 ) enable Networks to grow even more and thus memory consumption becomes a bottleneck . Gomez et al . ( 2017 ) propose a Reversible Residual Network ( RevNet ) where each layer ’ s activations can be reconstructed from the activations of the next layer . By replacing the residual blocks with coupling layers , they mimic the behaviour of residual blocks while being able to retrieve the original input of the layer . RevNet replaces the residual blocks of ResNets , but also accommodates non-invertible components to train more efficiently . By adding a downsampling operator to the coupling layer , i-RevNet circumvents these non-invertible modules ( Jacobsen et al. , 2018 ) . With this they show that losing information is not a necessary condition to learn representations that generalize well on complicated problems . Although i-RevNet circumvents non-invertible modules , data is not compressed and the model is only invertible up to the last layer . All their methods do not allow dimensionality reduction . In current research we build a pseudo invertible model which performs dimensionality reduction . 2.2 AUTOENCODERS . Auto-Encoders were first introduced by Rummerhart ( 1986 ) as an unsupervised learning algorithm . They are now widely used as a technique for dimension reduction by compressing input data . By training an encoder and a decoder network , and measuring the distance between original and reconstructed data , data can be represented in a latent space . This latent space can then be used for supervised learning algorithms . Instead of learning a compressed representation of the input data Kingma & Welling ( 2013 ) propose to learn the parameters of a probability distribution that represent the data . tol introduced new class of models - Wasserstein Auto Encoders , which use Optimal Transport to be trained . These methods require the optimization of the objective function which includes the terms defined on pixel level . Our model does not require such optimization . Moreover , it only perform encoding at training time . 3 THEORY . Here we introduce the approach for obtaining dimensionality reduction invertible mappings . Our method is based on the restriction of the mappings to low-dimensional manifolds , and extension of the inverse mappings with certain constraints ( Fig . 2 ) . 3.1 RESTRICTION-EXTENSION APPROACH . Given data xi ∈ X ⊂ RD . Assuming that X is a d-dimensional manifold , with d < D , we seek to find a mapping G : RD → Rd invertible on X . In other words , we are looking for a pair of associated functions G and G−1 such that { G ( X ) = Z ⊂ Rd G−1 ( Z ) = X ( 1 ) LetR be an open set in RD−d . We use this residual manifold in order to match the dimensionalities of the hidden and initial spaces . Here we introduce the function g : Rd → RD−d . With no loss of generality we can say thatR = g ( Z ) . We use the pair of extended functions G̃ : RD → Rd×RD−d and G̃−1 : Rd × RD−d → RD to rewrite Eq . 1 : { G̃ ( X ) = Z ×R G̃−1 ( Z ×R ) = X ( 2 ) Rather than searching for the invertible dimensionality reduction mapping directly , we seek to find G̃ , the invertible transformation with certain constraints , expressed byR . In search for G̃ , we focus on Fθ : RD → RD , Fθ ∈ F , where F is a parametric family of functions invertible on RD . We select the function Fθ with parameters θ which satisfy the constraint : F−1θ ◦ PRd×R ◦ Fθ = idX ( 3 ) where PRd×R is the orthogonal projection from Rd × RD−d to Rd ×R . Taking into account constraint 3 , we derive Fθ ( x ) = [ z , r ] , where z ∈ Z and r ∈ R. By combining this with Eq . 2 we have the desired pair of functions : { G ( x ) = z , G−1 ( z ) = F−1θ ( [ z , g ( z ) ] ) ( 4 ) The obtained function G is Pseudo Invertible Endocer , or shortly PIE . 3.2 LOG LIKELIHOOD MAXIMIZATION . As we are interested in high dimensional data such as images , the explicit choice of parameters θ is impossible . We choose θ∗ as a maximizer of the log likelihood of the observed data given the prior pθ ( x ) : θ∗ = argmax θ [ log pθ ( x ) ] ( 5 ) After a change of variables according to Eq . 4 we obtain p ( x ) = p ( Fθ ( x ) ) ∣∣∣ det ( ∂Fθ ∂xT ) ∣∣∣ ( 6 ) Taking into account the constraint 3 we derive the joint distribution for Fθ ( x ) = [ z , r ] p ( Fθ ( x ) ) = p ( z , r ) = p ( r|z ) p ( z ) ( 7 ) ∫ X p ( Fθ ( x ) ) dx = ∫ R=g ( Z ) ∫ Z p ( r|z ) p ( z ) drdz = ∫ R ∫ Z δ ( r− g ( z ) ) p ( z ) drdz ( 8 ) p ( Fθ ( x ) ) = δ ( r− g ( z ) ) p ( z ) ( 9 ) Dirac ’ s delta function can be viewed as a limit of sequence of Gaussians : δ ( x ) = lim →0 N ( x|0 , 2I ) ( 10 ) Let us fix 2 = 20 1 . Then δ ( x ) ≈ N ( x|0 , 20I ) ( 11 ) δ ( r− g ( z ) ) ≈ N ( r|g ( z ) , 20I ) ( 12 ) Finally , for the log likelihood we have : log p ( x ) ≈ log p ( z ) + logN ( r|g ( z ) , 20I ) + log ∣∣∣det ( ∂Fθ ∂xT ) ∣∣∣ ( 13 ) We choose prior distribution p ( z ) as Standard Gaussian . We search for the parameters by using Gradient Descent . 3.3 COMPOSITION OF BIJECTIVES . The method relies on the function Fθ . This choice is challenging by itself . The currently known classes of real-value bijectives are limited . To overcome this issue , we approximate Fθ with a composition of basic bijectives from certain classes : F = FK ◦ Fk−1 ◦ . . . ◦ F2 ◦ F1 ( 14 ) where Fj = Fj ( ·|θj ) ∈ Fj , j = 1 . . .K . Taking into account that a composition of PIE is also PIE , we create a final dimensionality reduction mapping from a sequence of PIEs : X ↔ Y1 ↔ Y2 ↔ · · · ↔ YL ↔ Z1 ( 15 ) such that D > dimY1 > dimY2 > . . . > dimYL > d ( 16 ) where L < D − d. Then the log likelihood is represented as log p ( x ) ≈ log p ( z ) + L∑ l=1 logN ( rl|gl ( zl ) , 20I ) + L∑ l=1 Kl∑ k=1 log |det ( Jkl ) | ( 17 ) where Jkl is the Jacobian of the k-th function of the l-th PIE . The approximation error here depends only on , according to the Eq . 10 . For the simplicity we will now refer to the whole model as PIE . The building blocks of this model are PIE blocks . 3.4 RELATION TO NORMALIZING FLOWS . If we choose the distribution p ( z ) in Eq . 17 as Standard Gaussian , gl ( · ) = 0 , ∀l and 0 = 1 , then the model can be viewed as Normalizing Flow with multi-scale architecture Dinh et al . ( 2016 ) Fig . 3 . It was demonstrated in Dinh et al . ( 2016 ) that the model with such architecture achieves semantic compression . 4 PSEUDO-INVERTIBLE ENCODER . This section introduces the basic bijectives for the Pseudo-Invertible Encoder ( PIE ) . We explain what each building bijective consists of and how it fits in the global architecture as shown in Fig . 4 . 4.1 ARCHITECTURE . PIE is composed of a series of convolutional blocks followed by linear blocks , as depicted in Fig . 4 . The convolutional PIE blocks consist of series of coupling layers and 1×1 convolutions . We perform invertible downsampling of the image at the beginning of the convolutional block , by reducing the spatial resolution and increasing the number of channels , keeping the overall number of the variables the same . At the end of the convolutional PIE block , the split of variables is performed . One part of the variables is projected to the residual manifold R while others is feed to the next block . The linear PIE blocks are constructed in the same manner . However , the downsampling is not performed and 1× 1 convolutions are replaced invertible linear mappings .

Summary: PIE extend NICE and Real NVP into situations which require having a smaller dimensionality of the latent variable (d) compared to the dimensionality of the observed variable (D), i.e. d < D. This is done by learning an extension function g(z) from R^d to R^{D-d} and then using the change of variables formula on x and [z, g(z)]. To model probabilistically the deterministic function g(z) is replaced by Normal distribution with mean g(z) and a small variance.