PIE: Pseudo-Invertible Encoder

1 INTRODUCTION . We consider the problem of information compression from high dimensional data . Where many studies consider the problem of compression by non-invertible transformations , we emphasize the importance of invertible compression as there are many cases where one can not or will not decide a priori what part of the information is important and what part is not . Compression of images for person ID in a small company requires less resolution then person ID at an airport . To loose part of the information without harm to the future purpose of viewing the picture requires knowing the purpose upfront . Therefore , the fundamental advantage of invertible information compression is that compression can be undone if a future purpose so requires . Recent advances of classification models have demonstrated that deep learning architectures of proper design do not lead to information loss while still being able to achieve state-of-the-art in classification performance . These i-RevNet models Jacobsen et al . ( 2018 ) implement a small but essential modification of the popular RevNet models while achieving invertibility and a performance similar to the standard RevNet Gomez et al . ( 2017 ) . This is of great interest as it contradicts the intuition that information loss is essential to achieve good performance in classification Tishby & Zaslavsky ( 2015 ) . Despite the requirement of the invertibility , flow-based generating models Dinh et al . ( 2014 ; 2016 ) ; Rezende & Mohamed ( 2015 ) ; Kingma & Dhariwal ( 2018 ) demonstrate that the combination of bijective mappings allows one to transform the raw distribution of the input data to any desired distribution and perform the manipulation of the data . On the other hand , Auto-Encoders have provided the ideal mechanism to reduce the data to the bare minimum while retaining all essential information for a specific task , the one implemented in the loss function . Variational Auto Encoders ( VAE ) Kingma & Welling ( 2013 ) and Wasserstein Auto Encoders ( WAE ) Tolstikhin et al . ( 2018 ) are performing best . They provide an approach for stable training of autoencoders , which demonstrate good results at reconstruction and generation . However , both of these methods involve the optimization of the objective defined on the pixel level . We would emphasise the importance of avoiding the separate decoder part and training the model without relying on the reconstuction quality directly . Combining the best of Invertible mappings and Auto-Encoders , we introduce Pseudo Invertible Encoder . Our model combines bijectives with restriction and extension of the mappings to the dependent sub-manifolds Fig . 1 . The main contributions of this paper are the following : • We introduce new class of likelihood-based Auto-Encoders , which we call Pseudo Invertible Encoders . We provide the theoretical explanation of their principles . • We demonstrate the properties of Gaussian Pseudo Invertible Encoder in manifold learning . • We compare our model with WAE and VAE on MNIST , and report that the sharpness of the images , generated by our models is better . 2 RELATED WORK . 2.1 INVERTIBLE MODELS . ResNets He et al . ( 2016 ) enable Networks to grow even more and thus memory consumption becomes a bottleneck . Gomez et al . ( 2017 ) propose a Reversible Residual Network ( RevNet ) where each layer ’ s activations can be reconstructed from the activations of the next layer . By replacing the residual blocks with coupling layers , they mimic the behaviour of residual blocks while being able to retrieve the original input of the layer . RevNet replaces the residual blocks of ResNets , but also accommodates non-invertible components to train more efficiently . By adding a downsampling operator to the coupling layer , i-RevNet circumvents these non-invertible modules ( Jacobsen et al. , 2018 ) . With this they show that losing information is not a necessary condition to learn representations that generalize well on complicated problems . Although i-RevNet circumvents non-invertible modules , data is not compressed and the model is only invertible up to the last layer . All their methods do not allow dimensionality reduction . In current research we build a pseudo invertible model which performs dimensionality reduction . 2.2 AUTOENCODERS . Auto-Encoders were first introduced by Rummerhart ( 1986 ) as an unsupervised learning algorithm . They are now widely used as a technique for dimension reduction by compressing input data . By training an encoder and a decoder network , and measuring the distance between original and reconstructed data , data can be represented in a latent space . This latent space can then be used for supervised learning algorithms . Instead of learning a compressed representation of the input data Kingma & Welling ( 2013 ) propose to learn the parameters of a probability distribution that represent the data . tol introduced new class of models - Wasserstein Auto Encoders , which use Optimal Transport to be trained . These methods require the optimization of the objective function which includes the terms defined on pixel level . Our model does not require such optimization . Moreover , it only perform encoding at training time . 3 THEORY . Here we introduce the approach for obtaining dimensionality reduction invertible mappings . Our method is based on the restriction of the mappings to low-dimensional manifolds , and extension of the inverse mappings with certain constraints ( Fig . 2 ) . 3.1 RESTRICTION-EXTENSION APPROACH . Given data xi ∈ X ⊂ RD . Assuming that X is a d-dimensional manifold , with d < D , we seek to find a mapping G : RD → Rd invertible on X . In other words , we are looking for a pair of associated functions G and G−1 such that { G ( X ) = Z ⊂ Rd G−1 ( Z ) = X ( 1 ) LetR be an open set in RD−d . We use this residual manifold in order to match the dimensionalities of the hidden and initial spaces . Here we introduce the function g : Rd → RD−d . With no loss of generality we can say thatR = g ( Z ) . We use the pair of extended functions G̃ : RD → Rd×RD−d and G̃−1 : Rd × RD−d → RD to rewrite Eq . 1 : { G̃ ( X ) = Z ×R G̃−1 ( Z ×R ) = X ( 2 ) Rather than searching for the invertible dimensionality reduction mapping directly , we seek to find G̃ , the invertible transformation with certain constraints , expressed byR . In search for G̃ , we focus on Fθ : RD → RD , Fθ ∈ F , where F is a parametric family of functions invertible on RD . We select the function Fθ with parameters θ which satisfy the constraint : F−1θ ◦ PRd×R ◦ Fθ = idX ( 3 ) where PRd×R is the orthogonal projection from Rd × RD−d to Rd ×R . Taking into account constraint 3 , we derive Fθ ( x ) = [ z , r ] , where z ∈ Z and r ∈ R. By combining this with Eq . 2 we have the desired pair of functions : { G ( x ) = z , G−1 ( z ) = F−1θ ( [ z , g ( z ) ] ) ( 4 ) The obtained function G is Pseudo Invertible Endocer , or shortly PIE . 3.2 LOG LIKELIHOOD MAXIMIZATION . As we are interested in high dimensional data such as images , the explicit choice of parameters θ is impossible . We choose θ∗ as a maximizer of the log likelihood of the observed data given the prior pθ ( x ) : θ∗ = argmax θ [ log pθ ( x ) ] ( 5 ) After a change of variables according to Eq . 4 we obtain p ( x ) = p ( Fθ ( x ) ) ∣∣∣ det ( ∂Fθ ∂xT ) ∣∣∣ ( 6 ) Taking into account the constraint 3 we derive the joint distribution for Fθ ( x ) = [ z , r ] p ( Fθ ( x ) ) = p ( z , r ) = p ( r|z ) p ( z ) ( 7 ) ∫ X p ( Fθ ( x ) ) dx = ∫ R=g ( Z ) ∫ Z p ( r|z ) p ( z ) drdz = ∫ R ∫ Z δ ( r− g ( z ) ) p ( z ) drdz ( 8 ) p ( Fθ ( x ) ) = δ ( r− g ( z ) ) p ( z ) ( 9 ) Dirac ’ s delta function can be viewed as a limit of sequence of Gaussians : δ ( x ) = lim →0 N ( x|0 , 2I ) ( 10 ) Let us fix 2 = 20 1 . Then δ ( x ) ≈ N ( x|0 , 20I ) ( 11 ) δ ( r− g ( z ) ) ≈ N ( r|g ( z ) , 20I ) ( 12 ) Finally , for the log likelihood we have : log p ( x ) ≈ log p ( z ) + logN ( r|g ( z ) , 20I ) + log ∣∣∣det ( ∂Fθ ∂xT ) ∣∣∣ ( 13 ) We choose prior distribution p ( z ) as Standard Gaussian . We search for the parameters by using Gradient Descent . 3.3 COMPOSITION OF BIJECTIVES . The method relies on the function Fθ . This choice is challenging by itself . The currently known classes of real-value bijectives are limited . To overcome this issue , we approximate Fθ with a composition of basic bijectives from certain classes : F = FK ◦ Fk−1 ◦ . . . ◦ F2 ◦ F1 ( 14 ) where Fj = Fj ( ·|θj ) ∈ Fj , j = 1 . . .K . Taking into account that a composition of PIE is also PIE , we create a final dimensionality reduction mapping from a sequence of PIEs : X ↔ Y1 ↔ Y2 ↔ · · · ↔ YL ↔ Z1 ( 15 ) such that D > dimY1 > dimY2 > . . . > dimYL > d ( 16 ) where L < D − d. Then the log likelihood is represented as log p ( x ) ≈ log p ( z ) + L∑ l=1 logN ( rl|gl ( zl ) , 20I ) + L∑ l=1 Kl∑ k=1 log |det ( Jkl ) | ( 17 ) where Jkl is the Jacobian of the k-th function of the l-th PIE . The approximation error here depends only on , according to the Eq . 10 . For the simplicity we will now refer to the whole model as PIE . The building blocks of this model are PIE blocks . 3.4 RELATION TO NORMALIZING FLOWS . If we choose the distribution p ( z ) in Eq . 17 as Standard Gaussian , gl ( · ) = 0 , ∀l and 0 = 1 , then the model can be viewed as Normalizing Flow with multi-scale architecture Dinh et al . ( 2016 ) Fig . 3 . It was demonstrated in Dinh et al . ( 2016 ) that the model with such architecture achieves semantic compression . 4 PSEUDO-INVERTIBLE ENCODER . This section introduces the basic bijectives for the Pseudo-Invertible Encoder ( PIE ) . We explain what each building bijective consists of and how it fits in the global architecture as shown in Fig . 4 . 4.1 ARCHITECTURE . PIE is composed of a series of convolutional blocks followed by linear blocks , as depicted in Fig . 4 . The convolutional PIE blocks consist of series of coupling layers and 1×1 convolutions . We perform invertible downsampling of the image at the beginning of the convolutional block , by reducing the spatial resolution and increasing the number of channels , keeping the overall number of the variables the same . At the end of the convolutional PIE block , the split of variables is performed . One part of the variables is projected to the residual manifold R while others is feed to the next block . The linear PIE blocks are constructed in the same manner . However , the downsampling is not performed and 1× 1 convolutions are replaced invertible linear mappings .

Summary: In this paper, an invertible encoding method is proposed for learning latent representations and deep generators via inverting the encoder. The proposed method can be seen as an autoencoder without the need to learn the decoder. This can be computed by inverting the encoder. To the best of my knowledge the proposed method is novel and its building blocks are described adequately.