Title: PIE: PSEUDO-INVERTIBLE ENCODER

Abstract: We consider the problem of information compression from high dimensional data. Where many studies consider the problem of compression by non-invertible transformations, we emphasize the importance of invertible compression. We introduce new class of likelihood-based Auto-Encoders with pseudo bijective architecture, which we call Pseudo Invertible Encoders. We provide the theoretical explanation of their principles. We evaluate Gaussian Pseudo Invertible Encoder on MNIST, where our model outperforms WAE and VAE in sharpness of the generated images.

1. INTRODUCTION:

We consider the problem of information compression from high dimensional data. Where many studies consider the problem of compression by non-invertible transformations, we emphasize the importance of invertible compression as there are many cases where one cannot or will not decide a priori what part of the information is important and what part is not. Compression of images for person ID in a small company requires less resolution then person ID at an airport. To loose part of the information without harm to the future purpose of viewing the picture requires knowing the purpose upfront. Therefore, the fundamental advantage of invertible information compression is that compression can be undone if a future purpose so requires.

Recent advances of classification models have demonstrated that deep learning architectures of proper design do not lead to information loss while still being able to achieve state-of-the-art in classification performance. These i-RevNet models Jacobsen et al. (2018) implement a small but essential modification of the popular RevNet models while achieving invertibility and a performance similar to the standard RevNet Gomez et al. (2017) . This is of great interest as it contradicts the intuition that information loss is essential to achieve good performance in classification Tishby & Zaslavsky (2015) . Despite the requirement of the invertibility, flow-based generating models Dinh et al. (2014; 2016) ; Rezende & Mohamed (2015) ; Kingma & Dhariwal (2018) demonstrate that the combination of bijective mappings allows one to transform the raw distribution of the input data to any desired distribution and perform the manipulation of the data.

On the other hand, Auto-Encoders have provided the ideal mechanism to reduce the data to the bare minimum while retaining all essential information for a specific task, the one implemented in the loss function. Variational Auto Encoders (VAE) Kingma & Welling (2013) and Wasserstein Auto Encoders (WAE) Tolstikhin et al. (2018) are performing best. They provide an approach for stable training of autoencoders, which demonstrate good results at reconstruction and generation. However, both of these methods involve the optimization of the objective defined on the pixel level. We would emphasise the importance of avoiding the separate decoder part and training the model without relying on the reconstuction quality directly.

Combining the best of Invertible mappings and Auto-Encoders, we introduce Pseudo Invertible Encoder. Our model combines bijectives with restriction and extension of the mappings to the dependent sub-manifolds Fig. 1 . The main contributions of this paper are the following:

• We introduce new class of likelihood-based Auto-Encoders, which we call Pseudo Invertible Encoders. We provide the theoretical explanation of their principles. • We demonstrate the properties of Gaussian Pseudo Invertible Encoder in manifold learning.

• We compare our model with WAE and VAE on MNIST, and report that the sharpness of the images, generated by our models is better. 2016) enable Networks to grow even more and thus memory consumption becomes a bottleneck. Gomez et al. (2017) propose a Reversible Residual Network (RevNet) where each layer's activations can be reconstructed from the activations of the next layer. By replacing the residual blocks with coupling layers, they mimic the behaviour of residual blocks while being able to retrieve the original input of the layer. RevNet replaces the residual blocks of ResNets, but also accommodates non-invertible components to train more efficiently. By adding a downsampling operator to the coupling layer, i-RevNet circumvents these non-invertible modules (Jacobsen et al., 2018) . With this they show that losing information is not a necessary condition to learn representations that generalize well on complicated problems. Although i-RevNet circumvents non-invertible modules, data is not compressed and the model is only invertible up to the last layer. All their methods do not allow dimensionality reduction. In current research we build a pseudo invertible model which performs dimensionality reduction.

2.2. AUTOENCODERS:

Auto-Encoders were first introduced by Rummerhart (1986) as an unsupervised learning algorithm.

They are now widely used as a technique for dimension reduction by compressing input data. By training an encoder and a decoder network, and measuring the distance between original and reconstructed data, data can be represented in a latent space. This latent space can then be used for supervised learning algorithms. Instead of learning a compressed representation of the input data Kingma & Welling (2013) propose to learn the parameters of a probability distribution that represent the data. tol introduced new class of models -Wasserstein Auto Encoders, which use Optimal Transport to be trained. These methods require the optimization of the objective function which includes the terms defined on pixel level. Our model does not require such optimization. Moreover, it only perform encoding at training time.

3. THEORY:

Here we introduce the approach for obtaining dimensionality reduction invertible mappings. Our method is based on the restriction of the mappings to low-dimensional manifolds, and extension of the inverse mappings with certain constraints (Fig. 2 ).

3.1. RESTRICTION-EXTENSION APPROACH:

Given data x i ∈ X ⊂ R D . Assuming that X is a d-dimensional manifold, with d < D, we seek to find a mapping G : R D → R d invertible on X . In other words, we are looking for a pair of associated functions G and

G -1 such that G(X ) = Z ⊂ R d G -1 (Z) = X (1)

Let R be an open set in R D-d . We use this residual manifold in order to match the dimensionalities of the hidden and initial spaces. Here we introduce the function g : R d → R D-d . With no loss of generality we can say that R = g(Z). We use the pair of extended functions G :

EQUATION

Rather than searching for the invertible dimensionality reduction mapping directly, we seek to find G, the invertible transformation with certain constraints, expressed by R.

In search for G, we focus on F θ : R D → R D , F θ ∈ F, where F is a parametric family of functions invertible on R D . We select the function F θ with parameters θ which satisfy the constraint:

F -1 θ • P R d ×R • F θ = id X (3) where P R d ×R is the orthogonal projection from R d × R D-d to R d × R.

Taking into account constraint 3, we derive F θ (x) = [z, r], where z ∈ Z and r ∈ R. By combining this with Eq. 2 we have the desired pair of functions:

EQUATION

The obtained function G is Pseudo Invertible Endocer, or shortly PIE.

3.2. LOG LIKELIHOOD MAXIMIZATION:

As we are interested in high dimensional data such as images, the explicit choice of parameters θ is impossible. We choose θ * as a maximizer of the log likelihood of the observed data given the prior p θ (x):

EQUATION

After a change of variables according to Eq. 4 we obtain Taking into account the constraint 3 we derive the joint distribution for

EQUATION

EQUATION

Dirac's delta function can be viewed as a limit of sequence of Gaussians:

EQUATION

Finally, for the log likelihood we have:

EQUATION

We choose prior distribution p(z) as Standard Gaussian. We search for the parameters by using Gradient Descent.

3.3. COMPOSITION OF BIJECTIVES:

The method relies on the function F θ . This choice is challenging by itself. The currently known classes of real-value bijectives are limited. To overcome this issue, we approximate F θ with a composition of basic bijectives from certain classes:

EQUATION

)

where

F j = F j (•|θ j ) ∈ F j , j = 1 . . . K.

Taking into account that a composition of PIE is also PIE, we create a final dimensionality reduction mapping from a sequence of PIEs:

EQUATION

x y

K l times x z K l times . . . . . . x z Convolutional Linear Downsampling 1 × 1 Conv, Linear Coupling Split method

Figure 4 : Architecture of the Pseudo-Invertible Encoder. PIE consists of convolutional and linear blocks which can be repeated multiple times, as denoted by the three dots between the block structure at the bottom. Each block has K l repetitions of coupling layers and 1 × 1 convolutions.

EQUATION

where L < D -d.

Then the log likelihood is represented as

EQUATION

where J kl is the Jacobian of the k-th function of the l-th PIE. The approximation error here depends only on , according to the Eq. 10. For the simplicity we will now refer to the whole model as PIE.

The building blocks of this model are PIE blocks.

3.4. RELATION TO NORMALIZING FLOWS:

If we choose the distribution p(z) in Eq. 17 as Standard Gaussian, g l (•) = 0, ∀l and 0 = 1, then the model can be viewed as Normalizing Flow with multi-scale architecture Dinh et al. ( 2016 

4. PSEUDO-INVERTIBLE ENCODER:

This section introduces the basic bijectives for the Pseudo-Invertible Encoder (PIE). We explain what each building bijective consists of and how it fits in the global architecture as shown in Fig. 4 .

4.1. ARCHITECTURE:

PIE is composed of a series of convolutional blocks followed by linear blocks, as depicted in Fig. 4 . The convolutional PIE blocks consist of series of coupling layers and 1×1 convolutions. We perform invertible downsampling of the image at the beginning of the convolutional block, by reducing the spatial resolution and increasing the number of channels, keeping the overall number of the variables the same. At the end of the convolutional PIE block, the split of variables is performed. One part of the variables is projected to the residual manifold R while others is feed to the next block. The linear PIE blocks are constructed in the same manner. However, the downsampling is not performed and 1 × 1 convolutions are replaced invertible linear mappings.

x 1 × + y 1 x P U y x 2 × + y 2 s 1 b 1 s 2 b 2 (a) Forward x 1 ÷ - y 1 x P -1 U -1 y x 2 ÷ - y 2 b 1 s 1 b 2 s 2 (b) Inverse

Figure 5 : Structure of a coupling block. P partitions the input into two groups of equal length. U unites these group together. In the inverse P -1 and U -1 are the reverse of these operations respectively.

4.2. COUPLING LAYER:

In order to enhance the flexibility of the model, we utilize affine coupling layers Fig. 5 . We modify the version, introduced in Dinh et al. (2016) .

Given input data x, the output y is obtained by using the mapping:

y 1 = s 1 (x 2 ) x 1 + b 1 (x 2 ) y 2 = s 2 (y 1 ) x 2 + b 2 (y 1 ) ⇐⇒ x 2 = (y 2 -b 2 (y 1 ))/s 2 (y 1 ) x 1 = (y 1 -b 1 (x 2 ))/s 1 (x 2 ) (18)

Here multiplication and division are performed element-wise. The scalings s 1 , s 2 and the biases b 1 , b 2 are the functions, parametrized with neural networks. The invertibility is not required for this functions. x 1 , x 2 are the non-intersecting partitions of x. For convolutional blocks we partition the tensors by splitting them into halves along the channels. In case of the linear blocks, we just split the features into halves.

The log determinant of the Jacobian of coupling layer is given by:

log det ∂F θ ∂x T = sum(log |s 1 |) + sum(log |s 2 |)

where log | • | is calculated element-wise.

4.3. INVERTIBLE 1 × 1 CONVOLUTION AND LINEAR TRANSFORMATION:

The affine couplings operate on non-intersecting parts of the tensor. In order to capture the various correlations between channels and features, the different mechanism of channel permutations were proposed. Kingma & Dhariwal (2018) demonstrated that invertible 1×1 convolutions perform better than fixed permutations and reversing of the order of channels Dinh et al. (2016) .

We parametrize Invertible 1 × 1 Convolutions and invertible linear mappings with Householder Matrices Householder (1958) . Given the vector v, the Householder Matrix is computed as:

H(v) = I -2 vv T v T v (19) z x P z r δ(r -g(z)) (a) Forward z x P -1 z r = g(z) (b) Inverse

Figure 6 : Structure of the split method. P partitions the input into two sub samples. P -1 unites these sub samples together.

The obtained matrix is orthogonal. Therefore, its inverse is just its transpose, which makes the computation of the inverse easier comparing to Kingma & Dhariwal (2018) . The log determinant of the Jacobian of such transformation is equal to 0.

4.4. DOWNSAMPLING:

We use invertible downsampling to progressively reduce the spatial size of the tensor and increase the number of its channels. The downsampling with the checkerboard patterns Jacobsen et al. ( 2018); Dinh et al. (2016) transforms the tensor of size C ×H ×W into a tensor of size 4C × H 2 × W 2 , where H, W are the height and the width of the image, and C is the number of the channels. The log determinant of the Jacobian of Downsampling is 0 as it just performs permutation.

4.5. SPLIT:

All the discussed blocks transform the data while preserving its dimensionality. Here we introduce Split block Fig. 6 , which is responsible for the projection, restrictions and extension, described in Section 3. It reduces the dimensionality of the data by splitting the variables into two nonintersecting parts z, r of dimensionalities d and D -d, respectively. z is kept and is to be processed by the subsequent blocks. r is constrained to match N (r|g(z), 2 0 I). The mappings is defined as

EQUATION

5. EXPERIMENTS:



5.1. MANIFOLD LEARNING:

For this experiment we trained a Gaussian PIE on the MNIST digits dataset. We build PIE with 2 convolutional blocks, each splitting the data in the last layer to 50% of the input size. Next we add three linear blocks to PIE, reducing the dimensions to 64, 10 and the last block does not reduce the dimensions any further. For each affine transformation we use the three biggest possible Householder reflections. For this experiment we set K l equal to 3. Optimization is done with the Adam optimizer Kingma & Ba (2014) . The model diminishes the number of dimensions from R 784 to R 10 .

This experiment shows the ability of PIE to learn a manifold with three different constraints; 2 = 0.01, 2 = 0.1 and 2 = 1.0. The results are shown in Fig. 7 . As the constraint gets to loose, as shown in the right column, the model is not able to reconstruct anymore (Fig. 7a ). Lower values for 2 perform better in terms of reconstruction. Too low values, however, sample fuzzy images (Fig. 7b ). Narrowing down the distribution to sample from increases the models probability to produce accurate images. This is shown in Fig. 7c where samples are taken from N (0, 0.5). For both 2 = 0.01 and 2 = 0.1 reconstructed images are more accurate. This experiment shows that tightening the constraint by decreasing 2 increases the power of the manifold learned by the model. This is shown again in Fig. 7e where we diminished the number of dimensions even further from R 10 to R 2 utilizing UMAP (McInnes et al., 2018) . With 2 = 1.0 UMAP created a manifold with a good Gaussian distribution. However, from the manifold created by PIE it was not able to separate distinct digits from each other. Tightening the constraint with a lower 2 moves the manifold created by UMAP further away from a Gaussian distribution, while it is better able to separate classes from each other.

5.2. IMAGE SHARPNESS:

It is a well-known problem in VAEs that generated images are smoothened. WAE Tolstikhin et al. (2018) improves over VAEs by utilizing Wasserstein distance function. To test the sharpness of generated images we convolve the grey-scaled images with the Laplace filter. This filter acts as an edge detector. We compute the variance of the activations and average them over 10000 sampled images. If an image is blurry, it means there are less edges and thus more activations will be close to zero, leading to a smaller variance. In this experiment we compare the sharpness of images generated by PIE with WAE, VAE and the sharpeness of the original images. For VAE and WAE we take the architecture as described in Radford et al. (2015) . For PIE we take the architecture as described in section 5.1.

Table 1 shows the results for this experiment. PIE outperforms both VAE and WAE in terms of sharpeness of generated images. Images generated by PIE are even more sharp then original images from the MNIST dataset. An explanation for this is the use of a checkerboard pattern in the downsampling layer of the PIE convolutional block. With this technique we capture intrinsic properties of the data and are thus able to reconstruct sharper images.

:

ACKNOWLEDGEMENTSOne of the authors is sponsored by STW project "Imagine: in search for the unknown".

Figures and Tables:

Figure 1: Schematic representation of the proposed mechanism of dimensionality reduction.

Figure 2: The schematic representation of the Restriction-Extension approach. The invertible mapping X ↔ Z is preformed by using the dependent sub-manifold R = g(Z) and a pair extended functions G, G-1 .

Figure 3: Schematic representation of three types of bijective mappings currently used in normalizing flows. The circles represent the variables. The basic invertible mappings are depicted with blue edges. Green edges represent the aggregation of the variables in objective function. In general normalizing flow (a) all the variables are mapped in the same manner and are propagated through the same number of flows. The multi-scale architecture used in RealNVP (b) transform different variables with different number of flows and afterwards map them to the same distribution. Our model (c) progressively discards part of the variables by hardly constraining their distributions.

) Fig. 3. It was demonstrated in Dinh et al. (2016) that the model with such architecture achieves semantic compression.

Figure 7: Experiment on MNIST dataset with dim(z) = 10. (a) shows reconstructions on test data. Row 1 and 3 are original images, row 2 and 4 are reconstructions from z-space. (b) and (c) both show reconstruction of a samples z-space. (b) is sampled from N (0, 1), (c) is sampled from N (0, 0.5). (d) is a linear interpolation between a picture on the left and the right of the image. All digits shown are reconstructed from z-space. At last (e) shows UMAP from dim(z) = 10.

Fig.7dshows for each model the linear interpolation from one latent space to another. Both lower values of 2 (0.01, 0.1) show digits that are quite accurate. When the constraint is loosened to = 1.0 the interpolation is unable to show distinct values.

Results for experiment on sharpness on three different models and original images. For all three models a sample of 8 dimensions was taken. The generated images where convolved with Laplace filter and then the variance of activations was averaged over 10000 samples images. Higher values are better.

Table content: <table><tr><td/><td>Sharpness</td></tr><tr><td>True</td><td>0.18</td></tr><tr><td>VAE</td><td>0.08</td></tr><tr><td>WAE</td><td>0.07</td></tr><tr><td>PIE</td><td>0.49</td></tr></table>

Peer review (iclr2019):
 PIE extend NICE and Real NVP into situations which require having a smaller dimensionality of the latent variable (d) compared to the dimensionality of the observed variable (D), i.e. d < D. This is done by learning an extension function g(z) from R^d to R^{D-d} and then using the change of variables formula on x and [z, g(z)]. To model probabilistically the deterministic function g(z) is replaced by Normal distribution with mean g(z) and a small variance. PIE is used to build deep generative models and trained on the MNIST dataset. The authors show that the models learnt via PIE produce sharper samples than VAEs and Wasserstein autoencoders (WAEs). No comparison to real NVP is made, which should be the main baseline of comparison to answer the question of "what is the advantage of having d < D?". Further MNIST is no longer a good enough benchmark to evaluate deep generative models. Most representative work in this literature use CIFAR-10, downsampled Imagenet, or Imagenet at 256x256. This work falls short of the standards of ICLR in a few ways: 1. The presentation is unclear. The explanation of the extension-restriction idea is overly complicated. Further, the paper does very little to properly contextualize this work in the literature. Real NVP and flow-based models are mentioned but the proposed technique is not compared to it. The authors say they "introduce new class of likelihood-based Auto-Encoders", but this is false as far as I understood. The technique is not even an autoencoder since a separate decoder is not trained, and is obtained by exactly inverting the encoder as in real NVP. 2. The experiments are weak. The samples shown are of poor quality, and on a very simplisitic dataset (MNIST). The authors compare with vanilla VAEs, but ignore more recent improvements to VAE such as VAE-IAF, flow-based models, and also autoregressive models. A heuristic is used to measure sharpness and only used to compare against VAE and WAE. Since all these models allow likelihood evaluation, likelihoods should also have been compared. 3. The technique itself is a small change over real NVP and it*s not clear whether this change brings any improvements or provides any insights about generative modeling.