Year: 2019

Abstract:
Deep autoregressive models are one of the most powerful models that exist today which achieve state-of-the-art bits per dim. However, they lie at a strict disadvantage when it comes to controlled sample generation compared to latent variable models. Latent variable models such as VAEs and normalizing flows allow meaningful semantic manipulations in latent space, which autoregressive models do not have. In this paper, we propose using Fisher scores as a method to extract embeddings from an autoregressive model to use for interpolation and show that our method provides more meaningful sample manipulation compared to alternate embeddings such as network activations.

heading: 1 INTRODUCTION
text: Over the last few decades, unsupervised learning has been a rapidly growing field, with the development of more complex and better probabilistic density models. Autoregressive generative models (Salimans et al., 2017; Oord et al., 2016a; Menick & Kalchbrenner, 2018) are one of the most powerful generative models to date, as they generally achieve the best bits per dim compared to other likelihood-based models such as Normalizing Flows or Variational Auto-encoders (VAEs) (Kingma & Welling, 2013; Dinh et al., 2016; Kingma & Dhariwal, 2018). However, it remains a difficult problem to perform any kind of controlled sample generation using autoregressive models. For example, flow models and VAEs are structured as latent variable models and allow meaningful manipulations in latent space, which can then be mapped back to original data distribution to produce generated samples either through an invertible map or a learned decoder.
In the context of natural image modelling, since discrete autoregressive models do not have continuous latent spaces, there is no natural method to apply controlled generation. When a latent space is not used, prior works generally perform controlled sample generation through training models conditioned on auxiliary information, such as class labels or facial attributes (Van den Oord et al., 2016). However, this requires a new conditional model to be trained for every new set of labels or features we want to manipulate, which is a time-consuming and tedious task. Ideally, we could structure an unconditional latent space that the autoregressive model could sample from, but where would this latent space come from?
In this paper, we propose a method of image interpolation and manipulation in a latent space defined by the Fisher score of both discrete and continuous autoregressive models. We use PixelCNNs to model the natural image distribution and use them to compute Fisher scores. In order to map
back from Fisher score space to natural images, we train a decoder by minimizing reconstruction error. We show that interpolations in Fisher score space provide higher-level semantic meaning compared to baselines such as interpolations in PixelCNN activation space, which produce blurry and incoherent intermediate interpolations similar in nature to interpolations using pixel values. In order to evaluate interpolations quantitatively, for different mixing coefficients α, we calculate FID (Heusel et al., 2017) of the images decoded from a large sample of convex combinations of latent vectors.
In summary, we present two key contributions in our paper:
• A novel method for natural image interpolation and semantic manipulation using autoregressive models through Fisher scores
• A new quantitative approach to evaluate interpolation quality of images

heading: 2 RELATED WORK
text: There exists a substantial amount of work on natural image manipulation using deep generative models. VAEs (Higgins et al., 2017), BigBiGANS (Donahue et al., 2016; Donahue & Simonyan, 2019; Brock et al., 2018) and normalizing flows Kingma & Dhariwal (2018) provide learned latent spaces in which realistic image manipulation is a natural task. Interpolating through a latent space allows more natural transitions between images compared to pixel value interpolations, which naively overlay images on top of each other. Other prior methods learn hierarchical latent spaces on GANs and VAEs to encourage semantic manipulations to disentangle different levels of global characteristics of facial features, such as skin color, gender, hair color, and facial features (Karras et al., 2019).
Aside from using a latent space for controlled image generation, prior methods have also trained generative methods conditioned on relevant auxiliary feature labels, such as class labels in ImageNet. Other prior works have also used facial feature embeddings and binary facial attributes from CelebA to manipulate facial characteristics of generated images (Van den Oord et al., 2016).
Similarly, there has been a large amount of work on using Fisher information in machine learning. Many prior methods use the Fisher Kernel in algorithms such as kernel discriminant analysis and kernel support vector machines (Mika et al., 1999; Jaakkola & Haussler, 1999). More recent works introduce the Fisher vector, which uses Fisher scores of mixtures of Gaussians as feature vectors for downstream tasks (Sánchez et al., 2013; Simonyan et al., 2013; Perronnin et al., 2010b; Gosselin et al., 2014; Perronnin et al., 2010a). However, to our knowledge, there has been no work in using Fisher scores for deep generative modelling.

heading: 3 BACKGROUND

heading: 3.1 PIXELCNN
text: PixelCNNs (Oord et al., 2016b) are powerful autoregressive models used to model complex image distributions. They are likelihood-based generative models that directly model the data distribution p(x). This allows us to exactly compute the Fisher score instead of using an approximation, which would be necessary for GANs (Goodfellow et al., 2014) or VAEs, as they either implicitly optimize likelihood, or optimize a variational lower bound.
PixelCNNs use a series of masked convolutions to define an autoregressive model over image data. Masked convolutions allow PixelCNNs to retain the autoregressive ordering when propagating values through layers. Over the past few years, many improved variants of PixelCNNs have been developed to address certain problems with the original PixelCNN design. Specifically for our work, we use Gated PixelCNNs (Van den Oord et al., 2016), which introduced horizontal and vertical convolutional blocks to remove the blind-spot in the original masked convolutions of PixelCNNs, and Pyramid PixelCNNs (Kolesnikov & Lampert, 2017), which use PixelCNN++ (Salimans et al., 2017) architectures in a hierarchical fashion, with each layer of the hierarchy modelling images at different down-sampled resolutions conditioned on the previous layer.

heading: 3.2 FISHER SCORE
text: The Fisher score ˙̀(x; θ) = ∇θ log pθ(x) is defined as the gradient of the log-probability of a sample with respect to the model parameters θ. Intuitively, Fisher scores describe the contributions of each parameter during the generation process. Similar samples in the data distribution should elicit similar Fisher scores. Similarity between samples can also be evaluated by taking a gradient step for one sample, and checking if the log-likelihood of another sample also increases. This intuition is re-affirmed in the underlying mathematical interpretation of Fisher information - the Fisher score maps data points onto a Riemannian manifold with a local metric given by the Fisher information matrix. The underlying kernel of this space is the Fisher kernel, defined as:
˙̀(xi; θ) TF−1 ˙̀(xj ; θ) = (F − 12 ˙̀(xi; θ)) T (F− 1 2 ˙̀(xj ; θ))
where F−1 is the inverse Fisher information matrix, and F− 1 2 is the Cholesky Decomposition. Applying F− 1 2 is a normalization process, so the Fisher kernel can be approximated by taking the dot product of standardized Fisher scores. This is useful since computing F−1 is normally difficult. As such, we can see the collection of Fisher scores as a high dimensional embedding space in which more meaningful information about the data distribution can be extracted compared to raw pixel values. More complex deep generative models may learn parameters that encode information at high-levels of abstraction which may be reflected as high-level features in Fisher scores.

heading: 3.3 SPARSE RANDOM PROJECTIONS
text: It would be cumbersome to work in the very high-dimensional parameter spaces of deep generative models, so we use dimensionality reduction methods to make our methods more scalable. Sparse random projections allow for memory-efficient and scalable projections of high dimensional vectors. The Johnson-Lindenstrauss Lemma (Dasgupta & Gupta, 1999) states that under a suitable orthogonal projection, a set of n points in a d-dimensional space can be accurately embedded to some k-dimensional vector space, where k depends only on log n. Therefore, for suitably large k, we can preserve the norms and relative distances between projected points, even for very high dimensional data. Since sparse random matrices are nearly orthogonal in high-dimensional settings, we can safely substantially reduce the dimensionality of our embedding spaces using this method.
We generate sparse random matrices according to Li et al. (2006). Given an n× k matrix to project, we define the minimum density of our sparse random matrix as d = 1√
k , and let s = 1d . Suppose
that we are projecting the data into p dimensions, then the n × p projection matrix P is generated according to the following distribution:
Pij =  − s√p with probability 1 2s
0 with probability 1− 1s s√ p with probability 1 2s
(1)
where Pij is the element of the ith row and jth column of the projection matrix.
Algorithm 1: The procedural generation of the new interpolated embedding dataset for quantitative evaluation. Following FID conventions, we use a sample size of 50000 images. Result: α-interpolated dataset Dα Input: The dataset D, projection matrix P , pre-trained autoregressive model pθ(x), and the learned decoder Dec Dα← {} for i← 0 to 50000 do
Sample x1, x2 ∼ D, a random pair of samples z1 ← P∇θ log pθ(x1) z2 ← P∇θ log pθ(x2) ẑ ← (1− α)z1 + αz2 x̂← Dec(ẑ) Dα ← Dα ∪ {x̂}
end

heading: 4 METHOD
text: We now describe our approach for natural image manipulation and interpolation using autoregressive models. Note that this method is not restricted to autoregressive models and can be applied to any likelihood-based models.

heading: 4.1 EMBEDDING SPACE CONSTRUCTION
text: Given a trained autoregressive model, pθ(x), we want to construct an embedding space that is more meaningful than raw pixel values. In this paper, our autoregressive models are exclusively variants of PixelCNNs since we are working in an image domain. Drawing inspiration off popular selfsupervised methods (Zhang et al., 2016; Gidaris et al., 2018; Doersch et al., 2015; Pathak et al., 2016; Noroozi & Favaro, 2016), it may seem natural to take the output activations of one of the last few convolutional layers. However in our experiments, we show that using output activations provides less meaningful embedding manipulation than using the Fisher score. This is especially the case for PixelCNNs, as PixelCNNs are known for generally encoding more local statistics than global features of images. However, Fisher scores lie in parameter space, which may encode more global semantics.
In order to construct the embedding space using Fisher scores, we initialize a sparse random matrix P , randomly generated following the distribution in equation 1. In practice, we found sparse random matrices the only feasible method for reasonable dimensionality reduction when scaling to more difficult datasets, where the sizes of corresponding PixelCNNs grew to tens of millions of parameters. Finally, the embedding space is constructed as follows: each element zi in the new embedding space is computed as zi = P∇θ log pθ(xi) for each sample xi in the dataset.

heading: 4.2 LEARNING A DECODER
text: Regardless of whether we use Fisher scores or network activations as embedding spaces, doing any sort of image manipulation in a generated embedding space requires a mapping back from zi to xi. To solve this problem, we learn a mapping back from zi to xi by training a network to model the density p(xi|zi). We try both supervised and unsupervised learning approaches, either directly learning a decoding model to minimize reconstruction error, or implicitly learn reconstruction by training a conditional generative model, such as another autoregressive model or a flow.

heading: 4.3 INTERPOLATION EVAUATION METRIC
text: We introduce an evaluation procedure to quantitatively evaluate image interpolation quality. The quality of image interpolations can roughly be measured by how realistic any intermediate interpolated image is. Existing popular methods to measure image quality can largely be attributed GAN metrics, such as Fréchet Inception Distance (FID) (Heusel et al., 2017) or Inception Score (IS) (Salimans et al., 2016). We choose to use FID as our evaluation metric since it is more generalizable to other datasets than IS. Interpolation quality is evaluated by computing FID for different mixing coefficients α. For α ∈ {0, 0.125, . . . , 0.5}, we generate a new dataset Dα according to Algorithm 1, and compute the FID between the true dataset and Dα. Under this evaluation metric, good interpolations result in lower FID scores, and bad interpolations produce peaked FID scores at α = 0.5.

heading: 5 EXPERIMENTS
text: We evaluate our method on MNIST and CelebA and design experiments to answer the following questions:
• How does image interpolation quality using Fisher scores compare to baseline embedding spaces such as PixelCNN activations?
• Do Fisher scores contain high-level semantic information about the original image data?

heading: 5.1 MNIST
text: Setup We trained a standard PixelCNN with masked convolutions on binarized MNIST images as our autoregressive model. Images were binarized by sampling from Bernoulli random variables biased by grayscale pixel values. The decoder model consisted of a dense layer following by 3 transposed convolutional layers that upscale the image to 28× 28.
Projection Method We primarily experimented with varying levels of density and projection dimensions for dense and sparse random projections. Figure 3 (a) shows reconstruction error on trained decoders for different hyperparameter combinations. For MNIST, we chose to use a dense matrix with a projection dimension of 1024.
Results Figure 2 shows a comparison between using the PixelCNN activations of the second-tolast convolutional layer and projected Fisher scores as embedding spaces. In general, reconstruction using the activations is slightly more accurate than using Fisher scores. However, for interpolations, using activations produces intermediate images that are similar to interpolations using raw pixel values. On the other hand, interpolations with our method using Fisher scores produces a more natural transition between samples. These qualitative observations are also reflected in our quantitative measurements shown in Figure 2. The interpolations using activations reaches a much higher FID (25.42) than using interpolation using Fisher scores (5.43). Note that we only show α from 0 to 0.5, as FID scores for α = 0.75 are the same as α = 0.25 since mixing coefficients are symmetric.

heading: 5.2 CELEBA
text: Dataset We evaluate our method 128x128 CelebA images. We follow the standard procedure of cropping raw 218 x 178 CelebA images into 128 x 128 CelebA images. All autoregressive models and decoders are trained to model 5-bit images, since this bit reduction results in faster learning with a negligible cost in visible image quality.
Autoregressive Architecture In order to investigate the effect of model complexity on Fisher score representations, we experimented with three different variations of PixelCNN architectures: a standard PixelCNN with 5 layers (same as MNIST), a Gated PixelCNN, and a Pyramid PixelCNN, each with about 1 million, 5 million, and 20 million parameters respectively. We trained all models with batch size 128 and learning rate 1e− 3 using an Adam optimizer for 50 epochs. More architecture details can be found in the appendix.
Decoder Architecture We experimented with different kinds of decoder architectures. We tested a convolutional decoder architecture with added residual blocks, and also various conditional generative models, such as conditional RealNVPs, Pyramid PixelCNNs, and WGANs (Gulrajani et al., 2017) which helped produce less noisy reconstructions. Out of all decoder architectures, we observed that the conditional RealNVP produced the best qualitative results. See the appendix for more samples of each decoder architecture.
Projection Method We use sparse random matrices to project the Fisher Scores with the default density 1√nfeatures , and projection dimension of 16384. We then apply PCA to reduce the dimensionality of the Fisher scores to 4096. We found that using PCA has minimal degradation on reconstruction quality, and significantly reduced the number of parameters in our decoder models. Using this process showed better quality reconstruction compared to directly applying a random projection to 4096 dimensions, which suggests that the random projection is the primary bottleneck in this method.
Reconstruction and Interpolation Figure 4 shows a comparison between reconstruction and interpolation for PixelCNN activations versus Fisher scores. We note that since we are using
a decoder to reconstruct images, interpolations do not begin and end with the true images of the dataset, and instead, are approximations constructed from the decoder. We see similar results an in our MNIST experiments. Reconstruction quality for activations is better than with Fisher scores. However, interpolations for Fisher scores is qualitatively more natural than those of activations. Figure 6 shows that similar interpolation characteristics were observed across different decoder architectures. Therefore, it is unlikely that the good interpolations arise solely from the stronger decoder networks, and instead meaningful semantic information is stored in the Fisher scores
themselves. These observations are supported by the quantitative results shown in Figure 7. For the RealNVP decoder, we can see that although using network activations began with a lower FID (better reconstruction), increasing levels of interpolation result in FID rising faster to a higher peak than the same interpolation level for Fisher scores.
Looking at the effect of model complexity, we see that there is not too much of a difference in reconstruction and interpolation quality, but generally, less complex models allow slightly more accurate image reconstructions. Good interpolations for even the simpler models suggest that they are still learning a substantial amount of information about the data including high level semantic information.
Semantic Manipulation Since CelebA has binary labels, we can also look at semantic manipulation in addition to interpolations. Given a binary attribute, such as the presence of black hair, a smile, or eyeglasses, we can extrapolate in embedding space in the same manner described in Glow (Kingma & Dhariwal, 2018). For some attribute, let zpos be the average of all embedding vectors with the attribute, and zneg be the average of all embedding vectors without the attribute. We can then apply or remove the attribute by manipulating a given embedding vector in the direction of δ = zpos − zneg or its negation. For our experiments, we found that scaling δ by a factor of 3 was enough to see visible change in our images. See Figure 5 for example of applying different attributes using both the activation and Fisher score embedding spaces. Overall, our method of using Fisher scores tends to encourage more natural semantic manipulations in adding a smile, or adding eyeglasses compared to activation embeddings, which tend to ”paste” generic smiles and glasses on top of each face.

heading: 6 CONCLUSION
text: We proposed a new method to allow image interpolation and manipulation using autoregressive models. Our approach used Fisher scores an as underlying embedding space, and showed more natural interpolation compared to our baselines using activations of PixelCNNs or raw pixel interpolations. In addition, we introduced a new evaluation metric for interpolations by using FID to measure images at different levels of interpolation. Lastly, we note that this method is not restricted to images, and generalizes to any autoregressive model on any kind of dataset.

heading: 7 APPENDIX

heading: 7.1 PIXELCNN ARCHITECTURES

heading: 7.1.1 MNIST
text: The PixelCNN architecture for MNIST was a standard PixeCNN Oord et al. (2016b), with 5 masked convolutional layers each kernel size 7, padding 3, and 64 filters. We used ReLU for our activation function.

heading: 7.1.2 CELEBA
text: 1-layer PixelCNN Architecture is the same as the PixelCNN from MNIST, but with only 1 masked convolutional layer.
5-layer PixelCNN Architecture is the same as the PixelCNN from MNIST
Gated PixelCNN The Gated PixelCNN architecture is the same as described in Van den Oord et al. (2016). We use a filter size of 120 dimensions, with 5 masked convolutional layers of kernel size 7 and padding 2. Each masked convolutional layer uses vertical and horizontal stack of convolutions, with residual connections and gating mechanisms.
Pyramid PixelCNN The Pyramid PixelCNN has 3 layers. The bottom layer is a PixelCNN++ on 8 × 8 down-sampled image. The second layer is a conditional PixelCNN++ that generates 32× 32 images conditioned on 8× 8 images. The final layer generates 128× 128 images conditioned on 32× 32 images.

heading: 7.2 DECODER ARCHITECTURES

heading: 7.2.1 MNIST
text: ResBlock Architecture
Conv2d (channels, k = 1, no bias) BatchNorm, ReLU
Conv2d (channels, k = 3, padding 1, no bias) BatchNorm, ReLU
Conv2d (channels, k = 1, no bias) BatchNorm, ReLU
Shortcut Connection, ReLU
Decoder Architecture
x ∈ Rn Linear (1024), ReLU
ConvTranspose (128 filters, k = 5, stride 2), ReLU ResBlock(128)
ConvTranspose (32 filters, k = 5, stride 2) BatchNorm, ReLU
ConvTranspose (2 filter, k = 5, stride 2)

heading: 7.2.2 CELEBA
text: Convolutional Model
x ∈ R1×1×4096 ConvTranspose (256 filters, k = 4)
4x ResBlock, Upsample 2x BatchNorm, ReLU
Conv2d (32 filters, k = 1)
WGAN Follows the same architecture described in https://github.com/LynnHo/ DCGAN-LSGAN-WGAN-GP-DRAGAN-Pytorch. To make it conditional, we concatenate the conditioning vector with the noise when generating. For the discriminator, we project the conditioning vector to the channel dimension of each ResBlock input, and add it broadcasted across the image as a bias.
RealNVP We use the Multiscale RealNVP architecture described in Dinh et al. (2016). We apply conditioning as follows: for each ResNet in the affine coupling layers, we project the conditioning vector to the channel dimension size, and add it as a bias to the ResNet input.

heading: 7.3 RECONSTRUCTIONS FOR DIFFERENT DECODERS

heading: 7.4 INTERPOLATIONS FOR DIFFERENT DECODERS
text: (a) Autoregressive model: Gated PixelCNN, Embedding Space: Activations, Decoder: Convolutional Network
(b) Autoregressive model: Gated PixelCNN, Embedding Space: Fisher Scores, Decoder: Convolutional Network
(a) Autoregressive model: Gated PixelCNN, Embedding Space: Activations, Decoder: RealNVP
(b) Autoregressive model: Gated PixelCNN, Embedding Space: Fisher Scores, Decoder: Conditional RealNVP
(a) Autoregressive model: Gated PixelCNN, Embedding Space: Fisher Scores, Decoder: Conditional WGAN
(b) Autoregressive model: Gated PixelCNN, Embedding Space: Fisher Scores, Decoder: Conditional Pyramid PixelCNN

heading: 7.5 FACIAL ATTRIBUTE MANIPULATION FOR DIFFERENT DECODERS
text: (a) Autoregressive model: Gated PixelCNN, Embedding Space: Activations, Decoder: Conditional RealNVP (b) Autoregressive model: Gated PixelCNN, Embedding Space: Activations, Decoder: Conditional RealNVP
(c) Autoregressive model: Gated PixelCNN, Embedding Space: Activations, Decoder: Conditional RealNVP (d) Autoregressive model: Gated PixelCNN, Embedding Space: Activations, Decoder: Conditional RealNVP
(a) Autoregressive model: Gated PixelCNN, Embedding Space: Fisher Score, Decoder: Conditional RealNVP (b) Autoregressive model: Gated PixelCNN, Embedding Space: Fisher Score, Decoder: Conditional RealNVP
(c) Autoregressive model: Gated PixelCNN, Embedding Space: Fisher Score, Decoder: Conditional RealNVP (d) Autoregressive model: Gated PixelCNN, Embedding Space: Fisher Score, Decoder: Conditional RealNVP

Review summaries:
Motivated by the observation that powerful deep autoregressive models such as PixelCNNs lack the ability to produce semantically meaningful latent embeddings and generate visually appealing interpolated images by latent representation manipulations, this paper proposes using Fisher scores projected to a reasonably low-dimensional space as latent embeddings for image manipulations. A decoder based on a CNN, a Conditional RealNVP, or a Conditional Pyramid PixelCNN is used to decode high-dimensional images from these projected Fisher score.  Experiments with different autoregressive and decoder architectures are conducted on MNIST and CelebA datasets are conducted. 

This paper focuses on the problem of interpolating between data points using neural autoregressive models. The core idea is that it is possible to use (a smaller-dimensional projection of) the Fisher score of the density function defined by the autoregressive model to represent data points in embedding space, and a neural decoder for mapping them back to input space. Experiments on both MNIST and Celeb suggest that this is a sensible method, and leads to smoother interpolations rather than just relying on the embeddings resulting from the network activations.