Natural Image Manipulation for Autoregressive Models Using Fisher Scores
Authors: Wilson Yan
Jonathan Ho
Pieter Abbeel
Venue:  (2020)
Keywords: fisher score
generative models
image interpolation

Abstract: Deep autoregressive models are one of the most powerful models that exist today which achieve state-of-the-art bits per dim. However, they lie at a strict disadvantage when it comes to controlled sample generation compared to latent variable models. Latent variable models such as VAEs and normalizing flows allow meaningful semantic manipulations in latent space, which autoregressive models do not have. In this paper, we propose using Fisher scores as a method to extract embeddings from an autoregressive model to use for interpolation and show that our method provides more meaningful sample manipulation compared to alternate embeddings such as network activations.

Under review as a conference paper at ICLR 2020
NATURAL IMAGE MANIPULATION FOR AUTOREGRES- SIVE MODELS USING FISHER SCORES
Anonymous authors Paper under double-blind review
ABSTRACT
Deep autoregressive models are one of the most powerful models that exist today which achieve fatt--of-theaar bits per dim. However, they lie at strict disadvan- tage when comes to controlled sample generation compared to latent variable models. Latent variable models such VAEs and normalizing flows allow mean- ingful semantic manipulations in latent space, which autoregressive models do not have. In this paper, propose using Fisher scores method extract embeddings from an autoregressive model to use for interpolation and show that our method provides more meaningful sample manipulation compared to alternate embeddings such as network activations.
INTRODUCTION
Over the last few decades, unsupervised learning has been a rapidly growing field, with the devel- opment of more complex and better probabilistic density models. Autoregressive generative mod- els (Salimans et al., 2017; Oord et al., 2016a; Menick & Kalchbrenner, 2018) are one of the most powerful generative models to date, as they generally achieve the best bits per dim compared to other likelihood based models such as Normalizing Flows or Variational Auto-encoders (VAEs) (Kingma & Welling, 2013; Dinh al., 2016; Kingma & Dhariwal, 2018). However, remains difficult problem to perform any kind of controlled sample generation using autoregressive models. For example, flow models and VAEs are structured as latent variable models and allow meaningful ma- nipulations in latent space, which can then be mapped back to original data distribution to produce generated samples either through an invertible map or learned decoder.
In the context of natural image modelling, since discrete autoregressive models do not have contin- uous latent spaces, there is no natural method to apply controlled generation. When latent space is not used, prior works generally perform controlled sample generation through training models conditioned on auxiliary information, such as class labels or facial attributes (Van den Oord al., 2016). However, this requires a new conditional model to be trained for every new set of labels or features we want to manipulate, which a time-consuming and tedious task. Idcally, we could structure an unconditional latent space that the autoregressive model could sample from, but where would this latent space come from?
In this paper, propose method of image interpolation and manipulation in latent space defined by the Fisher score of both discrete and continuous autoregressive models. We use PixeICNNs model the natural image distribution and use them to compute Fisher scores. In order map
Figure I: CelebA interpolation comparing PixcICNN activations (top row) versus Fisher scores (bottom row) embedding spaces for autoregressive models

---

Under review as a conference paper at ICLR 2020
back from Fisher score space to natural images, train a decoder by minimizing reconstruction error. We show that interpolations in Fisher score space provide higher-level semantic meaning compared to baselines such as interpolations in PixeICNN activation space, which produce blurry and incoherent intermediate interpolations similar in nature to interpolations using pixel values. In order evaluate interpolations quantitatively, for different mixing coefficients a, we calculate FID (Heusel et al., 2017) of the images decoded from a large sample of convex combinations of latent vectors.
In summary, we present two key contributions in our paper:
A novel method for natural image interpolation and semantic manipulation using autore- gressive models through Fisher scores
A new quantitative approach to evaluate interpolation quality of images
2 RELATED WORK
There exists a substantial amount of work on natural image manipulation using deep generative models. VAEs (Higgins et al., 2017), BigBiGANS (Donahue et al., 2016; Donahue & Simonyan, 2019; Brock et al., 2018) and normalizing flows Kingma & Dhariwal (2018) provide learned la- tent spaces in which realistic image manipulation is natural task. Interpolating through latent space allows more natural transitions between images compared to pixel value interpolations. which naively overlay images on top of each other. Other prior methods learn hierarchical latent spaces on GANs and VAEs to encourage semantic manipulations to disentangle different levels of global characteristics of facial features, such as skin color, gender, hair color, and facial features (Karras et al., 2019).
Aside from using a latent space for controlled image generation, prior methods have also trained generative methods conditioned on relevant auxiliary feature labels, such as class labels in ImageNet. Other prior works have also used facial feature embeddings and binary facial attributes from CelebA to manipulate facial characteristics of generated images (Van den Oord et al., 2016).
Similarly, there has been a large amount of work on using Fisher information in machine learning. Many prior methods use the Fisher Kernel in algorithms such as kernel discriminant analysis and kernel support vector machines (Mika et al., 1999; Jaakkola & Haussler, 1999). More recent works introduce the Fisher vector, which uses Fisher scores of mixtures of Gaussians as feature vectors for downstream tasks (Sánchez et al., 2013; Simonyan et al., 2013; Perronnin et al., 2010b; Gosselin al., 2014; Perronnin al., 2010a). However, to knowledge, there has been no work in using Fisher scores for deep generative modelling.
3 BACKGROUND
3.1 PIXELCCNN
PixeICNNs (Oord et al., 2016b) are powerful autoregressive models used model complex image distributions. They are likelihood-based generative models that directly model the data distribution p(x). This allows to exactly compute the Fisher score instead of using an approximation, which would be necessary for GANs (Goodfellow et al., 2014) or VAEs, as they either implicitly optimize likelihood, or optimize variational lower bound.
PixeICNNs use a series of masked convolutions to define an autoregressive model over image data. Masked convolutions allow PixeICNNs to retain the autoregressive ordering when propagating val- ues through layers. Over the past few years, many improved variants of PixeICNNs have been de- veloped to address certain problems with the original PixeICNN design. Specifically for our work, we use Gated PixelCNNs (Van den Oord et al., 2016), which introduced horizontal and vertical convolutional blocks to remove the blind-spot in the original masked convolutions of PixeICNNs, and Pyramid PixeICNNs (Kolesnikov & Lampert, 2017), which use PixeICNN+ (Salimans et al., 2017) architectures a hierarchical fashion, with each layer of the hierarchy modelling images at different down-sampled resolutions conditioned the previous layer.
2

---

Under review as a conference paper at ICLR 2020
3.2 FISHER SCORE
The Fisher score f(2; 0) V. log (o( is defined as the gradient of the log- prg-pbbbbbllity of sample with respect to the model parameters Intuitively, Fisher scores describe the contributions of each parameter during the generation process. Similar samples in the data distribution should elicit similar Fisher scores. Similarity between samples can also be evaluated by taking a gradient step for sample, and checking if the log-likelihood of another sample also increases. This intuition re-affirmed in the underlying mathematical interpretation of Fisher information the Fisher score maps data points onto Riemannian manifold with a local metric given by the Fisher information matrix. The underlying kernel this space is the Fisher kernel, defined as:
(x;00)² ((xx(=)))))))))))))) (F- ( (F=
where F-1 is the inverse Fisher information matrix, and F- is the Cholesky Decomposition. Applying F= is a normalization process, so the Fisher kernel can be approximated by taking the dot product of standardized Fisher scores. This is useful since computing |"-1 is normally difficult. As such, we can see the collection of Fisher scores as a high dimensional embedding space in which more meaningful information about the data distribution can be extracted compared raw pixel values. More complex deep generative models may learn parameters that encode information at high-levels of abstraction which may be reflected as high-level features in Fisher scores.
3.3 SPARSE RANDOM PROJECTIONS
It would be cumbersome to work in the very high-dimensional parameter spaces of deep generative models, so we use dimensionality reduction methods to make our methods more scalable. Sparse random projections allow for memory-efficiett and scalable projections of high dimensional vec- tors. The SaiinsoonLLndeestraane Lemma (Dasgupta & Gupta, 1999) states that under suitable orthogonal projection, set of points in d-dimensional space be accurately embedded to some k-dimensional vector space, where lo depends only on log n. Therefore, for suitably large k preserve the norms and relative distances between projected points, even for very high di- mensional data. Since sparse random matrices are nearly orthogonal in high-dimensional settings, safely substantially reduce the dimensionality of our embedding spaces using this method.
We generate sparse random matrices according to Li et al. (2006). Given an n X matrix to project, define the minimum density of our sparse random matrix as d + and let 1. Suppose that we are projecting the data into dimensions, then the n x p projection matrix / is generated according to the following distribution:
VE
with probability 1 2s with probability 1- with probability 1
Psj =
(I)
2s
where Pij is the element of the ith row and jth column of the projection matrix.
Algorithm 1: The procedural generation of the interpolated embedding dataset for quantitative
evaluation. Following FID conventions, we use a sample size of 50000 images.
Result: a-interpolated dataset DD Input: The dataset DD, projection matrix P, pre-trained autoregressive model po (o))), and the learned decoder Dec
Da - ( for i + 0 to 50000 do
Sample æ1, Ø2 ~ D, a random pair samples
21 <- PV, log Po
22 f- PV, log Pe
i (1 2) + 0822
à - Dec(2)
Da Da U (i}
end
3

---

Under review as a conference paper at ICLR 2020
4
METHOD
We now describe our approach for natural image manipulation and interpolation using autoregressive models. Note that this method is not restricted to autoregressive models and can be applied to any likelihood-bart based models.
4.1 EMBEDDING SPACE CONSTRUCTION
Given trained autoregressive model, ps (z), we want construct an embedding space that is more meaningful than raw pixel values. In this paper, our autoregressive models are exclusively variants of PixeICNNs since we are working in an image domain. Drawing inspiration off popular self- supervised methods (Zhang et al., 2016; Gidaris et al., 2018; Doersch et al., 2015; Pathak et al., 2016; Noroozi & Favaro, 2016), may seem natural to take the output activations of one of the last few convolutional layers. However in our experiments, we show that using output activations provides less meaningful embedding manipulation than using the Fisher score. This is especially the case for PixeICNNs, as PixcICNNs are known for generally encoding more local statistics than global features of images. However, Fisher scores lie in parameter space, which may encode more global semantics.
In order to construct the embedding space using Fisher scores, we initialize a sparse random matrix P. randomly generated following the distribution in equation 1. In practice, we found sparse random matrices the only feasible method for reasonable dimensionality reduction when scaling to more dif- ficult datasets, where the sizes of corresponding PixcICNNs grew to tens millions of parameters. Finally, the embedding space is constructed as follows: each element in the new embedding space computed as 2, PV, log po (Ix;) for each sample z; the dataset.
4.2 LEARNING A DECODER
Regardless of whether we use Fisher scores or network activations as embedding spaces. doing any sort of image manipulation in a generated embedding space requires mapping back from 2; to Tz. To solve this problem, we learn mapping back from 2, to x; by training a network to model the density p(x))=<) We try both supervised and unsupervised learning approaches, either directly learning a decoding model to minimize reconstruction error, or implicitly learn reconstruction by training a conditional generative model, such another autoregressive model or flow.
4.3 INTERPOLATION EVAUATION METRIC
We introduce evaluation procedure to quantitatively evaluate image interpolation quality. The quality of image interpolations can roughly be measured by how realistic any intermediate interpo- lated image is. Existing popular methods to measure image quality can largely be attributed GAN metrics, such as Fréchet Inception Distance (FID) (Heusel et al., 2017) or Inception Score (IS) (Sal- evaluation metric since it more generalizable to
imans et al., 2016). We choose other datasets than IS. Interpolation quality is evaluated by computing FID for different mixing co- efficients For {0, 0.15, ... 0.5), we generate new dataset D, according Algorithm 1, and compute the FID between the true dataset and D... Under this evaluation metric, good interpo- lations result lower FID scores, and bad interpolations produce peaked FID scores at a 0.5.
FID
5 EXPERIMENTS
We evaluate our method on MNIST and CelebA and design experiments answer the following questions:
How does image interpolation quality using Fisher scores compare baseline embedding spaces such as PixeICNN activations?
Do Fisher scores contain high-level semantic information about the original image data?
4

---

Under review as a conference paper at ICLR 2020
9
9
3
3
5 7
5 4
4
9
3
3
5 5
4
4 6
7
7
4
4
9 6
6
7
7
4
9
9
8
8 8 8 7 7 4 4
8
8
7
4
(a) Activations (Reconstruction)
(b) Fisher score (Reconstruction)
9
7
7 7 7 4 I 8
7
9 9
9
7
7
7
7 7 7
7
3
3
3
4 I o 6
4 I 9
4
3
3
3
3
7
7
7
4
4
5
5
5
5
5
/
4
4
4
4
4
4
a
5 9
7
7
7
7
7
7
4
4
4
4
3
3
3
3
4
4
4
4
4
4
3
3
3
7
7
5 3
3
3 3 3
7 9
9
9
9
3
3
3
3
3
(c) Activations (Interpolation)
(d) Fisher score (Interpolation)
Figure 2: Comparison between between using network activations and Fisher scores as underlying embedding spaces for reconstruction and interpolation. (a) and (b) show reconstructions, where for each image pair, the left image is the true image, and the right image the reconstruction. For (c) and (d), each row shows an interpolation from image in the MNIST dataset to another.
0.001
0.01
0.1
1.0
64
0.641
0.638
0.678
0.633
a
0.0
0.125
0.25
0.375
0.5
256
0.653
0.456
0.170
0.155
Activations
0.68
0.60
2.69
12.84
25.42
1024
0.142
0.154
0.129
0.122
Fisher Score (ours)
1.21
1.14
1.03
2.21
5.43
(a)
(b)
Figure 3: (a) shows reconstruction error our decoder for different different combinations of density and projection dimensions for sparse matrices. The top row is density, and the left column projection dimension. (b) compares the FID between activations and Fisher scores embedding spaces for incrcasing levels of interpolation. We can see that FID values for our method are much more stable compared to the bascline
5.1 MNIST
Setup We trained standard PixelCNN with masked convolutions binarized MNIST images as our autoregressive model. Images were binarized by sampling from Bernoulli random variables biased by grayscale pixel values. The decoder model consisted of dense layer following by transposed convolutional layers that upscale the image to 28 x 28.
Projection Method We primarily experimented with varying levels density and projection dimensions for dense and sparse random projections. Figure (a) shows reconstruction error trained decoders for different hyperparameter combinations. For MNIST, we chose to use dense matrix with projection dimension of 1024.
Results Figure 2 shows comparison between using the PixeICNN activations of the second-to- last convolutional layer and projected Fisher scores as embedding spaces. In general, reconstruction using the activations slightly more accurate than using Fisher scores. However, for interpolations, using activations produces intermediate images that are similar to interpolations using raw pixel values. On the other hand, interpolations with our method using Fisher scores produces a more natural transition between samples. These qualitative observations quantitative measurements shown in Figure 2. The interpolations using activations reaches a much higher FID (25.42) than using interpolation using Fisher scores (5.43). Note that we only show from to 0.5, as FID scores for a = 0.75 are the same as a = 0.25 since mixing coefficients are symmetric.
also reflected in our

---

Under review as a conference paper at ICLR 2020
(a) Activations (Reconstruction)
(b) Fisher score (Reconstruction)
(c) Activations (Interpolation)
(d) Fisher score (Interpolation)
Figure 4: Comparison between using network activations and Fisher scores as underlying embedding spaces for reconstruction and interpolation. (a) and (b) show reconstructions, where for each image pair, the left image the true image, and right image the reconstruction. from one image in the CelebA dataset to another. The left-most and right-most each row is the true image pair from the dataset.
(c) (d), each row shows an interpolation
5.2 CELEBA
Dataset We evaluate our method 128x128 CelebA images. We follow the standard procedure of cropping raw 218 x 178 CelebA images into 128 128 CelebA images. All autoregressive models and decoders are trained to model 5-bit images, since this bit reduction results in faster learning with a negligible cost visible image quality.
Autoregressive Architecture In order to investigate the effect of model complexity Fisher score representations, we experimented with three different variations of PixeICNN architectures: a standard PixcICNN with 5 layers (same MNIST), Gated PixeICNN, and Pyramid PixcICNN, million, and 20 million parameters respectively. We trained all models with batch size 128 and learning rate le 3 using Adam optimizer for 50 epochs. More architecture details can found in the appendix.
each with about 1 million,
Decoder Architecture We experimented with different kinds of decoder architectures. We tested convolutional decoder architecture with added residual blocks, and also various conditional generative models, such as conditional RealNVPs, Pyramid PixeICNNs, and WGANs (Gulrajani al., 2017) which helped produce less noisy reconstructions. Out of all decoder architectures, we observed that the conditional RealNVP produced the best qualitative results. See the appendix for more samples of each decoder architecture.
Projection Method We use sparse random matrices to project the Fisher Scores with the default density Vieemes and projection dimension of 16384. We then apply PCA to reduce the dimensionality of the Fisher scores to 4096. We found that using PCA has minimal degradation on reconstruction quality, and significantly reduced the number of parameters in our decoder models. Using this process showed better quality reconstruction compared to directly applying a random projection to 4096 dimensions, which suggests that the random projection is the primary bottleneck in this method.
Reconstruction and Interpolation Figure shows a comparison between reconstruction and interpolation for PixeICNN activations versus Fisher scores. We note that since using

---

Under review as a conference paper at ICLR 2020
Activations (Glasses)
(b) Fisher score (Glasses)
(c) Activations (Smiling)
(d) Fisher score (Smiling)
Figure 5: We compare semantic manipulation on glasses and smiling attributes for PixeICNN activation versus Fisher (ours) embedding spaces
2
2
Figure 6: The top two rows show interpolation using Conv and RealNVP decoders respectively with Pix- cICNN activations as embedding space. The bottom four rows show interpolations using Conv, WGAN, Pyramid PixeICNN, and RealNVP decoders with Fisher scores (our method) as embedding space. We can see that interpolations using our method are consistent across different decoders, which supports our claim that high-level semantic information is indeed stored in the Fisher scores, and not due to the decoder architectures themselves
decoder to reconstruct images, interpolations do not begin and end with the true images of the dataset, and instead, are approximations constructed from the decoder. We see similar results an in our MNIST experiments. Reconstruction quality for activations is better than with Fisher scores. However, interpolations for Fisher scores qualitatively more natural than those of activations. Figure 6 shows that similar interpolation characteristics architectures. Therefore, it is unlikely that the good interpolations arise solely from the stronger decoder networks, and instead meaningful semantic information is stored in the Fisher scores
observed across different decoder
7

---

Under review as a conference paper at ICLR 2020
Pyramid PixeICNN as Autoregressive Model
Decoder
Embedding
0.0
0.125
0.25
0.375
0.5
Activation
63.23
63.38
67.19
74.98
80.10
Conv
Fisher Score (ours)
136.05
130.70
127.98
127.62
128.36
Activation
24.52
28.32
36.81
46.98
51.75
RealNVP
Fisher Score (ours)
31.16
33.85
37.30
40.75
42.38
Gated PixeICNN as Autoregressive Model
Decoder
Embedding
0.0
a 0.125
0.25
= 0.375
Qt = 0.5
Activation
56.31
56.92
61.28
77.38
101.59
Conv
Fisher Score (ours)
129.45
128.99
129.90
132.50
134.38
Activation
24.42
30.20
41.76
53.87
59.00
RealNVP
Fisher Score (ours)
33.46
36.60
41.74
48.31
52.17
PixeICNN (5 layers) as Autoregressive Model
Decoder
Embedding
0.0
a 0.125
0.25
0.375
Q 0.5
Activation
54.40
55.75
61.63
89.38
127.85
Conv
Fisher Score (ours)
132.47
131.85
132.33
133.76
134.43
Activation
22.29
26.42
34.41
42.92
46.99
RealNVP
Fisher Score (ours)
30.27
32.14
35.53
39.65
41.83
Figure 7: Interpolation evaluation results for CelebA. We show quantitative results for varying com- binations of base autoregressive models paired with different decoders. We compare using network activations versus Fisher scores (our method) as embedding spaces.
themselves. These observations supported by the quantitative results shown in Figure 7. For the RealNVP decoder, we can see that although using network activations began with lower FID (better reconstruction), increasing levels of interpolation result in FID rising faster higher peak than the same interpolation level for Fisher scores.
Looking at the effect of model complexity. we see that there is not too much of a difference in reconstruction and interpolation quality, but generally, less complex models allow slightly more accurate image reconstructions. Good interpolations for even the simpler models suggest that they are still learning substantial amount of information about the data including high level semantic information.
Semantic Manipulation Since CelebA has binary labels, we can also look at semantic ma- nipulation in addition interpolations. Given binary attribute, such the presence of black hair, a smile, or eyeglasses, we can extrapolate in embedding space in the same manner described in Glow (Kingma Dhariwal, 2018). For some attribute, let Zpos be the average of all embedding vectors with the attribute, and &neg be the average of all embedding vectors without the attribute. We can then apply or remove the attribute by manipulating given embedding vector in the direction of 8 2pos Zneg its negation. For our experiments, we found that scaling 5 by a factor of 3 was enough see visible change in our images. See Figure for example of applying different attributes using both the activation and Fisher score embedding spaces. Overall, our method of using Fisher scores tends encourage more natural semantic manipulations adding a smile, or adding eyeglasses compared to activation embeddings, which tend to "paste" generie smiles and glasses on top of each face.
CONCLUSION
We proposed new method to allow image interpolation and manipulation using autoregressive models. Our approach used Fisher scores natural interpolation compared to our baselines using activations of PixelCNNs or raw pixel inter- polations. In addition, introduced new evaluation metric for interpolations by using FID to measure images at different levels of interpolation. Lastly, we note that this method not restricted to images, and generalizes any autoregressive model any kind of dataset.
underlying embedding space, and showed more
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint a809: 1096, 2018.
Sanjoy Dasgupta and Anupam Gupta. An elementary proof of the lindenstrauss lemma. International Computer Science Institute, Technical Report, 22(1):1-5, 1999.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint /605. 08803, 2016.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp- 1422-1430, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. arXiv preprint arXiv:1907.02544, 2019.
Jeff Donahue, Philipp Krähenbühh, and Trevor Darrell. Adversarial feature learning. arXiv preprint 165.v91605.09782, 2016.
Spyros Gidaris. Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.0778, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Philippe-Henri Gosselin, Naila Murray, Hervé Jégou, and Florent Perronnin. Revisiting the fisher vector for fine-grained classification. Putlern recognition letters, 49:92-98, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, pp- 5767-5777, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR, 2(5))6, 2017.
Tommi Jaakkola and David Haussler. Exploiting generative models in discriminative classifiers. In Advances in neural information processing systems, pp. 487-493, 1999.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Diederik P Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Alexander Kolesnikov and Christoph H Lampert. Pixelenn models with auxiliary variables for nat- ural image modeling. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1905-1914 4. JMLR. org, 2017.
Ping Li, Trevor J Hastie, and Kenneth W Church. Very sparse random projections. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 287-296. ACM, 2006.

---

Under review as a conference paper at ICLR 2020
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint arXiv:1812.01608, 2018.
Sebastian Mika, Gunnar Ratsch, Jason Weston, Bernhard Scholkopf, and Klaus-Robert Mullers. Fisher discriminant analysis with kernels. In Neural networks for signal processing IX: Proceed- ings of the 1999 IEEE signal processing society workshop (cat. no. 98th8468), pp. 41-48. leee, 1999.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 169,,33999 2016a.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukeuoglu. Pixel recurrent neural networks. arXiv preprint wrXiv:1601.0677 066599, 2016b.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Florent Perronnin, Yan Liu, Jorge Sánchez, and Hervé Poirier. Large-scale image retrieval with compressed fisher vectors. In 2010 IEEE Computer Society Conference on Computer Vision and Puttern Recognition, pp. 3384-3391. IEEE, 2010a.
Florent Perronnin, Jorge Sánchez, and Thomas Mensink. Improving the fisher kernel for large-scale image classification. In European conference on computer vision. pp. 143-156. Springer, 2010b.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234 2242, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint 1rXiv:1701.055// 2017.
Jorge Sánchez, Florent Perronnin, Thomas Mensink, and Jakob Verbeek. Image elassification with the fisher vector: Theory and practice. International journal of computer vision, 105(3):222-245, 2013.
Karen Simonyan, Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Fisher vector faces in the wild. In BMVC, volume pp. 4, 2013.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. In Advances in neural information processing systems, pp. 4790-4798. 2016.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
10

---

Under review as a conference paper at ICLR 2020
7 APPENDIX
7.1
PIXELCNN ARCHITECTURES
7.1.1
MNIST
The PixeICNN architecture for MNIST was a standard PixeCNN Oord et al. (2016b), with 5 masked convolutional layers each kernel size 7, padding and 64 filters. We used ReLU for activation function.
7.1.2 CELEBA
1-layer PixelCNN Architecture is the same as the PixelCNN from MNIST, but with only masked convolutional layer.
5-layer PixeICNN
Architecture is the same as the PixcICNN from MNIST
Gated PixeICNN The Gated PixeICNN architecture is the same as described in Van den Oord et al. (2016). We use filter size of 120 dimensions, with 5 masked convolutional layers of kernel size 7 and padding Each masked convolutional layer uses vertical and horizontal stack of convolutions, with residual connections and gating mechanisms.
Pyramid PixelCNN The Pyramid PixeICNN has layers. The bottom layer is Pixel- CNN++ on x 8 down-sampled image. The second layer is a conditional PixcICNN++ that generates 32 32 images conditioned on 8 8 images. The final layer generates 128 128 images conditioned on 32 x 32 images.
7.2 DECODER ARCHITECTURES
7.2.1
MNIST
ResBlock Architecture
Conv2d (channels, h: bias)
BatchNorm, ReLU
Conv2d (channels, &o = 3, padding I, no bias)
BatchNorm, ReLU
Conv2d (channels, k = no bias)
BatchNorm, ReLU
Shorteut Connection, ReLU
Decoder Architecture
R
Linear (1024), ReLU
Conv TonnSraasp (128 filters, = 5, stride 2), ReLU ResBlock(1 128) Conv ConvTranspose (32 filters, l = 5, stride 2) BatchNorm, ReLU
Conv ConyTranspose filter, k stride 2)
7.2.2 CELEBA
Convolutional Model
E IR 401x4096
Conv TonnSranspesee (256 filters, & = 4) 4x ResBlock, Upsample 2x
BatchNorm, ReLU
Conv2d (32 filters, & = I)
11

---

Under review as a conference paper at ICLR 2020
WGAN
Follows the same architecture described https://githip: Ltw.euuui..omm///nnoi..o.. LEA--N-GAN--GANN Www.-RAAA--POOOOO....... the conditioning vector with the noise when generating. For the discriminator, we project the conditioning vector the channel dimension of each ResBlock input, and add it broadcasted across the image as a bias.
To make it conditional, concatenate
RealNVP
We use the Multiscale RealNVP architecture described in Dinh et al. (2016). We apply conditioning as follows: for each ResNet in the affine coupling layers, we project the conditioning vector the channel dimension size, and add it as a bias to the ResNet input.
7.3 RECONSTRUCTIONS FOR DIFFERENT DECODERS
Figure 8: Autoregressive model: Gated PixeICNN, Embedding Space: Activations, Decoder: Convolutional
Network
12

---

Under review as a conference paper at ICLR 2020
Figure 9: Autoregressive model: Gated PixeICNN, Embedding Space: Activations, Decoder: RcaINVP
13

---

Under review as a conference paper at ICLR 2020
109
Figure 10: Autoregressive model: Gated PixeICNN, Embedding Space: Fisher Scores, Decoder: Convolu-
tional Network
14

---

Under review as a conference paper at ICLR 2020
Figure 11: Autoregressive model: Gated PixeICNN, Embedding Space: Fisher Scores, Decoder: Condi-
tional RealNVP
15

---

Under review as a conference paper at ICLR 2020
119
Figure 12: Autoregressive model: Gated PixeICNN, Embedding Space: Fisher Scores, Decoder: Condi-
tional WGAN
16

---

Under review as a conference paper at ICLR 2020
2
Figure 13: Autoregressive model: Gated PixeICNN, Embedding Space: Fisher Scores, Decoder: Condi-
tional Pyramid PixeICNN
17

---

Under review as a conference paper at ICLR 2020
7.4 INTERPOLATIONS FOR DIFFERENT DECODERS
(a) Autoregressive model: Gated PixeICNN, Embedding Space: Activations, Decoder: Convolutional Net- work
(b) Autoregressive model: Gated PixeICNN, Embedding Space: Fisher Scores, Decoder: Convolutional
Network
18

---

Under review as a conference paper at ICLR 2020
4
Autoregressive model: Gated PixeICNN, Embedding Space: Activations, Decoder: RealNVP
2
(b) Autoregressive model: Gated PixeICNN, Embedding Space: Fisher Scores, Decoder: Conditional Real-
NVP
19

---

Under review as a conference paper at ICLR 2020
(a) Autoregressive model: Gated PixeICNN, Embedding Space: Fisher Scores, Decoder: Conditional
WGAN
(b) Autoregressive model: Gated PixeICNN, Embedding Space: Fisher Scores, Decoder: Conditional Pyra-
mid PixeICNN
20

---

Under review as a conference paper at ICLR 2020
7.5 FACIAL ATTRIBUTE MANIPULATION FOR DIFFERENT DECODERS
(a) Autoregressive model: Gated Pixel- CNN, Embedding Space: Activations, De-
(b) Antoregressive model: Gated Pixel- CNN. Embedding Space: Activations, De- coder: Conditional RealNVP
coder:
Conditional RealNVP
(c) Autoregressive model: Gated Pixel- CNN, Embedding Space: Activations, De-
(d) Autoregressive model: Gated Pixel-
CNN, Embedding Space: Activations, De- coder: Conditional RealNVP
coder:
Conditional RealNVP
21

---

Under review as a conference paper at ICLR 2020
(a) Autoregressive model: Gated Pixel- CNN, Embedding Space: Fisher Score, De-
(b) Antoregressive model: Gated Pixel-
CNN, Embedding Space: Fisher Score, De- coder: Conditional RealNVP
coder:
Conditional RealNVP
(c) Autoregressive model: Gated Pixel- CNN, Embeddinggnnnn.... Space: Fisher Score, De-
(d) Autoregressive model: Gated Pixel-
CNN, Embedding Space: Fisher Score, De- coder: Conditional RealNVP
coder:
Conditional ReaINVP
22


Reviews:
[{"review_id": "rygwa-Oi5r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to learn a latent space for the PixelCNN by first computing the Fisher score of the PixelCNN model and then projecting it onto a lower-dimensional space using a sparse random matrix.\n\nMy first concern about this work is its novelty. Defining a feature space using the Fisher kernel of a generative model is a very well-known idea, and there is a large body of work around that. As the paper points out, the problem with the Fisher score for the recent deep generative model architectures is that Fisher score operates in the parameter space and the deep models have a very large number of parameters. The paper proposes to get around this problem by projecting the Fisher score onto a lower-dimensional space using random matrices. But I am not convinced that this random projection can learn useful representations, which brings me to my second concern about the evaluation metric. The paper uses the activation of the last layers of the PixelCNN as a baseline, which I consider to be a very weak baseline. Each activation at spatial position (i,j) only depends on the previous pixels and I believe they are not in general good high-level representations. For the evaluation, the paper only considers the FID score on the interpolated images and reconstructions. There are much better ways to compare the quality of unsupervised representations such as their performance on classifying images with a linear classifier as done in [1]. The paper would improve by comparing the quality of its latent representations with the recent unsupervised/self-supervised learning methods such as [1,2].\n\n[1] Data-Efficient Image Recognition with Contrastive Predictive Coding\n[2] Learning deep representations by mutual information estimation and maximization", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1gJnKcl9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the problem of interpolating between data points using neural autoregressive models. The core idea is that it is possible to use (a smaller-dimensional projection of) the Fisher score of the density function defined by the autoregressive model to represent data points in embedding space, and a neural decoder for mapping them back to input space. Experiments on both MNIST and Celeb suggest that this is a sensible method, and leads to smoother interpolations rather than just relying on the embeddings resulting from the network activations.\n\nMinor: the FID acronym on pg. 2 was not introduced beforehand.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJleU5H-qB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivated by the observation that powerful deep autoregressive models such as PixelCNNs lack the ability to produce semantically meaningful latent embeddings and generate visually appealing interpolated images by latent representation manipulations, this paper proposes using Fisher scores projected to a reasonably low-dimensional space as latent embeddings for image manipulations. A decoder based on a CNN, a Conditional RealNVP, or a Conditional Pyramid PixelCNN is used to decode high-dimensional images from these projected Fisher score.  Experiments with different autoregressive and decoder architectures are conducted on MNIST and CelebA datasets are conducted. \n\nPros:\n\nThis paper is well-written overall and the method is clearly presented.\n\n\nCons:\n\n1) It is well-known that the latent activations of deep autoregressive models don’t contain much semantically meaningful information. It is very obvious that either a CNN decoder, a conditional RealNVP decoder, or a conditional Pyramid PixelCNN decoder conditioned on projected Fisher scores will produce better images because the Fisher scores simply contain much more information about the images than the latent activations. When the $\\alpha$ is small, the learned decoder will function similarly to the original pixelCNN, therefore, latent activations produce smaller FID scores than projected Fisher scores for small $\\alpha$’s. These results are not surprising. Detailed explanations should be added here.\n\n2) The comparisons to baselines are unfair. As mentioned in 1), it’s obvious that Fisher scores contain more information than latent activations for deep autoregressive models and are better suited for manipulations. Fair comparisons should be performed against other latent variable models such as flow models and VAEs with more interesting tasks, which will make the paper much stronger.\n\n3) In Figure 3, how is the reconstruction error calculated? It’s squared error per pixel per image?\n\n4) On pp. 8, for semantic manipulations, some quantitative evaluations will strengthen this part.\n\nIn summary, this paper proposes a novel method based on projected Fisher scores for performing semantically meaningful image manipulations under the framework of deep autoregressive models. However, the experiments are not well-designed and the results are unconvincing. I like the idea proposed in the paper and strongly encourage the authors to seriously address the raised questions regarding experiments and comparisons.\n\n------------------\nAfter Rebuttal:\n\nI took back what I said. It's not that obvious that the \"latent activations of deep autoregressive models don’t contain much semantically meaningful information\". But the latent activations are indeed a weak baseline considering that PixelCNN is so powerful a generator. If the autoregressive generator is powerful enough, the latent activations can theoretically encode nothing.  I have spent a lot of time reviewing this paper and related papers, the technical explanation about the hidden activation calculation of PixelCNN  used in this paper is unclear and lacking (please use equations not just words). \n\nRelated paper:  The PixelVAE paper ( https://openreview.net/pdf?id=BJKYvt5lg ) explains that PixelCNN doesn't learn a good hidden representation for downstream tasks\n\nAnother paper combining VAE and PixelCNN also mentions this point:\n\nECML 2018: http://www.ecmlpkdd2018.org/wp-content/uploads/2018/09/455.pdf\n\nPlease also check the related arguments about PixcelCNN (and the \"Unconditional Decoder\" results) in Variational Lossy Autoencoder (https://arxiv.org/pdf/1611.02731.pdf )\n\nAs I mentioned in the response to the authors' rebuttal, training a separate powerful conditional generative model from some useful condition information (Fisher scores) is feasible to capture the global information in the condition, which is obvious to me. This separate powerful decoder has nothing to do with PixelCNN, which is the major reason that I vote reject.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes learning a latent embedding for image manipulation for PixelCNN by using Fisher scores projected to a low-dimensional space.
The reviewers have several concerns about this paper:
* Novelty
* Random projection doesn’t learn useful representation
* Weak evaluations
Since two expert reviewers are negative about this paper, I cannot recommend acceptance at this stage.

Decision: Reject