Natural Image Manipulation for Autoregressive Models Using Fisher Scores

1 INTRODUCTION . Over the last few decades , unsupervised learning has been a rapidly growing field , with the development of more complex and better probabilistic density models . Autoregressive generative models ( Salimans et al. , 2017 ; Oord et al. , 2016a ; Menick & Kalchbrenner , 2018 ) are one of the most powerful generative models to date , as they generally achieve the best bits per dim compared to other likelihood-based models such as Normalizing Flows or Variational Auto-encoders ( VAEs ) ( Kingma & Welling , 2013 ; Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ) . However , it remains a difficult problem to perform any kind of controlled sample generation using autoregressive models . For example , flow models and VAEs are structured as latent variable models and allow meaningful manipulations in latent space , which can then be mapped back to original data distribution to produce generated samples either through an invertible map or a learned decoder . In the context of natural image modelling , since discrete autoregressive models do not have continuous latent spaces , there is no natural method to apply controlled generation . When a latent space is not used , prior works generally perform controlled sample generation through training models conditioned on auxiliary information , such as class labels or facial attributes ( Van den Oord et al. , 2016 ) . However , this requires a new conditional model to be trained for every new set of labels or features we want to manipulate , which is a time-consuming and tedious task . Ideally , we could structure an unconditional latent space that the autoregressive model could sample from , but where would this latent space come from ? In this paper , we propose a method of image interpolation and manipulation in a latent space defined by the Fisher score of both discrete and continuous autoregressive models . We use PixelCNNs to model the natural image distribution and use them to compute Fisher scores . In order to map back from Fisher score space to natural images , we train a decoder by minimizing reconstruction error . We show that interpolations in Fisher score space provide higher-level semantic meaning compared to baselines such as interpolations in PixelCNN activation space , which produce blurry and incoherent intermediate interpolations similar in nature to interpolations using pixel values . In order to evaluate interpolations quantitatively , for different mixing coefficients α , we calculate FID ( Heusel et al. , 2017 ) of the images decoded from a large sample of convex combinations of latent vectors . In summary , we present two key contributions in our paper : • A novel method for natural image interpolation and semantic manipulation using autoregressive models through Fisher scores • A new quantitative approach to evaluate interpolation quality of images 2 RELATED WORK . There exists a substantial amount of work on natural image manipulation using deep generative models . VAEs ( Higgins et al. , 2017 ) , BigBiGANS ( Donahue et al. , 2016 ; Donahue & Simonyan , 2019 ; Brock et al. , 2018 ) and normalizing flows Kingma & Dhariwal ( 2018 ) provide learned latent spaces in which realistic image manipulation is a natural task . Interpolating through a latent space allows more natural transitions between images compared to pixel value interpolations , which naively overlay images on top of each other . Other prior methods learn hierarchical latent spaces on GANs and VAEs to encourage semantic manipulations to disentangle different levels of global characteristics of facial features , such as skin color , gender , hair color , and facial features ( Karras et al. , 2019 ) . Aside from using a latent space for controlled image generation , prior methods have also trained generative methods conditioned on relevant auxiliary feature labels , such as class labels in ImageNet . Other prior works have also used facial feature embeddings and binary facial attributes from CelebA to manipulate facial characteristics of generated images ( Van den Oord et al. , 2016 ) . Similarly , there has been a large amount of work on using Fisher information in machine learning . Many prior methods use the Fisher Kernel in algorithms such as kernel discriminant analysis and kernel support vector machines ( Mika et al. , 1999 ; Jaakkola & Haussler , 1999 ) . More recent works introduce the Fisher vector , which uses Fisher scores of mixtures of Gaussians as feature vectors for downstream tasks ( Sánchez et al. , 2013 ; Simonyan et al. , 2013 ; Perronnin et al. , 2010b ; Gosselin et al. , 2014 ; Perronnin et al. , 2010a ) . However , to our knowledge , there has been no work in using Fisher scores for deep generative modelling . 3 BACKGROUND . 3.1 PIXELCNN . PixelCNNs ( Oord et al. , 2016b ) are powerful autoregressive models used to model complex image distributions . They are likelihood-based generative models that directly model the data distribution p ( x ) . This allows us to exactly compute the Fisher score instead of using an approximation , which would be necessary for GANs ( Goodfellow et al. , 2014 ) or VAEs , as they either implicitly optimize likelihood , or optimize a variational lower bound . PixelCNNs use a series of masked convolutions to define an autoregressive model over image data . Masked convolutions allow PixelCNNs to retain the autoregressive ordering when propagating values through layers . Over the past few years , many improved variants of PixelCNNs have been developed to address certain problems with the original PixelCNN design . Specifically for our work , we use Gated PixelCNNs ( Van den Oord et al. , 2016 ) , which introduced horizontal and vertical convolutional blocks to remove the blind-spot in the original masked convolutions of PixelCNNs , and Pyramid PixelCNNs ( Kolesnikov & Lampert , 2017 ) , which use PixelCNN++ ( Salimans et al. , 2017 ) architectures in a hierarchical fashion , with each layer of the hierarchy modelling images at different down-sampled resolutions conditioned on the previous layer . 3.2 FISHER SCORE . The Fisher score ˙̀ ( x ; θ ) = ∇θ log pθ ( x ) is defined as the gradient of the log-probability of a sample with respect to the model parameters θ . Intuitively , Fisher scores describe the contributions of each parameter during the generation process . Similar samples in the data distribution should elicit similar Fisher scores . Similarity between samples can also be evaluated by taking a gradient step for one sample , and checking if the log-likelihood of another sample also increases . This intuition is re-affirmed in the underlying mathematical interpretation of Fisher information - the Fisher score maps data points onto a Riemannian manifold with a local metric given by the Fisher information matrix . The underlying kernel of this space is the Fisher kernel , defined as : ˙̀ ( xi ; θ ) TF−1 ˙̀ ( xj ; θ ) = ( F − 12 ˙̀ ( xi ; θ ) ) T ( F− 1 2 ˙̀ ( xj ; θ ) ) where F−1 is the inverse Fisher information matrix , and F− 1 2 is the Cholesky Decomposition . Applying F− 1 2 is a normalization process , so the Fisher kernel can be approximated by taking the dot product of standardized Fisher scores . This is useful since computing F−1 is normally difficult . As such , we can see the collection of Fisher scores as a high dimensional embedding space in which more meaningful information about the data distribution can be extracted compared to raw pixel values . More complex deep generative models may learn parameters that encode information at high-levels of abstraction which may be reflected as high-level features in Fisher scores . 3.3 SPARSE RANDOM PROJECTIONS . It would be cumbersome to work in the very high-dimensional parameter spaces of deep generative models , so we use dimensionality reduction methods to make our methods more scalable . Sparse random projections allow for memory-efficient and scalable projections of high dimensional vectors . The Johnson-Lindenstrauss Lemma ( Dasgupta & Gupta , 1999 ) states that under a suitable orthogonal projection , a set of n points in a d-dimensional space can be accurately embedded to some k-dimensional vector space , where k depends only on log n. Therefore , for suitably large k , we can preserve the norms and relative distances between projected points , even for very high dimensional data . Since sparse random matrices are nearly orthogonal in high-dimensional settings , we can safely substantially reduce the dimensionality of our embedding spaces using this method . We generate sparse random matrices according to Li et al . ( 2006 ) . Given an n× k matrix to project , we define the minimum density of our sparse random matrix as d = 1√ k , and let s = 1d . Suppose that we are projecting the data into p dimensions , then the n × p projection matrix P is generated according to the following distribution : Pij = − s√p with probability 1 2s 0 with probability 1− 1s s√ p with probability 1 2s ( 1 ) where Pij is the element of the ith row and jth column of the projection matrix . Algorithm 1 : The procedural generation of the new interpolated embedding dataset for quantitative evaluation . Following FID conventions , we use a sample size of 50000 images . Result : α-interpolated dataset Dα Input : The dataset D , projection matrix P , pre-trained autoregressive model pθ ( x ) , and the learned decoder Dec Dα← { } for i← 0 to 50000 do Sample x1 , x2 ∼ D , a random pair of samples z1 ← P∇θ log pθ ( x1 ) z2 ← P∇θ log pθ ( x2 ) ẑ ← ( 1− α ) z1 + αz2 x̂← Dec ( ẑ ) Dα ← Dα ∪ { x̂ } end 4 METHOD . We now describe our approach for natural image manipulation and interpolation using autoregressive models . Note that this method is not restricted to autoregressive models and can be applied to any likelihood-based models . 4.1 EMBEDDING SPACE CONSTRUCTION . Given a trained autoregressive model , pθ ( x ) , we want to construct an embedding space that is more meaningful than raw pixel values . In this paper , our autoregressive models are exclusively variants of PixelCNNs since we are working in an image domain . Drawing inspiration off popular selfsupervised methods ( Zhang et al. , 2016 ; Gidaris et al. , 2018 ; Doersch et al. , 2015 ; Pathak et al. , 2016 ; Noroozi & Favaro , 2016 ) , it may seem natural to take the output activations of one of the last few convolutional layers . However in our experiments , we show that using output activations provides less meaningful embedding manipulation than using the Fisher score . This is especially the case for PixelCNNs , as PixelCNNs are known for generally encoding more local statistics than global features of images . However , Fisher scores lie in parameter space , which may encode more global semantics . In order to construct the embedding space using Fisher scores , we initialize a sparse random matrix P , randomly generated following the distribution in equation 1 . In practice , we found sparse random matrices the only feasible method for reasonable dimensionality reduction when scaling to more difficult datasets , where the sizes of corresponding PixelCNNs grew to tens of millions of parameters . Finally , the embedding space is constructed as follows : each element zi in the new embedding space is computed as zi = P∇θ log pθ ( xi ) for each sample xi in the dataset .

Summary: This paper focuses on the problem of interpolating between data points using neural autoregressive models. The core idea is that it is possible to use (a smaller-dimensional projection of) the Fisher score of the density function defined by the autoregressive model to represent data points in embedding space, and a neural decoder for mapping them back to input space. Experiments on both MNIST and Celeb suggest that this is a sensible method, and leads to smoother interpolations rather than just relying on the embeddings resulting from the network activations.