Natural Image Manipulation for Autoregressive Models Using Fisher Scores

1 INTRODUCTION . Over the last few decades , unsupervised learning has been a rapidly growing field , with the development of more complex and better probabilistic density models . Autoregressive generative models ( Salimans et al. , 2017 ; Oord et al. , 2016a ; Menick & Kalchbrenner , 2018 ) are one of the most powerful generative models to date , as they generally achieve the best bits per dim compared to other likelihood-based models such as Normalizing Flows or Variational Auto-encoders ( VAEs ) ( Kingma & Welling , 2013 ; Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ) . However , it remains a difficult problem to perform any kind of controlled sample generation using autoregressive models . For example , flow models and VAEs are structured as latent variable models and allow meaningful manipulations in latent space , which can then be mapped back to original data distribution to produce generated samples either through an invertible map or a learned decoder . In the context of natural image modelling , since discrete autoregressive models do not have continuous latent spaces , there is no natural method to apply controlled generation . When a latent space is not used , prior works generally perform controlled sample generation through training models conditioned on auxiliary information , such as class labels or facial attributes ( Van den Oord et al. , 2016 ) . However , this requires a new conditional model to be trained for every new set of labels or features we want to manipulate , which is a time-consuming and tedious task . Ideally , we could structure an unconditional latent space that the autoregressive model could sample from , but where would this latent space come from ? In this paper , we propose a method of image interpolation and manipulation in a latent space defined by the Fisher score of both discrete and continuous autoregressive models . We use PixelCNNs to model the natural image distribution and use them to compute Fisher scores . In order to map back from Fisher score space to natural images , we train a decoder by minimizing reconstruction error . We show that interpolations in Fisher score space provide higher-level semantic meaning compared to baselines such as interpolations in PixelCNN activation space , which produce blurry and incoherent intermediate interpolations similar in nature to interpolations using pixel values . In order to evaluate interpolations quantitatively , for different mixing coefficients α , we calculate FID ( Heusel et al. , 2017 ) of the images decoded from a large sample of convex combinations of latent vectors . In summary , we present two key contributions in our paper : • A novel method for natural image interpolation and semantic manipulation using autoregressive models through Fisher scores • A new quantitative approach to evaluate interpolation quality of images 2 RELATED WORK . There exists a substantial amount of work on natural image manipulation using deep generative models . VAEs ( Higgins et al. , 2017 ) , BigBiGANS ( Donahue et al. , 2016 ; Donahue & Simonyan , 2019 ; Brock et al. , 2018 ) and normalizing flows Kingma & Dhariwal ( 2018 ) provide learned latent spaces in which realistic image manipulation is a natural task . Interpolating through a latent space allows more natural transitions between images compared to pixel value interpolations , which naively overlay images on top of each other . Other prior methods learn hierarchical latent spaces on GANs and VAEs to encourage semantic manipulations to disentangle different levels of global characteristics of facial features , such as skin color , gender , hair color , and facial features ( Karras et al. , 2019 ) . Aside from using a latent space for controlled image generation , prior methods have also trained generative methods conditioned on relevant auxiliary feature labels , such as class labels in ImageNet . Other prior works have also used facial feature embeddings and binary facial attributes from CelebA to manipulate facial characteristics of generated images ( Van den Oord et al. , 2016 ) . Similarly , there has been a large amount of work on using Fisher information in machine learning . Many prior methods use the Fisher Kernel in algorithms such as kernel discriminant analysis and kernel support vector machines ( Mika et al. , 1999 ; Jaakkola & Haussler , 1999 ) . More recent works introduce the Fisher vector , which uses Fisher scores of mixtures of Gaussians as feature vectors for downstream tasks ( Sánchez et al. , 2013 ; Simonyan et al. , 2013 ; Perronnin et al. , 2010b ; Gosselin et al. , 2014 ; Perronnin et al. , 2010a ) . However , to our knowledge , there has been no work in using Fisher scores for deep generative modelling . 3 BACKGROUND . 3.1 PIXELCNN . PixelCNNs ( Oord et al. , 2016b ) are powerful autoregressive models used to model complex image distributions . They are likelihood-based generative models that directly model the data distribution p ( x ) . This allows us to exactly compute the Fisher score instead of using an approximation , which would be necessary for GANs ( Goodfellow et al. , 2014 ) or VAEs , as they either implicitly optimize likelihood , or optimize a variational lower bound . PixelCNNs use a series of masked convolutions to define an autoregressive model over image data . Masked convolutions allow PixelCNNs to retain the autoregressive ordering when propagating values through layers . Over the past few years , many improved variants of PixelCNNs have been developed to address certain problems with the original PixelCNN design . Specifically for our work , we use Gated PixelCNNs ( Van den Oord et al. , 2016 ) , which introduced horizontal and vertical convolutional blocks to remove the blind-spot in the original masked convolutions of PixelCNNs , and Pyramid PixelCNNs ( Kolesnikov & Lampert , 2017 ) , which use PixelCNN++ ( Salimans et al. , 2017 ) architectures in a hierarchical fashion , with each layer of the hierarchy modelling images at different down-sampled resolutions conditioned on the previous layer . 3.2 FISHER SCORE . The Fisher score ˙̀ ( x ; θ ) = ∇θ log pθ ( x ) is defined as the gradient of the log-probability of a sample with respect to the model parameters θ . Intuitively , Fisher scores describe the contributions of each parameter during the generation process . Similar samples in the data distribution should elicit similar Fisher scores . Similarity between samples can also be evaluated by taking a gradient step for one sample , and checking if the log-likelihood of another sample also increases . This intuition is re-affirmed in the underlying mathematical interpretation of Fisher information - the Fisher score maps data points onto a Riemannian manifold with a local metric given by the Fisher information matrix . The underlying kernel of this space is the Fisher kernel , defined as : ˙̀ ( xi ; θ ) TF−1 ˙̀ ( xj ; θ ) = ( F − 12 ˙̀ ( xi ; θ ) ) T ( F− 1 2 ˙̀ ( xj ; θ ) ) where F−1 is the inverse Fisher information matrix , and F− 1 2 is the Cholesky Decomposition . Applying F− 1 2 is a normalization process , so the Fisher kernel can be approximated by taking the dot product of standardized Fisher scores . This is useful since computing F−1 is normally difficult . As such , we can see the collection of Fisher scores as a high dimensional embedding space in which more meaningful information about the data distribution can be extracted compared to raw pixel values . More complex deep generative models may learn parameters that encode information at high-levels of abstraction which may be reflected as high-level features in Fisher scores . 3.3 SPARSE RANDOM PROJECTIONS . It would be cumbersome to work in the very high-dimensional parameter spaces of deep generative models , so we use dimensionality reduction methods to make our methods more scalable . Sparse random projections allow for memory-efficient and scalable projections of high dimensional vectors . The Johnson-Lindenstrauss Lemma ( Dasgupta & Gupta , 1999 ) states that under a suitable orthogonal projection , a set of n points in a d-dimensional space can be accurately embedded to some k-dimensional vector space , where k depends only on log n. Therefore , for suitably large k , we can preserve the norms and relative distances between projected points , even for very high dimensional data . Since sparse random matrices are nearly orthogonal in high-dimensional settings , we can safely substantially reduce the dimensionality of our embedding spaces using this method . We generate sparse random matrices according to Li et al . ( 2006 ) . Given an n× k matrix to project , we define the minimum density of our sparse random matrix as d = 1√ k , and let s = 1d . Suppose that we are projecting the data into p dimensions , then the n × p projection matrix P is generated according to the following distribution : Pij = − s√p with probability 1 2s 0 with probability 1− 1s s√ p with probability 1 2s ( 1 ) where Pij is the element of the ith row and jth column of the projection matrix . Algorithm 1 : The procedural generation of the new interpolated embedding dataset for quantitative evaluation . Following FID conventions , we use a sample size of 50000 images . Result : α-interpolated dataset Dα Input : The dataset D , projection matrix P , pre-trained autoregressive model pθ ( x ) , and the learned decoder Dec Dα← { } for i← 0 to 50000 do Sample x1 , x2 ∼ D , a random pair of samples z1 ← P∇θ log pθ ( x1 ) z2 ← P∇θ log pθ ( x2 ) ẑ ← ( 1− α ) z1 + αz2 x̂← Dec ( ẑ ) Dα ← Dα ∪ { x̂ } end 4 METHOD . We now describe our approach for natural image manipulation and interpolation using autoregressive models . Note that this method is not restricted to autoregressive models and can be applied to any likelihood-based models . 4.1 EMBEDDING SPACE CONSTRUCTION . Given a trained autoregressive model , pθ ( x ) , we want to construct an embedding space that is more meaningful than raw pixel values . In this paper , our autoregressive models are exclusively variants of PixelCNNs since we are working in an image domain . Drawing inspiration off popular selfsupervised methods ( Zhang et al. , 2016 ; Gidaris et al. , 2018 ; Doersch et al. , 2015 ; Pathak et al. , 2016 ; Noroozi & Favaro , 2016 ) , it may seem natural to take the output activations of one of the last few convolutional layers . However in our experiments , we show that using output activations provides less meaningful embedding manipulation than using the Fisher score . This is especially the case for PixelCNNs , as PixelCNNs are known for generally encoding more local statistics than global features of images . However , Fisher scores lie in parameter space , which may encode more global semantics . In order to construct the embedding space using Fisher scores , we initialize a sparse random matrix P , randomly generated following the distribution in equation 1 . In practice , we found sparse random matrices the only feasible method for reasonable dimensionality reduction when scaling to more difficult datasets , where the sizes of corresponding PixelCNNs grew to tens of millions of parameters . Finally , the embedding space is constructed as follows : each element zi in the new embedding space is computed as zi = P∇θ log pθ ( xi ) for each sample xi in the dataset .

Summary: Motivated by the observation that powerful deep autoregressive models such as PixelCNNs lack the ability to produce semantically meaningful latent embeddings and generate visually appealing interpolated images by latent representation manipulations, this paper proposes using Fisher scores projected to a reasonably low-dimensional space as latent embeddings for image manipulations. A decoder based on a CNN, a Conditional RealNVP, or a Conditional Pyramid PixelCNN is used to decode high-dimensional images from these projected Fisher score.  Experiments with different autoregressive and decoder architectures are conducted on MNIST and CelebA datasets are conducted.