Improved Autoregressive Modeling with Distribution Smoothing

1 INTRODUCTION . Autoregressive models have exhibited promising results in a variety of downstream tasks . For instance , they have shown success in compressing images ( Minnen et al. , 2018 ) , synthesizing speech ( Oord et al. , 2016a ) and modeling complex decision rules in games ( Vinyals et al. , 2019 ) . However , the sample quality of autoregressive models on real-world image datasets is still lacking . Poor sample quality might be explained by the manifold hypothesis : many real world data distributions ( e.g . natural images ) lie in the vicinity of a low-dimensional manifold ( Belkin & Niyogi , 2003 ) , leading to complicated densities with sharp transitions ( i.e . high Lipschitz constants ) , which are known to be difficult to model for density models such as normalizing flows ( Cornish et al. , 2019 ) . Since each conditional of an autoregressive model is a 1-dimensional normalizing flow ( given a fixed context of previous pixels ) , a high Lipschitz constant will likely hinder learning of autoregressive models . Another reason for poor sample quality is the “ compounding error ” issue in autoregressive modeling . To see this , we note that an autoregressive model relies on the previously generated context to make a prediction ; once a mistake is made , the model is likely to make another mistake which compounds ( Kääriäinen , 2006 ) , eventually resulting in questionable and unrealistic samples . Intuitively , one would expect the model to assign low-likelihoods to such unrealistic images , however , this is not always the case . In fact , the generated samples , although appearing unrealistic , often are assigned high-likelihoods by the autoregressive model , resembling an “ adversarial example ” ( Szegedy et al. , 2013 ; Biggio et al. , 2013 ) , an input that causes the model to output an incorrect answer with high confidence . Inspired by the recent success of randomized smoothing techniques in adversarial defense ( Cohen et al. , 2019 ) , we propose to apply randomized smoothing to autoregressive generative modeling . More specifically , we propose to address a density estimation problem via a two-stage process . Unlike Cohen et al . ( 2019 ) which applies smoothing to the model to make it more robust , we apply smoothing to the data distribution . Specifically , we convolve a symmetric and stationary noise distribution with the data distribution to obtain a new “ smoother ” distribution . In the first stage , we model the smoothed version of the data distribution using an autoregressive model . In the second stage , we reverse the smoothing process—a procedure which can also be understood as “ denoising ” —by either applying a gradient-based denoising approach ( Alain & Bengio , 2014 ) or introducing another conditional autoregressive model to recover the original data distribution from the smoothed one . By choosing an appropriate smoothing distribution , we aim to make each step easier than the original learning problem : smoothing facilitates learning in the first stage by making the input distribution fully supported without sharp transitions in the density function ; generating a sample given a noisy one is easier than generating a sample from scratch . We show with extensive experimental results that our approach is able to drastically improve the sample quality of current autoregressive models on several synthetic datasets and real-world image datasets , while obtaining competitive likelihoods on synthetic datasets . We empirically demonstrate that our method can also be applied to density estimation , image inpainting , and image denoising . 2 BACKGROUND . We consider a density estimation problem . Given D-dimensional i.i.d samples { x1 , x2 , ... , xN } from a continuous data distribution pdata ( x ) , the goal is to approximate pdata ( x ) with a model pθ ( x ) parameterized by θ . A commonly used approach for density estimation is maximum likelihood estimation ( MLE ) , where the objective is to maximize L ( θ ) , 1N ∑N i=1 log pθ ( xi ) . 2.1 AUTOREGRESSIVE MODELS . An autoregressive model ( Larochelle & Murray , 2011 ; Salimans et al. , 2017 ) decomposes a joint distribution pθ ( x ) into the product of univariate conditionals : pθ ( x ) = D∏ i=1 pθ ( xi|x < i ) , ( 1 ) where xi stands for the i-th component of x , and x < i refers to the components with indices smaller than i . In general , an autoregressive model parameterizes each conditional pθ ( xi|x < i ) using a prespecified density function ( e.g . mixture of logistics ) . This bounds the capacity of the model by limiting the number of modes for each conditional . Although autoregressive models have achieved top likelihoods amongst all types of density based models , their sample quality is still lacking compared to energy-based models ( Du & Mordatch , 2019 ) and score-based models ( Song & Ermon , 2019 ) . We believe this can be caused by the following two reasons . 2.2 MANIFOLD HYPOTHESIS . Several existing methods ( Roweis & Saul , 2000 ; Tenenbaum et al. , 2000 ) rely on the manifold hypothesis , i.e . that real-world high-dimensional data tends to lie on a low-dimensional manifold ( Narayanan & Mitter , 2010 ) . If the manifold hypothesis is true , then the density of the data distribution is not well defined in the ambient space ; if the manifold hypothesis holds only approximately and the data lies in the vicinity of a manifold , then only points that are very close to the manifold would have high density , while all other points would have close to zero density . Thus we may expect the data density around the manifold to have large first-order derivatives , i.e . the density function has a high Lipschitz constant ( if not infinity ) . To see this , let us consider a 2-d example where the data distribution is a thin ring distribution ( almost a unit circle ) formed by rotating the 1-d Gaussian distribution N ( 1 , 0.012 ) around the origin . The density function of the ring has a high Lipschitz constant near the “ boundary ” . Let us focus on a data point travelling along the diagonal as shown in the leftmost panel in figure 2 . We plot the first-order directional derivatives of the density for the point as it approaches the boundary from the inside , then lands on the ring , and finally moves outside the ring ( see figure 2 ) . As we can see , when the point is far from the boundary , the derivative has a small magnitude . When the point moves closer to the boundary , the magnitude increases and changes significantly near the boundary even with small displacements in the trajectory . However , once the point has landed on the ring , the magnitude starts to decrease . As it gradually moves off the ring , the magnitude first increases and then decreases just like when the point approached the boundary from the inside . It has been observed that certain likelihood models , such as normalizing flows , exhibit pathological behaviors on data distributions whose densities have high Lipschitz constants ( Cornish et al. , 2019 ) . Since each conditional of an autoregressive model is a 1-d normalizing flow given a fixed context , a high Lipschitz constant on data density could also hinder learning of autoregressive models . 2.3 COMPOUNDING ERRORS IN AUTOREGRESSIVE MODELING . Autoregressive models can also be susceptible to compounding errors from the conditional distributions ( Lamb et al. , 2016 ) during sampling time . We notice that an autoregressive model pθ ( x ) learns the joint density pdata ( x ) by matching each of the conditional pθ ( xi|x < i ) with pdata ( xi|x < i ) . In practice , we typically have access to a limited amount of training data , which makes it hard for an autoregressive model to capture all the conditional distributions correctly due to the curse of dimensionality . During sampling , since a prediction is made based on the previously generated context , once a mistake is made at a previous step , the model is likely to make more mistakes in the later steps , eventually generating a sample x̂ that is far from being an actual image , but is mistakenly assigned a high-likelihood by the model . The generated image x̂ , being unrealistic but assigned a high-likelihood , resembles an adversarial example , i.e. , an input that causes the model to make mistakes . Recent works ( Cohen et al. , 2019 ) in adversarial defense have shown that random noise can be used to improve the model ’ s robustness to adversarial perturbations — a process during which adversarial examples that are close to actual data are generated to fool the model . We hypothesize that such approach can also be applied to improve an autoregressive modeling process by making the model less vulnerable to compounding errors occurred during density estimation . Inspired by the success of randomized smoothing in adversarial defense ( Cohen et al. , 2019 ) , we propose to apply smoothing to autoregressive modeling to address the problems mentioned above . 3 GENERATIVE MODELS WITH DISTRIBUTION SMOOTHING . In the following , we propose to decompose a density estimation task into a smoothed data modeling problem followed by an inverse smoothing problem where we recover the true data density from the smoothed one . 3.1 RANDOMIZED SMOOTHING PROCESS . Unlike Cohen et al . ( 2019 ) where randomized smoothing is applied to a model , we apply smoothing directly to the data distribution pdata ( x ) . To do this , we introduce a smoothing distribution q ( x̃|x ) — a distribution that is symmetric and stationary ( e.g . a Gaussian or Laplacian kernel ) — and convolve it with pdata ( x ) to obtain a new distribution q ( x̃ ) , ∫ q ( x̃|x ) pdata ( x ) dx . When q ( x̃|x ) is a normal distribution , this convolution process is equivalent to perturbing the data distribution with Gaussian noise , which , intuitively , will make the data distribution smoother . In the following , we formally prove that convolving a 1-d distribution pdata ( x ) with a suitable noise can indeed “ smooth ” pdata ( x ) . Theorem 1 . Given a continuous and bounded 1-d distribution pdata ( x ) that is supported on R , for any 1-d distribution q ( x̃|x ) that is symmetric ( i.e . q ( x̃|x ) = q ( x|x̃ ) ) , stationary ( i.e . translation invariant ) and satisfies limx→∞ pdata ( x ) q ( x|x̃ ) = 0 for any given x̃ , we have Lip ( q ( x̃ ) ) ≤ Lip ( pdata ( x ) ) , where q ( x̃ ) , ∫ q ( x̃|x ) pdata ( x ) dx and Lip ( · ) denotes the Lipschitz constant of the given 1-d function . Theorem 1 shows that convolving a 1-d data distribution pdata ( x ) with a suitable noise distribution q ( x̃|x ) ( e.g.N ( x̃|x , σ2 ) ) can reduce the Lipschitzness ( i.e . increase the smoothness ) of pdata ( x ) . We provide the proof of Theorem 1 in Appendix A . Given pdata ( x ) with a high Lipschitz constant , we empirically verify that density estimation becomes an easier task on the smoothed distribution q ( x̃ ) than directly on pdata ( x ) . To see this , we visualize a 1-d example in figure 3a , where we want to model a ten-mode data distribution with a mixture of logistics model . If our model has three logistic components , there is almost no way for the model , which only has three modes , to perfectly fit this data distribution , which has ten separate modes with sharp transitions . The model , after training ( see figure 3a ) , mistakenly assigns a much higher density to the low density regions between nearby modes . If we convolve the data distribution with q ( x̃|x ) = N ( x̃|x , 0.52 ) , the new distribution becomes smoother ( see figure 3b ) and can be captured reasonably well by the same mixture of logistics model with only three modes ( see figure 3b ) . Comparing the same model ’ s performance on the two density estimation tasks , we can see that the model is doing a better job at modeling the smoothed version of the data distribution than the original data distribution , which has a high Lipschitz constant . This smoothing process can also be understood as a regularization term for the original maximum likelihood objective ( on the un-smoothed data distribution ) , encouraging the learned model to be smooth , as formalized by the following statement : Proposition 1 ( Informal ) . Assume that the symmetric and stationary smoothing distribution q ( x̃|x ) has small variance and negligible higher order moments , then Epdata ( x ) Eq ( x̃|x ) [ log pθ ( x̃ ) ] ≈ Epdata ( x ) [ log pθ ( x ) + η 2 ∑ i ∂2 log pθ ∂x2i ] , for some constant η . Proposition 1 shows that our smoothing process provides a regularization effect on the original objective Epdata ( x ) [ log pθ ( x ) ] when no noise is added , where the regularization aims to maximize η 2 ∑ i ∂2 log pθ ∂x2i . Since samples from pdata should be close to a local maximum of the model , this encourages the second order gradients computed at a data point x to become closer to zero ( if it were positive then x will not be a local maximum ) , creating a smoothing effect . This extra term is also the trace of the score function ( up to a multiplicative constant ) that can be found in the score matching objective ( Hyvärinen , 2005 ) , which is closely related to many denoising methods ( Vincent , 2011 ; Hyvärinen , 2008 ) . This regularization effect can , intuitively , increase the generalization capability of the model . In fact , it has been demonstrated empirically that training with noise can lead to improvements in network generalization ( Sietsma & Dow , 1991 ; Bishop , 1995 ) . Our argument is also similar to that used in ( Bishop , 1995 ) except that we consider a more general generative modeling case as opposed to supervised learning with squared error . We provide the formal statement and proof of Proposition 1 in Appendix A .

Summary: .** Autoregressive models have demonstrate their potential utility for modeling images and other types of complex data with high flexibility (particularly in density estimation). However, its sampling ability is not that good as explained in the paper. Authors show that one of the main weaknesses of autoregressive models comes from the propagation of mistakes due to the mismatch of conditionals. Inspired in the promising results of randomized smoothing in adversarial models (Cohen et al. 2019), authors propose a similar strategy. The addition of Gaussian noise and posterior modeling of the smoother data makes easier to the autoregressive density to capture the true data distribution. The benefits of this strategy are empirically proved and shown in the experiments.