Improved Autoregressive Modeling with Distribution Smoothing
Authors: Chenlin Meng
Jiaming Song
Yang Song
Shengjia Zhao
Stefano Ermon
Venue: ICLR 2021 Oral (2021)
Keywords: generative models
autoregressive models

Abstract: While autoregressive models excel at image compression, their sample quality is often lacking. Although not realistic, generated images often have high likelihood according to the model, resembling the case of adversarial examples. Inspired by a successful adversarial defense method, we incorporate randomized smoothing into autoregressive generative modeling. We first model a smoothed version of the data distribution, and then reverse the smoothing process to recover the original data distribution. This procedure drastically improves the sample quality of existing autoregressive models on several synthetic and real-world image datasets while obtaining competitive likelihoods on synthetic datasets.

Published as a conference paper at ICLR 2021
IMPROVED AUTOREGRESSIVE MODELING WITH DISTRIBUTION SMOOTHING
Chenlin Meng, Jiaming Song, Yang Song, Shengjia Zhao & Stefano Ermon Stanford University
{chenl in, tsong,, yangsong, s sizho, samoonees.staaiiin edu
ABSTRACT
While autoregressive models excel at image compression, their sample quality is often lacking. Although realistic, generated images often have high likelihood according to the model, resembling the case of adversarial examples. Inspired by a successful adversarial defense method, we incorporate randomized smoothing into autoregressive generative modeling, We first model a smoothed version of the data distribution, and then reverse the smoothing process to recover the original data distribution. This procedure drastically improves the sample quality of existing autoregressive models on several synthetic and real-world image datasets while obtaining competitive likelihoods synthetic datasets.
INTRODUCTION
Autoregressive models have exhibited promising results in a variety of downstream tasks. For instance, they have shown success in compressing images (Minnen al., 2018), synthesizing speech (Oord et al., 2016a) and modeling complex decision rules games (Vinyals et al., 2019). However, the sample quality of autoregressive models on real-world image datasets still lacking.
Poor sample quality might be explained by the manifold hypothesis: many real world data distri- butions (e.g. natural images) lie in the vicinity of low-dimensional manifold (Belkin & Niyogi, 2003), leading to complicated densities with sharp transitions (i.e. high Lipschitz constants), which are known 2019). Since each conditional of an autoregressive model is I 1-dimensional normalizing flow (given fixed context previous pixels), a high Lipschitz constant will likely hinder learning of autoregressive models.
difficult to model for density models such as normalizing flows (Cornish et al.,
Another reason for poor sample quality the "compounding error" issue in autoregressive model- ing. To see this, we note that autoregressive model relies the previously generated context to make a prediction; once a mistake is made, the model is likely to make another mistake which com- pounds (Kääriäinen, 2006), eventually resulting in questionable and unrealistic samples. Intuitively, one would expect the model to assign lielliicls to such unrealistic images, however, this not always the case. In fact, the generated samples, although appearing unrealistic, often assigned high-likelihoods by the autoregressive model, resembling an "adversarial example" (Szegedy et al., 2013; Biggio al., 2013), input that causes the model to output an incorrect answer with high confidence.
Inspired by the recent success of randomized smoothing techniques in adversarial defense (Cohen et al., 2019), we propose to apply randomized smoothing to autoregressive generative modeling. More specifically, we propose to address a density estimation problem via a two-stage process. Unlike Cohen et al. (2019) which applies smoothing to the model to make it more robust, we apply smoothing to the data distribution. Specifically, we convolve a symmetric and stationary noise distri- bution with the data distribution to obtain new "smoother" distribution. In the first stage, we model the smoothed version of the data distribution using an autoregressive model. the second stage, reverse the smoothing process-2 procedure which can also be understood as "denoising" either applying a gradient-basee denoising approach (Alain & Bengio, 2014) or introducing another conditional autoregressive model to recover the original data distribution from the smoothed one. By choosing an appropriate smoothing distribution, we aim to make each step easier than the original learning problem: smoothing facilitates learning in the first stage by making the input distribution

---

Published as a conference paper at ICLR 2021
px(x
X
((x)
((x(x)
Overview of our method. From a data distribution (x) we inject noise (g(x)x)) which makes the distribution smoother (x); then we model the smoothed distribution (po (X)) as well the denoising step (pe (x) forming a two-step model.
Figure I:
fully supported without sharp transitions in the density function; generating a sample given a noisy one is easier than generating sample from scratch.
We show with extensive experimental results that our approach is able to drastically improve the sample quality of current autoregressive models several synthetic datasets and real-world image datasets, while obtaining competitive likelihoods on synthetic datasets. We empirically demonstrate that method can also be applied to density estimation, image inpainting, and image denoising.
BACKGROUND
We consider density estimation problem. Given D-dimensional i.i.d samples {x1, X2, ... XN) from a continuous data distribution (x), the goal is to approximate Pdata (x) with model (x( parameterized by 0. A commonly used approach for density estimation is maximum likelihood estimation (MLE), where the objective is to maximize T.(0) A ΣN lx,))(x;).
2.1 AUTOREGRESSIVE MODELS
An autoregressive model (Larochelle & Murray, 2011; Salimans al., 2017) decomposes joint distribution (x) into the product of univariate conditionals:
(x(x) II po (x. i=1
(I)
where stands for the i-th component of x, and x refers to the components with indices smaller than In general, an autoregressive model parameterizes each conditional po (x(((x<)) using pre- specified density function (e.g. mixture of logistics). This bounds the capacity of the model by limiting the number of modes for each conditional.
Although autoregressive models have achieved top likelihoods amongst all types of density based models, their sample quality is still lacking compared to energy-based models (Du Mordatch, 2019) and score-based models (Song & Ermon, 2019), We believe this can be caused by the follow- ing two reasons.
2.2 MANIFOLD HYPOTHESIS
Several existing methods (Roweis & Saul, 2000; Tenenbaum al., 2000) rely on the manifold hypothesis, i.e. that real-world nigh-dimensional data tends to lie on a low-dimensiona mani- fold (Narayanan & Mitter, 2010). If the manifold hypothesis is true, then the density of the data distribution is not well defined in the ambient space; if the manifold hypothesis holds only approx- imately and the data lies in the vicinity of a manifold, then only points that are very close to the manifold would have high density, while all other points would have close to zero density. Thus may expect the data density around the manifold to have large first-order derivatives, i.c the density function has high Lipschitz constant (if not infinity).
To see this, let us consider a 2-d example where the data distribution is thin ring distribution (almost unit circle) formed by rotating the 1-d Gaussian distribution (1, 0.013) around the origin. The density function of the ring has high Lipschitz constant near the "boundary". Let us focus on a data point travelling along the diagonal shown in the leftmost panel in figure 2. We plot the first-order
2

---

Published as a conference paper at ICLR 2021
Trjlectrry Data
Trajectory
-1e-1
-1.6e-2
-0.5e-2
-1e-3
0
1e-3 1e-2
1e-1
Figure 2: Manifold hypothesis illustration. The data point is travelling along the diagonal as shown in the leftmost panel. The white arrow stands for the direction and magnitude of the derivative of density at the data point. The data location for each figure is (v0.5 + c, v + c), where the number below each figure and (v0.5, v0.5) the upper right intersection of the trajectory with the unit circle.
directional derivatives of the density for the point as it approaches the boundary from the inside, then lands on the ring, and finally moves outside the ring (see figure 2). As we can see, when the point is far from the boundary, the derivative has small magnitude. When the point moves closer to the boundary, the magnitude increases and changes significantly near the boundary even with small displacements in the trajectory. However, once the point has landed on the ring, the magnitude starts to decrease. As gradually moves off the ring, the magnitude first increases and then decreases just like when the point approached the boundary from the inside. It has been observed that certain likelihood models, such as normalizing flows, exhibit pathological behaviors on data distributions whose densities have high Lipschitz constants (Cornish et al., 2019). Since each conditional of an autoregressive model is 1-d normalizing flow given a fixed context, a high Lipschitz constant on data density could also hinder learning of autoregressive models.
2.3 COMPOUNDING ERRORS IN AUTOREGRESSIVE MODELING
Autoregressive models can also be susceptible to compounding errors from the conditional distri- butions (Lamb et al., 2016) during sampling time. We notice that autoregressive model (x) learns the joint density Paua(xx (x) by matching each the conditional po (xx)x with Pama( (xx). In practice, typically have access to limited amount of training data, which makes it hard for an autoregressive model to capture all the conditional distributions correctly due to the curse of dimen- sionality. During sampling, since a prediction made based the previously generated context, once a mistake is made at a previous step, the model is likely to make more mistakes in the later steps, eventually generating sample that far from being an actual image, but is mistakenly assigned a high-likelihood by the model.
The generated image &, being unrealistic but assigned a high-likelihood, resembles an adversarial example, i.e., an input that causes the model to make mistakes. Recent works (Cohen et al., 2019) in adversarial defense have shown that random noise can be used improve the model's robustness to adversarial perturbations are generated to fool the model. We hypothesize that such approach can also be applied to improve an autoregressive modeling process by making the model less vulnerable to compounding errors occurred during density estimation. Inspired by the success of randomized smoothing in adversarial defense (Cohen al., 2019), propose apply smoothing to autoregressive modeling to address the problems mentioned above.
process during which adversarial examples that are close to actual data
3 GENERATIVE MODELS WITH DISTRIBUTION SMOOTHING
In the following, propose decompose density estimation task into smoothed data modeling problem followed by an inverse smoothing problem where we recover the true data density from the smoothed one.
3.1 RANDOMIZED SMOOTHING PROCESS
Unlike Cohen et al. (2019) where randomized smoothing is applied to model, we apply smoothing directly to the data distribution (x). To do this, we introduce smoothing distribution (x a distribution that is symmetric and stationary (e.g. a Gaussian or Laplacian kernel) - and convolve
with Pamuxx (x) to obtain a new distribution 9xx) & f "(x (())amxx)))a) When g(x x) is a normal distribution, this convolution process is equivalent perturbing the data distribution with Gaussian
3

---

Published as a conference paper at ICLR 2021
Data Model
0.25
Noisy data
1.0
Model
0.6
0.2
0.05
-2
2
10
14
-2
2
6
10
14
(a) Data
(b) Smoothed data
Figure 3: Visualization of a 1-d data distribution without smoothing (a) or with smoothing (b),
modeled by the same mixture of logistics model.
noise, which, intuitively, will make the data distribution smoother. In the following, formally prove that convolving a 1-d distribution Paaa(a) with a suitable noise can indeed "smooth" Paua (x).
Theorem 1. Given continuous and bounded 1-d distribution (z) that is supported R, for any 1-d distribution gā](z) that symmetric (i.e. q(ā)z) = ((z(²)). stationary (i.e. transla- tion invariant) and satisfies lim (1)aa())qqa8 = for any given 3. we have Láp(g(ö)) S Lip(páau (2)), where q(ī) & J'q(ā(x)) (2) da and Lip(-) denotes the Lipschitz constant of the given 1-d function.
Theorem 1 shows that convolving 1-d data distribution (x) with suitable noise distribution q() 1r) (e.g. (@ o²)) can reduce the Lipschitzness (i.e. increase the smoothness) Paau (z). We provide the proof of Theorem in Appendix A.
Given Pamna (x) with a high Lipschitz constant, we empirically verify that density estimation becomes easier task the smoothed distribution q(x) than directly Pdam To see this, visualize a 1-d example in figure 3a, where we want to model a ten-mode data distribution with a mixture of logistics model. If model has three logistic components, there is almost no way for the model, which only has three modes, to perfectly fit this data distribution, which has ten separate modes with sharp transitions. The model, after training (see figure 3a), mistakenly assigns much higher density to the low density regions between nearby modes. If we convolve the data distribution with ((ā(x) (ā 0.5"), the new distribution becomes smoother (see figure 3b) and can be captured reasonably well by the same mixture of logistics model with only three modes (see figure 3b). Comparing the same model's performance on the two density estimation tasks, we can see that the model is doing better job at modeling the smoothed version of the data distribution than the original data distribution, which has high Lipschitz constant.
This smoothing process also be understood regularization term for the original maximum likelihood objective (on the un-smoothed data distribution), encouraging the learned model to be smooth, as formalized by the following statement:
Proposition (Informal). Assume that the symmetric and stationary smoothing distribution q(x x) has small variance and negligible higher order moments, then
82 log Pa
E Exaa (x) (x1x) (log Po (xx) RJ E (x)
logpo(x) + 7
Ox?
for some constant y).
Proposition shows that our smoothing process provides regularization effect on the original objective (xx) when no noise is added, where the regularization aims to maximize
EE courages the second order gradients computed at a data point x to become closer to zero (if it were positive then x will not be a local maximum), creating a smoothing effect. This extra term is also the trace of the score function (up multiplicative constant) that can be found in the score matching objective (Hyvärinen, 2005), which is closely related to many denoising methods (Vincent, 201 I; Hyvärinen, 2008). This regularization effect can, intuitively, increase the generalization capability of the model. fact. it has been demonstrated empirically that training with noise can lead to im- provements in network generalization (Sietsma & Dow, 1991; Bishop, 1995). Our argument is also
0² lo Since samples from Pdata should be close to local maximum the model, this en- oz?
4

---

Published as a conference paper at ICLR 2021
similar that used in (Bishop, 1995) except that we consider more general generative modeling case as opposed to supervised learning with squared error. We provide the formal statement and proof of Proposition 1 in Appendix A.
3.2 AUTOREGRESSIVE DISTRIBUTION SMOOTHING MODELS
Motivated by the previous 1-d example, instead of directly modeling (x)m(X), which can have a high Lipschitz constant, we propose to first train autoregressive model on the smoothed version of the data distribution (()). Although the smoothing process makes the distribution easier to learn, also introduces bias. Thus, we need an extra step to debias the learned distribution by reverting the smoothing process.
If our goal is generate approximate samples for Pdata (x), when (({xx) = (x|xx, o² and is small, we can use the gradient of pa (x) for denoising (Alain & Bengio, 2014). More specifically, given smoothed samples x from po (x), we can "denoise" samples via:
(2) which only requires the knowledge of pe (x) and the ability to sample from it. However, this ap- proach does not provide likelihood estimate and Eq. (2) only works when (x{xx) is Gaussian (though alternative denoising updates for other smoothing processes could be derived under the Em- pirical Bayes framework (Raphan & Simoncelli, 201 I)). Although Eq. (2) could provide reasonable denoising results when the smoothing distribution has a small variance, obtained in this way is only point estimation of x E[x x] and does not capture the uncertainty of xx)
x x x log (x)),
To invert more general smoothing distributions (beyond Gaussians) and to obtain likelihood es- timations, we introduce a second autoregressive model (x)). The parameterized joint den- sity pa (x, x) can then be computed as pe (x, x) = pe (x) To obtain our approximation the joint distribution px(x, x) to obtain pe (x) Po (x, x) dS, which is in general intractable. However, we can easily obtain an evidence lower bound (ELBO):
of Pdna (x), we need to integrate over
log po (x) 2 (x [log po (x)) (x [log (x((x)) + Bugctee)
Note that when (((xx) is fixed, the entropy term Eg(x x) [log (xxxx) is a constant with respect the optimization parameters. Maximizing ELBO on ( (x) is then equivalent to maximizing:
(() = Epau(x) (lg + (x
[log ( pe (x{
(4)
From equation 4, we can that optimizing the two models (x and px(x(x) separately via maximum likelihood estimation is equivalent to optimizing J(0).
3.3 TRADEOFF IN MODELING
In general, there is trade-off between the difficulty of modeling Px( and (x(x To see this, let us consider two extreme cases for the variance of q(x when (xx[x) has a zero variance and an infinite variance. When q(x(x) has zero variance, ( xx]xx is a distribution with all its probability mass at x, meaning that no noise is added to the data distribution. In this case, modeling the smoothed distribution would be equivalent to modeling Panta (x). which can be hard as discussed above. The reverse smoothing process, however, would be easy since (x(x(x) can simply be an identity map perfectly invert the smoothing process. In the second case when ((xxx) has an infinite variance, modeling p(x) would be easy because all the information about the original data lost, and p(x) would be close the smoothing distribution. Modeling pxx(x). on the other hand, is equivalent to directly modeling Paam (x), which can be challenging.
Thus, the key here is to appropriately choose smoothing level so that both ((x) and p(x x) can be approximated relatively well by existing autoregressive models. In general, the optimal variance might be hard to find. Although one can train (x{[x) by jointly optimizing ELBO, in practice, we find this approach often assigns very large variance to q({[x), which can trade-off sample quality for better likelihoods on high dimensional image datasets. We find empirically that a pre-specified (({/x) chosen by heuristics (Saremi & Hyvarinen, 2019; Garreau et al., 2017) is able to generate much better samples than training (x(x) via ELBO. In this paper, we will focus on the sample quality and leave the training of (X{(x) for future work.
5

---

Published as a conference paper at ICLR 2021
Data
Baseline(2) Baseline(B)
Ours
-0.5
Lablace
1.2
ELDO
0.8
-1.
0.0
0.2 Variance
Figure 4: Density estimation on 1-d synthetic dataset. In the second figure, the digit in the paren- thesis denotes the number of mixture components used in the baseline mixture of logistics model. In comparison, our model the third figure uses only 2 mixture of logistics components for each univariate conditional distribution.
Table 1: Negative log-likelihoods on 2-d synthetic datasets (lower is better). We compare with MADE (Germain et al., 2015), RealNVP (Dinh et al., 2016), CIF-ReaINVP (Cornish et al., 2019).
Dataset
RealNVP
CeF-ReeINVP
MADE (3 mixtures)
MADE (6 mixtures)
Ours (3 mixtures)
Rings 2.81 1.80
2.81
3.26
2.81
2.71
Olympics
1.74
1.27
0.80
0.80
4 EXPERIMENTS
In this section, we demonstrate empirically that by appropriately choosing the smoothness level of randomized smoothing, our approach is able drastically improve the sample quality of exist- ing autoregressive models on several synthetic and real-world datasets while retaining competitive likelihoods on synthetic datasets. We also present results on image inpainting in Appendix C.2.
4.1 CHOOSING THE SMOOTHING DISTRIBUTION
To help us build insights into the selection of the smoothing distribution g x x), we first focus on a I-d multi molti-modal distribution (see figure leftmost panel). We use model-based methods to invert the smoothed distribution and provide analysis on "single-step denoising" in Appendix B.I. We start with the exploration of three different types of smoothing distributions Gaussian distribution, Laplace distribution, and uniform distribution. For each type of distribution, we perform grid search to find the optimal variance. Since our approach requires the modeling of both po(X) and po(x|x), stack and x together, and MADE model (Germain al., 2015) with a mixture of two logistic components to parameterize (x) and (=((x*] at the same time. For the baseline model. we train a mixture logistics model directly on Pdaaa (x). We compare the results in the middle two panels in figure 4.
We find that although the baseline with eight logistic components has the capacity to perfectly model the multi-modal data distribution, which has six modes, the baseline model still fails to do so. We believe this can be caused by optimization or initialization issues for modeling a distribution with a high Lipschitz constant. Our method, on the other hand, demonstrates more robustness by suc- cessfully modeling the different modes in the data distribution even when using only two mixture components for both (x(x) and (x))x)).
For all the three types of smoothing distributions, we observe a reverse U-shape correlation between the variance of q(x x) and ELBO values- with ELBO first increasing the variance increases and then decreasing as the variance grows beyond certain point. The results match our discussion on the off between modeling (x) and (x(x) in Section 3.3. We notice from the empirical results that Gaussian smoothing is able to obtain better ELBO than the other two distributions. Thus, will use (x([x) = (x|xx, x² I² for the later experiments.
4.2 2-D SYNTHETIC DATASETS In this section, we consider two challenging 2-d multi-modde modal synthetic datasets (see figure 5). We focus model-based denoising methods and present discussion "single-step denoising" in Ap- pendix B.2. We use a MADE model with comparable number of total parameters for both the baseline and our approach. For the baseline, we train the MADE model directly the data. For

---

Published as a conference paper at ICLR 2021
(a) Rings
(b) MADE (6)
Ours (3)
(d) Olympics
(e) MADE (6)
(1) Ours (3)
Figure 5: Samples on 2-d synthetic datasets. We use a MADE model with comparable number of parameters for both our method and the baseline. Our model uses mixture of logistics, while the baseline uses 6 (more) mixture of logistics.
7
5
5
4
8
5
7
3
5
4
8
5
7
3
5
4
8
5
3 3
7
5 S7
5 6 7 1
3
7
8
3
2 6
3
7
8
3
2
6
3
7
8
3
2 1
6
5 4/
6 3 b 6
2 4 6 6 %
σ 8 6 6
7
7
3
1 6 6 7 6 8
7
7
773
3
1
7
7
3
7
. 0
8
4
3
6
9
8
4
3
6
6
9
8
4
3
6
6
9
4 60 C
8
3
3
9
8 3
3
9
6
7
8 3
3
9
6 7
2
6
7
6
8
/
9
6
8
6
8
/
9
6
8
6
P
4
Column
Column
Column
Column 4
Figure 6: From left to right: Column 1: samples from (x).. Column 2: "single-step denoising"
samples from pa (x)).. Column 3: samples from po (x) Column 4: samples from the baseline
PixeICNN+ model. Samples in Column 2 ("single-step denoising") contain wild pixels and
less realistic compared to samples in Column 3 (modeled by another PixeICN+++). None of the samples are conditioned on class labels.
our randomized smoothing model, we choose (x{xx) = (x 0.331) to be the smoothing distri- bution. We observe that with this randomized smoothing approach, our model is able to generate better samples than the baseline (according to a human observer) even when using less logistic com- ponents (see figure 5). We provide more analysis the model's performance in Appendix B.2. We also provide the negative log- liglliikls in Tab.
4.3 IMAGE EXPERIMENTS
In this section, we focus on three common image datasets, namely MNIST, CIFAR-10 (Krizhevsky al., 2009) and CelebA (Liu et al., 2015). We seleet xxxx) ( 0²I) I) to be the smoothing distribution. We use PixeICNN+ (Salimans et al., 2017) as the model architecture for both (x) and (x(x)) We provide more details about settings in Appendix C.

---

Published as a conference paper at ICLR 2021
Image generation. For image datasets, select the of q(x[x) (X o²1) according to analysis in (Saremi & Hyvarinen, 2019) (see Appendix C for more details). Since ((x[x) is a Gaussian distribution, we can apply "single-step denoising"
reverse the smoothing process for samples drawn from po()). In this case, the model po( [ is not required for sampling since the be used to denoise samples (also from po(X)) (see equation 2). We present smoothed samples from px(X). reversed smoothing samples processed by "single-step denoising" and processed by pe (x|x) in figure 6. For comparison, we also present samples from a +ixeICNN+ with parameters comparable to the sum of total parameters of (x( and px(x We find that by using this randomized smoothing approach, we are able to drastically improve the sample quality of PixeICNN++ (see the rightmost panel in figure 6). We note that with only po( a PixelCNN++ optimized on the smoothed data, we already obtain more realistic samples compared the original PixelCNN+ method. However, Po (x(x)x) ( is needed to compute the likelihood lower bounds. We report the sample quality evaluated by Fenchel Inception Distance (FID (Heusel et al., 2017)), Kernel Inception Distance (KID (Birikowski et al., 2018)), and Inception scores (Salimans al., 2016) in Tab. 2. Although our method obtains better samples compared to the original PixeICNN++, model has worse likelihoods evaluated in BPDs. We believe this is because likelihood and sample quality are not always directly correlated as discussed in Theis et al. (2015). We also tried training the variance for (xxxx) by jointly optimizing ELBO. Although training the variance can produce better likelihoods. it does not generate samples with comparable quality as our method (i.e. choosing variance by heuristies). Thus, it is hard conclusively determine what is the best way of choosing (x)x). We provide more image samples in Appendix C.4 and nearest neighbors analysis in Appendix C.5.
gradient of pP)
Model
Inception
FID
KID +
BPD +
PixeICNN (Oord et al., 2016b)
4.60
65.93
3.14
PixellQn (Ostrovski et al., 2018)
5.29
49.46
PixeICNN+ (Salimans et al., 2017)
5.30
54.25
0.046
2.92
EBM (Du & Mordatch, 2019)
6.02
40.58
i-ResNet (Behrmann et al.. 2019)
65.01
3.45
MADE (Germain et al., 2015)
5.67
Glow (Kingma Dhariwal, 2018)
46.90
3.35
Single-step (Ours)
7.50 + .08
57.53
0.052
Two-step (Ours)
7.84 + .07
29.83
0.022
3.53
Table 2: Inception, FID and KID scores for unconditional CIFAR-10. "Single-step" samples are generated solely by (x). "Two-step" samples are generated by sampling from po(x) and then denoised by pe (x(x).. Although samples from "single-step" might appear visually similar to samples from the "two-step" method, there still a gap between their Inception, FID and KID scores.
ADDITIONAL EXPERIMENTS ON NORMALIZING FLOWS
In this section, we demonstrate empirically on 2-d synthetic datasets that randomized smoothing techniques can also be applied to improve the sample quality of normalizing flow models (Rezende & Mohamed, 2015). We focus on RealNVP (Dinh et al., 2016). We compare the RealNVP model trained with randomized smoothing, where we use po (x|x) (also RealNVP) revert the smoothing process, with a RealNVP trained with the original method but with comparable number of parame- ters. We observe that smoothing is able improve sample quality on the datasets we consider (see figure 7) while also obtaining competitive likelihoods. On the checkerboard dataset, our method has negative log-likellng ligeliiksssods 3.64 while the original RealNVP has 3.72; on the Olympics dataset, our method has negative log-likelihoods 1.32 while the original RealNVP has 1.80. This example demonstrates that randomized smoothing techniques can also be applied to normalizing flow models.
6 RELATED WORK
Our approach shares some similarities with denoising autoencoders (DAE, Vincent et al. (2008)) which recovers clean observation from corrupted one. However, unlike DAE which has train-
8

---

Published as a conference paper at ICLR 2021
(a) Data
(b) RealNVP
(c) Ours
(d) Data
(e) RealNVP
(f) Ours
Figure 7: RealNVP samples on 2-d synthetic datasets. The RealNVP model trained with randomized
smoothing is able to generate better samples according to human observers.
able encoder and fixed prior distribution, our approach fixes the encoder and models the prior using an autoregressive model. Generative stochastic networks (GSN, Bengio et al. (2014)) use DAEs to train a Markov chain whose equilibrium distribution matches the data distribution. However, GSN needs to start the chain from a sample that is very close to the training distribution. Denoising diffusion model (Sohl-Dickstein et al., 2015; Ho et al., 2020) and NCSN (Song & Ermon (2019; 2020)) address the issue of GSNs by considering sequence of distributions corresponding to data corrupted with various noise levels. By setting multiple noise levels that are close to each other, the sample from the previous level can serve next level. This way. the model can start from a distribution that is easy to model and gradually move the desired distribution. However, due the large number of noise levels, such approaches require many steps for the chain to converge to the right data distribution.
proper initialization for the sampling process the
In this paper, we instead propose use only one level of smoothing by modeling each step with a powerful autoregressive model instead of deterministic autoencoders. Motivated by the success of "randomized smoothing" techniques in adversarial defense (Cohen al., 2019), we perform randomized smoothing directly on the data distribution. Unlike denoising score matching (Vincent, 2011), a technique closely related to denoising diffusion models and NCSN, which requires the perturbed noise to be a Gaussian distribution, we are able to work with different noise distributions.
Our smoothing method is also relevant to "dequantization" approaches that are common in normal- izing flow models, where the discrete data distribution is converted to a continuous one by adding continuous noise (Uria et al., 2013; Ho et al., 2019). However the added noise for "dequantization" in flows is often indistinguishable to human eyes, and the reverse "dequantization" process is of- ten ignored. In contrast, we consider noise scales that are significantly larger and thus a denoising process is required.
Our method is also related to "quantization" approaches which reduce the number of "significant" bits that are modeled by generative model (Kingma & Dhariwal, 2018 8; Menick & Kalchbrenner, 2018). For instance, Glow (Kingma & Dhariwal, 2018) only models the 5 most significant bits of an image, which improves the visual quality of samples but decreases color fidelity. SPN (Menick & Kalchbrenner, 2018) introduces another network to predict the remaining bits conditioned on the 3 most significant bits already modeled. Modeling the most significant bits can be understood as capturing a data distribution perturbed by bit-wise correlated noise, similar to modeling smoothed data in our method. Modeling the remaining bits conditioned on the most significant ones in SPN is then similar to denoising. However, unlike these quantization approaches which process an image at the "significant" bits level, we apply continuous data independent Gaussian noise to the entire image with different motivation smooth the data density function.
7 DISCUSSION
In this paper, we propose to incorporate randomized smoothing techniques into autoregressive mod- eling. By choosing the smoothness level appropriately. this seemingly simple approach is able to drastically improve the sample quality of existing autoregressive models on several synthetic and real-world datasets while retaining reasonable likelihoods. Our work provides insights into how recent adversarial defense techniques can be leveraged to building more robust generative models. Since apply randomized smoothing technique directly to the target data distribution other than the model, we believe our approach is also applicable to other generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs).

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
The authors would like to thank Kristy Choi for reviewing the draft of the paper. This research was supported by NSF (#1651565, #1522054, #1733686), ONR (N00014.19-1-2145), AFOSR (FA9550-19-1-00024) ARO, and Amazon AWS.
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1):3563-3593 -5933 2014.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. International Conference on Machine Learning, pp- 573-582, 2019.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6)11373-139, 2003.
Yoshua Bengio, Eric Laufer, Guillaume Alain, and Jason Yosinski. Deep generative stochastic networks trainable by backprop. In International Conference on Machine Learning, pp. 226-234, 2014.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šmdićć Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Mikotaj Binkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint 1eXi.01801.0140 00101, 2018.
Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108-116, 1995.
Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint 1902:0099020299 2019.
Rob Cornish, Anthony L Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. arXiv preprint arXiv: /3833, 2019.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint
2016.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.08m6 2019.
Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median heuristic. arXiv preprint arXiv: 2017.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, pp. 881-889, 2015.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. arXiv preprint arXiv:/706. 1706.08000, 2017.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. arXiv preprint 10X000:902.00225.. 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. / 2020.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695-709, 2005.
10

---

Published as a conference paper at ICLR 2021
Aapo Hyvärinen. Optimal approximation of signal priors. Neural Computation, 20(12):3087-3110, 2008.
Matti Kääriäinen. Lower bounds for reductions. In Atomic Learning Workshop. 2006.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In
Advances in neural information processing systems, pp. 10215-10224, 2018.
Alex Krizhevsky et Learning multiple layers of features from tiny images. 2009.
Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In Advances in neural information processing systems, pp. 4601-4609, 2016.
Hugo Larochelle and lain Murray, The neural autoregressive distribution estimator. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 29-37. JMLR Workshop and Conference Proceedings, 2011.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pp. 3730-3738, 2015.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint arXiv:1812.01608, 2018.
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. In Advances in Neural Information Processing Systems, pp- 10771-10780, 2018
Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis. In Advances in neural information processing systems, pp. 1786-1794, 2010.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint 1609v 2016a.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. 2016b.
arXiv preprint
Georg Ostrovski, Will Dabney, and Rémi Munos. Autoregressive quantile networks for generative modeling. arXiv preprint arXiv:1806.05577 2018.
Martin Raphan and Eero P Simoncelli. Least squares estimation without priors or supervision. Neurul computation, 23(2):374-420, 2011.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.0577 2015.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embed- 2000.
ding. science, 290(5500):
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016.
Improved techniques for training gans. arXiv preprint
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.055/77 2017.
Saeed Saremi and Aapo Hyvarinen. Neural empirical bayes. Journal of Machine Learning Research, 20:1-23, 2019,
Jocelyn Sietsma and Robert JF Dow. Creating artificial neural networks that generalize. Neural networks, 4(1):67-79, 1991.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. arXiv preprint arXiv: 1503 035503585 2015.

---

Published as a conference paper at ICLR 2021
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems, pp. 11918-1 11930, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv preprini arXiv:2006.0900 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:/312.6199 2013.
Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500) 2000.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. arXiv preprini arXiv: /51 511.01844 2015.
Benigno Uria, lain Murray, and Hugo Larochelle. Rnade: The real-valued neural autoregressive density-estimator. In Advances in Neural Information Processing Systems, pp. 2175-215520 2183, 2013.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compu- 2011.
tation,
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czareekk, Michaël Mathicu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-33555 2019.
12

---

Published as a conference paper at ICLR 2021
A PROOFS
Theorem 1, Given a contimuous and bounded I-d distribution Pauu( that is supported on IR for any I-d distribution g((2z) that is symmetric (i.e. q(ā(xz) ((x(,)), stationary (i.e. transla- tion invariant) and satisfies lima--oo Patta ( ( = for anyy given Σ, we have Lip(g(T)) S Lip(Paau (2)). where g(z) _ (((()))) (2) dz and Lip(-) denotes the Lipschitz constant of the given 1-d function.
Proof. First, we have that:
Vap(<>) = [p(x) v, logp(x)) S Lip(p)
(5)
and if we assume symmetry, i.e. (x(()) - (( then by integration by parts we have:
Vaq( i) (a9(()) pp)()) [V
Ep( (g)p))((((())) v [og)(
(6)
Therefore,
Lip(g) = max - E p( V logp(x))
max
g( 2 (p))))))))))))))))))))) V ((z)
(7)
S max
((pp\)) Lip(p) max
[(+|)) = Lip(p)
(8)
which proves the result.
Proposition 1 (Formal). Given a D-dimensional data distribution Pdoto (x) and model distribution (x),,, assume that the smoothing distribution q(x(x) satisfies:
log Pa is infinitely differentiable on the support of pe (x)
xx x) is symmetric (i.e. q({xx) (x(x))
x x x) is stationary (i.e. translation invariant)
x x) is bounded and fully supported on Rº
xx x) is element-wise independent
1xx8/x) [(x x)2] is bounded, and T ( xz,²] rj each dimension i.
Denote = x then
2 log
Epsnut)) (x((/x)) [log (x() (x) logpo(x) + Σ
+
Ouy
where o( RDD R is function of such that lime
o(67) c²
0.
Thus when
f ooE" (x)att(x)p))dddde O, we have
E (x) (x(x))) (X)] + Epsn(x) logpo(x) + Σ
8 log po
(2)
Oz?
Proof. To see this, we first note that the new training objective for the smoothed data distribution is (xp (x(x/x (o))). Let e & x, because of the assumptions have, the PDF function (x(xx) can be reparameterized as p(e) which satisfies: p is bounded and fully supported on R. P p
plement-wisee independent and Ep(c) [ef) Ep(c.) [=?] = 17 at each dimension (2 1,..., D). Then we have
/
Expna()) By(()x) Pog (x)
log po e) (x) (x)p(e)
Using Taylor expansion, we have:
alog pa
8 log EiEj dx,Ox,
lo E) log (og))(x) Σ
Σ
o(E2).
E
Dx:
in
13

---

Published as a conference paper at ICLR 2021
Since is independent of and
000
e,de; O,
Exedde,de, = di,
where dúg is the Kronecker delta function, the right hand side of Equation 9 becomes
E pasn (lgglogggg [oo)) + /
0 log po
+ +o(e) Pina (x)((()p))
(10)
dx;
18 (x))
logpo(x) "
Σ
82 log po dx?
+ + o(E ((e)))
(11)
2
When
ol o(E)) (x)p(e)dxde + O, /
(12)
we have
(oocsto)loogr (loooop(()) Epen(x)) Eygato)
+ E Epenix))
log (x) + 77
Σ
0 log Po
(13)
Ox²
where the second term on the right hand side serves as regularization for the original objective pan (xc) [log pe (x)].
B DENOISING EXPERIMENTS
B.I ANALYSIS ON 1-D DENOISING
To provide more insights into denoising, we first study "single-step denoising" (see equation 2) on 1-d dataset. We choose the data distribution to be two mixture of Gaussian distribution 0-5. 0.12) + 0.5. 0.13) and the smoothing distribution to be ((([z) = N NN [,]x,, 0.3?) (see figure 8a). Since the convolution of two Gaussian distributions is also Gaussian distribu- tion, the smoothed data is a mixture of Gaussian distribution given by 0.5.(-0..3, 0.10 - 0.3²) + (0.3 0.12 + 0.3²). The ground truth of V log p(ø) can then be calculated in closed form. Thus given the smoothed data I, we can calculate the ground truth V p(ī) ) in equation 2 and obtain using "single-step denoising". We visualize the denoising results figure 8b. We find that the low density region between the two modes in Paata (z) are not modeled properly in figure 8b. However, this very expected since "single-step denosing" uses F E[a] [ the denoised result. When the smoothing distribution has a large variance (like in figure 8a where the smoothed data has merged into a one mode distribution), datapoints like To the middle low density region of (x) other points in the middle low density region of Pdaaa()) can come from both modes of Pdata with high probability before the smoothing process (see figure 11a), the denoised J = E[r = 2n) can still be located in the middle low density region (see figure 8b). Since a large proportion of the smoothed data is located in the middle low density region of Pasta (z), we would expect certain pro- portion of the density to remain in the low density region after "single-step denoising" just as shown in figure 8b. However, when the smoothing distribution has smaller variance, "single-step denois- ing" can achieve much better denoising results (see figure 9, where we use q( (())x)= = -M 0.1²)), Although denoising can be easier when the smoothing distribution has a smaller variance, modeling the smoothed distribution could be harder as we discussed before.
as the substitute for
have high density in the smoothed distribution. Since Tp. well as
In general, the right denoising results should be samples coming from p(=(ū), which is the reason
why samples from Po(a (zr(ā) (i.e. introducing the model po po(x(E)) is more ideal than using Eg [a]]]]] as a denoising substitute (i.e. "single-step denoising"). In general, the capacity of the denoising model p=(x(() also matters in terms of denoising results. Let us again consider the datapoint To shown in figure 8a. If the invert smoothing model po (ō) is a one mode logistic distribution, due to the mode covering property of maximum likelihood estimation, given the smoothed observation ão. the best the model can do is to center its only mode at ã0 for approximating p(x(ā [ ăn) (see figure 10c). Thus, like Zo, the smoothed datapoints the low density region between the modes of paat(
14

---

Published as a conference paper at ICLR 2021
still likely to remain between the two modes after denoising (see figure 10b). To solve this issue, we can increase the capacity of pa(x () by making a two mixture of logistics. In this case, the distribution Po Po (z| æo) can be captured in better way (see figure 1lc and figure 1 Ia). After the invert smoothing process, like To. most smoothed datapoints in the low density can be mapped to one of the two high density modes (see figure 11b), resulting in much better denoising effects.
Data
Data
Denoised
Smoothed data
TO
Data distribution (g(# (ā 0.3*)).
(b) Single-step denoising results.
Figure 8: 1-d single-step (gradient based) denoising.
Data
Data Smoothed data
Denoised
(a) Data distribution (g(ā)=) = .V (a ).13)).
(b) Single-step denoising results.
Figure 9: 1-d single-step (gradient based) denoising.
Data Denoised
Data Smoothed data
Data pa(x(&
TD
(a) Data distribution.
(b) Denoised with (z)(a))
(c) Distribution of po #o).
Figure 10: Denoising with pa (x) (x((). which is modeled by one mixture of logistics.
15

---

Published as a conference paper at ICLR 2021
(a) Data
(b) MADE (n = 5) (c) MADE (n = 6) (d) MADE (n = 7)
(e) Ours (n 2) (f) Ours (n 3)
Ours (n 4)
Figure 12: Samples on 2-d synthetic datasets. We use a MADE model with comparable number of parameters for both our method and the baseline. The models have mixture of logistics for each dimension. Our method is able to obtain reasonable samples when using fewer mixture components, while the baseline still has trouble modeling the two sides of the rings when n = 7.
Data
p(x|Ro]
Denoised
Data Pa(x|Rg)
To
(a) Ground truth p(a|ā To). (b) Denoising with (=) (c) Distribution of po = #o)-
Figure 11: Denoising with pa (x ž), which is modeled by two mixtures of logistics.
B.2 ANALYSIS ON 2-D DENOISING
On the 2-d Olympics dataset in section 4.2, find that the intersections between rings be poorly modeled with the proposed smoothing approach when only two mixture of logistics are used (see figure 12e). We believe this be caused if the denoising model is not flexible enough to capture the distribution p(x x). More specifically, we note that the ground truth distribution for p(x[*) at the intersections of the rings model which only has two mixtures of logistics for each dimension. If we increase the flexibility of Po (x x) by using three or four mixtures of logistics components (note that we still use fewer mixture components than the MADE baseline and we use comparable number of parameters), the intersection of the rings can modeled in improved way (see figure 12).
highly complicated distribution and can be hard capture using our
We also provide "single-step denoising" results for the experiments in Section 4.2 (see figure 13), where we use the same smoothing distribution, and the MADE model with three mixture compo- nents used in section 4.2. We note that "single-step denoising" results are not very good, which also expected. As discussed in section B.I, when the smoothing distribution has a relatively large variance, Eg [x|] is not a good approximation for the denoised result, and we want the de- noised sample to come from the distribution pe (x (x), in which case introducing a denoising model (x(x( could be a better option. Although could select (xxxx) have smaller variance that "single-step denoing" could work reasonably well, but modeling p(x) in this case could be more challenging.
16

---

Published as a conference paper at ICLR 2021
(a) Data
(b) Smoothed data distribution
(c) Single-step denoising results
(d) Data
(c) Smoothed data distribution
(f) Single-step denoising results
Figure 13: Single-step denoising results 2-d synthetic datasets. We the same MADE model with three mixture components and the same smoothing distribution as mentioned in Section 4.2.
C IMAGE EXPERIMENTS
C.I SETTINGS
For the image experiments, we first rescale images to [-1, and then perturb the images with q(x[x) [xxxx, I?I). We use 0.5 for MNIST and σ 0.3 for both CIFAR-10 and CelebA. The selection of is mainly based on analysis in (Saremi & Hyvarinen, 2019). More specifically, given an image, we consider the median value of the Euclidean distance between two data points a dataset, and then divide it by DvD, where D is the dimension of the data. This provides us with way of selecting the variance of q({)x), when ((((x) is Gaussian distribution. We find this selection of variance able to generate reasonably well samples practice. We train all the models with Adam optimizer with learning rate 0.0002. To model pe (x stack X and together at the second dimension to obtain = [x, x). which ensures that x comes before x in the pixel ordering. For instance, this stacking would provide an image & with size on a MNIST image, and an image with size 3 x (2 x 32) x 32 on CIFAR-10 10 image. Since PixeICNN+ consists of convolutional layers, can directly feed into the default architecture without modifying the model architecture. As the latter pixels of the input only depend on the previous pixels in an autoregressive model and comes before x, can parameterize po (x(xxx) by computing the likelihoods only on using the outputs from the autoregressive model.
(2 28) x 28
C.2 IMAGE INPAINTING
Since both po(Xi and po (x(x)) parameterized by autoregressive model, also perform image inpainting using our method. We present the inpainting results on CIFAR-10 10 in figure 14a and CelebA in figure 14b, where the bottom half of the input image is being inpainted.
17

---

Published as a conference paper at ICLR 2021
Original
Original
Masked
Masked
Inpainted
Inpainted
(a) CIFARIO inpainting
(b) CelchA inpainting
Figure 14: Inpainting results from our two-step method. The bottom half of the images are masked for inpainting.
C.3 IMAGE DENOISING
We notice that the reverse smoothing process can also be understood as denoising process. Besides the "single-step denoising" approach shown above,
also apply (x X) to denoise images. To visualize the denoising performance, we sample Xest from the test set and perturb Xwst with (xx(x) obtain noisy sample X We feed X into px Xiea) and draw samples from the model. We visualize the results in figure 15. As we can see, the model exhibits reasonable denoising results, which shows that the autoregressive model capable of learning the data distribution when conditioned on the smoothed data.
7
2
0
4
7 2
0
4
4
5
9
0
6
4
9
6
9
0 9
6
9
0
7
9
0
5 7
(a) Noisy MNIST
(b) MNIST denoising
(c) Noisy CIFARIO
(d) CIFARIO denoising
Figure 15: Denoising with (x(
18

---

Published as a conference paper at ICLR 2021
C.4 MORE SAMPLES
Figure 16: CIFAR-10 samples from pa px( (unconditioned on class labels).
19

---

Published as a conference paper at ICLR 2021
Figure 17: CIFAR-10 samples from (x (unconditioned on class labels).
20

---

Published as a conference paper at ICLR 2021
Figure 18: CIFAR-10 samples from the original PixeICNN+ method (unconditioned on class la- bels).
21

---

Published as a conference paper at ICLR 2021
Figure 19: CelebA samples from px( (unconditioned on class labels).
22

---

Published as a conference paper at ICLR 2021
Figure 20: CelebA samples from po (x(x) (unconditioned on class labels).
23

---

Published as a conference paper at ICLR 2021
Figure 21: CelebA samples from the original PixelCN++ method (unconditioned on class labels).
C.5 NEAREST NEIGHBORS
24

---

Published as a conference paper at ICLR 2021
Figure 22: Nearest neighbors measured by the l distance between images. Images on the left of the red vertical line are samples from our model. Images on the right are nearest neighbors in the training dataset.
25

---

Published as a conference paper at ICLR 2021
Figure 23: Nearest neighbors measured by the lg distance in the feature space of an Inception V3 network pretrained on ImageNet. Images on the left of the red vertical line are samples from our model. Images on the right are nearest neighbors in the training dataset.
26

---

Published as a conference paper at ICLR 2021
Figure 24: Nearest neighbors measured by the &a distance between images. Images on the left of the red vertical line are samples from our model. Images on the right are nearest neighbors in the training dataset.
27

---

Published as a conference paper at ICLR 2021
Figure 25: Nearest neighbors measured by the lg distance the feature space of an Inception V3 network pretrained on ImageNet. Images on the left of the red vertical line are samples from our model. Images on the right are nearest neighbors in the training dataset.
28

---

Published as a conference paper at ICLR 2021
C.6 ABLATION STUDIES
In this section, we show that gradient- based "single-step denoising" will not improve sample quali- ties without performing randomized smoothing. To see this, draw samples from PixeICNN++ (x(x) trained directly on Pama(X) (i.e. without smoothing). We perform "single-step denoising" update defined
x (x))..
(14)
We explore various values for o, and report the results in figure 26. This shows that "single-step denoising" alone (without randomized smoothing) will not improve sample quality of PixeICNN++.
(a) σ = 0 (Original)
(b) 0.01
(c) a = 0.05
(d) a = 0.1
Figure 26: "Single-step denoising" on PixeICNN+ trained on un-smoothed data. corre- sponds to the original samples.
29


Reviews:
[{"review_id": "ysYRKsGUOCA", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary of the paper :\nThe authors propose to improve the sample quality of autoregressive models. The authors propose to (1) - smooth the input data distribution leveraging methods that have shown success in adversarial defense, (2) recover input distribution by learning to reverse the smoothing process. The authors first demonstrate the efficiency of their method on 1d toy-problem, and extend the demonstration to more complex datasets such as MNIST, CIFAR-10 and CelebA with application such as image generation, inpaintting and denoising.\n\n#### Pros :\n* The idea to leverage a method previously used for adversarial defense to density estimation is interesting and novel.\n* The paper is well motivated through the manifold hypothesis approximation (which results in densities with high Lipschitz constants) and compounding errors.\n* The theory is strong \n\n#### Cons :\n* The experiments on denoising and inpainting are only qualitative and suffer from a lack of quantitative evaluation.\n\n#### Recommandation :\nThe article is clear, well motivated, and have a strong theoretical grounding. Therefore I would tend to accept the article.\n\n#### Detailed comments :\n\n* The experiment on 2d synthetic datasets (especially the olympic dataset) should be discussed more thoroughly. First, it is not clear that the proposed model is generating better sample than the MADE baseline on this specific dataset. Second, the intersection between rings, in the olympic dataset, seems to be much poorly modeled with the proposed approach compared to the MADE baseline. What is the reason ?\n \n* In the section 3.2 the authors are introducing 2 different debiasing methods (either a denoising step or another autoregressive model). In the rest of the article it is not clear which of the two methods the authors are using. In addition, in the 2d toy-problem (i.e. ring and olympic) as the authors are choosing a gaussian smoothing both debiasing methods are usable. Therefore it would be interesting to show both methods and to describe thoroughly the differences (in addition, it might provide an answer to my previous point). \n\n* The authors should not mention denoising and inpainting applications if there is no quantitative assessment (at least in appendix)… For the inpainting part, the corrupted input are not even shown (which part of the image has been predicted). The denoising and inpainting experiments sounds like it’s been rushed… \n\n#### Typos and suggestions to improve the paper :\n* Minor : Both theorems are provided with nice demonstrations, then the authors should refer to the demonstration in the core text of the article (e.g. see Appendix A).\n* Minor : Add small arrows in Table 2 to indicate that Inception score is better when lower, and opposite for FID\n* Typo : page 5, section 3.3, paragraph 2 : relative —> relatively\n* Figure3 : Right panel : What are the 3 shaded curves ? This should be shown in the legend or at least in the caption\n* Figure3 : Right panel: In the x-axis it should be specified ‘Variance of q(x^{\\tilde} \\mid x)\n* Page 7 : paragraph 1 : 'Thus, it is hard to conclusively determine what is the best way of choosing q(x ̃|x).’ —> I  think the authors actually give the key to properly choose the noise level (i.e. variance). It seems to depend on the task : if one wants to generate  good samples, then the variance has to be set by heuristic. If one needs a good likelihood (e.g. for subsequent downstream tasks) then the variance could be optimized.\n* Figure 6 :  On my understanding, the part ‘denoising’ is redundant with the section image generation. It is interesting to mention the denoising application, but I am not convinced of the utility of the figure 6.\n* Figure 7 : What is the corrupted input ? Which part of the input has been masked ??\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "vWh4UgeXY6u", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### SHORT DESCRIPTION \nThis paper proposes a two-stage generative modeling approach, first learning a distribution over noised data, then learning the original data distribution conditioned on this noised data. The paper demonstrates that this leads to improved sample quality compared to fitting the data distribution directly.\n\n\n#### DISCUSSION\nOverall, I like this paper: it's a straightforward idea, decently motivated and fairly well described, and has good supporting empirical evaluation. I didn't expect the sampling performance to improve substantially by adding just a single denoising step, and I think demonstrating this is a good contribution. However, I think the paper could be improved by some more careful discussion, and a better placement in the literature.\n\n\"Theorem 2 shows that our smoothing process provides a regularization effect on the original objective... This regularization effect can, intuitively, increase the generalization capability of the model.\" How does the extra term in the theorem lead to a regularization effect? Why does this 'intuitively' increase the generalization capability of the model? Unless I'm mistaken, the added term is (up to a constant) the Laplacian of the log-likelihood w.r.t. data. The objective maximizes this on average across observed data, which intuitively minimizes the 'curvature' or 'steepness' of the log-likelihood at observed data, thus presumably smoothing the maximum likelihood solution. This Laplacian term also appears in the score matching objective presented in Theorem 1 of 'Estimation of Non-Normalized Statistical Models by Score Matching, Hyvarinen 2005', where it is minimized instead of maximized. There are also known connections between score matching and denoising methods e.g. 'Optimal Approximation of Signal Priors, Hyvarinen 2006', and 'A Connection Between Score Matching and Denoising Autoencoders, Vincent 2011', which you've cited in passing later. Much of this material and how it relates to the objective in Theorem 2 might be discussed in more depth rather than passing over it as simply a 'regularization term'.   \n\n\"Our approach is related to two-stage VAE (Dai & Wipf, 2019) which introduces a second VAE to correct the errors made by the first VAE.\" I'm not sure I agree with this. The idea of the two-stage VAE in that paper is to clean up mismatch between the aggregate posterior q(z) and the prior p(z). On the other hand, your variational model is identical to the canonical VAE setup: x is data, z is noised data, the 'posterior' q(z | x) is fixed and adds Gaussian noise, the 'prior' p(z) is a powerful autoregressive model, and the observation likelihood p(x | z) is another powerful autoregressive model (the canonical VAE would have learned q(z | x), fixed simple p(z), and learned but simple p(x | z)). This is one of the reasons 'VAE' can a confusing term when used to describe latent variable generative models in general: assuming something should be 'encoded' and 'decoded' can sometimes obfuscate the actual probabilistic model. What you propose in this paper might generically be called a 'denoising VAE', but again that's maybe not the most accurate description. I think the most closely related work is probably the denoising diffusion and denoising score-matching approaches which have received attention recently and which you've mentioned, but you could also think of it as turning a denoising autoencoder into a generative model. In any case, I think a more careful discussion of these points would be beneficial for the paper.\n \nFinally, the approach isn't really tied to autoregressive models, apart from the motivation given in terms of smoothing 1D distributions. It's fairly likely that the same idea could readily be applied to e.g. normalizing flows and that it would work well there also, so it would have been nice to see experiments featuring flows included here. This would be especially useful since your best-performing two-stage method takes autoregressive models, which are already slow samplers, and effectively doubles their sampling time. \n\n\n#### EXTRA NOTES \nMaybe be careful with the word 'spurious' in the intro - I know what you mean, but samples from a model are by definition typical samples from that model, and there's nothing spurious about them. They're only questionable when compared to data. Similarly: \"The “erroneous” sample xˆ, in some sense, resembles an adversarial example, i.e., an input that causes the model to make mistakes.\" This seems to be implying that samples generated by a model are somehow pathological. By virtue of the fact that they are generated by the model, they are *by definition* typical samples from the model. There is nothing pathological whatsoever about them. Why the model specification and fitting procedure have resulted in such samples, and whether the samples resemble training data or not, is another issue entirely.\n\n'However, this approach bounds the capacity of the model by limiting the number of modes for each conditional.' All models have limited capacity -- what is particularly bad about the capacity of an autoregressive model being limited in this way? Do we have reason to believe inability to cover multiple nodes is a common bottleneck?\n\nFigure 2 & Figure 3: Axis ticks are too small (and there probably too many), whole figure could be made bigger (this would also help with legends cutting off a lot of the plots).\n\n'Proofs' for theorems 1 and 2 should be referred to in the main text. Theorem 2 should also have log p(x) on the LHS? \n\nFigure 5 caption: \"All the samples are not conditioned on class labels.\" -> None of the samples are conditioned on class labels.\n\nWhat exactly is being inpainted in Figures 7 (a) and (b)? \n\nSince a central claim of the paper is that the method results in improved sample quality, it might be good to add the Kernel Inception Distance (\"Demystifying MMD GANs\" Binkowski et al 2018) which has many favourable properties over FID, and is really no more difficult to compute.  \n\n#### CONCLUSION \nOverall, I think this paper is a nice submission, and would like to see it accepted given a few tweaks. \n\nUPDATE: I've upped my score to a 7, and would like to see the paper accepted. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "n2nNvRoTi71", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary : This paper proposes an approach to modeling distributions based on a two-step process which involves sampling a noisy x first and then applying a denoising function. The theory is grounded in other works in the literature that use the connection between denoising and the gradient of log p(x) with respect to x.\n\nI enjoyed reading the paper and I think that the authors are definitely working in an exciting area. The authors do a good job in section 3.3 \"Tradeoff in modeling\" to explain clearly that everything about this \"two step\" method relies on striking the right balance between the tasks of modeling p(noisy x) and p(x | noisy x). When one is trivial, the other is very hard.\n\nEverything about this paper hinges on the fact that the authors are learning p(noisy x). If they didn't, this whole paper would be trivial and a rather useless exercise. It took me a few passes to realize that they were indeed learning p(noisy x). This is such an important thing, and I think it might be worth insisting a bit more on this. Otherwise it's easy to look at the pictures and to conclude that they are taking x from the training distribution, adding a small amount of noise, and then showing that they can remove the noise. The authors are doing more than that, and this is why I like this paper.\n\n\nSome more specific comments about the text.\n\nSection 3.3 has \"q(noisy x| x) is simply zero\" when it's in fact a distribution with all its mass around x; not around zero. We understand what the authors meant by the context, but I recommend rephrasing this.\n\nI like figure 1-2 for how they illustrate the concepts well.\n\nI like that the authors took the time to write Theorem 1 instead of hand waving to appeal to how reasonable result feels.\n\nTheorem 2 contains a typo in its statement. The left side of the equation should be \"log p\" instead of \"p\".\n\nI could argue that \"Theorem 2\" might be more appropriately called simply a \"Proposition\" instead of a \"Theorem\", but I will leave it up to the authors to decide that. However, using the \\approx notation without clarifying its precise meaning is unbecoming for a theorem. Here the authors intend to communicate something that there is an equality, but with an extra term o(epsilon^2) on the right-hand side, as epsilon->0, with that epsilon having some role in the definition of q(noisy x | x). I think this deserves to be part of the statement of this theorem. Plenty of things are \"almost equal\" to something else, so the precise meaning matters when stating a theorem.\n\nFigure 5 refers to columns in a way that doesn't feel natural to me. To my eye, column 2 looks bad, column 3 is the best, and column 4 looks the worse. I suspect that the description of the figure either treats the first column as \"Column 0\", or it starts counting \"column 1\" after skipping the leftmost column, or we just don't have the same visual appreciation of thumbnails (especially when they contain wild pixels like column 2 does).\n\nInpainting at the bottom of page 7 doesn't really say what regions of the image are cut out to be inpainted.\n\nFigure 9 says \"unconditioned\", but it's about p(x | noisy x). Wouldn't that be the opposite of \"unconditioned\"?", "extra_scores": {}, "extra_text": {}}, {"review_id": "BgtnXaLSxF9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary.** Autoregressive models have demonstrate their potential utility for modeling images and other types of complex data with high flexibility (particularly in density estimation). However, its sampling ability is not that good as explained in the paper. Authors show that one of the main weaknesses of autoregressive models comes from the propagation of mistakes due to the mismatch of conditionals. Inspired in the promising results of randomized smoothing in adversarial models (Cohen et al. 2019), authors propose a similar strategy. The addition of Gaussian noise and posterior modeling of the smoother data makes easier to the autoregressive density to capture the true data distribution. The benefits of this strategy are empirically proved and shown in the experiments.\n\n**Strengths.** The quality of writing is high and the presentation of the paper facilitates the process of reading. I have to say that I enjoyed while reviewing it. The analysis and description of problems for sampling from autoregressive models is completely understandable to me and I agree with the manifold hypothesis held. \n\nResults with the “sharp” multimodal data looks reliable to me and I believe that the smoother process can also reduce the lipschitz constant as stated in Theorem 1. Until pp. 5, nothing is said about the data denoising process, so one could initially think that there is no way to recover the target density without noise, but authors also did an effort on this. Good point. It is important to remark that the randomized smoothing process can be reverted once learning finishes. \n\nAdditionally, I particularly like how authors first present the idea on 1-d examples, later in the experiments, the method is validated with 2-d rings and finally, as stated in the introduction, with different image datasets. \n\nFinally, I did not find any similar work that mixes the idea of smoothing for improving autoregressive modelling. \n\n**Weaknesses, Questions & Recommendations.**\nTo me, there are 3 main points of weakness:\n[W1]. A lack of analysis about the optimal noise for randomized smoothing.\n[W2]. Why just Gaussian noise, what if data is discrete, could we do this with another type of noise?\n[W3]. Comments about denoising are included a bit late in the manuscript. I think that authors should remark that this is a reversible process.\n\nMy main questions are:\n[Q1]. In section 2.2, I do not see why data closer to the manifold should have larger first order derivatives or even infinity. Is this a bit counter intuitive, or not? Like, better positioned, worse gradient values?\n[Q2]. Is the 1/N term in the global likelihood expression of 1st paragraph of section 2 correct?\n[Q3]. If I do not appropriately choose the \\sigma parameter for smoothing, do I have the risk of not capturing some modes of the original data? I have the opinion that adding too much or too less noise to data could “mask” modes and something could be lost. Am I correct? Did authors empirically analyzed this in the experiments?\n[Q4]. How could we assess that conditionals are now better fitted than before?\n\nA few recommendations for improvement:\n[Rec1]. I would explain a bit more the manifold hypothesis of section 2.2, maybe a diagram or figure would help for quicker comprehension of the problem.\n[Rec2]. Some acronym for “randomized smoothing” would help in the 1st paragraph of section 3.1. To avoid repetitive expressions.\n\n**Reasons for score.** I liked the idea, think that the paper is well written and I trust the results presented by the authors. Despite the randomized smoothing strategy is rather simple, it seems to work particularly well. For this reason I tend to vote for accept. If I not set a higher score, it is because a bit more of analysis on the optimal sigma, distribution for smoothing and lipschitz constant could have been included. \n\n**Post-rebuttal update.** Thanks to the authors for their response to all my questions and comments. I also read the updated version of the manuscript, which is clearly improved and the rest of reviews and comments by the AC. Looking to that, I agree with the rest of reviewers about the quality of the paper, so I raised my score and I recommend to accept it.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommend acceptance. Some concerns were raised about the precision of theorem 2 (now renamed to proposition 1), as well as the analysis of hyperparameter choices and quantitative evaluation, which I believe the authors have adequately addressed. Based on a suggestion of reviewer 1, experiments with flow-based models were also added, which demonstrates that the method is not strictly tied to autoregressive models. Personally, I was also curious about the connection between noise injection and quantisation, which the authors responded to by adding a paragraph discussing this connection in the manuscript.

I would recommend that the authors also add the kernel inception distance (KID) results reported in the comments to the manuscript.

This work stands out to me in that it combines a relatively simple, easy to understand idea with nice results, which is a trait of many impactful papers. I will therefore join the reviewers in recommending acceptance.
Decision: Accept (Oral)