Year: 2019

Abstract:
We propose a generic confidence-based approximation that can be plugged in and simplify the auto-regressive generation process with a proved convergence. We first assume that the priors of future samples can be generated in an independently and identically distributed (i.i.d.) manner using an efficient predictor. Given the past samples and future priors, the mother AR model can post-process the priors while the accompanied confidence predictor decides whether the current sample needs a resampling or not. Thanks to the i.i.d. assumption, the post-processing can update each sample in a parallel way, which remarkably accelerates the mother model. Our experiments on different data domains including sequences and images show that the proposed method can successfully capture the complex structures of the data and generate the meaningful future samples with lower computational cost while preserving the sequential relationship of the data.

heading: 1 INTRODUCTION
text: The auto-regressive (AR) model, which infers and predicts the causal relationship between the previous and future samples in a sequential data, has been widely studied since the beginning of machine learning research. The recent advances of the auto-regressive model brought by the neural network have achieved impressive success in handling complex data including texts (Sutskever et al., 2011), audio signals (Vinyals et al., 2012; Tamamori et al., 2017; van den Oord et al., 2016a), and images (van den Oord et al., 2016b; Salimans et al., 2017).
It is well known that AR model can learn a tractable data distribution p(x) and can be easily extended for both discrete and continuous data. Due to their nature, AR models have especially shown a good fit with a sequential data, such as voice generation (van den Oord et al., 2016a) and provide a stable training while they are free from the mode collapsing problem (van den Oord et al., 2016b). However, these models must infer each element xi of the data x = [x1, x2, · · · , xi, · · · , xN ] in a serial manner, requiring O(N) times more than the other non-sequential estimators, which outputs x at once (Garnelo et al., 2018; Kim et al., 2019; Kingma & Welling, 2014; Goodfellow et al., 2014). Moreover, it is difficult to employ recent parallel computation because AR models always require a previous time step by definition. This mostly limits the use of the AR models in practice despite their advantages.
To resolve the problem, we introduce a new and generic approximation method, Neural AutoRegressive model Approximator (NARA), which can be easily plugged into any AR model. We show that NARA can reduce the generation complexity of AR models by relaxing an inevitable AR nature and enables AR models to employ the powerful parallelization techniques in the sequential data generation, which was difficult previously.
NARA consists of three modules; (1) a prior-sample predictor, (2) a confidence predictor, and (3) the original AR model. To relax the AR nature, given a set of past samples, we first assume that each sample of the future sequence can be generated in an independent and identical manner. Thanks to the i.i.d. assumption, using the first module of NARA, we can sample a series of future priors and these future priors are post-processed by the original AR model, generating a set of raw predictions. The confidence predictor evaluates the credibility of these raw samples and decide whether the model needs re-sampling or not. The confidence predictor plays an important role in that the approximation errors can be accumulated during the sequential AR generation process if the
erroneous samples with low confidence are left unchanged. Therefore, in our model, the sample can be drawn either by the mixture of the AR model or the proposed approximation method, and finally the selection of the generated samples are guided by the predicted confidence.
We evaluate NARA with various baseline AR models and data domains including simple curves, image sequences (Yoo et al., 2017), CelebA (Liu et al., 2015a), and ImageNet (Deng et al., 2009). For the sequential data (simple curves and golf), we employed the Long Short-Term Memory models (LSTM) (Hochreiter & Schmidhuber, 1997) as a baseline AR model while PixelCNN++ (Salimans et al., 2017) is used for the image generation (CelebA and ImageNet). Our experiments show that NARA can largely reduce the sample inference complexity even with a heavy and complex model on a difficult data domain such as image pixels.
The main contributions of our work can be summarized as follows: (1) we introduce a new and generic approximation method that can accelerate any AR generation procedure. (2) Compared to a full AR generation, the quality of approximated samples remains reliable by the accompanied confidence prediction model that measures the sample credibility. (3) Finally, we show that this is possible because, under a mild condition, the approximated samples from our method can eventually converge toward the true future sample. Thus, our method can effectively reduce the generation complexity of the AR model by partially substituting it with the simple i.i.d. model.

heading: 2 PRELIMINARY: AUTO-REGRESSIVE MODELS
text: Auto-regressive generation model is a probabilistic model to assign a probability p(x) of the data x including n samples. This method considers the data x as a sequence {xi | i = 1, · · ·n}, and the probability p(x) is defined by an AR manner as follows:
p(x) = n∏ i=1 p(xi|x1, ..., xi−1), (1)
From the formulation, the AR model provides a tractable data distribution p(x). Recently, in training the model parameters using the training samples x̂T , the computation parallelization are actively employed for calculating the distance between the real sample x̂t ∈ x̂T and the generated sample xt from equation (1). Still, for generating the future samples, it requires O(N) by definition.

heading: 3 PROPOSED METHOD

heading: 3.1 OVERVIEW
text: Figure 9 shows the concept of the proposed approximator NARA. NARA consists of a prior-sample predictor fW and confidence predictor gV . Given samples x≤i = {x1, · · · , xi}, the prior-sample predictor predicts a chunk of M number of the prior values m(i,i+M ]. Afterward, using the prior samples, we draw the future samples x(i,i+M ] in parallel. We note that this is possible because the
prior m(i,i+M ] is i.i.d. variable from our assumption. Subsequently, for the predicted x(i,i+M ], the confidence predictor predicts confidence scores ν(i,i+M ]. Then, using the predicted confidence, our model decides whether the samples of interest should be redrawn by the AR model (re-sample xi+1) or they are just accepted. The detailed explanation will be described in the following sections.

heading: 3.2 APPROXIMATING SAMPLE DISTRIBUTION OF AR MODEL
text: Given the samples x≤i = {x1, · · · , xi}, a AR model defines the distribution of future samples x(i,j] = {xi+1, · · · , xj} as follows:
pθ(x(i,j]|x≤i) = j∏
l=i+1
pθ(xl|x1, ..., xl−1). (2)
Here, θ denotes the parameters of the AR model. The indices i, j are assumed to satisfy the condition j > i, ∀i, j ∈ [1, N ]. To approximate the distribution pθ(x(i,j]|x≤i), we introduce a set of prior samples m(i,j−1] = fW (x≤i;W ), where we assume that they are i.i.d. given the observation x≤i. Here, W is the model parameter of the prior-sample predictor fW (·). Based on this, we define an approximated distribution qθ,W (x(i,j−1]|x≤i,m(i,j−1]) characterized by the original AR model pθ and the prior-sample predictor fW (·) as follows:
qθ,W (x(i,j]|x≤i,m(i,j−1]) ≡ pθ(xi+1|x≤i) j∏
l=i+2 pθ(xl|x≤i,mi+1, . . .ml−1)︸ ︷︷ ︸ Compute in parallel (const time)
(A) ' pθ(xi+1|x≤i) j∏ l=i+2
pθ(xl|x≤i, xi+1, . . . xl−1)︸ ︷︷ ︸ Compute in sequential (linear time)
= pθ(x(i,j]|x≤i).
(3)
Here, approximation (A) is true when m(i,j−1] approaches to x(i,j−1]. Note that it becomes possible to compute qθ,W in a constant time because we assume the prior variable mi to be i.i.d. while pθ requires a linear time complexity.
Then, we optimize the network parameters θ and W by minimizing the negative log-likelihood (NLL) of qθ,W (x(i,j] = x̂(i,j]|x≤i, fW (x≤i)) where x̂(i,j] is a set of samples that are sampled from the baseline AR model. We guide the prior-sample predictor fW (·) to generate the prior samples that is likely to come from the distribution of original AR model m by minimizing the original AR model θ and prior-sample predictor W jointly as follows:
min θ,W − log pθ(x(g)(i,j]|x≤i)− Epθ(x(i,j]|x≤i)[log qθ,W (x(i,j]|x≤i, fW (x≤i))], (4)
where x(g) denotes the ground truth sample value in the generated region of the training samples. Note that both pθ(x) and its approximated distribution qθ,W (x) approaches to the true data distribution when (1) our prior-sample predictor generates the prior samples m close to the true samples x and (2) the NLL of the AR distribution approaches to the data distribution. Based on our analysis and experiments, we later show that our model can satisfy these conditions theoretically and empirically in the following sections.

heading: 3.3 CONFIDENCE PREDICTION
text: Using the prior-sample predictor fW (·), our model generates future samples based on the previous samples. However, accumulation of approximation errors in the AR generation may lead to an unsuccessful sample generation. To mitigate the problem, we introduce an auxiliary module that determines whether to accept or reject the approximated samples generated as described in the previous subsection, referred to as confidence predictor.
First, we define the confidence of the generated samples as follows: νk = qθ,W (xk = x̂k|x≤i, fW (x≤i)), (5)
where x̂k ∼ pθ(xk|x≤k−1) and k ∈ {1, · · · , j}. The confidence value νk provides a measure of how likely the generated samples from qθ,W (·) is drawn from pθ(xk|x≤k−1). Based on the confidence value νk, our model decides whether it can accept the sample xk or not. More specifically, we choose a threshold ∈ [0, 1] and accept samples which have the confidence score larger than the threshold . When the case = 1, our model always redraws the sample using the AR model no matter how our confidence is high. Note that our model becomes equivalent to the target AR model when = 1. When = 0, our model always accepts the approximated samples. In practice, we accept k̂( )− 1 samples among approximated M samples where k̂( ) = argmink νk > . Subsequently, we re-sample xk̂( )) from the original AR model and repeat approximation scheme until reach the maximum length.
However, it is impractical to calculate equation (5) directly because we need the samples x̂k from the original AR model. We first need to go forward using the AR model to see the next sample and come backward to calculate the confidence to decide whether we use the sample or not, which is nonsense.
To detour this problem, we introduce a network gV (·) that approximates the binary decision variable h k = I(νk ≥ ) as follows: h (i,j] ' gV (x≤i, fW (x≤i)), (6) where h (i,j] = {h i+1, · · · , h j}. The network gV (·) is implemented by a auto-encoder architecture with a sigmoid activation output that makes the equation (6) equivalent to the logistic regression.

heading: 3.4 TRAINING DETAILS
text: To train the proposed model, we randomly select the sample s(i) ∈ [1, N ] for the sequence x(i) in a training batch. Then, we predict l(i) = min(B,N − s(i)) sample values after s(i), where B denotes the number of samples the prediction considers. To calculate equation (4), we minimize the loss of the training sample x(k), k = 1 · · ·K, and the locations s(i) ∈ [1, N ] as,
min θ,W − 1 K K∑ i=1 log pθ(x (i) ≤s(i)+l(i))− 1 KM K∑ i=1 M∑ j=1 log qθ,W (x̂ (i)(j) (s(i),s(i)+l(i)] |x(i)≤s(i)). (7)
Here, x̂(i)(j) (s(i),s(i)+l(i)] for j ∈ {1 · · ·M} denotes M number of the sequences from the AR distribution pθ(x(s(i),s(i)+l(i)]|x (i)
≤s(i)) for i-th training data. From the experiment, we found that M = 1 sample is enough to train the model. This training scheme guides the distribution drawn by NARA to fit the original AR distribution as well as to generate future samples, simultaneously. To train gV (·), binary cross-entropy loss is used with h in equation (6), with freezing the other parameters.

heading: 3.5 THEORETICAL EXPLANATION
text: Here, we show that the proposed NARA is a regularized version of the original AR model. At the extremum, the approximated sample distribution from NARA is equivalent to that of the original AR model. In NARA, our approximate distribution q(x(i,j+1]|x≤i,m(i,j]) is reformulated as follows:
qφ,θ(x(i,j+1]|x≤i,m(i,j]) ≡ pθ(xi+1|x≤i) j+1∏ l=i+2 qφ(xl|x≤i,mi+1, ...,ml−1)
= pθ(x≤i+1) pθ(x≤i) · pθ(x≤i+2) pθ(x≤i+1) ·, · · · , ·pθ(x≤j+1) pθ(x≤j)︸ ︷︷ ︸
pθ(x(i,j+1]|x≤i)
· [ pθ(x≤i+1)
qφ(x≤i,mi+1) ·, · · · , ·
qφ(x≤i,m(i,j], xj+1)
pθ(x≤j+1) ] ︸ ︷︷ ︸
R(pθ,qφ,m(i,j])
,
(8)
where the parameter φ denotes the network parameters of the approximated distribution q(·). Therefore, our proposed cost function can be represented as the negative log-likelihood of the AR model with a regularizer − logR(pθ, qφ,m(i,j]):
min φ,θ −Epθ(x(i,j+1]|x≤i)[log pθ(x(i,j+1]|x≤i)) + logR(pθ, qφ,m(i,j])] (9)
Note that the proposed cost function is equivalent to that of the original AR model when logR(pθ, qφ,m(i,j]) = 0, which is true under the condition of m(i,j] = x(i,j] and qφ(·|·) = pθ(·|·). Here, m(i,j] = fW (x≤i). By minimizing the equation (9),R(pθ, qφ,m(i,j]) enforces the direction of the optimization to estimate the probability ratio of qφ and pθ while it minimize the gap between qφ(x≤i,m(i,j],xj+1)
qφ(x≤i,m(i,j]) so that m(i,j] = fW (x≤i) approaches to x(i,j].

heading: 4 RELATED WORK
text: Deep AR and regression models: After employing the deep neural network, the AR models handling sequential data has achieved significant improvements in handling the various sequential data including text (Sutskever et al., 2011), sound (Vinyals et al., 2012; Tamamori et al., 2017), and images (van den Oord et al., 2016b; Salimans et al., 2017). The idea has been employed to “Flow based model” which uses auto-regressive sample flows (Kingma & Dhariwal, 2018; Germain et al., 2015; Papamakarios et al., 2017; Kingma et al., 2016) to infer complex distribution, and reported meaningful progresses. Also, the attempts (Yoo et al., 2017; Garnelo et al., 2018; Kim et al., 2019) to replace the kernel function of the stochastic regression and prediction processes to neural network has been proposed to deal with semi-supervised data not imposing an explicit sequential relationship.
Approximated AR methods: Reducing the complexity of the deep AR model has been explored by a number of studies, either targeting multiple domain (Seo et al., 2018; Stern et al., 2018) or specific target such as machine translation (Wang et al., 2018; Ghazvininejad et al., 2019; Welleck et al., 2019; Wang et al., 2018; 2019) and image generation (Ramachandran et al., 2017).
Adding one step further to the previous studies, we propose a new general approximation method for AR methods by assuming the i.i.d. condition for the “easy to predict” samples. This differentiates our approach to (Seo et al., 2018) in that we do not sequentially approximate the future samples by using a smaller AR model but use a chunk-wise predictor to approximate the samples at once. In addition, our confidence prediction module can be seen as a stochastic version of the verification step in (Stern et al., 2018), which helps our model to converge toward the original solution. This confidence guided approximation can be easily augmented to the other domain specific AR approximation methods because our method is not limited to a domain specific selection queues such as quotation (Welleck et al., 2019; Ghazvininejad et al., 2019) or nearby convolutional features (Ramachandran et al., 2017).

heading: 5 EXPERIMENTS
text: In this section, we demonstrate the data generation results from the proposed NARA. To check the feasibility, we first test our method into time-series data generation problem, and second, into image generation. The detailed model structures and additional results are attached in the Supplementary material. The implementation of the methods will be available soon.

heading: 5.1 EXPERIMENTAL SETTING
text: Time-series data generation problem: In this problem, we used LSTM as the base model. First, we tested our method with a simple one-dimensional sinusoidal function. Second, we tested the video sequence data (golf swing) for demonstrating the more complicated case. In this case, we repeated the swing sequences 20 times to make the periodic image sequences and resize each image to 64× 64 resolution. Also, beside the LSTM, we used autoencoder structures to embed the images into latent space. The projected points for the image sequences are linked by LSTM, similar to (Yoo et al., 2017). For both cases, we used ADAM optimizer (Kingma & Ba, 2015) with a default setting and a learning rate 0.001.
Image generation: For the image generation task, we used PixelCNN++ (Salimans et al., 2017) as the base model. The number of channels of the network was set to 160 and the number of logistic mixtures was set to 10. See (Salimans et al., 2017) for the detailed explanation of the parameters. In this task, the baseline AR model (PixelCNN++) is much heavier than those used in the previous tasks. Here, we show that the proposed approximated model can significantly reduce the computational burden of the original AR model. The prior-sample predictor fW (·) and the confidence estimator gV (·) were both implemented by U-net structured autoencoder (Ronneberger et al., 2015). We optimized the models using ADAM with learning rate 0.0001. Every module was trained from scratch. We mainly used CelebA (Liu et al., 2015b) resizing the samples to 64× 64 resolution. In the experiments, we randomly pick 36, 000 images for training and 3, 000 images for validation.
Training and evaluation: For the first problem, we use single GPU (NVIDIA Titan XP), and for the second problem, four to eight GPUs (NVIDIA Tesla P40) were used1. The training and inference code used in this section are implemented by PyTorch library. For the quantitative evaluation, we measure the error between the true future samples and the generated one, and also employ Fréchet Inception Distance score (FID) (Heusel et al., 2017) as a measure of the model performance and visual quality of the generated images for the second image generation problem.

heading: 5.2 ANALYSIS

heading: 5.2.1 TIME-SERIES DATA GENERATION
text: Figure 2a shows the generation results of the one-dimensional time-series from our approximation model with different acceptance ratios (red, green, and blue) and the baseline LSTM models (black).
1The overall expreiments were conducted on NSML (Sung et al., 2017) GPU system.
From the figure, we can see that both models correctly generates the future samples. Please note that, from the prior sample generation result (magenta), the prior samples m converged to the true samples x as claimed in Section 3.5.
The graph in Figure 2b shows the acceptance ratio and the `1-error over the confidence threshold ∈ (0, 1]. The error denotes the distance between the ground truth samples x and the generated ones. As expected, our model accepted more samples as the threshold decreases. However, contrary to our initial expectations, the error-threshold graph shows that the less acceptance of samples does not always bring the more accurate generation results. From the graph, the generation with an intermediate acceptance ratio achieved the best result. Interestingly, we report that this tendency between the acceptance ratio and the generation quality was repeatedly observed in the other datasets as well.
Figure 3 shows the image sequence generation results from NARA. From the result, we can see that the proposed approximation method is still effective when the input data dimension becomes much larger and the AR model becomes more complicated. In the golf swing dataset, the proposed approximation model also succeeded to capture the periodic change of the image sequence. The table 3 shows that the proper amount of approximation can obtain better accuracy than the none, similar to the other previous experiments. One notable observation regarding the phenomenon is that the period of image sequence was slightly changed among different ratio of approximated sample acceptance (Figure 3). One possible explanation would be that the approximation module suppress the rapid change of the samples, and this affects the interval of a single cycle.

heading: 5.2.2 IMAGE GENERATION
text: Figure 4 shows that our method can be integrated into PixelCNN++ and generates images with the significant amount of the predicted sample acceptance (white region). We observed that the confidence was mostly low (blue) in the eyes, mouth, and boundary regions of the face, and the PixelCNN is used to generate those regions. This shows that compared to the other homogeneous regions of the image, the model finds it relatively hard to describe the details, which matches with our intuition.
The graphs in Figure 5 present the quantitative analysis regarding the inference time and the NLL in generating images. In Figure 5a, the relation between inference time and the skimming ratio is reported. The results show that the inference speed is significantly improved as more pixels are accepted. Table 2 further supports this that our approximation method generates a fair quality of images while it speeds up the generation procedure 5 ∼ 10 times faster than the base model. In the image generation example also, we found that the fair amount of acceptance can improve the perceptual visual quality of the generated images compared to the vanilla PixelCNN++ (Table 2). Our method benefits from increasing the acceptance ratio to some extent in terms of FID showing a U-shaped trend over the variation, similar to those in Figure 2b. Note that a lower FID score identifies a better model. Consistent with previous results, we can conjecture that the proposed approximation scheme learns the mean-prior of the images and guides the AR model to prevent generating erroneous images. The confidence maps and the graph illustrated in Figure 4, 5a, and 5c
support this conjecture. Complex details such as eyes, mouths and contours have largely harder than the backgrounds and remaining faces.
In Figure 5b and Figure 5c, the graphs show the results supporting the convergence of the proposed method. The graph in Figure 5b shows the NLL of the base PixelCNN++ and that of our proposed method under the full-accept case, i.e. we fully believe the approximation results. Note that the NLL of both cases converged and the PixelCNN achieved noticeably lower NLL compared to the fully accepting the pixels at every epoch. This is already expected in Section 3.2 that the baseline AR model approaches more closely to the data distribution than our module. This supports the necessity of re-generating procedure by using the PixelCNN++, especially when the approximation module finds the pixel has a low confidence.
The graph in Figure 5c presents the `1 distance between the generated prior pixel m and the corresponding ground-truth pixel x in the test data reconstruction. Again, similar to the previous time-series experiments, the model successfully converged to the original value (m approaches to x). Combined with the result in Figure 5b, this result supports the convergence conditions claimed in section 3.2. Regarding the convergence, we compared the NLL of the converged PixelCNN++ distribution from the proposed scheme and that of PixelCNN++ with CelebA dataset from the original paper (Salimans et al., 2017).

heading: 6 CONCLUSION
text: In this paper, we proposed the efficient neural auto-regressive model approximation method, NARA, which can be used in various auto-regressive (AR) models. By introducing the prior-sampling and confidence prediction modules, we showed that NARA can theoretically and empirically approximate the future samples under a relaxed causal relationships. This approximation simplifies the generation process and enables our model to use powerful parallelization techniques for the sample generation procedure. In the experiments, we showed that NARA can be successfully applied with different AR models in the various tasks from simple to complex time-series data and image pixel generation. These results support that the proposed method can introduce a way to use AR models in a more efficient manner.

heading: B SUPPLEMENTARY EXPLAIN ON PROPOSED SAMPLE GENERATION
text: Figure 6 shows the detailed process of sampling when the proposed NARA module is attached to the mother AR model. The diagram describes the Sinusoidal function generation example, which is the simplest example in our paper. In each auto-regressive step, we decide if we use the approximated value predicted by our sample predictor (predict the value in chunkwise manner assuming i.i.d condition) or sample the value with the mother auto-regressive model. This decision is conducted based on the confidence estimation by the Confidence predictor of the paper. The chunk-wise estimation step can be boosted by recent parallel computing methods different from the mother autoregressive model, and this can possibly relax the computation burden of the original auto-regressive model. More importantly, this scheme can be applied to various auto-regressive models.

heading: C SUPPLEMENTARY EXPERIMENTS
text: C.1 SUPPLEMENTARY GENERATION RESULTS
In addition to the results presented in the paper, we show supplement generation examples in below figures. Figure 3 and Table 3 present the image sequence generation result from the other golf swing sequence. In this case also, we can observe the similar swing cycle period changes and acceptance ratio-error tendencies reported in the paper. Our approximation slightly affects the cycle of the time-serious data, and the result also shows that the approximation can achieve even better prediction results than “none-acceptance” case.
Figure 11 and 12 shows the additional facial image generation results among ∈ [0.0, 1.0]. We can see that pixels from boundary region were more frequently re-sampled compared to other relatively simple face parts such as cheek or forehead. Also, we tested our model with ImageNet classes, and the results were presented in Figure 10.
C.2 ANALYSIS ON WHEN CONFIDENCE MODULE FAILS.
If the prior-sample predictor fW performs worse than our expectation, the confidence module gV will reject all prior samples; hence, in this case, the model repeats to draw samples using the original AR model in a sample-by-sample manner. However, in an even worse situation, the confidence module could always accept prior-samples, including low-quality prior samples. To simulate the failure or both fW and gV , we manually fix the “accept region” regardless of the predicted confidence score. We select ten samples with the lowest confidence scores and report the results in Figure 8. In the figure, we can observe that the drastic error can occur when both the prior-sample predictor fW and confidence module gV fail. However, in practice, our confidence module gV can detect the drastic error during the sampling stage as shown in Figure 8b; hence, such extreme error will occur rarely.
C.3 APPLYING NARA WITH FASTER PIXELCNN++
To show that our method can be added to diverse AR mode, we also combined our skimming method to the fast version of PixelCNN++2. Figure 9 and Table 4 show the generated samples and the generation time using the incorporated model (skim+Fast PixelCNN++). Due to the lack of time to fully investigate all datasets with the new implementation, we conducted the experiment on CIFAR-10, which is the most widely used dataset in PixelCNN works and is more diverse than the CelebA dataset. From the results, we show that our method, augmented with Fast PixelCNN++, can make the baseline algorithm much faster as we suggested in the paper.
2Ramachandran, Prajit, et al. ”Fast generation for convolutional autoregressive models.” arXiv preprint arXiv:1704.06001 (2017).

Review summaries:
This paper addresses the sequential limitation of autoregressive model when doing sampling. Specifically, instead of sampling next observations in a sequential fashion, this paper generates future observations in parallel, with the help of i.i.d. future priors. With the help of learned confidence model, the model is able to get trade-off between speed and approximation accuracy. Experiments on synthetic data and image generation with PixelCNN++ show the comparable results while being significantly faster than baseline.