Neural Approximation of an Auto-Regressive Process through Confidence Guided Sampling

1 INTRODUCTION . The auto-regressive ( AR ) model , which infers and predicts the causal relationship between the previous and future samples in a sequential data , has been widely studied since the beginning of machine learning research . The recent advances of the auto-regressive model brought by the neural network have achieved impressive success in handling complex data including texts ( Sutskever et al. , 2011 ) , audio signals ( Vinyals et al. , 2012 ; Tamamori et al. , 2017 ; van den Oord et al. , 2016a ) , and images ( van den Oord et al. , 2016b ; Salimans et al. , 2017 ) . It is well known that AR model can learn a tractable data distribution p ( x ) and can be easily extended for both discrete and continuous data . Due to their nature , AR models have especially shown a good fit with a sequential data , such as voice generation ( van den Oord et al. , 2016a ) and provide a stable training while they are free from the mode collapsing problem ( van den Oord et al. , 2016b ) . However , these models must infer each element xi of the data x = [ x1 , x2 , · · · , xi , · · · , xN ] in a serial manner , requiring O ( N ) times more than the other non-sequential estimators , which outputs x at once ( Garnelo et al. , 2018 ; Kim et al. , 2019 ; Kingma & Welling , 2014 ; Goodfellow et al. , 2014 ) . Moreover , it is difficult to employ recent parallel computation because AR models always require a previous time step by definition . This mostly limits the use of the AR models in practice despite their advantages . To resolve the problem , we introduce a new and generic approximation method , Neural AutoRegressive model Approximator ( NARA ) , which can be easily plugged into any AR model . We show that NARA can reduce the generation complexity of AR models by relaxing an inevitable AR nature and enables AR models to employ the powerful parallelization techniques in the sequential data generation , which was difficult previously . NARA consists of three modules ; ( 1 ) a prior-sample predictor , ( 2 ) a confidence predictor , and ( 3 ) the original AR model . To relax the AR nature , given a set of past samples , we first assume that each sample of the future sequence can be generated in an independent and identical manner . Thanks to the i.i.d . assumption , using the first module of NARA , we can sample a series of future priors and these future priors are post-processed by the original AR model , generating a set of raw predictions . The confidence predictor evaluates the credibility of these raw samples and decide whether the model needs re-sampling or not . The confidence predictor plays an important role in that the approximation errors can be accumulated during the sequential AR generation process if the erroneous samples with low confidence are left unchanged . Therefore , in our model , the sample can be drawn either by the mixture of the AR model or the proposed approximation method , and finally the selection of the generated samples are guided by the predicted confidence . We evaluate NARA with various baseline AR models and data domains including simple curves , image sequences ( Yoo et al. , 2017 ) , CelebA ( Liu et al. , 2015a ) , and ImageNet ( Deng et al. , 2009 ) . For the sequential data ( simple curves and golf ) , we employed the Long Short-Term Memory models ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) as a baseline AR model while PixelCNN++ ( Salimans et al. , 2017 ) is used for the image generation ( CelebA and ImageNet ) . Our experiments show that NARA can largely reduce the sample inference complexity even with a heavy and complex model on a difficult data domain such as image pixels . The main contributions of our work can be summarized as follows : ( 1 ) we introduce a new and generic approximation method that can accelerate any AR generation procedure . ( 2 ) Compared to a full AR generation , the quality of approximated samples remains reliable by the accompanied confidence prediction model that measures the sample credibility . ( 3 ) Finally , we show that this is possible because , under a mild condition , the approximated samples from our method can eventually converge toward the true future sample . Thus , our method can effectively reduce the generation complexity of the AR model by partially substituting it with the simple i.i.d . model . 2 PRELIMINARY : AUTO-REGRESSIVE MODELS . Auto-regressive generation model is a probabilistic model to assign a probability p ( x ) of the data x including n samples . This method considers the data x as a sequence { xi | i = 1 , · · ·n } , and the probability p ( x ) is defined by an AR manner as follows : p ( x ) = n∏ i=1 p ( xi|x1 , ... , xi−1 ) , ( 1 ) From the formulation , the AR model provides a tractable data distribution p ( x ) . Recently , in training the model parameters using the training samples x̂T , the computation parallelization are actively employed for calculating the distance between the real sample x̂t ∈ x̂T and the generated sample xt from equation ( 1 ) . Still , for generating the future samples , it requires O ( N ) by definition . 3 PROPOSED METHOD . 3.1 OVERVIEW . Figure 9 shows the concept of the proposed approximator NARA . NARA consists of a prior-sample predictor fW and confidence predictor gV . Given samples x≤i = { x1 , · · · , xi } , the prior-sample predictor predicts a chunk of M number of the prior values m ( i , i+M ] . Afterward , using the prior samples , we draw the future samples x ( i , i+M ] in parallel . We note that this is possible because the prior m ( i , i+M ] is i.i.d . variable from our assumption . Subsequently , for the predicted x ( i , i+M ] , the confidence predictor predicts confidence scores ν ( i , i+M ] . Then , using the predicted confidence , our model decides whether the samples of interest should be redrawn by the AR model ( re-sample xi+1 ) or they are just accepted . The detailed explanation will be described in the following sections . 3.2 APPROXIMATING SAMPLE DISTRIBUTION OF AR MODEL . Given the samples x≤i = { x1 , · · · , xi } , a AR model defines the distribution of future samples x ( i , j ] = { xi+1 , · · · , xj } as follows : pθ ( x ( i , j ] |x≤i ) = j∏ l=i+1 pθ ( xl|x1 , ... , xl−1 ) . ( 2 ) Here , θ denotes the parameters of the AR model . The indices i , j are assumed to satisfy the condition j > i , ∀i , j ∈ [ 1 , N ] . To approximate the distribution pθ ( x ( i , j ] |x≤i ) , we introduce a set of prior samples m ( i , j−1 ] = fW ( x≤i ; W ) , where we assume that they are i.i.d . given the observation x≤i . Here , W is the model parameter of the prior-sample predictor fW ( · ) . Based on this , we define an approximated distribution qθ , W ( x ( i , j−1 ] |x≤i , m ( i , j−1 ] ) characterized by the original AR model pθ and the prior-sample predictor fW ( · ) as follows : qθ , W ( x ( i , j ] |x≤i , m ( i , j−1 ] ) ≡ pθ ( xi+1|x≤i ) j∏ l=i+2 pθ ( xl|x≤i , mi+1 , . . .ml−1 ) ︸ ︷︷ ︸ Compute in parallel ( const time ) ( A ) ' pθ ( xi+1|x≤i ) j∏ l=i+2 pθ ( xl|x≤i , xi+1 , . . . xl−1 ) ︸ ︷︷ ︸ Compute in sequential ( linear time ) = pθ ( x ( i , j ] |x≤i ) . ( 3 ) Here , approximation ( A ) is true when m ( i , j−1 ] approaches to x ( i , j−1 ] . Note that it becomes possible to compute qθ , W in a constant time because we assume the prior variable mi to be i.i.d . while pθ requires a linear time complexity . Then , we optimize the network parameters θ and W by minimizing the negative log-likelihood ( NLL ) of qθ , W ( x ( i , j ] = x̂ ( i , j ] |x≤i , fW ( x≤i ) ) where x̂ ( i , j ] is a set of samples that are sampled from the baseline AR model . We guide the prior-sample predictor fW ( · ) to generate the prior samples that is likely to come from the distribution of original AR model m by minimizing the original AR model θ and prior-sample predictor W jointly as follows : min θ , W − log pθ ( x ( g ) ( i , j ] |x≤i ) − Epθ ( x ( i , j ] |x≤i ) [ log qθ , W ( x ( i , j ] |x≤i , fW ( x≤i ) ) ] , ( 4 ) where x ( g ) denotes the ground truth sample value in the generated region of the training samples . Note that both pθ ( x ) and its approximated distribution qθ , W ( x ) approaches to the true data distribution when ( 1 ) our prior-sample predictor generates the prior samples m close to the true samples x and ( 2 ) the NLL of the AR distribution approaches to the data distribution . Based on our analysis and experiments , we later show that our model can satisfy these conditions theoretically and empirically in the following sections . 3.3 CONFIDENCE PREDICTION . Using the prior-sample predictor fW ( · ) , our model generates future samples based on the previous samples . However , accumulation of approximation errors in the AR generation may lead to an unsuccessful sample generation . To mitigate the problem , we introduce an auxiliary module that determines whether to accept or reject the approximated samples generated as described in the previous subsection , referred to as confidence predictor . First , we define the confidence of the generated samples as follows : νk = qθ , W ( xk = x̂k|x≤i , fW ( x≤i ) ) , ( 5 ) where x̂k ∼ pθ ( xk|x≤k−1 ) and k ∈ { 1 , · · · , j } . The confidence value νk provides a measure of how likely the generated samples from qθ , W ( · ) is drawn from pθ ( xk|x≤k−1 ) . Based on the confidence value νk , our model decides whether it can accept the sample xk or not . More specifically , we choose a threshold ∈ [ 0 , 1 ] and accept samples which have the confidence score larger than the threshold . When the case = 1 , our model always redraws the sample using the AR model no matter how our confidence is high . Note that our model becomes equivalent to the target AR model when = 1 . When = 0 , our model always accepts the approximated samples . In practice , we accept k̂ ( ) − 1 samples among approximated M samples where k̂ ( ) = argmink νk > . Subsequently , we re-sample xk̂ ( ) ) from the original AR model and repeat approximation scheme until reach the maximum length . However , it is impractical to calculate equation ( 5 ) directly because we need the samples x̂k from the original AR model . We first need to go forward using the AR model to see the next sample and come backward to calculate the confidence to decide whether we use the sample or not , which is nonsense . To detour this problem , we introduce a network gV ( · ) that approximates the binary decision variable h k = I ( νk ≥ ) as follows : h ( i , j ] ' gV ( x≤i , fW ( x≤i ) ) , ( 6 ) where h ( i , j ] = { h i+1 , · · · , h j } . The network gV ( · ) is implemented by a auto-encoder architecture with a sigmoid activation output that makes the equation ( 6 ) equivalent to the logistic regression .

Summary: This paper addresses the sequential limitation of autoregressive model when doing sampling. Specifically, instead of sampling next observations in a sequential fashion, this paper generates future observations in parallel, with the help of i.i.d. future priors. With the help of learned confidence model, the model is able to get trade-off between speed and approximation accuracy. Experiments on synthetic data and image generation with PixelCNN++ show the comparable results while being significantly faster than baseline.