Neural Rendering Model: Joint Generation and Prediction for Semi-Supervised Learning

1 INTRODUCTION . Unsupervised and semi-supervised learning have still lagged behind compared to performance leaps we have seen in supervised learning over the last five years . This is partly due to a lack of good generative models that can capture all latent variations in complex domains such as natural images and provide useful structures that help learning . When it comes to probabilistic generative models , it is hard to design good priors for the latent variables that drive the generation . Instead , recent approaches avoid the explicit design of image priors . For instance , the Generative Adversarial Networks ( GANs ) use implicit feedback from an additional discriminator that distinguishes real from fake images ( Goodfellow et al. , 2014 ) . Using such feedback helps GANs to generate visually realistic images , but it is not clear if this is the most effective form of feedback for predictive tasks . Moreover , due to separation of generation and discrimination in GANs , there are typically more parameters to train , and this might make it harder to obtain gains for semi-supervised learning in the low ( labeled ) sample setting . We propose an alternative approach to GANs by designing a class of probabilistic generative models , such that inference in those models also has good performance on predictive tasks . This approach is well-suited for semi-supervised learning since it eliminates the need for a separate prediction network . Specifically , we answer the following question : what generative processes output Convolutional Neural Networks ( CNNs ) when inference is carried out ? This is natural to ask since CNNs are stateof-the-art ( SOTA ) predictive models for images , and intuitively , such powerful predictive models should capture some essence of image generation . However , standard CNNs are not directly reversible and likely do not have all the information for generation since they are trained for predictive tasks such as image classification . We can instead invert the irreversible operations in CNNs , e.g. , the rectified linear units ( ReLUs ) and spatial pooling , by assigning auxiliary latent variables to account for uncertainty in the CNN ’ s inversion process due to the information loss . Contribution 1 – Neural Rendering Model : We develop the Neural Rendering Model ( NRM ) whose bottom-up inference corresponds to a CNN architecture of choice ( see Figure 1a ) . The “ reverse ” top-down process of image generation is through coarse-to-fine rendering , which progressively increases the resolution of the rendered image . This is intuitive since the reverse process of bottom-up inference reduces the resolution ( and dimension ) through operations such as spatial pooling . We also introduce structured stochasticity in the rendering process through a small set of discrete latent variables , which capture the uncertainty in reversing the CNN feed-forward process . The rendering in NRM follows a product of linear transformations , which can be considered as the transpose of the inference process in CNNs . In particular , the rendering weights in NRM are proportional to the transposes of the filters in CNNs . Furthermore , the bias terms in the ReLU units at each layer ( after the convolution operator ) make the latent variables in different network layers dependent ( when the bias terms are non-zero ) . This design of image prior has an interesting interpretation from a predictive-coding perspective in neuroscience : the dependency between latent variables can be considered as a form of backward connections that captures prior knowledge from coarser levels in NRM and helps adjust the estimation at the finer levels ( Rao & Ballard , 1999 ; Friston , 2018 ) . NRM is a likelihood-based framework , where unsupervised learning can be derived by maximizing the expected complete-data log-likelihood of the model while supervised learning is done through optimizing the class-conditional log-likelihood . Semi-supervised learning unifies both log-likelihoods into an objective cost for learning from both labeled and unlabeled data . The NRM prior has the desirable property of being a conjugate prior , which makes learning in NRM computationally efficient . Interestingly , we derive the popular cross-entropy loss used to train CNNs for supervised learning as an upper bound of the NRM ’ s negative class-conditional log-likelihood . This new interpretation of cross-entropy allow us to develop better losses for training CNNs . An example is the Max-Min cross-entropy discussed in Contribution 2 and Section 3 . Contribution 2 – New regularization , loss function , architecture and generalization bounds : The joint nature of generation , inference , and learning in NRM allows us to develop new training procedures for semi-supervised and supervised learning , as well as new theoretical ( statistical ) guarantees for learning . In particular , for training , we derive a new form of regularization termed as the Rendering Path Normalization ( RPN ) from the NRM ’ s conjugate prior . A rendering path is a set of latent variable values in NRM . Unlike the path-wise regularizer in ( Neyshabur et al. , 2015 ) , RPN uses information from a generative model to penalizes the number of the possible rendering paths , encouraging the network to be compact in terms of representing the image . It also helps enforce the dependency among different layers in NRM during training and improves classification performance . We provide a new theoretical bound based on NRM . In particular , we prove that NRM is statistically consistent and derive a generalization bound of NRM for ( semi- ) supervised learning tasks . Our generalization bound is proportional to the number of active rendering paths that generate close-to-real images . This suggests that RPN regularization may help in generalization since RPN enforces the dependencies among latent variables in NRM and , therefore , reduces the number of active rendering paths . We observe that RPN helps improve generalization in our experiments . Max-Min cross-entropy and network : We propose the new Max-Min cross-entropy loss function for learning , based on negative class-conditional log-likelihood in NRM . It combines the traditional cross-entropy with another loss , which we term as the Min cross-entropy . While the traditional ( Max ) cross-entropy maximizes the probability of correct labels , the Min cross-entropy minimizes the probability of incorrect labels . We show that the Max-Min cross-entropy is also an upper bound to the negative conditional log-likehood of NRM , just like the cross-entropy loss . The Max-Min cross-entropy is realized through a new CNN architecture , namely the Max-Min network , which is a CNN with an additional branch sharing weights with the original CNN but containing minimum pooling ( MinPool ) operator and negative rectified linear units ( NReLUs ) , i.e. , min ( · , 0 ) ( see Figure 4 ) . Although the Max-Min network is derived from NRM , it is a meta-architecture that can be applied independently on any CNN architecture . We show empirically that Max-Min networks and crossentropy help improve the SOTA on object classification for supervised and semi-supervised learning . Contribution 3 – State-of-the-art empirical results for semi-supervised and supervised learning : We show strong results for semi-supervised learning over CIFAR10 , CIFAR100 and SVHN benchmarks in comparison with SOTA methods that use and do not use consistency regularization . Consistency regularization , such as those used in Temporal Ensembling ( Laine & Aila , 2017 ) and Mean Teacher ( Tarvainen & Valpola , 2017 ) , enforces the networks to learn representation invariant to realistic perturbations of the data . NRM alone outperforms most SOTA methods which do not use consistency regularization ( Salimans et al. , 2016 ; Dumoulin et al. , 2017 ) in most settings . Max-Min cross-entropy then helps improves NRM ’ s semi-supervised learning results significantly . When combining the NRM , Max-Min cross-entropy , and Mean Teacher , we achieve SOTA results or very close to those on CIFAR10 , CIFAR100 , and SVHN ( see Table 2 , 3 , and 4 ) . Interestingly , compared to the other competitors , our method is consistently good , achieving either best or second best results in all experiments . Furthermore , Max-Min cross-entropy also helps supervised learning . Using the Max-Min cross-entropy , we achieve SOTA result for supervised learning on CIFAR10 ( 2.30 % test error ) . Similarly , Max-Min cross-entropy helps improve supervised training on ImageNet . Despite good classification results , there is a caveat that NRM may not generate good looking images since that objective is not “ baked ” into its training . NRM is primarily aimed at improving semi-supervised and supervised learning through better regularization . Potentially , an adversarial loss can be added to NRM to improve visual characteristics of the image . Related Work : In addition to GANs , other recently developed deep generative models include the Variational Autoencoders ( VAE ) ( Kingma & Welling , 2013 ) and the Deep Generative Networks ( Kingma et al. , 2014 ) . Unlike these models , which replace complicated or intractable inference by CNNs , NRM derives CNNs as its inference . This advantage allows us to develop better learning algorithms for CNNs with statistical guarantees , as being discussed in Section 2.3 and 2.4 . Recent works including the Bidirectional GANs ( Donahue et al. , 2017 ) and the Adversarially Learned Inference model ( Dumoulin et al. , 2017 ) try to make the discriminators and generators in GANs reversible of each other , thereby providing an alternative way to invert CNNs . These approaches , nevertheless , still employ a separate network to bypass the irreversible operators in CNNs . NRM is also close in spirit to the Deep Rendering Model ( DRM ) ( Patel et al. , 2016 ) and the Multi-layered ConvolutionalSparse-Coding Model ( ML-CSC ) ( Papyan et al. , 2018 ) but markedly different . Compared to NRM , DRM and ML-CSC have several limitations . In particular , latent variables in DRM are assumed to be independent , which is rather unrealistic . This lack of dependency causes the missing of the bias terms in the ReLUs of the CNN derived from DRM . Furthermore , the cross-entropy loss used in training CNNs for supervised learning tasks is not captured naturally by DRM and ML-CSC . Due to these limitations , model consistency and generalization bounds are not derived for DRM and ML-CSC . Notation : To facilitate the presentation , the NRM ’ s notations are deferred to Table 5 in Appendix A . 2 THE NEURAL RENDERING MODEL . 2.1 GENERATIVE MODEL . The NRM attempts to invert the CNNs as its inference by employing the structure of its latent variables . The dependencies among latent variables in the model are implicitly captured by their conjugate prior distribution . More precisely , NRM can be defined as follows : Definition 2.1 . [ Neural Rendering Model ( NRM ) ] NRM is a deep generative model in which the latent variables z ( ` ) = { t ( ` ) , s ( ` ) } L ` =1 at different layers ` are dependent . Let x be the input image and y ∈ { 1 , . . . , K } be the target variable , e.g . object category . Generation in NRM takes the form πz|y , Softmax ( 1 σ2 η ( y , z ) ) ; z|y ∼ Cat ( πz|y ) ( 1 ) h ( y , z ; 0 ) , Λ ( z ; 1 ) Λ ( z ; 2 ) · · ·Λ ( z ; L ) µ ( y ) ; x|z , y ∼ N ( h ( 0 ) , σ21D ( 0 ) ) , ( 2 ) ⇥ < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > zero-pad translate zero-pad translate ... < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > Rendering Noise ... < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > h ( ` 1 ) < latexit sha1_base64= '' tf2d+fCOcTNB4hDa1H9joi9ZuXo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUgyURQY8FLx4r2A9IQ9lsN+3STTbsToQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8MJXCoOt+O2vrG5tb26Wd8u7e/sFh5ei4bVSmGW8xJZXuhtRwKRLeQoGSd1PNaRxK3gnHdzO/88S1ESp5xEnKg5gOExEJRtFK/qjW41KSS+Jd9CtVt+7OQVaJV5AqFGj2K1+9gWJZzBNkkhrje26KQU41Cib5tNzLDE8pG9Mh9y1NaMxNkM9PnpJzqwxIpLStBMlc/T2R09iYSRzazpjiyCx7M/E/z88wug1ykaQZ8oQtFkWZJKjI7H8yEJozlBNLKNPC3krYiGrK0KZUtiF4yy+vkvZV3XPr3sN1tdEt4ijBKZxBDTy4gQbcQxNawEDBM7zCm4POi/PufCxa15xi5gT+wPn8AS2Kj+4= < /latexit > < latexit sha1_base64= '' tf2d+fCOcTNB4hDa1H9joi9ZuXo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUgyURQY8FLx4r2A9IQ9lsN+3STTbsToQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8MJXCoOt+O2vrG5tb26Wd8u7e/sFh5ei4bVSmGW8xJZXuhtRwKRLeQoGSd1PNaRxK3gnHdzO/88S1ESp5xEnKg5gOExEJRtFK/qjW41KSS+Jd9CtVt+7OQVaJV5AqFGj2K1+9gWJZzBNkkhrje26KQU41Cib5tNzLDE8pG9Mh9y1NaMxNkM9PnpJzqwxIpLStBMlc/T2R09iYSRzazpjiyCx7M/E/z88wug1ykaQZ8oQtFkWZJKjI7H8yEJozlBNLKNPC3krYiGrK0KZUtiF4yy+vkvZV3XPr3sN1tdEt4ijBKZxBDTy4gQbcQxNawEDBM7zCm4POi/PufCxa15xi5gT+wPn8AS2Kj+4= < /latexit > < latexit sha1_base64= '' tf2d+fCOcTNB4hDa1H9joi9ZuXo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUgyURQY8FLx4r2A9IQ9lsN+3STTbsToQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8MJXCoOt+O2vrG5tb26Wd8u7e/sFh5ei4bVSmGW8xJZXuhtRwKRLeQoGSd1PNaRxK3gnHdzO/88S1ESp5xEnKg5gOExEJRtFK/qjW41KSS+Jd9CtVt+7OQVaJV5AqFGj2K1+9gWJZzBNkkhrje26KQU41Cib5tNzLDE8pG9Mh9y1NaMxNkM9PnpJzqwxIpLStBMlc/T2R09iYSRzazpjiyCx7M/E/z88wug1ykaQZ8oQtFkWZJKjI7H8yEJozlBNLKNPC3krYiGrK0KZUtiF4yy+vkvZV3XPr3sN1tdEt4ijBKZxBDTy4gQbcQxNawEDBM7zCm4POi/PufCxa15xi5gT+wPn8AS2Kj+4= < /latexit > < latexit sha1_base64= '' tf2d+fCOcTNB4hDa1H9joi9ZuXo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUgyURQY8FLx4r2A9IQ9lsN+3STTbsToQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8MJXCoOt+O2vrG5tb26Wd8u7e/sFh5ei4bVSmGW8xJZXuhtRwKRLeQoGSd1PNaRxK3gnHdzO/88S1ESp5xEnKg5gOExEJRtFK/qjW41KSS+Jd9CtVt+7OQVaJV5AqFGj2K1+9gWJZzBNkkhrje26KQU41Cib5tNzLDE8pG9Mh9y1NaMxNkM9PnpJzqwxIpLStBMlc/T2R09iYSRzazpjiyCx7M/E/z88wug1ykaQZ8oQtFkWZJKjI7H8yEJozlBNLKNPC3krYiGrK0KZUtiF4yy+vkvZV3XPr3sN1tdEt4ijBKZxBDTy4gQbcQxNawEDBM7zCm4POi/PufCxa15xi5gT+wPn8AS2Kj+4= < /latexit > x < latexit sha1_base64= '' TQGEygQLk4MpTONXYfpVxGVfRf8= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeCF48t2A9oQ9lsJ+3azSbsbsQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKLSPJb3ZpqgH9GR5CFn1Fip+TQoV9yquwBZJ15OKpCjMSh/9YcxSyOUhgmqdc9zE+NnVBnOBM5K/VRjQtmEjrBnqaQRaj9bHDojF1YZkjBWtqQhC/X3REYjradRYDsjasZ61ZuL/3m91IQ3fsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXvK7Uu3kcRTiDc7gED2pQhztoQAsYIDzDK7w5D86L8+58LFsLTj5zCn/gfP4A7NeNEg== < /latexit > < latexit sha1_base64= '' TQGEygQLk4MpTONXYfpVxGVfRf8= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeCF48t2A9oQ9lsJ+3azSbsbsQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKLSPJb3ZpqgH9GR5CFn1Fip+TQoV9yquwBZJ15OKpCjMSh/9YcxSyOUhgmqdc9zE+NnVBnOBM5K/VRjQtmEjrBnqaQRaj9bHDojF1YZkjBWtqQhC/X3REYjradRYDsjasZ61ZuL/3m91IQ3fsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXvK7Uu3kcRTiDc7gED2pQhztoQAsYIDzDK7w5D86L8+58LFsLTj5zCn/gfP4A7NeNEg== < /latexit > < latexit sha1_base64= '' TQGEygQLk4MpTONXYfpVxGVfRf8= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeCF48t2A9oQ9lsJ+3azSbsbsQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKLSPJb3ZpqgH9GR5CFn1Fip+TQoV9yquwBZJ15OKpCjMSh/9YcxSyOUhgmqdc9zE+NnVBnOBM5K/VRjQtmEjrBnqaQRaj9bHDojF1YZkjBWtqQhC/X3REYjradRYDsjasZ61ZuL/3m91IQ3fsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXvK7Uu3kcRTiDc7gED2pQhztoQAsYIDzDK7w5D86L8+58LFsLTj5zCn/gfP4A7NeNEg== < /latexit > < latexit sha1_base64= '' TQGEygQLk4MpTONXYfpVxGVfRf8= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeCF48t2A9oQ9lsJ+3azSbsbsQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKLSPJb3ZpqgH9GR5CFn1Fip+TQoV9yquwBZJ15OKpCjMSh/9YcxSyOUhgmqdc9zE+NnVBnOBM5K/VRjQtmEjrBnqaQRaj9bHDojF1YZkjBWtqQhC/X3REYjradRYDsjasZ61ZuL/3m91IQ3fsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXvK7Uu3kcRTiDc7gED2pQhztoQAsYIDzDK7w5D86L8+58LFsLTj5zCn/gfP4A7NeNEg== < /latexit > h ( ` ) < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > p < latexit sha1_base64= '' fhC54FVT5NGEMJ6yFbTk8xYGdwc= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqpYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qjW8RRhjM4h0vw4AYacA9N8IEBwjO8wpvz6Lw4787HsrXkFDOn8AfO5w/gt40K < /latexit > < latexit sha1_base64= '' fhC54FVT5NGEMJ6yFbTk8xYGdwc= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqpYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qjW8RRhjM4h0vw4AYacA9N8IEBwjO8wpvz6Lw4787HsrXkFDOn8AfO5w/gt40K < /latexit > < latexit sha1_base64= '' fhC54FVT5NGEMJ6yFbTk8xYGdwc= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqpYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qjW8RRhjM4h0vw4AYacA9N8IEBwjO8wpvz6Lw4787HsrXkFDOn8AfO5w/gt40K < /latexit > < latexit sha1_base64= '' fhC54FVT5NGEMJ6yFbTk8xYGdwc= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqpYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qjW8RRhjM4h0vw4AYacA9N8IEBwjO8wpvz6Lw4787HsrXkFDOn8AfO5w/gt40K < /latexit > p + 1 < latexit sha1_base64= '' /ZAggRAtMCulDnXktOjUB3UP/A8= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh4LXjxWtB/QhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zQHbX0w8Hhvhpl5QSKFQdf9dkpr6xubW+Xtys7u3v5B9fCobeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqEacJ9yM6UiIUjKKVHpILb1CtuXU3B1klXkFqUKA5qH71hzFLI66QSWpMz3MT9DOqUTDJZ5V+anhC2YSOeM9SRSNu/Cw/dUbOrDIkYaxtKSS5+nsio5Ex0yiwnRHFsVn25uJ/Xi/F8MbPhEpS5IotFoWpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nYkPwll9eJe3LuufWvfurWqNbxFGGEziFc/DgGhpwB01oAYMRPMMrvDnSeXHenY9Fa8kpZo7hD5zPH7cLjXo= < /latexit > < latexit sha1_base64= '' /ZAggRAtMCulDnXktOjUB3UP/A8= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh4LXjxWtB/QhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zQHbX0w8Hhvhpl5QSKFQdf9dkpr6xubW+Xtys7u3v5B9fCobeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqEacJ9yM6UiIUjKKVHpILb1CtuXU3B1klXkFqUKA5qH71hzFLI66QSWpMz3MT9DOqUTDJZ5V+anhC2YSOeM9SRSNu/Cw/dUbOrDIkYaxtKSS5+nsio5Ex0yiwnRHFsVn25uJ/Xi/F8MbPhEpS5IotFoWpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nYkPwll9eJe3LuufWvfurWqNbxFGGEziFc/DgGhpwB01oAYMRPMMrvDnSeXHenY9Fa8kpZo7hD5zPH7cLjXo= < /latexit > < latexit sha1_base64= '' /ZAggRAtMCulDnXktOjUB3UP/A8= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh4LXjxWtB/QhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zQHbX0w8Hhvhpl5QSKFQdf9dkpr6xubW+Xtys7u3v5B9fCobeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqEacJ9yM6UiIUjKKVHpILb1CtuXU3B1klXkFqUKA5qH71hzFLI66QSWpMz3MT9DOqUTDJZ5V+anhC2YSOeM9SRSNu/Cw/dUbOrDIkYaxtKSS5+nsio5Ex0yiwnRHFsVn25uJ/Xi/F8MbPhEpS5IotFoWpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nYkPwll9eJe3LuufWvfurWqNbxFGGEziFc/DgGhpwB01oAYMRPMMrvDnSeXHenY9Fa8kpZo7hD5zPH7cLjXo= < /latexit > < latexit sha1_base64= '' /ZAggRAtMCulDnXktOjUB3UP/A8= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh4LXjxWtB/QhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zQHbX0w8Hhvhpl5QSKFQdf9dkpr6xubW+Xtys7u3v5B9fCobeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqEacJ9yM6UiIUjKKVHpILb1CtuXU3B1klXkFqUKA5qH71hzFLI66QSWpMz3MT9DOqUTDJZ5V+anhC2YSOeM9SRSNu/Cw/dUbOrDIkYaxtKSS5+nsio5Ex0yiwnRHFsVn25uJ/Xi/F8MbPhEpS5IotFoWpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nYkPwll9eJe3LuufWvfurWqNbxFGGEziFc/DgGhpwB01oAYMRPMMrvDnSeXHenY9Fa8kpZo7hD5zPH7cLjXo= < /latexit > render at the next location ⇥ < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > h ( ` ) < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > Figure 3 : Rendering from level ` to level ` − 1 in the NRM . At each pixel p in the intermediate image h ( ` ) , NRM renders iff the template selects latent variable s ( ` , p ) = 1 . If rendering , the template Γ ( ` , p ) is multiplied by the pixel value h ( ` , p ) . Then the matrix B ( ` , p ) zero-pads the result to the size of the intermediate image at level ` − 1 . Next , the translation matrix T ( ` , p ) locally translates the rendered image to location specified by the latent variable t ( ` , p ) . The same rendering repeats at other pixels of h ( ` ) . NRM adds all rendered images to obtain the intermediate image h ( ` − 1 ) . where η ( y , z ) , ∑L ` =1 〈b ( t ; ` ) , s ( ` ) h ( ` ) 〉 and Softmax ( η ) , exp ( η ) / ∑ η exp ( η ) . The generation process in the NRM can be summarized in the following steps ( details are in Algorithm 1 in Appendix A ) : 1 ) Sample the latent variables z from a categorical distribution whose prior is πz|y . 2 ) Render its coarsest image , the object template h ( L ) = µ ( y ) of class y . 3 ) Incorporate a set of of latent variations z ( ` ) into h ( y ; ` ) at each layer ` via a linear transformation Λ ( z ; ` ) to render the finer image h ( y , z ; ` − 1 ) . 4 ) Add Gaussian pixel noise into h ( y , z ; 0 ) to render the final image x . In Eqn.2 , if Λ ( z ; ` ) is a linear transformation , it will yield many parameters to learn . As a result , we would like to introduce new structures into NRM so that CNNs can be derived as NRM ’ s inference . This way , NRM will be informed by the prior knowledge of natural images captured by CNNs . In our attempt to invert CNNs , we constrain the latent variables z ( ` ) at layer ` in NRM to a set of template selecting latent variables s ( ` ) and local translation latent variables t ( ` ) . As been shown later in Section 2.2 , during inference of NRM , the ReLU non- linearity at layer ` “ inverts ” s ( ` ) to find if particular features are in the image or not . Similarly , the MaxPool operator “ inverts ” t ( ` ) to locate where particular features , if exist , are in the image . Both s ( ` ) and t ( ` ) are vectors indexed by p ( ` ) ∈ P ( ` ) ≡ { pixels in level ` } . The rendering process from layer ` to layer ` − 1 is given by : h ( ` − 1 ) , Λ ( ` ) h ( ` ) = ∑ p∈P ( ` ) s ( ` , p ) T ( t ; ` , p ) B ( ` , p ) Γ ( ` , p ) h ( ` , p ) . ( 3 ) The rendering process in Eqn.3 is illustrated in Figure 3 . At each pixel p in the intermediate image h ( ` ) at layer ` , NRM decides to use that pixel to render or not according to the value of the template selecting latent variable s ( p ; ` ) at that pixel location . If s ( p ; ` ) = 1 , then NRM renders ; otherwise , it does not . If rendering , the pixel value h ( ` , p ) is used to scale the local rendering template Γ ( ` , p ) , which has the same number of feature maps as the next rendered image h ( ` − 1 ) , but is of smaller size , e.g. , 3× 3 or 5× 5 . Next , the padding matrix n B ( ` , p ) pads the resultant patch to the size of the image h ( ` − 1 ) with zeros , and the translation matrix T ( t ; ` , p ) translates the result to a local location . NRM then keeps rendering at the next pixel location p+ 1 following the same process . All rendered images are added to form the final rendered image h ( ` − 1 ) at layer ` − 1 . Note that we can constrain NRM by enforcing all pixels in the same feature maps of h ( ` ) share the same rendering template . This constraint helps to yield convolutions in CNNs during the inference of NRM , and the rendering template in NRM now corresponds to the convolution filters in CNNs . Reconstructed images at each layer of a 5-layered NRM trained on MNIST are visualized in Figure 2 . The network is trained using the semi-supervised learning framework discussed in Section 2.3 . While s ( ` ) and t ( ` ) can be independent , we further constrain the model by enforcing the dependency among s ( ` ) and t ( ` ) at different layers in NRM . This constraint is motivated from realistic rendering of natural objects : different parts of a natural object are dependent on each other . NRM captures such dependency in natural objects by imposing more structures into the joint prior of latent variables at all layer in the model . The form of the join prior πz|y in Eqn . 1 might look mysterious at first , but NRM parametrizes πz|y in this particular way so that πz|y is the conjugate prior of the model likelihood as being proven in Appendix C.11 . The conjugate form of πz|y allows efficient inference in the NRM . Parameters b ( t ; ` ) of the conjugate prior πz|y will become the bias terms after the CNN ’ s convolutions as will be shown in Theorem 2.2 . When training in an unsupervised setup , the conjugate prior results in the RPN regularization as shown in Theorem 2.3 ( b ) . This RPN regularization helps enforce the dependencies among latent variables and increases the likelihood of latent configuration presents in the data during training . NRM with skip connections : We derive ResNet and DenseNet by adding skip connections into the rendering matrices Λ ( ` ) of NRM ( see Appendix B ) .

Summary: The paper claims to propose a novel generative probabilistic neural network model such that its encoder (classifying an image) can be approximated by a convolutional neural network with ReLU activations and MaxPooling layers. Besides the standard parameters of the units (weights and biases), the model has two additional latent variables per unit, which decide whether and where to put the template (represented by the weights of the neuron) in the subsequent layer, when generating an image from the class. Furthermore, the authors claim to derive new learning criteria for semi-supervised learning of the model including a novel regulariser and claim to prove its consistency.