Neural Rendering Model: Joint Generation and Prediction for Semi-Supervised Learning

1 INTRODUCTION . Unsupervised and semi-supervised learning have still lagged behind compared to performance leaps we have seen in supervised learning over the last five years . This is partly due to a lack of good generative models that can capture all latent variations in complex domains such as natural images and provide useful structures that help learning . When it comes to probabilistic generative models , it is hard to design good priors for the latent variables that drive the generation . Instead , recent approaches avoid the explicit design of image priors . For instance , the Generative Adversarial Networks ( GANs ) use implicit feedback from an additional discriminator that distinguishes real from fake images ( Goodfellow et al. , 2014 ) . Using such feedback helps GANs to generate visually realistic images , but it is not clear if this is the most effective form of feedback for predictive tasks . Moreover , due to separation of generation and discrimination in GANs , there are typically more parameters to train , and this might make it harder to obtain gains for semi-supervised learning in the low ( labeled ) sample setting . We propose an alternative approach to GANs by designing a class of probabilistic generative models , such that inference in those models also has good performance on predictive tasks . This approach is well-suited for semi-supervised learning since it eliminates the need for a separate prediction network . Specifically , we answer the following question : what generative processes output Convolutional Neural Networks ( CNNs ) when inference is carried out ? This is natural to ask since CNNs are stateof-the-art ( SOTA ) predictive models for images , and intuitively , such powerful predictive models should capture some essence of image generation . However , standard CNNs are not directly reversible and likely do not have all the information for generation since they are trained for predictive tasks such as image classification . We can instead invert the irreversible operations in CNNs , e.g. , the rectified linear units ( ReLUs ) and spatial pooling , by assigning auxiliary latent variables to account for uncertainty in the CNN ’ s inversion process due to the information loss . Contribution 1 – Neural Rendering Model : We develop the Neural Rendering Model ( NRM ) whose bottom-up inference corresponds to a CNN architecture of choice ( see Figure 1a ) . The “ reverse ” top-down process of image generation is through coarse-to-fine rendering , which progressively increases the resolution of the rendered image . This is intuitive since the reverse process of bottom-up inference reduces the resolution ( and dimension ) through operations such as spatial pooling . We also introduce structured stochasticity in the rendering process through a small set of discrete latent variables , which capture the uncertainty in reversing the CNN feed-forward process . The rendering in NRM follows a product of linear transformations , which can be considered as the transpose of the inference process in CNNs . In particular , the rendering weights in NRM are proportional to the transposes of the filters in CNNs . Furthermore , the bias terms in the ReLU units at each layer ( after the convolution operator ) make the latent variables in different network layers dependent ( when the bias terms are non-zero ) . This design of image prior has an interesting interpretation from a predictive-coding perspective in neuroscience : the dependency between latent variables can be considered as a form of backward connections that captures prior knowledge from coarser levels in NRM and helps adjust the estimation at the finer levels ( Rao & Ballard , 1999 ; Friston , 2018 ) . NRM is a likelihood-based framework , where unsupervised learning can be derived by maximizing the expected complete-data log-likelihood of the model while supervised learning is done through optimizing the class-conditional log-likelihood . Semi-supervised learning unifies both log-likelihoods into an objective cost for learning from both labeled and unlabeled data . The NRM prior has the desirable property of being a conjugate prior , which makes learning in NRM computationally efficient . Interestingly , we derive the popular cross-entropy loss used to train CNNs for supervised learning as an upper bound of the NRM ’ s negative class-conditional log-likelihood . This new interpretation of cross-entropy allow us to develop better losses for training CNNs . An example is the Max-Min cross-entropy discussed in Contribution 2 and Section 3 . Contribution 2 – New regularization , loss function , architecture and generalization bounds : The joint nature of generation , inference , and learning in NRM allows us to develop new training procedures for semi-supervised and supervised learning , as well as new theoretical ( statistical ) guarantees for learning . In particular , for training , we derive a new form of regularization termed as the Rendering Path Normalization ( RPN ) from the NRM ’ s conjugate prior . A rendering path is a set of latent variable values in NRM . Unlike the path-wise regularizer in ( Neyshabur et al. , 2015 ) , RPN uses information from a generative model to penalizes the number of the possible rendering paths , encouraging the network to be compact in terms of representing the image . It also helps enforce the dependency among different layers in NRM during training and improves classification performance . We provide a new theoretical bound based on NRM . In particular , we prove that NRM is statistically consistent and derive a generalization bound of NRM for ( semi- ) supervised learning tasks . Our generalization bound is proportional to the number of active rendering paths that generate close-to-real images . This suggests that RPN regularization may help in generalization since RPN enforces the dependencies among latent variables in NRM and , therefore , reduces the number of active rendering paths . We observe that RPN helps improve generalization in our experiments . Max-Min cross-entropy and network : We propose the new Max-Min cross-entropy loss function for learning , based on negative class-conditional log-likelihood in NRM . It combines the traditional cross-entropy with another loss , which we term as the Min cross-entropy . While the traditional ( Max ) cross-entropy maximizes the probability of correct labels , the Min cross-entropy minimizes the probability of incorrect labels . We show that the Max-Min cross-entropy is also an upper bound to the negative conditional log-likehood of NRM , just like the cross-entropy loss . The Max-Min cross-entropy is realized through a new CNN architecture , namely the Max-Min network , which is a CNN with an additional branch sharing weights with the original CNN but containing minimum pooling ( MinPool ) operator and negative rectified linear units ( NReLUs ) , i.e. , min ( · , 0 ) ( see Figure 4 ) . Although the Max-Min network is derived from NRM , it is a meta-architecture that can be applied independently on any CNN architecture . We show empirically that Max-Min networks and crossentropy help improve the SOTA on object classification for supervised and semi-supervised learning . Contribution 3 – State-of-the-art empirical results for semi-supervised and supervised learning : We show strong results for semi-supervised learning over CIFAR10 , CIFAR100 and SVHN benchmarks in comparison with SOTA methods that use and do not use consistency regularization . Consistency regularization , such as those used in Temporal Ensembling ( Laine & Aila , 2017 ) and Mean Teacher ( Tarvainen & Valpola , 2017 ) , enforces the networks to learn representation invariant to realistic perturbations of the data . NRM alone outperforms most SOTA methods which do not use consistency regularization ( Salimans et al. , 2016 ; Dumoulin et al. , 2017 ) in most settings . Max-Min cross-entropy then helps improves NRM ’ s semi-supervised learning results significantly . When combining the NRM , Max-Min cross-entropy , and Mean Teacher , we achieve SOTA results or very close to those on CIFAR10 , CIFAR100 , and SVHN ( see Table 2 , 3 , and 4 ) . Interestingly , compared to the other competitors , our method is consistently good , achieving either best or second best results in all experiments . Furthermore , Max-Min cross-entropy also helps supervised learning . Using the Max-Min cross-entropy , we achieve SOTA result for supervised learning on CIFAR10 ( 2.30 % test error ) . Similarly , Max-Min cross-entropy helps improve supervised training on ImageNet . Despite good classification results , there is a caveat that NRM may not generate good looking images since that objective is not “ baked ” into its training . NRM is primarily aimed at improving semi-supervised and supervised learning through better regularization . Potentially , an adversarial loss can be added to NRM to improve visual characteristics of the image . Related Work : In addition to GANs , other recently developed deep generative models include the Variational Autoencoders ( VAE ) ( Kingma & Welling , 2013 ) and the Deep Generative Networks ( Kingma et al. , 2014 ) . Unlike these models , which replace complicated or intractable inference by CNNs , NRM derives CNNs as its inference . This advantage allows us to develop better learning algorithms for CNNs with statistical guarantees , as being discussed in Section 2.3 and 2.4 . Recent works including the Bidirectional GANs ( Donahue et al. , 2017 ) and the Adversarially Learned Inference model ( Dumoulin et al. , 2017 ) try to make the discriminators and generators in GANs reversible of each other , thereby providing an alternative way to invert CNNs . These approaches , nevertheless , still employ a separate network to bypass the irreversible operators in CNNs . NRM is also close in spirit to the Deep Rendering Model ( DRM ) ( Patel et al. , 2016 ) and the Multi-layered ConvolutionalSparse-Coding Model ( ML-CSC ) ( Papyan et al. , 2018 ) but markedly different . Compared to NRM , DRM and ML-CSC have several limitations . In particular , latent variables in DRM are assumed to be independent , which is rather unrealistic . This lack of dependency causes the missing of the bias terms in the ReLUs of the CNN derived from DRM . Furthermore , the cross-entropy loss used in training CNNs for supervised learning tasks is not captured naturally by DRM and ML-CSC . Due to these limitations , model consistency and generalization bounds are not derived for DRM and ML-CSC . Notation : To facilitate the presentation , the NRM ’ s notations are deferred to Table 5 in Appendix A . 2 THE NEURAL RENDERING MODEL . 2.1 GENERATIVE MODEL . The NRM attempts to invert the CNNs as its inference by employing the structure of its latent variables . The dependencies among latent variables in the model are implicitly captured by their conjugate prior distribution . More precisely , NRM can be defined as follows : Definition 2.1 . [ Neural Rendering Model ( NRM ) ] NRM is a deep generative model in which the latent variables z ( ` ) = { t ( ` ) , s ( ` ) } L ` =1 at different layers ` are dependent . Let x be the input image and y ∈ { 1 , . . . , K } be the target variable , e.g . object category . Generation in NRM takes the form πz|y , Softmax ( 1 σ2 η ( y , z ) ) ; z|y ∼ Cat ( πz|y ) ( 1 ) h ( y , z ; 0 ) , Λ ( z ; 1 ) Λ ( z ; 2 ) · · ·Λ ( z ; L ) µ ( y ) ; x|z , y ∼ N ( h ( 0 ) , σ21D ( 0 ) ) , ( 2 ) ⇥ < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > zero-pad translate zero-pad translate ... < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > Rendering Noise ... < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > < latexit sha1_base64= '' ca4An8D0SY3sUfhv6vlZZoxXxlo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg== < /latexit > h ( ` 1 ) < latexit sha1_base64= '' tf2d+fCOcTNB4hDa1H9joi9ZuXo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUgyURQY8FLx4r2A9IQ9lsN+3STTbsToQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8MJXCoOt+O2vrG5tb26Wd8u7e/sFh5ei4bVSmGW8xJZXuhtRwKRLeQoGSd1PNaRxK3gnHdzO/88S1ESp5xEnKg5gOExEJRtFK/qjW41KSS+Jd9CtVt+7OQVaJV5AqFGj2K1+9gWJZzBNkkhrje26KQU41Cib5tNzLDE8pG9Mh9y1NaMxNkM9PnpJzqwxIpLStBMlc/T2R09iYSRzazpjiyCx7M/E/z88wug1ykaQZ8oQtFkWZJKjI7H8yEJozlBNLKNPC3krYiGrK0KZUtiF4yy+vkvZV3XPr3sN1tdEt4ijBKZxBDTy4gQbcQxNawEDBM7zCm4POi/PufCxa15xi5gT+wPn8AS2Kj+4= < /latexit > < latexit sha1_base64= '' tf2d+fCOcTNB4hDa1H9joi9ZuXo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUgyURQY8FLx4r2A9IQ9lsN+3STTbsToQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8MJXCoOt+O2vrG5tb26Wd8u7e/sFh5ei4bVSmGW8xJZXuhtRwKRLeQoGSd1PNaRxK3gnHdzO/88S1ESp5xEnKg5gOExEJRtFK/qjW41KSS+Jd9CtVt+7OQVaJV5AqFGj2K1+9gWJZzBNkkhrje26KQU41Cib5tNzLDE8pG9Mh9y1NaMxNkM9PnpJzqwxIpLStBMlc/T2R09iYSRzazpjiyCx7M/E/z88wug1ykaQZ8oQtFkWZJKjI7H8yEJozlBNLKNPC3krYiGrK0KZUtiF4yy+vkvZV3XPr3sN1tdEt4ijBKZxBDTy4gQbcQxNawEDBM7zCm4POi/PufCxa15xi5gT+wPn8AS2Kj+4= < /latexit > < latexit sha1_base64= '' tf2d+fCOcTNB4hDa1H9joi9ZuXo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUgyURQY8FLx4r2A9IQ9lsN+3STTbsToQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8MJXCoOt+O2vrG5tb26Wd8u7e/sFh5ei4bVSmGW8xJZXuhtRwKRLeQoGSd1PNaRxK3gnHdzO/88S1ESp5xEnKg5gOExEJRtFK/qjW41KSS+Jd9CtVt+7OQVaJV5AqFGj2K1+9gWJZzBNkkhrje26KQU41Cib5tNzLDE8pG9Mh9y1NaMxNkM9PnpJzqwxIpLStBMlc/T2R09iYSRzazpjiyCx7M/E/z88wug1ykaQZ8oQtFkWZJKjI7H8yEJozlBNLKNPC3krYiGrK0KZUtiF4yy+vkvZV3XPr3sN1tdEt4ijBKZxBDTy4gQbcQxNawEDBM7zCm4POi/PufCxa15xi5gT+wPn8AS2Kj+4= < /latexit > < latexit sha1_base64= '' tf2d+fCOcTNB4hDa1H9joi9ZuXo= '' > AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUgyURQY8FLx4r2A9IQ9lsN+3STTbsToQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8MJXCoOt+O2vrG5tb26Wd8u7e/sFh5ei4bVSmGW8xJZXuhtRwKRLeQoGSd1PNaRxK3gnHdzO/88S1ESp5xEnKg5gOExEJRtFK/qjW41KSS+Jd9CtVt+7OQVaJV5AqFGj2K1+9gWJZzBNkkhrje26KQU41Cib5tNzLDE8pG9Mh9y1NaMxNkM9PnpJzqwxIpLStBMlc/T2R09iYSRzazpjiyCx7M/E/z88wug1ykaQZ8oQtFkWZJKjI7H8yEJozlBNLKNPC3krYiGrK0KZUtiF4yy+vkvZV3XPr3sN1tdEt4ijBKZxBDTy4gQbcQxNawEDBM7zCm4POi/PufCxa15xi5gT+wPn8AS2Kj+4= < /latexit > x < latexit sha1_base64= '' TQGEygQLk4MpTONXYfpVxGVfRf8= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeCF48t2A9oQ9lsJ+3azSbsbsQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKLSPJb3ZpqgH9GR5CFn1Fip+TQoV9yquwBZJ15OKpCjMSh/9YcxSyOUhgmqdc9zE+NnVBnOBM5K/VRjQtmEjrBnqaQRaj9bHDojF1YZkjBWtqQhC/X3REYjradRYDsjasZ61ZuL/3m91IQ3fsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXvK7Uu3kcRTiDc7gED2pQhztoQAsYIDzDK7w5D86L8+58LFsLTj5zCn/gfP4A7NeNEg== < /latexit > < latexit sha1_base64= '' TQGEygQLk4MpTONXYfpVxGVfRf8= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeCF48t2A9oQ9lsJ+3azSbsbsQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKLSPJb3ZpqgH9GR5CFn1Fip+TQoV9yquwBZJ15OKpCjMSh/9YcxSyOUhgmqdc9zE+NnVBnOBM5K/VRjQtmEjrBnqaQRaj9bHDojF1YZkjBWtqQhC/X3REYjradRYDsjasZ61ZuL/3m91IQ3fsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXvK7Uu3kcRTiDc7gED2pQhztoQAsYIDzDK7w5D86L8+58LFsLTj5zCn/gfP4A7NeNEg== < /latexit > < latexit sha1_base64= '' TQGEygQLk4MpTONXYfpVxGVfRf8= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeCF48t2A9oQ9lsJ+3azSbsbsQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKLSPJb3ZpqgH9GR5CFn1Fip+TQoV9yquwBZJ15OKpCjMSh/9YcxSyOUhgmqdc9zE+NnVBnOBM5K/VRjQtmEjrBnqaQRaj9bHDojF1YZkjBWtqQhC/X3REYjradRYDsjasZ61ZuL/3m91IQ3fsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXvK7Uu3kcRTiDc7gED2pQhztoQAsYIDzDK7w5D86L8+58LFsLTj5zCn/gfP4A7NeNEg== < /latexit > < latexit sha1_base64= '' TQGEygQLk4MpTONXYfpVxGVfRf8= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeCF48t2A9oQ9lsJ+3azSbsbsQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKLSPJb3ZpqgH9GR5CFn1Fip+TQoV9yquwBZJ15OKpCjMSh/9YcxSyOUhgmqdc9zE+NnVBnOBM5K/VRjQtmEjrBnqaQRaj9bHDojF1YZkjBWtqQhC/X3REYjradRYDsjasZ61ZuL/3m91IQ3fsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXvK7Uu3kcRTiDc7gED2pQhztoQAsYIDzDK7w5D86L8+58LFsLTj5zCn/gfP4A7NeNEg== < /latexit > h ( ` ) < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > p < latexit sha1_base64= '' fhC54FVT5NGEMJ6yFbTk8xYGdwc= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqpYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qjW8RRhjM4h0vw4AYacA9N8IEBwjO8wpvz6Lw4787HsrXkFDOn8AfO5w/gt40K < /latexit > < latexit sha1_base64= '' fhC54FVT5NGEMJ6yFbTk8xYGdwc= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqpYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qjW8RRhjM4h0vw4AYacA9N8IEBwjO8wpvz6Lw4787HsrXkFDOn8AfO5w/gt40K < /latexit > < latexit sha1_base64= '' fhC54FVT5NGEMJ6yFbTk8xYGdwc= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqpYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qjW8RRhjM4h0vw4AYacA9N8IEBwjO8wpvz6Lw4787HsrXkFDOn8AfO5w/gt40K < /latexit > < latexit sha1_base64= '' fhC54FVT5NGEMJ6yFbTk8xYGdwc= '' > AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqpYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qjW8RRhjM4h0vw4AYacA9N8IEBwjO8wpvz6Lw4787HsrXkFDOn8AfO5w/gt40K < /latexit > p + 1 < latexit sha1_base64= '' /ZAggRAtMCulDnXktOjUB3UP/A8= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh4LXjxWtB/QhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zQHbX0w8Hhvhpl5QSKFQdf9dkpr6xubW+Xtys7u3v5B9fCobeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqEacJ9yM6UiIUjKKVHpILb1CtuXU3B1klXkFqUKA5qH71hzFLI66QSWpMz3MT9DOqUTDJZ5V+anhC2YSOeM9SRSNu/Cw/dUbOrDIkYaxtKSS5+nsio5Ex0yiwnRHFsVn25uJ/Xi/F8MbPhEpS5IotFoWpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nYkPwll9eJe3LuufWvfurWqNbxFGGEziFc/DgGhpwB01oAYMRPMMrvDnSeXHenY9Fa8kpZo7hD5zPH7cLjXo= < /latexit > < latexit sha1_base64= '' /ZAggRAtMCulDnXktOjUB3UP/A8= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh4LXjxWtB/QhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zQHbX0w8Hhvhpl5QSKFQdf9dkpr6xubW+Xtys7u3v5B9fCobeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqEacJ9yM6UiIUjKKVHpILb1CtuXU3B1klXkFqUKA5qH71hzFLI66QSWpMz3MT9DOqUTDJZ5V+anhC2YSOeM9SRSNu/Cw/dUbOrDIkYaxtKSS5+nsio5Ex0yiwnRHFsVn25uJ/Xi/F8MbPhEpS5IotFoWpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nYkPwll9eJe3LuufWvfurWqNbxFGGEziFc/DgGhpwB01oAYMRPMMrvDnSeXHenY9Fa8kpZo7hD5zPH7cLjXo= < /latexit > < latexit sha1_base64= '' /ZAggRAtMCulDnXktOjUB3UP/A8= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh4LXjxWtB/QhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zQHbX0w8Hhvhpl5QSKFQdf9dkpr6xubW+Xtys7u3v5B9fCobeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqEacJ9yM6UiIUjKKVHpILb1CtuXU3B1klXkFqUKA5qH71hzFLI66QSWpMz3MT9DOqUTDJZ5V+anhC2YSOeM9SRSNu/Cw/dUbOrDIkYaxtKSS5+nsio5Ex0yiwnRHFsVn25uJ/Xi/F8MbPhEpS5IotFoWpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nYkPwll9eJe3LuufWvfurWqNbxFGGEziFc/DgGhpwB01oAYMRPMMrvDnSeXHenY9Fa8kpZo7hD5zPH7cLjXo= < /latexit > < latexit sha1_base64= '' /ZAggRAtMCulDnXktOjUB3UP/A8= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh4LXjxWtB/QhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zQHbX0w8Hhvhpl5QSKFQdf9dkpr6xubW+Xtys7u3v5B9fCobeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqEacJ9yM6UiIUjKKVHpILb1CtuXU3B1klXkFqUKA5qH71hzFLI66QSWpMz3MT9DOqUTDJZ5V+anhC2YSOeM9SRSNu/Cw/dUbOrDIkYaxtKSS5+nsio5Ex0yiwnRHFsVn25uJ/Xi/F8MbPhEpS5IotFoWpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nYkPwll9eJe3LuufWvfurWqNbxFGGEziFc/DgGhpwB01oAYMRPMMrvDnSeXHenY9Fa8kpZo7hD5zPH7cLjXo= < /latexit > render at the next location ⇥ < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > < latexit sha1_base64= '' ZshQRDCewTBC9osbqPvttQWWNL4= '' > AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K < /latexit > h ( ` ) < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > < latexit sha1_base64= '' e/CdV7XsMf2Xh+89wrgoEpUAHLQ= '' > AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg= < /latexit > Figure 3 : Rendering from level ` to level ` − 1 in the NRM . At each pixel p in the intermediate image h ( ` ) , NRM renders iff the template selects latent variable s ( ` , p ) = 1 . If rendering , the template Γ ( ` , p ) is multiplied by the pixel value h ( ` , p ) . Then the matrix B ( ` , p ) zero-pads the result to the size of the intermediate image at level ` − 1 . Next , the translation matrix T ( ` , p ) locally translates the rendered image to location specified by the latent variable t ( ` , p ) . The same rendering repeats at other pixels of h ( ` ) . NRM adds all rendered images to obtain the intermediate image h ( ` − 1 ) . where η ( y , z ) , ∑L ` =1 〈b ( t ; ` ) , s ( ` ) h ( ` ) 〉 and Softmax ( η ) , exp ( η ) / ∑ η exp ( η ) . The generation process in the NRM can be summarized in the following steps ( details are in Algorithm 1 in Appendix A ) : 1 ) Sample the latent variables z from a categorical distribution whose prior is πz|y . 2 ) Render its coarsest image , the object template h ( L ) = µ ( y ) of class y . 3 ) Incorporate a set of of latent variations z ( ` ) into h ( y ; ` ) at each layer ` via a linear transformation Λ ( z ; ` ) to render the finer image h ( y , z ; ` − 1 ) . 4 ) Add Gaussian pixel noise into h ( y , z ; 0 ) to render the final image x . In Eqn.2 , if Λ ( z ; ` ) is a linear transformation , it will yield many parameters to learn . As a result , we would like to introduce new structures into NRM so that CNNs can be derived as NRM ’ s inference . This way , NRM will be informed by the prior knowledge of natural images captured by CNNs . In our attempt to invert CNNs , we constrain the latent variables z ( ` ) at layer ` in NRM to a set of template selecting latent variables s ( ` ) and local translation latent variables t ( ` ) . As been shown later in Section 2.2 , during inference of NRM , the ReLU non- linearity at layer ` “ inverts ” s ( ` ) to find if particular features are in the image or not . Similarly , the MaxPool operator “ inverts ” t ( ` ) to locate where particular features , if exist , are in the image . Both s ( ` ) and t ( ` ) are vectors indexed by p ( ` ) ∈ P ( ` ) ≡ { pixels in level ` } . The rendering process from layer ` to layer ` − 1 is given by : h ( ` − 1 ) , Λ ( ` ) h ( ` ) = ∑ p∈P ( ` ) s ( ` , p ) T ( t ; ` , p ) B ( ` , p ) Γ ( ` , p ) h ( ` , p ) . ( 3 ) The rendering process in Eqn.3 is illustrated in Figure 3 . At each pixel p in the intermediate image h ( ` ) at layer ` , NRM decides to use that pixel to render or not according to the value of the template selecting latent variable s ( p ; ` ) at that pixel location . If s ( p ; ` ) = 1 , then NRM renders ; otherwise , it does not . If rendering , the pixel value h ( ` , p ) is used to scale the local rendering template Γ ( ` , p ) , which has the same number of feature maps as the next rendered image h ( ` − 1 ) , but is of smaller size , e.g. , 3× 3 or 5× 5 . Next , the padding matrix n B ( ` , p ) pads the resultant patch to the size of the image h ( ` − 1 ) with zeros , and the translation matrix T ( t ; ` , p ) translates the result to a local location . NRM then keeps rendering at the next pixel location p+ 1 following the same process . All rendered images are added to form the final rendered image h ( ` − 1 ) at layer ` − 1 . Note that we can constrain NRM by enforcing all pixels in the same feature maps of h ( ` ) share the same rendering template . This constraint helps to yield convolutions in CNNs during the inference of NRM , and the rendering template in NRM now corresponds to the convolution filters in CNNs . Reconstructed images at each layer of a 5-layered NRM trained on MNIST are visualized in Figure 2 . The network is trained using the semi-supervised learning framework discussed in Section 2.3 . While s ( ` ) and t ( ` ) can be independent , we further constrain the model by enforcing the dependency among s ( ` ) and t ( ` ) at different layers in NRM . This constraint is motivated from realistic rendering of natural objects : different parts of a natural object are dependent on each other . NRM captures such dependency in natural objects by imposing more structures into the joint prior of latent variables at all layer in the model . The form of the join prior πz|y in Eqn . 1 might look mysterious at first , but NRM parametrizes πz|y in this particular way so that πz|y is the conjugate prior of the model likelihood as being proven in Appendix C.11 . The conjugate form of πz|y allows efficient inference in the NRM . Parameters b ( t ; ` ) of the conjugate prior πz|y will become the bias terms after the CNN ’ s convolutions as will be shown in Theorem 2.2 . When training in an unsupervised setup , the conjugate prior results in the RPN regularization as shown in Theorem 2.3 ( b ) . This RPN regularization helps enforce the dependencies among latent variables and increases the likelihood of latent configuration presents in the data during training . NRM with skip connections : We derive ResNet and DenseNet by adding skip connections into the rendering matrices Λ ( ` ) of NRM ( see Appendix B ) .

Summary: This paper introduces the Neural Rendering Model (NRM), a generative model in which the computations involved in inference correspond to those of a CNN forward pass. The NRM’s supervised learning objective is lower bounded by a variant of the cross-entropy objective. This objective is used to formulate a max-min network, which has a particular type of weight sharing between a standard branch with max pooling / ReLUs and a second branch with min pooling / NReLUs. The max-min objective and network show strong performance on semi-supervised learning tasks.