Year: 2018

Abstract:
Unsupervised and semi-supervised learning are important problems that are especially challenging with complex data like natural images. Progress on these problems would accelerate if we had access to appropriate generative models under which to pose the associated inference tasks. Inspired by the success of Convolutional Neural Networks (CNNs) for supervised prediction in images, we design the Neural Rendering Model (NRM), a new hierarchical probabilistic generative model whose inference calculations correspond to those in a CNN. The NRM introduces a small set of latent variables at each level of the model and enforces dependencies among all the latent variables via a conjugate prior distribution. The conjugate prior yields a new regularizer for learning based on the paths rendered in the generative model for training CNNs–the Rendering Path Normalization (RPN). We demonstrate that this regularizer improves generalization both in theory and in practice. Likelihood estimation in the NRM yields the new Max-Min cross entropy training loss, which suggests a new deep network architecture–the MaxMin network–which exceeds or matches the state-of-art for semi-supervised and supervised learning on SVHN, CIFAR10, and CIFAR100.

heading: 1 INTRODUCTION
text: Unsupervised and semi-supervised learning have still lagged behind compared to performance leaps we have seen in supervised learning over the last five years. This is partly due to a lack of good generative models that can capture all latent variations in complex domains such as natural images and provide useful structures that help learning. When it comes to probabilistic generative models, it is hard to design good priors for the latent variables that drive the generation.
Instead, recent approaches avoid the explicit design of image priors. For instance, the Generative Adversarial Networks (GANs) use implicit feedback from an additional discriminator that distinguishes real from fake images (Goodfellow et al., 2014). Using such feedback helps GANs to generate visually realistic images, but it is not clear if this is the most effective form of feedback for predictive tasks. Moreover, due to separation of generation and discrimination in GANs, there are typically more parameters to train, and this might make it harder to obtain gains for semi-supervised learning in the low (labeled) sample setting.
We propose an alternative approach to GANs by designing a class of probabilistic generative models, such that inference in those models also has good performance on predictive tasks. This approach is well-suited for semi-supervised learning since it eliminates the need for a separate prediction network. Specifically, we answer the following question: what generative processes output Convolutional Neural Networks (CNNs) when inference is carried out? This is natural to ask since CNNs are stateof-the-art (SOTA) predictive models for images, and intuitively, such powerful predictive models should capture some essence of image generation. However, standard CNNs are not directly reversible and likely do not have all the information for generation since they are trained for predictive tasks such as image classification. We can instead invert the irreversible operations in CNNs, e.g., the rectified linear units (ReLUs) and spatial pooling, by assigning auxiliary latent variables to account for uncertainty in the CNN’s inversion process due to the information loss.
Contribution 1 – Neural Rendering Model: We develop the Neural Rendering Model (NRM) whose bottom-up inference corresponds to a CNN architecture of choice (see Figure 1a). The “reverse”
top-down process of image generation is through coarse-to-fine rendering, which progressively increases the resolution of the rendered image. This is intuitive since the reverse process of bottom-up inference reduces the resolution (and dimension) through operations such as spatial pooling. We also introduce structured stochasticity in the rendering process through a small set of discrete latent variables, which capture the uncertainty in reversing the CNN feed-forward process. The rendering in NRM follows a product of linear transformations, which can be considered as the transpose of the inference process in CNNs. In particular, the rendering weights in NRM are proportional to the transposes of the filters in CNNs. Furthermore, the bias terms in the ReLU units at each layer (after the convolution operator) make the latent variables in different network layers dependent (when the bias terms are non-zero). This design of image prior has an interesting interpretation from a predictive-coding perspective in neuroscience: the dependency between latent variables can be considered as a form of backward connections that captures prior knowledge from coarser levels in NRM and helps adjust the estimation at the finer levels (Rao & Ballard, 1999; Friston, 2018).
NRM is a likelihood-based framework, where unsupervised learning can be derived by maximizing the expected complete-data log-likelihood of the model while supervised learning is done through optimizing the class-conditional log-likelihood. Semi-supervised learning unifies both log-likelihoods into an objective cost for learning from both labeled and unlabeled data. The NRM prior has the desirable property of being a conjugate prior, which makes learning in NRM computationally efficient.
Interestingly, we derive the popular cross-entropy loss used to train CNNs for supervised learning as an upper bound of the NRM’s negative class-conditional log-likelihood. This new interpretation of cross-entropy allow us to develop better losses for training CNNs. An example is the Max-Min cross-entropy discussed in Contribution 2 and Section 3.
Contribution 2 – New regularization, loss function, architecture and generalization bounds: The joint nature of generation, inference, and learning in NRM allows us to develop new training procedures for semi-supervised and supervised learning, as well as new theoretical (statistical) guarantees for learning. In particular, for training, we derive a new form of regularization termed as the Rendering Path Normalization (RPN) from the NRM’s conjugate prior. A rendering path is a set of latent variable values in NRM. Unlike the path-wise regularizer in (Neyshabur et al., 2015), RPN uses information from a generative model to penalizes the number of the possible rendering paths, encouraging the network to be compact in terms of representing the image. It also helps enforce the dependency among different layers in NRM during training and improves classification performance.
We provide a new theoretical bound based on NRM. In particular, we prove that NRM is statistically consistent and derive a generalization bound of NRM for (semi-)supervised learning tasks. Our generalization bound is proportional to the number of active rendering paths that generate close-to-real images. This suggests that RPN regularization may help in generalization since RPN enforces the dependencies among latent variables in NRM and, therefore, reduces the number of active rendering paths. We observe that RPN helps improve generalization in our experiments.
Max-Min cross-entropy and network: We propose the new Max-Min cross-entropy loss function for learning, based on negative class-conditional log-likelihood in NRM. It combines the traditional cross-entropy with another loss, which we term as the Min cross-entropy. While the traditional (Max) cross-entropy maximizes the probability of correct labels, the Min cross-entropy minimizes
the probability of incorrect labels. We show that the Max-Min cross-entropy is also an upper bound to the negative conditional log-likehood of NRM, just like the cross-entropy loss. The Max-Min cross-entropy is realized through a new CNN architecture, namely the Max-Min network, which is a CNN with an additional branch sharing weights with the original CNN but containing minimum pooling (MinPool) operator and negative rectified linear units (NReLUs), i.e., min(·, 0) (see Figure 4). Although the Max-Min network is derived from NRM, it is a meta-architecture that can be applied independently on any CNN architecture. We show empirically that Max-Min networks and crossentropy help improve the SOTA on object classification for supervised and semi-supervised learning.
Contribution 3 – State-of-the-art empirical results for semi-supervised and supervised learning: We show strong results for semi-supervised learning over CIFAR10, CIFAR100 and SVHN benchmarks in comparison with SOTA methods that use and do not use consistency regularization. Consistency regularization, such as those used in Temporal Ensembling (Laine & Aila, 2017) and Mean Teacher (Tarvainen & Valpola, 2017), enforces the networks to learn representation invariant to realistic perturbations of the data. NRM alone outperforms most SOTA methods which do not use consistency regularization (Salimans et al., 2016; Dumoulin et al., 2017) in most settings. Max-Min cross-entropy then helps improves NRM’s semi-supervised learning results significantly. When combining the NRM, Max-Min cross-entropy, and Mean Teacher, we achieve SOTA results or very close to those on CIFAR10, CIFAR100, and SVHN (see Table 2, 3, and 4). Interestingly, compared to the other competitors, our method is consistently good, achieving either best or second best results in all experiments. Furthermore, Max-Min cross-entropy also helps supervised learning. Using the Max-Min cross-entropy, we achieve SOTA result for supervised learning on CIFAR10 (2.30% test error). Similarly, Max-Min cross-entropy helps improve supervised training on ImageNet.
Despite good classification results, there is a caveat that NRM may not generate good looking images since that objective is not “baked” into its training. NRM is primarily aimed at improving semi-supervised and supervised learning through better regularization. Potentially, an adversarial loss can be added to NRM to improve visual characteristics of the image.
Related Work: In addition to GANs, other recently developed deep generative models include the Variational Autoencoders (VAE) (Kingma & Welling, 2013) and the Deep Generative Networks (Kingma et al., 2014). Unlike these models, which replace complicated or intractable inference by CNNs, NRM derives CNNs as its inference. This advantage allows us to develop better learning algorithms for CNNs with statistical guarantees, as being discussed in Section 2.3 and 2.4. Recent works including the Bidirectional GANs (Donahue et al., 2017) and the Adversarially Learned Inference model (Dumoulin et al., 2017) try to make the discriminators and generators in GANs reversible of each other, thereby providing an alternative way to invert CNNs. These approaches, nevertheless, still employ a separate network to bypass the irreversible operators in CNNs. NRM is also close in spirit to the Deep Rendering Model (DRM) (Patel et al., 2016) and the Multi-layered ConvolutionalSparse-Coding Model (ML-CSC) (Papyan et al., 2018) but markedly different. Compared to NRM, DRM and ML-CSC have several limitations. In particular, latent variables in DRM are assumed to be independent, which is rather unrealistic. This lack of dependency causes the missing of the bias terms in the ReLUs of the CNN derived from DRM. Furthermore, the cross-entropy loss used in training CNNs for supervised learning tasks is not captured naturally by DRM and ML-CSC. Due to these limitations, model consistency and generalization bounds are not derived for DRM and ML-CSC.
Notation: To facilitate the presentation, the NRM’s notations are deferred to Table 5 in Appendix A.

heading: 2 THE NEURAL RENDERING MODEL

heading: 2.1 GENERATIVE MODEL
text: The NRM attempts to invert the CNNs as its inference by employing the structure of its latent variables. The dependencies among latent variables in the model are implicitly captured by their conjugate prior distribution. More precisely, NRM can be defined as follows: Definition 2.1. [Neural Rendering Model (NRM)] NRM is a deep generative model in which the latent variables z(`) = {t(`), s(`)}L`=1 at different layers ` are dependent. Let x be the input image and y ∈ {1, . . . ,K} be the target variable, e.g. object category. Generation in NRM takes the form
πz|y , Softmax ( 1
σ2 η(y, z)
) ; z|y ∼ Cat(πz|y) (1)
h(y, z; 0) , Λ(z; 1)Λ(z; 2) · · ·Λ(z;L)µ(y); x|z, y ∼ N (h(0), σ21D(0)), (2)
⇥<latexit sha1_base64="ZshQRDCewTBC9osbqPvttQWWNL4=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K</latexit><latexit sha1_base64="ZshQRDCewTBC9osbqPvttQWWNL4=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K</latexit><latexit sha1_base64="ZshQRDCewTBC9osbqPvttQWWNL4=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K</latexit><latexit sha1_base64="ZshQRDCewTBC9osbqPvttQWWNL4=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K</latexit> zero-pad translate
zero-pad translate ...<latexit sha1_base64="ca4An8D0SY3sUfhv6vlZZoxXxlo=">AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg==</latexit><latexit sha1_base64="ca4An8D0SY3sUfhv6vlZZoxXxlo=">AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg==</latexit><latexit sha1_base64="ca4An8D0SY3sUfhv6vlZZoxXxlo=">AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg==</latexit><latexit sha1_base64="ca4An8D0SY3sUfhv6vlZZoxXxlo=">AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg==</latexit>
Rendering Noise
...<latexit sha1_base64="ca4An8D0SY3sUfhv6vlZZoxXxlo=">AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg==</latexit><latexit sha1_base64="ca4An8D0SY3sUfhv6vlZZoxXxlo=">AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg==</latexit><latexit sha1_base64="ca4An8D0SY3sUfhv6vlZZoxXxlo=">AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg==</latexit><latexit sha1_base64="ca4An8D0SY3sUfhv6vlZZoxXxlo=">AAAB8nicbVBNS8NAEJ34WetX1aOXYBE8lUQEPRa8eKxgP6AJZbPZtEs3u2F3UiihP8OLB0W8+mu8+W/ctjlo64OBx3szzMyLMsENet63s7G5tb2zW9mr7h8cHh3XTk47RuWasjZVQuleRAwTXLI2chSsl2lG0kiwbjS+n/vdCdOGK/mE04yFKRlKnnBK0Er9IEqKYBIrNLNBre41vAXcdeKXpA4lWoPaVxArmqdMIhXEmL7vZRgWRCOngs2qQW5YRuiYDFnfUklSZsJicfLMvbRK7CZK25LoLtTfEwVJjZmmke1MCY7MqjcX//P6OSZ3YcFlliOTdLkoyYWLyp3/78ZcM4piagmhmttbXToimlC0KVVtCP7qy+ukc93wvYb/eFNv9so4KnAOF3AFPtxCEx6gBW2goOAZXuHNQefFeXc+lq0bTjlzBn/gfP4AzZaRpg==</latexit>
h(` 1) <latexit sha1_base64="tf2d+fCOcTNB4hDa1H9joi9ZuXo=">AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUgyURQY8FLx4r2A9IQ9lsN+3STTbsToQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8MJXCoOt+O2vrG5tb26Wd8u7e/sFh5ei4bVSmGW8xJZXuhtRwKRLeQoGSd1PNaRxK3gnHdzO/88S1ESp5xEnKg5gOExEJRtFK/qjW41KSS+Jd9CtVt+7OQVaJV5AqFGj2K1+9gWJZzBNkkhrje26KQU41Cib5tNzLDE8pG9Mh9y1NaMxNkM9PnpJzqwxIpLStBMlc/T2R09iYSRzazpjiyCx7M/E/z88wug1ykaQZ8oQtFkWZJKjI7H8yEJozlBNLKNPC3krYiGrK0KZUtiF4yy+vkvZV3XPr3sN1tdEt4ijBKZxBDTy4gQbcQxNawEDBM7zCm4POi/PufCxa15xi5gT+wPn8AS2Kj+4=</latexit><latexit sha1_base64="tf2d+fCOcTNB4hDa1H9joi9ZuXo=">AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUgyURQY8FLx4r2A9IQ9lsN+3STTbsToQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8MJXCoOt+O2vrG5tb26Wd8u7e/sFh5ei4bVSmGW8xJZXuhtRwKRLeQoGSd1PNaRxK3gnHdzO/88S1ESp5xEnKg5gOExEJRtFK/qjW41KSS+Jd9CtVt+7OQVaJV5AqFGj2K1+9gWJZzBNkkhrje26KQU41Cib5tNzLDE8pG9Mh9y1NaMxNkM9PnpJzqwxIpLStBMlc/T2R09iYSRzazpjiyCx7M/E/z88wug1ykaQZ8oQtFkWZJKjI7H8yEJozlBNLKNPC3krYiGrK0KZUtiF4yy+vkvZV3XPr3sN1tdEt4ijBKZxBDTy4gQbcQxNawEDBM7zCm4POi/PufCxa15xi5gT+wPn8AS2Kj+4=</latexit><latexit sha1_base64="tf2d+fCOcTNB4hDa1H9joi9ZuXo=">AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUgyURQY8FLx4r2A9IQ9lsN+3STTbsToQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8MJXCoOt+O2vrG5tb26Wd8u7e/sFh5ei4bVSmGW8xJZXuhtRwKRLeQoGSd1PNaRxK3gnHdzO/88S1ESp5xEnKg5gOExEJRtFK/qjW41KSS+Jd9CtVt+7OQVaJV5AqFGj2K1+9gWJZzBNkkhrje26KQU41Cib5tNzLDE8pG9Mh9y1NaMxNkM9PnpJzqwxIpLStBMlc/T2R09iYSRzazpjiyCx7M/E/z88wug1ykaQZ8oQtFkWZJKjI7H8yEJozlBNLKNPC3krYiGrK0KZUtiF4yy+vkvZV3XPr3sN1tdEt4ijBKZxBDTy4gQbcQxNawEDBM7zCm4POi/PufCxa15xi5gT+wPn8AS2Kj+4=</latexit><latexit sha1_base64="tf2d+fCOcTNB4hDa1H9joi9ZuXo=">AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUgyURQY8FLx4r2A9IQ9lsN+3STTbsToQS+jO8eFDEq7/Gm//GbZuDtj4YeLw3w8y8MJXCoOt+O2vrG5tb26Wd8u7e/sFh5ei4bVSmGW8xJZXuhtRwKRLeQoGSd1PNaRxK3gnHdzO/88S1ESp5xEnKg5gOExEJRtFK/qjW41KSS+Jd9CtVt+7OQVaJV5AqFGj2K1+9gWJZzBNkkhrje26KQU41Cib5tNzLDE8pG9Mh9y1NaMxNkM9PnpJzqwxIpLStBMlc/T2R09iYSRzazpjiyCx7M/E/z88wug1ykaQZ8oQtFkWZJKjI7H8yEJozlBNLKNPC3krYiGrK0KZUtiF4yy+vkvZV3XPr3sN1tdEt4ijBKZxBDTy4gQbcQxNawEDBM7zCm4POi/PufCxa15xi5gT+wPn8AS2Kj+4=</latexit>
x <latexit sha1_base64="TQGEygQLk4MpTONXYfpVxGVfRf8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeCF48t2A9oQ9lsJ+3azSbsbsQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKLSPJb3ZpqgH9GR5CFn1Fip+TQoV9yquwBZJ15OKpCjMSh/9YcxSyOUhgmqdc9zE+NnVBnOBM5K/VRjQtmEjrBnqaQRaj9bHDojF1YZkjBWtqQhC/X3REYjradRYDsjasZ61ZuL/3m91IQ3fsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXvK7Uu3kcRTiDc7gED2pQhztoQAsYIDzDK7w5D86L8+58LFsLTj5zCn/gfP4A7NeNEg==</latexit><latexit sha1_base64="TQGEygQLk4MpTONXYfpVxGVfRf8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeCF48t2A9oQ9lsJ+3azSbsbsQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKLSPJb3ZpqgH9GR5CFn1Fip+TQoV9yquwBZJ15OKpCjMSh/9YcxSyOUhgmqdc9zE+NnVBnOBM5K/VRjQtmEjrBnqaQRaj9bHDojF1YZkjBWtqQhC/X3REYjradRYDsjasZ61ZuL/3m91IQ3fsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXvK7Uu3kcRTiDc7gED2pQhztoQAsYIDzDK7w5D86L8+58LFsLTj5zCn/gfP4A7NeNEg==</latexit><latexit sha1_base64="TQGEygQLk4MpTONXYfpVxGVfRf8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeCF48t2A9oQ9lsJ+3azSbsbsQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKLSPJb3ZpqgH9GR5CFn1Fip+TQoV9yquwBZJ15OKpCjMSh/9YcxSyOUhgmqdc9zE+NnVBnOBM5K/VRjQtmEjrBnqaQRaj9bHDojF1YZkjBWtqQhC/X3REYjradRYDsjasZ61ZuL/3m91IQ3fsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXvK7Uu3kcRTiDc7gED2pQhztoQAsYIDzDK7w5D86L8+58LFsLTj5zCn/gfP4A7NeNEg==</latexit><latexit sha1_base64="TQGEygQLk4MpTONXYfpVxGVfRf8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeCF48t2A9oQ9lsJ+3azSbsbsQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKLSPJb3ZpqgH9GR5CFn1Fip+TQoV9yquwBZJ15OKpCjMSh/9YcxSyOUhgmqdc9zE+NnVBnOBM5K/VRjQtmEjrBnqaQRaj9bHDojF1YZkjBWtqQhC/X3REYjradRYDsjasZ61ZuL/3m91IQ3fsZlkhqUbLkoTAUxMZl/TYZcITNiagllittbCRtTRZmx2ZRsCN7qy+ukfVX13KrXvK7Uu3kcRTiDc7gED2pQhztoQAsYIDzDK7w5D86L8+58LFsLTj5zCn/gfP4A7NeNEg==</latexit>
h(`) <latexit sha1_base64="e/CdV7XsMf2Xh+89wrgoEpUAHLQ=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg=</latexit><latexit sha1_base64="e/CdV7XsMf2Xh+89wrgoEpUAHLQ=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg=</latexit><latexit sha1_base64="e/CdV7XsMf2Xh+89wrgoEpUAHLQ=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg=</latexit><latexit sha1_base64="e/CdV7XsMf2Xh+89wrgoEpUAHLQ=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg=</latexit>
p <latexit sha1_base64="fhC54FVT5NGEMJ6yFbTk8xYGdwc=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqpYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qjW8RRhjM4h0vw4AYacA9N8IEBwjO8wpvz6Lw4787HsrXkFDOn8AfO5w/gt40K</latexit><latexit sha1_base64="fhC54FVT5NGEMJ6yFbTk8xYGdwc=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqpYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qjW8RRhjM4h0vw4AYacA9N8IEBwjO8wpvz6Lw4787HsrXkFDOn8AfO5w/gt40K</latexit><latexit sha1_base64="fhC54FVT5NGEMJ6yFbTk8xYGdwc=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqpYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qjW8RRhjM4h0vw4AYacA9N8IEBwjO8wpvz6Lw4787HsrXkFDOn8AfO5w/gt40K</latexit><latexit sha1_base64="fhC54FVT5NGEMJ6yFbTk8xYGdwc=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqpYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qjW8RRhjM4h0vw4AYacA9N8IEBwjO8wpvz6Lw4787HsrXkFDOn8AfO5w/gt40K</latexit>
p + 1 <latexit sha1_base64="/ZAggRAtMCulDnXktOjUB3UP/A8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh4LXjxWtB/QhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zQHbX0w8Hhvhpl5QSKFQdf9dkpr6xubW+Xtys7u3v5B9fCobeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqEacJ9yM6UiIUjKKVHpILb1CtuXU3B1klXkFqUKA5qH71hzFLI66QSWpMz3MT9DOqUTDJZ5V+anhC2YSOeM9SRSNu/Cw/dUbOrDIkYaxtKSS5+nsio5Ex0yiwnRHFsVn25uJ/Xi/F8MbPhEpS5IotFoWpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nYkPwll9eJe3LuufWvfurWqNbxFGGEziFc/DgGhpwB01oAYMRPMMrvDnSeXHenY9Fa8kpZo7hD5zPH7cLjXo=</latexit><latexit sha1_base64="/ZAggRAtMCulDnXktOjUB3UP/A8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh4LXjxWtB/QhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zQHbX0w8Hhvhpl5QSKFQdf9dkpr6xubW+Xtys7u3v5B9fCobeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqEacJ9yM6UiIUjKKVHpILb1CtuXU3B1klXkFqUKA5qH71hzFLI66QSWpMz3MT9DOqUTDJZ5V+anhC2YSOeM9SRSNu/Cw/dUbOrDIkYaxtKSS5+nsio5Ex0yiwnRHFsVn25uJ/Xi/F8MbPhEpS5IotFoWpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nYkPwll9eJe3LuufWvfurWqNbxFGGEziFc/DgGhpwB01oAYMRPMMrvDnSeXHenY9Fa8kpZo7hD5zPH7cLjXo=</latexit><latexit sha1_base64="/ZAggRAtMCulDnXktOjUB3UP/A8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh4LXjxWtB/QhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zQHbX0w8Hhvhpl5QSKFQdf9dkpr6xubW+Xtys7u3v5B9fCobeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqEacJ9yM6UiIUjKKVHpILb1CtuXU3B1klXkFqUKA5qH71hzFLI66QSWpMz3MT9DOqUTDJZ5V+anhC2YSOeM9SRSNu/Cw/dUbOrDIkYaxtKSS5+nsio5Ex0yiwnRHFsVn25uJ/Xi/F8MbPhEpS5IotFoWpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nYkPwll9eJe3LuufWvfurWqNbxFGGEziFc/DgGhpwB01oAYMRPMMrvDnSeXHenY9Fa8kpZo7hD5zPH7cLjXo=</latexit><latexit sha1_base64="/ZAggRAtMCulDnXktOjUB3UP/A8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBZBEEoigh4LXjxWtB/QhrLZbtqlm03YnQgl9Cd48aCIV3+RN/+N2zQHbX0w8Hhvhpl5QSKFQdf9dkpr6xubW+Xtys7u3v5B9fCobeJUM95isYx1N6CGS6F4CwVK3k00p1EgeSeY3M79zhPXRsTqEacJ9yM6UiIUjKKVHpILb1CtuXU3B1klXkFqUKA5qH71hzFLI66QSWpMz3MT9DOqUTDJZ5V+anhC2YSOeM9SRSNu/Cw/dUbOrDIkYaxtKSS5+nsio5Ex0yiwnRHFsVn25uJ/Xi/F8MbPhEpS5IotFoWpJBiT+d9kKDRnKKeWUKaFvZWwMdWUoU2nYkPwll9eJe3LuufWvfurWqNbxFGGEziFc/DgGhpwB01oAYMRPMMrvDnSeXHenY9Fa8kpZo7hD5zPH7cLjXo=</latexit>
render at the next location
⇥<latexit sha1_base64="ZshQRDCewTBC9osbqPvttQWWNL4=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K</latexit><latexit sha1_base64="ZshQRDCewTBC9osbqPvttQWWNL4=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K</latexit><latexit sha1_base64="ZshQRDCewTBC9osbqPvttQWWNL4=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K</latexit><latexit sha1_base64="ZshQRDCewTBC9osbqPvttQWWNL4=">AAAB7XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bjw4jGCeUCyhNnJbDJmdmaZ6RXCkn/w4kERr/6PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+nfntJ26s0OoBJykPEzpUIhaMopNaPRQJt/1K1a/5c5BVEhSkCgUa/cpXb6BZlnCFTFJru4GfYphTg4JJPi33MstTysZ0yLuOKuqWhPn82ik5d8qAxNq4Ukjm6u+JnCbWTpLIdSYUR3bZm4n/ed0M45swFyrNkCu2WBRnkqAms9fJQBjOUE4cocwIdythI2ooQxdQ2YUQLL+8SlqXtcCvBfdX1XqniKMEp3AGFxDANdThDhrQBAaP8Ayv8OZp78V79z4WrWteMXMCf+B9/gC8yY9K</latexit>h(`) <latexit sha1_base64="e/CdV7XsMf2Xh+89wrgoEpUAHLQ=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg=</latexit><latexit sha1_base64="e/CdV7XsMf2Xh+89wrgoEpUAHLQ=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg=</latexit><latexit sha1_base64="e/CdV7XsMf2Xh+89wrgoEpUAHLQ=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg=</latexit><latexit sha1_base64="e/CdV7XsMf2Xh+89wrgoEpUAHLQ=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBahXkoigh4LXjxWsB/QhrLZTtqlm03Y3Qgl9Ed48aCIV3+PN/+N2zQHbX0w8Hhvhpl5QSK4Nq777ZQ2Nre2d8q7lb39g8Oj6vFJR8epYthmsYhVL6AaBZfYNtwI7CUKaRQI7AbTu4XffUKleSwfzSxBP6JjyUPOqLFSd1IfoBCXw2rNbbg5yDrxClKDAq1h9WswilkaoTRMUK37npsYP6PKcCZwXhmkGhPKpnSMfUsljVD7WX7unFxYZUTCWNmShuTq74mMRlrPosB2RtRM9Kq3EP/z+qkJb/2MyyQ1KNlyUZgKYmKy+J2MuEJmxMwSyhS3txI2oYoyYxOq2BC81ZfXSeeq4bkN7+G61uwVcZThDM6hDh7cQBPuoQVtYDCFZ3iFNydxXpx352PZWnKKmVP4A+fzB6AMjyg=</latexit>
Figure 3: Rendering from level ` to level `− 1 in the NRM. At each pixel p in the intermediate image h(`), NRM renders iff the template selects latent variable s(`, p) = 1. If rendering, the template Γ(`, p) is multiplied by the pixel value h(`, p). Then the matrix B(`, p) zero-pads the result to the size of the intermediate image at level `− 1. Next, the translation matrix T (`, p) locally translates the rendered image to location specified by the latent variable t(`, p). The same rendering repeats at other pixels of h(`). NRM adds all rendered images to obtain the intermediate image h(`− 1).
where η(y, z) , ∑L `=1 〈b(t; `), s(`) h(`)〉 and Softmax (η) , exp(η)/ ∑ η exp(η).
The generation process in the NRM can be summarized in the following steps (details are in Algorithm 1 in Appendix A):
1) Sample the latent variables z from a categorical distribution whose prior is πz|y. 2) Render its coarsest image, the object template h(L) = µ(y) of class y. 3) Incorporate a set of of latent variations z(`) into h(y; `) at each layer ` via a linear transformation Λ(z; `) to render the finer image h(y, z; `− 1). 4) Add Gaussian pixel noise into h(y, z; 0) to render the final image x. In Eqn.2, if Λ(z; `) is a linear transformation, it will yield many parameters to learn. As a result, we would like to introduce new structures into NRM so that CNNs can be derived as NRM’s inference. This way, NRM will be informed by the
prior knowledge of natural images captured by CNNs. In our attempt to invert CNNs, we constrain the latent variables z(`) at layer ` in NRM to a set of template selecting latent variables s(`) and local translation latent variables t(`). As been shown later in Section 2.2, during inference of NRM, the ReLU non- linearity at layer ` “inverts” s(`) to find if particular features are in the image or not. Similarly, the MaxPool operator “inverts” t(`) to locate where particular features, if exist, are in the image. Both s(`) and t(`) are vectors indexed by p(`) ∈ P(`) ≡ {pixels in level `}. The rendering process from layer ` to layer `− 1 is given by:
h(`− 1) , Λ(`)h(`) = ∑
p∈P(`) s(`, p)T (t; `, p)B(`, p)Γ(`, p)h(`, p). (3)
The rendering process in Eqn.3 is illustrated in Figure 3. At each pixel p in the intermediate image h(`) at layer `, NRM decides to use that pixel to render or not according to the value of the template selecting latent variable s(p; `) at that pixel location. If s(p; `) = 1, then NRM renders; otherwise, it does not. If rendering, the pixel value h(`, p) is used to scale the local rendering template Γ(`, p), which has the same number of feature maps as the next rendered image h(`− 1), but is of smaller size, e.g., 3× 3 or 5× 5. Next, the padding matrix n B(`, p) pads the resultant patch to the size of the image h(`− 1) with zeros, and the translation matrix T (t; `, p) translates the result to a local location. NRM then keeps rendering at the next pixel location p+ 1 following the same process. All rendered images are added to form the final rendered image h(`− 1) at layer `− 1. Note that we can constrain NRM by enforcing all pixels in the same feature maps of h(`) share the same rendering template. This constraint helps to yield convolutions in CNNs during the inference of NRM, and the rendering template in NRM now corresponds to the convolution filters in CNNs. Reconstructed images at each layer of a 5-layered NRM trained on MNIST are visualized in Figure 2. The network is trained using the semi-supervised learning framework discussed in Section 2.3.
While s(`) and t(`) can be independent, we further constrain the model by enforcing the dependency among s(`) and t(`) at different layers in NRM. This constraint is motivated from realistic rendering of natural objects: different parts of a natural object are dependent on each other. NRM captures such dependency in natural objects by imposing more structures into the joint prior of latent variables at all layer in the model. The form of the join prior πz|y in Eqn. 1 might look mysterious at first, but NRM parametrizes πz|y in this particular way so that πz|y is the conjugate prior of the model likelihood as being proven in Appendix C.11. The conjugate form of πz|y allows efficient inference in the NRM. Parameters b(t; `) of the conjugate prior πz|y will become the bias terms after the CNN’s convolutions as will be shown in Theorem 2.2. When training in an unsupervised setup, the conjugate prior results in the RPN regularization as shown in Theorem 2.3(b). This RPN regularization helps enforce the dependencies among latent variables and increases the likelihood of latent configuration presents in the data during training.
NRM with skip connections: We derive ResNet and DenseNet by adding skip connections into the rendering matrices Λ(`) of NRM (see Appendix B).

heading: 2.2 INFERENCE
text: We show that calculations in the bottom-up inference in NRM corresponds to a CNN (see Figure 1b) and, therefore, is tractable and efficient. The impact of this correspondence goes beyond a reverseengineering effort. First, it provides probabilistic semantics for components in CNNs, justifying their usage, and providing an opportunity to employ probabilistic inference methods in the context of CNNs. Second, such a correspondence offers a flexible framework to design CNNs. Instead of directly engineering CNNs for new tasks and datasets, we can modify NRM to incorporate our knowledge of the tasks and datasets into the model and perform JMAP inference to achieve a new CNN architecture. The following theorem establishes the NRM-CNN correspondence: Theorem 2.2. The JMAP inference of latent variable z in NRM is the feedforward step in CNNs. Particularly, we have:
max z {p(z|x, y)} = max z
1 σ2 {〈h(y, z; 0), x〉+ η(y, z)}+ const ≥ 1 σ2 〈µ(y), ψ(L)〉+ const (4)
where ψ(L) is computed recursively. In particular, ψ(0) = x and
ψ(`) = MaxPool(ReLu(Conv(Γ>(`), ψ(`− 1))) + b(`)). (5) The equality holds in Eqn. 4 when the parameters θ in NRM satisfy the non-negativity assumption that the intermediate rendered image h(`) ≥ 0, ∀` = 1, 2, . . . , L. There are four key results in Theorem 2.2. First, ReLU non-linearities in CNNs find the optimal value for the template selecting latent variables s(`) at each layer ` in NRM, detecting if particular features exist in the image or not. Second, MaxPool operators in CNNs find the optimal value for the local translation latent variables t(`) at each layer ` in NRM, locating where features are rendered in the image. Third, bias terms after each convolution in CNNs are from the prior knowledge of latent variables in the model. Those bias terms update the posterior estimation of latent variables from data using the knowledge encoded in the prior distribution of those latent variables. Fourth, convolutions in CNNs result from reversing the local rendering operator using template Γ(`) in NRM. Instead of rendering as in NRM, convolutions in CNNs perform template matching. The convolution weights W (`) in CNNs are proportional to the transposes of the rendering templates Γ(`). The proofs for these correspondences and derivations for leaky ReLU and batch normalization are in Appendix C.

heading: 2.3 LEARNING
text: Learning in NRM can be posed as likelihood estimation problems in which we find the optimal values for parameters in NRM to optimize the appropriate likelihood functions. The optimization can be done by gradient-based methods (Robbins & Monro, 1985). The following theorem derives the learning objectives for NRM, and more details and proofs is provided in Appendix B and C. Theorem 2.3. For any n ≥ 1, let x1, . . . , xn be i.i.d. samples from the NRM. Assume that the final rendered template h(y, z; 0) is normalized such that its norm is constant. The following holds: (a) Cross-entropy loss for training CNNs with labeled data
max (zi)ni=1,θ
1
n n∑ i=1 log p(yi|xi, zi; θ) ≥ max θ 1 n n∑ i=1 log q(yi|xi) = − min θ∈Aγ Hp,q(y|x) (6)
where q(y|x) is the posterior estimated by CNN, and Hp,q(y|x) is the cross-entropy between q(y|x) and the true posterior p(y|x) given by the ground truth. (b) Reconstruction loss with RPN for unsupervised training of CNNs with labeled and unlabeled data
min θ
1
n n∑ i=1 E [log p(xi, zi|yi)] asymp≈ min θ 1 n n∑ i=1 ‖xi − h(yi, z∗i ; 0)‖2 2 + RPN, when σ → 0 (7)
where the latent variable z∗i is estimated by the CNN as described in Theorem 2.2, h(yi, z ∗ i ; 0) is the reconstructed image, and the RPN regularization is the negative log prior defined as follows:
RPN = − 1 n n∑ i=1 log p(z∗i |yi) = − 1 n n∑ i=1 Softmax (η(yi, z∗i )) . (8)
Cross-Entropy Loss for Training CNNs with Labeled Data: Theorem 2.3(a) establishes the cross-entropy loss in the context of CNNs as an upper bound of the NRM’s negative conditional log-likelihood Lsup := − 1n ∑n i=1 log p(yi|xi, zi; θ). In contrast to other derivations of cross-entropy loss via logistic regression, we derives the cross-entropy loss in conjunction with the architecture of CNNs since the estimation of the optimal latent variables z∗ is part of the optimization in Eqn. 6. In other word, Theorem 2.3(a) ties feature extraction and learning for classification in CNNs into an end-to-end conditional likelihood estimation problem in NRM. This new interpretation of the cross-entropy loss suggests an interesting direction in which better losses for training CNNs with labeled data for supervised classification tasks can be derived from other upper bounds of Lsup. The Max-Min cross-entropy in Section 3 is an example.
Reconstruction Loss with RPN Regularization: Theorem 2.3(b) suggests that NRM learns without labels by maximizing its expected complete-data log-likelihood. One term in this objective function is the reconstruction loss between the input image xi and the reconstructed template h(yi, z∗i ; 0). Another term is the RPN from Eqn. 8. RPN encourages the (yi, z∗i ) inferred in the bottom-up E-step to have higher prior among all possible values of (yi, zi) and, thanks to its structure, enforces the dependencies among latent variables (s(`), t(`)) at different layers in NRM.
Semi-Supervised Learning in NRM: Let x1, . . . , xn be i.i.d. samples from NRM and assume that the labels y1, . . . , yn1 are unknown for some 0 ≤ n1 ≤ n, NRM determines optimal parameters employed for the semi-supervised classification task via the following model:
min θ { αRC n n∑ i=1 (‖xi − h(yi, z∗i ; 0)‖2 2 + RPN ) − αCE n− n1 n∑ i=n1+1 log qθ(yi|xi) } , (9)
where αRC and αCE are non-negative weights associated with the reconstruction loss with RPN regularization and the cross-entropy loss, respectively. Again, the optimal latent variables z∗i is estimated by CNN as in Theorem 2.2. For unlabeled data, yi is set to the label estimated by CNN.

heading: 2.4 GENERALIZATION BOUND FOR CLASSIFICATION
text: Our generalization bound for classification with NRM is proportional to the ratio of the number of active rendering paths and the total number of rendering paths in the trained NRM. A rendering path is a configuration of all latent variables in NRM, and active optimal rendering paths are those among optimal rendering paths (ŷ, ẑ) whose corresponding rendered image is sufficiently close to one of the data point from the input data distribution. Let LA and LD denote the population and empirical losses on the data population A and the training set D of NRM, respectively. Our key result is summarized below. More details and proofs are deferred to Appendix B and C.
Theorem 2.4. Under the margin-based loss, with high probability, the following result on the generalization bound of the classification framework with optimal solutions from Eqn. 9 holds:
LA ≤ LD + τn|L|/ √ n.
Here, τn ∈ (0, 1) denotes the ratio of active optimal rendering paths among all the optimal rendering paths, |L| is the total number of rendering paths, and n is the number of training data samples. The dependence of generalization bound on the number of active rendering paths τn|L| helps to justify our modeling assumptions. In particular, NRM helps to reduce the number of active rendering paths thanks to the dependencies among its latent variables, thereby tightening the generalization gap. Nevertheless, there is a limitation regarding the current generalization bound. In particular, the bound involves the number of rendering paths |L|, which is usually large. This is mainly because our bound has not fully taken into account the structure of CNNs, which is the limitation shared among other latest generalization bounds for CNN. It is interesting to explore if techniques in works by Bartlett et al. (2017) and Golowich et al. (2018) can be employed to improve the term |L| in our bound.

heading: 3 NEW MAX-MIN CROSS ENTROPY FROM THE NEURAL RENDERING MODEL
text: In this section, we explore a particular way to derive an alternative to cross-entropy inspired by the results in Theorem 2.3(a). In particular, denoting zmax , arg maxz {〈h(y, z; 0), x〉+ η(y, z)} and zmin , arg minz {〈h(y, z; 0), x〉+ η(y, z)}, the new cross-entropy HM&M , which is called the Max-Min cross-entropy, is the weighted average of the cross-entropy losses from zmax and zmin:
HM&M , αmaxHp,q(y|x, zmax) + αminHp,q(y|x, zmin) = αmaxHmaxp,q (y|x) + αminHminp,q (y|x).
Here the Max cross-entropyHmaxp,q and Min cross entropyHminp,q maximizes the correct target posterior and and minimizes the incorrect target posterior, respectively. Similar to the cross-entropy loss, the Max-Min cross-entropy can also be shown to be an upper bound for the negative conditional log-likelihood Lsup of the NRM and has the same generalization bound derived in Section 2.4. The Max-Min networks in Figure 4
realize this new loss. These networks have two CNN-like branches that share weights. The max branch estimates zmax using ReLU and Max-Pooling, and the min branch estimates zmin using the Negative ReLU, i.e., min(·, 0), and Min-Pooling. The Max-Min networks can be interpreted as a form of knowledge distillation like the Born Again networks (Furlanello et al., 2018) and the Mean Teacher networks. However, instead of a student network learning from a teacher network, in Max-Min networks, two students networks, the Max and the Min networks, cooperate and learn from each other during the training.

heading: 4 EXPERIMENTS

heading: 4.1 SEMI-SUPERVISED LEARNING
text: We show NRM armed with Max-Min cross-entropy and Mean Teacher regularizer achieves SOTA on benchmark datasets. We discuss the experimental results for CIFAR10 and CIFAR100 here. The results for SVHN, training losses, and training details, can be found in the Appendix A & D.
CIFAR-10: Table 2 shows comparable results of NRM to SOTA methods. NRM is also better than the best methods that do not use consistency regularization like GAN, Ladder network, and ALI when using only Nl=2K and 4K labeled images. NRM outperforms DRM in all settings. Also, among methods in our comparison, NRM achieves the best test accuracy when using all available labeled data (Nl=50K). NRM has the advantage over consistency regularization methods like Temporal Ensembling and Mean Teacher when there are enough labeled data because the consistency regularization in those methods tries to match the activations in the network, but does not take into account the available class labels. On the contrary, NRM employs the class labels, if they are available, in its reconstruction loss and RPN regularization as in Eqns. 7 and 8. In all settings, RPN regularizer improves NRM performance. Even though the improvement from RPN is small, it is consistent across the experiments. Furthermore, using Max-Min cross-entropy significantly reduces the test errors. When combining with Mean-Teacher, our Max-Min NRM improves upon Mean-Teacher and consistently achieves either SOTA results or second best results in all settings.
Table 2: Error rate percentage on CIFAR-10 over 3 runs. 1K labels
50K images
2K labels
50K images
4K labels
50K images
50K labels 50K images
Adversarial Learned Inference (Dumoulin et al., 2017) 19.98± 0.89 19.09± 0.44 17.99± 1.62 Improved GAN (Salimans et al., 2016) 21.83± 2.01 19.61± 2.09 18.63± 2.32 Ladder Network (Rasmus et al., 2015) 20.40± 0.47 Π model (Laine & Aila, 2017) 27.36± 1.20 18.02± 0.60 13.20± 0.27 6.06± 0.11 Temporal Ensembling (Laine & Aila, 2017) 12.16± 0.31 5.60± 0.10 Mean Teacher (Tarvainen & Valpola, 2017) 21.55± 1.48 15.73± 0.31 12.31± 0.28 5.94± 0.15 VAT+EntMin (Miyato et al., 2018) 10.55 DRM (Patel et al., 2016; 2015) 27.67± 1.86 20.71± 0.30 15.36± 0.34 5.75± 0.24 Supervised-only 46.43± 1.21 33.94± 0.73 20.66± 0.57 5.82± 0.15 NRM without RPN 24.88± 0.76 18.97± 0.80 14.41± 0.19 5.57± 0.07 NRM+RPN 24.48± 0.43 18.62± 0.70 14.18± 0.46 5.35± 0.08 NRM+RPN+Max-Min 21.55± 0.46 16.24± 0.17 12.50± 0.35 4.85± 0.10 NRM+RPN+Max-Min+Mean Teacher 19.79± 0.74 15.11± 0.51 11.81± 0.13 4.88± 0.09
This consistency in performance is only observed in our method and Mean-Teacher. Also, like with Mean-Teacher, NRM can potentially be combined with other consistency regularization methods, e.g., the Virtual Adversarial Training (VAT) (Miyato et al., 2018), to obtain better results.
CIFAR-100: Table 3 shows NRM’s comparable results to Π model and Temporal Ensembling, as well as better results than DRM. Same as with CIFAR10, using the RPN regularizer results in a slightly better test accuracy, and NRM achieves better results than Π model and Temporal Ensembling method when using all available labeled data.
50K labels
Notice that combining with Mean-Teacher just slightly improves NRM’s performance when training with 10K labeled data. This is again because consistency regularization methods like Mean-Teacher do not add much advantage when there are enough labeled data. However, NRM+Max-Min still yields
better test errors and achieves SOTA result in all settings.

heading: 4.2 SUPERVISED LEARNING WITH MAX-MIN CROSS-ENTROPY
text: The Max-Min cross-entropy can be applied not only to improve semi-supervised learning on deep models including CNNs but also to enhance their supervised learning performance. In our experiments, we indeed observe Max-Min cross-entropy reduces the test error for supervised object classification on CIFAR10. In particular, using the Max-Min cross-entropy loss on a 29-layer ResNet (Xie et al., 2017) trained with the Shake-Shake regularization (Gastaldi, 2017) and Cutout data augmentation (DeVries & Taylor, 2017), we are able to achieve SOTA test error of 2.30% on CIFAR10, an improvement of 0.26% over the test error of the baseline architecture trained with the traditional cross-entropy loss. While 0.26% improvement seems small, it is a meaningful enhancement given that our baseline architecture (ResNeXt + Shake-Shake + Cutout) is the second best model for supervised learning on CIFAR10. Such small improvement over an already very accurate model is significant in applications in which high accuracy is demanded such as self-driving cars or medical diagnostics. Similarly, we observe Max-Min improves the top-5 test error of the Squeeze-and-Excitation ResNeXt50 network (Hu et al., 2018) on ImageNet by 0.17% compared to the baseline (7.04% vs. 7.21%). For a fair comparison, we re-train the baseline models and report the scores in the re-implementation.

heading: 5 CONCLUSIONS
text: We present the NRM, a general and an effective framework for semi-supervised learning that combines generation and prediction in an end-to-end optimization. Using NRM, we can explain operations used in CNNs and develop new features that help learning in CNNs. For example, we derive the new Max-Min cross-entropy loss for training CNNs, which outperforms the traditional cross-entropy. Despite promising results in this paper, NRM can still be improved. For instance, an adversarial loss like in GANs can be incorporated into NRM so that the model can generate realistic images. Furthermore, more knowledge of image generation from graphics and physics can be integrated in NRM so that the model can employ more structures to help learning and generation.

heading: Appendix A
text: This appendix contains several key tables with simulation studies and notations as well as figures for the connection between NRM and CNN in the main text.
Semi-Supervised Learning Results on SVHN
Rendering Process in NRM
Algorithm 1 Rendering Process in NRM Input: Object category y. Output: Rendered image x given the object category y.
Parameters: θ = ( {µ(y)}Ky=1 , {Γ(`)} L `=1 , {b(`)} L `=1 ) where µ(y) is the class template, Γ(`) is
the rendering template at layer `, and b(`) is the parameters of the conjugate prior p(z|y, x) at layer `, which turn out to be the bias terms in the ReLU after convolutions at each layer in CNNs.
1. Use Markov chain Monte Carlo method to sample the latent variables z(`) in NRM, ` = 1, 2, . . . , L from πz|y = Softmax ( 1 σ2 η(y, z) ) , where η(y, z) , ∑L `=1 〈b(t; `), s(`) h(`)〉. 2. Render h(`), ` = 0, 1, . . . , L − 1 using the recursion h(` − 1) =∑ p∈P(`) s(`, p)T (t; `, p)B(`, p)Γ(`, p)h(`, p) in Eqn. 3, in which h(L) = µ(y) and T (t; `, p) and B(`, p) are the local translation matrix and the zero-padding matrix at pixel location p in layer ` as described above.
3. Add Gaussian pixel noiseN (0, σ21D(0)) into h(0) to achieve the final rendered image x, where D(0) is the dimension, i.e. the number of pixels, of h(0), as well as of x.

heading: Appendix B
text: In this appendix, we give proofs for Theorem 2.3. We also give further connection of NRM to cross entropy as well as additional derivation of NRM to various models under both unsupervised and (semi)supervised setting of data being mentioned in the main text. We also formally present our results on consistency and generalization bounds for NRM in supervised and semi-supervised learning settings. In addition, we explain how to extend NRM to derive ResNet and DenseNet. For the simplicity of
the presentation, we denote θ = ( {µ(y)}Ky=1 , {Γ(`)} L `=1 , {πy} K y=1 , {b(`)} L `=1 ) to represent all the
parameters that we would like to estimate from NRM where L is the set of all possible values of latent (nuisance) variables z = (t(`), s(`))L`=1. Additionally, for each (y, z) ∈ J := {1, . . . ,K} × L, we denote θy,z = ( µ(y), {Γ(`)}L`=1 , {b(`)} ) , i.e., the subset of parameters corresponding to specific
label y and latent variable z. Furthermore, to stress the dependence of η(y, z) on θ, we define the following function
τ(θy,z) := η(y, z) = L∑ `=1 b>(`) (s(`) z(`)) = L∑ `=1 b>(`)M(s; `)z(`)
for each (y, z) ∈ J where M(s; `) = diag(s(`)) is a masking matrix associated with s(`). Throughout this supplement, we will use τ(θy,z) and η(y, z) interchangeably as long as the context is clear. Furthermore, we assume that Γ(`) ∈ θ`, which is a subset of RF (`)×D(`) for any 1 ≤ ` ≤ L, µ(y) ∈ Ω, which is a subset of RD(`), for 1 ≤ y ≤ K, and b(t; `) ∈ Ξ(`), which is a subset of RD(`) for all choices of t(`) and 1 ≤ ` ≤ L. Last but not least, we say that θ satisfies the non-negativity assumption if the intermediate rendered images z(`) satisfy z(`) ≥ 0 for all 1 ≤ ` ≤ L. Notation: In this appendix, we use A> to denote transpose of the matrix A.
.1 CONNECTION BETWEEN NRM AND CROSS ENTROPY
As being established in part (a) of Theorem 2.3, the cross entropy is the upper bound of the NRM’s negative conditional log likelihood. In the following full theorem, we will show both the upper bound and the lower bound of maximizing the conditional log likelihood in terms of the cross entropy. Theorem .1. Given any γ > 0, we denote Aγ = {θ : ‖h(y, z; 0)‖ = γ}. For any n ≥ 1 and σ > 0, let x1, . . . , xn be i.i.d. samples from the NRM. Then, the following holds (a) (Lower bound)
max (zi)ni=1,θ∈Aγ
1
n n∑ i=1 log p(yi|xi, zi; θ)
≥ max θ∈Aγ
1
n n∑ i=1 log ( Softmax ( max zi ( h>(yi, zi; 0)xi + η(yi, zi) σ2 ) + byi ))
= max θ∈Aγ
1
n n∑ i=1 log q(yi|xi) = − min θ∈AΓ Hp,q(y|x)
where by = log πy for all 1 ≤ y ≤ K, q(y|x) = Softmax (
max z
( h>(y, z; 0)x+η(y, z) ) /σ2 +by ) for all (x, y), and Hp,q(y|x) is the cross-entropy between the estimated posterior q(y|x) and the true posterior given by the ground-true labels p(y|x). (b) (Upper bound)
max (zi)ni=1,θ∈Aγ
1
n n∑ i=1 log p(yi|xi, zi; θ)
≤ max θ∈Aγ
1
n n∑ i=1 { log q(yi|xi) + max y ( max zi ( h>(y, zi; 0)xi + η(y, zi) σ2 ) −h >(y, zi; 0)xi + η(y, zi)
σ2
)} + logK
where zi = arg max zi p(yi|xi, zi; ~θ) for 1 ≤ i ≤ n.
Remark .2. As being demonstrated in Theorem 2.2, max z
( h>(y, z; 0)x+ η(y, z) ) /σ2, approxi-
mately, has the form of a CNN. If we further have the non-negativity assumption with θ, then this is exact. Therefore, the cross entropy Hp,q obtained in Theorem .1 has a strong connection with CNN. Remark .3. The gap between the upper bound and the lower bound of maximizing the conditional log likelihood in terms of cross entropy function suggests how good the estimation in Theorem .1 is. In particular, this gap is given by:
1
n n∑ i=1 { max y ( max zi ( h>(y, zi; 0)xi + η(y, zi) σ2 ) − h >(y, zi; 0)xi + η(y, zi) σ2 ) ︸ ︷︷ ︸
trade-off loss
} + logK,
where zi = arg max zi p(yi|xi, zi; θ) for 1 ≤ i ≤ n. As long as the number of labels is not too large and the trade-off loss is sufficiently small, the gap between the upper bound and the lower bound in Theorem .1 is small.
.2 LEARNING IN THE NRM WITHOUT NOISE FOR UNSUPERVISED SETTING UNDER NON-NEGATIVITY ASSUMPTION
To ease the presentation of the inference with NRM without noise, we first assume that rendered images h(y, z; 0) satisfy the non-negativity assumption (Later in Section .3, we will discuss the relaxation of this assumption for the inference of the NRM). With this assumption, as being demonstrated in Theorem 2.2, we have:
max y,z
{ h>(y, z; 0)x+ η(y, z) } = max
y h>(y)ψ(L) (10)
where we define
ψ(L) = max z(L)
Λ>(z(L)) ( max z(L−1) ( Λ>(z;L− 1) · · · ( max z(1) Λ>(z; 1)x+ b(1) ) . . . ) + b(L− 1) ) +b(`).
Now, we will provide careful derivation of part (b) of Theorem 2.3 in the main text. Remind that, for the unsupervised setting, we have data x1, . . . , xn are i.i.d. samples from NRM. The complete-data log-likelihood of the NRM is given as follows:
Ey,z [log p(x, (y, z))] = n∑ i=1 ∑ (y,z)∈J P (y, z|xi){log πy,z + logN (xi|h(y, z; 0))}
where we have
P (y, z|xi) = πy,zN (xi|h(y, z; 0))∑
(y′,z′)∈J πy′,z′N (xi|h(y′, z′; 0))
=
πy exp
( − ‖xi − h(y, z; 0)‖
2 − 2η(y, z) 2σ2 ) ∑
(y′,z′)∈J πy′ exp
( − ‖xi − h(y
′, z′; 0)‖2 − 2η(y′, z′) 2σ2 ) . At the zero-noise limit, i.e., σ → 0, it is clear that P (y, z|xi) = 1 as (y, z) = arg min
(y′,z′)∈J
{ ‖xi −
h(y′, z′; 0)‖2 − 2η(y′, z′) }
and P (y, z|xi) = 0 otherwise. Therefore, we can asymptotically view the complete log-likelihood of the NRM under the zero-noise limit as
n∑ i=1 ∑ (y,z)∈J ry,z ( log πy,z − 1 2 ‖xi − h(y, z; 0)‖2 )
= n∑ i=1 ∑ (y,z)∈J
−1 2 ry,z‖xn − h(y, z; 0)‖2︸ ︷︷ ︸
Reconstruction Loss
+ n∑ i=1 ∑ (y,z)∈J
ry,z log πy,z︸ ︷︷ ︸ Path Normalization Regularizer
where
ry,z ≡  1, if (y, z) = arg min(y′,z′)∈J { ‖xi − h(y′, z′; 0)‖2 − 2η(y′, z′) } 0, otherwise
With the above formulation, we have the following objective function
Un = min θ
1
n n∑ i=1 ∑ (y,z)∈J ry,z ( 1 2 ‖xi − h(y, z; 0)‖2 − log πy,z ) (11)
where θ = ( {µ(y)}Ky=1 , {Γ(`)} L `=1 , {πy} K y=1 , {b(`)} L `=1 ) . We call the above objective function to
be unsupervised NRM without noise.
Relaxation of unsupervised NRM without noise Unfortunately, the inference with unsupervised NRM without noise is intractable in practice due to two elements: the involvement of ‖h(y′, z′; 0)‖2 to determine the value of ry,z and the summation ∑ (y′,z′)∈J exp(η(y′, z′)+log πy′) in the denominator of πy,z for all (y, z) ∈ J . Therefore, we need to develop a tractable version of this objective function. Theorem .4. (Relaxation of unsupervised NRM without noise) Assume that πy ≥ γ for all 1 ≤ y ≤ K for some given γ ∈ (0, 1/2). Denote
Vn := min θ
1
n n∑ i=1 ∑ (y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( h>(y,z;0)xi+η(y,z) ) (‖xi − h(y′, z′; 0)‖2 2 − log(πy′,z′) )
where py′,z′ = exp ( η(y′, z′) + log πy′ ) / ( K∑ y=1 exp ( max z∈L η(y, z) + log πy )) for all (y′, z′) ∈ J . For any θ, we define
(yi, zi) = arg min (y,z)∈J
{ ‖xi − h(y, z; 0)‖2 − 2η(y, z) } and
(ỹi, z̃i) = arg max (y,z)∈J
( h>(y, z; 0)xi + η(y, z) ) as 1 ≤ i ≤ n. Then, the following holds (a) Upper bound:
Un ≤ min θ
1
n n∑ i=1 {(‖xi − h(ỹi, z̃i; 0)‖2 2 − log(πỹi,z̃i) ) + ( log πỹi − log πyi ) ︸ ︷︷ ︸
prior loss
} + log |L|
≤ Vn + log ( 1 γ − 1 ) + log |L|
(b) Lower bound:
Un ≥ min θ
1
n n∑ i=1 {(‖xi − h(ỹi, z̃i; 0)‖2 2 − log(πỹi,z̃i) ) + ( log πyi − log πỹi ) ︸ ︷︷ ︸
prior loss
+ 1
2
( ‖h(yi, zi; 0)‖2 − ‖h(ỹi, z̃i; 0)‖2 ) ︸ ︷︷ ︸
norm loss
}
≥ Vn + log ( γ
1− γ
) + min
θ
1
n n∑ i=1 1 2 ( ‖h(yi, zi; 0)‖2 − ‖h(ỹi, z̃i; 0)‖2 )
Unlike Un, the inference with objective function of Vn is tractable. According to the upper bound and lower bound of Un in terms of Vn, we can use Vn as a tractable approximation of Un for the inference purpose with unsupervised setting of data when the noise is treated to be 0. Therefore, we achieve the conclusion of part (b) of Theorem 2.3 in the main text. The algorithm for determined (local) minima of Vn is summarized in Algorithm 2.
.3 RELAXATION OF NON-NEGATIVITY ASSUMPTION WITH RENDERED IMAGES
It is clear that the inference with Vn relies on the non-negativity assumption such that equation equation 10 holds. Now, we will argue that when the non-negativity assumption with rendered images h(y, z; 0) does not hold, we can relax Vn to a more tractable version under that setting. Theorem .5. (Relaxation of objective function Vn when non-negativity assumption does not hold) Assume that πy ≥ γ for all 1 ≤ y ≤ K for some given γ ∈ (0, 1/2). Denote
Wn := min θ
1
n n∑ i=1 K∑ y′=1 1{ y′=arg max
y∈J g(y,xi)
}(‖xi − g(y′, zi)‖2 2 − log(πy′,zi) )
where
g(y, x) = h>(y) MaxPool ( ReLu ( Conv ( Γ(`), · · ·MaxPool ( ReLu ( Conv ( Γ(1), I ) + b(1) ))) · · ·+ b(`)
)) for all (x, y). Additionally, zi is the maximal value of z in the CNN structure of g(y, xi) for 1 ≤ i ≤ n. For any θ, we define
(ỹi, z̃i) = arg max (y,z)∈J
( h>(y, z; 0)xi + η(y, z) ) and yi = arg max
y g(y, xi) as 1 ≤ i ≤ n. Then, the following holds
(a) Upper bound:
Vn ≤ min θ
1
n n∑ i=1 {(‖xi − h(yi, zi; 0)‖2 2 − log(πyi,zi) ) + ( log πyi − log πỹi ) ︸ ︷︷ ︸
prior loss
+ 1
2
( ‖h(ỹi, z̃i; 0)‖2 − ‖h(yi, zi; 0)‖2 ) ︸ ︷︷ ︸
norm loss
}
≤ Wn + log ( 1 γ − 1 ) + min θ 1 n n∑ i=1 1 2 ( ‖h(ỹi, z̃i; 0)‖2 − ‖h(yi, zi; 0)‖2 ) (b) Lower bound:
Vn ≥ min θ
1
n n∑ i=1 {(‖xi − h(yi, zi; 0)‖2 2 − log(πyi,zi) ) + ( log πyi − log πỹi ) ︸ ︷︷ ︸
prior loss
+ 1
2
( ‖h(ỹi, z̃i; 0)‖2 − ‖h(yi, zi; 0)‖2 ) ︸ ︷︷ ︸
norm loss
+ ( g(yi, xi)− { h(ỹi, z̃i; 0) >xi + η(ỹi, z̃i) })
︸ ︷︷ ︸ CNN loss
}
≥Wn + log ( γ
1− γ
) + min
θ
1
n n∑ i=1 1 2 ( ‖h(ỹi, z̃i; 0)‖2 − ‖h(yi, zi; 0)‖2 )
+ min θ
1
n n∑ i=1 ( g(yi, xi)− { h(ỹi, z̃i; 0) >xi + η(ỹi, z̃i) })
Algorithm 2 Relaxation of unsupervised NRM without noise Input: Data xi, translation matrices T (t; `), zero padding matrices B(`), number of labels K, number of layers L. Output: Parameters θ.
Initialize θ = ( {µ(y)}Ky=1 , {Γ(`)} L `=1 , {πy} K y=1 , {b(`)} L `=1 ) .
while θ has not converged do 1. E-Step: Update labels (y, z) of each data for i = 1 to n do
(ŷi, ẑi) = arg max y,z
( h>(y, z)xi + log(πy,z) ) .
end for 2. M-Step: By using Stochastic Gradient Descent (SGD), update θ that minimizes n∑ i=1 (‖xi − h(ŷi, ẑi)‖2 2 − log(πŷi,ẑi) ) .
end while
The proof argument of the above theorem is similar to that of Theorem .4; therefore, it is omitted. The upper bound and lower bound of Vn in terms of Wn in Theorem .5 implies that we can use Wn as a relaxation of Vn when the non-negativity assumption with rendered images h(y, z; 0) does not hold. The algorithm for achieving the (local) minima of Wn is similar to Algorithm 2.
.4 NRM WITH (SEMI)-SUPERVISED SETTING
In this section, we consider the application of NRM to the (semi)-supervised setting of the data. Under that setting, only a (full) portion of labels of data x1, . . . , xn is available. Without loss of generality, we also assume that the rendering path h(y, z; 0) satisfies the non-negativity assumption. For the case that h(y, z; 0) does not satisfy this assumption, we can argue in the same fashion as that of Theorem .5. Now, we assume that only the labels (yn1+1, . . . , yn) are unknown for some n1 ≥ 0. When n1 = 0, we have the supervised setting of data while we have the semi-supervised setting of data when n− n1 is small. Our goal is to build a semi-supervised model based on NRM such that the clustering information from data x1, . . . , xn1 can be used efficiently to increase the accuracy of classifying the labels of data xn1+1, . . . , xn. For the sake of simple inference with that purpose, we only consider the setting of NRM when the noise goes to 0. Our idea of constructing the semi-supervised model based on NRM is insprired by an approximation of the upper bound of maximizing the conditional log likelihood of NRM in terms of the cross entropy and reconstruction loss in part (b) of Theorem 2.3. In particular, we combine the tractable version of reconstruction loss from the unsupervised setting in Theorem 2.3b and the cross entropy of approximate posterior in Theorem 2.3a, which can be formulated as follows
min θ αRC n { n1∑ i=1 ∑ (y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( h>(y,z;0)xi+τ(θy,z) ) (‖xi − h(y′, z′; 0)‖2 2 − log(πy′,z′) )
+ ( n∑ i=n1+1 ∑ z′∈L 1z′=arg maxz∈L ( h>(yi,z;0)xi+τ(θyi,z) ) (‖xi − h(yi, z′; 0)‖2 2 − log(πyi,z′) )}
− αCE n− n1 n∑ i=n1+1 log qθ(yi|xi)
where αRC and αCE are non-negative weights associated with reconstruction loss and cross entropy respectively. Additionally, the approximate posterior qθ(y|xi) is chosen as
qθ(y|xi) = exp
( max z∈L { h>(y, z; 0)xi + τ(θy,z) } + log πy ) K∑ y′=1 exp ( max z∈L { h>(y′, z; 0)xi + τ(θy′,z) } + log πy′ ) .
Note that, since the labels (yn1+1, . . . , yn) are known, the reconstruction loss for clustering data xn1+1, . . . , xn in the above objective function indeeds incorporate these information to improve the accuracy of estimating the parameters. We call the above objective function to be (semi)-supervised NRM without noise.
Boosting the accuracy of (semi)-supervised NRM without noise In practice, it may happen that the accuracy of classifying data by using the parameters from (semi)-supervised NRM without noise is not very high. To account for that problem, we consider the following general version of (semi)-supervised NRM without noise that includes the variational inference term and the moment matching term
min θ αRC n { n1∑ i=1 ∑ (y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( h>(y,z;0)xi+τ(θy,z) ) (‖xi − h(y′, z′; 0)‖2 2 − log(πy′,z′) )
+ ( n∑ i=n1+1 ∑ z′∈L 1z′=arg maxz∈L ( h>(yi,z;0)xi+τ(θyi,z) ) (‖xi − h(yi, z′; 0)‖2 2 − log(πyi,z′) )}
− αCE n− n1 n∑ i=n1+1 log qθ(yi|xi) + αKL n n∑ i=1 K∑ y=1 qθ(y|xi) log ( qθ(y|xi) πy )
+αMM L∑ `=1 DKL ( N (µh(`), σ2h(`))||N (µψ(`), σ2ψ(`)) ) . (12)
Here, αKL and αMM are non-negative weights associated with the variational inference loss and moment matching loss respectively. Additionally, µh(`), σ2h(`), µψ(`), σ 2 ψ(`) in the moment matching loss are defined as follows:
µh(`) = 1
n n∑ i=1 ĥ(`)i, σ 2 h(`) = 1 n n∑ i=1 (ĥ(`)i − µh(`))2
µψ(`) = 1
n n∑ i=1 ψ(`)i, σ 2 ψ(`) = 1 n n∑ i=1 (ψ(`)i − µψ(`))2 (13)
where ĥ(`)i is the estimated value of h(`) given the optimal latent variables ŝ(`) and t̂(`) inferred from the image xi for 1 ≤ i ≤ n. It is clear that when αKL = αMM = 0, we return to (semi)-supervised NRM without noise. In Appendix C, we provide careful theoretical analyses regarding statistical guarantees of model equation 12.
Now, we will provide heuristic explanations about the improvement in terms of performance of model equation 12 based on the variational inference term and the moment matching term.
Regarding the variational term: The DRMM inference algorithm developed thus far ignores uncertainty in the latent nuisance posterior p(y, z|x) due to the max-marginalization over (y, z) in the E-step bottom-up inference. We would like to properly account for this uncertainty for two main reasons: (i) our fundamental hypothesis is that the brain performs probabilistic inference and (ii) uncertainty accounting is very important for good generalization in the semi-supervised setting since we have very little labeled data.
One approach attempts to approximate the true class posterior p(y|x) for the DRMM. We employ variational inference, a technique that enables the approximate inference of the latent posterior. Mathematically, for the DRMM this means we would like to approximate the true class posterior p(y|x) ≈ q(y|x), where the approximate posterior q is restricted to some tractable family of distributions (e.g. Gaussian or Categorical). We strategically choose the tractable family to be q(y|x) ≡ p(y|ẑ, x), where ẑ ≡ argmax
z p(y, z|x). In other words, we choose q to be restricted
to the DRMM family of nuisance max-marginalized class posteriors. Note that this is indeed an approximation, since the true DRMM class posterior has nuisances that are sum-marginalized out p(y|x) = ∑
z p(y, z|x), whereas the approximating variational family has nuisances that are max-
marginalized out.
Given our choice of variational family q, we derive the variational term for the loss function, starting from the principled goal of minimizing the KL-distance DKL[q(y|x)||p(y|x)] between the true and approximate posteriors with respect to the parameters of q. As a result, such an optimized q will tilt towards better approximating p(y|x), which in turn means that it will account for some of the uncertainty in p(z|x). The variational terms in the loss are defined as Blei et al. (2017):
LV I ≡ LRC + βKLLKL ≡ −Eq [ln p(x|y)] + βKLDKL[q(y|x)||p(y)]. (14)
This term is quite similar to that used in variational autoencoders (VAE) Kingma & Welling (2013), except for two key differences: (i) here the latent variable y is discrete categorical rather than continuous Gaussian and (ii) we have employed a slight relaxation of the VAE by allowing for a penalty parameter βKL 6= 1. The latter is motivated by recent experimental results showing that such freedom enables optimal disentangling of the true intrinsic latent variables from the data.
Regarding the moment matching term: Batch Normalization can potentially be derived by normalizing the intermediate rendered images h(`), ` = 1, 2, . . . , L in the NRM by subtracting their means and dividing by their standard deviations under the assumption that the means and standard derivations of h(`) are close to those of the activation ψ(`) in the CNNs. From this intuition, in Section .4 of Appendix A, we introduce the moment-matching loss to improve the performance of the NRM/CNNs trained for semi-supervised learning tasks.
.5 STATISTICAL GUARANTEES FOR (SEMI)-SUPERVISED SETTING
For the sake of simplicity with proof argument, we only provide detail theoretical analysis for statistical guarantee with the setting of equation 12 when the moment matching term is skipped. In particular, we are interested in the following (semi)-supervised model
Yn := min θ αRC n { n1∑ i=1 ∑ (y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( h>(y,z;0)xi+τ(θy,z)
) × (‖xi − h(y′, z′; 0)‖2
2 − log(πy′,z′) ) +
( n∑ i=n1+1 ∑ z′∈L 1z′=arg maxz∈L ( h>(yi,z;0)xi+τ(θyi,z) ) (‖xi − h(yi, z′; 0)‖2 2 − log(πyi,z′) )}
− αCE n− n1 n∑ i=n1+1 log qθ(yi|xi) + αKL n n∑ i=1 K∑ y=1 qθ(y|xi) log ( qθ(y|xi) πy ) (15)
where the approximate posterior qθ(y|xi) is chosen as
qθ(y|xi) := exp
( max z∈L { h>(y, z; 0)xi + τ(θy,z) } + log πy ) K∑ y′=1 exp ( max z∈L { h>(y′, z; 0)xi + τ(θy′,z) } + log πy′
) . Here, αRC, αCE, and αKL are non-negative weights associated with reconstruction loss, cross entropy, and variational inference respectively. As being indicated in the formulation of objection function Yn, the only difference between Yn and equation 12 is the weight αMM regarding moment matching loss in equation 12 is set to be 0. To ease the presentation with theoretical analyses later, we call the objective function with Yn to be partially labeled latent dependence regularized cross entropy (partially labeled LDCE).
Consistency of partially labeled LDCE Firstly, we demonstrate that the objective function of partially labeled LDCE enjoys the consistency guarantee.
Theorem .6. (Consistency of objective function of partially labeled LDCE) Assume that n1 is a function of n such that n1/n→ λ as n→∞. Furthermore, P(‖x‖ ≤ R) = 1 as x ∼ P for some
given R > 0. We denote the population version of partially labeled LDCE as follows
Y := min θ αRC
{ λ (∫ ∑ (y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( h>(y,z;0)x+τ(θy,z) ) (‖x− h(y′, z′; 0)‖2 2
− log(πy′,z′) ) dP (x) ) + (1− λ) (∫ ∑ z′∈L 1z′=arg maxz∈L ( h>(y,z;0)x+τ(θy,z) ) (‖x− h(y, z′; 0)‖2 2
− log(πy,z′) ) dQ(x, c) )} − αCE ∫ log qθ(y|x)dQ(x, c) + αKL ∫ K∑ y=1 qθ(y|x) log ( qθ(y|x) πy ) dP (x).
Then, we obtain that Yn → Y almost surely as n→∞.
The detail proof of Theorem .6 is deferred to Appendix C. Now, we denote θ̃ :=( {µ̃(y)}Ky=1 , { Γ̃(`) }L `=1 , {π̃y}Ky=1 , { b̃(`) }L `=1 ) the optimal solutions of objective function equation 15. Note that, the existence of these optimal solutions is guaranteed due to the compactness assumption of the parameter spaces Θ`, Ω, and Ξl for 1 ≤ ` ≤ L. The optimal solutions {µ̃(y)}Ky=1 and { Γ̃(`) }L `=1 lead to corresponding set of optimal rendered images S̃n. Similar to the case of SPLD
regularized K-means, our goal is to guarantee the consistency of S̃n as well as {π̃y}Ky=1 , { b̃(`) }L `=1 .
In particular, we denote F̃0 the set of all optimal solutions θ̃0 of population partially labeled LDCE where θ̃0 := ({ µ̃0(y) }K y=1 , { Γ̃0(`) }L `=1 , { π̃0y }K y=1 , { b̃0(`) }L `=1 ) . For each θ̃0 ∈ F̃0, we define
S̃0 the set of optimal rendered images associated with θ̃0. We denote G(F̃0) the corresponding set of all optimal rendered images S̃0, optimal prior probabilities { π̃0y }K y=1 , and optimal biases { b̃0(`) }L `=1 . Theorem .7. (Consistency of optimal rendering paths and optimal solutions of partially labeled LDCE) Assume that P(‖x‖ ≤ R) = 1 as x ∼ P for some given R > 0. Then, we obtain that
inf (S̃0,{π̃0y},{b̃0(`)})∈G(F̃0)
{ H(S̃n, S̃0) + K∑ y=1 |π̃y − π̃0y|+ L∑ `=1 ‖b̃(`)− b̃0(`)‖ } → 0
almost surely as n→∞.
The detail proof of Theorem .7 is postponed to Appendix C.
.6 GENERALIZATION BOUND FOR CLASSIFICATION FRAMEWORK WITH (SEMI)-SUPERVISED SETTING
In this section, we provide a simple generalization bound for certain classification function with the optimal solutions θ̃ = ( {µ̃(y)}Ky=1 , { Γ̃(`) }L `=1 , {π̃y}Ky=1 , { b̃(`) }L `=1 ) of equation 12. In particular, we denote the following function f : RD(0) × {1, . . . ,K} → R as
f(x, y) = max z∈L
{ h̃>(y, z; 0)x+ τ(θ̃y,z) } + log π̃y
for all (x, y) ∈ RD(0) × {1, . . . ,K} where h̃(y, z; 0) := Λ̃(z; 1) . . . Λ̃(z;L)µ̃(y),
Λ̃(z; `) := ∑
p∈P(`) s(`, p)T (t; `, p)B(`, p)Γ̃(`, p)
for all (y, z) and 1 ≤ ` ≤ L. To achieve the generalization bound regarding that classification function, we rely on the study of generalization bound with margin loss. For the simplicity of
argument, we assume that the true labels of x1, . . . , xn are y1, . . . , yn while y1, . . . , yn1 are not available to train. The margin of a labeled example (x, y) based on f can be defined as
ρ(f, x, y) = f(x, y)−max l 6=y f(x, l).
Therefore, the classification function f misspecifies the labeled example (x, y) as long as ρ(f, x, y) ≤ 0. The empirical margin error of f at margin coefficient Γ ≥ 0 is
Rn,γ(f) = 1
n n∑ i=1 1{ρ(f,xi,yi)≤γ}.
It is clear that Rn,0(f) is the empirical risk of 0-1 loss, i.e., we have
Rn,0(f) = 1
n n∑ i=1 1{ arg max 1≤y≤K f(xi,y) 6=yi }.
Similar to the argument in the case of SPLD regularized K-means, the optimal solutions θ̃ of partially labeled LDCE lead to a set of rendered images h̃(y, z; 0) for all (y, z) ∈ J . However, only a small fraction of rendering paths are indeed active in the following sense. There exists a subset Ln of L such that |Ln| ≤ τn|L| where τn ∈ (0, 1], which is independent of data (x1, y1), . . . , (xn, yn), and the following holds
max z∈L
{ h̃>(y, z; 0)x+ τ(θ̃y,z) } + log π̃y = max
z∈Ln
{ h̃>(y, z; 0)x+ τ(θ̃y,z) } + log π̃y
for all 1 ≤ y ≤ K. The above equation implies that Rn,Γ(f) = Rn,γ(fγn)
for all Γ ≥ 0 and n ≥ 1 where fτn(x, y) = max z∈Ln
{ h̃>(y, z; 0)x+ τ(θ̃y,z) } + log π̃y for all (x, y).
With that connection, we denote the expected margin error of classification function fτn at margin coefficient Γ ≥ 0 is
Rγ(fτn) = E1{ρ(fτn ,x,y)≤γ}. The generalization bound that we establish in this section will base on the gap between the expected margin error R0(fτn) and its corresponding empirical version Rn,Γ(fτn), which is also Rn,Γ(f). Theorem .8. (Generalization bound for margin-based classification) Assume that P (‖x‖ ≤ R) = 1 for some given R > 0 and x ∼ P . Additionally, the parameter spaces Θ` and Ω are chosen such that ‖h(y, z; 0)‖ ≤ R for all (y, z) ∈ J . For any δ > 0, with probability at least 1− δ, we have
R0(fτn) ≤ inf γ∈(0,1]
{ Rn,γ(fτn) +
8K(2K − 1) Γ √ n
( 2τn|L|(R2 + 1) + | log γ| ) + ( log log2(2γ
−1) n
)1/2 + √ log(2δ−1)
2n } where γ is the lower bound of prior probability πy for all y. Remark .9. The result of Theorem .8 gives a simple characterization for the generalization bound of classification setup from optimal solutions of partially labeled LDCE based on the number of active rendering paths, which is inherent to the structure of NRM. Such dependence of generalization bound on the number of active rendering configurations τn|L| is rather interesting and may provide a new perspective on understanding the generalization bound. Nevertheless, there are certain limitations regarding the current generalization gap: (1) the active ratio τn may change with the sample size unless we put certain constraints on the sparsity of switching variables a to reduce the number of active optimal rendering configurations; (2) the generalization bound is depth- dependent due to the involvement of the number of rendering configurations |L|. This is mainly because we have not fully taken into account all the structures of CNNs for the studying of generalization bound. Given some current progress on depth-independent generalization bound (Bartlett et al., 2017; Golowich et al., 2018), it is an interesting direction to explore whether the techniques in these work can be employed to improve |L| in the generalization bound in Theorem .8.
.7 NEURAL RENDERING MODEL IS THE UNIFYING FRAMEWORK FOR NETWORKS IN THE CONVNET FAMILY
The structure of the rendering matrices Λ(`) gives rise to MaxPooling, ReLU, and convolution operators in the CNNs. By modifying the structure of Λ(`), we can derive different types of networks in the convolutional neural network family. In this section, we define and explore several other interesting variants of NRM: the Residual NRM (ResNRM) and the Dense NRM (DenseNRM). Inference algorithms in these NRMs yield ResNet He et al. (2016) and DenseNet Huang et al. (2017), respectively. Proofs for these correspondence are given in Appendix C. Both ResNet and DenseNet are among state-of-the-art neural networks for object recognition and popularly used for other visual perceptual inference tasks. These two architectures employ skip connections (a.k.a., shortcuts) to create short paths from early layers to later layers. During training, the short paths help avoid the vanishing-gradient problem and allow the network to propagate and reuse features.
.7.1 RESIDUAL NEURAL RENDERING MODEL YIELDS RESNET
In a ResNet, layers learn residual functions with reference to the layer inputs. In particular, as illustrated in Fig. 5, layers in a ResNet are reformulated to represent the mapping F (ψ) +Wskipψ and the layers try to fit the residual mapping F (ψ) where ψ is the input feature. The term Wskipψ accounts for the skip connections/shortcuts He et al. (2016). In order to derive the ResNet, we rewrite the rendering matrix Λ(`) as the sum of a shortcut matrix Λskip(`) and a rendering matrix, both of which can be updated during the training. The shortcut matrices yields skip connections in He et al. (2016). Note that Λskip(`) depends on the template selecting latent variables s(`). In the rest of this section, for clarity, we will refer to Λ(`) and Λskip(`) as Λ(t, s; `) and Λskip(s; `), respectively, to show their dependency on latent variables in NRM. We define the Residual Neural Rendering Model as follows: Definition .10. The Residual Neural Rendering Model (ResNRM) is the Neural Rendering Model whose rendering process from layer ` to layer ` − 1, for some ` ∈ {1, 2, · · · , L}, has the residual form as follows:
h(`− 1) := (Λ(t, s; `) + Λskip(s; `))h(`), (16) where Λskip(t, s; `) is the shorcut matrices that results in skip connections in the corresponding ResNet. In particular, Λskip(t, s; `) has the following form:
Λskip(s; `) = Λ̃skip(`)M(s; `), (17)
where M(s; `) ≡ diag (s(`)) ∈ RD(`)×D(`) is a diagonal matrix whose diagonal is the vector s(`). This matrix selects the templates for rendering. Furthermore, Λ̃skip(`) is a rendering matrix that is independent of latent variables t and s.
The following theorem show that similar to how CNNs can be derived from NRM, ResNet can be derived as a bottom-up inference in ResNRM. Theorem .11. Inference in ResNRM yields skip connetions. In particular, if the rendering process at layer ` has the residual form as in Definition .10, the inference at this layer takes the following form:
ψ(`) ≡ max t(`),s(`)
{( Λ>(t, s; `) + Λ>skip(s; `) ) ψ(`− 1) + b(`) } = MaxPool ReLu Conv(Γ>(`), ψ(`− 1)) + b(`) + Λ̃>skip(`)ψ(`− 1)︸ ︷︷ ︸
skip connection
 
d = MaxPool ReLu Conv(W (`), ψ(`− 1)) + b(`) +Wskip(`)ψ(`− 1)︸ ︷︷ ︸
skip connection
  . (18)
Here, when ψ(`−1) and ψ(`) have the same dimensions, Λ̃skip(`) is chosen to be an constant identity matrix in order to derive the parameter-free, identity shortcut among layers of the same size in the ResNet. When ψ(`− 1) and ψ(`) have the different dimensions, Λ̃skip(`) is chosen to be a learnable shortcut matrix which yields the projection shortcut Wskip(`) among layers of different sizes in
the ResNet. As mentioned above, identity shortcuts and projection shortcuts are two types of skip connections in the ResNet. The operator d= implies that discriminative relaxation is applied.
In practice, the skip connections are usually across two layers or three layers. This is indeed a straightforward extension from the ResNRM. In particular, the ResNRM building block that corresponds to the ResNet building block in He et al. (2016) (see Fig. 5) takes the following form:
h(`− 2) := (Λ(t, s; `− 1)Λ(t, s; `) + Λskip(s; `))h(`).
In inference, this ResNRM building block yields the ResNet building block in Fig. 5:
ψ(`) = ReLu ( Conv ( Γ>(`),ReLu ( Conv ( Γ>(`− 1), ψ(`− 2) ) + b(`− 1) )) + b(`) + Λ̃>skip(`)ψ(`− 2)
) d = ReLu ( Conv ( W (`),ReLu ( Conv ( W (`− 1), ψ(`− 2) ) + b(`− 1)
)) + b(`) +Wskip(`)ψ(`− 2) ) . (19)
.7.2 DENSE NEURAL RENDERING MODEL YIELDS DENSENET
In a DenseNet Huang et al. (2017), instead of combining features through summation, the skip connections concatenate features. In addition, within a building block, all layers are connected to each other (see Fig. 6). Similar to how ResNet can be derived from ResNRM, DenseNet can also be derived from a variant of NRM, which we call the Dense Neural Rendering Model (DenseNRM). In DenseNRM, the rendering matrix Λ(`) is concatenated by an identity matrix. This extra identity matrix, in inference, yields the skip connections that concatenate features at different layers in a DenseNet. We define DenseNRM as follows.
Definition .12. The Dense Neural Rendering Model (DenseNRM) is the Neural Rendering Model whose rendering process from layer ` to layer ` − 1, for some ` ∈ {1, 2, · · · , L}, has the residual form as follows:
h(`− 1) := [ Λ(t, s; `)h(`), 1D(`)h(`) ] . (20)
We again denote Λ(`) as Λ(t, s; `) to show the dependency of Λ(`) on the latent variables t(`) and s(`). The following theorem establishes the connections between DenseNet and DenseNRM.
Theorem .13. Inference in DenseNRM yields DenseNet building blocks. In particular, if the rendering process at layer ` has the dense form as in Definition .12, the inference at this layer takes
the following form: ψ(`) ≡ [ max t(`),s(`) { Λ>(t, s; `)ψ(`− 1) + b(`) } ψ(`− 1) ]
=
[ MaxPool(ReLu(Conv(Γ>(`), ψ(`− 1)) + b(`)))
ψ(`− 1) ] d = [ MaxPool(ReLu(Conv(W (`), ψ(`− 1)) + b(`))) ψ(`− 1) ] . (21)
In Eqn. 21, we concatenate the output MaxPool ReLu(Conv(W (`), ψ(` − 1)) + b(`))) at layer ` with the input feature ψ(`− 1) at layer `− 1 to generate the input to the next layer ψ(`), just like in the DenseNet. Proofs for Theorem .11 and .13 can be found in Appendix B. The approach to proving Theorem .11 can be used to prove the result in Eqn. 19.

heading: Appendix C
text: In this appendix, we provide the proofs for key results in the paper.
.8 PROOF FOR THEOREM 2.2: DERIVING CONVOLUTIONAL NEURAL NETWORKS FROM THE NEURAL RENDERING MODEL
To ease the clarity of the proof presentation, we ignore the normalizing factor 1σ2 and only consider the proof for two layers, i.e., L = 2. The argument for L ≥ 3 is similar and can be derived recursively from the proof for L = 2. Similar proof holds when the normalizing factor 1σ2 is considered. Now, we obtain that
max z
{ h>(y, z; 0)x+ η(y, z) } = max t(1),s(1) t(2),s(2) { ∑ p1∈P(1) h(1, p1)s(1, p1)Γ >(1, p1)B >(1, p1)T >(t; 1, p1)x
+ ∑
p1∈P(1) h(1, p1)s(1, p1)b(t; 1, p1) + ∑ p2∈P(2) µ(y; p2)s(2, p2)b(t; 2, p2)
}
= max t(1),s(1) t(2),s(2) { ∑ p1∈P(1) h(1, p1)s(1, p1) ( Γ>(1, p1)B >(1, p1)T >(t; 1, p1)x+ b(t; 1, p1) ) +
∑ p2∈P(2) µ(y; p2)s(2, p2)b(t; 2, p2) } (a) = max
t(2),s(2) { ∑ p1∈P(1) h(1, p1) max t(1,p1),s(1,p1) s(1, p1) ( Γ>(1, p1)B >(1, p1)T >(t; 1, p1)x+ b(t; 1, p1) ) +
∑ p2∈P(2) µ(y; p2)s(2, p2)b(t; 2, p2) } = A,
where equation in (a) is due to the non-negativity assumption that h(1, p1) ≥ 0, just as in max-sum and max-product message passing. We define ψ(1, p1) as follows:
ψ(1, p1) = max t(1,p1),s(1,p1)
s(1, p1) ( Γ>(1, p1)B >(1, p1)T >(t; 1, p1)x+ b(t; 1, p1) ) ,
and let ψ(1) = (ψ(1, p1))p1∈P(1) be the vector of ψ(1, p1). The following holds:
A = max t(2),s(2)
{ h>(1)ψ(1) + ∑ p2∈P(2) µ(y; p2)s(2, p2)b(t; 2, p2) } (b) = max t(2),s(2) { ∑ p2∈P(2) µ(y; p2)s(2, p2)Γ >(2, p2)B >(2, p2)T >(t; 2, p2)ψ(1)
+ ∑
p2∈P(2) µ(y; p2)s(2, p2)b(t; 2, p2) } (c) =
∑ p2∈P(2) µ(y; p2) max t(2),s(2) s(2, p2) ( Γ>(2, p2)B >(2, p2)T >(t; 2, p2)ψ(1) + b(t; 2, p2) ) ︸ ︷︷ ︸
ψ(2,p2)
=µ>(y)ψ(2).
Here ψ(2) = (ψ(2, p2))p2∈P(2) is the vector of ψ(2, p2), and in line (b) we substitute h(1) by: h(1) = ∑
p∈P(2) s(2, p2)T (t; 2, p2)B(2, p2)Γ(2, p2)µ(y; p2).
Notice that line (a) and (c) form a recursion. Therefore, we finish proving that the feedforward step in CNNs is the latent variable inference in NRM if ψ(1) has the structure of the building block of
CNNs, i.e. MaxPool (ReLu (Conv())). Indeed,
ψ(1) = (ψ(1, p(1)))p(1)∈P(1)
= ( max
t(1,p1),s(1,p1) s(1, p1)
( Γ>(1, p1)B >(1, p1)T >(t; 1, p1)x+ b(t; 1, p1) )) p(1)∈P(1)
= ( max (( ReLu ( Γ>(1, p1)B >(1, p1)T >(t; 1, p1)x+ b(t; 1, p1) )) t(1,p1)=0,1,2,3 )) p(1)∈P(1)
(22) = MaxPool ( ReLu ( Conv ( Γ>(1), x ) + b(t; 1) )) d = MaxPool (ReLu (Conv (W (1), x) + b(t; 1))) ,
where W (1) = Γ>(1) corresponds to the weights at layer 1 of the CNN, and d= implies that discriminative relaxation is applied. In Eqn. 22, since s(1, p1) ∈ {0, 1}, max
s(1,p1) (s(1, p1)× .) is equivalent to
max( . , 0) and, therefore, yields the ReLU function. Also, in order to compute max t(1,p1) (), we take the brute force approach, computing the function inside the parentheses for all possible values of t(1, p1) and pick the maximum one. This procedure is equivalent to the MaxPool operator in CNNs. Here we can make the bias term b(t; 1) independent of t and b(t; 1, p1) are the same for all pixels p1 in the same feature map of h(1) as in CNNs. Similarly, ψ(2) = MaxPool (ReLu (Conv (W (2), ψ(1)) + b(2))). Thus, we obtain the conclusion of the theorem for L = 2.
.9 PROOF FOR THEOREM .11: DERIVING THE RESIDUAL NETWORKS FROM THE RESIDUAL NEURAL RENDERING MODEL
Similar to the proof in Section .8 above, when the rendering process at each layer in ResNRM is as in Eqn. 16, the activations ψ(`) is given by:
ψ(`) = ( max
t(`,p),s(`,p) s(`, p)
(( Γ>(`, p)B>(`, p)T>(t; `, p) + Λ̃>skip(`, p) ) ψ(`− 1) + b(t; `, p) )) p∈P(`)
= ( max (( ReLu ( Γ>(`, p)B>(`, p)T>(t; `, p)ψ(`− 1) + b(t; `, p)
+ Λ̃>skip(`, p)ψ(`− 1) ))
t(`,p)=0,1,2,3 )) p∈P(`)
= MaxPool ( ReLu ( Conv ( Γ>(`, p), ψ(`− 1) ) + b(t; `) + Λ̃>skip(`)ψ(`− 1) )) d = MaxPool ReLu Conv(W (`), ψ(`− 1)) + b(t; `) +Wskip(`)ψ(`− 1)︸ ︷︷ ︸
skip connection
  . (23)
Here we can again make the bias term b(t; `) independent of t and b(t; `, p) are the same for all pixels p` in the same feature map of h(`) as in CNNs. We obtain the conclusion of the Theorem .11.
.10 PROOF FOR THEOREM .13: DERIVING THE DENSELY CONNECTED NETWORKS FROM THE DENSE NEURAL RENDERING MODEL
Similar to the proof for Theorem .11 above, when the rendering process at each layer in ResNRM is as in Eqn. 20, the activations ψ(`) is given by:
ψ(`) ≡
 ( max s(`,p),t(`,p) ( s(`, p) ( Γ>(`, p)B>(`, p)T>(t; `, p)ψ(`− 1) + b(t; `, p) ))) p∈P(`)
ψ(`− 1)  = [ ( max (( ReLu ( Γ>(`, p)B>(`, p)T>(t; `, p)ψ(`− 1) + b(t; `, p) )) t(`,p)=0,1,2,3 )) p∈P(`)
ψ(`− 1)
]
=
[ MaxPool(ReLu(Conv(Γ>(`, p), ψ(`− 1)) + b(t; `)))
ψ(`− 1) ] d = [ MaxPool(ReLu(Conv(W (`), ψ(`− 1)) + b(t; `)))
ψ(`− 1)
] . (24)
Again we can make the bias term b(t; `) independent of t and b(t; `, p) are the same for all pixels p` in the same feature map of h(`) as in CNNs. We obtain the conclusion of the Theorem .13.
.11 PROVING THAT THE PARAMETRIZED JOINT PRIOR P(Y,Z) IS A CONJUGATE PRIOR
Again, for simplicity, we only consider the proof for two layers. The argument for L ≥ 3 is similar and can be derived recursively from the proof for L = 2. In the derivation below, h(2) is µ(y). As in Eqn.1 in the definition of the NRM, the joint prior of y and z is given by:
p(y, z) ∝ exp ( 1
σ2 b>(t; 2)(s(2) h(2)) + 1 σ2 b>(t; 1)(s(1) h(1)) + lnπy ) = exp ( 1
σ2 h>(2)(b(t; 2) s(2)) + 1 σ2 h>(1)(b(t; 1) s(1)) + lnπy ) = exp ( 1
σ2 h>(2)(b(t; 2) s(2)) + 1 σ2 h>(2)Λ>(2)(b(t; 1) s(1)) + lnπy ) = exp ( 1
σ2 h>(2)
[ Λ>(2)(b(t; 1) s(1)) + b(t; 2) s(2) ] + lnπy ) (25)
Furthermore, as explained in Section .14 of Appendix D, due to the constant norm assumption with h(y, z; 0), the likelihood p(x|y, z) is estimated as follows:
p(x|y, z) ∝ exp ( 1
σ2 h>(y, z; 0)x ) = exp ( 1
σ2 h>(2)Λ>(2)Λ>(1)x ) The posterior p(y, z|x) is given by:
p(y, z|x) ∝ exp ( 1
σ2 h>(2)
[ Λ>(2)(b(t; 1) s(1)) + Λ>(2)Λ>(1)x+ b(t; 2) s(2) ] + lnπy ) = exp ( 1
σ2 h>(2)
[ Λ>(2)(b(t; 1) s(1) + Λ>(1)x) + b(t; 2) s(2) ] + lnπy ) (26)
Comparing Eqn. 25 and Eqn. 26, we see that the prior and the posterior have the same functional form. This completes the proof.
.12 DERIVING OTHER COMPONENTS IN THE CONVOLUTIONAL NEURAL NETWORKS
.12.1 DERIVING THE LEAKY RECTIFIED LINEAR UNIT (LEAKY RELU)
The Leaky ReLU can be derived from the NRM in the same way as we derive the ReLU in Section .8, but instead of s(`) ∈ {0, 1}, we now let s(`) ∈ {α, 1}, where α is a small positive constant. Then, in
Eqn. 22, max s(`,p) (s(`, p) × .) is equivalent to [. < 0](α × .) + [. >= 0](.), which is the LeakyReLU function. Note that compared to Eqn. 22, here we replace the layer number (1) by (`) since the same derivation can be applied at any layer.
.13 PROOFS FOR BATCH NORMALIZATION
In this section we will derive the batch normalization from a 2-layer DRM. The same derivation can be generalized to the case of K-layer DRM.
In order to derive the Batch Normalization, we normalize the intermediate rendering image h(`).
h(y, z; 0) = ∑
p1∈P(1) s(1, p1)T (t; 1, p1)B(1, p1)Γ(1, p1)
1
σh(1, p1) (h(1, p1)− Eh(1, p1))− Eh(y,z;0)
= ∑
p1∈P(1) s(1, p1)T (t; 1, p1)B(1, p1)Γ(1, p1)
1
σh(1, p1) × (( ∑
p2∈P(2) s(1, p2)T (t; 2, p2)B(2, p2)Γ(2, p2)
1
σh(y, z; 0) h(y, z; 0)
) (1, p1)− Eh(1, p1) ) − Eh(y,z;0) (27)
During inference, we de-mean the input image and find z∗ = arg max z h>(y, z; 0)(x− Eh(y,z;0)). In particular, the inference can be derived as follows:
max z h>(y, z; 0)(x− Eh(y,z;0))
= max z ( ∑ p1∈P(1) s(1, p1)T (t; 1, p1)B(1, p1)Γ(1, p1) 1 σh(1, p1) ( h(1, p1)− Eh(1, p1) ))> (x− Eh(y,z;0))
− E>h(y,z;0)(x− Eh(y,z;0))︸ ︷︷ ︸ const w.r.t y,z
a≈max z ∑ p1∈P(1) ( s(1, p1) ( h(1, p1)− Eh(1, p1) ) 1
σh(1, p1)︸ ︷︷ ︸ scalar
Γ>(1, p1)B >(1, p1)T >(t; 1, p1)︸ ︷︷ ︸ row vector (x− Ei)︸ ︷︷ ︸ column vector︸ ︷︷ ︸
scalar
)
︸ ︷︷ ︸ dot product
+ const := A+ const.
Direct computation leads to
A = max z ∑ p1∈P(1) ( s(1, p1) ( h(1, p1)− Eh(1, p1) ) σψ(1, p1)
σh(1, p1)︸ ︷︷ ︸ α(1,p1)
1
σψ(1, p1)︸ ︷︷ ︸ normalize
× (Γ>(1, p1)B>(1, p1)T>(t; 1, p1)x− E[Γ>(1, p1)B>(1, p1)T>(t; 1, p1)x])︸ ︷︷ ︸ de-mean
)
= max z(2) ∑ p1∈P(1) max s(1,p1),t(1,p1) ( s(1, p1) ( h(1, p1)− Eh(1, p1) )
× BatchNorm(Γ>(1, p1)B>(1, p1)T>(t; 1, p1)x;α(1, p1)) )
= max z(2)
( h(1)− Eh(1) )>( MaxPool(ReLu(BatchNorm(Conv(Γ(1), x);α(1), 0))) ) (28)
In line (a), we approximate Eh(y,z;0) by its empirical value Ei. The de-mean and normalize operators with the scale parameter α(1) and the shift parameter β(1) = 0 in the equations above already have the form of batch normalization. Note that when the model is trained with Stochastic Gradient Descent (SGD), the scale and shift parameters at each layer also account for the error in evaluating statistics of the activations using the mini-batches. Thus, β(1) is not 0 any more, but a parameter learned during the training. Also, in the equations above, ψ(1) is the activations at layer 1 in CNNs and given by:
ψ(1) = MaxPool(ReLu(Normalize(Demean(Conv(Γ(1), x))))), (29) where σψ(1) is the standard deviation of the ψ(1). Eqn.28 can be expressed in term of ψ(1) as follows:
max z(2)
( h(1)− Eh(1) )> ψ(1) + const
= max z(2)
h>(1)(ψ(1)− Eψ(1)) + (h>(1)Eψ(1)− E>h (1)ψ(1)) + const
= max z(2) ∑ p2∈P(2) ( s(2, p2)µ(y, p2) σψ(2, p2)
σµ(y)(p2)︸ ︷︷ ︸ α(2,p2)
1
σψ(2, p2)︸ ︷︷ ︸ normalize
× ( Γ>(2, p2)B >(2, p2)T
>(t; 2, p2)ψ(1)− E[Γ>(2, p2)B>(2, p2)T>(t; 2, p2)ψ(1)]︸ ︷︷ ︸ de-mean
+ h(1, p1)Eψ(1, p1)− Eh(1, p1)ψ(1, p1)︸ ︷︷ ︸ β(2,p2)
)) + const
= max z(2)
µ>(y) ( MaxPool(ReLu(BatchNorm(Conv(Γ(2), ψ(1));α(2), α(2) β(2)))) ) + const
(30)
The batch normalization at this layer of the CNN has the scale parameter α(2) and the shift parameter is the element-wise product of α(2) and β(2).
.14 PROOFS FOR CONNECTION BETWEEN NRM AND CROSS ENTROPY
PROOF OF THEOREM .1 (a) To ease the presentation of proof, we denote the following key notation
A := max (zi)ni=1,θ∈Aγ
1
n n∑ i=1 ln p(yi|xi, zi; θ).
From the definition of Aγ = {θ : ‖h(y, z; 0)‖ = γ}, we achieve the following equations
A = max θ∈Aγ
1
n n∑ i=1 max zi log p(yi|xi, zi; θ)
= max θ∈Aγ
1
n n∑ i=1 max zi
log p(xi|yi, zi; θ)p(yi|zi; θ) K∑ y=1 p(xi|y, zi; θ)p(y|zi; θ)
= max θ∈Aγ
1
n n∑ i=1 max zi { log p(xi|yi, zi; θ) + log p(yi, zi|θ)− log ( K∑ y=1 p(xi|y, zi; θ)p(y, zi|θ) )} .
From the formulation of NRM, we have the following formulation of prior probabilities p(y, z|θ)
p(y, z|θ) = exp
( η(y, z)
σ2 ) πy∑
y′,z′ exp ( η(y′, z′) σ2 ) πy′ .
for all (y, z). By means of the previous equations, we eventually obtain that
A = max θ∈Aγ
1
n n∑ i=1 max zi { log p(xi|yi, zi; θ) + log  exp ( η(yi, zi) σ2 ) πyi∑
y,z exp
( η(y, z)
σ2
) πy

− log K∑ y=1 ( p(xi|y, zi; θ) exp
( η(y, zi)
σ2 ) πy∑
y′,z′ exp ( η(y′, z′) σ2 ) πy′
)}
= max θ∈Aγ
1
n n∑ i=1 max zi ( log p(xi|yi, zi; θ) + log ( exp ( η(yi, zi) σ2 ) πyi ))
− log ( K∑ y=1 ( p(xi|y, zi; θ) exp ( η(y, zi) σ2 ) πy ) .
By defining ψi(y, zi) := log p(xi|y, zi; θ) + log ( exp ( η(y, zi)
σ2
) πy ) for all 1 ≤ i ≤ n, the above
equation can be rewritten as
A = max θ∈Aγ
1
n n∑ i=1 max zi
( log (exp(ψi(yi, zi)))− log ( K∑ y=1 exp(ψi(y, zi)) )) (31)
= max θ∈Aγ
1
n n∑ i=1 max zi log (Softmax(ψi(yi, zi)))
≥ max θ∈Aγ
1
n n∑ i=1 log ( Softmax ( max zi ψi(yi, zi)) )) = B. (32)
By means of direct computation, the following equations hold
B = max θ∈Aγ
1
n n∑ i=1 log ( Softmax ( max zi log p(xi|yi, zi; θ) + η(yi, zi)/σ2 + log πyi ))
= max θ∈Aγ
1
n n∑ i=1 log ( Softmax ( max zi −‖xi − µyi,zi‖ 2 2σ2 + η(yi, zi) σ2 + log πyi )) (i) = max
θ∈Aγ
1
n n∑ i=1 log ( Softmax ( max zi ( h>(yi, zi; 0)xi + η(yi, zi) σ2 ) + byi ))
= − min θ∈Aγ
1
n n∑ i=1 − log ( Softmax ( max zi ( h>(yi, zi; 0)xi + η(yi, zi) σ2 ) + byi ))
= − min θ∈Aγ
1
n n∑ i=1 − log q(yi|xi) = − min θ∈AΓ Hp,q(y|x)
where equation (i) is due to the constant norm assumption with rendered images h(y, z; 0). Therefore, we achieve the conclusion of part (a) of the theorem.
(b) Regarding the upper bound, from the definition of zi, we obtain that
max zi
log (Softmax(ψi(yi, zi)))− log ( Softmax (
max zi ψi(yi, zi)) )) = log (Softmax(ψi(yi, zi)))− log ( Softmax ( max zi ψi(yi, zi)) ))
≤ log max zi exp(ψi(yi, zi))
K∑ y=1 exp(ψi(y, zi))
− log ( Softmax (
max zi ψi(yi, zi))
))
= log ( K∑ y=1 exp(max zi ψi(y, zi)) ) − log ( K∑ y=1 exp(ψi(y, zi)) ) ≤ logK + max
y max zi ψi(y, zi)−max y ψi(y, zi)
≤ logK + max y ( max zi ψi(y, zi)− ψi(y, zi) )
for any 1 ≤ i ≤ n. As a consequence, we obtain the conclusion of part (b) of the theorem.
PROOF OF THEOREM .4 (a) From the definitions of (yi, zi) and (ỹi, z̃i), we obtain that
Un = min θ
1
n n∑ i=1 ( 1 2 ‖xi − h(yi, zi; 0)‖2 − log πyi,zi )
= min θ
{ 1
n n∑ i=1 ( 1 2 ‖xi − h(yi, zi; 0)‖2 − η(yi, zi)− log πyi ) + log
( ∑ (y′,z′)∈J exp(η(y′, z′) + log πy′) )} = A
By means of direct computation, the following inequality holds
A ≤ min θ
{ 1
n n∑ i=1 ( 1 2 ‖xi − h(ỹi, z̃i; 0)‖2 − η(ỹi, z̃i)− log πyi ) + log
( ∑ (y′,z′)∈J exp(η(y′, z′) + log πy′) )} .
It is clear that∑ (y′,z′)∈J exp(η(y′, z′) + log πy′) ≤ |L| ∑ y′ exp(max z′∈L η(y′, z′) + log πy′).
Combining this inequality with the inequality of the term A in the above display, we have
Un ≤ min θ
{ 1
n n∑ i=1 ( 1 2 ‖xi − h(ỹi, z̃i; 0)‖2 − η(ỹi, z̃i)− log πỹi ) + (log πỹi − log πyi)
+ log (∑ y′ exp(max z′∈L η(y′, z′) + log πy′) ) + log |L| }
= min θ
1
n n∑ i=1 {(‖xi − h(ỹi, z̃i; 0)‖2 2 − log(πỹi,z̃i) ) + ( log πỹi − log πyi )} + log |L|
≤ min θ
1
n n∑ i=1 (‖xi − h(ỹi, z̃i; 0)‖2 2 − log(πỹi,z̃i) ) + log ( 1 γ − 1 ) + log |L|
= Vn + log
( 1 γ − 1 ) + log |L|
where the final inequality is due to the fact that πỹi/πyi ≤ (1− γ)/γ for all 1 ≤ i ≤ n. Therefore, we achieve the conclusion of part (a) of the theorem. (b) Similar to the proof argument with part (a), we have
Un = min θ
{ 1
n n∑ i=1 ( 1 2 ‖xi − h(yi, zi; 0)‖2 − η(yi, zi)− log πyi ) + log
( ∑ (y′,z′)∈J exp(η(y′, z′) + log πy′) )}
≥ min θ
{ 1
n n∑ i=1 ( 1 2 ‖xi‖2 − h(ỹi, z̃i; 0)>xi − η(ỹi, z̃i) + ‖h(yi, zi; 0)‖2/2− log πyi ) + log
(∑ y′ exp(max z′∈L η(y′, z′) + log πy′) ) = B.
Direct computation with B leads to
B = min θ
1
n n∑ i=1 {(‖xi − h(ỹi, z̃i; 0)‖2 2 − log(πỹi,z̃i) ) + ( log πyi − log πỹi ) +‖h(yi, zi; 0)‖2 − ‖h(ỹi, z̃i; 0)‖2
} ≥ min
θ
1
n n∑ i=1 (‖xi − h(ỹi, z̃i; 0)‖2 2 − log(πỹi,z̃i) ) + min θ 1 n n∑ i=1 ( log πyi − log πỹi )
+ min θ
1
n n∑ i=1 ( ‖h(yi, zi; 0)‖2 − ‖h(ỹi, z̃i; 0)‖2 )
≥ Vn + log ( γ
1− γ
) + min
θ
1
n n∑ i=1 ( ‖h(yi, zi; 0)‖2 − ‖h(ỹi, z̃i; 0)‖2 ) where the final inequality is due to the fact that πỹi/πyi ≥ γ/(1 − γ) for all 1 ≤ i ≤ n. As a consequence, we achieve the conclusion of part (b) of the theorem.
.15 PROOFS FOR STATISTICAL GUARANTEE AND GENERALIZATION BOUND FOR (SEMI)-SUPERVISED LEARNING
PROOF OF THEOREM .6 The proof of this theorem relies on several results with uniform laws of large numbers. In particular, we will need to demonstrate the following results
sup θ ∣∣∣∣ 1n1 n1∑ i=1 max (y,z)∈J ( h>(y, z; 0)xi + τ(θy,z) ) − ∫
max (y,z)∈J
( h>(y, z; 0)x+ τ(θy,z) ) dP (x) ∣∣∣∣→ 0, (33) Ln1 = sup
θ ∣∣∣∣ 1n1 n1∑ i=1 ∑ (y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( h>(y,z;0)xi+τ(θy,z) ) (‖h(y′, z′; 0)‖2 2 − log πy′ )
− ∫ ( ∑
(y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( h>(y,z;0)x+τ(θy,z) ) (‖h(y′, z′; 0)‖2 2 − log πy′ )) dP (x) ∣∣∣∣→ 0,(34) sup θ ∣∣∣∣ 1n− n1 n∑
i=n1+1
max z∈L
( h>(yi, z; 0)xi + τ(θyi,z) ) − ∫
max z∈L
( h>(y, z; 0)x+ τ(θy,z) ) dQ(x, c) ∣∣∣∣→ 0, (35) E(1)n = sup
θ ∣∣∣∣ 1n− n1 n∑
i=n1+1 ∑ z′∈L 1z′=arg maxz∈L ( h>(yi,z;0)xi+τ(θyi,z) ) (‖h(yi, z′; 0)‖2 2 − log πyi )
− ∫ ( ∑
z′∈L 1z′=arg maxg∈J ( h>(y,z;0)x+τ(θy,z) ) (‖h(y, z′; 0)‖2 2 − log πy )) dQ(x, c) ∣∣∣∣→ 0, (36) E(2)n = sup
θ ∣∣∣∣ 1n− n1 n∑
i=n1+1
log qθ(yi|xi)− ∫ log qθ(y|x)dQ(x, c) ∣∣∣∣→ 0, (37)
E(3)n = sup θ ∣∣∣∣ 1n n∑ i=1 K∑ y=1 qθ(y|xi) log ( qθ(y|xi) πy )
− ∫ K∑
y=1
qθ(y|x)dQ(x, c) log ( qθ(y|x) πy ) dP (x) ∣∣∣∣→ 0, (38) almost surely as n→∞. The proof for equation 35 is similar to that of equation 33; therefore, it is omitted.
Proof of equation 33: It is clear that
sup θ ∣∣∣∣ 1n1 n1∑ i=1 max (y,z)∈J ( h>(y, z; 0)xi + τ(θy,z) ) − ∫ max (y,z)∈J ( h>(y, z; 0)x+ τ(θy,z) ) dP (x) ∣∣∣∣ ≤ sup |S′|≤|J | ∣∣∣∣ 1n1 n1∑ i=1 max s∈S′ s>[xi, 1]− ∫ max s∈S′ s>[x, 1]dP (x)
∣∣∣∣ where [x, 1] ∈ RD(0)+1 denotes the vector forms by concatenating 1 to x ∈ RD(0) and S′ in the above supremum is the set of finite elements in RD(0)+1. Therefore, to achieve the result of equation 33, it is sufficient to show that
sup |S′|≤|J | ∣∣∣∣ 1n1 n1∑ i=1 max s∈S′ s>[xi, 1]− ∫ max s∈S′ s>[x, 1]dP (x) ∣∣∣∣→ 0. (39)
To obtain the conclusion of equation 39, we utilize the classical result with bracketing entropy to establish the uniform laws of large number (cf. Lemma 3.1 in van de Geer (2000)). In particular, we denote G to be the family of function on RD(0) with the form fS′(x) = max
s∈S′ s>[x, 1] where
S′ ∈ O|J |, which contains all sets that have at most |J | elements in RD (0)+1. Due to the assumption with distribution P , we can restrict O|J | to contain only set S′ with elements in B(R), which is a closed ball of radius R on RD(0)+1. By means of Lemma 2.5 in (van de Geer, 2000), we can find a finite set Eδ such that each element in B(R) is within distance δ to some element of Eδ for all δ > 0. We denote O|J |(δ) to be the subset of O|J | such that it only contains sets with elements in Eδ for all δ > 0. Now, for each set S′ = {s1, . . . , sk} ∈ O|J |, we can choose corresponding set S = {s′1, . . . , s′k} ∈ O|J |(δ) such that ‖si − s′i‖ ≤ δ for all 1 ≤ i ≤ k. Now, we denote
fS(x) = max s∈S
s>[x, 1] + δ‖[x, 1]‖,
f S (x) = max s∈S
s>[x, 1]− δ‖[x, 1]‖
for any S ∈ O|J |(δ). It is clear that fS(x) ≤ fS(x) ≤ fS(x) for all x ∈ R D(0) . Furthermore, we also have that∫ (fS(x)− fS(x))dP (x) = 2δ ∫ ‖[x, 1]‖dP (x) ≤ 2δ (∫ ‖x‖dP (x) + 1 ) .
For any > 0, by choosing δ < 2( ∫ ‖x‖dP (x) + 1) then we will have that ∫ (fS(x) −
f S (x))dP (x) < . It implies that the -bracketing entropy of G is finite for the L1 norm with distribution P (for the definition of bracketing entropy, see Definition 2.2 in van de Geer (2000)). According to Lemma 3.1 in van de Geer (2000), it implies that G satisfies the uniform law of large numbers, i.e., equation 39 holds.
Proof of equation 34: To achieve the conclusion of this claim, we will need to rely on the control of Rademacher complexity based on Vapnik-Chervonenkis (VC) dimension. In particular, we firstly demonstrate that
sup |S′|=k ∣∣∣∣ 1n1 n1∑ i=1 1{ j=arg max
1≤l≤|S′| [xi,1]>s′l
} − ∫
1{ j=arg max
1≤l≤|S′| [x,1]>s′l
}dP (x) ∣∣∣∣→ 0 (40)
almost surely as n1 →∞ for each 1 ≤ j ≤ k and k ≥ 1 where the supremum is taken with respect to S′ = {s′1, . . . , s′k}. For each j, we denote the Rademacher complexity as follows
Rn1 = E sup |S′|=k ∣∣∣∣ 1n1 n1∑ i=1 σi1{ j=arg max
1≤l≤|S′| [xi,1]>s′l }∣∣∣∣ where σ1, . . . , σn1 are i.i.d. Rademacher random variables, i.e., P(σi = −1) = P(σi = 1) = 1/2 for 1 ≤ i ≤ n1. Then, for any n1 ≥ 1 and δ ≥ 0, according to standard argument with Rademacher complexity (Vershynin, 2011),
sup |S′|=k ∣∣∣∣ 1n1 n1∑ i=1 1{ j=arg max
1≤l≤|S′| [xi,1]>s′l
} − ∫
1{ j=arg max
1≤l≤|S′| [x,1]>s′l
}dP (x) ∣∣∣∣ ≤ 2Rn1 + δ
with probability at least 1 − 2 exp ( − n1δ 2
8
) . According to Borel-Cantelli’s lemma, to achieve
equation 40, it is sufficient to demonstrate that Rn1 → 0 as n1 →∞. To achieve that result, we will utilize the study of VC dimension with partitions (cf. Section 21.5 in Devroye et al. (1996)). In particular, for each set S′ = (s′1, . . . , s ′ k), it gives rise to
the partition Ai = { x ∈ RD(0) : [x, 1]>s′i ≥ [x, 1]>s′l ∀ l ∈ {1, . . . , k} } as 1 ≤ i ≤ k. For
our purpose with equation 40, it is sufficient to consider Pn1 = { Aj , ⋃ i 6=j Ai } , which is a partition of B(R), for each set S′ with k elements. We denote F to be the collection of all Pn for all S′ with k elements and B(Pn) the collection of all sets obtained from the unions of elements of Pn. For each data (x1, . . . , xn1), we let NF (x1, . . . , xn1) the number of different sets in {(x1, . . . , xn1) ∩A : A ∈ B(Pn1) for Pn1 ∈ F}. The shatter coefficient of F is defined as
∆n1(F) = s(F , n1) = max (x1,...,xn1 ) NF (x1, . . . , xn1).
According to Lemma 21.1 in Devroye et al. (1996), ∆n1(F) ≤ 4∆∗n1(F) where ∆∗n1(F) is the maximal number of different ways that n1 points can be partitioned by members of F . Now, for
each element Pn1 = { Aj , ⋃ i 6=j Ai } of F , it is clear that the boundaries between Aj and ⋃ i 6=j Ai are subsets of hyperplanes. From the formulation of Aj , we have at most k − 1 boundaries between Aj and
⋃ i 6=j Ai. From the classical result of Dudley (1978), each n1 points in B(R) can be splitted
by a hyperplane in at most nD (0)+1
1 different ways as the VC dimension of the hyperplane is at
most D(0) + 1. As a consequence, we would have ∆∗n1(F) ≤ n (D(0)+1)(k−1) 1 , which leads to ∆n1(F) ≤ 4n(D (0)+1)(k−1) 1 .
Going back to our evaluation with Rademacher complexity Rn1 , by means of Massart’s lemma, we have that
Rn1 = E ( Eσ sup |S′|=k ∣∣∣∣ 1n1 n1∑ i=1 σi1{ j=arg max
1≤l≤|S′| [xi,1]>s′l
}∣∣∣∣|x1, . . . , xn1)
≤ E (√
2 log 2NF (x1, . . . , xn1) n1
) ≤ √
2(log 8 + (D(0) + 1)(k − 1) log n1) n1 → 0 (41)
as n1 →∞. Therefore, equation 40 is proved.
Proof of equation 36: To achieve the conclusion of this claim, we firstly demonstrate that
sup |S′|=k ∣∣∣∣ 1n− n1 n∑
i=n1+1
1{ j=arg max
1≤l≤|S′| [xi,1]>s′l
}1{yi=l}
− ∫
1{ j=arg max
1≤l≤|S′| [x,1]>s′l
}1{c=l}dQ(x, c) ∣∣∣∣→ 0 (42)
almost surely as n→∞ for each 1 ≤ j ≤ k and 1 ≤ l ≤ K where k ≥ 1 and the supremum is taken with respect to S′ = {s′1, . . . , s′k}. The proof of the above result will rely on VC dimension with Voronoi partitions being established in equation 34. In particular, according to the standard argument with Rademacher complexity, it is sufficient to demonstrate that
R′n = E sup |S′|=k ∣∣∣∣ 1n− n1 n∑
i=n1+1
σi1{ j=arg max
1≤l≤|S′| [xi,1]>s′l
}1{yi=l} ∣∣∣∣→ 0.
By means of the inequality with Rademacher complexity in equation 41, we obtain that
R′n = n−n1∑ v=0 ∑ ~c∈Av E ( Eσ sup |S′|=k ∣∣∣∣ 1n− n1 n∑ i=n1+1 σi1{ j=arg max 1≤l≤|S′| [xi,1]>s′l }1{yi=l} ∣∣∣∣|~c ∈ Av)P(~c ∈ Av)
= n−n1∑ v=0 E sup |S′|=k ∣∣∣∣ 1n− n1 n1+v∑ i=n1+1 σi1{ j=arg max
1≤l≤|S′| [xi,1]>s′l
}∣∣∣∣pvl (1− pl)n−n1−v(n− n1v )
≤ n−n1∑ v=1
v
n− n1
√ 2(log 8 + (D(0) + 1)(k − 1) log v)
v pvl (1− pl)n−n1−v ( n− n1 v )
≤ √ 2 log 8
n− n1 n−n1∑ v=1 √ v n− n1 pvl (1− pl)n−n1−v ( n− n1 v )
+
√ 2(D(0) + 1)(k − 1) log(n− n1)
n− n1
n−n1∑ v=1
√ v log v
(n− n1) log(n− n1) pvl (1− pl)n−n1−v ( n− n1 v )
≤ (√ 2 log 8
n− n1 +
√ 2(D(0) + 1)(k − 1) log(n− n1)
n− n1
) n−n1∑ v=1 pvl (1− pl)n−n1−v ( n− n1 v )
=
(√ 2 log 8
n− n1 +
√ 2(D(0) + 1)(k − 1) log(n− n1)
n− n1
) (1− (1− pl)n−n1)→ 0
as n→∞ where ~c = (cn−n1+1, . . . , cn) and Av is the set of ~c such that there are exactly v values of yi to be l for 0 ≤ v ≤ n − n1. The final inequality is due to the fact that v/(n − 1) ≤ 1 and v log v/ {(n− 1) log(n− 1)} ≤ 1 for all 1 ≤ v ≤ n− n1. Therefore, we achieve the conclusion of equation 42.
Now, coming back to equation 36, by means of triangle inequality, we achieve that
E(1)n ≤ K∑ l=1 sup θ ∣∣∣∣ 1n− n1 n∑ i=n1+1 ∑ z′∈L 1z′=arg maxz∈L ( h>(yi,z;0)xi+τ(θyi,z) )1{yi=l} (‖h(yi, z′; 0)‖2 2 − log πyi )
− ∫ ( ∑
z′∈L 1z′=arg maxg∈J ( h>(y,z;0)x+τ(θy,z) )1{y=l} (‖h(y, z′; 0)‖2 2 − log πy )) dQ(x, y) ∣∣∣∣ ≤
K∑ l=1 sup θ ∑ z′∈L ∣∣∣∣‖h(l, z′; 0)‖22 − log πl ∣∣∣∣∣∣∣∣ 1n− n1 n∑ i=n1+1 1z′=arg maxz∈L ( h>(yi,z;0)xi+τ(θyi,z)
)1{yi=l} − ∫
1z′=arg maxg∈J ( h>(y,z;0)x+τ(θy,z) )1{y=l}dQ(x, y) ∣∣∣∣→ 0
where the last inequality is due to the results with uniform laws of large numbers from equation 42 and the fact that ‖h(l, z′; 0)‖2
2 − log πl is bounded for all l and z′ ∈ L. Hence, we obtain the conclusion
of equation 36.
Proof of equation 37: For this claim, we have the following inequality
E(2)n ≤ sup {S′y},{πy} ∣∣∣∣ 1n− n1 n∑ i=n1+1 log exp
( max s∈S′yi s>[xi, 1] + log πyi ) K∑̀ =1 exp ( max s∈S′l s>[xi, 1] + log πl )
− ∫ log exp ( max s∈S′y s>[x, 1] + log πy ) K∑̀ =1 exp ( max s∈S′l s>[x, 1] + log πl )dQ(x, y)∣∣∣∣
≤ K∑ `=1 sup {S′y},{πy} ∣∣∣∣ 1n− n1 n∑ i=n1+1 log exp
( max s∈S′yi s>[xi, 1] + log πyi ) K∑̀ =1 exp ( max s∈S′l s>[xi, 1] + log πl )1{yi=l}
− ∫ log exp ( max s∈S′y s>[x, 1] + log πy ) K∑̀ =1 exp ( max s∈S′l s>[x, 1] + log πl )1{c=l}dQ(x, y)∣∣∣∣ = K∑ `=1 Fn,l
where { S′y } in the above supremum stands for the collection of sets S′1, . . . , S ′ K such that |S′y| ≤ |L| and elements in S′y are in RD (0)+1. Therefore, to achieve the conclusion of equation 37, it is sufficient to demonstrate that Fn,l → 0 almost surely as n→∞ for each 1 ≤ l ≤ K.
In fact, for each 1 ≤ l ≤ K, we denote G to be the family of function on RD(0) × {1, . . . ,K} of the form
f{S′y},{πy}(x, y) = log exp
( max s∈S′y s>[x, 1] + log πy ) K∑̀ =1 exp ( max s∈S′l s>[x, 1] + log πl )1{y=l}
for all (x, y) where S′1, . . . , S ′ K ∈ O|L|, which contains all sets that have at most |L| elements in
RD(0)+1, and {πy} satisfy that K∑ y=1 πy = 1 and πy ≥ γ for all 1 ≤ y ≤ K. Due to the assumption with distribution P , we can restrict O|L| to contain only set S′ with elements in the ball B(R) of radius R on RD(0)+1. By means of Lemma 2.5 in (van de Geer, 2000), we can find a finite set Eδ such that each element in B(R) is within distance δ to some element of Eδ for all δ > 0. Additionally, there exists a set ∆(δ) such that for each (π1, . . . , πK), we can find a corresponding element (π′1, . . . , π ′ K) ∈ ∆(δ) such that ‖(π1, . . . , πK)− (π′1, . . . , π′K)‖ ≤ δ. We denote
F(δ) = {{ S′y } , {πy} : elements of S′y in Eδ, and(π1, . . . , πK) ∈ ∆(δ) }
for all δ > 0. For each element { S′y }K y=1
, {πy}Ky=1, we can choose the corresponding element{ S ′ y }K y=1 , {πy}Ky=1 ∈ F(δ) such that S′y = { sy1, . . . , syky } , S ′ y = { s′y1, . . . , s ′ yky } satisfy ‖syj − s′yj‖ ≤ δ for all 1 ≤ y ≤ K and 1 ≤ j ≤ ky. Additionally,
‖(π1, . . . , πK)− (π1, . . . , πK)‖ ≤ δ. With these notations, we define
f{S′y},{πy}(x, y) = log ( exp(max s∈S′y s>[x, 1] + log πy)
K∑ τ=1 exp(max s∈S′τ s>[x, 1] + log πτ )
) 1{y=l} + 2δ‖[x, 1]‖+ 2δ/γ,
f{S′y},{πy}(x, y) = log ( exp(max s∈S′y s>[x, 1] + log πy)
K∑ τ=1 exp(max s∈S′τ s>[x, 1] + log πτ )
) 1{y=l} − 2δ‖[x, 1]‖ − 2δ/γ
for any ({ S ′ y } , {πy} ) ∈ F(δ). By means of Cauchy-Schwarz’s inequality, we have
s>yi[x, 1]1{y=l} − (s′yi)>[x, 1]1{y=l} ≤ ‖sci − s′yi‖‖[x, 1]‖1{y=l} ≤ δ‖[x, 1]‖
for all x and 1 ≤ i ≤ k. Additionally, the following also holds s>yi[x, 1]1{y=l} − (s′yi)>[x, 1]1{y=l} ≥ −δ‖[x, 1]‖.
Furthermore, | log πy − log πy| ≤ log(1 + δ/γ) ≤ δ/γ. Hence, we obtain that
exp(max s∈S′y
s>[x, 1] + log πy)
K∑ τ=1 exp(max s∈S′τ s>[x, 1] + log πτ )
≤ exp(max
s∈S′y s>[x, 1] + log πy + δ‖[x, 1]‖+ δ/γ)
K∑ τ=1 exp(max s∈S′τ s>[x, 1] + log πτ − δ‖[x, 1]‖ − δ/γ)
≤ exp(max
s∈S′y s>[x, 1] + log πy)
K∑ τ=1 exp(max s∈S′τ s>[x, 1] + log πτ ) exp(2δ‖[x, 1]‖+ 2δ/γ).
Similarly, we also have
exp(max s∈S′y
s>[x, 1] + log πy)
K∑ τ=1 exp(max s∈S′τ s>[x, 1] + log πτ )
≥ exp(max
s∈S′y s>[x, 1] + log πy)
K∑ τ=1 exp(max s∈S′τ s>[x, 1] + log πτ ) exp(−2δ‖[x, 1]‖ − 2δ/γ).
As a consequence, we achieve that
f{S′y},{πy}(x, y) ≤ f{S′y},{πy}(x, y) ≤ f{S′y},{πy}(x, y)
for all (x, y). With the formulations of f{S′y},{πy}(x, c) and f{S′y},{πy}(x, y), we have∫ ( f{S′y},{πy}(x, y)− f{S′y},{πy}(x, y) ) dQ(x, y)
= 4δ ∫ ‖[x, 1]‖dQ(x, y) + 4δ/γ ≤ 4δ (∫ ‖x‖dQ(x, y) + 1 ) + 4δ/γ.
For any > 0, by choosing δ < 4 ∫ ‖x‖dQ(x, y) + 4 + 4/γ then we will have∫
(f{S′y},{πy}(x, y) − f{S′y},{πy}(x, y))dQ(x, y) < . It implies that the -bracketing entropy of G is finite for the L1 norm with distribution Q. Therefore, it implies that G satisfies the uniform law of large numbers, i.e., Fn,l → 0 almost surely as n→∞ for all 1 ≤ l ≤ K. As a consequence, the uniform law of large number result equation 37 holds.
Going back to the original problem, denote θ̃0 = ({ µ̃0(y) }K y=1 , { Γ̃0(`) }L `=1 , { π̃0y }K y=1 , { b̃0(`) }L `=1 ) the optimal solutions of population partially labeled LDCE (Note that, the existence of these optimal
solutions is guaranteed due to the compact assumptions with the parameter spaces Θ`, Ω, and Ξ` for all 1 ≤ ` ≤ L). Then, according to the formulation of partially labeled LDCE, we will have that
Yn ≤ min θ αRC n { n1∑ i=1 ∑ (y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( x>i h̃ 0(y,z;0)+τ(θ̃0y,z) ) (‖xi − h̃0(y′, z′; 0)‖2 2
− log(p̃0(y′, z′)) ) + n∑ i=n1+1 ∑ z′∈L 1z′=arg maxz∈L ( x>i h̃ 0(y,z;0)+τ(θ̃0y,z) ) (‖xi − h̃0(yi, z′; 0)‖2 2
− log(p̃0(yi, z′)) )} − αCE n− n1 n∑ i=n1+1 log qθ̃0(yi|xi) + αKL n n∑ i=1 K∑ y=1 qθ̃0(y|xi) log ( qθ̃0(y|xi) πy ) = Dn
for all n ≥ 1 where we have the following formulations
h̃0(y, z; 0) = Λ̃0(z; 1) . . . Λ̃0(z;L)µ̃ 0(y), Λ̃0(z; `) = ∑
p∈P(`) s(`, p)T (t; `, p)B(`, p)Γ̃0(`, p),
p̃0(y′, z′) = exp ( τ(θ̃0y′,z′) + log π̃ 0 y′ ) / ( K∑ y=1 exp ( max z∈L τ(θ̃0y,z) + log π̃ 0 y )) .
From the results with uniform laws of large numbers in equation 33, equation 34, equation 35, equation 36, equation 37, and equation 38, we obtain that
1
n1 n1∑ i=1 ∑ (y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( x>i h̃ 0(y,z;0)+τ(θ̃0y,z) ) (‖xi − h̃0(y′, z′; 0)‖2 2 − log(p̃0(y′, z′)) )
→ ∫ ∑
(y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( x>h̃0(y,z;0)+τ(θ̃0y,z) ) (‖x− h̃0(y′, z′; 0)‖2 2 − log(p̃0(y′, z′)) ) dP (x),
1
n− n1 n∑ i=n1+1 ∑ z′∈L 1z′=arg maxz∈L ( x>i h̃ 0(y,z;0)+τ(θ̃0y,z) ) (‖xi − h̃0(yi, z′; 0)‖2 2 − log(p̃0(yi, z′)) )
→ ∫ ∑
z′∈L 1z′=arg maxz∈L ( x>h̃0(y,z;0)+τ(θ̃0y,z) ) (‖x− h̃0(y, z′; 0)‖2 2 − log(p̃0(y, z′)) ) dQ(x, y),
1
n− n1 n∑ i=n1+1 log qθ̃0(yi|xi)→ ∫ log qθ̃0(y|x)dQ(x, y),
1
n n∑ i=1 K∑ y=1 qθ̃0(y|xi) log ( qθ̃0(y|xi) πy ) → ∫ K∑ y=1 qθ̃0(y|x) log ( qθ̃0(y|x) πy ) dP (x)
almost surely as n→∞. Combining with the fact that n1/n→ λ, the above results lead to Dn → Y almost surely as n→∞. Therefore, we have lim
n→∞ Yn ≤ Y almost surely as n→∞. On the other
hand, with the results of uniform laws of large numbers in equation 33, equation 34, equation 35,
equation 36, equation 37, and equation 38, we also have that
1
n1 n1∑ i=1 ∑ (y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( x>i h̃(y,z;0)+τ(θ̃y,z) ) (‖xi − h̃(y′, z′; 0)‖2 2 − log(p̃(y′, z′)) )
→ ∫ ∑
(y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( x>h̃(y,z;0)+τ(θ̃y,z) ) (‖x− h̃(y′, z′; 0)‖2 2 − log(p̃(y′, z′)) ) dP (x),
1
n− n1 n∑ i=n1+1 ∑ z′∈L 1z′=arg maxz∈L ( x>i h̃(y,z;0)+τ(θ̃y,z) ) (‖xi − h̃(yi, z′; 0)‖2 2 − log(p̃(yi, z′)) )
→ ∫ ∑
z′∈L 1z′=arg maxz∈L ( x>h̃0(y,z;0)+τ(θ̃y,z) ) (‖x− h̃(y, z′; 0)‖2 2 − log(p̃(y, z′)) ) dQ(x, y),
1
n− n1 n∑ i=n1+1 log qθ̃(yi|xi)→ ∫ log qθ̃(y|x)dQ(x, y),
1
n n∑ i=1 K∑ y=1 qθ̃(y|xi) log ( qθ̃(y|xi) πy ) → ∫ K∑ y=1 qθ̃(y|x) log ( qθ̃(y|x) πy ) dP (x)
almost surely as n→∞where θ̃ = ( {µ̃(y)}Ky=1 , { Γ̃(`) }L `=1 , {π̃y}Ky=1 , { b̃(`) }L `=1 ) is the optimal solution of partially labeled LDCE and
h̃(y, z; 0) = Λ̃(z; 1) . . . Λ̃(z;L)µ̃(y), Λ̃(z; `) = ∑
p∈P(`) s(`, p)T (t; `, p)B(`, p)Γ̃(`, p),
p̃(y′, z′) = exp ( τ(θ̃y′,z′) + log π̃y′ ) / ( K∑ y=1 exp ( max z∈L τ(θ̃y′,z′) + log π̃y )) .
Hence, we eventually achieve that Yn → αRC { λ (∫ ∑ (y′,z′)∈J 1(y′,z′)=arg max(y,z)∈J ( x>h̃(y,z;0)+τ(θ̃y,z) ) (‖x− h̃(y′, z′; 0)‖2 2
− log(p̃(y′, z′)) ) dP (x) ) + (1− λ) (∫ ∑ z′∈L 1z′=arg maxz∈L ( x>h̃0(y,z;0)+τ(θ̃y,z) ) (‖x− h̃(y, z′; 0)‖2 2
− log(p̃(y, z′)) ) dQ(x, y) )} − αCE ∫ log qθ̃(y|x)dQ(x, y)
+αKL ∫ K∑ y=1 qθ̃(y|x) log ( qθ̃(y|x) πy ) dP (x) ≥ Y
almost surely as n → ∞. As a consequence, we achieve Yn → Y almost surely as n → ∞. We reach the conclusion of the theorem.
PROOF OF THEOREM .7 The proof argument of this theorem is a direct application of the results with uniform laws of large numbers in the proof of Theorem .6. In fact, we define
F̃0( ) = { (S, {πy} , {b(`)}) : S = { h(y, z; 0) : (y, z) ∈ J } ,
and inf (S̃0,{π̃0y},{b̃0(`)})∈G(F̃0)
{ H(S, S̃0) + K∑ y=1 |πy − π̃0y|+ ∑ z∈L ‖b(`)− b̃0(`)‖ } ≥ }
for any > 0. Since the parameter spaces of θ are compact sets, the set F̃0( ) is also a compact set for all > 0. Denote
g (S, {πy} , {b(`)})
= αRC
{ λ (∫ ∑ (y′,z′)∈J 1(y′,z′)= ( h>(y,z;0)x+τ(θy,z) ) (‖x− h(y′, z′; 0)‖2 2 − log(πy′,z′) ) dP (x) )
+(1− λ) (∫ ∑
z′∈L 1z′=arg maxz∈L ( h>(y,z;0)x+τ(θy,z) ) (‖x− h(y, z′; 0)‖2 2 − log(πy,z′) ) dQ(x, y) }
−αCE ∫ log qθ(y|x)dQ(x, y) + αKL ∫ K∑
y=1
qθ(y|x)dQ(x, c) log ( qθ(y|x) πy ) dP (x)
for all (S, {πy} , {b(`)}). From the definition of F̃0( ), we have that g (S, {πy} , {b(`)}) > g ( S̃0, { π̃0y } , { b̃0(`) }) for all (S, {πy} , {b(`)}) ∈ F̃0( ) and ( S̃0, { π̃0y } , { b̃0(`) }) ∈ G(F̃0). As F̃0( ) is a compact set,
we further have that
inf (S,{πy},{b(`)})∈F̃0( )
g (S, {πy} , {b(`)}) > g ( S̃0, { π̃0y } , { b̃0(`) }) for all ( S̃0, { π̃0y } , { b̃0(`) }) ∈ G(F̃0) and > 0.
Now, according to the uniform laws of large numbers established in the proof of Theorem .6, we have that Yn → g ( S̃n, {π̃y} , { b̃(`) }) almost surely as n → ∞. According to the result of
Theorem .6, it implies that g ( S̃n, {π̃y} , { b̃(`) }) → g ( S̃0, { π̃0y } , { b̃0(`) }) almost surely for all(
S̃0, { π̃0y } , { b̃0(`) }) ∈ G(F̃0). Therefore, for each > 0 we can find sufficiently large N such that
we have
inf (S̃0,{π̃0y},{b̃0(`)})∈G(F̃0)
{ H(S̃n, S̃0) + K∑ y=1 |π̃y − π̃0y|+ ∑ z∈L ‖b̃(`)− b̃0(`)‖ } <
almost surely for all n ≥ N . As a consequence, we achieve the conclusion of the theorem.
PROOF OF THEOREM .8 The proof of the theorem is an application of Theorem 11 for generalization bound with margin from Koltchinskii & Panchenko (2002) based on an evaluation of Rademacher complexity. In particular, we denote
Jn = { hτn(x, y) : RD (0) × {1, . . . ,K} → R|
hτn(x, y) = max z∈L(τn)
{ h>(y, z; 0)x+ τ(θy,z) } + log πy ∀ (x, y) for some |L(τn)| ≤ τn|L| } .
Now, we denote J̃n = {hτn(., y) : y ∈ {1, . . . ,K} , hτn ∈ Jn}. For any δ > 0, using the same argument as that of the proof of Theorem 11 in (Koltchinskii & Panchenko, 2002), with probability at least 1− δ, we have
R0(fτn) ≤ inf Γ∈(0,1]
{ Rn,Γ(fτn) +
8K(2K − 1) Γ <n(J̃n)
+
( log log2(2Γ
−1) n
)1/2 + √ log(2δ−1)
2n
} (43)
where <n(J̃n) is Rademacher complexity of J̃n, which in our case is defined as
<n(J̃n) = E sup θ sup |L(τn)|≤τn|L| ∣∣∣∣ 1n n∑ i=1 σi ( max g∈L(τn) { h>(y, z; 0)xi + τ(θy,z) } + log πy )∣∣∣∣ where σ1, . . . , σn are i.i.d. Rademacher random variables. Since γ is the lower bound of πy for all 1 ≤ y ≤ K, we obtain that
<n(J̃ ) ≤ E sup θ sup |L(τn)|≤τn|L| ∣∣∣∣ 1n n∑ i=1 σi ( max g∈L(τn) { h>(y, z; 0)xi + τ(θy,z) })∣∣∣∣ + E sup
θ sup
|L(τn)|≤τn|L| ∣∣∣∣ 1n n∑ i=1 σi log πy ∣∣∣∣ ≤ E sup
θ sup
|L(τn)|≤τn|L| ∣∣∣∣ 1n n∑ i=1 σi ( max g∈L(τn) { h>(y, z; 0)xi + τ(θy,z) })∣∣∣∣+ | log γ|√n Furthermore, we have the following inequalities
E sup θ sup |L(τn)|≤τn|L| ∣∣∣∣ 1n n∑ i=1 σi ( max g∈L(τn) { h>(y, z; 0)xi + τ(θy,z) })∣∣∣∣ ≤ E sup
|S′|≤cn|L| ∣∣∣∣ 1n n∑ i=1 σi max s∈S′ s>[xi, 1] ∣∣∣∣ ≤ 2cn|L|E sup
s∈B(R) ∣∣∣∣ 1n n∑ i=1 σis >[xi, 1] ∣∣∣∣ ≤ 2cn|L|RE
∥∥∥∥ 1n n∑ i=1 σi[xi, 1] ∥∥∥∥ ≤ 2cn|L|(R
2 + 1)√ n
where the final inequality is due to Cauchy-Schwartz’s inequality. Combining the above results with equation 43, we achieve the conclusion of the theorem.

heading: Appendix D
text: In this appendix, we provide detail descriptions for several simulation studies in the main text.
.16 ARCHITECTURE OF THE NETWORK USED IN OUR SEMI-SUPERVISED EXPERIMENTS
.17 TRAINING DETAILS
.17.1 SEMI-SUPERVISED LEARNING EXPERIMENTS ON CIFAR10, CIFAR100, AND SVHN
The training losses are discussed in Section 2.3. In addition to the cross-entropy loss, the recontruction loss, and the RPN regularization, in order to further improve the performance of NRM, we introduce two new losses for training the model. Those two new losses are from our derivation of batch normalization using the NRM framework and from applying variational inference on the NRM. More details on these new training losses can be found in Appendix A.4. We compare NRM with state-ofthe-art methods on semi-supervised object classification tasks which use consistency regularization, such as the Π model (Laine & Aila, 2017), the Temporal Ensembling (Laine & Aila, 2017), the Mean Teacher (Tarvainen & Valpola, 2017), the Virtual Adversarial Training (VAT), and the Ladder Network (Rasmus et al., 2015). We also compare NRM with methods that do not use consistency regularization including the improved GAN (Salimans et al., 2016) and the Adversarially Learned Inference (ALI) (Dumoulin et al., 2017).
All networks were trained using Adam with learning rate of 0.001 for the first 20 epochs. Adam momentum parameters were set to beta1 = 0.9 and beta2 = 0.999. Then we used SGD with decayed learning rate to train the networks for another 380 epochs. The starting learning rate for SGD is 0.15 and the end learning rate at epoch 400 is 0.0001. We use batch size 128. Let the weights for the cross-entropy loss, the reconsruction loss, the KL divergence loss, the moment matching loss, and the RPN regularization be αCE , αRC , αKL, αMM , and αPN , respectively. In our training, αCE = 1.0, αRC = 0.5, αKL = 0.5, αMM = 0.5, and αPN = 1.0. For Max-Min cross-entropy, we use αmax = αmin = 0.5.
.17.2 SUPERVISED LEARNING EXPERIMENTS WITH MAX-MIN CROSS ENTROPY
Training on CIFAR10 We use the 26 2 x 96d ”Shake-Shake-Image” ResNet in Gastaldi (2017) with the Cutout data augmentation suggested in DeVries & Taylor (2017) as our baseline. We implement the Max-Min cross-entropy on top of this baseline and turn it into a Max-Min network. In addition to Cutout data augmentation, standard translation and flipping data augmentation is applied on the 32 x 32 x 3 input image. Training procedure are the same as in (Gastaldi, 2017). In particular, the models were trained for 1800 epochs. The learning rate is initialized at 0.2 and is annealed using a Cosine
function without restart (see (Loshchilov & Hutter, 2016)). We train our models on 1 GPU with a mini-batch size of 128.
Training on CIFAR10 We use the Squeeze-and-Excitation ResNeXt-50 as in Hu et al. (2018) as our baseline. As with CIFAR10, we implement the Max-Min cross-entropy for the baseline and turn it into a Max-Min network. During training, we follow standard practice and perform data augmentation with random-size cropping (Szegedy et al., 2015) to 224 x 224 x 3 pixels. We train the network with the Nesterov accelerated SGD for 125 epochs. The intial learning rate is 0.1 with momentum 0.9. We divide the learning rate by 10 at epoch 30, 60, 90, 95, 110, 120. Our network is trained on 8 GPUs with batch size of 32.
.17.3 SEMI-SUPERVISED TRAINING ON MNIST WITH 50K LABELED TO GET THE TRAINED MODEL FOR GENERATING RECONSTRUCTED IMAGE IN FIGURE 2
The architecture of the baseline CNN we use is given in the Table 7. We use the same training procedure as in Section .17.1

Review summaries:
This paper introduces the Neural Rendering Model (NRM), a generative model in which the computations involved in inference correspond to those of a CNN forward pass. The NRM’s supervised learning objective is lower bounded by a variant of the cross-entropy objective. This objective is used to formulate a max-min network, which has a particular type of weight sharing between a standard branch with max pooling / ReLUs and a second branch with min pooling / NReLUs. The max-min objective and network show strong performance on semi-supervised learning tasks. 

The paper claims to propose a novel generative probabilistic neural network model such that its encoder (classifying an image) can be approximated by a convolutional neural network with ReLU activations and MaxPooling layers. Besides the standard parameters of the units (weights and biases), the model has two additional latent variables per unit, which decide whether and where to put the template (represented by the weights of the neuron) in the subsequent layer, when generating an image from the class. Furthermore, the authors claim to derive new learning criteria for semi-supervised learning of the model including a novel regulariser and claim to prove its consistency.