Title: NEURAL RENDERING MODEL: JOINT GENERATION AND PREDICTION FOR SEMI-SUPERVISED LEARNING

Abstract: Unsupervised and semi-supervised learning are important problems that are especially challenging with complex data like natural images. Progress on these problems would accelerate if we had access to appropriate generative models under which to pose the associated inference tasks. Inspired by the success of Convolutional Neural Networks (CNNs) for supervised prediction in images, we design the Neural Rendering Model (NRM), a new hierarchical probabilistic generative model whose inference calculations correspond to those in a CNN. The NRM introduces a small set of latent variables at each level of the model and enforces dependencies among all the latent variables via a conjugate prior distribution. The conjugate prior yields a new regularizer for learning based on the paths rendered in the generative model for training CNNs-the Rendering Path Normalization (RPN). We demonstrate that this regularizer improves generalization both in theory and in practice. Likelihood estimation in the NRM yields the new Max-Min cross entropy training loss, which suggests a new deep network architecture-the Max-Min network-which exceeds or matches the state-of-art for semi-supervised and supervised learning on SVHN, CIFAR10, and CIFAR100.

1. INTRODUCTION:

Unsupervised and semi-supervised learning have still lagged behind compared to performance leaps we have seen in supervised learning over the last five years. This is partly due to a lack of good generative models that can capture all latent variations in complex domains such as natural images and provide useful structures that help learning. When it comes to probabilistic generative models, it is hard to design good priors for the latent variables that drive the generation.

Instead, recent approaches avoid the explicit design of image priors. For instance, the Generative Adversarial Networks (GANs) use implicit feedback from an additional discriminator that distinguishes real from fake images (Goodfellow et al., 2014) . Using such feedback helps GANs to generate visually realistic images, but it is not clear if this is the most effective form of feedback for predictive tasks. Moreover, due to separation of generation and discrimination in GANs, there are typically more parameters to train, and this might make it harder to obtain gains for semi-supervised learning in the low (labeled) sample setting.

We propose an alternative approach to GANs by designing a class of probabilistic generative models, such that inference in those models also has good performance on predictive tasks. This approach is well-suited for semi-supervised learning since it eliminates the need for a separate prediction network. Specifically, we answer the following question: what generative processes output Convolutional Neural Networks (CNNs) when inference is carried out? This is natural to ask since CNNs are stateof-the-art (SOTA) predictive models for images, and intuitively, such powerful predictive models should capture some essence of image generation. However, standard CNNs are not directly reversible and likely do not have all the information for generation since they are trained for predictive tasks such as image classification. We can instead invert the irreversible operations in CNNs, e.g., the rectified linear units (ReLUs) and spatial pooling, by assigning auxiliary latent variables to account for uncertainty in the CNN's inversion process due to the information loss.

Contribution 1 -Neural Rendering Model: We develop the Neural Rendering Model (NRM) whose bottom-up inference corresponds to a CNN architecture of choice (see Figure 1a ). The "reverse" x < l a t e x i t s h a 1 _ b a s e 6 4 = " T Q G E y g Q L k 4 M p T O N X Y f p V x G V f R f 8 = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e C F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b s Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 7 d z v P K L S P J b 3 Z p q g H 9 G R 5 C F n 1 F i p + T Q o V 9 y q u w B Z J 1 5 O K p C j M S h / 9 Y c x S y O U h g m q d c 9 z E + N n V B n O B M 5 K / V R j Q t m E j r B n q a Q R a j 9 b H D o j F 1 Y Z k j B W t q Q h C / X 3 R E Y j r a d R Y D s j a s Z 6 1 Z u L / 3 m 9 1 I Q 3 f s Z l k h q U b L k o T A U x M Z l / T Y Z c I T N i a g l l i t t b C R t T R Z m x 2 Z R s C N 7 q y + u k f V X 1 3 K r X v K 7 U u 3 k c R T i D c 7 g E D 2 p Q h z t o Q A s Y I D z D K 7 w 5 D 8 6 L 8 + 5 8 L F s L T j 5 z C n / g f P 4 A 7 N e N E g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T Q G E y g Q L k 4 M p T O N X Y f p V x G V f R f 8 = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e C F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b s Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 7 d z v P K L S P J b 3 Z p q g H 9 G R 5 C F n 1 F i p + T Q o V 9 y q u w B Z J 1 5 O K p C j M S h / 9 Y c x S y O U h g m q d c 9 z E + N n V B n O B M 5 K / V R j Q t m E j r B n q a Q R a j 9 b H D o j F 1 Y Z k j B W t q Q h C / X 3 R E Y j r a d R Y D s j a s Z 6 1 Z u L / 3 m 9 1 I Q 3 f s Z l k h q U b L k o T A U x M Z l / T Y Z c I T N i a g l l i t t b C R t T R Z m x 2 Z R s C N 7 q y + u k f V X 1 3 K r X v K 7 U u 3 k c R T i D c 7 g E D 2 p Q h z t o Q A s Y I D z D K 7 w 5 D 8 6 L 8 + 5 8 L F s L T j 5 z C n / g f P 4 A 7 N e N E g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T Q G E y g Q L k 4 M p T O N X Y f p V x G V f R f 8 = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e C F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b s Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 7 d z v P K L S P J b 3 Z p q g H 9 G R 5 C F n 1 F i p + T Q o V 9 y q u w B Z J 1 5 O K p C j M S h / 9 Y c x S y O U h g m q d c 9 z E + N n V B n O B M 5 K / V R j Q t m E j r B n q a Q R a j 9 b H D o j F 1 Y Z k j B W t q Q h C / X 3 R E Y j r a d R Y D s j a s Z 6 1 Z u L / 3 m 9 1 I Q 3 f s Z l k h q U b L k o T A U x M Z l / T Y Z c I T N i a g l l i t t b C R t T R Z m x 2 Z R s C N 7 q y + u k f V X 1 3 K r X v K 7 U u 3 k c R T i D c 7 g E D 2 p Q h z t o Q A s Y I D z D K 7 w 5 D 8 6 L 8 + 5 8 L F s L T j 5 z C n / g f P 4 A 7 N e N E g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T Q G E y g Q L k 4 M p T O N X Y f p V x G V f R f 8 = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e C F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b s Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 7 d z v P K L S P J b 3 Z p q g H 9 G R 5 C F n 1 F i p + T Q o V 9 y q u w B Z J 1 5 O K p C j M S h / 9 Y c x S y O U h g m q d c 9 z E + N n V B n O B M 5 K / V R j Q t m E j r B n q a Q R a j 9 b H D o j F 1 Y Z k j B W t q Q h C / X 3 R E Y j r a d R Y D s j a s Z 6 1 Z u L / 3 m 9 1 I Q 3 f s Z l k h q U b L k o T A U x M Z l / T Y Z c I T N i a g l l i t t b C R t T R Z m x 2 Z R s C N 7 q y + u k f V X 1 3 K r X v K 7 U u 3 k c R T i D c 7 g E D 2 p Q h z t o Q A s Y I D z D K 7 w 5 D 8 6 L 8 + 5 8 L F s L T j 5 z C n / g f P 4 A 7 N e N E g = = < / l a t e x i t > y < l a t e x i t s h a 1 _ b a s e 6 4 = " z X / n e h u C + f K 5 + A T 4 o 3 l 1 J M U r C Q Q = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P B i 8 c W 7 A e 0 o W y 2 k 3 b t Z h N 2 N 0 I I / Q V e P C j i 1 Z / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 6 W N z a 3 t n f J u Z W / / 4 P C o e n z S 0 X G q G L Z Z L G L V C 6 h G w S W 2 D T c C e 4 l C G g U C u 8 H 0 b u 5 3 n 1 B p H s s H k y X o R 3 Q s e c g Z N V Z q Z c N q z a 2 7 C 5 B 1 4 h W k B g W a w + r X Y B S z N E J p m K B a 9 z 0 3 M X 5 O l e F M 4 K w y S D U m l E 3 p G P u W S h q h 9 v P F o T N y Y Z U R C W N l S x q y U H 9 P 5 D T S O o s C 2 x l R M 9 G r 3 l z 8 z + u n J r z 1 c y 6 T 1 K B k y 0 V h K o i J y f x r M u I K m R G Z J Z Q p b m 8 l b E I V Z c Z m U 7 E h e K s v r 5 P O V d 1 z 6 1 7 r u t b o F X G U 4 Q z O 4 R I 8 u I E G 3 E M T 2 s A A 4 R l e 4 c 1 5 d F 6 c d + d j 2 V p y i p l T + A P n 8 w f u W 4 0 T < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " z X / n e h u C + f K 5 + A T 4 o 3 l 1 J M U r C Q Q = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P B i 8 c W 7 A e 0 o W y 2 k 3 b t Z h N 2 N 0 I I / Q V e P C j i 1 Z / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 6 W N z a 3 t n f J u Z W / / 4 P C o e n z S 0 X G q G L Z Z L G L V C 6 h G w S W 2 D T c C e 4 l C G g U C u 8 H 0 b u 5 3 n 1 B p H s s H k y X o R 3 Q s e c g Z N V Z q Z c N q z a 2 7 C 5 B 1 4 h W k B g W a w + r X Y B S z N E J p m K B a 9 z 0 3 M X 5 O l e F M 4 K w y S D U m l E 3 p G P u W S h q h 9 v P F o T N y Y Z U R C W N l S x q y U H 9 P 5 D T S O o s C 2 x l R M 9 G r 3 l z 8 z + u n J r z 1 c y 6 T 1 K B k y 0 V h K o i J y f x r M u I K m R G Z J Z Q p b m 8 l b E I V Z c Z m U 7 E h e K s v r 5 P O V d 1 z 6 1 7 r u t b o F X G U 4 Q z O 4 R I 8 u I E G 3 E M T 2 s A A 4 R l e 4 c 1 5 d F 6 c d + d j 2 V p y i p l T + A P n 8 w f u W 4 0 T < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " z X / n e h u C + f K 5 + A T 4 o 3 l 1 J M U r C Q Q = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P B i 8 c W 7 A e 0 o W y 2 k 3 b t Z h N 2 N 0 I I / Q V e P C j i 1 Z / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 6 W N z a 3 t n f J u Z W / / 4 P C o e n z S 0 X G q G L Z Z L G L V C 6 h G w S W 2 D T c C e 4 l C G g U C u 8 H 0 b u 5 3 n 1 B p H s s H k y X o R 3 Q s e c g Z N V Z q Z c N q z a 2 7 C 5 B 1 4 h W k B g W a w + r X Y B S z N E J p m K B a 9 z 0 3 M X 5 O l e F M 4 K w y S D U m l E 3 p G P u W S h q h 9 v P F o T N y Y Z U R C W N l S x q y U H 9 P 5 D T S O o s C 2 x l R M 9 G r 3 l z 8 z + u n J r z 1 c y 6 T 1 K B k y 0 V h K o i J y f x r M u I K m R G Z J Z Q p b m 8 l b E I V Z c Z m U 7 E h e K s v r 5 P O V d 1 z 6 1 7 r u t b o F X G U 4 Q z O 4 R I 8 u I E G 3 E M T 2 s A A 4 R l e 4 c 1 5 d F 6 c d + d j 2 V p y i p l T + A P n 8 w f u W 4 0 T < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " z X / n e h u C + f K 5 + A T 4 o 3 l 1 J M U r C Q Q = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P B i 8 c W 7 A e 0 o W y 2 k 3 b t Z h N 2 N 0 I I / Q V e P C j i 1 Z / k z X / j t s 1 B W x 8 M P N 6 b Y W Z e k A i u j e t + O 6 W N z a 3 t n f J u Z W / / 4 P C o e n z S 0 X G q G L Z Z L G L V C 6 h G w S W 2 D T c C e 4 l C G g U C u 8 H 0 b u 5 3 n 1 B p H s s H k y X o R 3 Q s e c g Z N V Z q Z c N q z a 2 7 C 5 B 1 4 h W k B g W a w + r X Y B S z N E J p m K B a 9 z 0 3 M X 5 O l e F M 4 K w y S D U m l E 3 p G P u W S h q h 9 v P F o T N y Y Z U R C W N l S x q y U H 9 P 5 D T S O o s C 2 x l R M 9 G r 3 l z 8 z + u n J r z 1 c y 6 T 1 K B k y 0 V h K o i J y f x r M u I K m R G Z J Z Q p b m 8 l b E I V Z c Z m U 7 E h e K s v r 5 P O V d 1 z 6 1 7 r u t b o F X G U 4 Q z O 4 R I 8 u I E G 3 E M T 2 s A A 4 R l e 4 c 1 5 d F 6 c d + d j 2 V p y i p l T + A P n 8 w f u W 4 0 T < / l a t e x i t > Noise < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 3 p u J A q h D 1 0 g 3 Y H 6 c y 9 a c I T U D N U = " > A A A B 8 3 i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 B I v g q S Q i 6 L H o x Z N U s L X Q h L L Z T t q l m 9 2 w O x F L 6 N / w 4 k E R r / 4 Z b / 4 b t 2 0 O 2 v p g 4 P H e D D P z o l R w g 5 7 3 7 Z R W V t f W N 8 q b l a 3 t n d 2 9 6 v 5 B 2 6 h M M 2 g x J Z T u R N S A 4 B J a y F F A J 9 V A k 0 j A Q z S 6 n v o P j 6 A N V / I e x y m E C R 1 I H n N G 0 U p B g P C E + a 3 i B i a 9 a s 2 r e z O 4 y 8 Q v S I 0 U a P a q X 0 F f s S w B i U x Q Y 7 q + l 2 K Y U 4 2 c C Z h U g s x A S t m I D q B r q a Q J m D C f 3 T x x T 6 z S d 2 O l b U l 0 Z + r v i Z w m x o y T y H Y m F I d m 0 Z u K / 3 n d D O P L M O c y z R A k m y + K M + G i c q c B u H 2 u g a E Y W 0 K Z 5 v Z W l w 2 p p g x t T B U b g r / 4 8 j J p n 9 V 9 r + 7 f n d c a V 0 U c Z X J E j s k p 8 c k F a Z A b 0 i Q t w k h K n s k r e X M y 5 8 V 5 d z 7 m r S W n m D k k f + B 8 / g C Z l Z I J < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 3 p u J A q h D 1 0 g 3 Y H 6 c y 9 a c I T U D N U = " > A A A B 8 3 i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 B I v g q S Q i 6 L H o x Z N U s L X Q h L L Z T t q l m 9 2 w O x F L 6 N / w 4 k E R r / 4 Z b / 4 b t 2 0 O 2 v p g 4 P H e D D P z o l R w g 5 7 3 7 Z R W V t f W N 8 q b l a 3 t n d 2 9 6 v 5 B 2 6 h M M 2 g x J Z T u R N S A 4 B J a y F F A J 9 V A k 0 j A Q z S 6 n v o P j 6 A N V / I e x y m E C R 1 I H n N G 0 U p B g P C E + a 3 i B i a 9 a s 2 r e z O 4 y 8 Q v S I 0 U a P a q X 0 F f s S w B i U x Q Y 7 q + l 2 K Y U 4 2 c C Z h U g s x A S t m I D q B r q a Q J m D C f 3 T x x T 6 z S d 2 O l b U l 0 Z + r v i Z w m x o y T y H Y m F I d m 0 Z u K / 3 n d D O P L M O c y z R A k m y + K M + G i c q c B u H 2 u g a E Y W 0 K Z 5 v Z W l w 2 p p g x t T B U b g r / 4 8 j J p n 9 V 9 r + 7 f n d c a V 0 U c Z X J E j s k p 8 c k F a Z A b 0 i Q t w k h K n s k r e X M y 5 8 V 5 d z 7 m r S W n m D k k f + B 8 / g C Z l Z I J < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 3 p u J A q h D 1 0 g 3 Y H 6 c y 9 a c I T U D N U = " > A A A B 8 3 i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 B I v g q S Q i 6 L H o x Z N U s L X Q h L L Z T t q l m 9 2 w O x F L 6 N / w 4 k E R r / 4 Z b / 4 b t 2 0 O 2 v p g 4 P H e D D P z o l R w g 5 7 3 7 Z R W V t f W N 8 q b l a 3 t n d 2 9 6 v 5 B 2 6 h M M 2 g x J Z T u R N S A 4 B J a y F F A J 9 V A k 0 j A Q z S 6 n v o P j 6 A N V / I e x y m E C R 1 I H n N G 0 U p B g P C E + a 3 i B i a 9 a s 2 r e z O 4 y 8 Q v S I 0 U a P a q X 0 F f s S w B i U x Q Y 7 q + l 2 K Y U 4 2 c C Z h U g s x A S t m I D q B r q a Q J m D C f 3 T x x T 6 z S d 2 O l b U l 0 Z + r v i Z w m x o y T y H Y m F I d m 0 Z u K / 3 n d D O P L M O c y z R A k m y + K M + G i c q c B u H 2 u g a E Y W 0 K Z 5 v Z W l w 2 p p g x t T B U b g r / 4 8 j J p n 9 V 9 r + 7 f n d c a V 0 U c Z X J E j s k p 8 c k F a Z A b 0 i Q t w k h K n s k r e X M y 5 8 V 5 d z 7 m r S W n m D k k f + B 8 / g C Z l Z I J < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 3 p u J A q h D 1 0 g 3 Y H 6 c y 9 a c I T U D N U = " > A A A B 8 3 i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 B I v g q S Q i 6 L H o x Z N U s L X Q h L L Z T t q l m 9 2 w O x F L 6 N / w 4 k E R r / 4 Z b / 4 b t 2 0 O 2 v p g 4 P H e D D P z o l R w g 5 7 3 7 Z R W V t f W N 8 q b l a 3 t n d 2 9 6 v 5 B 2 6 h M M 2 g x J Z T u R N S A 4 B J a y F F A J 9 V A k 0 j A Q z S 6 n v o P j 6 A N V / I e x y m E C R 1 I H n N G 0 U p B g P C E + a 3 i B i a 9 a s 2 r e z O 4 y 8 Q v S I 0 U a P a q X 0 F f s S w B i U x Q Y 7 q + l 2 K Y U 4 2 c C Z h U g s x A S t m I D q B r q a Q J m D C f 3 T x x T 6 z S d 2 O l b U l 0 Z + r v i Z w m x o y T y H Y m F I d m 0 Z u K / 3 n d D O P L M O c y z R A k m y + K M + G i c q c B u H 2 u g a E Y W 0 K Z 5 v Z W l w 2 p p g x t T B U b g r / 4 8 j J p n 9 V 9 r + 7 f n d c a V 0 U c Z X J E j s k p 8 c k F a Z A b 0 i Q t w k h K n s k r e X M y 5 8 V 5 d z 7 m r S W n m D k k f + B 8 / g C Z l Z I J < / l a t e x i t > . . . . . .

One Layer Rendering in NRM:

Is_render=Yes Location=LowerRight image at level image at level -1

< l a t e x i t s h a 1 _ b a s e 6 4 = " r C G B M z n 4 w B g 0 9 W D 3 n 0 k B j 8 Y d x s Q = " > A A A B 6 3 i c d V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S R C f B U 0 F F X d d H d F q e D G + v 6 H t 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 2 v C 7 9 9 j 9 r w R N 3 Z S Y q h p E P F Y 8 6 o L a Q e C t G v V I O a P w P x f 5 E v q w o L N P q V 9 9 4 g Y Z l E Z Z m g x n Q D P 7 V h T r X l T O C 0 3 M s M p p S N 6 R C 7 j i o q 0 Y T 5 7 

N Y p O X X K g M S J d q U s m a n L E z m V x k x k 5 D o l t S P z 0 y v E v 7 x u Z u O r M O c q z S w q N l 8 U Z 4 L Y h B S P k w H X y K y Y O E K Z 5 u 5 W w k Z U U 2 Z d P O X l E P 4 n r f N a 4 N e C 2 4 t q v b O I o w T H c A J n E M A l 1 O E G G t A E B i N 4 g C d 4 9 q T 3 6 L 1 4 r / P W F W 8 x c w T f 4 L 1 9 A h W E j l I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r C G B M z n 4 w B g 0 9 W D 3 n 0 k B j 8 Y d x s Q = " > A A A B 6 3 i c d V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S R C f B U 0 F F X d d H d F q e D G + v 6 H t 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 2 v C 7 9 9 j 9 r w R N 3 Z S Y q h p E P F Y 8 6 o L a Q e C t G v V I O a P w P x f 5 E v q w o L N P q V 9 9 4 g Y Z l E Z Z m g x n Q D P 7 V h T r X l T O C 0 3 M s M p p S N 6 R C 7 j i o q 0 Y T 5 7 N Y p O X X K g M S J d q U s m a n L E z m V x k x k 5 D o l t S P z 0 y v E v 7 x u Z u O r M O c q z S w q N l 8 U Z 4 L Y h B S P k w H X y K y Y O E K Z 5 u 5 W w k Z U U 2 Z d P O X l E P 4 n r f N a 4 N e C 2 4 t q v b O I o w T H c A J n E M A l 1 O E G G t A E B i N 4 g C d 4 9 q T 3 6 L 1 4 r / P W F W 8 x c w T f 4 L 1 9 A h W E j l I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r C G B M z n 4 w B g 0 9 W D 3 n 0 k B j 8 Y d x s Q = " > A A A B 6 3 i c d V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S R C f B U 0 F F X d d H d F q e D G + v 6 H t 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 2 v C 7 9 9 j 9 r w R N 3 Z S Y q h p E P F Y 8 6 o L a Q e C t G v V I O a P w P x f 5 E v q w o L N P q V 9 9 4 g Y Z l E Z Z m g x n Q D P 7 V h T r X l T O C 0 3 M s M p p S N 6 R C 7 j i o q 0 Y T 5 7 N Y p O X X K g M S J d q U s m a n L E z m V x k x k 5 D o l t S P z 0 y v E v 7 x u Z u O r M O c q z S w q N l 8 U Z 4 L Y h B S P k w H X y K y Y O E K Z 5 u 5 W w k Z U U 2 Z d P O X l E P 4 n r f N a 4 N e C 2 4 t q v b O I o w T H c A J n E M A l 1 O E G G t A E B i N 4 g C d 4 9 q T 3 6 L 1 4 r / P W F W 8 x c w T f 4 L 1 9 A h W E j l I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r C G B M z n 4 w B g 0 9 W D 3 n 0 k B j 8 Y d x s Q = " > A A A B 6 3 i c d V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S R C f B U 0 F F X d d H d F q e D G + v 6 H t 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 2 v C 7 9 9 j 9 r w R N 3 Z S Y q h p E P F Y 8 6 o L a Q e C t G v V I O a P w P x f 5 E v q w o L N P q V 9 9 4 g Y Z l E Z Z m g x n Q D P 7 V h T r X l T O C 0 3 M s M p p S N 6 R C 7 j i o q 0 Y T 5 7 N Y p O X X K g M S J d q U s m a n L E z m V x k x k 5 D o l t S P z 0 y v E v 7 x u Z u O r M O c q z S w q N l 8 U Z 4 L Y h B S P k w H X y K y Y O E K Z 5 u 5 W w k Z U U 2 Z d P O X l E P 4 n r f N a 4 N e C 2 4 t q v b O I o w T H c A J n E M A l 1 O E G G t A E B i N 4 g C d 4 9 q T 3 6 L 1 4 r / P W F W 8 x c w T f 4 L 1 9 A h W E j l I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r C G B M z n 4 w B g 0 9 W D 3 n 0 k B j 8 Y d x s Q = " > A A A B 6 3 i c d V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S R C f B U 0 F F X d d H d F q e D G + v 6 H t 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 2 v C 7 9 9 j 9 r w R N 3 Z S Y q h p E P F Y 8 6 o L a Q e C t G v V I O a P w P x f 5 E v q w o L N P q V 9 9 4 g Y Z l E Z Z m g x n Q D P 7 V h T r X l T O C 0 3 M s M p p S N 6 R C 7 j i o q 0 Y T 5 7 N Y p O X X K g M S J d q U s m a n L E z m V x k x k 5 D o l t S P z 0 y v E v 7 x u Z u O r M O c q z S w q N l 8 U Z 4 L Y h B S P k w H X y K y Y O E K Z 5 u 5 W w k Z U U 2 Z d P O X l E P 4 n r f N a 4 N e C 2 4 t q v b O I o w T H c A J n E M A l 1 O E G G t A E B i

= " > A A A B 6 3 i c d V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S R C f B U 0 F F X d d H d F q e D G + v 6 H t 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 2 v C 7 9 9 j 9 r w R N 3 Z S Y q h p E P F Y 8 6 o L a Q e C t G v V I O a P w P x f 5 E v q w o L N P q V 9 9 4 g Y Z l E Z Z m g x n Q D P 7 V h T r X l T O C 0 3 M s M p p S N 6 R C 7 j i o q 0 Y T 5 7 N Y p O X X K g M S J d q U s m a n L E z m V x k x k 5 D o l t S P z 0 y v E v 7 x u Z u O r M O c q z S w q N l 8 U Z 4 L Y h B S P k w H X y K y Y O E K Z 5 u 5 W w k Z U U 2 Z d P O X l E P 4 n r f N a 4 N e C 2 4 t q v b O I o w T H c A J n E M A l 1 O E G G t A E B i N 4 g C d 4 9 q T 3 6 L 1 4 r / P W F W 8 x c w T f 4 L 1 9 A h W E j l I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r C G B M z n 4 w B g 0 9 W D 3 n 0 k B j 8 Y d x s Q = " > A A A B 6 3 i c d V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S R C f B U 0 F F X d d H d F q e D G + v 6 H t 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 2 v C 7 9 9 j 9 r w R N 3 Z S Y q h p E P F Y 8 6 o L a Q e C t G v V I O a P w P x f 5 E v q w o L N P q V 9 9 4 g Y Z l E Z Z m g x n Q D P 7 V h T r X l T O C 0 3 M s M p p S N 6 R C 7 j i o q 0 Y T 5 7 N Y p O X X K g M S J d q U s m a n L E z m V x k x k 5 D o l t S P z 0 y v E v 7 x u Z u O r M O c q z S w q N l 8 U Z 4 L Y h B S P k w H X y K y Y O E K Z 5 u 5 W w k Z U U 2 Z d P O X l E P 4 n r f N a 4 N e C 2 4 t q v b O I o w T H c A J n E M A l 1 O E G G t A E B i N 4 g C d 4 9 q T 3 6 L 1 4 r / P W F W 8 x c w T f 4 L 1 9 A h W E j l I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r C G B M z n 4 w B g 0 9 W D 3 n 0 k B j 8 Y d x s Q = " > A A A B 6 3 i c d V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S R C f B U 0 F F X d d H d F q e D G + v 6 H t 7 K 6 t r 6 x W d o q b + / s 7 u 1 X D g 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 2 v C 7 9 9 j 9 r w R N 3 Z S Y q h p E P F Y 8 6 o L a Q e C t G v V I O a P w P x f 5 E v q w o L N P q V 9 9 4 g Y Z l E Z Z m g x n Q D P 7 V h T r X l T O C 0 3 M s M p p S N 6 R C 7 j i o q 0 Y T 5 7 N Y p O X X K g M S J d q U s m a n L E z m V x k x k 5 D o l t S P z 0 y v E v 7 x u Z u O r M O c q z S w q N l 8 U Z 4 L Y h B S P k w H X y K y Y O E K Z 5 u 5 W w k Z U U 2 Z d P O X l E P 4 n r f N a 4 N e C 2 4 t q v b O I o w T H c A J n E M A l 1 O E G G t A E B i N 4 g C d 4 9 q T 3 6 L 1 4 r / P W F W 8 x c w T f 4 L 1 9 A h W E j l I = < / l a t e x i t > (`)

< l a t e x i t s h a 1 _ b a s e 6 4 = " 4 B j 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " v o 8 q 8 O C g 4 q C n 9 D q 2 8 x Q w F a W I F

1 P b I L T A 1 W 2 K I v n D b q a O U f G A = " > A A A B 8 3 i c d V D L S g N B E J y N r x h f U Y 9 e F o M Q L 2 F X B D 0 G P O g x g n l A d g m 9 k 9 5 k y M z s M j M r h C W / 4 c W D I l 7 9 G W / + j Z O H E F 8 F D U V V N 1 1 U l H K m j e d 9 O I W V 1 b X 1 j e J m a W t 7 Z 3 e v v H / Q 0 k m m K D Z p w h P V i U A j Z x K b h h m O n V Q h i I h j O x p d T f 3 2 P S r N E n l n x i m G A g a S x Y y C s V I Q X I M Q U A 2 Q 8 9 N e

I L T A 1 W 2 K I v n D b q a O U f G A = " > A A A B 8 3 i c d V D L S g N B E J y N r x h f U Y 9 e F o M Q L 2 F X B D 0 G P O g x g n l A d g m 9 k 9 5 k y M z s M j M r h C W / 4 c W D I l 7 9 G W / + j Z O H E F 8 F D U V V N 1 1 U l H K m j e d 9 O I W V 1 b X 1 j e J m a W t 7 Z 3 e v v H / Q 0 k m m K D Z p w h P V i U A j Z x K b h h m O n V Q h i I h j O x p d T f 3 2 P S r N E n l n x i m G A g a S x Y y C s V I Q X I M Q U A 2 Q 8 9 N e

I L T A 1 W 2 K I v n D b q a O U f G A = " > A A A B 8 3 i c d V D L S g N B E J y N r x h f U Y 9 e F o M Q L 2 F X B D 0 G P O g x g n l A d g m 9 k 9 5 k y M z s M j M r h C W / 4 c W D I l 7 9 G W / + j Z O H E F 8 F D U V V N 1 1 U l H K m j e d 9 O I W V 1 b X 1 j e J m a W t 7 Z 3 e v v H / Q 0 k m m K D Z p w h P V i U A j Z x K b h h m O n V Q h i I h j O x p d T f 3 2 P S r N E n l n x i m G A g a S x Y y C s V I Q X I M Q U A 2 Q 8 9 N e

I L T A 1 W 2 K I v n D b q a O U f G A = " > A A A B 8 3 i c d V D L S g N B E J y N r x h f U Y 9 e F o M Q L 2 F X B D 0 G P O g x g n l A d g m 9 k 9 5 k y M z s M j M r h C W / 4 c W D I l 7 9 G W / + j Z O H E F 8 F D U V V N 1 1 U l H K m j e d 9 O I W V 1 b X 1 j e J m a W t 7 Z 3 e v v H / Q 0 k m m K D Z p w h P V i U A j Z x K b h h m O n V Q h i I h j O x p d T f 3 2 P S r N E n l n x i m G A g a S x Y y C s V I Q X I M Q U A 2 Q 8 9 N e

A q S / a v u c 1 u r k G v g o X 6 N Z T A = " > A A A B 6 3 i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S S I J h Y 0 F F X d d H d F q e D G + v 6 X t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 1 v C 7 / 9 i N r w R D 3 Y S Y q h p E P F Y 8 6 o L a Q e C t G v V P 2 a P w P 5 I c E y q c I C j X 7 l s z d I W C Z R W S a o M d 3 A T 2 2 Y U 2 0 5 E z g t 9 z K D K W V j O s S u o 4 p K N G E + u 3 V K z p 0 y I H G i X S l L Z u r v i Z x K Y y Y y c p 2 S 2 p F Z 9 g r x P 6 + b 2 f g m z L l K M 4 u K z R f F m S A 2 I c X j Z M A 1 M i s m j l C m u b u V s B H V l F k X T 9 m F s P L y K m l d 1 g K / F t x f V e u d R R w l O I U z u I A A r q E O d 9 C A J j A Y w R O 8 w K s n v W f v z X u f t 6 5 5 i 5 k T + A P v 4 x s V c o 5 S < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W 6 w Q y A q S / a v u c 1 u r k G v g o X 6 N Z T A = " > A A A B 6 3 i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S S I J h Y 0 F F X d d H d F q e D G + v 6 X t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 1 v C 7 / 9 i N r w R D 3 Y S Y q h p E P F Y 8 6 o L a Q e C t G v V P 2 a P w P 5 I c E y q c I C j X 7 l s z d I W C Z R W S a o M d 3 A T 2 2 Y U 2 0 5 E z g t 9 z K D K W V j O s S u o 4 p K N G E + u 3 V K z p 0 y I H G i X S l L Z u r v i Z x K Y y Y y c p 2 S 2 p F Z 9 g r x P 6 + b 2 f g m z L l K M 4 u K z R f F m S A 2 I c X j Z M A 1 M i s m j l C m u b u V s B H V l F k X T 9 m F s P L y K m l d 1 g K / F t x f V e u d R R w l O I U z u I A A r q E O d 9 C A J j A Y w R O 8 w K s n v W f v z X u f t 6 5 5 i 5 k T + A P v 4 x s V c o 5 S < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W 6 w Q y A q S / a v u c 1 u r k G v g o X 6 N Z T A = " > A A A B 6 3 i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S S I J h Y 0 F F X d d H d F q e D G + v 6 X t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 1 v C 7 / 9 i N r w R D 3 Y S Y q h p E P F Y 8 6 o L a Q e C t G v V P 2 a P w P 5 I c E y q c I C j X 7 l s z d I W C Z R W S a o M d 3 A T 2 2 Y U 2 0 5 E z g t 9 z K D K W V j O s S u o 4 p K N G E + u 3 V K z p 0 y I H G i X S l L Z u r v i Z x K Y y Y y c p 2 S 2 p F Z 9 g r x P 6 + b 2 f g m z L l K M 4 u K z R f F m S A 2 I c X j Z M A 1 M i s m j l C m u b u V s B H V l F k X T 9 m F s P L y K m l d 1 g K / F t x f V e u d R R w l O I U z u I A A r q E O d 9 C A J j A Y w R O 8 w K s n v W f v z X u f t 6 5 5 i 5 k T + A P v 4 x s V c o 5 S < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W 6 w Q y A q S / a v u c 1 u r k G v g o X 6 N Z T A = " > A A A B 6 3 i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S S I J h Y 0 F F X d d H d F q e D G + v 6 X t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 1 v C 7 / 9 i N r w R D 3 Y S Y q h p E P F Y 8 6 o L a Q e C t G v V P 2 a P w P 5 I c E y q c I C j X 7 l s z d I W C Z R W S a o M d 3 A T 2 2 Y U 2 0 5 E z g t 9 z K D K W V j O s S u o 4 p K N G E + u 3 V K z p 0 y I H G i X S l L Z u r v i Z x K Y y Y y c p 2 S 2 p F Z 9 g r x P 6 + b 2 f g m z L l K M 4 u K z R f F m S A 2 I c X j Z M A 1 M i s m j l C m u b u V s B H V l F k X T 9 m F s P L y K m l d 1 g K / F t x f V e u d R R w l O I U z u I A A r q E O d 9 C A J j A Y w R O 8 w K s n v W f v z X u f t

V s = " > A A A B 8 X i c b V D L S g N B E O y N r x h f U Y 9 e B o M Q L 2 F X A n o R A l 4 8 R j A P T J Y w O + l N h s z O L j O z Q l j y F 1 4 8 K O L V v / H m 3 z h 5 I J p Y 0 F B U d d P d F S S C a + O 6 X 0 5 u b X 1 j c y u / X d j Z 3 d s / K B 4 e N X W c K o Y N F o t Y t Q O q U X C J D c O N w H a i k E a B w F Y w u p n 6 r U d U m s f y 3 o w T 9 C M 6 k D z k j B o r P b T K X R T i n F y T X r H k V t w Z y A / x l k k J F q j 3 i p / d f s z S C K V h g m r d 8 d z E + B l V h j O B k 0 I 3 1 Z h Q N q I D 7 F g q a Y T a z 2 Y X T 8 i Z V f o k j J U t a c h M / T 2 R 0 U j r c R T Y z o i a o V 7 2 p u J / X i c 1 4 Z W f c Z m k B i W b L w p T Q U x M p u + T P l f I j B h b Q p n i 9 l b C h l R R Z m x I B R v C y s u r p H l R 8 d y K d 1 c t 1 d q L O P J w A q d Q B g 8 u o Q a 3 U I c G M J D w B C

/ w 6 m j n 2 X l z 3 u e t O W c x c w x / 4 H x 8 A 7 r p j 7 M = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v o 8 q 8 O C g 4 q C n 9 D q 2 8 x Q w F a W I F

V s = " > A A A B 8 X i c b V D L S g N B E O y N r x h f U Y 9 e B o M Q L 2 F X A n o R A l 4 8 R j A P T J Y w O + l N h s z O L j O z Q l j y F 1 4 8 K O L V v / H m 3 z h 5 I J p Y 0 F B U d d P d F S S C a + O 6 X 0 5 u b X 1 j c y u / X d j Z 3 d s / K B 4 e N X W c K o Y N F o t Y t Q O q U X C J D c O N w H a i k E a B w F Y w u p n 6 r U d U m s f y 3 o w T 9 C M 6 k D z k j B o r P b T K X R T i n F y T X r H k V t w Z y A / x l k k J F q j 3 i p / d f s z S C K V h g m r d 8 d z E + B l V h j O B k 0 I 3 1 Z h Q N q I D 7 F g q a Y T a z 2 Y X T 8 i Z V f o k j J U t a c h M / T 2 R 0 U j r c R T Y z o i a o V 7 2 p u J / X i c 1 4 Z W f c Z m k B i W b L w p T Q U x M p u + T P l f I j B h b Q p n i 9 l b C h l R R Z m x I B R v C y s u r p H l R 8 d y K d 1 c t 1 d q L O P J w A q d Q B g 8 u o Q a 3 U I c G M J D w B C

/ w 6 m j n 2 X l z 3 u e t O W c x c w x / 4 H x 8 A 7 r p j 7 M = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v o 8 q 8 O C g 4 q C n 9 D q 2 8 x Q w F a W I F

V s = " > A A A B 8 X i c b V D L S g N B E O y N r x h f U Y 9 e B o M Q L 2 F X A n o R A l 4 8 R j A P T J Y w O + l N h s z O L j O z Q l j y F 1 4 8 K O L V v / H m 3 z h 5 I J p Y 0 F B U d d P d F S S C a + O 6 X 0 5 u b X 1 j c y u / X d j Z 3 d s / K B 4 e N X W c K o Y N F o t Y t Q O q U X C J D c O N w H a i k E a B w F Y w u p n 6 r U d U m s f y 3 o w T 9 C M 6 k D z k j B o r P b T K X R T i n F y T X r H k V t w Z y A / x l k k J F q j 3 i p / d f s z S C K V h g m r d 8 d z E + B l V h j O B k 0 I 3 1 Z h Q N q I D 7 F g q a Y T a z 2 Y X T 8 i Z V f o k j J U t a c h M / T 2 R 0 U j r c R T Y z o i a o V 7 2 p u J / X i c 1 4 Z W f c Z m k B i W b L w p T Q U x M p u + T P l f I j B h b Q p n i 9 l b C h l R R Z m x I B R v C y s u r p H l R 8 d y K d 1 c t 1 d q L O P J w A q d Q B g 8 u o Q a 3 U I c G M J D w B C

/ w 6 m j n 2 X l z 3 u e t O W c x c w x / 4 H x 8 A 7 r p j 7 M = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v o 8 q 8 O C g 4 q C n 9 D q 2 8 x Q w F a W I F

V s = " > A A A B 8 X i c b V D L S g N B E O y N r x h f U Y 9 e B o M Q L 2 F X A n o R A l 4 8 R j A P T J Y w O + l N h s z O L j O z Q l j y F 1 4 8 K O L V v / H m 3 z h 5 I J p Y 0 F B U d d P d F S S C a + O 6 X 0 5 u b X 1 j c y u / X d j Z 3 d s / K B 4 e N X W c K o Y N F o t Y t Q O q U X C J D c O N w H a i k E a B w F Y w u p n 6 r U d U m s f y 3 o w T 9 C M 6 k D z k j B o r P b T K X R T i n F y T X r H k V t w Z y A / x l k k J F q j 3 i p / d f s z S C K V h g m r d 8 d z E + B l V h j O B k 0 I 3 1 Z h Q N q I D 7 F g q a Y T a z 2 Y X T 8 i Z V f o k j J U t a c h M / T 2 R 0 U j r c R T Y z o i a o V 7 2 p u J / X i c 1 4 Z W f c Z m k B i W b L w p T Q U x M p u + T P l f I j B h b Q p n i 9 l b C h l R R Z m x I B R v C y s u r p H l R 8 d y K d 1 c t 1 d q L O P J w A q d Q B g 8 u o Q a 3 U I c G M J D w B C

/ w 6 m j n 2 X l z 3 u e t O W c x c w x / 4 H x 8 A 7 r p j 7 M = < / l a t e x i t > > (`)

< l a t e x i t s h a 1 _ b a s e 6 4 = " top-down process of image generation is through coarse-to-fine rendering, which progressively increases the resolution of the rendered image. This is intuitive since the reverse process of bottom-up inference reduces the resolution (and dimension) through operations such as spatial pooling. We also introduce structured stochasticity in the rendering process through a small set of discrete latent variables, which capture the uncertainty in reversing the CNN feed-forward process. The rendering in NRM follows a product of linear transformations, which can be considered as the transpose of the inference process in CNNs. In particular, the rendering weights in NRM are proportional to the transposes of the filters in CNNs. Furthermore, the bias terms in the ReLU units at each layer (after the convolution operator) make the latent variables in different network layers dependent (when the bias terms are non-zero). This design of image prior has an interesting interpretation from a predictive-coding perspective in neuroscience: the dependency between latent variables can be considered as a form of backward connections that captures prior knowledge from coarser levels in NRM and helps adjust the estimation at the finer levels (Rao & Ballard, 1999; Friston, 2018) .

d 2 U O G X L 6 k Q p Q q s + u Y G a m F h m o m O s = " > A A A B / H i c d V D L S g M x F M 3 4 r P U 1 2 q W b Y B H q p s y I o M u C C 1 1 W s A / o 1 J J J M 2 1 o M h m S O 8 I w 1 F 9 x 4 0 I R t 3 6 I O / / G 9 C H U 1 4 E L h 3 P u 5 R 5 O m A h u w P M + n K X l l d W 1 9 c J G c X N r e 2 f X 3 d t v G p V q y h p U C a X b I T F M 8 J g 1 g I N g 7 U Q z I k P B W u H o Y u K 3 7 p g 2 X M U 3 k C W s K 8 k g 5 h G n B K z U c 0 v B J Z G S 3 O Y B q G R c C Z g Q x z 2 3 7 F e 9 K b D 3 i 3 x Z Z T R H v e e + B 3 1 F U 8 l i o I I Y 0 / G 9 B L o 5 0 c C p Y O N i k B q W E D o i A 9 a x N C a S m W 4 + D T / G R 1 b p 4 0 h p O z H g q b p 4 k R N p T C Z D u y k J D M 1 P b y L + 5 X V S i M 6 7 O Y + T F F h M Z 4 + i V G B Q e N I E 7 n P N K I j M E k I 1 t 1 k x H R J N K N i + i o s l / E + a J 1 X f q / r X p + V a e 1 5 H A R 2 g Q 1 R B P j p D N X S F 6 q i B K M r Q A 3 p C z 8 6 9 8 + i 8 O K + z 1 S V n f l N C 3 + C 8 f Q J u b 5 S u < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d 2 U O G X L 6 k Q p Q q s + u Y G a m F h m o m O s = " > A A A B / H i c d V D L S g M x F M 3 4 r P U 1 2 q W b Y B H q p s y I o M u C C 1 1 W s A / o 1 J J J M 2 1 o M h m S O 8 I w 1 F 9 x 4 0 I R t 3 6 I O / / G 9 C H U 1 4 E L h 3 P u 5 R 5 O m A h u w P M + n K X l l d W 1 9 c J G c X N r e 2 f X 3 d t v G p V q y h p U C a X b I T F M 8 J g 1 g I N g 7 U Q z I k P B W u H o Y u K 3 7 p g 2 X M U 3 k C W s K 8 k g 5 h G n B K z U c 0 v B J Z G S 3 O Y B q G R c C Z g Q x z 2 3 7 F e 9 K b D 3 i 3 x Z Z T R H v e e + B 3 1 F U 8 l i o I I Y 0 / G 9 B L o 5 0 c C p Y O N i k B q W E D o i A 9 a x N C a S m W 4 + D T / G R 1 b p 4 0 h p O z H g q b p 4 k R N p T C Z D u y k J D M 1 P b y L + 5 X V S i M 6 7 O Y + T F F h M Z 4 + i V G B Q e N I E 7 n P N K I j M E k I 1 t 1 k x H R J N K N i + i o s l / E + a J 1 X f q / r X p + V a e 1 5 H A R 2 g Q 1 R B P j p D N X S F 6 q i B K M r Q A 3 p C z 8 6 9 8 + i 8 O K + z 1 S V n f l N C 3 + C 8 f Q J u b 5 S u < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d 2 U O G X L 6 k Q p Q q s + u Y G a m F h m o m O s = " > A A A B / H i c d V D L S g M x F M 3 4 r P U 1 2 q W b Y B H q p s y I o M u C C 1 1 W s A / o 1 J J J M 2 1 o M h m S O 8 I w 1 F 9 x 4 0 I R t 3 6 I O / / G 9 C H U 1 4 E L h 3 P u 5 R 5 O m A h u w P M + n K X l l d W 1 9 c J G c X N r e 2 f X 3 d t v G p V q y h p U C a X b I T F M 8 J g 1 g I N g 7 U Q z I k P B W u H o Y u K 3 7 p g 2 X M U 3 k C W s K 8 k g 5 h G n B K z U c 0 v B J Z G S 3 O Y B q G R c C Z g Q x z 2 3 7 F e 9 K b D 3 i 3 x Z Z T R H v e e + B 3 1 F U 8 l i o I I Y 0 / G 9 B L o 5 0 c C p Y O N i k B q W E D o i A 9 a x N C a S m W 4 + D T / G R 1 b p 4 0 h p O z H g q b p 4 k R N p T C Z D u y k J D M 1 P b y L + 5 X V S i M 6 7 O Y + T F F h M Z 4 + i V G B Q e N I E 7 n P N K I j M E k I 1 t 1 k x H R J N K N i + i o s l / E + a J 1 X f q / r X p + V a e 1 5 H A R 2 g Q 1 R B P j p D N X S F 6 q i B K M r Q A 3 p C z 8 6 9 8 + i 8 O K + z 1 S V n f l N C 3 + C 8 f Q J u b 5 S u < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " h P + 6 L r U f 2 d 3 t Z a l d q a Q Q v E K M X y w = " > A A A B 2 X i c b Z D N S g M x F I X v 1 L 8 6 V q 1 r N 8 E i u C o z b n Q p u H F Z w b Z C O 5 R M 5 k 4 b m s k M y R 2 h D H 0 B F 2 5 E f C 9 3 v o 3 p z 0 J b D w Q + z k n I v S c u l L Q U B N

Y C 9 X + T L q q A F G n 3 3 P R j E N J V M A R X E m K 7 v J d D L i Q Z O B Z u U g t S w h N A x G b K u p Y p I Z n r 5 L P w E H 1 t l g K N Y 2 1 G A Z + r y R U 6 k M Z k M 7 a Y k M D I / v a n 4 l 9 d N I T r v 5 V w l K T B F 5 4 + i V G C I 8 b Q J P O C a U R C Z J Y R q b r N i O i K a U L B 9 l Z Z L + J + 0 T m u + V / O v v U q 9 s 6 i j i A 7 R E a o i H 5 2 h O r p C D d R E F G X o A T 2 h Z + f e e X R e n N f 5 a s F Z 3 J T R N z h v n 2 0 v l K o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d 2 U O G X L 6 k Q p Q q s + u Y G a m F h m o m O s = " > A A A B / H i c d V D L S g M x F M 3 4 r P U 1 2 q W b Y B H q p s y I o M u C C 1 1 W s A / o 1 J J J M 2 1 o M h m S O 8 I w 1 F 9 x 4 0 I R t 3 6 I O / / G 9 C H U 1 4 E L h 3 P u 5 R 5 O m A h u w P M + n K X l l d W 1 9 c J G c X N r e 2 f X 3 d t v G p V q y h p U C a X b I T F M 8 J g 1 g I N g 7 U Q z I k P B W u H o Y u K 3 7 p g 2 X M U 3 k C W s K 8 k g 5 h G n B K z U c 0 v B J Z G S 3 O Y B q G R c C Z g Q x z 2 3 7 F e 9 K b D 3 i 3 x Z Z T R H v e e + B 3 1 F U 8 l i o I I Y 0 / G 9 B L o 5 0 c C p Y O N i k B q W E D o i A 9 a x N C a S m W 4 + D T / G R 1 b p 4 0 h p O z H g q b p 4 k R N p T C Z D u y k J D M 1 P b y L + 5 X V S i M 6 7 O Y + T F F h M Z 4 + i V G B Q e N I E 7 n P N K I j M E k I 1 t 1 k x H R J N K N i + i o s l / E + a J 1 X f q / r X p + V a e 1 5 H A R 2 g Q 1 R B P j p D N X S F 6 q i B K M r Q A 3 p C z 8 6 9 8 + i 8 O K + z 1 S V n f l N C 3 + C 8 f Q J u b 5 S u < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d 2 U O G X L 6 k Q p Q q s + u Y G a m F h m o m O s = " > A A A B / H i c d V D L S g M x F M 3 4 r P U 1 2 q W b Y B H q p s y I o M u C C 1 1 W s A / o 1 J J J M 2 1 o M h m S O 8 I w 1 F 9 x 4 0 I R t 3 6 I O / / G 9 C H U 1 4 E L h 3 P u 5 R 5 O m A h u w P M + n K X l l d W 1 9 c J G c X N r e 2 f X 3 d t v G p V q y h p U C a X b I T F M 8 J g 1 g I N g 7 U Q z I k P B W u H o Y u K 3 7 p g 2 X M U 3 k C W s K 8 k g 5 h G n B K z U c 0 v B J Z G S 3 O Y B q G R c C Z g Q x z 2 3 7 F e 9 K b D 3 i 3 x Z Z T R H v e e + B 3 1 F U 8 l i o I I Y 0 / G 9 B L o 5 0 c C p Y O N i k B q W E D o i A 9 a x N C a S m W 4 + D T / G R 1 b p 4 0 h p O z H g q b p 4 k R N p T C Z D u y k J D M 1 P b y L + 5 X V S i M 6 7 O Y + T F F h M Z 4 + i V G B Q e N I E 7 n P N K I j M E k I 1 t 1 k x H R J N K N i + i o s l / E + a J 1 X f q / r X p + V a e 1 5 H A R 2 g Q 1 R B P j p D N X S F 6 q i B K M r Q A 3 p C z 8 6 9 8 + i 8 O K + z 1 S V n f l N C 3 + C 8 f Q J u b 5 S u < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d 2 U O G X L 6 k Q p Q q s + u Y G a m F h m o m O s = " > A A A B / H i c d V D L S g M x F M 3 4 r P U 1 2 q W b Y B H q p s y I o M u C C 1 1 W s A / o 1 J J J M 2 1 o M h m S O 8 I w 1 F 9 x 4 0 I R t 3 6 I O / / G 9 C H U 1 4 E L h 3 P u 5 R 5 O m A h u w P M + n K X l l d W 1 9 c J G c X N r e 2 f X 3 d t v G p V q y h p U C a X b I T F M 8 J g 1 g I N g 7 U Q z I k P B W u H o Y u K 3 7 p g 2 X M U 3 k C W s K 8 k g 5 h G n B K z U c 0 v B J Z G S 3 O Y B q G R c C Z g Q x z 2 3 7 F e 9 K b D 3 i 3 x Z Z T R H v e e + B 3 1 F U 8 l i o I I Y 0 / G 9 B L o 5 0 c C p Y O N i k B q W E D o i A 9 a x N C a S m W 4 + D T / G R 1 b p 4 0 h p O z H g q b p 4 k R N p T C Z D u y k J D M 1 P b y L + 5 X V S i M 6 7 O Y + T F F h M Z 4 + i V G B Q e N I E 7 n P N K I j M E k I 1 t 1 k x H R J N K N i + i o s l / E + a J 1 X f q / r X p + V a e 1 5 H A R 2 g Q 1 R B P j p D N X S F 6 q i B K M r Q A 3 p C z 8 6 9 8 + i 8 O K + z 1 S V n f l N C 3 + C 8 f Q J u b 5 S u < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d 2 U O G X L 6 k Q p Q q s + u Y G a m F h m o m O s = " > A A A B / H i c d V D L S g M x F M 3 4 r P U 1 2 q W b Y B H q p s y I o M u C C 1 1 W s A / o 1 J J J M 2 1 o M h m S O 8 I w 1 F 9 x 4 0 I R t 3 6 I O / / G 9 C H U 1 4 E L h 3 P u 5 R 5 O m A h u w P M + n K X l l d W 1 9 c J G c X N r e 2 f X 3 d t v G p V q y h p U C a X b I T F M 8 J g 1 g I N g 7 U Q z I k P B W u H o Y u K 3 7 p g 2 X M U 3 k C W s K 8 k g 5 h G n B K z U c 0 v B J Z G S 3 O Y B q G R c C Z g Q x z 2 3 7 F e 9 K b D 3 i 3 x Z Z T R H v e e + B 3 1 F U 8 l i o I I Y 0 / G 9 B L o 5 0 c C p Y O N i k B q W E D o i A 9 a x N C a S m W 4 + D T / G R 1 b p 4 0 h p O z H g q b p 4 k R N p T C Z D u y k J D M 1 P b y L + 5 X V S i M 6 7 O Y + T F F h M Z 4 + i V G B Q e N I E 7 n P N K I j M E k I 1 t 1 k x H R J N K N i + i o s l / E + a J 1 X f q / r X p + V a e 1 5 H A R 2 g Q 1 R B P j p D N X S F 6 q i B K M r Q A 3 p C z 8 6 9 8 + i 8 O K + z 1 S V n f l N C 3 + C 8 f Q J u b 5 S u < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d 2 U O G X L 6 k Q p Q q s + u Y G a m F h m o m O s = " > A A A B / H i c d V D L S g M x F M 3 4 r P U 1 2 q W b Y B H q p s y I o M u C C 1 1 W s A / o 1 J J J M 2 1 o M h m S O 8 I w 1 F 9 x 4 0 I R t 3 6 I O / / G 9 C H U 1 4 E L h 3 P u 5 R 5 O m A h u w P M + n K X l l d W 1 9 c J G c X N r e 2 f X 3 d t v G p V q y h p U C a X b I T F M 8 J g 1 g I N g 7 U Q z I k P B W u H o Y u K 3 7 p g 2 X M U 3 k C W s K 8 k g 5 h G n B K z U c 0 v B J Z G S 3 O Y B q G R c C Z g Q x z 2 3 7 F e 9 K b D 3 i 3 x Z Z T R H v e e + B 3 1 F U 8 l i o I I Y 0 / G 9 B L o 5 0 c C p Y O N i k B q W E D o i A 9 a x N C a S m W 4 + D T / G R 1 b p 4 0 h p O z H g q b p 4 k R N p T C Z D u y k J D M 1 P b y L + 5 X V S i M 6 7 O Y + T F F h M Z 4 + i V G B Q e N I E 7 n P N K I j M E k I 1 t 1 k x H R J N K N i + i o s l / E + a J 1 X f q / r X p + V a e 1 5 H A R 2 g Q 1 R B P j p D N X S F 6 q i B K M r Q A 3 p C z 8 6 9 8 + i 8 O K + z 1 S V n f l N C 3 + C 8 f Q J u b 5 S u < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d 2 U O G X L 6 k Q p Q q s + u Y G a m F h m o m O s = " > A A A B / H i c d V D L S g M x F M 3 4 r P U 1 2 q W b Y B H q p s y I o M u C C 1 1 W s A / o 1 J J J M 2 1 o M h m S O 8 I w 1 F 9 x 4 0 I R t 3 6 I O / / G 9 C H U 1 4 E L h 3 P u 5 R 5 O m A h u w P M + n K X l l d W 1 9 c J G c X N r e 2 f X 3 d t v G p V q y h p U C a X b I T F M 8 J g 1 g I N g 7 U Q z I k P B W u H o Y u K 3 7 p g 2 X M U 3 k C W s K 8 k g 5 h G n B K z U c 0 v B J Z G S 3 O Y B q G R c C Z g Q x z 2 3 7 F e 9 K b D 3 i 3 x Z Z T R H v e e + B 3 1 F U 8 l i o I I Y 0 / G 9 B L o 5 0 c C p Y O N i k B q W E D o i A 9 a x N C a S m W 4 + D T / G R 1 b p 4 0 h p O z H g q b p 4 k R N p T C Z D u y k J D M 1 P b y L + 5 X V S i M 6 7 O Y + T F F h M Z 4 + i V G B Q e N I E 7 n P N K I j M E k I 1 t 1 k x H R J N K N i + i o s l / E + a J 1 X f q / r X p + V a e 1 5 H A R 2 g Q 1 R B P j p D N X S F 6 q i B K M r Q A 3 p C z 8 6 9 8 + i 8 O K + z 1 S V n f l N C 3 + C 8 f Q J u b 5 S u < / l a t e x i t > / < l a t e x i t s h a 1 _ b a s e 6 4 = " U P X T k Y q C X z y / M q d S g f F H s L 9 y Z c 8 = " > A A A B 7 n i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N n J J B k y O z P M 9 A p h y U d 4 8 a C I V 7 / H m 3 / j 5 I F o Y k F D U d V N d 1 d i p H A Y h l / B 2 v r G 5 t Z 2 Y a e 4 u 7 d / c F g 6 O m 4 4 n V n G 6 0 x L b V s J d V w K x e s o U P K W s Z y m i e T N Z H Q 7 9 Z u P 3 D q h 1 Q O O D Y 9 T O l C i L x h F L z U 7 x m q D u l s q h 5 V w B v J D o m V S h g V q 3 d J n p 6 d Z l n K F T F L n 2 l F o M M 6 p R c E k n x Q 7 m e O G s h E d 8 L a n i q b c x f n s 3 A k 5 9 0 q P 9 L X 1 p Z D M 1 N 8 T O U 2 d G 6 e J 7 0 w p D t 2 y N x X / 8 9 o Z 9 m / i X C i T I V d s v q i f S Y K a T H 8 n P W E 5 Q z n 2 h D I r / K 2 E D a m l D H 1 C R R / C y s u r p H F Z i c J K d H 9 V r r Y W c R T g F M 7 g A i K 4 h i r c Q Q 3 q w G A E T / A C r 4 E J n o O 3 4 H 3 e u h Y s Z k 7 g D 4 K P b 6 r n j 9 c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U P X T k Y q C X z y / M q d S g f F H s L 9 y Z c 8 = " > A A A B 7 n i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N n J J B k y O z P M 9 A p h y U d 4 8 a C I V 7 / H m 3 / j 5 I F o Y k F D U d V N d 1 d i p H A Y h l / B 2 v r G 5 t Z 2 Y a e 4 u 7 d / c F g 6 O m 4 4 n V n G 6 0 x L b V s J d V w K x e s o U P K W s Z y m i e T N Z H Q 7 9 Z u P 3 D q h 1 Q O O D Y 9 T O l C i L x h F L z U 7 x m q D u l s q h 5 V w B v J D o m V S h g V q 3 d J n p 6 d Z l n K F T F L n 2 l F o M M 6 p R c E k n x Q 7 m e O G s h E d 8 L a n i q b c x f n s 3 A k 5 9 0 q P 9 L X 1 p Z D M 1 N 8 T O U 2 d G 6 e J 7 0 w p D t 2 y N x X / 8 9 o Z 9 m / i X C i T I V d s v q i f S Y K a T H 8 n P W E 5 Q z n 2 h D I r / K 2 E D a m l D H 1 C R R / C y s u r p H F Z i c J K d H 9 V r r Y W c R T g F M 7 g A i K 4 h i r c Q Q 3 q w G A E T / A C r 4 E J n o O 3 4 H 3 e u h Y s Z k 7 g D 4 K P b 6 r n j 9 c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U P X T k Y q C X z y / M q d S g f F H s L 9 y Z c 8 = " > A A A B 7 n i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N n J J B k y O z P M 9 A p h y U d 4 8 a C I V 7 / H m 3 / j 5 I F o Y k F D U d V N d 1 d i p H A Y h l / B 2 v r G 5 t Z 2 Y a e 4 u 7 d / c F g 6 O m 4 4 n V n G 6 0 x L b V s J d V w K x e s o U P K W s Z y m i e T N Z H Q 7 9 Z u P 3 D q h 1 Q O O D Y 9 T O l C i L x h F L z U 7 x m q D u l s q h 5 V w B v J D o m V S h g V q 3 d J n p 6 d Z l n K F T F L n 2 l F o M M 6 p R c E k n x Q 7 m e O G s h E d 8 L a n i q b c x f n s 3 A k 5 9 0 q P 9 L X 1 p Z D M 1 N 8 T O U 2 d G 6 e J 7 0 w p D t 2 y N x X / 8 9 o Z 9 m / i X C i T I V d s v q i f S Y K a T H 8 n P W E 5 Q z n 2 h D I r / K 2 E D a m l D H 1 C R R / C y s u r p H F Z i c J K d H 9 V r r Y W c R T g F M 7 g A i K 4 h i r c Q Q 3 q w G A E T / A C r 4 E J n o O 3 4 H 3 e u h Y s Z k 7 g D 4 K P b 6 r n j 9 c = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U P X T k Y q C X z y / M q d S g f F H s L 9 y Z c 8 = " > A A A B 7 n i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N n J J B k y O z P M 9 A p h y U d 4 8 a C I V 7 / H m 3 / j 5 I F o Y k F D U d V N d 1 d i p H A Y h l / B 2 v r G 5 t Z 2 Y a e 4 u 7 d / c F g 6 O m 4 4 n V n G 6 0 x L b V s J d V w K x e s o U P K W s Z y m i e T N Z H Q 7 9 Z u P 3 D q h 1 Q O O D Y 9 T O l C i L x h F L z U 7 x m q D u l s q h 5 V w B v J D o m V S h g V q 3 d J n p 6 d Z l n K F T F L n 2 l F o M M 6 p R c E k n x Q 7 m e O G s h E d 8 L a n i q b c x f n s 3 A k 5 9 0 q P 9 L X 1 p Z D M 1 N 8 T O U 2 d G 6 e J 7 0 w p D t 2 y N x X / 8 9 o Z 9 m / i X C i T I V d s v q i f S Y K a T H 8 n P W E 5 Q z n 2 h D I r / K 2 E D a m l D H 1 C R R / C y s u r p H F Z i c J K d H 9 V r r Y W c R T g F M 7 g A i K 4 h i r c Q Q 3 q w G A E T / A C r 4 E J n o O 3 4 H 3 e u h Y

NRM is a likelihood-based framework, where unsupervised learning can be derived by maximizing the expected complete-data log-likelihood of the model while supervised learning is done through optimizing the class-conditional log-likelihood. Semi-supervised learning unifies both log-likelihoods into an objective cost for learning from both labeled and unlabeled data. The NRM prior has the desirable property of being a conjugate prior, which makes learning in NRM computationally efficient.

Interestingly, we derive the popular cross-entropy loss used to train CNNs for supervised learning as an upper bound of the NRM's negative class-conditional log-likelihood. This new interpretation of cross-entropy allow us to develop better losses for training CNNs. An example is the Max-Min cross-entropy discussed in Contribution 2 and Section 3.

Contribution 2 -New regularization, loss function, architecture and generalization bounds:

The joint nature of generation, inference, and learning in NRM allows us to develop new training procedures for semi-supervised and supervised learning, as well as new theoretical (statistical) guarantees for learning. In particular, for training, we derive a new form of regularization termed as the Rendering Path Normalization (RPN) from the NRM's conjugate prior. A rendering path is a set of latent variable values in NRM. Unlike the path-wise regularizer in (Neyshabur et al., 2015) , RPN uses information from a generative model to penalizes the number of the possible rendering paths, encouraging the network to be compact in terms of representing the image. It also helps enforce the dependency among different layers in NRM during training and improves classification performance.

We provide a new theoretical bound based on NRM. In particular, we prove that NRM is statistically consistent and derive a generalization bound of NRM for (semi-)supervised learning tasks. Our generalization bound is proportional to the number of active rendering paths that generate close-to-real images. This suggests that RPN regularization may help in generalization since RPN enforces the dependencies among latent variables in NRM and, therefore, reduces the number of active rendering paths. We observe that RPN helps improve generalization in our experiments.

Max-Min cross-entropy and network::

We propose the new Max-Min cross-entropy loss function for learning, based on negative class-conditional log-likelihood in NRM. It combines the traditional cross-entropy with another loss, which we term as the Min cross-entropy. While the traditional (Max) cross-entropy maximizes the probability of correct labels, the Min cross-entropy minimizes the probability of incorrect labels. We show that the Max-Min cross-entropy is also an upper bound to the negative conditional log-likehood of NRM, just like the cross-entropy loss. The Max-Min cross-entropy is realized through a new CNN architecture, namely the Max-Min network, which is a CNN with an additional branch sharing weights with the original CNN but containing minimum pooling (MinPool) operator and negative rectified linear units (NReLUs), i.e., min(•, 0) (see Figure 4 ). Although the Max-Min network is derived from NRM, it is a meta-architecture that can be applied independently on any CNN architecture. We show empirically that Max-Min networks and crossentropy help improve the SOTA on object classification for supervised and semi-supervised learning.

Contribution 3 -State-of-the-art empirical results for semi-supervised and supervised learning: We show strong results for semi-supervised learning over CIFAR10, CIFAR100 and SVHN benchmarks in comparison with SOTA methods that use and do not use consistency regularization.

Consistency regularization, such as those used in Temporal Ensembling (Laine & Aila, 2017) and Mean Teacher (Tarvainen & Valpola, 2017) , enforces the networks to learn representation invariant to realistic perturbations of the data. NRM alone outperforms most SOTA methods which do not use consistency regularization (Salimans et al., 2016; Dumoulin et al., 2017) Despite good classification results, there is a caveat that NRM may not generate good looking images since that objective is not "baked" into its training. NRM is primarily aimed at improving semi-supervised and supervised learning through better regularization. Potentially, an adversarial loss can be added to NRM to improve visual characteristics of the image.

Related Work: In addition to GANs, other recently developed deep generative models include the Variational Autoencoders (VAE) (Kingma & Welling, 2013) and the Deep Generative Networks (Kingma et al., 2014) . Unlike these models, which replace complicated or intractable inference by CNNs, NRM derives CNNs as its inference. This advantage allows us to develop better learning algorithms for CNNs with statistical guarantees, as being discussed in Section 2.3 and 2.4. Recent works including the Bidirectional GANs (Donahue et al., 2017) and the Adversarially Learned Inference model (Dumoulin et al., 2017) try to make the discriminators and generators in GANs reversible of each other, thereby providing an alternative way to invert CNNs. These approaches, nevertheless, still employ a separate network to bypass the irreversible operators in CNNs. NRM is also close in spirit to the Deep Rendering Model (DRM) (Patel et al., 2016) and the Multi-layered Convolutional-Sparse-Coding Model (ML-CSC) (Papyan et al., 2018) but markedly different. Compared to NRM, DRM and ML-CSC have several limitations. In particular, latent variables in DRM are assumed to be independent, which is rather unrealistic. This lack of dependency causes the missing of the bias terms in the ReLUs of the CNN derived from DRM. Furthermore, the cross-entropy loss used in training CNNs for supervised learning tasks is not captured naturally by DRM and ML-CSC. Due to these limitations, model consistency and generalization bounds are not derived for DRM and ML-CSC.

Notation: To facilitate the presentation, the NRM's notations are deferred to 

EQUATION

⇥ < l a t e x i t s h a 1 _ b a s e 6 4 = " Z s h Q R D C e w T B C 9 o s b q P v t t Q W W N L 4 = " > A A A B 7 X i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N n J b D J m d m a Z 6 R X C k n / w 4 k E R r / 6 P N / / G S b I H T S x o K K q 6 6 e 6 K U i k s + v 6 3 t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b V m e G 8 S b T U p t O R C 2 X Q v E m C p S 8 k x p O k 0 j y d j S + n f n t J 2 6 s 0 O o B J y k P E z p U I h a M o p N a P R Q J t / 1 K 1 a / 5 c 5 B V E h S k C g U a / c p X b 6 B Z l n C F T F J r u 4 G f Y p h T g 4 J J P i 3 3 M s t T y s Z 0 y L u O K u q W h P n 8 2 i k 5 d 8 q A x N q 4 U k j m 6 u +

J n C b W T p L I d S Y U R 3 b Z m 4 n / e d 0 M 4 5 s w F y r N k C u 2 W B R n k q A m s 9 f J Q B j O U E 4 c o c w I d y t h I 2 o o Q x d Q 2 Y U Q L L + 8 S l q X t c C v B f d X 1 X q n i K M E p 3 A G F x D A N d T h D h r Q B A a P 8

A y v 8 O Z p 7 8 V 7 9 z 4 W r W t e M X M C f + B 9 / g C 8 y Y 9 K < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z s ⇥ < l a t e x i t s h a 1 _ b a s e 6 4 = " Z s h Q R D C e w T B C 9 o s b q P v t t Q W W N L 4 = " > A A A B 7 X i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N n J b D J m d m a Z 6 R X C k n / w 4 k E R r / 6 P N / / G S b I H T S x o K K q 6 6 e 6 K U i k s + v 6 3 t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b V m e G 8 S b T U p t O R C 2 X Q v E m C p S 8 k x p O k 0 j y d j S + n f n t J 2 6 s 0 O o B J y k P E z p U I h a M o p N a P R Q J t / 1 K 1 a / 5 c 5 B V E h S k C g U a / c p X b 6 B Z l n C F T F J r u 4 G f Y p h T g 4 J J P i 3 3 M s t T y s Z 0 y L u O K u q W h P n 8 2 i k 5 d 8 q A x N q 4 U k j m 6 u + J n C b W T p L I d S Y U R 3 b Z m 4 n / e d 0 M 4 5 s w F y r N k C u 2 W B R n k q A m s 9 f J Q B j O U E 4 c o c w I d y t h I 2 o o Q x d Q 2 Y U Q L L + 8 S l q X t c C v B f d X 1 X q n i K M E p 3 A G F x D A N d T h D h r Q B A a P 8 A y v 8 O Z p 7 8 V 7 9 z 4 W r W t e M X M C f + B 9 / g C 8 y Y 9 K < / l a t e x i t > Original images are in the rightmost column. Early layers in the rendering such as layer 4 and 3 capture coarse-scale features of the image while later layers such as layer 2 and 1 capture finer-scale features. From layer 2, we begin to see the gist of the rendered digits.

h Q R D C e w T B C 9 o s b q P v t t Q W W N L 4 = " > A A A B 7 X i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N n J b D J m d m a Z 6 R X C k n / w 4 k E R r / 6 P N / / G S b I H T S x o K K q 6 6 e 6 K U i k s + v 6 3 t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b V m e G 8 S b T U p t O R C 2 X Q v E m C p S 8 k x p O k 0 j y d j S + n f n t J 2 6 s 0 O o B J y k P E z p U I h a M o p N a P R Q J t / 1 K 1 a / 5 c 5 B V E h S k C g U a / c p X b 6 B Z l n C F T F J r u 4 G f Y p h T g 4 J J P i 3 3 M s t T y s Z 0 y L u O K u q W h P n 8 2 i k 5 d 8 q A x N q 4 U k j m 6 u + J n C b W T p L I d S Y U R 3 b Z m 4 n / e d 0 M 4 5 s w F y r N k C u 2 W B R n k q A m s 9 f J Q B j O U E 4 c o c w I d y t h I 2 o o Q x d Q 2 Y U Q L L + 8 S l q X t c C v B f d X 1 X q n i K M E p 3 A G F x D A N d T h D h r Q B A a P 8 A y v 8 O Z p 7 8 V 7 9 z 4 W r W t e M X M C f + B 9 / g C 8 y Y 9 K < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z s h Q R D C e w T B C 9 o s b q P v t t Q W W N L 4 = " > A A A B 7 X i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N n J b D J m d m a Z 6 R X C k n / w 4 k E R r / 6 P N / / G S b I H T S x o K K q 6 6 e 6 K U i k s + v 6 3 t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b V m e G 8 S b T U p t O R C 2 X Q v E m C p S 8 k x p O k 0 j y d j S + n f n t J 2 6 s 0 O o B J y k P E z p U I h a M o p N a P R Q J t / 1 K 1 a / 5 c 5 B V E h S k C g U a / c p X b 6 B Z l n C F T F J r u 4 G f Y p h T g 4 J J P i 3 3 M s t T y s Z 0 y L u O K u q W h P n 8 2 i k 5 d 8 q A x N q 4 U k j m 6 u + J n C b W T p L I d S Y U R 3 b Z m 4 n / e d 0 M 4 5 s w F y r N k C u 2 W B R n k q A m s 9 f J Q B j O U E 4 c o c w I d y t h I 2 o o Q x d Q 2 Y U Q L L + 8 S l q X t c C v B f d X 1 X q n i K M E p 3 A G F x D A N d T h D h r Q B A a P 8 A y v 8 O Z p 7 8 V 7 9 z 4 W r W t e M X M C f + B 9 / g C 8 y Y 9 K < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z s h Q R D C e w T B C 9 o s b q P v t t Q W W N L 4 = " > A A A B 7 X i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N n J b D J m d m a Z 6 R X C k n / w 4 k E R r / 6 P N / / G S b I H T S x o K K q 6 6 e 6 K U i k s + v 6 3 t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b V m e G 8 S b T U p t O R C 2 X Q v E m C p S 8 k x p O k 0 j y d j S + n f n t J 2 6 s 0 O o B J y k P E z p U I h a M o p N a P R Q J t / 1 K 1 a / 5 c 5 B V E h S k C g U a / c p X b 6 B Z l n C F T F J r u 4 G f Y p h T g 4 J J P i 3 3 M s t T y s Z 0 y L u O K u q W h P n 8 2 i k 5 d 8 q A x N q 4 U k j m 6 u + J n C b W T p L I d S Y U R 3 b Z m 4 n / e d 0 M 4 5 s w F y r N k C u 2 W B R n k q A m s 9 f J Q B j O U E 4 c o c w I d y t h I 2 o o Q x d Q 2 Y U Q L L + 8 S l q X t c C v B f d X 1 X q n i K M E p 3 A G F x D A N d T h D h r Q B A

U f h v 6 v l Z Z o x X x l o = " > A A A B 8 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X Y B E 8 l U Q E P R a 8 e K x g P 6 A J Z b P Z t E s 3 u 2 F 3 U i i h P 8 O L B 0 W 8 + m u 8 + W / c t j l o 6 4 O B x 3 s z z M y L M s E N e t 6 3 s 7 G 5 t b 2 z W 9 m r 7 h 8 c H h 3 X T k 4 7 R u W a s j Z V Q u l e R A w T X L I 2 c h S s l 2 l G 0 k i w b j S + n / v d C d O G K / m E 0 4 y F K R l K n n B K 0 E r 9 I E q K Y B I r N L N B r e 4 1 v A X c d e K X p A 4 l W o P a V x A r m q d M I h X E m L 7 v Z R g W R C O n g s 2 q Q W 5 Y R u i Y D F n f U k l S Z s J i c f L M v b R K 7 C Z K 2 5 L o L t T f E w V J j Z m m k e 1 M C Y 7 M q j c X / / P 6 O S Z 3 Y c F l l i O T d L k o y Y W L y p 3 / 7 8 Z c M 4 p i a g m h m t t b X T o i m l C 0 K V V t C P 7 q y + u k c 9 3 w v Y b / e F N v 9 s o 4 K n A O F 3 A F P t x C E x 6 g B W 2 g o O A Z X u H N Q e f F e X c + l q 0 b T j l z B n / g f P 4 A z Z a R p g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c a 4 A n 8 D 0 S Y 3 s U f h v 6 v l Z Z o x X x l o = " > A A A B 8 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X Y B E 8 l U Q E P R a 8 e K x g P 6 A J Z b P Z t E s 3 u 2 F 3 U i i h P 8 O L B 0 W 8 + m u 8 + W / c t j l o 6 4 O B x 3 s z z M y L M s E N e t 6 3 s 7 G 5 t b 2 z W 9 m r 7 h 8 c H h 3 X T k 4 7 R u W a s j Z V Q u l e R A w T X L I 2 c h S s l 2 l G 0 k i w b j S + n / v d C d O G K / m E 0 4 y F K R l K n n B K 0 E r 9 I E q K Y B I r N L N B r e 4 1 v A X c d e K X p A 4 l W o P a V x A r m q d M I h X E m L 7 v Z R g W R C O n g s 2 q Q W 5 Y R u i Y D F n f U k l S Z s J i c f L M v b R K 7 C Z K 2 5 L o L t T f E w V J j Z m m k e 1 M C Y 7 M q j c X / / P 6 O S Z 3 Y c F l l i O T d L k o y Y W L y p 3 / 7 8 Z c M 4 p i a g m h m t t b X T o i m l C 0 K V V t C P 7 q y + u k c 9 3 w v Y b / e F N v 9 s o 4 K n A O F 3 A F P t x C E x 6 g B W 2 g o O A Z X u H N Q e f F e X c + l q 0 b T j l z B n / g f P 4 A z Z a R p g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c a 4 A n 8 D 0 S Y 3 s U f h v 6 v l Z Z o x X x l o = " > A A A B 8 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X Y B E 8 l U Q E P R a 8 e K x g P 6 A J Z b P Z t E s 3 u 2 F 3 U i i h P 8 O L B 0 W 8 + m u 8 + W / c t j l o 6 4 O B x 3 s z z M y L M s E N e t 6 3 s 7 G 5 t b 2 z W 9 m r 7 h 8 c H h 3 X T k 4 7 R u W a s j Z V Q u l e R A w T X L I 2 c h S s l 2 l G 0 k i w b j S + n / v d C d O G K / m E 0 4 y F K R l K n n B K 0 E r 9 I E q K Y B I r N L N B r e 4 1 v A X c d e K X p A 4 l W o P a V x A r m q d M I h X E m L 7 v Z R g W R C O n g s 2 q Q W 5 Y R u i Y D F n f U k l S Z s J i c f L M v b R K 7 C Z K 2 5 L o L t T f E w V J j Z m m k e 1 M C Y 7 M q j c X / / P 6 O S Z 3 Y c F l l i O T d L k o y Y W L y p 3 / 7 8 Z c M 4 p i a g m h m t t b X T o i m l C 0 K V V t C P 7 q y + u k c 9 3 w v Y b / e F N v 9 s o 4 K n A O F 3 A F P t x C E x 6 g B W 2 g o O A Z X u H N Q e f F e X c + l q 0 b T j l z B n / g f P 4 A z Z a R p g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c a 4 A n 8 D 0 S Y 3 s U f h v 6 v l Z Z o x X x l o = " > A A A B 8 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X Y B E 8 l U Q E P R a 8 e K x g P 6 A J Z b P Z t E s 3 u 2 F 3 U i i h P 8 O L B 0 W 8 + m u 8 + W / c t j l o 6 4 O B x 3 s z z M y L M s E N e t 6 3 s 7 G 5 t b 2 z W 9 m r 7 h 8 c H h 3 X T k 4 7 R u W a s j Z V Q u l e R A w T X L I 2 c h S s l 2 l G 0 k i w b j S + n / v d C d O G K / m E 0 4 y F K R l K n n B K 0 E r 9 I E q K Y B I r N L N B r e 4 1 v A X c d e K X p A 4 l W o P a V x A r m q d M I h X E m L 7 v Z R g W R C O n g s 2 q Q W 5 Y R u i Y D F n f U k l S Z s J i c f L M v b R K 7 C Z K 2 5 L o L t T f E w V J j Z m m k e 1 M C Y 7 M q j c X / / P 6 O S Z 3 Y c F l l i O T d L k o y Y W L y p 3 / 7 8 Z c M 4 p i a g m h m t t b X T o i m l C 0 K V V t C P 7 q y + u k c 9 3 w v Y b / e F N v 9 s o 4 K n A O F 3 A F P t x C E x 6 g B W 2 g o O A Z X u H N Q e f F e X c + l q 0 b T j l z B n / g f P 4 A z Z a R p g = = < / l a t e x i t > Rendering Noise . . . < l a t e x i t s h a 1 _ b a s e 6 4 = " c a 4 A n 8 D 0 S Y 3 s U f h v 6 v l Z Z o x X x l o = " > A A A B 8 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X Y B E 8 l U Q E P R a 8 e K x g P 6 A J Z b P Z t E s 3 u 2 F 3 U i i h P 8 O L B 0 W 8 + m u 8 + W / c t j l o 6 4 O B x 3 s z z M y L M s E N e t 6 3 s 7 G 5 t b 2 z W 9 m r 7 h 8 c H h 3 X T k 4 7 R u W a s j Z V Q u l e R A w T X L I 2 c h S s l 2 l G 0 k i w b j S + n / v d C d O G K / m E 0 4 y F K R l K n n B K 0 E r 9 I E q K Y B I r N L N B r e 4 1 v A X c d e K X p A 4 l W o P a V x A r m q d M I h X E m L 7 v Z R g W R C O n g s 2 q Q W 5 Y R u i Y D F n f U k l S Z s J i c f L M v b R K 7 C Z K 2 5 L o L t T f E w V J j Z m m k e 1 M C Y 7 M q j c X / / P 6 O S Z 3 Y c F l l i O T d L k o y Y W L y p 3 / 7 8 Z c M 4 p i a g m h m t t b X T o i m l C 0 K V V t C P 7 q y + u k c 9 3 w v Y b / e F N v 9 s o 4 K n A O F 3 A F P t x C E x 6 g B W 2 g o O A Z X u H N Q e f F e X c + l q 0 b T j l z B n / g f P 4 A z Z a R p g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c a 4 A n 8 D 0 S Y 3 s U f h v 6 v l Z Z o x X x l o = " > A A A B 8 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X Y B E 8 l U Q E P R a 8 e K x g P 6 A J Z b P Z t E s 3 u 2 F 3 U i i h P 8 O L B 0 W 8 + m u 8 + W / c t j l o 6 4 O B x 3 s z z M y L M s E N e t 6 3 s 7 G 5 t b 2 z W 9 m r 7 h 8 c H h 3 X T k 4 7 R u W a s j Z V Q u l e R A w T X L I 2 c h S s l 2 l G 0 k i w b j S + n / v d C d O G K / m E 0 4 y F K R l K n n B K 0 E r 9 I E q K Y B I r N L N B r e 4 1 v A X c d e K X p A 4 l W o P a V x A r m q d M I h X E m L 7 v Z R g W R C O n g s 2 q Q W 5 Y R u i Y D F n f U k l S Z s J i c f L M v b R K 7 C Z K 2 5 L o L t T f E w V J j Z m m k e 1 M C Y 7 M q j c X / / P 6 O S Z 3 Y c F l l i O T d L k o y Y W L y p 3 / 7 8 Z c M 4 p i a g m h m t t b X T o i m l C 0 K V V t C P 7 q y + u k c 9 3 w v Y b / e F N v 9 s o 4 K n A O F 3 A F P t x C E x 6 g B W 2 g o O A Z X u H N Q e f F e X c + l q 0 b T j l z B n / g f P 4 A z Z a R p g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c a 4 A n 8 D 0 S Y 3 s U f h v 6 v l Z Z o x X x l o = " > A A A B 8 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X Y B E 8 l U Q E P R a 8 e K x g P 6 A J Z b P Z t E s 3 u 2 F 3 U i i h P 8 O L B 0 W 8 + m u 8 + W / c t j l o 6 4 O B x 3 s z z M y L M s E N e t 6 3 s 7 G 5 t b 2 z W 9 m r 7 h 8 c H h 3 X T k 4 7 R u W a s j Z V Q u l e R A w T X L I 2 c h S s l 2 l G 0 k i w b j S + n / v d C d O G K / m E 0 4 y F K R l K n n B K 0 E r 9 I E q K Y B I r N L N B r e 4 1 v A X c d e K X p A 4 l W o P a V x A r m q d M I h X E m L 7 v Z R g W R C O n g s 2 q Q W 5 Y R u i Y D F n f U k l S Z s J i c f L M v b R K 7 C Z K 2 5 L o L t T f E w V J j Z m m k e 1 M C Y 7 M q j c X / / P 6 O S Z 3 Y c F l l i O T d L k o y Y W L y p 3 / 7 8 Z c M 4 p i a g m h m t t b X T o i m l C 0 K V V t C P 7 q y + u k c 9 3 w v Y b / e F N v 9 s o 4 K n A O F 3 A F P t x C E x 6 g B W 2 g o O A Z X u H N Q e f F e X c + l q 0 b T j l z B n / g f P 4 A z Z a R p g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c a 4 A n 8 D 0 S Y 3 s U f h v 6 v l Z Z o x X x l o = " > A A A B 8 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X Y B E 8 l U Q E P R a 8 e K x g P 6 A J Z b P Z t E s 3 u 2 F 3 U i i h P 8 O L B 0 W 8 + m u 8 + W / c t j l o 6 4 O B x 3 s z z M y L M s E N e t 6 3 s 7 G 5 t b 2 z W 9 m r 7 h 8 c H h 3 X T k 4 7 R u W a s j Z V Q u l e R A w T X L I 2 c h S s l 2 l G 0 k i w b j S + n / v d C d O G K / m E 0 4 y F K R l K n n B K 0 E r 9 I E q K Y B I r N L N B r e 4 1 v A X c d e K X p A 4 l W o P a V x A r m q d M I h X E m L 7 v Z R g W R C O n g s 2 q Q W 5 Y R u i Y D F n f U k l S Z s J i c f L M v b R K 7 C Z K 2 5 L o L t T f E w V J j Z m m k e 1 M C Y 7 M q j c

S m G W 8 x J Z X u h t R w K R L e Q o G S d 1 P N a R x K 3 g n H d z O / 8 8 S 1 E S p 5 x E n K g 5 g O E x E J R t F K / q j W 4 1 K S S + J d 9 C t V t + 7 O Q V a J V 5 A q F G j 2 K 1 + 9 g W J Z z B N k k h r j e 2 6 K Q U 4 1 C i b 5 t N z L D E 8 p G 9 M h 9 y 1 N a M x N k M 9 P n p J z q w x I p L S t B M l c / T 2 R 0 9 i Y S R z a z p j i y C x 7 M / E / z 8 8 w u g 1 y k a Q Z 8 o Q t F k W Z J K j I 7 H 8 y E J o z l B N L K N P C 3 k r Y i G r K 0 K Z U t i F 4 y y + v k v Z V 3 X P r 3 s N 1 t d E t 4 i j B K Z x B D T y 4 g Q b c Q x N

S m G W 8 x J Z X u h t R w K R L e Q o G S d 1 P N a R x K 3 g n H d z O / 8 8 S 1 E S p 5 x E n K g 5 g O E x E J R t F K / q j W 4 1 K S S + J d 9 C t V t + 7 O Q V a J V 5 A q F G j 2 K 1 + 9 g W J Z z B N k k h r j e 2 6 K Q U 4 1 C i b 5 t N z L D E 8 p G 9 M h 9 y 1 N a M x N k M 9 P n p J z q w x I p L S t B M l c / T 2 R 0 9 i Y S R z a z p j i y C x 7 M / E / z 8 8 w u g 1 y k a Q Z 8 o Q t F k W Z J K j I 7 H 8 y E J o z l B N L K N P C 3 k r Y i G r K 0 K Z U t i F 4 y y + v k v Z V 3 X P r 3 s N 1 t d E t 4 i j B K Z x B D T y 4 g Q b c Q x N

S m G W 8 x J Z X u h t R w K R L e Q o G S d 1 P N a R x K 3 g n H d z O / 8 8 S 1 E S p 5 x E n K g 5 g O E x E J R t F K / q j W 4 1 K S S + J d 9 C t V t + 7 O Q V a J V 5 A q F G j 2 K 1 + 9 g W J Z z B N k k h r j e 2 6 K Q U 4 1 C i b 5 t N z L D E 8 p G 9 M h 9 y 1 N a M x N k M 9 P n p J z q w x I p L S t B M l c / T 2 R 0 9 i Y S R z a z p j i y C x 7 M / E / z 8 8 w u g 1 y k a Q Z 8 o Q t F k W Z J K j I 7 H 8 y E J o z l B N L K N P C 3 k r Y i G r K 0 K Z U t i F 4 y y + v k v Z V 3 X P r 3 s N 1 t d E t 4 i j B K Z x B D T y 4 g Q b c Q x N

S m G W 8 x J Z X u h t R w K R L e Q o G S d 1 P N a R x K 3 g n H d z O / 8 8 S 1 E S p 5 x E n K g 5 g O E x E J R t F K / q j W 4 1 K S S + J d 9 C t V t + 7 O Q V a J V 5 A q F G j 2 K 1 + 9 g W J Z z B N k k h r j e 2 6 K Q U 4 1 C i b 5 t N z L D E 8 p G 9 M h 9 y 1 N a M x N k M 9 P n p J z q w x I p L S t B M l c / T 2 R 0 9 i Y S R z a z p j i y C x 7 M / E / z 8 8 w u g 1 y k a Q Z 8 o Q t F k W Z J K j I 7 H 8 y E J o z l B N L K N P C 3 k r Y i G r K 0 K Z U t i F 4 y y + v k v Z V 3 X P r 3 s N 1 t d E t 4 i j B K Z x B D T y 4 g Q b c Q x N

Q G E y g Q L k 4 M p T O N X Y f p V x G V f R f 8 = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e C F

S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 7 d z v P K L S P J b 3 Z p q g H 9 G R 5 C F n 1 F i p + T Q o V 9 y q u w B Z J 1 5 O K p C j M S h / 9 Y c x S y O U h g m q d c 9 z E + N n V B n O B M 5 K / V R j Q t m E j r B n q a Q R a j 9 b H D o j F 1 Y Z k j B W t q Q h C / X 3 R E Y j r a d R Y D s j a s Z 6 1 Z u L / 3 m 9 1 I Q 3 f s Z l k h q U b L k o T A U x M Z l / T Y Z c I T N i a g l l i t t b C R t T R Z m x 2 Z R s C N 7 q y + u k f V X 1 3 K r X v K 7 U u 3 k c R T i D c 7 g E D 2 p Q h z t o Q A s Y I D z

Q G E y g Q L k 4 M p T O N X Y f p V x G V f R f 8 = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e C F

S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 7 d z v P K L S P J b 3 Z p q g H 9 G R 5 C F n 1 F i p + T Q o V 9 y q u w B Z J 1 5 O K p C j M S h / 9 Y c x S y O U h g m q d c 9 z E + N n V B n O B M 5 K / V R j Q t m E j r B n q a Q R a j 9 b H D o j F 1 Y Z k j B W t q Q h C / X 3 R E Y j r a d R Y D s j a s Z 6 1 Z u L / 3 m 9 1 I Q 3 f s Z l k h q U b L k o T A U x M Z l / T Y Z c I T N i a g l l i t t b C R t T R Z m x 2 Z R s C N 7 q y + u k f V X 1 3 K r X v K 7 U u 3 k c R T i D c 7 g E D 2 p Q h z t o Q A s Y I D z

Q G E y g Q L k 4 M p T O N X Y f p V x G V f R f 8 = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e C F

S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 7 d z v P K L S P J b 3 Z p q g H 9 G R 5 C F n 1 F i p + T Q o V 9 y q u w B Z J 1 5 O K p C j M S h / 9 Y c x S y O U h g m q d c 9 z E + N n V B n O B M 5 K / V R j Q t m E j r B n q a Q R a j 9 b H D o j F 1 Y Z k j B W t q Q h C / X 3 R E Y j r a d R Y D s j a s Z 6 1 Z u L / 3 m 9 1 I Q 3 f s Z l k h q U b L k o T A U x M Z l / T Y Z c I T N i a g l l i t t b C R t T R Z m x 2 Z R s C N 7 q y + u k f V X 1 3 K r X v K 7 U u 3 k c R T i D c 7 g E D 2 p Q h z t o Q A s Y I D z

Q G E y g Q L k 4 M p T O N X Y f p V x G V f R f 8 = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e C F

S q G 1 C N g k t s G W 4 E d h O F N A o E d o L J 7 d z v P K L S P J b 3 Z p q g H 9 G R 5 C F n 1 F i p + T Q o V 9 y q u w B Z J 1 5 O K p C j M S h / 9 Y c x S y O U h g m q d c 9 z E + N n V B n O B M 5 K / V R j Q t m E j r B n q a Q R a j 9 b H D o j F 1 Y Z k j B W t q Q h C / X 3 R E Y j r a d R Y D s j a s Z 6 1 Z u L / 3 m 9 1 I Q 3 f s Z l k h q U b L k o T A U x M Z l / T Y Z c I T N i a g l l i t t b C R t T R Z m x 2 Z R s C N 7 q y + u k f V X 1 3 K r X v K 7 U u 3 k c R T i D c 7 g E D 2 p Q h z t o Q A s Y I D z

E p U A H L Q = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h X k o i g h 4 L X j x W s B / Q h r L Z T t q l m 0 3 Y 3 Q g l 9 E d 4 8 a C I V 3 + P N / + N 2 z Q H b X 0 w 8 H h v h p l 5 Q S K 4 N q 7 7 7 Z Q 2 N r e 2 d 8 q 7 l b 3 9 g 8 O j 6 v F J R 8 e p Y t h m s Y h V L 6 A a B Z f Y N t w I 7 C U K a R Q I 7 A b T u 4 X f f U K l e S w f z S x B P 6 J j y U P O q L F S d 1 I f o B C X w 2 r N b b g 5 y D r x C l K D A q 1 h 9 W s w i l k a o T R M U K 3 7 n p s Y P 6 P K c C Z w X h m k G h P K p n S M f U s l j V D 7 W X 7 u n F x Y Z U T C W N m S h u T q 7 4 m M R l r P o s B 2 R t R M 9 K q 3 E P / z + q k J b / 2 M y y Q 1 K N l y U Z g K Y m K y + J 2 M u E J m x M w S y h S 3 t x I 2 o Y o y Y x O q 2 B C 8 1 Z f X S e e q 4 b k N 7 + G 6 1 u w V c Z T h D M 6 h D h 7 c Q B P u o Q V t Y D C F Z 3 i F N y d x X p x 3 5 2 P Z W n K K m V P 4 A + f z B 6 A M j y g = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e / C d V 7 X s M f 2 X h + 8 9 w r g o E p U A H L Q = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h X k o i g h 4 L X j x W s B / Q h r L Z T t q l m 0 3 Y 3 Q g l 9 E d 4 8 a C I V 3 + P N / + N 2 z Q H b X 0 w 8 H h v h p l 5 Q S K 4 N q 7 7 7 Z Q 2 N r e 2 d 8 q 7 l b 3 9 g 8 O j 6 v F J R 8 e p Y t h m s Y h V L 6 A a B Z f Y N t w I 7 C U K a R Q I 7 A b T u 4 X f f U K l e S w f z S x B P 6 J j y U P O q L F S d 1 I f o B C X w 2 r N b b g 5 y D r x C l K D A q 1 h 9 W s w i l k a o T R M U K 3 7 n p s Y P 6 P K c C Z w X h m k G h P K p n S M f U s l j V D 7 W X 7 u n F x Y Z U T C W N m S h u T q 7 4 m M R l r P o s B 2 R t R M 9 K q 3 E P / z + q k J b / 2 M y y Q 1 K N l y U Z g K Y m K y + J 2 M u E J m x M w S y h S 3 t x I 2 o Y o y Y x O q 2 B C 8 1 Z f X S e e q 4 b k N 7 + G 6 1 u w V c Z T h D M 6 h D h 7 c Q B P u o Q V t Y D C F Z 3 i F N y d x X p x 3 5 2 P Z W n K K m V P 4 A + f z B 6 A M j y g = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e / C d V 7 X s M f 2 X h + 8 9 w r g o E p U A H L Q = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h X k o i g h 4 L X j x W s B / Q h r L Z T t q l m 0 3 Y 3 Q g l 9 E d 4 8 a C I V 3 + P N / + N 2 z Q H b X 0 w 8 H h v h p l 5 Q S K 4 N q 7 7 7 Z Q 2 N r e 2 d 8 q 7 l b 3 9 g 8 O j 6 v F J R 8 e p Y t h m s Y h V L 6 A a B Z f Y N t w I 7 C U K a R Q I 7 A b T u 4 X f f U K l e S w f z S x B P 6 J j y U P O q L F S d 1 I f o B C X w 2 r N b b g 5 y D r x C l K D A q 1 h 9 W s w i l k a o T R M U K 3 7 n p s Y P 6 P K c C Z w X h m k G h P K p n S M f U s l j V D 7 W X 7 u n F x Y Z U T C W N m S h u T q 7 4 m M R l r P o s B 2 R t R M 9 K q 3 E P / z + q k J b / 2 M y y Q 1 K N l y U Z g K Y m K y + J 2 M u E J m x M w S y h S 3 t x I 2 o Y o y Y x O q 2 B C 8 1 Z f X S e e q 4 b k N 7 + G 6 1 u w V c Z T h D M 6 h D h 7 c Q B P u o Q V t Y D C F Z 3 i F N y d x X p x 3 5 2 P Z W n K K m V P 4 A + f z B 6 A M j y g = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e / C d V 7 X s M f 2 X h + 8 9 w r g o E p U A H L Q = " > A A A B 7 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h X k o i g h 4 L X j x W s B / Q h r L Z T t q l m 0 3 Y 3 Q g l 9 E d 4 8 a C I V 3 + P N / + N 2 z Q H b X 0 w 8 H h v h p l 5 Q S K 4 N q 7 7 7 Z Q 2 N r e 2 d 8 q 7 l b 3 9 g 8 O j 6 v F J R 8 e p Y t h m s Y h V L 6 A a B Z f Y N t w I 7 C U K a R Q I 7 A b T u 4 X f f U K l e S w f z S x B P 6 J j y U P O q L F S d 1 I f o B C X w 2 r N b b g 5 y D r x C l K D A q 1 h 9 W s w i l k a o T R M U K 3 7 n p s Y P 6 P K c C Z w X h m k G h P K p n S M f U s l j V D 7 W X 7 u n F x Y Z U T C W N m S h u T q 7 4 m M R l r P o s B 2 R t R M 9 K q 3 E P / z + q k J b / 2 M y y Q 1 K N l y U Z g K Y m K y + J 2 M u E J m x M w S y h S 3 t x I 2 o Y o y Y x O q 2 B C 8 1 Z f X S e e q 4 b k N 7 + G 6 1 u w V c Z T h D M 6 h D h 7 c Q B P u o Q V t Y D C F Z 3 i F N

i k r Z N M M f R Z I h L V D a l G w S X 6 h h u B 3 V Q h j U O B n X B y N / c 7 T 6 g 0 T + S D m a Y Y x H Q k e c Q Z N V Z q p Y N q z a 2 7 C 5 B 1 4 h W k B g W a g + p X f 5 i w L E Z p m K B a 9 z w 3 N U F O l e F M 4 K z S z z S m l E 3 o C H u W S h q j D v L F o T N y Y Z U h i R J l S x q y U H 9 P 5 D T W e h q H t j O m Z q x X v b n 4 n 9 f L T H Q b 5 F y m m U H J l o u i T B C T k P n X Z M g V M i O m l l C m u L 2 V s D F V l B m b T c W G 4 K 2 + v E 7 a V 3 X P r X u t 6 1 q j W 8 R

i k r Z N M M f R Z I h L V D a l G w S X 6 h h u B 3 V Q h j U O B n X B y N / c 7 T 6 g 0 T + S D m a Y Y x H Q k e c Q Z N V Z q p Y N q z a 2 7 C 5 B 1 4 h W k B g W a g + p X f 5 i w L E Z p m K B a 9 z w 3 N U F O l e F M 4 K z S z z S m l E 3 o C H u W S h q j D v L F o T N y Y Z U h i R J l S x q y U H 9 P 5 D T W e h q H t j O m Z q x X v b n 4 n 9 f L T H Q b 5 F y m m U H J l o u i T B C T k P n X Z M g V M i O m l l C m u L 2 V s D F V l B m b T c W G 4 K 2 + v E 7 a V 3 X P r X u t 6 1 q j W 8 R

i k r Z N M M f R Z I h L V D a l G w S X 6 h h u B 3 V Q h j U O B n X B y N / c 7 T 6 g 0 T + S D m a Y Y x H Q k e c Q Z N V Z q p Y N q z a 2 7 C 5 B 1 4 h W k B g W a g + p X f 5 i w L E Z p m K B a 9 z w 3 N U F O l e F M 4 K z S z z S m l E 3 o C H u W S h q j D v L F o T N y Y Z U h i R J l S x q y U H 9 P 5 D T W e h q H t j O m Z q x X v b n 4 n 9 f L T H Q b 5 F y m m U H J l o u i T B C T k P n X Z M g V M i O m l l C m u L 2 V s D F V l B m b T c W G 4 K 2 + v E 7 a V 3 X P r X u t 6 1 q j W 8 R

i k r Z N M M f R Z I h L V D a l G w S X 6 h h u B 3 V Q h j U O B n X B y N / c 7 T 6 g 0 T + S D m a Y Y x H Q k e c Q Z N V Z q p Y N q z a 2 7 C 5 B 1 4 h W k B g W a g + p X f 5 i w L E Z p m K B a 9 z w 3 N U F O l e F M 4 K z S z z S m l E 3 o C H u W S h q j D v L F o T N y Y Z U h i R J l S x q y U H 9 P 5 D T W e h q H t j O m Z q x X v b n 4 n 9 f L T H Q b 5 F y m m U H J l o u i T B C T k P n X Z M g V M i O m l l C m u L 2 V s D F V l B m b T c W G 4 K 2 + v E 7 a V 3 X P r X u t 6 1 q j W 8 R

o Q x d Q 2 Y U Q L L + 8 S l q X t c C v B f d X 1 X q n i K M E p 3 A G F x D A N d T h D h

o Q x d Q 2 Y U Q L L + 8 S l q X t c C v B f d X 1 X q n i K M E p 3 A G F x D A N d T h D h

o Q x d Q 2 Y U Q L L + 8 S l q X t c C v B f d X 1 X q n i K M E p 3 A G F x D A N d T h D h

1) Sample the latent variables z from a categorical distribution whose prior is π z|y .

2) Render its coarsest image, the object template h(L) = µ(y) of class y. 3) Incorporate a set of of latent variations z( ) into h(y; ) at each layer via a linear transformation Λ(z; ) to render the finer image h(y, z; -1). 4) Add Gaussian pixel noise into h(y, z; 0) to render the final image x. In Eqn.2, if Λ(z; ) is a linear transformation, it will yield many parameters to learn. As a result, we would like to introduce new structures into NRM so that CNNs can be derived as NRM's inference. This way, NRM will be informed by the prior knowledge of natural images captured by CNNs. In our attempt to invert CNNs, we constrain the latent variables z( ) at layer in NRM to a set of template selecting latent variables s( ) and local translation latent variables t( ). As been shown later in Section 2.2, during inference of NRM, the ReLU non-linearity at layer "inverts" s( ) to find if particular features are in the image or not. Similarly, the MaxPool operator "inverts" t( ) to locate where particular features, if exist, are in the image. Both s( ) and t( ) are vectors indexed by p( ) ∈ P( ) ≡ {pixels in level }. The rendering process from layer to layer -1 is given by: h( -1) Λ( )h( ) = p∈P( ) s( , p)T (t; , p)B( , p)Γ( , p)h( , p).

(3)

The rendering process in Eqn.3 is illustrated in Figure 3 . At each pixel p in the intermediate image h( ) at layer , NRM decides to use that pixel to render or not according to the value of the template selecting latent variable s(p; ) at that pixel location. If s(p; ) = 1, then NRM renders; otherwise, it does not. If rendering, the pixel value h( , p) is used to scale the local rendering template Γ( , p), which has the same number of feature maps as the next rendered image h( -1), but is of smaller size, e.g., 3 × 3 or 5 × 5. Next, the padding matrix n B( , p) pads the resultant patch to the size of the image h( -1) with zeros, and the translation matrix T (t; , p) translates the result to a local location. NRM then keeps rendering at the next pixel location p + 1 following the same process. All rendered images are added to form the final rendered image h( -1) at layer -1. Note that we can constrain NRM by enforcing all pixels in the same feature maps of h( ) share the same rendering template. This constraint helps to yield convolutions in CNNs during the inference of NRM, and the rendering template in NRM now corresponds to the convolution filters in CNNs. Reconstructed images at each layer of a 5-layered NRM trained on MNIST are visualized in Figure 2 . The network is trained using the semi-supervised learning framework discussed in Section 2.3. 

2.2. INFERENCE:

We show that calculations in the bottom-up inference in NRM corresponds to a CNN (see Figure 1b ) and, therefore, is tractable and efficient. The impact of this correspondence goes beyond a reverseengineering effort. First, it provides probabilistic semantics for components in CNNs, justifying their usage, and providing an opportunity to employ probabilistic inference methods in the context of CNNs. Second, such a correspondence offers a flexible framework to design CNNs. Instead of directly engineering CNNs for new tasks and datasets, we can modify NRM to incorporate our knowledge of the tasks and datasets into the model and perform JMAP inference to achieve a new CNN architecture. The following theorem establishes the NRM-CNN correspondence: Theorem 2.2. The JMAP inference of latent variable z in NRM is the feedforward step in CNNs.

Particularly, we have:

max z {p(z|x, y)} = max z 1 σ 2 { h(y, z; 0), x + η(y, z)} + const ≥ 1 σ 2 µ(y), ψ(L) + const (4) where ψ(L) is computed recursively. In particular, ψ(0) = x and ψ( ) = MaxPool(ReLu(Conv(Γ ( ), ψ( -1))) + b( )).

(5) The equality holds in Eqn. 4 when the parameters θ in NRM satisfy the non-negativity assumption that the intermediate rendered image h( ) ≥ 0, ∀ = 1, 2, . . . , L. There are four key results in Theorem 2.2. First, ReLU non-linearities in CNNs find the optimal value for the template selecting latent variables s( ) at each layer in NRM, detecting if particular features exist in the image or not. Second, MaxPool operators in CNNs find the optimal value for the local translation latent variables t( ) at each layer in NRM, locating where features are rendered in the image. Third, bias terms after each convolution in CNNs are from the prior knowledge of latent variables in the model. Those bias terms update the posterior estimation of latent variables from data using the knowledge encoded in the prior distribution of those latent variables. Fourth, convolutions in CNNs result from reversing the local rendering operator using template Γ( ) in NRM. Instead of rendering as in NRM, convolutions in CNNs perform template matching. The convolution weights W ( ) in CNNs are proportional to the transposes of the rendering templates Γ( ). The proofs for these correspondences and derivations for leaky ReLU and batch normalization are in Appendix C.

2.3. LEARNING:

Learning in NRM can be posed as likelihood estimation problems in which we find the optimal values for parameters in NRM to optimize the appropriate likelihood functions. The optimization can be done by gradient-based methods (Robbins & Monro, 1985) . The following theorem derives the learning objectives for NRM, and more details and proofs is provided in Appendix B and C. Theorem 2.3. For any n ≥ 1, let x 1 , . . . , x n be i.i.d. samples from the NRM. Assume that the final rendered template h(y, z; 0) is normalized such that its norm is constant. The following holds: (a) Cross-entropy loss for training CNNs with labeled data

max (zi) n i=1 ,θ 1 n n i=1 log p(y i |x i , z i ; θ) ≥ max θ 1 n n i=1 log q(y i |x i ) = -min θ∈Aγ H p,q (y|x) (6)

where q(y|x) is the posterior estimated by CNN, and H p,q (y|x) is the cross-entropy between q(y|x) and the true posterior p(y|x) given by the ground truth.

(b) Reconstruction loss with RPN for unsupervised training of CNNs with labeled and unlabeled data

min θ 1 n n i=1 E [log p(x i , z i |y i )] asymp ≈ min θ 1 n n i=1 x i -h(y i , z * i ; 0) 2 2 + RPN, when σ → 0 (7)

where the latent variable z * i is estimated by the CNN as described in Theorem 2.2, h(y i , z * i ; 0) is the reconstructed image, and the RPN regularization is the negative log prior defined as follows:

EQUATION

)

Cross-Entropy Loss for Training CNNs with Labeled Data: Theorem 2.3(a) establishes the cross-entropy loss in the context of CNNs as an upper bound of the NRM's negative conditional log-likelihood

L sup := -1 n n i=1 log p(y i |x i , z i ; θ).

In contrast to other derivations of cross-entropy loss via logistic regression, we derives the cross-entropy loss in conjunction with the architecture of CNNs since the estimation of the optimal latent variables z * is part of the optimization in Eqn. 6. In other word, Theorem 2.3(a) ties feature extraction and learning for classification in CNNs into an end-to-end conditional likelihood estimation problem in NRM. This new interpretation of the cross-entropy loss suggests an interesting direction in which better losses for training CNNs with labeled data for supervised classification tasks can be derived from other upper bounds of L sup . The Max-Min cross-entropy in Section 3 is an example.

Reconstruction Loss with RPN Regularization: Theorem 2.3(b) suggests that NRM learns without labels by maximizing its expected complete-data log-likelihood. One term in this objective function is the reconstruction loss between the input image x i and the reconstructed template h(y i , z * i ; 0). Another term is the RPN from Eqn. 8. RPN encourages the (y i , z * i ) inferred in the bottom-up E-step to have higher prior among all possible values of (y i , z i ) and, thanks to its structure, enforces the dependencies among latent variables (s( ), t( )) at different layers in NRM.

Semi-Supervised Learning in NRM: Let x 1 , . . . , x n be i.i.d. samples from NRM and assume that the labels y 1 , . . . , y n1 are unknown for some 0 ≤ n 1 ≤ n, NRM determines optimal parameters employed for the semi-supervised classification task via the following model:

EQUATION

where α RC and α CE are non-negative weights associated with the reconstruction loss with RPN regularization and the cross-entropy loss, respectively. Again, the optimal latent variables z * i is estimated by CNN as in Theorem 2.2. For unlabeled data, y i is set to the label estimated by CNN.

2.4. GENERALIZATION BOUND FOR CLASSIFICATION:

Our generalization bound for classification with NRM is proportional to the ratio of the number of active rendering paths and the total number of rendering paths in the trained NRM. A rendering path is a configuration of all latent variables in NRM, and active optimal rendering paths are those among optimal rendering paths ( y, z) whose corresponding rendered image is sufficiently close to one of the data point from the input data distribution. Let L A and L D denote the population and empirical losses on the data population A and the training set D of NRM, respectively. Our key result is summarized below. More details and proofs are deferred to Appendix B and C. Theorem 2.4. Under the margin-based loss, with high probability, the following result on the generalization bound of the classification framework with optimal solutions from Eqn. 9 holds:

L A ≤ L D + τ n |L|/ √ n.

Here, τ n ∈ (0, 1) denotes the ratio of active optimal rendering paths among all the optimal rendering paths, |L| is the total number of rendering paths, and n is the number of training data samples. The dependence of generalization bound on the number of active rendering paths τ n |L| helps to justify our modeling assumptions. In particular, NRM helps to reduce the number of active rendering paths thanks to the dependencies among its latent variables, thereby tightening the generalization gap. Nevertheless, there is a limitation regarding the current generalization bound. In particular, the bound involves the number of rendering paths |L|, which is usually large. This is mainly because our bound has not fully taken into account the structure of CNNs, which is the limitation shared among other latest generalization bounds for CNN. It is interesting to explore if techniques in works by Bartlett et al. (2017) and Golowich et al. (2018) can be employed to improve the term |L| in our bound.

3. NEW MAX-MIN CROSS ENTROPY FROM THE NEURAL RENDERING MODEL:

In this section, we explore a particular way to derive an alternative to cross-entropy inspired by the results in Theorem 2.3(a). In particular, denoting z max arg max z { h(y, z; 0), x + η(y, z)} and z min arg min z { h(y, z; 0), x + η(y, z)}, the new cross-entropy H M &M , which is called the Max-Min cross-entropy, is the weighted average of the cross-entropy losses from z max and z min :

H M&M α max H p,q (y|x, z max ) + α min H p,q (y|x, z min ) = α max H max p,q (y|x) + α min H min p,q (y|x). Here the Max cross-entropy H max p,q and Min cross entropy H min p,q maximizes the correct target posterior and and minimizes the incorrect target posterior, respectively. Similar to the cross-entropy loss, the Max-Min cross-entropy can also be shown to be an upper bound for the negative conditional log-likelihood L sup of the NRM and has the same generalization bound derived in Section 2.4. The Max-Min networks in Figure 4 realize this new loss. These networks have two CNN-like branches that share weights. The max branch estimates z max using ReLU and Max-Pooling, and the min branch estimates z min using the Negative ReLU, i.e., min(•, 0), and Min-Pooling. The Max-Min networks can be interpreted as a form of knowledge distillation like the Born Again networks (Furlanello et al., 2018) and the Mean Teacher networks. However, instead of a student network learning from a teacher network, in Max-Min networks, two students networks, the Max and the Min networks, cooperate and learn from each other during the training.

Input Image Max Xentropy Min Xentropy:



Max-Min Xentropy:



Shared weights:



4. EXPERIMENTS:



4.1. SEMI-SUPERVISED LEARNING:

We show NRM armed with Max-Min cross-entropy and Mean Teacher regularizer achieves SOTA on benchmark datasets. We discuss the experimental results for CIFAR10 and CIFAR100 here. The results for SVHN, training losses, and training details, can be found in the Appendix A & D.

CIFAR-10: Table 2 shows comparable results of NRM to SOTA methods. NRM is also better than the best methods that do not use consistency regularization like GAN, Ladder network, and ALI when using only N l =2K and 4K labeled images. NRM outperforms DRM in all settings. Also, among methods in our comparison, NRM achieves the best test accuracy when using all available labeled data (N l =50K). NRM has the advantage over consistency regularization methods like Temporal Ensembling and Mean Teacher when there are enough labeled data because the consistency regularization in those methods tries to match the activations in the network, but does not take into account the available class labels. On the contrary, NRM employs the class labels, if they are available, in its reconstruction loss and RPN regularization as in Eqns. 7 and 8. In all settings, RPN regularizer improves NRM performance. Even though the improvement from RPN is small, it is consistent across the experiments. Furthermore, using Max-Min cross-entropy significantly reduces the test errors. When combining with Mean-Teacher, our Max-Min NRM improves upon Mean-Teacher and consistently achieves either SOTA results or second best results in all settings. This consistency in performance is only observed in our method and Mean-Teacher. Also, like with Mean-Teacher, NRM can potentially be combined with other consistency regularization methods, e.g., the Virtual Adversarial Training (VAT) (Miyato et al., 2018) , to obtain better results.

CIFAR-100: Table 3 shows NRM's comparable results to Π model and Temporal Ensembling, as well as better results than DRM. Same as with CIFAR10, using the RPN regularizer results in a slightly better test accuracy, and NRM achieves better results than Π model and Temporal Ensembling method when using all available labeled data. Notice that combining with Mean-Teacher just slightly improves NRM's performance when training with 10K labeled data. This is again because consistency regularization methods like Mean-Teacher do not add much advantage when there are enough labeled data. However, NRM+Max-Min still yields better test errors and achieves SOTA result in all settings.

4.2. SUPERVISED LEARNING WITH MAX-MIN CROSS-ENTROPY:

The Max-Min cross-entropy can be applied not only to improve semi-supervised learning on deep models including CNNs but also to enhance their supervised learning performance. In our experiments, we indeed observe Max-Min cross-entropy reduces the test error for supervised object classification on CIFAR10. In particular, using the Max-Min cross-entropy loss on a 29-layer ResNet (Xie et al., 2017) trained with the Shake-Shake regularization (Gastaldi, 2017) and Cutout data augmentation (DeVries & Taylor, 2017), we are able to achieve SOTA test error of 2.30% on CIFAR10, an improvement of 0.26% over the test error of the baseline architecture trained with the traditional cross-entropy loss. While 0.26% improvement seems small, it is a meaningful enhancement given that our baseline architecture (ResNeXt + Shake-Shake + Cutout) is the second best model for supervised learning on CIFAR10. Such small improvement over an already very accurate model is significant in applications in which high accuracy is demanded such as self-driving cars or medical diagnostics.

Similarly, we observe Max-Min improves the top-5 test error of the Squeeze-and-Excitation ResNeXt-50 network (Hu et al., 2018) on ImageNet by 0.17% compared to the baseline (7.04% vs. 7.21%). For a fair comparison, we re-train the baseline models and report the scores in the re-implementation.

5. CONCLUSIONS:

We present the NRM, a general and an effective framework for semi-supervised learning that combines generation and prediction in an end-to-end optimization. Using NRM, we can explain operations used in CNNs and develop new features that help learning in CNNs. For example, we derive the new Max-Min cross-entropy loss for training CNNs, which outperforms the traditional cross-entropy. Despite promising results in this paper, NRM can still be improved. For instance, an adversarial loss like in GANs can be incorporated into NRM so that the model can generate realistic images. Furthermore, more knowledge of image generation from graphics and physics can be integrated in NRM so that the model can employ more structures to help learning and generation. 73257 images 500 labels 73257 images 1000 labels 73257 images 73257 labels 73257 images ALI Dumoulin et al. (2017) 7.42 ± 0.65 Improved GAN (Salimans et al., 2016) 18.44 ± 4.8 8.11 ± 1.3 + Jacob.-reg + Tangents (Kumar et al., 2017) 4.87 ± 1.60 4.39 ± 1.20 Π model (Laine & Aila, 2017) 9.69 ± 0.92 6.83 ± 0.66 4.95 ± 0.26 2.50 ± 0.07 Temporal Ensembling (Laine & Aila, 2017) 5.12 ± 0.13 4.42 ± 0.16 2.74 ± 0.06 Mean Teacher (Tarvainen & Valpola, 2017) 4.35 ± 0.50 4.18 ± 0.27 3.95 ± 0.19 2.50 ± 0.05 2.50 ± 0.05 2.50 ± 0.05 VAT+EntMin (Miyato et al., 2018) 3.86 DRM (Nguyen et al., 2016) 9 Output: Rendered image x given the object category y.

Parameters: θ = {µ(y)} K y=1 , {Γ( )} L =1 , {b( )} L =1

where µ(y) is the class template, Γ( ) is the rendering template at layer , and b( ) is the parameters of the conjugate prior p(z|y, x) at layer , which turn out to be the bias terms in the ReLU after convolutions at each layer in CNNs.

1. Use Markov chain Monte Carlo method to sample the latent variables z( ) in NRM, = 1, 2, . . . , L from π z|y = Softmax 1 σ 2 η(y, z) , where η(y, z)

L =1 b(t; ), s( ) h( ) .

2. Render h( ), = 0, 1, . . . , L -1 using the recursion h( -1) = p∈P( ) s( , p)T (t; , p)B( , p)Γ( , p)h( , p) in Eqn. 3, in which h(L) = µ(y) and T (t; , p) and B( , p) are the local translation matrix and the zero-padding matrix at pixel location p in layer as described above.

3. Add Gaussian pixel noise N (0, σ 2 1 D(0) ) into h(0) to achieve the final rendered image x, where D(0) is the dimension, i.e. the number of pixels, of h(0), as well as of x. 

) = h( ) intermediate rendered image of size D( ) in level h(y, z; 0) = h(0)

rendered image of size D(0) from NRM before adding noise ψ( ) corresponding feature maps in layer in CNNs.

Parameters

µ(y) = h(y; L) = h(L)

the template of class y, as well as the coarsest image of size D(L) determined by the category y at the top of NRM before adding any fine detail. µ(y) is learned from the data. Λ( ) rendering matrix of size D( -1) × D( ) at layer . Γ( ) dictionary of D( ) rendering template Γ( , p) of size F ( ) × 1 at layer . Γ( ) is learned from the data. W ( ) = Γ ( , p) corresponding weight at the layer in CNNs B( )

set of zero-padding matrices B( , p) ∈ R D( -1)×F ( ) ) at layer T ( ) set of local translation matrices T ( , p) ∈ R D( -1)×D( -1) ) at layer . T ( , p) is chosen according to value of t( , p) b(t; ) = b( )

parameter of the conjugate prior p(z|x, y) at layer . This term is of size D( ) and becomes the bias term after convolutions in CNNs. It can be made independent of t, which is equivalent to using the same bias in each feature map in CNNs. Here, b(t; ) is learned from data. π y probability of object category y. σ 2 pixel noise variance Other Notations η(y, z)

L =1 1 σ 2 b(t; ), s( ) h( ) . Softmax (η) exp(η) η exp(η ) . RPN -1 n n i=1 log p(z * i |y i ) = -1 n n i=1 Softmax (η(y i , z * i )). (y,z(L), . . . , z(1)) rendering configuration.

where

z i = arg max zi p(y i |x i , z i ; θ) for 1 ≤ i ≤ n.

Remark .2. As being demonstrated in Theorem 2.2, max z h (y, z; 0)x + η(y, z) /σ 2 , approximately, has the form of a CNN. If we further have the non-negativity assumption with θ, then this is exact. Therefore, the cross entropy H p,q obtained in Theorem .1 has a strong connection with CNN.

Remark .3. The gap between the upper bound and the lower bound of maximizing the conditional log likelihood in terms of cross entropy function suggests how good the estimation in Theorem .1 is.

In particular, this gap is given by:

1 n n i=1 max y max zi h (y, z i ; 0)x i + η(y, z i ) σ 2 - h (y, z i ; 0)x i + η(y, z i ) σ 2 trade-off loss + log K,

where

z i = arg max zi p(y i |x i , z i ; θ) for 1 ≤ i ≤ n.

As long as the number of labels is not too large and the trade-off loss is sufficiently small, the gap between the upper bound and the lower bound in Theorem .1 is small.

.

2. LEARNING IN THE NRM WITHOUT NOISE FOR UNSUPERVISED SETTING UNDER NON-NEGATIVITY ASSUMPTION:

To ease the presentation of the inference with NRM without noise, we first assume that rendered images h(y, z; 0) satisfy the non-negativity assumption (Later in Section .3, we will discuss the relaxation of this assumption for the inference of the NRM). With this assumption, as being demonstrated in Theorem 2.2, we have: max

EQUATION

where we define

ψ(L) = max z(L) Λ (z(L)) max z(L-1) Λ (z; L -1) • • • max z(1) Λ (z; 1)x + b(1) . . . + b(L -1)

+b( ). Now, we will provide careful derivation of part (b) of Theorem 2.3 in the main text. Remind that, for the unsupervised setting, we have data x 1 , . . . , x n are i.i.d. samples from NRM. The complete-data log-likelihood of the NRM is given as follows:

E y,z [log p(x, (y, z))] = n i=1 (y,z)∈J P (y, z|x i ){log π y,z + log N (x i |h(y, z; 0))}

where we have

P (y, z|x i ) = π y,z N (x i |h(y, z; 0)) (y ,z )∈J π y ,z N (x i |h(y , z ; 0)) = π y exp - x i -h(y, z; 0) 2 -2η(y, z) 2σ 2 (y ,z )∈J π y exp - x i -h(y , z ; 0) 2 -2η(y , z ) 2σ 2

. At the zero-noise limit, i.e., σ → 0, it is clear that P (y, z|x i ) = 1 as (y, z) = arg min (y ,z )∈J

x ih(y , z ; 0) 2 -2η(y , z ) and P (y, z|x i ) = 0 otherwise. Therefore, we can asymptotically view the complete log-likelihood of the NRM under the zero-noise limit as

n i=1 (y,z)∈J r y,z log π y,z - 1 2 x i -h(y, z; 0) 2 = n i=1 (y,z)∈J - 1 2 r y,z x n -h(y, z; 0) 2 Reconstruction Loss + n i=1 (y,z)∈J r y,z log π y,z

Path Normalization Regularizer where

r y,z ≡    1, if (y, z) = arg min (y ,z )∈J x i -h(y , z ; 0) 2 -2η(y , z ) 0, otherwise

With the above formulation, we have the following objective function

EQUATION

where

θ = {µ(y)} K y=1 , {Γ( )} L =1 , {π y } K y=1 , {b( )} L =1

. We call the above objective function to be unsupervised NRM without noise.

Relaxation of unsupervised NRM without noise Unfortunately, the inference with unsupervised NRM without noise is intractable in practice due to two elements: the involvement of h(y , z ; 0) 2 to determine the value of r y,z and the summation (y ,z )∈J exp(η(y , z )+log π y ) in the denominator of π y,z for all (y, z) ∈ J . Therefore, we need to develop a tractable version of this objective function. Theorem .4. (Relaxation of unsupervised NRM without noise) Assume that π y ≥ γ for all 1 ≤ y ≤ K for some given γ ∈ (0, 1/2). Denote

V n := min θ 1 n n i=1 (y ,z )∈J 1    (y ,z )=arg max (y,z)∈J h (y,z;0)xi+η(y,z)    x i -h(y , z ; 0) 2 2 -log(π y ,z )

where p y ,z = exp η(y , z ) + log π y / K y=1 exp max z∈L η(y, z) + log π y for all (y , z ) ∈ J .

For any θ, we define

(y i , z i ) = arg min (y,z)∈J x i -h(y, z; 0) 2 -2η(y, z)

and

( y i , z i ) = arg max (y,z)∈J h (y, z; 0)x i + η(y, z)

as 1 ≤ i ≤ n. Then, the following holds (a) Upper bound:

U n ≤ min θ 1 n n i=1 x i -h( y i , z i ; 0) 2 2 -log(π yi, zi ) + log π yi -log π y i prior loss + log |L| ≤ V n + log 1 γ -1 + log |L| (b) Lower bound: U n ≥ min θ 1 n n i=1 x i -h( y i , z i ; 0) 2 2 -log(π yi, zi ) + log π y i -log π yi prior loss + 1 2 h(y i , z i ; 0) 2 -h( y i , z i ; 0) 2 norm loss ≥ V n + log γ 1 -γ + min θ 1 n n i=1 1 2 h(y i , z i ; 0) 2 -h( y i , z i ; 0) 2

Unlike U n , the inference with objective function of V n is tractable. According to the upper bound and lower bound of U n in terms of V n , we can use V n as a tractable approximation of U n for the inference purpose with unsupervised setting of data when the noise is treated to be 0. Therefore, we achieve the conclusion of part (b) of Theorem 2.3 in the main text. The algorithm for determined (local) minima of V n is summarized in Algorithm 2.

.

3. RELAXATION OF NON-NEGATIVITY ASSUMPTION WITH RENDERED IMAGES:

It is clear that the inference with V n relies on the non-negativity assumption such that equation equation 10 holds. Now, we will argue that when the non-negativity assumption with rendered images h(y, z; 0) does not hold, we can relax V n to a more tractable version under that setting.

Theorem .5. (Relaxation of objective function V n when non-negativity assumption does not hold) Assume that π y ≥ γ for all 1 ≤ y ≤ K for some given γ ∈ (0, 1/2). Denote

W n := min θ 1 n n i=1 K y =1 1 y =arg max y∈J g(y,xi) x i -g(y , z i ) 2 2 -log(π y ,zi )

where

g(y, x) = h (y) MaxPool ReLu Conv Γ( ), • • • MaxPool ReLu Conv Γ(1), I + b(1) • • • + b( )

for all (x, y). Additionally, z i is the maximal value of z in the CNN structure of g(y, x i ) for 1 ≤ i ≤ n.

For any θ, we define

( y i , z i ) = arg max (y,z)∈J h (y, z; 0)x i + η(y, z)

and y i = arg max y g(y, x i ) as 1 ≤ i ≤ n. Then, the following holds (a) Upper bound:

V n ≤ min θ 1 n n i=1 x i -h(y i , z i ; 0) 2 2 -log(π y i ,zi ) + log π y i -log π yi prior loss + 1 2 h( y i , z i ; 0) 2 -h(y i , z i ; 0) 2 norm loss ≤ W n + log 1 γ -1 + min θ 1 n n i=1 1 2 h( y i , z i ; 0) 2 -h(y i , z i ; 0) 2 (b) Lower bound: V n ≥ min θ 1 n n i=1 x i -h(y i , z i ; 0) 2 2 -log(π y i ,zi ) + log π y i -log π yi prior loss + 1 2 h( y i , z i ; 0) 2 -h(y i , z i ; 0) 2 norm loss + g(y i , x i ) -h( y i , z i ; 0) x i + η( y i , z i ) CNN loss ≥ W n + log γ 1 -γ + min θ 1 n n i=1 1 2 h( y i , z i ; 0) 2 -h(y i , z i ; 0) 2 + min θ 1 n n i=1 g(y i , x i ) -h( y i , z i ; 0) x i + η( y i , z i )

Algorithm 2 Relaxation of unsupervised NRM without noise Input: Data x i , translation matrices T (t; ), zero padding matrices B( ), number of labels K, number of layers L.

Output: Parameters θ.

Initialize θ = {µ(y)} K y=1 , {Γ( )} L =1 , {π y } K y=1 , {b( )} L =1 . while θ has not converged do 1. E-Step: Update labels (y, z) of each data for i = 1 to n do ( y i , z i ) = arg max y,z h (y, z)x i + log(π y,z ) .

end for 2. M-Step: By using Stochastic Gradient Descent (SGD), update θ that minimizes

n i=1 x i -h( y i , z i ) 2 2 -log(π yi, zi ) .

end while:

The proof argument of the above theorem is similar to that of Theorem .4; therefore, it is omitted.

The upper bound and lower bound of V n in terms of W n in Theorem .5 implies that we can use W n as a relaxation of V n when the non-negativity assumption with rendered images h(y, z; 0) does not hold. The algorithm for achieving the (local) minima of W n is similar to Algorithm 2.

.4 NRM WITH (SEMI)-SUPERVISED SETTING

In this section, we consider the application of NRM to the (semi)-supervised setting of the data. Under that setting, only a (full) portion of labels of data x 1 , . . . , x n is available. Without loss of generality, we also assume that the rendering path h(y, z; 0) satisfies the non-negativity assumption.

For the case that h(y, z; 0) does not satisfy this assumption, we can argue in the same fashion as that of Theorem .5. Now, we assume that only the labels (y n1+1 , . . . , y n ) are unknown for some n 1 ≥ 0. When n 1 = 0, we have the supervised setting of data while we have the semi-supervised setting of data when nn 1 is small. Our goal is to build a semi-supervised model based on NRM such that the clustering information from data x 1 , . . . , x n1 can be used efficiently to increase the accuracy of classifying the labels of data x n1+1 , . . . , x n . For the sake of simple inference with that purpose, we only consider the setting of NRM when the noise goes to 0. Our idea of constructing the semi-supervised model based on NRM is insprired by an approximation of the upper bound of maximizing the conditional log likelihood of NRM in terms of the cross entropy and reconstruction loss in part (b) of Theorem 2.3. In particular, we combine the tractable version of reconstruction loss from the unsupervised setting in Theorem 2.3b and the cross entropy of approximate posterior in Theorem 2.3a, which can be formulated as follows

min θ α RC n n1 i=1 (y ,z )∈J 1    (y ,z )=arg max (y,z)∈J h (y,z;0)xi+τ (θy,z)    x i -h(y , z ; 0) 2 2 -log(π y ,z ) + n i=n1+1 z ∈L 1    z =arg max z∈L h (yi,z;0)xi+τ (θy i ,z )    x i -h(y i , z ; 0) 2 2 -log(π yi,z ) - α CE n -n 1 n i=n1+1 log q θ (y i |x i )

where α RC and α CE are non-negative weights associated with reconstruction loss and cross entropy respectively. Additionally, the approximate posterior q θ (y|x i ) is chosen as

q θ (y|x i ) = exp max z∈L h (y, z; 0)x i + τ (θ y,z ) + log π y K y =1

exp max z∈L h (y , z; 0)x i + τ (θ y ,z ) + log π y .

Given our choice of variational family q, we derive the variational term for the loss function, starting from the principled goal of minimizing the KL-distance D KL [q(y|x)||p(y|x)] between the true and approximate posteriors with respect to the parameters of q. As a result, such an optimized q will tilt towards better approximating p(y|x), which in turn means that it will account for some of the uncertainty in p(z|x). The variational terms in the loss are defined as Blei et al. (2017) :

EQUATION

)

This term is quite similar to that used in variational autoencoders (VAE) Kingma & Welling (2013) , except for two key differences: (i) here the latent variable y is discrete categorical rather than continuous Gaussian and (ii) we have employed a slight relaxation of the VAE by allowing for a penalty parameter β KL = 1. The latter is motivated by recent experimental results showing that such freedom enables optimal disentangling of the true intrinsic latent variables from the data.

Regarding the moment matching term: Batch Normalization can potentially be derived by normalizing the intermediate rendered images h( ), = 1, 2, . . . , L in the NRM by subtracting their means and dividing by their standard deviations under the assumption that the means and standard derivations of h( ) are close to those of the activation ψ( ) in the CNNs. From this intuition, in Section .4 of Appendix A, we introduce the moment-matching loss to improve the performance of the NRM/CNNs trained for semi-supervised learning tasks.

.5 STATISTICAL GUARANTEES FOR (SEMI)-SUPERVISED SETTING For the sake of simplicity with proof argument, we only provide detail theoretical analysis for statistical guarantee with the setting of equation 12 when the moment matching term is skipped. In particular, we are interested in the following (semi)-supervised model

EQUATION

where the approximate posterior q θ (y|x i ) is chosen as

q θ (y|x i ) := exp max z∈L h (y, z; 0)x i + τ (θ y,z ) + log π y K y =1 exp max z∈L h (y , z; 0)x i + τ (θ y ,z ) + log π y .

Here, α RC , α CE , and α KL are non-negative weights associated with reconstruction loss, cross entropy, and variational inference respectively. As being indicated in the formulation of objection function Y n , the only difference between Y n and equation 12 is the weight α MM regarding moment matching loss in equation 12 is set to be 0. To ease the presentation with theoretical analyses later, we call the objective function with Y n to be partially labeled latent dependence regularized cross entropy (partially labeled LDCE).

Consistency of partially labeled LDCE Firstly, we demonstrate that the objective function of partially labeled LDCE enjoys the consistency guarantee.

Theorem .6. (Consistency of objective function of partially labeled LDCE) Assume that n 1 is a function of n such that n 1 /n → λ as n → ∞. Furthermore, P( x ≤ R) = 1 as x ∼ P for some given R > 0. We denote the population version of partially labeled LDCE as follows

Y := min θ α RC λ (y ,z )∈J 1    (y ,z )=arg max (y,z)∈J h (y,z;0)x+τ (θy,z)    x -h(y , z ; 0) 2 2 -log(π y ,z ) dP (x) + (1 -λ) z ∈L 1    z =arg max z∈L h (y,z;0)x+τ (θy,z)    x -h(y, z ; 0) 2 2 -log(π y,z ) dQ(x, c) -α CE log q θ (y|x)dQ(x, c) + α KL K y=1

q θ (y|x) log q θ (y|x) π y dP (x).

Then, we obtain that Y n → Y almost surely as n → ∞.

The detail proof of Theorem .6 is deferred to Appendix C. Now, we denote θ :=

{ µ(y)} K y=1 , Γ( ) L =1 , { π y } K y=1 , b( ) L =1

the optimal solutions of objective function equation 15. Note that, the existence of these optimal solutions is guaranteed due to the compactness assumption of the parameter spaces Θ , Ω, and Ξ l for 1 ≤ ≤ L. The optimal solutions { µ(y)}

K y=1:

and Γ( )

L =1

lead to corresponding set of optimal rendered images S n . Similar to the case of SPLD regularized K-means, our goal is to guarantee the consistency of S n as well as

{ π y } K y=1 , b( ) L =1

.

In particular, we denote F 0 the set of all optimal solutions θ 0 of population partially labeled LDCE where θ 0 := µ 0 (y)

K y=1 , Γ 0 ( ) L =1 , π 0 y K y=1 , b 0 ( ) L =1

. For each θ 0 ∈ F 0 , we define S 0 the set of optimal rendered images associated with θ 0 . We denote G( F 0 ) the corresponding set of all optimal rendered images S 0 , optimal prior probabilities π 0 y K y=1 , and optimal biases b 0 ( ) L

=1:

.

Theorem .7. (Consistency of optimal rendering paths and optimal solutions of partially labeled LDCE) Assume that P( x ≤ R) = 1 as x ∼ P for some given R > 0. Then, we obtain that inf ( S0,{ π 0 y },{ b0( )})∈G( F0)

H( S n , S 0 ) + K y=1 | π y -π 0 y | + L =1 b( ) -b 0 ( ) → 0 almost surely as n → ∞.

The detail proof of Theorem .7 is postponed to Appendix C. for all (y, z) and 1 ≤ ≤ L. To achieve the generalization bound regarding that classification function, we rely on the study of generalization bound with margin loss. For the simplicity of argument, we assume that the true labels of x 1 , . . . , x n are y 1 , . . . , y n while y 1 , . . . , y n1 are not available to train. The margin of a labeled example (x, y) based on f can be defined as

ρ(f, x, y) = f (x, y) -max l =y f (x, l).

Therefore, the classification function f misspecifies the labeled example (x, y) as long as ρ(f, x, y) ≤ 0. The empirical margin error of f at margin coefficient

Γ ≥ 0 is R n,γ (f ) = 1 n n i=1 1 {ρ(f,xi,yi)≤γ} .

It is clear that R n,0 (f ) is the empirical risk of 0-1 loss, i.e., we have

R n,0 (f ) = 1 n n i=1 1 arg max 1≤y≤K f (xi,y) =yi

.

Similar to the argument in the case of SPLD regularized K-means, the optimal solutions θ of partially labeled LDCE lead to a set of rendered images h(y, z; 0) for all (y, z) ∈ J . However, only a small fraction of rendering paths are indeed active in the following sense. There exists a subset L n of L such that |L n | ≤ τ n |L| where τ n ∈ (0, 1], which is independent of data (x 1 , y 1 ), . . . , (x n , y n ), and the following holds max z∈L h (y, z; 0)x + τ ( θ y,z ) + log π y = max z∈Ln h (y, z; 0)x + τ ( θ y,z ) + log π y for all 1 ≤ y ≤ K. The above equation implies that

R n,Γ (f ) = R n,γ (f γ n )

for all Γ ≥ 0 and n ≥ 1 where f τ n (x, y) = max z∈Ln h (y, z; 0)x + τ ( θ y,z ) + log π y for all (x, y).

With that connection, we denote the expected margin error of classification function

f τ n at margin coefficient Γ ≥ 0 is R γ (f τ n ) = E1 {ρ(fτ n ,x,y)≤γ} .

The generalization bound that we establish in this section will base on the gap between the expected margin error R 0 (f τ n ) and its corresponding empirical version R n,Γ (f τ n ), which is also R n,Γ (f ). Theorem .8. (Generalization bound for margin-based classification) Assume that P ( x ≤ R) = 1 for some given R > 0 and x ∼ P . Additionally, the parameter spaces Θ and Ω are chosen such that h(y, z; 0) ≤ R for all (y, z) ∈ J . For any δ > 0, with probability at least 1δ, we have

R 0 (f τ n ) ≤ inf γ∈(0,1] R n,γ (f τ n ) + 8K(2K -1) Γ √ n 2τ n |L|(R 2 + 1) + | log γ| + log log 2 (2γ -1 ) n 1/2 + log(2δ -1 ) 2n

where γ is the lower bound of prior probability π y for all y.

Remark .9. The result of Theorem .8 gives a simple characterization for the generalization bound of classification setup from optimal solutions of partially labeled LDCE based on the number of active rendering paths, which is inherent to the structure of NRM. Such dependence of generalization bound on the number of active rendering configurations τ n |L| is rather interesting and may provide a new perspective on understanding the generalization bound. Nevertheless, there are certain limitations regarding the current generalization gap: (1) the active ratio τ n may change with the sample size unless we put certain constraints on the sparsity of switching variables a to reduce the number of active optimal rendering configurations; (2) the generalization bound is depth-dependent due to the involvement of the number of rendering configurations |L|. This is mainly because we have not fully taken into account all the structures of CNNs for the studying of generalization bound. Given some current progress on depth-independent generalization bound (Bartlett et al., 2017; Golowich et al., 2018) , it is an interesting direction to explore whether the techniques in these work can be employed to improve |L| in the generalization bound in Theorem .8.

. 

where Λ skip (t, s; ) is the shorcut matrices that results in skip connections in the corresponding ResNet. In particular, Λ skip (t, s; ) has the following form:

EQUATION

where M (s; ) ≡ diag (s( )) ∈ R D( )×D( ) is a diagonal matrix whose diagonal is the vector s( ). This matrix selects the templates for rendering. Furthermore, Λskip ( ) is a rendering matrix that is independent of latent variables t and s.

The following theorem show that similar to how CNNs can be derived from NRM, ResNet can be derived as a bottom-up inference in ResNRM.

Theorem .11. Inference in ResNRM yields skip connetions. In particular, if the rendering process at layer has the residual form as in Definition .10, the inference at this layer takes the following form:

EQUATION

Here, when ψ( -1) and ψ( ) have the same dimensions, Λskip ( ) is chosen to be an constant identity matrix in order to derive the parameter-free, identity shortcut among layers of the same size in the ResNet. When ψ( -1) and ψ( ) have the different dimensions, Λskip ( ) is chosen to be a learnable shortcut matrix which yields the projection shortcut W skip ( ) among layers of different sizes in In practice, the skip connections are usually across two layers or three layers. This is indeed a straightforward extension from the ResNRM. In particular, the ResNRM building block that corresponds to the ResNet building block in He et al. (2016) (see Fig. 5 ) takes the following form:

h( -2) := (Λ(t, s; -1)Λ(t, s; ) + Λ skip (s; )) h( ).

In inference, this ResNRM building block yields the ResNet building block in Fig. 5 :

EQUATION

.7.2 DENSE NEURAL RENDERING MODEL YIELDS DENSENET In a DenseNet Huang et al. (2017) , instead of combining features through summation, the skip connections concatenate features. In addition, within a building block, all layers are connected to each other (see Fig. 6 ). Similar to how ResNet can be derived from ResNRM, DenseNet can also be derived from a variant of NRM, which we call the Dense Neural Rendering Model (DenseNRM).

In DenseNRM, the rendering matrix Λ( ) is concatenated by an identity matrix. This extra identity matrix, in inference, yields the skip connections that concatenate features at different layers in a DenseNet. We define DenseNRM as follows.

Definition .12. The Dense Neural Rendering Model (DenseNRM) is the Neural Rendering Model whose rendering process from layer to layer -1, for some ∈ {1, 2, • • • , L}, has the residual form as follows:

EQUATION

We again denote Λ( ) as Λ(t, s; ) to show the dependency of Λ( ) on the latent variables t( ) and s( ). The following theorem establishes the connections between DenseNet and DenseNRM.

Theorem .13. Inference in DenseNRM yields DenseNet building blocks. In particular, if the rendering process at layer has the dense form as in Definition .12, the inference at this layer takes the following form: (s(1, p 1 ) × .) is equivalent to max( . , 0) and, therefore, yields the ReLU function. Also, in order to compute max t(1,p1) (), we take the brute force approach, computing the function inside the parentheses for all possible values of t(1, p 1 ) and pick the maximum one. This procedure is equivalent to the MaxPool operator in CNNs. Here we can make the bias term b(t; 1) independent of t and b(t; 1, p 1 ) are the same for all pixels p 1 in the same feature map of h(1) as in CNNs. Similarly, ψ(2) = MaxPool (ReLu (Conv (W (2), ψ(1)) + b(2))). Thus, we obtain the conclusion of the theorem for L = 2.

EQUATION

.9 PROOF FOR THEOREM .11: DERIVING THE RESIDUAL NETWORKS FROM THE RESIDUAL NEURAL RENDERING MODEL Similar to the proof in Section .8 above, when the rendering process at each layer in ResNRM is as in Eqn. 16, the activations ψ( ) is given by:

EQUATION

Here we can again make the bias term b(t; ) independent of t and b(t; , p) are the same for all pixels p in the same feature map of h( ) as in CNNs. We obtain the conclusion of the Theorem .11.

. 

EQUATION

)

Again we can make the bias term b(t; ) independent of t and b(t; , p) are the same for all pixels p in the same feature map of h( ) as in CNNs. We obtain the conclusion of the Theorem .13.

.11 PROVING THAT THE PARAMETRIZED JOINT PRIOR P(Y,Z) IS A CONJUGATE PRIOR Again, for simplicity, we only consider the proof for two layers. The argument for L ≥ 3 is similar and can be derived recursively from the proof for L = 2. In the derivation below, h(2) is µ(y). As in Eqn.1 in the definition of the NRM, the joint prior of y and z is given by:

EQUATION

Furthermore, as explained in Section .14 of Appendix D, due to the constant norm assumption with h(y, z; 0), the likelihood p(x|y, z) is estimated as follows:

p(x|y, z) ∝ exp 1 σ 2 h (y, z; 0)x = exp 1 σ 2 h (2)Λ (2)Λ (1)x

The posterior p(y, z|x) is given by:

EQUATION

Comparing Eqn. 25 and Eqn. 26, we see that the prior and the posterior have the same functional form. This completes the proof.

.12 DERIVING OTHER COMPONENTS IN THE CONVOLUTIONAL NEURAL NETWORKS .12.1 DERIVING THE LEAKY RECTIFIED LINEAR UNIT (LEAKY RELU)

The Leaky ReLU can be derived from the NRM in the same way as we derive the ReLU in Section .8, but instead of s( ) ∈ {0, 1}, we now let s( ) ∈ {α, 1}, where α is a small positive constant. Then, in

In line (a), we approximate E h(y,z;0) by its empirical value E i . The de-mean and normalize operators with the scale parameter α(1) and the shift parameter β(1) = 0 in the equations above already have the form of batch normalization. Note that when the model is trained with Stochastic Gradient Descent (SGD), the scale and shift parameters at each layer also account for the error in evaluating statistics of the activations using the mini-batches. Thus, β(1) is not 0 any more, but a parameter learned during the training. Also, in the equations above, ψ(1) is the activations at layer 1 in CNNs and given by: ψ(1) = MaxPool(ReLu(Normalize(Demean(Conv(Γ(1), x))))), (29) where σ ψ (1) is the standard deviation of the ψ(1). Eqn.28 can be expressed in term of ψ(1) as follows:

max z(2) h(1) -E h (1) ψ(1) + const = max z(2) h (1)(ψ(1) -E ψ (1)) + (h (1)E ψ (1) -E h (1)ψ(1)) + const = max z(2) p2∈P(2) s(2, p 2 )µ(y, p 2 ) σ ψ (2, p 2 ) σ µ(y) (p 2 ) α(2,p2) 1 σ ψ (2, p 2 ) normalize × Γ (2, p 2 )B (2, p 2 )T (t; 2, p 2 )ψ(1) -E[Γ (2, p 2 )B (2, p 2 )T (t; 2, p 2 )ψ(1)] de-mean + h(1, p 1 )E ψ (1, p 1 ) -E h (1, p 1 )ψ(1, p 1 ) β(2,p2) + const = max z(2) µ (y) MaxPool(ReLu(BatchNorm(Conv(Γ(2), ψ(1)); α(2), α(2) β(2)))) + const (30)

The batch normalization at this layer of the CNN has the scale parameter α(2) and the shift parameter is the element-wise product of α(2) and β(2).

.14 PROOFS FOR CONNECTION BETWEEN NRM AND CROSS ENTROPY PROOF OF THEOREM .1 (a) To ease the presentation of proof, we denote the following key notation

A := max (zi) n i=1 ,θ∈Aγ 1 n n i=1 ln p(y i |x i , z i ; θ).

From the definition of A γ = {θ : h(y, z; 0) = γ}, we achieve the following equations

A = max θ∈Aγ 1 n n i=1 max zi log p(y i |x i , z i ; θ) = max θ∈Aγ 1 n n i=1 max zi log p(x i |y i , z i ; θ)p(y i |z i ; θ) K y=1 p(x i |y, z i ; θ)p(y|z i ; θ) = max θ∈Aγ 1 n n i=1 max zi log p(x i |y i , z i ; θ) + log p(y i , z i |θ) -log K y=1 p(x i |y, z i ; θ)p(y, z i |θ) .

From the formulation of NRM, we have the following formulation of prior probabilities p(y, z|θ)

p(y, z|θ) = exp η(y, z) σ 2 π y y ,z exp η(y , z ) σ 2 π y .

for all (y, z). By means of the previous equations, we eventually obtain that

A = max θ∈Aγ 1 n n i=1 max zi log p(x i |y i , z i ; θ) + log      exp η(y i , z i ) σ 2 π yi y,z exp η(y, z) σ 2 π y      -log K y=1 p(x i |y, z i ; θ) exp η(y, z i ) σ 2 π y y ,z exp η(y , z ) σ 2 π y = max θ∈Aγ 1 n n i=1 max zi log p(x i |y i , z i ; θ) + log exp η(y i , z i ) σ 2 π yi -log K y=1 p(x i |y, z i ; θ) exp η(y, z i ) σ 2 π y .

By defining ψ i (y, z i ) := log p(x i |y, z i ; θ) + log exp η(y, z i ) σ 2 π y for all 1 ≤ i ≤ n, the above equation can be rewritten as

EQUATION

By means of direct computation, the following equations hold

B = max θ∈Aγ 1 n n i=1 log Softmax max zi log p(x i |y i , z i ; θ) + η(y i , z i )/σ 2 + log π yi = max θ∈Aγ 1 n n i=1 log Softmax max zi - x i -µ yi,zi 2 2σ 2 + η(y i , z i ) σ 2 + log π yi (i) = max θ∈Aγ 1 n n i=1 log Softmax max zi h (y i , z i ; 0)x i + η(y i , z i ) σ 2 + b yi = -min θ∈Aγ 1 n n i=1 -log Softmax max zi h (y i , z i ; 0)x i + η(y i , z i ) σ 2 + b yi = -min θ∈Aγ 1 n n i=1 -log q(y i |x i ) = -min θ∈AΓ H p,q (y|x)

where equation (i) is due to the constant norm assumption with rendered images h(y, z; 0). Therefore, we achieve the conclusion of part (a) of the theorem.

where the final inequality is due to the fact that π yi /π y i ≤ (1γ)/γ for all 1 ≤ i ≤ n. Therefore, we achieve the conclusion of part (a) of the theorem. (b) Similar to the proof argument with part (a), we have

U n = min θ 1 n n i=1 1 2 x i -h(y i , z i ; 0) 2 -η(y i , z i ) -log π y i + log (y ,z )∈J exp(η(y , z ) + log π y ) ≥ min θ 1 n n i=1 1 2 x i 2 -h( y i , z i ; 0) x i -η( y i , z i ) + h(y i , z i ; 0) 2 /2 -log π y i + log y exp(max z ∈L η(y , z ) + log π y ) = B.

Direct computation with B leads to

B = min θ 1 n n i=1 x i -h( y i , z i ; 0) 2 2 -log(π yi, zi ) + log π y i -log π yi + h(y i , z i ; 0) 2 -h( y i , z i ; 0) 2 ≥ min θ 1 n n i=1 x i -h( y i , z i ; 0) 2 2 -log(π yi, zi ) + min θ 1 n n i=1 log π y i -log π yi + min θ 1 n n i=1 h(y i , z i ; 0) 2 -h( y i , z i ; 0) 2 ≥ V n + log γ 1 -γ + min θ 1 n n i=1 h(y i , z i ; 0) 2 -h( y i , z i ; 0) 2

where the final inequality is due to the fact that π yi /π y i ≥ γ/(1γ) for all 1 ≤ i ≤ n. As a consequence, we achieve the conclusion of part (b) of the theorem.

.15 PROOFS FOR STATISTICAL GUARANTEE AND GENERALIZATION BOUND FOR (SEMI)-SUPERVISED LEARNING PROOF OF THEOREM .6 The proof of this theorem relies on several results with uniform laws of large numbers. In particular, we will need to demonstrate the following results

EQUATION

EQUATION

EQUATION

EQUATION

EQUATION

almost surely as n → ∞. The proof for equation 35 is similar to that of equation 33; therefore, it is omitted.

Proof of equation 33: It is clear that

sup θ 1 n 1 n1 i=1 max (y,z)∈J h (y, z; 0)x i + τ (θ y,z ) - max (y,z)∈J h (y, z; 0)x + τ (θ y,z ) dP (x) ≤ sup |S |≤|J | 1 n 1 n1 i=1 max s∈S s [x i , 1] -max s∈S s [x, 1]dP (x)

where [x, 1] ∈ R D (0) +1 denotes the vector forms by concatenating 1 to x ∈ R D (0) and S in the above supremum is the set of finite elements in R D (0) +1 . Therefore, to achieve the result of equation 33, it is sufficient to show that

EQUATION

To obtain the conclusion of equation 39, we utilize the classical result with bracketing entropy to establish the uniform laws of large number (cf. Lemma 3.1 in van de Geer ( 2000)). In particular, we denote G to be the family of function on R 

} ∈ O |J | (δ) such that s i -s i ≤ δ for all 1 ≤ i ≤ k. Now, we denote f S (x) = max s∈S s [x, 1] + δ [x, 1] , f S (x) = max s∈S s [x, 1] -δ [x, 1] for any S ∈ O |J | (δ). It is clear that f S (x) ≤ f S (x) ≤ f S (x) for all x ∈ R D (0)

. Furthermore, we also have that

(f S (x) -f S (x))dP (x) = 2δ [x, 1] dP (x) ≤ 2δ x dP (x) + 1 .

For any > 0, by choosing δ < 2( x dP (x) + 1) then we will have that (f S (x)f S (x))dP (x) < . It implies that the -bracketing entropy of G is finite for the L 1 norm with distribution P (for the definition of bracketing entropy, see Definition 2.2 in van de Geer ( 2000)).

According to Lemma 3.1 in van de Geer (2000) , it implies that G satisfies the uniform law of large numbers, i.e., equation 39 holds.

Proof of equation 34::

To achieve the conclusion of this claim, we will need to rely on the control of Rademacher complexity based on Vapnik-Chervonenkis (VC) dimension. In particular, we firstly demonstrate that

EQUATION

almost surely as n 1 → ∞ for each 1 ≤ j ≤ k and k ≥ 1 where the supremum is taken with respect to S = {s 1 , . . . , s k }. For each j, we denote the Rademacher complexity as follows

R n1 = E sup |S |=k 1 n 1 n1 i=1 σ i 1 j=arg max 1≤l≤|S | [xi,1] s l

where σ 1 , . . . , σ n1 are i.i.d. Rademacher random variables, i.e., P(σ i = -1) = P(σ i = 1) = 1/2 for 1 ≤ i ≤ n 1 . Then, for any n 1 ≥ 1 and δ ≥ 0, according to standard argument with Rademacher complexity (Vershynin, 2011) , To achieve that result, we will utilize the study of VC dimension with partitions (cf. Section 21.5 in Devroye et al. (1996) ). In particular, for each set S = (s 1 , . . . , s k ), it gives rise to the partition A i = x ∈ R D (0) : [x, 1] s i ≥ [x, 1] s l ∀ l ∈ {1, . . . , k} as 1 ≤ i ≤ k. For our purpose with equation 40, it is sufficient to consider P n1 = A j , i =j

A i , which is a partition of B(R), for each set S with k elements. We denote F to be the collection of all P n for all S with k elements and B(P n ) the collection of all sets obtained from the unions of elements of P n . For each data (x 1 , . . . , x n1 ), we let N F (x 1 , . . . , x n1 ) the number of different sets in {(x 1 , . . . , x n1 ) ∩ A : A ∈ B(P n1 ) for P n1 ∈ F}. The shatter coefficient of F is defined as ∆ n1 (F) = s(F, n 1 ) = max (x1,...,xn 1 )

N F (x 1 , . . . , x n1 ).

According to Lemma 21.1 in Devroye et al. (1996) , ∆ n1 (F) ≤ 4∆ * n1 (F) where ∆ * n1 (F) is the maximal number of different ways that n 1 points can be partitioned by members of F. Now, for each element P n1 = A j , i =j A i of F, it is clear that the boundaries between A j and i =j A i are subsets of hyperplanes. From the formulation of A j , we have at most k -1 boundaries between A j and i =j A i . From the classical result of Dudley (1978) , each n 1 points in B(R) can be splitted By means of the inequality with Rademacher complexity in equation 41, we obtain that

R n = n-n1 v=0 c∈Av E E σ sup |S |=k 1 n -n 1 n i=n1+1 σ i 1 j=arg max 1≤l≤|S | [xi,1] s l 1 {yi=l} | c ∈ A v P( c ∈ A v ) = n-n1 v=0 E sup |S |=k 1 n -n 1 n1+v i=n1+1 σ i 1 j=arg max 1≤l≤|S | [xi,1] s l p v l (1 -p l ) n-n1-v n -n 1 v ≤ n-n1 v=1 v n -n 1 2(log 8 + (D (0) + 1)(k -1) log v) v p v l (1 -p l ) n-n1-v n -n 1 v ≤ 2 log 8 n -n 1 n-n1 v=1 v n -n 1 p v l (1 -p l ) n-n1-v n -n 1 v + 2(D (0) + 1)(k -1) log(n -n 1 ) n -n 1 n-n1 v=1 v log v (n -n 1 ) log(n -n 1 ) p v l (1 -p l ) n-n1-v n -n 1 v ≤ 2 log 8 n -n 1 + 2(D (0) + 1)(k -1) log(n -n 1 ) n -n 1 n-n1 v=1 p v l (1 -p l ) n-n1-v n -n 1 v = 2 log 8 n -n 1 + 2(D (0) + 1)(k -1) log(n -n 1 ) n -n 1 (1 -(1 -p l ) n-n1 ) → 0

as n → ∞ where c = (c n-n1+1 , . . . , c n ) and A v is the set of c such that there are exactly v values of y i to be l for 0 ≤ v ≤ nn 1 . The final inequality is due to the fact that v/(n -1) ≤ 1 and v log v/ {(n -1) log(n -1)} ≤ 1 for all 1 ≤ v ≤ nn 1 . Therefore, we achieve the conclusion of equation 42. , 2000) , we can find a finite set E δ such that each element in B(R) is within distance δ to some element of E δ for all δ > 0.

Additionally, there exists a set ∆(δ) such that for each (π 1 , . . . , π K ), we can find a corresponding element (π 1 , . . . , π K ) ∈ ∆(δ) such that (π 1 , . . . , π K ) -(π 1 , . . . , π K ) ≤ δ. We denote F(δ) = S y , {π y } : elements of S y in E δ , and(π 1 , . . . , π K ) ∈ ∆(δ) for all δ > 0. For any > 0, by choosing δ < 4

x dQ(x, y) + 4 + 4/γ then we will have (f {S y },{πy} (x, y)f {S y },{πy} (x, y))dQ(x, y) < . It implies that the -bracketing entropy of G is finite for the L 1 norm with distribution Q. Therefore, it implies that G satisfies the uniform law of large numbers, i.e., F n,l → 0 almost surely as n → ∞ for all 1 ≤ l ≤ K. As a consequence, the uniform law of large number result equation 37 holds.

Going back to the original problem, denote θ 0 = µ 0 (y) the optimal solutions of population partially labeled LDCE (Note that, the existence of these optimal solutions is guaranteed due to the compact assumptions with the parameter spaces Θ , Ω, and Ξ for all 1 ≤ ≤ L). Then, according to the formulation of partially labeled LDCE, we will have that xh 0 (y, z ; 0) 2 2 log( p 0 (y, z )) dQ(x, y),

Y n ≤ min

1 n -n 1 n i=n1+1

log q θ 0 (y i |x i ) → log q θ 0 (y|x)dQ(x, y),

1 n n i=1 K y=1 q θ 0 (y|x i ) log q θ 0 (y|x i ) π y → K y=1

q θ 0 (y|x) log q θ 0 (y|x) xh(y, z ; 0) 2 2 log( p(y, z )) dQ(x, y), 1 nn 1 n i=n1+1 log q θ (y i |x i ) → log q θ (y|x)dQ(x, y),

1 n n i=1 K y=1 q θ (y|x i ) log q θ (y|x i ) π y → K y=1

q θ (y|x) log q θ (y|x) π y dP (x)

almost surely as n → ∞ where θ = { µ(y)} xh(y, z ; 0) 2 2 log( p(y, z )) dQ(x, y) α CE log q θ (y|x)dQ(x, y)

+α KL K y=1

q θ (y|x) log q θ (y|x) π y dP (x) ≥ Y almost surely as n → ∞. As a consequence, we achieve Y n → Y almost surely as n → ∞. We reach the conclusion of the theorem.

PROOF OF THEOREM .7:

The proof argument of this theorem is a direct application of the results with uniform laws of large numbers in the proof of Theorem .6. In fact, we define where n ( J n ) is Rademacher complexity of J n , which in our case is defined as h (y, z; 0)x i + τ (θ y,z )

n ( J n ) = E

≤ E sup |S |≤cn|L| 1 n n i=1 σ i max s∈S s [x i , 1] ≤ 2c n |L|E sup s∈B(R) 1 n n i=1 σ i s [x i , 1] ≤ 2c n |L|RE 1 n n i=1 σ i [x i , 1] ≤ 2c n |L|(R 2 + 1) √ n

where the final inequality is due to Cauchy-Schwartz's inequality. Combining the above results with equation 43, we achieve the conclusion of the theorem.

:

labels

Figures and Tables:

N 4 g C d 4 9 q T 3 6 L 1 4 r / P W F W 8 x c w T f 4 L 1 9 A h W E j l I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r C G B M z n 4 w B g 0 9 W D 3 n 0 k B j 8 Y d x s Q

u e L X v B l c 7 x f 5 s i p k g U a v / B 7 0 E 5 o J l I Z y 0 L r r e 6 k J c 1 C G U Y 6 T U p B p T I G O Y I B d S y U I 1 G E + y z x x T 6 z S d + N E 2 Z H G n a n L F z k I r c c i s p s C z F D / 9 K b i X 1 4 3 M / F l m D O Z Z g Y l n T + K M + 6 a x J 0 W 4 P a Z Q m r 4 2 B K g i t m s L h 2 C A m p s T a X l E v 4 n r b O a 7 9 X 8 2 / N K v b O o o 0 i O y D G p E p 9 c k D q 5 I Q 3 S J J S k 5 I E 8 k W c n c x 6 d F + d 1 v l p w F j e H 5 B u c t 0 8 0 x J E y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 B j 3 1 P b

u e L X v B l c 7 x f 5 s i p k g U a v / B 7 0 E 5 o J l I Z y 0 L r r e 6 k J c 1 C G U Y 6 T U p B p T I G O Y I B d S y U I 1 G E + y z x x T 6 z S d + N E 2 Z H G n a n L F z k I r c c i s p s C z F D / 9 K b i X 1 4 3 M / F l m D O Z Z g Y l n T + K M + 6 a x J 0 W 4 P a Z Q m r 4 2 B K g i t m s L h 2 C A m p s T a X l E v 4 n r b O a 7 9 X 8 2 / N K v b O o o 0 i O y D G p E p 9 c k D q 5 I Q 3 S J J S k 5 I E 8 k W c n c x 6 d F + d 1 v l p w F j e H 5 B u c t 0 8 0 x J E y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 B j 3 1 P b

u e L X v B l c 7 x f 5 s i p k g U a v / B 7 0 E 5 o J l I Z y 0 L r r e 6 k J c 1 C G U Y 6 T U p B p T I G O Y I B d S y U I 1 G E + y z x x T 6 z S d + N E 2 Z H G n a n L F z k I r c c i s p s C z F D / 9 K b i X 1 4 3 M / F l m D O Z Z g Y l n T + K M + 6 a x J 0 W 4 P a Z Q m r 4 2 B K g i t m s L h 2 C A m p s T a X l E v 4 n r b O a 7 9 X 8 2 / N K v b O o o 0 i O y D G p E p 9 c k D q 5 I Q 3 S J J S k 5 I E 8 k W c n c x 6 d F + d 1 v l p w F j e H 5 B u c t 0 8 0 x J E y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 B j 3 1 P b

u e L X v B l c 7 x f 5 s i p k g U a v / B 7 0 E 5 o J l I Z y 0 L r r e 6 k J c 1 C G U Y 6 T U p B p T I G O Y I B d S y U I 1 G E + y z x x T 6 z S d + N E 2 Z H G n a n L F z k I r c c i s p s C z F D / 9 K b i X 1 4 3 M / F l m D O Z Z g Y l n T + K M + 6 a x J 0 W 4 P a Z Q m r 4 2 B K g i t m s L h 2 C A m p s T a X l E v 4 n r b O a 7 9 X 8 2 / N K v b O o o 0 i O y D G p E p 9 c k D q 5 I Q 3 S J J S k 5 I E 8 k W c n c x 6 d F + d 1 v l p w F j e H 5 B u c t 0 8 0 x J E y < / l a t e x i t > level < l a t e x i t s h a 1 _ b a s e 6 4 = " W 6 w Q y

6 5 5 i 5 k T + A P v 4 x s V c o 5 S < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W 6 w Q y A q S / a v u c 1 u r k G v g o X 6 N Z T A = " > A A A B 6 3 i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S S I J hY 0 F F X d d H d F q e D G + v 6 X t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 1 v C 7 / 9 i N r w R D 3 Y S Y q h p E P F Y 8 6 o L a Q e C t G v V P 2 a P w P 5 I c E y q c I C j X 7 l s z d I W C Z R W S a o M d 3 A T 2 2 Y U 2 0 5 E z g t 9 z K D K W V j O s S u o 4 p K N G E + u 3 V K z p 0 y I H G i X S l L Z u r v i Z x K Y y Y y c p 2 S 2 p F Z 9 g r x P 6 + b 2 f g m z L l K M 4 u K z R f F m S A 2 I c X j Z M A 1 M i s m j l C m u b u V s B H V l F k X T 9 m F s P L y K m l d 1 g K / F t x f V e u d R R w l O I U z u I A A r q E O d 9 C A J j A Y w R O 8 w K s n v W f v z X u f t 6 5 5 i 5 k T + A P v 4 x s V c o 5 S < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W 6 w Q y A q S / a v u c 1 u r k G v g o X 6 N Z T A = " > A A A B 6 3 i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S S I J h Y 0 F F X d d H d F q e D G + v 6 X t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 1 v C 7 / 9 i N r w R D 3 Y S Y q h p E P F Y 8 6 o L a Q e C t G v V P 2 a P w P 5 I c E y q c I C j X 7 l s z d I W C Z R W S a o M d 3 A T 2 2 Y U 2 0 5 E z g t 9 z K D K W V j O s S u o 4 p K N G E + u 3 V K z p 0 y I H G i X S l L Z u r v i Z x K Y y Y y c p 2 S 2 p F Z 9 g r x P 6 + b 2 f g m z L l K M 4 u K z R f F m S A 2 I c X j Z M A 1 M i s m j l C m u b u V s B H V l F k X T 9 m F s P L y K m l d 1 g K / F t x f V e u d R R w l O I U z u I A A r q E O d 9 C A J j A Y w R O 8 w K s n v W f vz X u f t 6 5 5 i 5 k T + A P v 4 x s V c o 5 S < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W 6 w Q y A q S / a v u c 1u r k G v g o X 6 N Z T A = " > A A A B 6 3 i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S S I J h Y 0 F F X d d H d F q e D G + v 6 X t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 1 v C 7 / 9 i N r w R D 3 Y S Y q h p E P F Y 8 6 o L a Q e C t G v V P 2 a P w P 5 I c E y q c I C j X 7 l s z d I W C Z R W S a o M d 3 A T 2 2 Y U 2 0 5 E z g t 9 z K D K W V j O s S u o 4 p K N G E + u 3 V K z p 0 y I H G i X S l L Z u r v i Z x K Y y Y y c p 2 S 2 p F Z 9 g r x P 6 + b 2 f g m z L l K M 4 u K z R f F m S A 2 I c X j Z M A 1 M i s m j l C m u b u V s B H V l F k X T 9 m F s P L y K m l d 1 g K / F t x f V e u d R R w l O I U z u I A A r q E O d 9 C A J j A Y w R O 8 w K s n v W f v z X u f t6 5 5 i 5 k T + A P v 4 x s V c o 5 S < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " W 6 w Q y A q S / a v u c 1 u r k G v g o X 6 N Z T A = " > A A A B 6 3 i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N l J b z J k Z n a Z m R X C k l / w 4 k E R r / 6 Q N / / G 2 S S I J h Y 0 F F X d d H d F q e D G + v 6 X t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b J s k 0 w y Z L R K I 7 E T U o u M K m 5 V Z g J 9 V I Z S S w H Y 1 v C 7 / 9 i N r w R D 3 Y S Y q h p E P F Y 8 6 o L a Q e C t G v V P 2 a P w P 5 I c E y q c I C j X 7 l s z d I W C Z R W S a o M d 3 A T 2 2 Y U 2 0 5 E z g t 9 z K D K W V j O s S u o 4 p K N G E + u 3 V K z p 0 y I H G i X S l L Z u r v i Z x K Y y Y y c p 2 S 2 p F Z 9 g r x P 6 + b 2 f g m z L l K M 4 u K z R f F m S A 2 I c X j Z M A 1 M i s m j l C m u b u V s B H V l F k X T 9 m F s P L y K m l d 1 g K / F t x f V e u d R R w l O I U z u I A A r q E O d 9 C A J j A Y w R O 8 w K s n v W f v z X u f t 6 5 5 i 5 k T + A P v 4 x s V c o 5 S < / l a t e x i t > Convolution W (`) =

9 e b W d 3 b / + g f u g f N f z j k 9 N m o 2 f z 0 g j s i l z l 5 j n m F p X U 2 C V J C p 8 L g z y L F f b j 6 f 0 i 7 7 + g s T L X T z Q r M M r 4 W M t U C k 7 O 6 o y a r a A d L M W 2 I V x D C 9 Y a N b + G S S 7 K D D U J x a 0 d h E F B U c U N S a F w 7 g 9L i w U X U z 7 G g U P N M 7 R R t R x z z i 6 d k 7 A 0 N + 5 o Y k v 3 9 4 u K Z 9 b O s t j d z D h N 7 G a 2 M P / L B i W l t 1 E l d V E S a r H 6 K C 0 V o 5 w t d m a J N C h I z R x w Y a S b l Y k J N 1 y Q a 8 Z 3 H Y S b G 2 9 D 7 7 o d B u 3 w M Y A 6 n M M F X E E I N 3 A H D 9 C B L g h I 4B X e v Y n 3 5 n 2 s u q p 5 6 9 L O 4 I + 8 z x 8 4 x I o 4 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " v 5 ji O u Z U G S S 4 3 d c Z A u 3 A E k k 8 B H Y = " > A A A B 8 X i c b Z B L S w M x F I X v + K y 1 6 q h L N 8 E i 1 E 2 Z c a N L w Y U u K 9 g H d M a S S T N t a B 5 D k h H K U P + K G x e K + G f c + W 9 M H w t t P R D 4 O C f h 3 p w k 4 8 z Y I P j 2 1 t Y 3 N r e 2 S z v l 3 c r e / o F / W G k Z l W t C m 0 R x p T s J N p Q z S Z u W W U 4 7 m a Z Y J J y 2 k 9 H N N G 8 / U W 2 Y k g 9 2 n N F Y 4 I F k K S P Y O q v n H 0 e 3 W A j 8 W E R W Z Z N a R D k / 7 / n V o B 7 M h F Y h X E A V F m r 0 / K + o r 0 g u q L S E Y 2 O 6 Y Z D Z u M D a M s L p p B z l h m a Y j P C A d h 1 K L K i J i 9 n y E 3 T m n D 5 K l X Z H W j R z f 7 8 o s D B m L B J 3 U 2 A 7 N M v Z 1 P w v 6 + Y 2 v Y o L J r P c U k n m g 9 K c I 6 v Q t A n U Z5 o S y 8 c O M N H M 7 Y r I E G t M r O u r 7 E o I l 7 + 8 C q 2 L e h j U w / s A S n A C p 1 C D E C 7 h G u 6 g A U 0 g M I Y X e I N 3 7 9 l 7 9 T 7 m d a 1 5 i 9 6 O 4 I + 8 z x 8 B m p M w < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B n U Y m 3 e d 4 D w X E R Q a P Q h c b S p R F R g = " > A A A B 8 X i c d Z B L S w M x F I X v 1 F e t V U d d u g k W o W 7 K j B t d C i 5 0 W c E + o D O W T J p p Q 5 P M k G S E Y a h / x Y 0 L R f w z 7 v w 3 p g + h v g 4 E D u f c S y 5 f l H K m j e d 9 O K W V 1 b X 1 j f J m Z a u 6 v b P r 7 l X b O s k U o S 2 S 8 E R 1 I 6 w p Z 5 K 2 D D O c d l N F s Y g 4 7 U T j y 2 n f u a d K s 0 T e m j y l o c B D y W J G s L F R 3 z 0 I r r A Q + K 4 I T J J O 6 g H l / K T v 1 v y G N x P y f p m v q g Y L N f v u e z B I S C a o N I R j r X u + l 5 q w w M o w w u m k E m S a p p i M 8 Z D 2 r J V Y U B 0 W s + M n 6 N g m A x Q n y j 5 p 0 C x d 3 i i w 0 D o X k Z 0 U 2 I z 0 z 2 4 a / t X 1 M h O f h w W T a W a o J P O P 4 o w j k 6 A p C T R g i h L D c 2 s w U c z e i s g I K 0 y M 5 V V Z h v C / a Z 8 2 f K / h 3 3 h Q h k M 4 g j r 4 c A Y X c A 1 N a A G B H B 7 h G V 6 c B + f J e Z 3 j K j k L b v v w T c 7 b J w M P k z E = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " b p x D T X 4 S H H 5 i u E z A F g P H H B x 4 r g s = " > A A A B / H i c d V D L S g M x F M 3 U V 6 2 v 0 S 7 d B I t Q N 2 X G j S 4 L L n R Z w T 6 g U 0 s m z b S h S W Z I 7 g j D U H / F j Q t F 3 P o h 7 v w b 0 4 d Q X w c u H M 6 5 l 3 s 4 Y S K 4 A c / 7 c A o r q 2 v r G 8 X N 0 t b 2 z u 6 e u 3 / Q M n G q K W v S W M S 6 E x L D B F e s C R w E 6 y S a E R k K 1 g 7 H F 1 O / f c e 0 4 b G 6 g S x h P U m G i k e c E r B S 3 y 0 H l 0 R K c p s H E C e T a s C E O O m 7 F b / m z

Figure 1: (a) The Neural Rendering Model (NRM) captures latent variations in the data and yields CNNs as its inference. Costs for training CNNs are derived from likelihood estimations in NRM. (b) Graphical model depiction of NRM. Latent variables in NRM depend on each other. Object category y decides the class template, and then new latent variables are incorporated at each layer to render intermediate images (red) with finer details. Finally, pixel noise is added to render the image x.

a P 8 A y v 8 O Z p 7 8 V 7 9 z 4 W r W t e M X M C f + B 9 / g C 8 y Y 9 K < / l a t e x i t > zero-pad translate zero-pad translate . . . < l a t e x i t s h a 1 _ b a s e 6 4 = " c a 4 A n 8 D 0 S Y 3 s

X / / P 6 O S Z 3 Y c F l l i O T d L k o y Y W L y p 3 / 7 8 Z c M 4 p i a g m h m t t b X T o i m l C 0 K V V t C P 7 q y + u k c 9 3 w v Y b / e F N v 9 s o 4 K n A O F 3 A F P t x C E x 6 g B W 2 g o O A Z X u H N Q e f F e X c + l q 0 b T j l z B n / g f P 4 A z Z a R p g = = < / l a t e x i t > h(` 1) < l a t e x i t s h a 1 _ b a s e 6 4 = " t f 2 d + f C O c T N B 4 h D a 1 H 9 j o i 9 Z u X o = " > A A A B 8 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S L U g y U R Q Y 8 F L x 4 r 2 A 9 I Q 9 l s N + 3 S T T b s T o Q S + j O 8 e F D E q 7 / G m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M J X C o O t + O 2 v r G 5 t b 2 6 W d 8 u 7 e / s F h 5 e i 4 b V

a w E D B M 7 z C m 4 P O i / P u f C x a 1 5 x i 5 g T + w P n 8 A S 2 K j + 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " t f 2d + f C O c T N B 4 h D a 1 H 9 j o i 9 Z u X o = " > A A A B 8 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S L U g y U R Q Y 8 F L x 4r 2 A 9 I Q 9 l s N + 3 S T T b s T o Q S + j O 8 e F D E q 7 / G m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M J X C o O t + O 2 v r G 5 t b 2 6 W d 8 u 7 e / s F h 5 e i 4 b V

a w E D B M 7 z C m 4 P O i / P u f C x a 1 5 x i 5 g T + w P n 8 A S 2 K j + 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " t f 2d + f C O c T N B 4 h D a 1 H 9 j o i 9 Z u X o = " > A A A B 8 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S L U g y U R Q Y 8 F L x 4r 2 A 9 I Q 9 l s N + 3 S T T b s T o Q S + j O 8 e F D E q 7 / G m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M J X C o O t + O 2 v r G 5 t b 2 6 W d 8 u 7 e / s F h 5 e i 4 b V

a w E D B M 7 z C m 4 P O i / P u f C x a 1 5 x i 5 g T + w P n 8 A S 2 K j + 4 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " t f 2d + f C O c T N B 4 h D a 1 H 9 j o i 9 Z u X o = " > A A A B 8 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S L U g y U R Q Y 8 F L x 4r 2 A 9 I Q 9 l s N + 3 S T T b s T o Q S + j O 8 e F D E q 7 / G m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M J X C o O t + O 2 v r G 5 t b 2 6 W d 8 u 7 e / s F h 5 e i 4 b V

a w E D B M 7 z C m 4 P O i / P u f C x a 1 5 x i 5 g T + w P n 8 A S 2 K j + 4 = < / l a t e x i t >x < l a t e x i t s h a 1 _ b a s e 6 4 = " T

4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b s Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M

D K 7 w 5 D 8 6 L 8 + 5 8 L F s L T j 5 z C n / g f P 4 A 7 N e N E g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T

4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b s Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M

D K 7 w 5 D 8 6 L 8 + 5 8 L F s L T j 5 z C n / g f P 4 A 7 N e N E g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T

4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b s Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M

D K 7 w 5 D 8 6 L 8 + 5 8 L F s L T j 5 z C n / g f P 4 A 7 N e N E g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T

4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b s Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e F R + f i k r e N U M W y x W M

D K 7 w 5 D 8 6 L 8 + 5 8 L F s L T j 5 z C n / g f P 4 A 7 N e N E g = = < / l a t e x i t > h(`) < l a t e x i t s h a 1 _ b a s e 6 4 = " e / C d V 7 X s M f 2 X h + 8 9 w r g o

y d x X p x 3 5 2 P Z W n K K m V P 4 A + f z B 6 A M j y g = < / l a t e x i t > p < l a t e x i t s h a 1 _ b a s e 6 4 = " f h C 5 4 F V T 5 N G E M J 6 y F b T k 8 x Y G d w c = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P B i 8 c W T F t o Q 9 l s J + 3 a z S b s b o Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M B V c G 9 f 9 d k o b m 1 v b O + X d y t 7 + w e F R 9 f

R h j M 4 h 0 v w 4 A Y a c A 9 N 8 I E B w j O 8 w p v z 6 L w 4 7 8 7 H s r X k F D O n 8 A f O 5 w / g t 4 0 K < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f h C 5 4 F V T 5 N G E M J 6 y F b T k 8 x Y G d w c = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P B i 8 c W T F t o Q 9 l s J + 3 a z S b s b o Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M B V c G 9 f 9 d k o b m 1 v b O + X d y t 7 + w e F R 9 f

R h j M 4 h 0 v w 4 A Y a c A 9 N 8 I E B w j O 8 w p v z 6 L w 4 7 8 7 H s r X k F D O n 8 A f O 5 w / g t 4 0 K < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f h C 5 4 F V T 5 N G E M J 6 y F b T k 8 x Y G d w c = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P B i 8 c W T F t o Q 9 l s J + 3 a z S b s b o Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M B V c G 9 f 9 d k o b m 1 v b O + X d y t 7 + w e F R 9 f

R h j M 4 h 0 v w 4 A Y a c A 9 N 8 I E B w j O 8 w p v z 6 L w 4 7 8 7 H s r X k F D O n 8 A f O 5 w / g t 4 0 K < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f h C 5 4 F V T 5 N G E M J 6 y F b T k 8 x Y G d w c = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 0 G P B i 8 c W T F t o Q 9 l s J + 3 a z S b s b o Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M B V c G 9 f 9 d k o b m 1 v b O + X d y t 7 + w e F R 9 f

R h j M 4 h 0 v w 4 A Y a c A 9 N 8 I E B w j O 8 w p v z 6 L w 4 7 8 7 H s r X k F D O n 8 A f O 5 w / g t 4 0 K < / l a t e x i t > p + 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " / Z A g g R A t M C u l D n X k t O j U B 3 U P / A 8 = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B Z B E E o i g h 4 L X j x W t B / Q h r L Z b t q l m 03 Y n Q g l 9 C d 4 8 a C I V 3 + R N / + N 2 z Q H b X 0 w 8 H h v h p l 5 Q S K F Q d f 9 d k p r 6 x u b W + X t y s 7 u 3 v 5 B 9 f C o b e J U M 9 5 i s Y x 1 N 6 C G S 6 F 4 C w V K 3 k 0 0 p 1 E g e S e Y 3 M 7 9 z h P X R s T q E a c J 9 y M 6 U i I U j K K V H p I L b 1 C t u X U 3 B 1 k l X k F q U K A 5 q H 7 1 h z F L I 6 6 Q S W p M z 3 M T 9 D O q U T D J Z 5 V + a n h C 2 Y S O e M 9 S R S N u / C w / d U b O r D I k Y a x t K S S 5 + n s i o 5 E x 0 y i w n R H F s V n 2 5 u J / X i / F 8 M b P h E p S 5 I o t F o W p J B i T + d 9 k K D R n K K e W U K a F v Z W w M d W U o U 2 n Y k P w l l 9 e J e 3 L u u f W v f u r W q N b x F G G E z i F c / D g G h p w B 0 1 o A Y M R P M M r v D n S e X H e n Y 9 F a 8 k p Z o 7 h D 5 z P H 7 c L j X o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / Z A g g R A t M C u l D n X k t O j U B 3 U P / A 8 = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B Z B E E o i g h 4 L X j x W t B / Q h r L Z b t q l m 0 3 Y n Q g l 9 C d 4 8 a C I V 3 + R N / + N 2 z Q H b X 0 w 8 H h v h p l 5 Q S K F Q d f 9 d k p r 6 x u b W + X t y s 7 u 3 v 5 B 9 f C o b e J U M 9 5 i s Y x 1 N 6 C G S 6 F 4 C w V K 3 k 0 0 p 1 E g e S e Y 3 M 7 9 z h P X R s T q E a c J 9 y M 6 U i I U j K K V H p I L b 1 C t u X U 3 B 1 k l X k F q U K A 5 q H 7 1 h z F L I 6 6 Q S W p M z 3 M T 9 D O q U T D J Z 5 V + a n h C 2 Y S O e M 9 S R S N u / C w / d U b O r D I k Y a x t K S S 5 + n s i o 5 E x 0 y i w n R H F s V n 2 5 u J / X i / F 8 M b P h E p S 5 I o t F o W p J B i T + d 9 k K D R n K K e W U K a F v Z W w M d W U o U 2 n Y k P w l l 9 e J e 3 L u u f W v f u r W q N b x F G G E z i F c / D g G h p w B 0 1 o A Y M R P M M r v D n S eX H e n Y 9 F a 8 k p Z o 7 h D 5 z P H 7 c L j X o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / Z A g g R A t M C u l D n X k t O j U B 3 U P / A 8 = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9L B Z B E E o i g h 4 L X j x W t B / Q h r L Z b t q l m 0 3 Y n Q g l 9 C d 4 8 a C I V 3 + R N / + N 2 z Q H b X 0 w 8 H h v h p l 5 Q S K F Q d f 9 d k p r 6 x u b W + X t y s 7 u 3 v 5 B 9 f C o b e J U M 9 5 i s Y x 1 N 6 C G S 6 F 4 C w V K 3 k 0 0 p 1 E g e S e Y 3 M 7 9 z h P X R s T q E a c J 9 y M 6 U i I U j K K V H p I L b 1 C t u X U 3 B 1 k l X k F q U K A 5 q H 7 1 h z F L I 6 6 Q S W p M z 3 M T 9 D O q U T D J Z 5 V + a n h C 2 Y S O e M 9 S R S N u / C w / d U b O r D I k Y a x t K S S 5 + n s i o 5 E x 0 y i w n R H F s V n 2 5 u J / X i / F 8 M b P h E p S 5 I o t F o W p J B i T + d 9 k K D R n K K e W U K a F v Z W w M d W U o U 2 n Y k P w l l 9 e J e 3 L u u f W v f u r W q N b x F G G E z i F c / D g G h p w B 0 1 o A Y M R P M M r v D n Se X H e n Y 9 F a 8 k p Z o 7 h D 5 z P H 7 c L j X o = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / Z A g g R A t M C u l D n X k t O j U B 3 U P / A 8 = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B Z B E E o i g h 4 L X j x W t B / Q h r L Z b t q l m 0 3 Y n Q g l 9 C d 4 8 a C I V 3 + R N / + N 2 z Q H b X 0 w 8 H h v h p l 5 Q S K F Q d f 9 d k p r 6 x u b W + X t y s 7 u 3 v 5 B 9 f C o b e J U M 9 5 i s Y x 1 N 6 C G S 6 F 4 C w V K 3 k 0 0 p 1 E g e S e Y 3 M 7 9 z h P X R s T q E a c J 9 y M 6 U i IU j K K V H p I L b 1 C t u X U 3 B 1 k l X k F q U K A 5 q H 7 1 h z F L I 6 6 Q S W p M z 3 M T 9 D O q U T D J Z 5 V + a n h C 2 Y S Oe M 9 S R S N u / C w / d U b O r D I k Y a x t K S S 5 + n s i o 5 E x 0 y i w n R H F s V n 2 5 u J / X i / F 8 M b P h E p S 5 I o t F o W p J B i T + d 9 k K D R n K K e W U K a F v Z W w M d W U o U 2 n Y k P w l l 9 e J e 3 L u u f W v f u r W q N b x F G G E z i F c / D g G h p w B 0 1 o A Y M R P M M r v D n S e X H e n Y 9 F a 8 k p Z o 7 h D 5 z P H 7 c L j X o = < / l a t e x i t > render at the next location

l a t e x i t s h a 1 _ b a s e 6 4 = " Z s h Q R D C e w T B C 9 o s b q P v t t Q W W N L 4 = " > A A A B 7 X i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N n J b D J m d m a Z 6 R X C k n / w 4 k E R r / 6 P N / / G S b I H T S x o K K q 6 6 e 6 K U i k s + v 6 3 t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b V m e G 8 S b T U p t O R C 2 X Q v E m C p S 8 k x p O k 0 j y d j S + n f n t J 2 6 s 0 O o B J y k P E z p U I h a M o p N a P R Q J t / 1 K 1 a / 5 c 5 B V E h S k C g U a / c p X b 6 B Z l n C F T F J r u 4 G f Y p h T g 4 J J P i 3 3 M s t T y s Z 0 y L u O K u q W h P n 8 2 i k 5 d 8 q A x N q 4 U k j m 6 u + J n C b W T p L I d S Y U R 3 b Z m 4 n / e d 0 M 4 5 s w F y r N k C u 2 W B R n k q A m s 9 f J Q B j O U E 4 c o c w I d y t h I 2 o

r Q B A a P 8 A y v 8 O Z p 7 8 V 7 9 z 4 W r W t e M X M C f + B 9 / g C 8 y Y 9 K < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z sh Q R D C e w T B C 9 o s b q P v t t Q W W N L 4 = " > A A A B 7 X i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N n J b D J m d m a Z 6 R X C k n / w 4 k E Rr / 6 P N / / G S b I H T S x o K K q 6 6 e 6 K U i k s + v 6 3 t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b V m e G 8 S b T U p t O R C 2 X Q v E m C p S 8 k x p O k 0 j y d j S + n f n t J 2 6 s 0 O o B J y k P E z p U I h a M o p N a P R Q J t / 1 K 1 a / 5 c 5 B V E h S k C g U a / c p X b 6 B Z l n C F T F J r u 4 G f Y p h T g 4 J J P i 3 3 M s t T y s Z 0 y L u O K u q W h P n 8 2 i k 5 d 8 q A x N q 4 U k j m 6 u + J n C b W T p L I d S Y U R 3 b Z m 4 n / e d 0 M 4 5 s w F y r N k C u 2 W B R n k q A m s 9 f J Q B j O U E 4 c o c w I d y t h I 2 o

r Q B A a P 8 A y v 8 O Z p 7 8 V 7 9 z 4 W r W t e M X M C f + B 9 / g C 8 y Y 9 K < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " Z sh Q R D C e w T B C 9 o s b q P v t t Q W W N L 4 = " > A A A B 7 X i c b V D L S g N B E O z 1 G e M r 6 t H L Y B A 8 h V 0 R 9 B j w 4 j G C e U C y h N n J b D J m d m a Z 6 R X C k n / w 4 k E Rr / 6 P N / / G S b I H T S x o K K q 6 6 e 6 K U i k s + v 6 3 t 7 a + s b m 1 X d o p 7 + 7 t H x x W j o 5 b V m e G 8 S b T U p t O R C 2 X Q v E m C p S 8 k x p O k 0 j y d j S + n f n t J 2 6 s 0 O o B J y k P E z p U I h a M o p N a P R Q J t / 1 K 1 a / 5 c 5 B V E h S k C g U a / c p X b 6 B Z l n C F T F J r u 4 G f Y p h T g 4 J J P i 3 3 M s t T y s Z 0 y L u O K u q W h P n 8 2 i k 5 d 8 q A x N q 4 U k j m 6 u + J n C b W T p L I d S Y U R 3 b Z m 4 n / e d 0 M 4 5 s w F y r N k C u 2 W B R n k q A m s 9 f J Q B j O U E 4 c o c w I d y t h I 2 o

Figure 3: Rendering from level to level -1 in the NRM. At each pixel p in the intermediate image h( ), NRM renders iff the template selects latent variable s( , p) = 1. If rendering, the template Γ( , p) is multiplied by the pixel value h( , p). Then the matrix B( , p) zero-pads the result to the size of the intermediate image at level -1. Next, the translation matrix T ( , p) locally translates the rendered image to location specified by the latent variable t( , p). The same rendering repeats at other pixels of h( ). NRM adds all rendered images to obtain the intermediate image h( -1).where η(y, z)

Figure 2: Reconstructed images at each layer in a 5layer NRM trained on MNIST with 50K labeled data. Original images are in the rightmost column. Early layers in the rendering such as layer 4 and 3 capture coarse-scale features of the image while later layers such as layer 2 and 1 capture finer-scale features. From layer 2, we begin to see the gist of the rendered digits.

Figure 4: The Max-Min network

GENERALIZATION BOUND FOR CLASSIFICATION FRAMEWORK WITH (SEMI)-SUPERVISED SETTING In this section, we provide a simple generalization bound for certain classification function with the optimal solutions θ = { µ(ydenote the following function f : R D (0) × {1, . . . , K} → R as f (x, y) = max z∈L h (y, z; 0)x + τ ( θ y,z ) + log π y for all (x, y) ∈ R D (0) × {1, . . . , K} where h(y, z; 0) := Λ(z; 1) . . . Λ(z; L) µ(y), Λ(z; ) := p∈P( ) s( , p)T (t; , p)B( , p) Γ( , p)

Figure5: ResNet building block as inHe et al. (2016)

s l dP (x) ≤ 2R n1 + δ with probability at least 1 -2 exp -n 1 δ 2 8. According to Borel-Cantelli's lemma, to achieve equation 40, it is sufficient to demonstrate that R n1 → 0 as n 1 → ∞.

by a hyperplane in at most n D (0) +1 1 different ways as the VC dimension of the hyperplane is at most D (0) + 1. As a consequence, we would have∆ * n1 (F) ≤ n (D (0) +1)(k-1) 1, which leads to∆ n1 (F) ≤ 4n (D (0) +1)(k-1) 1.Going back to our evaluation with Rademacher complexity R n1 , by means of Massart's lemma, we have thatR n1 = E E σ sup F (x 1 , . . . , x n1 ) n 1 ≤ 2(log 8 + (D (0) + 1)(k -1) log n 1 ) n 1 → 0 (41)as n 1 → ∞. Therefore, equation 40 is proved.Proof of equation 36:To achieve the conclusion of this claim, we firstly demonstrate that sup n → ∞ for each 1 ≤ j ≤ k and 1 ≤ l ≤ K where k ≥ 1 and the supremum is taken with respect to S = {s 1 , . . . , s k }. The proof of the above result will rely on VC dimension with Voronoi partitions being established in equation 34. In particular, according to the standard argument with Rademacher complexity, it is sufficient to demonstrate thatR n = E sup 1] s l 1 {yi=l} → 0.

Now, coming back to equation 36, by means of triangle inequality, we achieve that E {y=l} dQ(x, y) → 0 where the last inequality is due to the results with uniform laws of large numbers from equation 42 and the fact that h(l, z ; 0) 2 2 log π l is bounded for all l and z ∈ L. Hence, we obtain the conclusion of equation 36.Proof of equation 37: For this claim, we have the following inequalityE , 1] + log π l 1 {c=l} dQ(x, y) = K =1 F n,lwhere S y in the above supremum stands for the collection of sets S 1 , . . . , S K such that |S y | ≤ |L| and elements in S y are in R D (0) +1 . Therefore, to achieve the conclusion of equation 37, it is sufficient to demonstrate that F n,l → 0 almost surely as n → ∞ for each 1 ≤ l ≤ K.In fact, for each 1 ≤ l ≤ K, we denote G to be the family of function on R D (0) × {1, . . . , K} of the form f {S y },{πy} (x, y) , 1] + log π l 1 {y=l} for all (x, y) where S 1 , . . . , S K ∈ O |L| , which contains all sets that have at most |L| elements in R D (0) +1 , and {π y } satisfy that K y=1 π y = 1 and π y ≥ γ for all 1 ≤ y ≤ K. Due to the assumption with distribution P , we can restrict O |L| to contain only set S with elements in the ball B(R) of radius R on R D (0) +1 . By means of Lemma 2.5 in (van de Geer

y } K y=1 , we can choose the corresponding element S y K y=1 , {π y } K y=1 ∈ F(δ) such that S y = s y1 , . . . , s yky , S y = s y1 , . . . , s yky satisfy s yjs yj ≤ δ for all 1 ≤ y ≤ K and 1 ≤ j ≤ k y . Additionally, (π 1 , . . . , π K ) -(π 1 , . . . , π K ) ≤ δ. With these notations, we definef {S y },{πy} (x, y) + log π τ ) 1 {y=l} -2δ [x, 1] -2δ/γfor any S y , {π y } ∈ F(δ). By means of Cauchy-Schwarz's inequality, we haves yi [x, 1]1 {y=l} -(s yi ) [x, 1]1 {y=l} ≤ s cis yi [x, 1] 1 {y=l} ≤ δ [x, 1]for all x and 1 ≤ i ≤ k. Additionally, the following also holdss yi [x, 1]1 {y=l} -(s yi ) [x, 1]1 {y=l} ≥ -δ [x, 1] . Furthermore, | log π ylog π y | ≤ log(1 + δ/γ) ≤ δ/γ.Hence, we obtain that exp(maxs∈S y s [x, 1] + log π y ) , 1] + log π y + δ [x, 1] + δ/γ) , 1] + log π τ ) exp(-2δ [x, 1] -2δ/γ).As a consequence, we achieve thatf {S y },{πy} (x, y) ≤ f {S y },{πy} (x, y) ≤ f {S y },{πy} (x, y)for all (x, y). With the formulations of f {S y },{πy} (x, c) and f {S y },{πy} (x, y), we have f {S y },{πy} (x, y)f {S y },{πy} (x, y) dQ(x, y)= 4δ [x, 1] dQ(x, y) + 4δ/γ ≤ 4δx dQ(x, y) + 1 + 4δ/γ.

(y|x i ) log q θ 0 (y|x i ) π y = D nfor all n ≥ 1 where we have the following formulationsh 0 (y, z; 0) = Λ 0 (z; 1) . . . Λ 0 (z; L) µ 0 (y), Λ 0 (z; ) = p∈P( ) s( , p)T (t; , p)B( , p) Γ0 ( , p), p 0 (y , z ) = exp τ ( θ 0 y ,z ) + log π 0From the results with uniform laws of large numbers in equation 33, equation 34, equation 35, equation 36, equation 37, and equation 38, we obtain that h 0 (y , z ; 0) 2 2 log( p 0 (y , z )) dP (xh 0 (y i , z ; 0) 2 2 log( p 0 (y i , z ))

π y dP (x) almost surely as n → ∞. Combining with the fact that n 1 /n → λ, the above results lead to D n → Y almost surely as n → ∞. Therefore, we have lim n→∞ Y n ≤ Y almost surely as n → ∞. On the other hand, with the results of uniform laws of large numbers in equation 33, equation 34, equation 35, equation 36, equation 37, and equation 38, we also have that 1

p)T (t; , p)B( , p) Γ( , p), p(y , z ) = exp τ ( θ y ,z ) + log π y / log( p(y , z )) dP (x) + (1λ)

0 ( ) = (S, {π y } , {b( )}) : S = h(y, z; 0) : (y, z) ∈ J , and inf ( S0,{ π 0 y },{ b0( )})∈G( F0) H(S, S 0 ) + K y=1 |π yπ 0 y | + z∈L b( )b 0 ( ) ≥

in most settings. Max-Min cross-entropy then helps improves NRM's semi-supervised learning results significantly. When combining the NRM, Max-Min cross-entropy, and Mean Teacher, we achieve SOTA results or very close to those on CIFAR10, CIFAR100, and SVHN (see Table2, 3, and 4). Interestingly, compared to the other competitors, our method is consistently good, achieving either best or second best results in all experiments. Furthermore, Max-Min cross-entropy also helps supervised learning. Using the Max-Min cross-entropy, we achieve SOTA result for supervised learning on CIFAR10 (2.30% test error). Similarly, Max-Min cross-entropy helps improve supervised training on ImageNet.

Table content: <table/>



Table content: <table><tr><td>in Appendix A.</td></tr></table>

Correspondence between NRM and CNN. ) and t( ) can be independent, we further constrain the model by enforcing the dependency among s( ) and t( ) at different layers in NRM. This constraint is motivated from realistic rendering of natural objects: different parts of a natural object are dependent on each other. NRM captures such dependency in natural objects by imposing more structures into the joint prior of latent variables at all layer in the model. The form of the join prior π z|y in Eqn. 1 might look mysterious at first, but NRM parametrizes π z|y in this particular way so that π z|y is the conjugate prior of the model likelihood as being proven in Appendix C.11. The conjugate form of π z|y allows efficient inference in the NRM. Parameters b(t; ) of the conjugate prior π z|y will become the bias terms after the CNN's convolutions as will be shown in Theorem 2.2.

Table content: <table><tr><td>NRM (Generation)</td><td>CNN (Inference)</td></tr><tr><td>Rendering templates Γ( )</td><td>Transpose of weights W ( )</td></tr><tr><td>Class templates µ(y)</td><td>Softmax weights</td></tr><tr><td>Parameters b(t; ) of</td><td>Bias terms b( ) after</td></tr><tr><td>the conjugate prior π y|x</td><td>each convolution</td></tr><tr><td>Intermediate image h( ) = h(y, z; ) of size D( ) in layer</td><td>Feature maps in layer in CNNs</td></tr><tr><td>Latent variables z( , p) = {s( , p), t( , p)} at pixel p, layer</td><td>States of ReLUs &amp; MaxPools</td></tr><tr><td>Max over template selecting variable s( , p)</td><td>ReLU</td></tr><tr><td>Max over local translation variable t( , p)</td><td>MaxPool</td></tr><tr><td>Zero-padding matrices</td><td>Downsampling</td></tr><tr><td>B( , p) in layer</td><td>in MaxPool</td></tr><tr><td>Conditional log-likelihood</td><td>Cross-entropy loss</td></tr><tr><td>Expected complete-data log-likelihood</td><td>Reconstruction loss</td></tr></table>

Error rate percentage on CIFAR-100 over 3 runs.

Table content: <table><tr><td>10K labels</td><td>50K labels</td></tr><tr><td>50K images</td><td>50K images</td></tr></table>

Table of notations for NRM

Table content: <table><tr><td/><td>Variables</td></tr><tr><td>x</td><td>input image of size D(0)</td></tr><tr><td>y</td><td>object category</td></tr><tr><td>z( ) = {s( ), t( )} s( , p)</td><td>all latent variables of size D( ) in level switching latent variable at pixel location p in level</td></tr><tr><td>t( , p)</td><td>local translation latent variable at pixel location p in level</td></tr><tr><td>h(y, z;</td><td/></tr></table>

7 NEURAL RENDERING MODEL IS THE UNIFYING FRAMEWORK FOR NETWORKS IN THE CONVNET FAMILYThe structure of the rendering matrices Λ( ) gives rise to MaxPooling, ReLU, and convolution operators in the CNNs. By modifying the structure of Λ( ), we can derive different types of networks in the convolutional neural network family. In this section, we define and explore several other interesting variants of NRM: the Residual NRM (ResNRM) and the Dense NRM (DenseNRM). Inference algorithms in these NRMs yield ResNetHe et al. (2016) and DenseNetHuang et al. (2017), respectively. Proofs for these correspondence are given in Appendix C. Both ResNet and DenseNet are among state-of-the-art neural networks for object recognition and popularly used for other visual perceptual inference tasks. These two architectures employ skip connections (a.k.a., shortcuts) to create short paths from early layers to later layers. During training, the short paths help avoid the vanishing-gradient problem and allow the network to propagate and reuse features..7.1 RESIDUAL NEURAL RENDERING MODEL YIELDS RESNET In a ResNet, layers learn residual functions with reference to the layer inputs. In particular, as illustrated in Fig.5, layers in a ResNet are reformulated to represent the mapping F (ψ) + W skip ψ and the layers try to fit the residual mapping F (ψ) where ψ is the input feature. The term W skip ψ accounts for the skip connections/shortcutsHe et al. (2016). In order to derive the ResNet, we rewrite the rendering matrix Λ( ) as the sum of a shortcut matrix Λ skip ( ) and a rendering matrix, both of which can be updated during the training. The shortcut matrices yields skip connections inHe et al. (2016). Note that Λ skip ( ) depends on the template selecting latent variables s( ). In the rest of this section, for clarity, we will refer to Λ( ) and Λ skip ( ) as Λ(t, s; ) and Λ skip (s; ), respectively, to show their dependency on latent variables in NRM. We define the Residual Neural Rendering Model as follows: Definition .10. The Residual Neural Rendering Model (ResNRM) is the Neural Rendering Model whose rendering process from layer to layer -1, for some ∈ {1, 2, • • • , L}, has the residual form as follows:

Table content: <table><tr><td>h( -1) := (Λ(t, s; ) + Λ skip (s; )) h( ),</td></tr></table>

In Eqn. 21, we concatenate the output MaxPool ReLu(Conv(W ( ), ψ( -1)) + b( ))) at layer with the input feature ψ( -1) at layer -1 to generate the input to the next layer ψ( ), just like in the DenseNet. Proofs for Theorem .11 and .13 can be found in Appendix B. The approach to proving Theorem .11 can be used to prove the result in Eqn. 19.

Table content: <table><tr><td colspan="3">CNNs, i.e. MaxPool (ReLu (Conv())). Indeed,</td><td/></tr><tr><td colspan="3">ψ(1) = (ψ(1, p(1))) p(1)∈P(1)</td><td/></tr><tr><td>=</td><td>max t(1,p1),s(1,p1)</td><td colspan="2">s(1, p 1 ) Γ (1, p 1 )B (1, p 1 )T (t; 1, p 1 )x + b(t; 1, p 1 )</td><td>p(1)∈P(1)</td></tr><tr><td colspan="5">= max ReLu Γ (1, p 1 )B (1, p 1 )T (t; 1, p 1 )x + b(t; 1, p 1 ) t(1,p1)=0,1,2,3 p(1)∈P(1)</td></tr><tr><td/><td/><td/><td/><td>(22)</td></tr><tr><td colspan="3">= MaxPool ReLu Conv Γ (1), x + b(t; 1)</td><td/></tr><tr><td colspan="3">d = MaxPool (ReLu (Conv (W (1), x) + b(t; 1))) ,</td><td/></tr><tr><td/><td/><td>x0</td><td/></tr><tr><td/><td/><td>H1</td><td/></tr><tr><td/><td/><td>x1</td><td/></tr><tr><td/><td/><td>H2</td><td/></tr><tr><td/><td/><td>x2</td><td/></tr><tr><td colspan="3">H3 where W (1) = Γ (1) corresponds to the weights at layer 1 of the CNN, and x3</td><td colspan="2">d = implies that discrimi-</td></tr><tr><td colspan="3">H4 native relaxation is applied. In Eqn. 22, since s(1, p 1 ) ∈ {0, 1}, max x4 s(1,p1)</td><td/></tr><tr><td/><td colspan="3">Figure 6: DenseNet building block as in Huang et al. (2017)</td></tr></table>

10 PROOF FOR THEOREM .13: DERIVING THE DENSELY CONNECTED NETWORKS FROM THE DENSE NEURAL RENDERING MODEL Similar to the proof for Theorem .11 above, when the rendering process at each layer in ResNRM is as in Eqn. 20, the activations ψ( ) is given by:

Table content: <table><tr><td>ψ( ) ≡</td><td> </td><td>max s( ,p),t( ,p)</td><td>ψ( -1) s( , p) Γ ( , p)B ( , p)T (t; , p)ψ( -1) + b(t; , p)</td><td>p∈P( )</td><td> </td></tr><tr><td>=</td><td/><td colspan="4">max ReLu Γ ( , p)B ( , p)T (t; , p)ψ( -1) + b(t; , p) t( ,p)=0,1,2,3 p∈P( )</td></tr></table>

D (0) with the form f S (x) = max |J | , which contains all sets that have at most |J | elements in R D (0) +1 . Due to the assumption with distribution P , we can restrict O |J | to contain only set S with elements in B(R), which is a closed ball of radius R on R D (0) +1 . By means of Lemma 2.5 in (van de Geer, 2000), we can find a finite set E δ such that each element in B(R) is within distance δ to some element of E δ for all δ > 0. We denote O |J | (δ) to be the subset of O |J | such that it only contains sets with elements in E δ for all δ > 0. Now, for each set S = {s 1 , . . . , s k } ∈ O |J | , we can choose corresponding set S = {s 1 , . . . , s k

Table content: <table/>

sup (y, z; 0)x i + τ (θ y,z ) + log π y where σ 1 , . . . , σ n are i.i.d. Rademacher random variables. Since γ is the lower bound of π y for all 1 ≤ y ≤ K, we obtain that

Table content: <table><tr><td/><td>θ</td><td colspan="5">sup |L(τ n)|≤τ n|L|</td><td>1 n</td><td>n i=1</td><td>σ i</td><td>max g∈L(τ n )</td></tr><tr><td/><td/><td>θ</td><td colspan="5">sup |L(τ n )|≤τ n |L|</td><td>1 n</td><td>n i=1</td><td>σ i</td><td>max g∈L(τ n)</td><td>h (y, z; 0)x i + τ (θ y,z )</td></tr><tr><td/><td>+ E</td><td>θ</td><td colspan="5">sup |L(τ n )|≤τ n |L|</td><td>1 n</td><td>n i=1</td><td>σ i log π y</td></tr><tr><td/><td colspan="2">≤ E sup θ</td><td colspan="5">sup |L(τ n )|≤τ n |L|</td><td>1 n</td><td>n i=1</td><td>σ i</td><td>max g∈L(τ n)</td><td>h (y, z; 0)x i + τ (θ y,z )</td><td>+</td><td>| log γ| √ n</td></tr><tr><td colspan="10">Furthermore, we have the following inequalities</td></tr><tr><td>E sup θ</td><td colspan="3">sup |L(τ n )|≤τ n |L|</td><td>1 n</td><td>n i=1</td><td>σ i</td><td/><td colspan="2">max g∈L(τ n )</td></tr></table>

Peer review (iclr2019):
 Summary: This paper introduces the Neural Rendering Model (NRM), a generative model in which the computations involved in inference correspond to those of a CNN forward pass. The NRM’s supervised learning objective is lower bounded by a variant of the cross-entropy objective. This objective is used to formulate a max-min network, which has a particular type of weight sharing between a standard branch with max pooling / ReLUs and a second branch with min pooling / NReLUs. The max-min objective and network show strong performance on semi-supervised learning tasks. Posing a CNN as inference in a generative model is an interesting direction, and could be very useful for probabilistic inference in the context of neural nets. However, the paper is rather difficult to follow and requires frequent reference to the appendix to understand the main body. Some important components (like those relating to rendering paths and RPNs) are given good intuitive explanations early on but remain a bit ambiguous throughout the paper. I would recommend improving the presentation before publication. Question: “we can modify NRM to incorporate our knowledge of the tasks and datasets into the model and perform JMAP inference to achieve a new CNN architecture.“ I appreciate the CNN / NRM correspondence in Table 1, and see how the NRM may be modified to produce modified CNN architectures. That being said, I am not sure I understand what sorts of task-specific knowledge are being referred to here. Could you give an example of a type of knowledge that the NRM would allow you to bake into a CNN architecture, but would otherwise be difficult to incorporate? Minor: “As been shown later in Section 2.2…” “…is part of the optimization in equation equation 6.”