Title: PREDICTIVE UNCERTAINTY THROUGH QUANTIZATION

Abstract: High-risk domains require reliable confidence estimates from predictive models. Deep latent variable models provide these, but suffer from the rigid variational distributions used for tractable inference, which err on the side of overconfidence. We propose Stochastic Quantized Activation Distributions (SQUAD), which imposes a flexible yet tractable distribution over discretized latent variables. The proposed method is scalable, self-normalizing and sample efficient. We demonstrate that the model fully utilizes the flexible distribution, learns interesting non-linearities, and provides predictive uncertainty of competitive quality.

:

In high-risk domains, prediction errors come at high costs. Luckily such domains often provide a fail-safe: self-driving cars perform an emergency stop, doctors run another diagnostic test, and industrial processes are temporarily halted. For deep learning models, this can be achieved by rejecting datapoints with a confidence score below a predetermined threshold. This way, a low error rate can be guaranteed at the cost of rejecting some predictions. However, estimating high quality confidence scores from neural networks, which create well-ordered rankings of correct and incorrect predictions, remains an active area of research.

Deep Latent Variable Models (DLVMs, fig. 1 ) approach this by postulating latent variables z for which the uncertainty in p(z|x) influences the confidence in the target prediction. Recently, efficient inference algorithms have been proposed in the form of variational inference, where an inference neural network is optimized to predict parameters of a variational distribution that approximates an otherwise intractable distribution (Kingma & Welling (2013) q G g Y M Q L 7 X 3 o w u 1 F F t K Z 8 U j K 5 Q k w 1 c t V N 9 U g K n 6 j s M y C 2 9 y 1 G j O o 5 N B J G y u 1 Q q E K k g 7 S a z E 9 w U 8 5 l B r k C S Q 9 m c q j m o S t Y u Z Q b / p 8 + p F i K 7 E T p 0 r X M i I l E U T A v q f 2 Y s c g E / x Z z M s M i D B H 3 E n u I 0 8 T O O m P E k m G a l m m v S H t V 2 i 3 S b p V e n C + o 6 y c X V T g s w F t d T w v w t A r n B T i v Q h 0 U 6 P U D q k o u C 4 r L K j w r w L M q n B X g r A o P 3 S V 0 k 0 M D 6 1 u w i z g l 7 K W C y P N o 9 q j L X 1 X T 6 W W l i O c t i s g w 8 a o d p i w d t Z 2 k O / 5 7 5 Z 9 6 q c 2 I r + E I N i x 4 d Q U b b W h L O g k 0 d C r m Q W H v g 3 L l V z a m 0 k w G b 2 S e q p 3 t a r S c W / v b O E C y m f 3 d c Z J 6 8 b X Z X H h k p A I U k f 1 r / 6 4 n 0 W y X c k 4 k N K f c b 5 u p A / M C O z B p W O k 7 0 9 Z M M 6 s 6 u 2 4 H J + 2 m 1 W p a n 1 8 3 D t 4 v 5 t o 6 e A 5 e g G 1 g g T f g A H w E R 6 A P M G D g O / g J f t X e 1 r 7 W J j V 6 L V 1 d W X i e g d K q y T 9 6 X P W P < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o m h p C f m S D g P n t J K R 3 V 0 0 E O H 6 5 f s = " > A A A F u H i c l Z T f b t M w F M a 9 s c I o / z a 4 5 M a i m t j Q q G g Y M Q L 7 X 3 o w u 1 F F t K Z 8 U j K 5 Q k w 1 c t V N 9 U g K n 6 j s M y C 2 9 y 1 G j O o 5 N B J G y u 1 Q q E K k g 7 S a z E 9 w U 8 5 l B r k C S Q 9 m c q j m o S t Y u Z Q b / p 8 + p F i K 7 E T p 0 r X M i I l E U T A v q f 2 Y s c g E / x Z z M s M i D B H 3 E n u I 0 8 T O O m P E k m G a l m m v S H t V 2 i 3 S b p V e n C + o 6 y c X V T g s w F t d T w v w t A r n B T i v Q h 0 U 6 P U D q k o u C 4 r L K j w r w L M q n B X g r A o P 3 S V 0 k 0 M D 6 1 u w i z g l 7 K W C y P N o 9 q j L X 1 X T 6 W W l i O c t i s g w 8 a o d p i w d t Z 2 k O / 5 7 5 Z 9 6 q c 2 I r + E I N i x 4 d Q U b b W h L O g k 0 d C r m Q W H v g 3 L l V z a m 0 k w G b 2 S e q p 3 t a r S c W / v b O E C y m f 3 d c Z J 6 8 b X Z X H h k p A I U k f 1 r / 6 4 n 0 W y X c k 4 k N K f c b 5 u p A / M C O z B p W O k 7 0 9 Z M M 6 s 6 u 2 4 H J + 2 m 1 W p a n 1 8 3 D t 4 v 5 t o 6 e A 5 e g G 1 g g T f g A H w E R 6 A P M G D g O / g J f t X e 1 r 7 W J j V 6 L V 1 d W X i e g d K q y T 9 6 X P W P < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o m h p C f m S D g P n t J K R 3 V 0 0 E O H 6 5 f s = " > A A A F u H i c l Z T f b t M w F M a 9 s c I o / z a 4 5 M a i m t j Q 

V j U V E h V o 0 g S d h E Q v B l

V j U V E h V o 0 g S d h E Q v B l

F k S 8 1 C q G g Y M Q L 7 X 3 o w u 1 F F t K Z 8 U j K 5 Q k w 1 c t V N 9 U g K n 6 j s M y C 2 9 y 1 G j O o 5 N B J G y u 1 Q q E K k g 7 S a z E 9 w U 8 5 l B r k C S Q 9 m c q j m o S t Y u Z Q b / p 8 + p F i K 7 E T p 0 r X M i I l E U T A v q f 2 Y s c g E / x Z z M s M i D B H 3 E n u I 0 8 T O O m P E k m G a l m m v S H t V 2 i 3 S b p V e n C + o 6 y c X V T g s w F t d T w v w t A r n B T i v Q h 0 U 6 P U D q k o u C 4 r L K j w r w L M q n B X g r A o P 3 S V 0 k 0 M D 6 1 u w i z g l 7 K W C y P N o 9 q j L X 1 X T 6 W W l i O c t i s g w 8 a o d p i w d t Z 2 k O / 5 7 5 Z 9 6 q c 2 I r + E I N i x 4 d Q U b b W h L O g k 0 d C r m Q W H v g 3 L l V z a m 0 k w G b 2 S e q p 3 t a r S c W / v b O E C y m f 3 d c Z J 6 8 b X Z X H h k p A I U k f 1 r / 6 4 n 0 W y X c k 4 k N K f c b 5 u p A / M C O z B p W O k 7 0 9 Z M M 6 s 6 u 2 4 H J + 2 m 1 W p a n 1 8 3 D t 4 v 5 t o 6 e A 5 e g G 1 g g T f g A H w E R 6 A P M G D g O / g J f t X e 1 r 7 W J j V 6 L V 1 d W X i e g d K q y T 9 6 X P W P < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " o m h p C f m S D g P n t J K R 3 V 0 0 E O H 6 5 f s = " > A A A F u H i c l Z T f b t M w F M a 9 s c I o / z a 4 5 M a i m t j Q V j U V E h V o 0 g S d h E Q v B l r X b k 1 U H M d p r D p 2 s J 1 1 X Z a X 4 B J 4 M d 4 G J 2 t Z k o k L L F U 6 P b / v O 8 d 2 5 O N G j C r d a v 1 e W b 2 z V r t 7 b / 1 + / c H D R 4 + f b G w + P V E i l p j 0 s W B C D l 2 k C K O c 9 D X V j A w j S V D o M j J w p x 8 y P j g n U l H B j / U 8 I k 6 I J p z 6 F C N t U s N o b O u A a D T e a L S a r X z B 2 4 G 1 C B p g s Y 7 G m 2 s / b E / g O C R c Y 4 a U G l m t S D s J k p p i R t K 6 H S s S I T x F E z I y I U c h U U 6 S b z i F W y b j Q V 9 I 8 + M a 5 t m y Q / s d J 6 E 8 i j X h 2 B g g Y k z M o M n v d W C e L + m P L S f J S u V i s 7 a g g b C z 5 1 I N j 6 2 8 i y o a k s D c h Z T E z 9 W 5 I c + Y a 5 y W h b F k S 8 1 C q G g Y M Q L 7 X 3 o w u 1 F F t K Z 8 U j K 5 Q k w 1 c t V N 9 U g K n 6 j s M y C 2 9 y 1 G j O o 5 N B J G y u 1 Q q E K k g 7 S a z E 9 w U 8 5 l B r k C S Q 9 m c q j m o S t Y u Z Q b / p 8 + p F i K 7 E T p 0 r X M i I l E U T A v q f 2 Y s c g E / x Z z M s M i D B H 3 E n u I 0 8 T O O m P E k m G a l m m v S H t V 2 i 3 S b p V e n C + o 6 y c X V T g s w F t d T w v w t A r n B T i v Q h 0 U 6 P U D q k o u C 4 r L K j w r w L M q n B X g r A o P 3 S V 0 k 0 M D 6 1 u w i z g l 7 K W C y P N o 9 q j L X 1 X T 6 W W l i O c t i s g w 8 a o d p i w d t Z 2 k O / 5 7 5 Z 9 6 q c 2 I r + E I N i x 4 d Q U b b W h L O g k 0 d C r m Q W H v g 3 L l V z a m 0 k w G b 2 S e q p 3 t a r S c W / v b O E C y m f 3 d c Z J 6 8 b X Z X H h k p A I U k f 1 r / 6 4 n 0 W y X c k 4 k N K f c b 5 u p A / M C O z B p W O k 7 0 9 Z M M 6 s 6 u 2 4 H J + 2 m 1 W p a n 1 8 3 D t 4 v 5 t o 6 e A 5 e g G 1 g g T f g A H w E R 6 A P M G D g O / g J f t X e 1 r 7 W J j V 6 L V 1 d W X i e g d K

q y T 9 6 X P W P < / l a t e x i t > q < l a t e x i t s h a 1 _ b a s e 6 4 = " 0

I O t 3 c b E R N C Z O J 4 B 7 q y w M + J N p d s = " > A A A F t n i c l Z T f b 9 M w E M c 9 W G G U X x s 8 8 m J R T W x o q 5 o K i U p o 0 g S d h E Q f B l r X b k 2 o H M d p T B 0 7 s 9 1 1 X Z b / g V f 2 n / H f 4 G Q p S z L x w E m R z v f 5 3 p 3 t y O d G j C r d a v 1 e u X d / t f b g 4 d q j + u M n T 5 8 9 X 9 9 4 c a z E T G L S x 4 I J O X S R I o x y 0 t d U M z K M J E G h y 8 j A n X 5 K + e C c S E U F P 9 K L i D g h m n D q U 4 y 0 C R 2 f j e 0 o o O P 1 R q v Z y g z e d a z c a Y D c D s c b q 9 e 2 J / A s J F x j h p Q a W a 1 I O z G S m m J G k r o 9 U y R C e I o m Z G R c j k K i n D j b b g I 3 T c S D v p D m 4 x p m 0 X K G 9 j t O T H k 0 0 4 R j k w A R Y 2 I O T X y 3 A 7 N 4 S X 9 k O X F a K h M b 2 4 Q G w s 6 u S z U 8 s r I u q p g Q B + Y m p C R + p s 4 S s o i 5 x G l Z O J N s q c m F i o Y R I 7 D / r Q f T + 1 R E a 8 o n p S R X i K l G r r q t H k n h E 5 X + B M R 2 z 2 a I U b 2 A R s J I u R 0 K V Y h 0 k F S D 2 Q l u y 7 n M I F c g 6 c F U D t U i d A U r l 3 L D / 9 O H F E u R n i h Z Z i 0 j Y i J R F C x K a n / G W G S c f 4 s 5 m W M R h o h 7 s T 3 E S W y n n T F i 8 T B J y r R X p L 0 q 7 R Z p t 0 o v z n P q + v F F F Q 4 L 8 E 7 X k w I 8 q c J F A S 6 q U A c F a l Z E o 6 r k s q C 4 r M L T A j y t w n k B z q v w w F 1 C N z 4 w s L 4 J u 4 h T w t 4 o i D y P p k + 6 / F c 1 n V 5 W i n h e X k S G s V f t M G X J q O 3 E 3 f H f K / / S S 2 x G f A 1 H s G H B q y v Y a E N b 0 k m g o V N J H h T 2 P i h X f m t j K s 1 k 8 E b m q d r p r k b L q b W 3 h Q M k m + l y 2 4 n r x d d m c + G R k Q p Q R P Z u 8 n c 8 i e Y 7 l H M i o T n l X t t M H Z g V 2 I Z x w 0 o + m L Z m m l n V 2 X X X O W 4 3 r V b T + v q u s f 8 x n 2 t r 4 B V 4 D b a A B d 6 D f f A Z H I I + w O A H + A l + g e t a p / a 9 R m q T G + m 9 l T z n J S h Z L f o D v 4 7 0 p w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 0 I O t 3 c b E R N C Z O J 4 B 7 q y w M + J N p d s = " > A A A F t n i c l Z T f b 9 M w E M c 9 W G G U X x s 8 8 m J R T W x o q 5 o K i U p o 0 g S d h E Q f B l r X b k 2 o H M d p T B 0 7 s 9 1 1 X Z b / g V f 2 n / H f 4 G Q p S z L x w E m R z v f 5 3 p 3 t y O d G j C r d a v 1 e u X d / t f b g 4 d q j + u M n T 5 8 9 X 9 9 4 c a z E T G L S x 4 I J O X S R I o x y 0 t d U M z K M J E G h y 8 j A n X 5 K + e C c S E U F P 9 K L i D g h m n D q U 4 y 0 C R 2 f j e 0 o o O P 1 R q v Z y g z e d a z c a Y D c D s c b q 9 e 2 J / A s J F x j h p Q a W a 1 I O z G S m m J G k r o 9 U y R C e I o m Z G R c j k K i n D j b b g I 3 T c S D v p D m 4 x p m 0 X K G 9 j t O T H k 0 0 4 R j k w A R Y 2 I O T X y 3 A 7 N 4 S X 9 k O X F a K h M b 2 4 Q G w s 6 u S z U 8 s r I u q p g Q B + Y m p C R + p s 4 S s o i 5 x G l Z O J N s q c m F i o Y R I 7 D / r Q f T + 1 R E a 8 o n p S R X i K l G r r q t H k n h E 5 X + B M R 2 z 2 a I U b 2 A R s J I u R 0 K V Y h 0 k F S D 2 Q l u y 7 n M I F c g 6 c F U D t U i d A U r l 3 L D / 9 O H F E u R n i h Z Z i 0 j Y i J R F C x K a n / G W G S c f 4 s 5 m W M R h o h 7 s T 3 E S W y n n T F i 8 T B J y r R X p L 0 q 7 R Z p t 0 o v z n P q + v F F F Q 4 L 8 E 7 X k w I 8 q c J F A S 6 q U A c F a l Z E o 6 r k s q C 4 r M L T A j y t w n k B z q v w w F 1 C N z 4 w s L 4 J u 4 h T w t 4 o i D y P p k + 6 / F c 1 n V 5 W i n h e X k S G s V f t M G X J q O 3 E 3 f H f K / / S S 2 x G f A 1 H s G H B q y v Y a E N b 0 k m g o V N J H h T 2 P i h X f m t j K s 1 k 8 E b m q d r p r k b L q b W 3 h Q M k m + l y 2 4 n r x d d m c + G R k Q p Q R P Z u 8 n c 8 i e Y 7 l H M i o T n l X t t M H Z g V 2 I Z x w 0 o + m L Z m m l n V 2 X X X O W 4 3 r V b T + v q u s f 8 x n 2 t r 4 B V 4 D b a A B d 6 D f f A Z H I I + w O A H + A l +

g e t a p / a 9 R m q T G + m 9 l T z n J S h Z L f o D v 4 7 0 p w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 0

I O t 3 c b E R N C Z O J 4 B 7 q y w M + J N p d s = " > A A A F t n i c l Z T f b 9 M w E M c 9 W G G U X x s 8 8 m J R T W x o q 5 o K i U p o 0 g S d h E Q f B l r X b k 2 o H M d p T B 0 7 s 9 1 1 X Z b / g V f 2 n / H f 4 G Q p S z L x w E m R z v f 5 3 p 3 t y O d G j C r d a v 1 e u X d / t f b g 4 d q j + u M n T 5 8 9 X 9 9 4 c a z E T G L S x 4 I J O X S R I o x y 0 t d U M z K M J E G h y 8 j A n X 5 K + e C c S E U F P 9 K L i D g h m n D q U 4 y 0 C R 2 f j e 0 o o O P 1 R q v Z y g z e d a z c a Y D c D s c b q 9 e 2 J / A s J F x j h p Q a W a 1 I O z G S m m J G k r o 9 U y R C e I o m Z G R c j k K i n D j b b g I 3 T c S D v p D m 4 x p m 0 X K G 9 j t O T H k 0 0 4 R j k w A R Y 2 I O T X y 3 A 7 N 4 S X 9 k O X F a K h M b 2 4 Q G w s 6 u S z U 8 s r I u q p g Q B + Y m p C R + p s 4 S s o i 5 x G l Z O J N s q c m F i o Y R I 7 D / r Q f T + 1 R E a 8 o n p S R X i K l G r r q t H k n h E 5 X + B M R 2 z 2 a I U b 2 A R s J I u R 0 K V Y h 0 k F S D 2 Q l u y 7 n M I F c g 6 c F U D t U i d A U r l 3 L D / 9 O H F E u R n i h Z Z i 0 j Y i J R F C x K a n / G W G S c f 4 s 5 m W M R h o h 7 s T 3 E S W y n n T F i 8 T B J y r R X p L 0 q 7 R Z p t 0 o v z n P q + v F F F Q 4 L 8 E 7 X k w I 8 q c J F A S 6 q U A c F a l Z E o 6 r k s q C 4 r M L T A j y t w n k B z q v w w F 1 C N z 4 w s L 4 J u 4 h T w t 4 o i D y P p k + 6 / F c 1 n V 5 W i n h e X k S G s V f t M G X J q O 3 E 3 f H f K / / S S 2 x G f A 1 H s G H B q y v Y a E N b 0 k m g o V N J H h T 2 P i h X f m t j K s 1 k 8 E b m q d r p r k b L q b W 3 h Q M k m + l y 2 4 n r x d d m c + G R k Q p Q R P Z u 8 n c 8 i e Y 7 l H M i o T n l X t t M H Z g V 2 I Z x w 0 o + m L Z m m l n V 2 X X X O W 4 3 r V b T + v q u s f 8 x n 2 t r 4 B V 4 D b a A B d 6 D f f A Z H I I + w O A H + A l +

g e t a p / a 9 R m q T G + m 9 l T z n J S h Z L f o D v 4 7 0 p w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 0

I O t 3 c b E R N C Z O J 4 B 7 q y w M + J N p d s = " > A A A F t n i c l Z T f b 9 M w E M c 9 W G G U X x s 8 8 m J R T W x o q 5 o K i U p o 0 g S d h E Q f B l r X b k 2 o H M d p T B 0 7 s 9 1 1 X Z b / g V f 2 n / H f 4 G Q p S z L x w E m R z v f 5 3 p 3 t y O d G j C r d a v 1 e u X d / t f b g 4 d q j + u M n T 5 8 9 X 9 9 4 c a z E T G L S x 4 I J O X S R I o x y 0 t d U M z K M J E G h y 8 j A n X 5 K + e C c S E U F P 9 K L i D g h m n D q U 4 y 0 C R 2 f j e 0 o o O P 1 R q v Z y g z e d a z c a Y D c D s c b q 9 e 2 J / A s J F x j h p Q a W a 1 I O z G S m m J G k r o 9 U y R C e I o m Z G R c j k K i n D j b b g I 3 T c S D v p D m 4 x p m 0 X K G 9 j t O T H k 0 0 4 R j k w A R Y 2 I O T X y 3 A 7 N 4 S X 9 k O X F a K h M b 2 4 Q G w s 6 u S z U 8 s r I u q p g Q B + Y m p C R + p s 4 S s o i 5 x G l Z O J N s q c m F i o Y R I 7 D / r Q f T + 1 R E a 8 o n p S R X i K l G r r q t H k n h E 5 X + B M R 2 z 2 a I U b 2 A R s J I u R 0 K V Y h 0 k F S D 2 Q l u y 7 n M I F c g 6 c F U D t U i d A U r l 3 L D / 9 O H F E u R n i h Z Z i 0 j Y i J R F C x K a n / G W G S c f 4 s 5 m W M R h o h 7 s T 3 E S W y n n T F i 8 T B J y r R X p L 0 q 7 R Z p t 0 o v z n P q + v F F F Q 4 L 8 E 7 X k w I 8 q c J F A S 6 q U A c F a l Z E o 6 r k s q C 4 r M L T A j y t w n k B z q v w w F 1 C N z 4 w s L 4 J u 4 h T w t 4 o i D y P p k + 6 / F c 1 n V 5 W i n h e X k S G s V f t M G X J q O 3 E 3 f H f K / / S S 2 x G f A 1 H s G H B q y v Y a E N b 0 k m g o V N J H h T 2 P i h X f m t j K s 1 k 8 E b m q d r p r k b L q b W 3 h Q M k m + l y 2 4 n r x d d m c + G R k Q p Q R P Z u 8 n c 8 i e Y 7 l H M i o T n l X t t M H Z g V 2 I Z x w 0 o + m L Z m m l n V 2 X X X O W 4 3 r V b T + v q u s f 8 x n 2 t r 4 B V 4 D b a A B d 6 D f f A Z H I I + w O A H + A l + g e t a p / a 9 R m q T G + m 9 l T z n J S h Z L f o D v 4 7 0 p w = = < / l a t e x i t > ✓ ⇤ < l a t e x i t s h a 1 _ b a s e 6 4 = " L z S x C 6 2 q K q 7 o 4 x a W W X m a V V s w m 3 4 = " > A A A F x 3 i c l Z T f b t M w F M Y 9 W G G U f x u I K 2 4 s q o l t 2 q q m m s Q k N G k S n Q S i F w O t a 7 c m V I 7 j N F a d O N j O u s 7 L B a / B H b f w R L w N T t a y J B M X W I p 0 c n 7 f + U 7 s y M e N G Z W q 1 f q 9 d O f u c u 3 e / Z U H 9 Y e P H j 9 5 u r r 2 7 E T y R G D S w 5 x x M X C R J I x G p K e o Y m Q Q C 4 J C l 5 G + O 3 m X 8 f 4 5 E Z L y 6 F j N Y u K E a B x R n 2 K k T G q 0 + s L O P b T L E p J C W w V E o S 9 b o 9 V G q 9 n K F 7 w d W P O g A e b r a L S 2 / N 3 2 O E 5 C E i n M k J R D q x U r R y O h K G Y k r d u J J D H C E z Q m Q x N G K C T S 0 X n v F K 6 b j A d 9 L s w T K Z h n y x X K 3 3 M 0 j e J E k Q i b A o g Y 4 1 N o 8 j t 7 M M + X 9 M e W o z O r X G z W O j Q Q 7 u 2 4 V M F j K + 8 i i w U 6 M E c j B P F z d V 6 Q Z 8 y p T s r C R L C F Z i 6 U N I w Z g b 3 P X Z g d s C R K 0 W h c K n I 5 n y j k y h v 3 W H C f y O y v I L b z N U G M q h k 0 E k b K 7 V A o Q 6 S C t J r M d 3 B j 5 z K D X I 6 E B z M 5 l L P Q 5 a x s 5 Y b / p w 8 p F j z b U b q o W m T 4 W K A 4 m J X U f s J Y b I J / i y M y x T w M U e R p e 4 B T b W e d M W J 6 k K Z l 2 i 3 S b p V 2 i r R T p R f n c + r 6 + q I K B w V 4 q + t p A Z 5 W 4 a w A Z 1 W o g g K 9 v k J V y W V B c V m F Z w V 4 V o X T A p x W 4 a G 7 g K 4 + N L C + D j s o o o S 9 l h B 5 H s 3 u e P m v K j q 5 r J h 4 3 t x E h N q r d p i w d N h 2 d G f 0 9 8 g / d l O b E V / B I W x Y 8 O o K N t r Q F n Q c K O h U i v u F b + + X n b d s T I W Z D N 7 Q X F U 7 + 6 r h Y o z t b + A A i W b 2 u u n o e v G 2 2 R H 3 y F A G K C b 7 1 / X b n k D T b R p F R E C z y / 2 2 m T o w N 9 i E u m G l b 0 1 b M 8 2 s 6 u y 6 H Z y 0 m 1 a r a X 3 a b R z s z u f a C n g J X o E N Y I E 3 4 A C 8 B 0 e g B z D Q 4 A f 4 C X 7 V P t R 4 7 b x 2 c S 2 9 s z S v e Q 5 K q / b t D w N o + y g = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L z S x C 6 2 q K q 7 o 4 x a W W X m a V V s w m 3 4 = " > A A A F x 3 i c l Z T f b t M w F M Y 9 W G G U f x u I K 2 4 s q o l t 2 q q m m s Q k N G k S n Q S i F w O t a 7 c m V I 7 j N F a d O N j O u s 7 L B a / B H b f w R L w N T t a y J B M X W I p 0 c n 7 f + U 7 s y M e N G Z W q 1 f q 9 d O f u c u 3 e / Z U H 9 Y e P H j 9 5 u r r 2 7 E T y R G D S w 5 x x M X C R J I x G p K e o Y m Q Q C 4 J C l 5 G + O 3 m X 8 f 4 5 E Z L y 6 F j N Y u K E a B x R n 2 K k T G q 0 + s L O P b T L E p J C W w V E o S 9 b o 9 V G q 9 n K F 7 w d W P O g A e b r a L S 2 / N 3 2 O E 5 C E i n M k J R D q x U r R y O h K G Y k r d u J J D H C E z Q m Q x N G K C T S 0 X n v F K 6 b j A d 9 L s w T K Z h n y x X K 3 3 M 0 j e J E k Q i b A o g Y 4 1 N o 8 j t 7 M M + X 9 M e W o z O r X G z W O j Q Q 7 u 2 4 V M F j K + 8 i i w U 6 M E c j B P F z d V 6 Q Z 8 y p T s r C R L C F Z i 6 U N I w Z g b 3 P X Z g d s C R K 0 W h c K n I 5 n y j k y h v 3 W H C f y O y v I L b z N U G M q h k 0 E k b K 7 V A o Q 6 S C t J r M d 3 B j 5 z K D X I 6 E B z M 5 l L P Q 5 a x s 5 Y b / p w 8 p F j z b U b q o W m T 4 W K A 4 m J X U f s J Y b I J / i y M y x T w M U e R p e 4 B T b W e d M W J 6 k K Z l 2 i 3 S b p V 2 i r R T p R f n c + r 6 + q I K B w V 4 q + t p A Z 5 W 4 a w A Z 1 W o g g K 9 v k J V y W V B c V m F Z w V 4 V o X T A p x W 4 a G 7 g K 4 + N L C + D j s o o o S 9 l h B 5 H s 3 u e P m v K j q 5 r J h 4 3 t x E h N q r d p i w d N h 2 d G f 0 9 8 g / d l O b E V / B I W x Y 8 O o K N t r Q F n Q c K O h U i v u F b + + X n b d s T I W Z D N 7 Q X F U 7 + 6 r h Y o z t b + A A i W b 2 u u n o e v G 2 2 R H 3 y F A G K C b 7 1 / X b n k D T b R p F R E C z y / 2 2 m T o w N 9 i E u m G l b 0 1 b M 8 2 s 6 u y 6 H Z y 0 m 1 a r a X 3 a b R z s z u f a C n g J X o E N Y I E 3 4 A C 8 B 0 e g B z D Q 4 A f 4 C X 7 V P t R 4 7 b x 2 c S 2 9 s z S v e Q 5 K q / b t D w N o + y g = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L z S x C 6 2 q K q 7 o 4 x a W W X m a V V s w m 3 4 = " > A A A F x 3 i c l Z T f b t M w F M Y 9 W G G U f x u I K 2 4 s q o l t 2 q q m m s Q k N G k S n Q S i F w O t a 7 c m V I 7 j N F a d O N j O u s 7 L B a / B H b f w R L w N T t a y J B M X W I p 0 c n 7 f + U 7 s y M e N G Z W q 1 f q 9 d O f u c u 3 e / Z U H 9 Y e P H j 9 5 u r r 2 7 E T y R G D S w 5 x x M X C R J I x G p K e o Y m Q Q C 4 J C l 5 G + O 3 m X 8 f 4 5 E Z L y 6 F j N Y u K E a B x R n 2 K k T G q 0 + s L O P b T L E p J C W w V E o S 9 b o 9 V G q 9 n K F 7 w d W P O g A e b r a L S 2 / N 3 2 O E 5 C E i n M k J R D q x U r R y O h K G Y k r d u J J D H C E z Q m Q x N G K C T S 0 X n v F K 6 b j A d 9 L s w T K Z h n y x X K 3 3 M 0 j e J E k Q i b A o g Y 4 1 N o 8 j t 7 M M + X 9 M e W o z O r X G z W O j Q Q 7 u 2 4 V M F j K + 8 i i w U 6 M E c j B P F z d V 6 Q Z 8 y p T s r C R L C F Z i 6 U N I w Z g b 3 P X Z g d s C R K 0 W h c K n I 5 n y j k y h v 3 W H C f y O y v I L b z N U G M q h k 0 E k b K 7 V A o Q 6 S C t J r M d 3 B j 5 z K D X I 6 E B z M 5 l L P Q 5 a x s 5 Y b / p w 8 p F j z b U b q o W m T 4 W K A 4 m J X U f s J Y b I J / i y M y x T w M U e R p e 4 B T b W e d M W J 6 k K Z l 2 i 3 S b p V 2 i r R T p R f n c + r 6 + q I K B w V 4 q + t p A Z 5 W 4 a w A Z 1 W o g g K 9 v k J V y W V B c V m F Z w V 4 V o X T A p x W 4 a G 7 g K 4 + N L C + D j s o o o S 9 l h B 5 H s 3 u e P m v K j q 5 r J h 4 3 t x E h N q r d p i w d N h 2 d G f 0 9 8 g / d l O b E V / B I W x Y 8 O o K N t r Q F n Q c K O h U i v u F b + + X n b d s T I W Z D N 7 Q X F U 7 + 6 r h Y o z t b + A A i W b 2 u u n o e v G 2 2 R H 3 y F A G K C b 7 1 / X b n k D T b R p F R E C z y / 2 2 m T o w N 9 i E u m G l b 0 1 b M 8 2 s 6 u y 6 H Z y 0 m 1 a r a X 3 a b R z s z u f a C n g J X o E N Y I E 3 4 A C 8 B 0 e g B z D Q 4 A f 4 C X 7 V P t R 4 7 b x 2 c S 2 9 s z S v e Q 5 K q / b t D w N o + y g = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L z S x C 6 2 q K q 7 o 4 x a W W X m a V V s w m 3 4 = " > A A A F x 3 i c l Z T f b t M w F M Y 9 W G G U f x u I K 2 4 s q o l t 2 q q m m s Q k N G k S n Q S i F w O t a 7 c m V I 7 j N F a d O N j O u s 7 L B a / B H b f w R L w N T t a y J B M X W I p 0 c n 7 f + U 7 s y M e N G Z W q 1 f q 9 d O f u c u 3 e / Z U H 9 Y e P H j 9 5 u r r 2 7 E T y R G D S w 5 x x M X C R J I x G p K e o Y m Q Q C 4 J C l 5 G + O 3 m X 8 f 4 5 E Z L y 6 F j N Y u K E a B x R n 2 K k T G q 0 + s L O P b T L E p J C W w V E o S 9 b o 9 V G q 9 n K F 7 w d W P O g A e b r a L S 2 / N 3 2 O E 5 C E i n M k J R D q x U r R y O h K G Y k r d u J J D H C E z Q m Q x N G K C T S 0 X n v F K 6 b j A d 9 L s w T K Z h n y x X K 3 3 M 0 j e J E k Q i b A o g Y 4 1 N o 8 j t 7 M M + X 9 M e W o z O r X G z W O j Q Q 7 u 2 4 V M F j K + 8 i i w U 6 M E c j B P F z d V 6 Q Z 8 y p T s r C R L C F Z i 6 U N I w Z g b 3 P X Z g d s C R K 0 W h c K n I 5 n y j k y h v 3 W H C f y O y v I L b z N U G M q h k 0 E k b K 7 V A o Q 6 S C t J r M d 3 B j 5 z K D X I 6 E B z M 5 l L P Q 5 a x s 5 Y b / p w 8 p F j z b U b q o W m T 4 W K A 4 m J X U f s J Y b I J / i y M y x T w M U e R p e 4 B T b W e d M W J 6 k K Z l 2 i 3 S b p V 2 i r R T p R f n c + r 6 + q I K B w V 4 q + t p A Z 5 W 4 a w A Z 1 W o g g K 9 v k J V y W V B c V m F Z w V 4 V o X T A p x W 4 a G 7 g K 4 + N L C + D j s o o o S 9 l h B 5 H s 3 u e P m v K j q 5 r J h 4 3 t x E h N q r d p i w d N h 2 d G f 0 9 8 g / d l O b E V / B I W x Y 8 O o K N t r Q F n Q c K O h U i v u F b + + X n b d s T I W Z D N 7 Q X F U 7 + 6 r h Y o z t b + A A i W b 2 u u n o e v G 2 2 R H 3 y F A G K C b 7 1 / X b n k D T b R p F R E C z y / 2 2 m T o w N 9 i E u m G l b 0 1 b M 8 2 s 6 u y 6 H Z y 0 m 1 a r a X 3 a b R z s z u f a C n g J X o E N Y I E 3 4 A C 8 B 0 e g B z D Q 4 A f 4 C X 7 V P t R 4 7 b x 2 c S 2 9 s z S v e Q 5 K q / b t D w N o + y g = < / l a t e x i t > ⇤ < l a t e x i t s h a 1 _ b a s e 6 4 = " h A n G Z A t K F J W 9 d F F A V o f a A 1 m I V l g = " > A A A F x H i c l Z T f b t M w F M Y 9 W G G U f 9 u 4 5 M a i m t i m r W q q S U x C k y b R I S R 6 M d C 6 d m t C 5 S R O Y 9 W x g + 2 s 6 7 z w I F x w C 6 / E 2 + B k L U s y c Y G l S C f n 9 3 3 n x I 5 8 3 J g S q V q t 3 0 v 3 7 i / X H j x c e V R / / O T p s + e r a + u n k i f C w z 2 P U y 4 G L p K Y E o Z 7 i i i K B 7 H A K H I p 7 r u T d x n v X 2 A h C W c n a h Z j J 0 J j R g L i I W V S o 9 V 1 O 6 + h B f Z T a M c h + b I 9 W m 2 0 m q 1 8 w b u B N Q 8 a Y L 6 O R 2 v L 3 2 2 f e 0 m E m f I o k n J o t W L l a C Q U 8 S h O 6 3 Y i c Y y 8 C R r j o Q k Z i r B 0 d N 4 4 h R s m 4 8 O A C / M w B f N s 2 a G C f U c T F i c K M 8 8 Y I K K U T 6 H J 7 + 7 D P F / S n 1 i O z k r l Y r M 2 o I F w f 9 c l C p 5 Y e R d Z N O j Q n I s Q O M j V u S H P m C O d l I W J o A v N X C h J F F M M e 5 + 7 M D t d i Z U i b F w y u Z x P F H L l b f V Y 8 A D L 7 J c g u v s 1 Q Z S o G T Q S i s v t U C Q j p M K 0 m s x 3 c F v O p Q a 5 H A k f Z n I o Z 5 H L a b m U G / 2 f P i K e 4 N m O 0 o V r k e F j g e J w V l I H C a W x C f 4 t Z n j q 8 S h C z N f 2 w E u 1 n X X 2 E N W D N C 3 T b p F 2 q 7 R T p J 0 q v b y Y U z f Q l 1 U 4 K M A 7 X c 8 K 8 K w K Z w U 4 q 0 I V F q h 5 w w p V J V c F x V U V n h f g e R V O C 3 B a h U f u A r r 6 y M D 6 B u w g R j B 9 L S H y f Z J d 8 P J f V W R y V S n i + / M i I t J + t c O E p s O 2 o z u j v 0 f + s Z v a F A c K D m H D g t f X s N G G t i D j U E G n Y u 4 X v r 1 f r r x t e 0 S Y y e A P z V W 1 s 6 8 a L m b Y w a Y X I t H M X r c c X S / e N p t x H w 9 l i G J 8 c O P f 8 Q W a 7 h D G s I B m l w d t M 3 V g X m A L 6 o a V v j V t z T S z q r P r b n D a b l q t p v V p r 3 G 4 N 5 9 r K + A l e A U 2 g Q X e g E P w A R y D H v D A J f g B f o J f t f c 1 W p O 1 5 E Z 6 b 2 n u e Q F K q / b t D 8 X o + c g = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " h A n G Z A t K F J W 9 d F F A V o f a A 1 m I V l g = " > A A A F x H i c l Z T f b t M w F M Y 9 W G G U f 9 u 4 5 M a i m t i m r W q q S U x C k y b R I S R 6 M d C 6 d m t C 5 S R O Y 9 W x g + 2 s 6 7 z w I F x w C 6 / E 2 + B k L U s y c Y G l S C f n 9 3 3 n x I 5 8 3 J g S q V q t 3 0 v 3 7 i / X H j x c e V R / / O T p s + e r a + u n k i f C w z 2 P U y 4 G L p K Y E o Z 7 i i i K B 7 H A K H I p 7 r u T d x n v X 2 A h C W c n a h Z j J 0 J j R g L i I W V S o 9 V 1 O 6 + h B f Z T a M c h + b I 9 W m 2 0 m q 1 8 w b u B N Q 8 a Y L 6 O R 2 v L 3 2 2 f e 0 m E m f I o k n J o t W L l a C Q U 8 S h O 6 3 Y i c Y y 8 C R r j o Q k Z i r B 0 d N 4 4 h R s m 4 8 O A C / M w B f N s 2 a G C f U c T F i c K M 8 8 Y I K K U T 6 H J 7 + 7 D P F / S n 1 i O z k r l Y r M 2 o I F w f 9 c l C p 5 Y e R d Z N O j Q n I s Q O M j V u S H P m C O d l I W J o A v N X C h J F F M M e 5 + 7 M D t d i Z U i b F w y u Z x P F H L l b f V Y 8 A D L 7 J c g u v s 1 Q Z S o G T Q S i s v t U C Q j p M K 0 m s x 3 c F v O p Q a 5 H A k f Z n I o Z 5 H L a b m U G / 2 f P i K e 4 N m O 0 o V r k e F j g e J w V l I H C a W x C f 4 t Z n j q 8 S h C z N f 2 w E u 1 n X X 2 E N W D N C 3 T b p F 2 q 7 R T p J 0 q v b y Y U z f Q l 1 U 4 K M A 7 X c 8 K 8 K w K Z w U 4 q 0 I V F q h 5 w w p V J V c F x V U V n h f g e R V O C 3 B a h U f u A r r 6 y M D 6 B u w g R j B 9 L S H y f Z J d 8 P J f V W R y V S n i + / M i I t J + t c O E p s O 2 o z u j v 0 f + s Z v a F A c K D m H D g t f X s N G G t i D j U E G n Y u 4 X v r 1 f r r x t e 0 S Y y e A P z V W 1 s 6 8 a L m b Y w a Y X I t H M X r c c X S / e N p t x H w 9 l i G J 8 c O P f 8 Q W a 7 h D G s I B m l w d t M 3 V g X m A L 6 o a V v j V t z T S z q r P r b n D a b l q t p v V p r 3 G 4 N 5 9 r K + A l e A U 2 g Q X e g E P w A R y D H v D A J f g B f o J f t f c 1 W p O 1 5 E Z 6 b 2 n u e Q F K q / b t D 8 X o + c g = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " h A n G Z A t K F J W 9 d F F A V o f a A 1 m I V l g = " > A A A F x H i c l Z T f b t M w F M Y 9 W G G U f 9 u 4 5 M a i m t i m r W q q S U x C k y b R I S R 6 M d C 6 d m t C 5 S R O Y 9 W x g + 2 s 6 7 z w I F x w C 6 / E 2 + B k L U s y c Y G l S C f n 9 3 3 n x I 5 8 3 J g S q V q t 3 0 v 3 7 i / X H j x c e V R / / O T p s + e r a + u n k i f C w z 2 P U y 4 G L p K Y E o Z 7 i i i K B 7 H A K H I p 7 r u T d x n v X 2 A h C W c n a h Z j J 0 J j R g L i I W V S o 9 V 1 O 6 + h B f Z T a M c h + b I 9 W m 2 0 m q 1 8 w b u B N Q 8 a Y L 6 O R 2 v L 3 2 2 f e 0 m E m f I o k n J o t W L l a C Q U 8 S h O 6 3 Y i c Y y 8 C R r j o Q k Z i r B 0 d N 4 4 h R s m 4 8 O A C / M w B f N s 2 a G C f U c T F i c K M 8 8 Y I K K U T 6 H J 7 + 7 D P F / S n 1 i O z k r l Y r M 2 o I F w f 9 c l C p 5 Y e R d Z N O j Q n I s Q O M j V u S H P m C O d l I W J o A v N X C h J F F M M e 5 + 7 M D t d i Z U i b F w y u Z x P F H L l b f V Y 8 A D L 7 J c g u v s 1 Q Z S o G T Q S i s v t U C Q j p M K 0 m s x 3 c F v O p Q a 5 H A k f Z n I o Z 5 H L a b m U G / 2 f P i K e 4 N m O 0 o V r k e F j g e J w V l I H C a W x C f 4 t Z n j q 8 S h C z N f 2 w E u 1 n X X 2 E N W D N C 3 T b p F 2 q 7 R T p J 0 q v b y Y U z f Q l 1 U 4 K M A 7 X c 8 K 8 K w K Z w U 4 q 0 I V F q h 5 w w p V J V c F x V U V n h f g e R V O C 3 B a h U f u A r r 6 y M D 6 B u w g R j B 9 L S H y f Z J d 8 P J f V W R y V S n i + / M i I t J + t c O E p s O 2 o z u j v 0 f + s Z v a F A c K D m H D g t f X s N G G t i D j U E G n Y u 4 X v r 1 f r r x t e 0 S Y y e A P z V W 1 s 6 8 a L m b Y w a Y X I t H M X r c c X S / e N p t x H w 9 l i G J 8 c O P f 8 Q W a 7 h D G s I B m l w d t M 3 V g X m A L 6 o a V v j V t z T S z q r P r b n D a b l q t p v V p r 3 G 4 N 5 9 r K + A l e A U 2 g Q X e g E P w A R y D H v D A J f g B f o J f t f c 1 W p O 1 5 E Z 6 b 2 n u e Q F K q / b t D 8 X o + c g = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " h A n G Z A t K F J W 9 d F F A V o f a A 1 m I V l g = " > A A A F x H i c l Z T f b t M w F M Y 9 W G G U f 9 u 4 5 M a i m t i m r W q q S U x C k y b R I S R 6 M d C 6 d m t C 5 S R O Y 9 W x g + 2 s 6 7 z w I F x w C 6 / E 2 + B k L U s y c Y G l S C f n 9 3 3 n x I 5 8 3 J g S q V q t 3 0 v 3 7 i / X H j x c e V R / / O T p s + e r a + u n k i f C w z 2 P U y 4 G L p K Y E o Z 7 i i i K B 7 H A K H I p 7 r u T d x n v X 2 A h C W c n a h Z j J 0 J j R g L i I W V S o 9 V 1 O 6 + h B f Z T a M c h + b I 9 W m 2 0 m q 1 8 w b u B N Q 8 a Y L 6 O R 2 v L 3 2 2 f e 0 m E m f I o k n J o t W L l a C Q U 8 S h O 6 3 Y i c Y y 8 C R r j o Q k Z i r B 0 d N 4 4 h R s m 4 8 O A C / M w B f N s 2 a G C f U c T F i c K M 8 8 Y I K K U T 6 H J 7 + 7 D P F / S n 1 i O z k r l Y r M 2 o I F w f 9 c l C p 5 Y e R d Z N O j Q n I s Q O M j V u S H P m C O d l I W J o A v N X C h J F F M M e 5 + 7 M D t d i Z U i b F w y u Z x P F H L l b f V Y 8 A D L 7 J c g u v s 1 Q Z S o G T Q S i s v t U C Q j p M K 0 m s x 3 c F v O p Q a 5 H A k f Z n I o Z 5 H L a b m U G / 2 f P i K e 4 N m O 0 o V r k e F j g e J w V l I H C a W x C f 4 t Z n j q 8 S h C z N f 2 w E u 1 n X X 2 E N W D N C 3 T b p F 2 q 7 R T p J 0 q v b y Y U z f Q l 1 U 4 K M A 7 X c 8 K 8 K w K Z w U 4 q 0 I V F q h 5 w w p V J V c F x V U V n h f g e R V O C 3 B a h U f u A r r 6 y M D 6 B u w g R j B 9 L S H y f Z J d 8 P J f V W R y V S n i + / M i I t J + t c O E p s O 2 o z u j v 0 f + s Z v a F A c K D m H D g t f X s N G G t i D j U E G n Y u 4 X v r 1 f r r x t e 0 S Y y e A P z V W 1 s 6 8 a L m b Y w a Y X I t H M X r c c X S / e N p t x H w 9 l i G J 8 c O P f 8 Q W a 7 h D G s I B m l w d t M 3 V g X m A L 6 o a V v j V t z T S z q r P r b n D a b l q t p v V p r 3 G 4 N 5 9 r K + A l e A U 2 g Q X e g E P w A R y D H v D A J f g B f o J f t f c 1 W p O 1 5 E Z 6 b 2 n u e Q F K q / b t D 8 X o + c g = < / l a t e x i t > Figure 2: A dis- tribution q φ is op- timized to approxi- mate p θ * .

Variational inference relies on a tractable class of distributions that can be optimized to closely resemble the true distribution (fig. 2 ), and it's hypothesized that more flexible classes lead to more faithful approximations and thus better performance (Jordan et al. (1999) ). To explore this hypothesis, we propose a novel tractable class of highly flexible variational distributions. Considering that neural networks with low-precision activations exhibit good performance (Holi & Hwang (1993) ; Hubara et al. ( 2016)), we make the modeling assumption that latent variables can be expressed under a strong quantization scheme, without loss of predictive fidelity. If this assumption holds, it becomes tractable to model a scalar latent variable with a flexible multinomial distribution over the quantization bins (fig. 3 ).

p(z):

< l a t e x i t s h a 1 _ b a s e 6 4 = " 9 e V B 2 E 4

x U d H l x k Z 7 u Z H 9 d P W h H X w = " > A A A F v X i c l Z T d b t M w F M e 9 s c I o X x t c c m N R T W x o q 5 o K i U p o Y o J O Q q I X A 6 1 r t y S a H M d p r T p 2 Z r v r 2 i z P g c Q V P B Z v g 5 O l L M n E B Z Y i H Z / f / 5 z j j / h 4 E a N K t 1 q / V 1 b v r d X u P 1 h / W H / 0 + M n T Z x u b z 0 + U m E p M + l g w I Y c e U o R R T v q a a k a G k S Q o 9 B g Z e J N P K R 9 c E q m o 4 M d 6 H h E 3 R C N O A 4 q R N i 4 3 2 n Z C p M d e E C + S n f O N R q v Z y g a 8 a 1 i 5 0 Q D 5 O D r f X P v h + A J P Q 8 I 1 Z k g p 2 2 p F 2 o 2 R 1 B Q z k t S d q S I R w h M 0 I r Y x O Q q J c u N s 1 Q n c M h 4 f B k K a j 2 u Y e c s R O u i 4 M e X R V B O O T Q B E j I k Z N P 6 9 D s z 8 J f 2 x 5 c Z p q k x s x h Y 0 E H b 2 P K r h s Z V V U c W A e G w O R E o S Z O o s I P O Y s 5 y U h V P J l p p c q G g Y M Q L 7 3 3 o w P V Z F t K Z 8 V A r y h J h o 5 K n b 7 J E U A V H p X S C 2 d z F F j O o 5 N B J G y u V Q q N J b S a r O b A e 3 6 T x m k C e Q 9 G E q h 2 o e e o K V U 3 n h / + l D i q V I d 5 Q s o 5 Y e M Z I o G s 9 L 6 m D K W G S M f 4 s 5 m W E R h o j 7 s T P E S Z z 9 b h i x e J g k Z d o r 0 l 6 V d o u 0 W 6 V X l z k 1 P / J V F Q 4 L 8 E 7 V 0 w I 8 r c J 5 A c 6 r U I 8 L 1 M y I R l X J o q B Y V O F Z A Z 5 V 4 a w A Z 1 V 4 6 C 2 h F x 8 a W N + C X c Q p Y a 8 V R L 5 P 0 5 d d v l V N J 4 t K E t / P k 8 g w 9 q s V J i y x 2 2 7 c P f 9 7 5 F 9 6 i c N I o K E N G x a 8 v o a N N n Q k H Y 0 1 d C v B g 8 L a B + X M b x x M p e k M v m 2 e q p O u y l 4 2 r / 1 t P E a y m U 5 3 3 L h e f G 0 O F z 6 x 1 R h F Z P 8 m f t e X a L Z L O S c S m l 3 u t 0 3 X g V m C H R g 3 r O S 9 K W u 6 m V X t X X e N k 3 b T a j W t r 2 8 b B x / z v r Y O X o J X Y B t Y 4 B 0 4 A J / B E

e g D D C 7 A d / A T / K p 9 q J E a q / E b 6 e p K H v M C l E Z t 9 g e f f v e B < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 e V B 2 E 4

x U d H l x k Z 7 u Z H 9 d P W h H X w = " > A A A F v X i c l Z T d b t M w F M e 9 s c I o X x t c c m N R T W x o q 5 o K i U p o Y o J O Q q I X A 6 1 r t y S a H M d p r T p 2 Z r v r 2 i z P g c Q V P B Z v g 5 O l L M n E B Z Y i H Z / f / 5 z j j / h 4 E a N K t 1 q / V 1 b v r d X u P 1 h / W H / 0 + M n T Z x u b z 0 + U m E p M + l g w I Y c e U o R R T v q a a k a G k S Q o 9 B g Z e J N P K R 9 c E q m o 4 M d 6 H h E 3 R C N O A 4 q R N i 4 3 2 n Z C p M d e E C + S n f O N R q v Z y g a 8 a 1 i 5 0 Q D 5 O D r f X P v h + A J P Q 8 I 1 Z k g p 2 2 p F 2 o 2 R 1 B Q z k t S d q S I R w h M 0 I r Y x O Q q J c u N s 1 Q n c M h 4 f B k K a j 2 u Y e c s R O u i 4 M e X R V B O O T Q B E j I k Z N P 6 9 D s z 8 J f 2 x 5 c Z p q k x s x h Y 0 E H b 2 P K r h s Z V V U c W A e G w O R E o S Z O o s I P O Y s 5 y U h V P J l p p c q G g Y M Q L 7 3 3 o w P V Z F t K Z 8 V A r y h J h o 5 K n b 7 J E U A V H p X S C 2 d z F F j O o 5 N B J G y u V Q q N J b S a r O b A e 3 6 T x m k C e Q 9 G E q h 2 o e e o K V U 3 n h / + l D i q V I d 5 Q s o 5 Y e M Z I o G s 9 L 6 m D K W G S M f 4 s 5 m W E R h o j 7 s T P E S Z z 9 b h i x e J g k Z d o r 0 l 6 V d o u 0 W 6 V X l z k 1 P / J V F Q 4 L 8 E 7 V 0 w I 8 r c J 5 A c 6 r U I 8 L 1 M y I R l X J o q B Y V O F Z A Z 5 V 4 a w A Z 1 V 4 6 C 2 h F x 8 a W N + C X c Q p Y a 8 V R L 5 P 0 5 d d v l V N J 4 t K E t / P k 8 g w 9 q s V J i y x 2 2 7 c P f 9 7 5 F 9 6 i c N I o K E N G x a 8 v o a N N n Q k H Y 0 1 d C v B g 8 L a B + X M b x x M p e k M v

m 2 e q p O u y l 4 2 r / 1 t P E a y m U 5 3 3 L h e f G 0 O F z 6 x 1 R h F Z P 8 m f t e X a L Z L O S c S m l 3 u t 0 3 X g V m C H R g 3 r O S 9 K W u 6 m V X t X X e N k 3 b T a j W t r 2 8 b B x / z v r Y O X o J X Y B t Y 4 B 0 4 A J / B E e g D D C 7 A d / A T / K p 9 q J E a q / E b 6 e p K H v M C l E Z t 9 g e f f v e B < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 e V B 2 E 4

x U d H l x k Z 7 u Z H 9 d P W h H X w = " > A A A F v X i c l Z T d b t M w F M e 9 s c I o X x t c c m N R T W x o q 5 o K i U p o Y o J O Q q I X A 6 1 r t y S a H M d p r T p 2 Z r v r 2 i z P g c Q V P B Z v g 5 O l L M n E B Z Y i H Z / f / 5 z j j / h 4 E a N K t 1 q / V 1 b v r d X u P 1 h / W H / 0 + M n T Z x u b z 0 + U m E p M + l g w I Y c e U o R R T v q a a k a G k S Q o 9 B g Z e J N P K R 9 c E q m o 4 M d 6 H h E 3 R C N O A 4 q R N i 4 3 2 n Z C p M d e E C + S n f O N R q v Z y g a 8 a 1 i 5 0 Q D 5 O D r f X P v h + A J P Q 8 I 1 Z k g p 2 2 p F 2 o 2 R 1 B Q z k t S d q S I R w h M 0 I r Y x O Q q J c u N s 1 Q n c M h 4 f B k K a j 2 u Y e c s R O u i 4 M e X R V B O O T Q B E j I k Z N P 6 9 D s z 8 J f 2 x 5 c Z p q k x s x h Y 0 E H b 2 P K r h s Z V V U c W A e G w O R E o S Z O o s I P O Y s 5 y U h V P J l p p c q G g Y M Q L 7 3 3 o w P V Z F t K Z 8 V A r y h J h o 5 K n b 7 J E U A V H p X S C 2 d z F F j O o 5 N B J G y u V Q q N J b S a r O b A e 3 6 T x m k C e Q 9 G E q h 2 o e e o K V U 3 n h / + l D i q V I d 5 Q s o 5 Y e M Z I o G s 9 L 6 m D K W G S M f 4 s 5 m W E R h o j 7 s T P E S Z z 9 b h i x e J g k Z d o r 0 l 6 V d o u 0 W 6 V X l z k 1 P / J V F Q 4 L 8 E 7 V 0 w I 8 r c J 5 A c 6 r U I 8 L 1 M y I R l X J o q B Y V O F Z A Z 5 V 4 a w A Z 1 V 4 6 C 2 h F x 8 a W N + C X c Q p Y a 8 V R L 5 P 0 5 d d v l V N J 4 t K E t / P k 8 g w 9 q s V J i y x 2 2 7 c P f 9 7 5 F 9 6 i c N I o K E N G x a 8 v o a N N n Q k H Y 0 1 d C v B g 8 L a B + X M b x x M p e k M v

m 2 e q p O u y l 4 2 r / 1 t P E a y m U 5 3 3 L h e f G 0 O F z 6 x 1 R h F Z P 8 m f t e X a L Z L O S c S m l 3 u t 0 3 X g V m C H R g 3 r O S 9 K W u 6 m V X t X X e N k 3 b T a j W t r 2 8 b B x / z v r Y O X o J X Y B t Y 4 B 0 4 A J / B E e g D D C 7 A d / A T / K p 9 q J E a q / E b 6 e p K H v M C l E Z t 9 g e f f v e B < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 e V B 2 E 4

x U d H l x k Z 7 u Z H 9 d P W h H X w = " > A A A F v X i c l Z T d b t M w F M e 9 s c I o X x t c c m N R T W x o q 5 o K i U p o Y o J O Q q I X A 6 1 r t y S a H M d p r T p 2 Z r v r 2 i z P g c Q V P B Z v g 5 O l L M n E B Z Y i H Z / f / 5 z j j / h 4 E a N K t 1 q / V 1 b v r d X u P 1 h / W H / 0 + M n T Z x u b z 0 + U m E p M + l g w I Y c e U o R R T v q a a k a G k S Q o 9 B g Z e J N P K R 9 c E q m o 4 M d 6 H h E 3 R C N O A 4 q R N i 4 3 2 n Z C p M d e E C + S n f O N R q v Z y g a 8 a 1 i 5 0 Q D 5 O D r f X P v h + A J P Q 8 I 1 Z k g p 2 2 p F 2 o 2 R 1 B Q z k t S d q S I R w h M 0 I r Y x O Q q J c u N s 1 Q n c M h 4 f B k K a j 2 u Y e c s R O u i 4 M e X R V B O O T Q B E j I k Z N P 6 9 D s z 8 J f 2 x 5 c Z p q k x s x h Y 0 E H b 2 P K r h s Z V V U c W A e G w O R E o S Z O o s I P O Y s 5 y U h V P J l p p c q G g Y M Q L 7 3 3 o w P V Z F t K Z 8 V A r y h J h o 5 K n b 7 J E U A V H p X S C 2 d z F F j O o 5 N B J G y u V Q q N J b S a r O b A e 3 6 T x m k C e Q 9 G E q h 2 o e e o K V U 3 n h / + l D i q V I d 5 Q s o 5 Y e M Z I o G s 9 L 6 m D K W G S M f 4 s 5 m W E R h o j 7 s T P E S Z z 9 b h i x e J g k Z d o r 0 l 6 V d o u 0 W 6 V X l z k 1 P / J V F Q 4 L 8 E 7 V 0 w I 8 r c J 5 A c 6 r U I 8 L 1 M y I R l X J o q B Y V O F Z A Z 5 V 4 a w A Z 1 V 4 6 C 2 h F x 8 a W N + C X c Q p Y a 8 V R L 5 P 0 5 d d v l V N J 4 t K E t / P k 8 g w 9 q s V J i y x 2 2 7 c P f 9 7 5 F 9 6 i c N I o K E N G x a 8 v o a N N n Q k H Y 0 1 d C v B g 8 L a B + X M b x x M p e k M v

m 2 e q p O u y l 4 2 r / 1 t P E a y m U 5 3 3 L h e f G 0 O F z 6 x 1 R h F Z P 8 m f t e X a L Z L O S c S m l 3 u t 0 3 X g V m C H R g 3 r O S 9 K W u 6 m V X t X X e N k 3 b T a j W t r 2 8 b B x / z v r Y O X o J X Y B t Y 4 B 0 4 A J / B E e g D D C 7 A d / A T / K p 9 q J E a q / E b 6 e p K H v M C l E Z t 9 g e f f v e B < / l a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " S 2 w By re-positioning the variational distribution from a potentially limited description of moments, as found in commonly applied conjugate distributions, to a direct expression of probabilities per value, a variety of benefits arise. As the output domain is constrained, the method becomes self-normalizing, relieving the model from hard-to-parallelize batch normalization techniques (Ioffe & Szegedy (2015) ). More interesting priors can be explored and the model is able to learn unique activation functions per neuron.

x D Z m M e u O 5 q f Y a c j L a U A h x z u M = " > A A A F u n i c l Z T d b t M w F M c 9 W G G U r w 0 u u b G o J j a 0 V U 2 F x C S Y N E E n I d G L g d a 1 W x N N t u O 0 V h w 7 2 O 6 6 N s t b c I U E 7 8 X b 4 G Q t S z J x g a V I x + f 3 P + f 4 I z 4 4 5 k y b V u v 3 y p 2 7 q 7 V 7 9 9 c e 1 B 8 + e v z k 6 f r G s x M t J 4 r Q H p F c q g F G m n I m a M 8 w w + k g V h R F m N M + D j 9 m v H 9 B l W Z S H J t Z T L 0 I j Q Q L G E H G u s 7 c C J k x D p J 5 e r 7 e a D V b + Y C 3 D W d h N M B i H J 1 v r P 5 w f U k m E R W G c K T 1 0 G n F x k u Q M o x w m t b d i a Y x I i E a 0 a E 1 B Y q o 9 p J 8 y S n c t B 4 f B l L Z T x i Y e 8 s R J t j z E i b i i a G C 2 A C I O J d T a P 2 7 e z D 3 l / T H j p d k q X K x H Z v Q Q r i 3 i 5 m B x 0 5 e R R c D k r E 9 D a V o k K v z g N x j D z I s C y e K L z U L o W Z R z C n s f e 3 C 7 E w 1 N Y a J U S k I S x k a h P V N 9 l j J g O r s I h D f / T Z B n J k Z t B J O y + V Q p L N L S a v O f A c 3 6 T C 3 C E u k f J j J o Z 5 F W P J y K h z 9 n z 5 i R M l s R + k y a u m R I 4 X i 8 a y k D i a c x 9 b 4 t 1 j Q K Z F R h I S f u A O S J v n f R h B P B m l a p t 0 i 7 V Z p p 0 g 7 V X p 5 s a D 2 P 7 6 s w k E B 3 q p 6 W o C n V T g r w F k V m n G B 2 h k 1 q C q Z F x T z K j w r w L M q n B b g t A o P 8 R L i 5 N D C + i b s I M E o f 6 U h 8 n 2 W P e v y r R o W z i t J f H + R R E W J X 6 0 Q 8 n T Y 9 p L O + d 8 j / 9 x N X U 4 D A 4 e w 4 c C r K 9 h o Q 1 e x 0 d h A r x L c L 6 y 9 X 8 7 8 2 i V M 2 c 7 g D + 1 T d b N V D Z e d a 3 + L j J F q Z t N t L 6 k X X 5 s r p E + H e o x i u n 8 d v + M r N N 1 h Q l A F 7 S 7 3 2 7 b r w D z B N k w a T v r O l r X d z K n 2 r t v G S b v p t J r O l z e N g w + L v r Y G X o C X Y A s 4 4 C 0 4 A J / A E e g B A g T 4 D n 6 C X 7 X 3 N V x j t f B a e m d l E f M c l E b N / A E a u v a i < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S 2 w x D Z m M e u O 5 q f Y a c j L a U A h x z u M = " > A A A F u n i c l Z T d b t M w F M c 9 W G G U r w 0 u u b G o J j a 0 V U 2 F x C S Y N E E n I d G L g d a 1 W x N N t u O 0 V h w 7 2 O 6 6 N s t b c I U E 7 8 X b 4 G Q t S z J x g a V I x + f 3 P + f 4 I z 4 4 5 k y b V u v 3 y p 2 7 q 7 V 7 9 9 c e 1 B 8 + e v z k 6 f r G s x M t J 4 r Q H p F c q g F G m n I m a M 8 w w + k g V h R F m N M + D j 9 m v H 9 B l W Z S H J t Z T L 0 I j Q Q L G E H G u s 7 c C J k x D p J 5 e r 7 e a D V b + Y C 3 D W d h N M B i H J 1 v r P 5 w f U k m E R W G c K T 1 0 G n F x k u Q M o x w m t b d i a Y x I i E a 0 a E 1 B Y q o 9 p J 8 y S n c t B 4 f B l L Z T x i Y e 8 s R J t j z E i b i i a G C 2 A C I O J d T a P 2 7 e z D 3 l / T H j p d k q X K x H Z v Q Q r i 3 i 5 m B x 0 5 e R R c D k r E 9 D a V o k K v z g N x j D z I s C y e K L z U L o W Z R z C n s f e 3 C 7 E w 1 N Y a J U S k I S x k a h P V N 9 l j J g O r s I h D f / T Z B n J k Z t B J O y + V Q p L N L S a v O f A c 3 6 T C 3 C E u k f J j J o Z 5 F W P J y K h z 9 n z 5 i R M l s R + k y a u m R I 4 X i 8 a y k D i a c x 9 b 4 t 1 j Q K Z F R h I S f u A O S J v n f R h B P B m l a p t 0 i 7 V Z p p 0 g 7 V X p 5 s a D 2 P 7 6 s w k E B 3 q p 6 W o C n V T g r w F k V m n G B 2 h k 1 q C q Z F x T z K j w r w L M q n B b g t A o P 8 R L i 5 N D C + i b s I M E o f 6 U h 8 n 2 W P e v y r R o W z i t J f H + R R E W J X 6 0 Q 8 n T Y 9 p L O + d 8 j / 9 x N X U 4 D A 4 e w 4 c C r K 9 h o Q 1 e x 0 d h A r x L c L 6 y 9 X 8 7 8 2 i V M 2 c 7 g D + 1 T d b N V D Z e d a 3 + L j J F q Z t N t L 6 k X X 5 s r p E + H e o x i u n 8 d v + M r N N 1 h Q l A F 7 S 7 3 2 7 b r w D z B N k w a T v r O l r X d z K n 2 r t v G S b v p t J r O l z e N g w + L v r Y G X o C X Y A s 4 4 C 0 4 A J / A E e g B A g T 4 D n 6 C X 7 X 3 N V x j t f B a e m d l E f M c l E b N / A E a u v a i < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S 2 w x D Z m M e u O 5 q f Y a c j L a U A h x z u M = " > A A A F u n i c l Z T d b t M w F M c 9 W G G U r w 0 u u b G o J j a 0 V U 2 F x C S Y N E E n I d G L g d a 1 W x N N t u O 0 V h w 7 2 O 6 6 N s t b c I U E 7 8 X b 4 G Q t S z J x g a V I x + f 3 P + f 4 I z 4 4 5 k y b V u v 3 y p 2 7 q 7 V 7 9 9 c e 1 B 8 + e v z k 6 f r G s x M t J 4 r Q H p F c q g F G m n I m a M 8 w w + k g V h R F m N M + D j 9 m v H 9 B l W Z S H J t Z T L 0 I j Q Q L G E H G u s 7 c C J k x D p J 5 e r 7 e a D V b + Y C 3 D W d h N M B i H J 1 v r P 5 w f U k m E R W G c K T 1 0 G n F x k u Q M o x w m t b d i a Y x I i E a 0 a E 1 B Y q o 9 p J 8 y S n c t B 4 f B l L Z T x i Y e 8 s R J t j z E i b i i a G C 2 A C I O J d T a P 2 7 e z D 3 l / T H j p d k q X K x H Z v Q Q r i 3 i 5 m B x 0 5 e R R c D k r E 9 D a V o k K v z g N x j D z I s C y e K L z U L o W Z R z C n s f e 3 C 7 E w 1 N Y a J U S k I S x k a h P V N 9 l j J g O r s I h D f / T Z B n J k Z t B J O y + V Q p L N L S a v O f A c 3 6 T C 3 C E u k f J j J o Z 5 F W P J y K h z 9 n z 5 i R M l s R + k y a u m R I 4 X i 8 a y k D i a c x 9 b 4 t 1 j Q K Z F R h I S f u A O S J v n f R h B P B m l a p t 0 i 7 V Z p p 0 g 7 V X p 5 s a D 2 P 7 6 s w k E B 3 q p 6 W o C n V T g r w F k V m n G B 2 h k 1 q C q Z F x T z K j w r w L M q n B b g t A o P 8 R L i 5 N D C + i b s I M E o f 6 U h 8 n 2 W P e v y r R o W z i t J f H + R R E W J X 6 0 Q 8 n T Y 9 p L O + d 8 j / 9 x N X U 4 D A 4 e w 4 c C r K 9 h o Q 1 e x 0 d h A r x L c L 6 y 9 X 8 7 8 2 i V M 2 c 7 g D + 1 T d b N V D Z e d a 3 + L j J F q Z t N t L 6 k X X 5 s r p E + H e o x i u n 8 d v + M r N N 1 h Q l A F 7 S 7 3 2 7 b r w D z B N k w a T v r O l r X d z K n 2 r t v G S b v p t J r O l z e N g w + L v r Y G X o C X Y A s 4 4 C 0 4 A J / A E e g B A g T 4 D n 6 C X 7 X 3 N V x j t f B a e m d l E f M c l E b N / A E a u v a i < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S 2 w x D Z m M e u O 5 q f Y a c j L a U A h x z u M = " > A A A F u n i c l Z T d b t M w F M c 9 W G G U r w 0 u u b G o J j a 0 V U 2 F x C S Y N E E n I d G L g d a 1 W x N N t u O 0 V h w 7 2 O 6 6 N s t b c I U E 7 8 X b 4 G Q t S z J x g a V I x + f 3 P + f 4 I z 4 4 5 k y b V u v 3 y p 2 7 q 7 V 7 9 9 c e 1 B 8 + e v z k 6 f r G s x M t J 4 r Q H p F c q g F G m n I m a M 8 w w + k g V h R F m N M + D j 9 m v H 9 B l W Z S H J t Z T L 0 I j Q Q L G E H G u s 7 c C J k x D p J 5 e r 7 e a D V b + Y C 3 D W d h N M B i H J 1 v r P 5 w f U k m E R W G c K T 1 0 G n F x k u Q M o x w m t b d i a Y x I i E a 0 a E 1 B Y q o 9 p J 8 y S n c t B 4 f B l L Z T x i Y e 8 s R J t j z E i b i i a G C 2 A C I O J d T a P 2 7 e z D 3 l / T H j p d k q X K x H Z v Q Q r i 3 i 5 m B x 0 5 e R R c D k r E 9 D a V o k K v z g N x j D z I s C y e K L z U L o W Z R z C n s f e 3 C 7 E w 1 N Y a J U S k I S x k a h P V N 9 l j J g O r s I h D f / T Z B n J k Z t B J O y + V Q p L N L S a v O f A c 3 6 T C 3 C E u k f J j J o Z 5 F W P J y K h z 9 n z 5 i R M l s R + k y a u m R I 4 X i 8 a y k D i a c x 9 b 4 t 1 j Q K Z F R h I S f u A O S J v n f R h B P B m l a p t 0 i 7 V Z p p 0 g 7 V X p 5 s a D 2 P 7 6 s w k E B 3 q p 6 W o C n V T g r w F k V m n G B 2 h k 1 q C q Z F x T z K j w r w L M q n B b g t A o P 8 R L i 5 N D C + i b s I M E o f 6 U h 8 n 2 W P e v y r R o W z i t J f H + R R E W J X 6 0 Q 8 n T Y 9 p L O + d 8 j / 9 x N X U 4 D A 4 e w 4 c C r K 9 h o Q 1 e x 0 d h A r x L c L 6 y 9 X 8 7 8 2 i V M 2 c 7 g D + 1 T d b N V D Z e d a 3 + L j J F q Z t N t L 6 k X X 5 s r p E + H e o x i u n 8 d v + M r N N 1 h Q l A F 7 S 7 3 2 7 b r w D z B N k w a T v r O l r X d z K n 2 r t v G S b v p t J r O l z e N g w + L v r Y G X o C X Y A s 4 4 C 0 4 A J / A E e g B A g T 4 D n 6 C X 7 X 3 N V x j t f B a e m d l E f M c l E b N / A E a u v a i < / l a t e x i t >

More concretely, the contributions of this work are as follows:

• We propose a novel variational inference method by leveraging multinomial distributions on quantized latent variables. • We show that the emerging predicted distributions are multi-modal, motivating the need for flexible distributions in variational inference. • We demonstrate that the proposed method applied to the information bottleneck objective computes competitive uncertainty over the predictions and that this manifests in better performance under strong risk guarantees.

2. BACKGROUND:

In this work, we explore deep neural networks for regression and classification. We have data-points consisting of inputs x and targets y in a dataset D = {(x i , y i ) | i ∈ [1, ..., N ]} and postulate latent variables z that represent the data. We focus on the Information Bottleneck (IB) perspective: first proposed by Tishby et al. (2000) , the information bottleneck objective I(y, z; θ) -βI(x, z; θ) is optimized to maximize the mutual information between z and y, whilst minimizing the mutual information between z and x. The objective can be efficiently optimized using a variational inference scheme as shown concurrently by both Alemi et al. ( 2016) and Achille & Soatto (2016) . Under the Markov assumption P (z, x, y) = p(z|x)p(y|x)p(x), they derive the following lower bound:

EQUATION

where E p(z|xn) is commonly estimated using a single Monte Carlo sample and r(z) is a variational approximation to the marginal distribution of z. In practice r(z) is fixed to a simple distribution such as a spherical Gaussian. 2014) is a special case of the IB bound when y = x and β = 1:

EQUATION

where i represents the identity of data-point x i . Interestingly, the VAE perspective considers the bound to optimize a variational distribution q(z|x), whilst the IB perspective prescribes that q(z|x) in the ELBO is not a variational posterior but the true encoder p(z|x), and instead p(y|z) and p(z) are the distributions approximated with variational counterparts. However, deriving the ELBO on a conditional distribution p(y|x) leads to a KL divergence between the approximate posterior q(z|x) and the real posteriorp(z|x), which is not available at test time, rather than the prior p(z). Hence, the information bottleneck perspective provides a more fundamental motivation for the bound in equation 1.

Alternatively, equation 1 can be interpreted as a domain-translating beta-VAE (Higgins et al. (2016)) , where an input image is encoded into a latent space and decoded into the target domain. The Lagrange multiplier β then controls the trade-off between rate and distortion, as argued by Alemi et al. (2017) . However, this does not naturally arise from deriving the ELBO using Jensen's inequality on the conditional likelihood p(y|x).

3. METHOD:

In this work, we follow the IB interpretation of the bound in equation 1 and leave the evaluation of our proposed variational inference scheme in other models such as the VAE for further work. At the heart of our proposal lies the assumption that neuron networks can be effective under strong activation quantization schemes. We start with presenting the derivation if the model in the context of a single latent-layer information bottleneck, following the single data-point loss in equation 1 and dropping the subscript n for clarity, with figure 4 for visual reference:

EQUATION

To impose a flexible, multi-modal distribution over z, we first make a mean-field assumption p(z|x) = k p(z k |x). We then quantize the domain of each of the K scalar latent variables z k such that only a small set of potential values remain:

z k ∈ v = {v 1 , . . . , v C } ∀k with e.g. C = 11, see fig.

3. We use the same quantization scheme, defined by v, for all the latents. The specific scheme is considered part of the prior, and is discussed in figure 5 .

Although the number of values of z is relatively small, the space of values over all the latent variables grows exponentially. We thus take to a monte-carlo estimation scheme. To optimize the parameters θ with Stochastic Gradient Descent (SGD), we need to derive a fully differentiable sampling scheme that allows us to sample values of z. . To formulate this, we re-parametrize the expectation over z in equation 3 using a set of variables s k ∈ {1, . . . , C} which index the value vector v, allowing us to use a softmax function to represent the distribution over z:

EQUATION

These indexing values s are then used in conjunction with values v as in input for q θ (y|z), which is modelled with a small network f θ (•): (abusing notation to indicate element-wise indexing with v[s]):

L error = E s∼p θ [log f θ (v[s])].

To enable sampling from the discrete variables s, we use the Gumbel-Max trick (Gumbel (1954) ), denoted gumb(), re-parameterizing the expectation E s∼p with uniform noise ∼ U (0, 1):

EQUATION

As the argmax is not differentiable, we approximate this expectation using the Gumbel-Softmax trick (Maddison et al. (2016) ; Jang et al. (2016) ), which generates samples that smoothly deform into one-hot samples as the softmax temperature τ approaches 0. Using the inner product (denoted •) of the approximate one-hot samples and v, we create samples from z:

EQUATION

In practice, we anneal τ from 1.0 to 0.5 during the training process, as proposed by Yang et al. (2017) to reduce gradient variance initially, at the risk of introducing bias.

To conclude our derivation, we use a fixed SQUAD distribution to model the variational marginal r(z) as shown in figure 5 . We can then derive the KL term analytically following the definition for discrete distributions. Using the fact that the KL divergence is additive for independent variables, we get our final loss:

EQUATION

For the remainder of this work, we will refer to the latent variables as z in lieu of s, for clarity.

At test time, we can approximate the predictive function p(y * |x * ) for a new data-point x * by taking T samples from the latent variables z i.e. ẑt ∼ p(z|x * ), and averaging the predictions for y * :

EQUATION

A downside of mean-field variational approximations is that latents are assumed to be uncorrelated.

We can bring some correlation to the latents, and extend the flexibility of the proposed model, by creating a hierarchical set of latent variables. We maintain a mean-field assumption for the prior, and the joint posterior distribution of L layers of latents factorizes as follows:

EQUATION

With q θ (y|z 1 , . . . , z L ) = q θ (y|z L ). This is straightforwardly implemented with a simple ancestral sampling scheme.

Interestingly, the strong quantization proposed in our method can itself be considered an additional information bottleneck, as it exactly upper-bounds the number of bits per latent variable. Such bottlenecks are theorized to have a beneficial effect on generalization (Tishby et al. (2000) ; Achille & Soatto (2016); Alemi et al. (2017; 2016) ), and we can directly control this bottleneck by varying the number of quantization bins.

The computational complexity of the method, as well as the number of model parameters θ, scale linearly in C, i.e. O(C) (with C the number of quantization bins). It is thus suitable for large-scale inference. We would like to stress that the proposed method differs from work that leverages the Gumbel-Softmax trick to model categorical latent variables: our proposal models continuous scalar latent variables by quantizing their domain and modeling belief with a multinomial distribution. Categorical latent variable models would incur a much larger polynomial complexity penalty of O(C 2 ).

The method is easily integrated into existing deep neural network architectures, and SQUAD layer implementations are provided at github.com/anonymized.

Matrix-factorization variant:

In the natural image domain, we anticipate a need for a high amount of information per variable, for which a small number of bins does not suffice. To improving the tractability of using a large number of quantization bins, we propose a variant of SQUAD that uses a matrix factorization scheme to improve the parameter efficiency. Formally, equation 4 becomes:

p(s k = c|x) = softmax(w k,c (w k x + b k ) + b k,c ),

with full layer weights W and W respectively of shape (K, B, C) and (|X|, K, B), where K denotes the number of neurons, B the number of factorization inputs, C number of quantization bins and |X| the input dimensionality. To improve the parameter efficiency, we can learn W per layer as well, resulting in shape (|X|, 1, B), which is found to be beneficial for large C by the hyperparameter search presented in section 5. We depict this alternative model on the right side of figure 4 and will refer to it as SQUAD-factorized. We leave further extensions such as Network-in-Network (Lin et al. (2013) ) for future work. Figure 5 : In the IB bound, the marginal p(z) is approximated with a fixed distribution r(z). Using our proposed SQUAD distribution we can impose a variety of interesting forms for r(z) via the spacing v and weighting r(z k = v c ) of the quantization bins. For the values v, we compare linearly spaced bins (left) versus bins with equal probability mass under a normal distribution (right). Furthermore, we explore the effect of allowing the bin values to be optimized with SGD on a per-neuron or per-layer basis, to allow the model to optimize the quantization scheme with the highest fidelity. For the prior probabilities, we explore a uniform prior (left) and probability mass of the bins under a normal distribution (middle). MCdropout model with a similar number of weights, SQUAD appears to have a slight -though not significant-advantage, despite the strong quantization scheme, especially at low risk guarantees. Deep ensembles improve results for all methods, which fits the hypothesis that ensembles integrate over a form of weight uncertainty. When comparing the optimal hyperparameters found for fash-ionMNIST on the similar notMNIST dataset, we find that SQUAD shows strong performance, as shown in table 2. This provides some evidence that despite the increase in number hyperparameters in SQUAD, the optimal settings are more robust to a change in the data distribution. 

4. RELATED WORK:



5.2. NATURAL IMAGES:

To explore larger models trained on natural image datasets, we re-tune hyper-parameters on 256latent 2-layer models over 100 TPE evaluations. As SVHN contains natural images in color, we anticipate a need for a higher amount of information per variable. We thus explore the effect of the matrix-factorized variant.

As shown in table 3 , SQUAD-factorized outperforms the non-factorized variant. Considering the computational cost at the optimum of a 4-neuron factorization (B = 4) with C=37 quantization bins, the model clocks 3.4 million weights. In comparison, the optimum for the presented MCdropout results has C=11, using 9.0 million weights. On an NVIDIA Titan Xp, the dropout baseline takes 13s per epoch on average, while SQUAD-factorized spans just 9s.

To evaluate the sample efficiency of the methods, we compare results at T = 4 samples. We find that SQUAD's results suffer less from under-sampling the predictive distribution than MCdropout. We tentatively attribute the sample efficiency to the flexible approximating posterior on the activations, which is in stark contrast to the rigid approximating distribution that MCdropout imposes on the weights. In conclusion, SQUAD comes out favorably in a resource-constrained environment.

5.3. ANALYSIS OF LATENT VARIABLE DISTRIBUTIONS:

In order to evaluate if the proposed variational distribution does not simply collapse into single mode predictions, we want to find out what type of distributions the model predicts over the latent variables. We visualize the forms of predicted distributions in figure 7a . Although this showcases only a small subset of potential multi-modal behavior that emerges, this demonstrates that the model indeed utilizes the distribution to its full potential. To provide an intuition on how these predicted distributions emerge, we present figure 9 in the appendix. In figure 8 we visualize one of the activation functions that the method learns for a 1-dimensional input SQUAD-factorized model. The learned activation functions resemble "peaked" sigmoid activations, which can be interpreted as a combination of an RBF kernel and sigmoid. This provides food for thought on how non-linearity's for conventional neural networks can be designed, and the effect of using such a non-linearity can be studied in further work.

6. DISCUSSION:

In this work, we have proposed a new flexible class of variational distributions. To measure the effectiveness for real world classification, we applied the class to a deep variational information bottleneck model. By placing a quantization-based distribution on the activations, we can compute uncertainty estimates over the outputs. We proposed an evaluation scheme motivated by the need in real-world domains to guarantee a minimal risk. The results presented indicate that SQUAD provides an improvement over plain neural networks and Gaussian information bottleneck models.

In comparison to a MCDropout model, which approximates a Bayesian neural network, we get competitive performance. Moreover, qualitatively we find that the flexible distribution is used to its full advantage is sample efficient. The method learns interesting non-linearity's, is tractable and scale-able, and as the output domain is constrained, no batch normalization techniques are required.

Various directions for future work arise. The improvement of ensemble methods over individual models indicates that there remains room for improvement for capturing the full uncertainty of the output, and thus a fully Bayesian approach to SQUAD which would include weight uncertainty, shows promise. The flexible class allows us to define a wide variety of interesting priors, which provides opportunity to study interesting priors that are hard to define as a continuous density. Likewise, more effective initialization of parameters for the proposed method requires further attention.

Orthogonally, the proposed class can be applied to other variational objectives as well, such as the variational auto-encoder. Finally, the discretized nature of the variables allows for the analytical computation of other divergences such as mutual information and the Jensen-Shannon divergence, the effectiveness of which remains to be studied.

Figure 9 : This figure serves to provide intuition on how a variety of distributions come about in our model. We show the set of weights used to predict the probability for the C bins of a randomly selected latent variable z l=1,k=12 from the first layer in a converged 2-layer SQUAD model (reshaped to a 28x28 squares for comparison with the data). We then present 5 data-points for which the neuron predicts a stereotypical distribution, as visualized in the last bar-plot.

We deviate slightly fromGeifman & El-Yaniv (2017) in that we use Softmax Response (SR) -the probability taken from the softmax output for the most likely class -as the confidence score for all methods. Geifman & El-Yaniv (2017) instead proposed to use the variance of the probabilities for MCdropout, but our experiments showed that SR paints MCDropout in a more favorable light.

2 All models are optimized with ADAM (Kingma & Ba (2014)), weight initialization as proposed byHe et al. (2015), a weight decay of 10 -5 and adaptive learning rate decay scheme -10x reduction after 10 epochs of no validation accuracy improvement-and use early stopping after 20 epochs of no

improvement. 3 We found this baseline to perform stronger in comparison to conventional ReLU MCdropout models, under equal number of latent variables.

Figures and Tables:

Figure 1: DLVMs have layers of stochastic latent variables.

; Rezende et al. (2014); Alemi et al. (2016); Achille & Soatto (2016)).

t e x i t s h a 1 _ b a s e 6 4 = " o m h p C f m S D g P n tJ K R 3 V 0 0 E O H 6 5 f s = " > A A A F u H i c l Z T f b t M w F M a 9 s c I o / z a 4 5 M a i m t j Q V j U V E h V o 0 g S d h E Q v B l r X b k 1 U H M d p r D p 2 s J 1 1 X Z a X 4 B J 4 M d 4 G J 2 t Z k o k L L F U 6 P b / v O 8 d 2 5 O N G j C r d a v 1 e W b 2 z V r t 7 b / 1 + / c H D R 4 + f b G w + P V E i l p j 0 s W B C D l 2 k C K O c 9 D X V j A w j S V D o M j J w p x 8 y P j g n U l H B j / U 8 I k 6 I J p z 6 F C N t U s N o b O u A a D T e a L S a r X z B 2 4 G 1 C B p g s Y 7 G m 2 s / b E / g O C R c Y 4 a U G l m t S D s J k p p i R t K 6 H S s S I T x F E z I y I U c h U U 6 S b z i F W y b j Q V 9 I 8 + M a 5 t m y Q / s d J 6 E 8 i j X h 2 B g g Y k z M o M n v d W C e L + m P L S f J S u V is 7 a g g b C z 5 1 I N j 6 2 8 i y o a k s D c h Z T E z 9 W 5 I c + Y a 5 y W h b F k S 8 1 C

r X b k 1 U H M d p r D p 2 s J 1 1 X Z a X 4 B J 4 M d 4 G J 2 t Z k o k L L F U 6 P b / v O 8 d 2 5 O N G j C r d a v 1 e W b 2 z V r t 7 b / 1 + / c H D R 4 + f b G w + P V E i l p j 0 s W B C D l 2 k C K O c 9 D X V j A w j S V D o M j J w p x 8 y P j g n U l H B j / U 8 I k 6 I J p z 6 F C N t U s N o b O u A a D T e a L S a r X z B 2 4 G 1 C B p g s Y 7 G m 2 s / b E / g O C R c Y 4 a U G l m t S D s J k p p i R t K 6 H S s S I T x F E z I y I U c h U U 6 S b z i F W y b j Q V 9 I 8 + M a 5 t m y Q / s d J 6 E 8 i j X h 2 B g g Y k z M o M n v d W C e L + m P L S f J S u V is 7 a g g b C z 5 1 I N j 6 2 8 i y o a k s D c h Z T E z 9 W 5 I c + Y a 5 y W h b F k S 8 1 C

r X b k 1 U H M d p r D p 2 s J 1 1 X Z a X 4 B J 4 M d 4 G J 2 t Z k o k L L F U 6 P b / v O 8 d 2 5 O N G j C r d a v 1 e W b 2 z V r t 7 b / 1 + / c H D R 4 + f b G w + P V E i l p j 0 s W B C D l 2 k C K O c 9 D X V j A w j S V D o M j J w p x 8 y P j g n U l H B j / U 8 I k 6 I J p z 6 F C N t U s N o b O u A a D T e a L S a r X z B 2 4 G 1 C B p g s Y 7 G m 2 s / b E / g O C R c Y 4 a U G l m t S D s J k p p i R t K 6 H S s S I T x F E z I y I U c h U U 6 S b z i F W y b j Q V 9 I 8 + M a 5 t m y Q / s d J 6 E 8 i j X h 2 B g g Y k z M o M n v d W C e L + m P L S f J S u V is 7 a g g b C z 5 1 I N j 6 2 8 i y o a k s D c h Z T E z 9 W 5 I c + Y a 5 y W h b

Figure 3: SQUAD quantizes the domain of z to model a flexible and tractable variational distribution.

Figure 4: The left diagram visualizes the computational graph of SQUAD at training time, providing a detailed view on how an individual latent variable is sampled. The right diagram visualizes how the proposed matrix-factorization variant improves the parameter efficiency of the model.

Figure6: Risk/coverage curve (with log-axes for discernibility) of 2-layer models on notMNIST. Lines closer to the lower-right are better. As the selective classifier lowers the confidence threshold, coverage increase at the cost of greater classification risk. Area around curves represents 90% confidence bounds computed using 10 initializations/splits.

Figure 7: By analyzing the emerging predicted distributions of individual neurons in a converged SQUAD model, we find that the flexible variational distribution is used to its full advantage. Figure (A) visualizes a subset of interesting stereotypical distributions we hope to find in the model. Figure (B) summarizes distributions predicted by the model similar to stereotypes, discovered by looking at predicted distributions with low KL.Figure (C) shows how often distributions similar to stereotypes arise, as measured by the KL distance (lower KL is closer to stereotypes).

Figure8: By using a 1-dimensional matrix factorization for a SQUAD-factorized distribution, we can visualize the type of (stochastic) activation functions learned by the method. After training the model as usual on fashion-MNIST, we take a random neuron from the first layer. We visualize how the predicted distribution of the output changes as a function of the 1-dimensional input. The left y-axis indicates the prob(ability per value as shown using the green line. The right y-axis indicates the value and in blue the most likely value is shown, and the gray dots represent samples from the neuron. The red line depicts the expected output of the neuron. The shape of the expected output is akin to a peaky sigmoid activation, and similar shapes are found in the other neurons of the network as well. This provides food for thought on the design of activation functions for conventional neural networks.

Outside the realm of DLVMs, other methods have been explored for predictive uncertainty.Lakshminarayanan et al. (2017)  propose deep ensembles: straightforward averaging of predictions from a

Table content: <table><tr><td/><td colspan="2">r(z)</td><td/><td/><td/><td/><td/><td colspan="2">r(z)</td><td/><td/><td/><td/><td/><td>r(z)</td><td/><td/><td/><td/><td/><td/></tr><tr><td>z</td><td>-1.6</td><td>-1.2</td><td>-.8</td><td>-.4</td><td>0</td><td>.4 .8 1.2 1.6</td><td>z</td><td>-1.6</td><td>-1.2</td><td>-.8</td><td>-.4</td><td>0</td><td>.4 .8 1.2 1.6</td><td>z</td><td>-1.4</td><td>-.8</td><td>-.5</td><td>0</td><td>.5</td><td>.8</td><td>1.4</td></tr></table>

SGR coverage results on Fashion MNIST. We present coverage percentage of the test dataset for three pre-determined risk-guarantees (one per column), where higher coverage is better, as well as negative log-likelihood and overall accuracy. The results indicate that SQUAD provides competitive uncertainty, especially at low-risk guarantees. Bayesian approximations via deep ensembles improve coverage all over the board, and for SQUAD in particular. When SGR can not guarantee the required risk level at high probability, 0% coverage is reported. (2 std. deviations shown in parentheses, optimal results in bold.)

Table content: <table><tr><td>Fashion MNIST</td><td colspan="4">cov@risk .5% cov@risk 1% cov@risk 2% NLL</td><td>Acc.</td></tr><tr><td>Plain MLP</td><td colspan="3">29.1 (±20.71) 45.9 (±4.04) 60.4 (±3.17)</td><td colspan="2">0.408 (±.036) 87.7 (±.42)</td></tr><tr><td>Maxout MCDropout</td><td>41.9 (±9.86)</td><td>56.5 (±2.30)</td><td colspan="3">69.9 (±1.48) 0.299 (±.008) 89.5 (±.28)</td></tr><tr><td>DLGM</td><td>0.0 (±.00)</td><td colspan="2">33.5 (±2.42) 47.0 (±1.47)</td><td colspan="2">0.446 (±.007) 84.3 (±.15)</td></tr><tr><td>SQUAD</td><td>42.9 (±7.19)</td><td colspan="2">58.3 (±3.06) 69.5 (±1.55)</td><td colspan="2">0.293 (±.008) 89.5 (±.35)</td></tr><tr><td>Deep Ensemble</td><td colspan="4">cov@risk .5% cov@risk 1% cov@risk 2% NLL</td><td>Acc.</td></tr><tr><td>Plain MLP Ensemble</td><td>40.6</td><td>58.3</td><td>70.2</td><td>0.296</td><td>89.3</td></tr><tr><td colspan="2">Max. MCD. Ensemble 48.2</td><td>59.1</td><td>72.2</td><td>0.271</td><td>90.2</td></tr><tr><td>DLGM Ensemble</td><td>0.0</td><td>34.3</td><td>47.8</td><td>0.435</td><td>84.7</td></tr><tr><td>SQUAD Ensemble</td><td>47.5</td><td>61.6</td><td>73.1</td><td>0.273</td><td>90.1</td></tr></table>

SQUAD exhibits strong performance on notMNIST using the optimal hyperparameters found for fashionMNIST.

Table content: <table><tr><td>notMNIST</td><td colspan="4">cov@risk .5% cov@risk 1% cov@risk 2% NLL</td><td>Acc.</td></tr><tr><td>Plain MLP</td><td>77.4 (±5.20)</td><td colspan="2">85.5 (±1.91) 90.3 (±.87)</td><td colspan="2">0.228 (±.009) 93.3 (±.29)</td></tr><tr><td>Maxout MCDropout</td><td>85.7 (±1.14)</td><td>90.6 (±.62)</td><td>94.2 (±.36)</td><td colspan="2">0.165 (±.003) 95.3 (±.21)</td></tr><tr><td>SQUAD</td><td>87.1 (±1.60)</td><td>91.1 (±.86)</td><td>94.5 (±.50)</td><td colspan="2">0.161 (±.006) 95.4 (±.22)</td></tr><tr><td>Plain MLP Ensemble</td><td>85.9</td><td>90.6</td><td>93.5</td><td>0.175</td><td>94.9</td></tr><tr><td colspan="2">Max. MCD. Ensemble 88.5</td><td>92.8</td><td>95.7</td><td>0.148</td><td>96.0</td></tr><tr><td>SQUAD Ensemble</td><td>90.7</td><td>93.5</td><td>96.1</td><td>0.137</td><td>96.2</td></tr></table>

Results on SVHN indicate that the quantization scheme imposed by SQUAD models might hinder performance, but that this is effectively compensated by the SQUAD-factorized variant using a larger amount of bins. Even with T = 4 MC samples at test time, SQUAD performs well.

Table content: <table><tr><td>MLP K=256 (SVHN)</td><td colspan="4">cov@risk .5% cov@risk 1% cov@risk 2% NLL</td><td>Acc.</td></tr><tr><td>Plain MLP</td><td>0.0 (±.00)</td><td>0.0 (±.00)</td><td colspan="2">36.3 (±2.84) 0.758 (±.065) 83.1 (±.42)</td></tr><tr><td>Maxout MCDropout</td><td>0.0 (±.00)</td><td colspan="3">50.7 (±1.42) 65.0 (±2.32) 0.480 (±.020) 86.4 (±.71)</td></tr><tr><td>SQUAD-factorized</td><td colspan="4">18.4 (±30.33) 53.9 (±1.81) 66.7 (±2.10) 0.454 (±.021) 86.7 (±.88)</td></tr><tr><td>SQUAD</td><td>1.7 (±6.75)</td><td colspan="3">42.8 (±2.25) 59.3 (±1.38) 0.534 (±.005) 84.6 (±.13)</td></tr><tr><td>Max. MCDropout T=4</td><td>0.0 (±.00)</td><td colspan="3">38.5 (±3.85) 57.6 (±2.66) 0.562 (±.016) 84.9 (±.73)</td></tr><tr><td colspan="5">SQUAD-factorized T=4 10.7 (±26.25) 49.9 (±3.51) 64.5 (±2.45) 0.480 (±.024) 86.2 (±.67)</td></tr><tr><td>SQUAD T=4</td><td>0.0 (±.00)</td><td colspan="2">38.0 (±1.86) 55.6 (±.84)</td><td>0.569 (±.019) 83.7 (±.52)</td></tr></table>

Peer review (iclr2019):
 The authors propose “Stochastic Quantized Activation Distributions” (SQUAD). It quantizes the continuous values of a network activation under a finite number of discrete (non-ordinal) values, and is distributed according to a Gumbel-Softmax distribution. While the topic is interesting, the work could improve by making more precise the benefit of (relaxed) discrete random variables. This will also allow the authors to more precisely display in the experiments why this particular approach is more natural than other baselines (e.g., if multimodality is the issue, compare to a mixture model; if correlation is a difficulty, compare to any structured distribution such as a flow). Derivation-wise, the method ends up resembling Gumbel-Softmax VAEs but under an information bottleneck (discriminative model) setup rather than under a generative model. Unfortunately, that in and of itself is not original. The idea of quantizing a continuous distribution over activations using a multinomial is interesting. However, by ultimately adding Gumbel noise (and requiring a binning procedure), the resulting network ends up looking a lot like continuous values but now constrained under a simplex rather than the real line. Given either the model bias against a true Categorical latent variable, or continuous simplex-valued codes, it seems more natural as a baseline to compare against a mixture of Gaussians. They have a number of hyperparameters that make it difficult to compare without a more rigorous sensitivity analysis (e.g., bin size). Given that the number of bins they use is only 11, I’m also unclear on what the matrix factorization approach benefits from. Is this experimented with and without?