Title: VARIATIONAL AUTOENCODERS WITH JOINTLY OPTIMIZED LATENT DEPENDENCY STRUCTURE

Abstract: We propose a method for learning the dependency structure between latent variables in deep latent variable models. Our general modeling and inference framework combines the complementary strengths of deep generative models and probabilistic graphical models. In particular, we express the latent variable space of a variational autoencoder (VAE) in terms of a Bayesian network with a learned, flexible dependency structure. The network parameters, variational parameters as well as the latent topology are optimized simultaneously with a single objective. Inference is formulated via a sampling procedure that produces expectations over latent variable structures and incorporates top-down and bottom-up reasoning over latent variable values. We validate our framework in extensive experiments on MNIST, Omniglot, and CIFAR-10. Comparisons to state-of-the-art structured variational autoencoder baselines show improvements in terms of the expressiveness of the learned model.

1. INTRODUCTION:

Deep latent variable models offer an effective method for automatically learning structure from data. By explicitly modeling the data distribution using latent variables, these models are capable of learning compressed representations that are then relevant for downstream tasks. Such models have been applied across a wide array of domains, such as images (Gregor et al., 2014; Kingma & Welling, 2014; Rezende et al., 2014; Gregor et al., 2015) , audio (Chung et al., 2015; Fraccaro et al., 2016) , video (He et al., 2018; Yingzhen & Mandt, 2018) , and text (Bowman et al., 2016; Krishnan et al., 2017) . However, despite their success, latent variable models are often formulated with simple (e.g. Gaussian) distributions, making independence assumptions among the latent variables. That is, each latent variable is sampled independently. Ignoring the dependencies between latent variables limits the flexibility of these models, negatively impacting the model's ability to fit the data.

In general, structural dependencies can be incorporated into all phases of a forward process, including inference, latent model, and output space: Normalizing flows (Rezende & Mohamed, 2015) , for instance, accounts for dependencies during inference by learning a mapping from a simple distribution to a more complex distribution that contains these dependencies. Structured output networks (Lehrmann & Sigal, 2017) , on the other hand, directly predict an expressive non-parametric output distribution. On the modeling side, one can add dependencies by constructing a hierarchical latent representation (Dayan et al., 1995) . These structures consist of conditional (empirical) priors, in which one latent variable forms a prior on another latent variable. While this conditional distribution may take a simple form, marginalizing over the parent variable can result in an arbitrarily complex distribution. Models with these more flexible latent dependency structures have been shown to result in improved performance (Sønderby et al., 2016; Burda et al., 2016; Kingma et al., 2016) . However, despite the benefits of including additional structure in these models, their dependency structures have so far been predefined, potentially limiting the performance of this approach.

In this work, we propose a method for learning dependency structures in latent variable models. Structure learning is a difficult task with a long history in the graphical models community (Koller & Friedman, 2009) . Over the years, it has been tackled from several perspectives, including constraint-based approaches (Cheng et al., 2002; Lehmann & Romano, 2008) , optimization of structure scores (Kass & Raftery, 1995; Heckerman et al., 1995; Barron et al., 1998) , Bayesian model averaging (Heckerman et al., 1999; Koivisto & Sood, 2004) , and many more. Unfortunately, the underlying objectives are often limited to graphs of a particular form (e.g., limited tree width), prohibitively expensive, or difficult to integrate with the gradient-based optimization techniques of modern neural networks. Here, we discuss an end-to-end approach for general graph structures introducing minimal complexity overhead. In particular, we introduce a set of binary global variables to gate the latent dependencies. The whole model (including its structure) is jointly optimized with a single stochastic variational inference objective. In our experimental validation, we show that the learned dependency structures result in models that more accurately model the data distribution, outperforming several common predefined latent dependency structures.

2. BACKGROUND 2.1 VARIATIONAL INFERENCE & VARIATIONAL AUTOENCODERS:

A latent variable model, defined by the joint distribution, p θ (x, z) = p θ (x z)p θ (z), models each data example, x, using a local latent variable, z, and global parameters, θ. p θ (x z) denotes the conditional likelihood, and p θ (z) denotes the prior. Latent variable models are capable of capturing the structure present in data, with z forming a compressed representation of each data example. Unfortunately, inferring the posterior, p θ (z x), is typically computationally intractable, prompting the use of approximate inference techniques. Variational inference (Jordan et al., 1999) introduces an approximate posterior, q φ (z x), and optimizes variational parameters, φ, to minimize the KL-divergence to the true posterior, KL(q φ (z x) p θ (z x)). As this quantity cannot be evaluated directly, the following relation is used: log p θ (x) = KL(q φ (z x) p θ (z x)) + L(x; θ, φ),

where L(x; θ, φ) is the evidence lower bound (ELBO), defined as L(x; θ, φ) = E q φ (z x) [log p θ (x z)] -KL(q φ (z x) p θ (z)).

(2)

In Eq. (1), log p θ (x) is independent of φ, so we can minimize the KL divergence term, i.e. perform approximate inference, by maximizing L(x; θ, φ) w.r.t. q φ (z x). Further, because KL divergence is non-negative, L(x; θ, φ) is a lower bound on log p θ (x), meaning we can then learn the model parameters by maximizing L(x; θ, φ) w.r.t. θ.

Variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) amortize inference optimization across data examples by parameterizing q φ (z x) as a separate inference model, then jointly optimizing the model parameters θ and φ. VAEs instantiate both the inference model and latent variable model with deep networks, allowing them to scale to high-dimensional data. However, VAEs are typically implemented with basic graphical structures and simple, unimodal distributions (e.g. Gaussians). For instance, the dimensions of the prior are often assumed to be independent, p θ (z) = ∏ m p θ (z m ), with a common assumption being a fixed standard Gaussian: p θ (z) = N (z; 0, I). Similarly, approximate posteriors often make the mean field assumption, q φ (z x) = ∏ m q φ (z m x). Independence assumptions such as these may be overly restrictive, thereby limiting modeling capabilities.

2.2. EMPIRICAL PRIORS THROUGH LATENT DEPENDENCY STRUCTURE:

One technique for improving the expressive capacity of latent variable models is through incorporating dependency structure among the latent variables, forming a hierarchy (Dayan et al., 1995;  zm < l a t e x i t s h a 1 _ b a s e 6 4 = " F Q U F B 8 L N x t S L j q O l + X L S 2 P 4 F 8 M M = " > A A A C K H i c b V D N T g I x G G z 9 R R Q F P X p p B B N P Z J e L H k m 8 e M T E B R L Y k G 7 p Q k P b 3 b R d I 2 7 2 G b z q K / g 0 3 g x X n 8 S y c H D B S Z p M Z r 4 v 3 3 S C m D N t H G c B d 3 b 3 9 g 8 O S 0 f l 4 5 P K 6 V m 1 d t 7 V U a I I 9 U j E I 9 U P s K a c S e o Z Z j j t x 4 p i E X D a C 2 b 3 S 7 / 3 T J V m k X w y 8 5 j 6 A k 8 k C x n B x k p e 4 3 U k G q N q 3 W k 6 O d A 2 c d e k D t b o j G q w M h x H J B F U G s K x 1 g P X i Y 2 f Y m U Y 4 T Q r D x N N Y 0 x m e E I H l k o s q P b T P G 2 G r q 0 y R m G k 7 J M G 5 e r f j R Q L r e c i s J M C m 6 n e 9 J b i f 9 4 g M e G d n z I Z J 4 Z K s j o U J h y Z C C 2 / j s Z M U W L 4 3 B J M F L N Z E Z l i h Y m x B R W u T B S O p 4 y 8 Z E W V y j x X Q Q x E V r Y F u p t 1 b Z N u q + k 6 T f e x V W + 3 1 l W W w C W 4 A j f A B b e g D R 5 A B 3 i A A A b e w D v 4 g J / w C 3 7 D x W p 0 B 6 5 3 L k A B 8 O c X c d K m E A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " F Q U F B 8 L N x t S L j q O l + X L S 2 P 4 F 8 M M = " > A A A C K H i c b V D N T g I x G G z 9 R R Q F P X p p B B N P Z J e L H k m 8 e M T E B R L Y k G 7 p Q k P b 3 b R d I 2 7 2 G b z q K / g 0 3 g x X n 8 S y c H D B S Z p M Z r 4 v 3 3 S C m D N t H G c B d 3 b 3 9 g 8 O S 0 f l 4 5 P K 6 V m 1 d t 7 V U a I I 9 U j E I 9 U P s K a c S e o Z Z j j t x 4 p i E X D a C 2 b 3 S 7 / 3 T J V m k X w y 8 5 j 6 A k 8 k C x n B x k p e 4 3 U k G q N q 3 W k

6 O d A 2 c d e k D t b o j G q w M h x H J B F U G s K x 1 g P X i Y 2 f Y m U Y 4 T Q r D x N N Y 0 x m e E I H l k o s q P b T P G 2 G r q 0 y R m G k 7 J M G 5 e r f j R Q L r e c i s J M C m 6 n e 9 J b i f 9 4 g M e G d n z I Z J 4 Z K s j o U J h y Z C C 2 / j s Z M U W L 4 3 B J M F L N Z E Z l i h Y m x B R W u T B S O p 4 y 8 Z E W V y j x X Q Q x E V r Y F u p t 1 b Z N u q + k 6 T f e x V W + 3 1 l W W w C W 4 A j f A B b e g D R 5 A B 3 i A A A b e w D v 4 g J / w C 3 7 D x W p 0 B 6 5 3 L k A B 8 O c X c d K m E A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " F Q U F B 8 L N x t S L j q O l + X L S 2 P 4 F 8 M M = " > A A A C K H i c b V D N T g I x G G z 9 R R Q F P X p p B B N P Z J e L H k m 8 e M T E B R L Y k G 7 p Q k P b 3 b R d I 2 7 2 G b z q K / g 0 3 g x X n 8 S y c H D B S Z p M Z r 4 v 3 3 S C m D N t H G c B d 3 b 3 9 g 8 O S 0 f l 4 5 P K 6 V m 1 d t 7 V U a I I 9 U j E I 9 U P s K a c S e o Z Z j j t x 4 p i E X D a C 2 b 3 S 7 / 3 T J V m k X w y 8 5 j 6 A k 8 k C x n B x k p e 4 3 U k G q N q 3 W k 6 O d A 2 c d e k D t b o j G q w M h x H J B F U G s K x 1 g P X i Y 2 f Y m U Y 4 T Q r D x N N Y 0 x m e E I H l k o s q P b T P G 2 G r q 0 y R m G k 7 J M G 5 e r f j R Q L r e c i s J M C m 6 n e 9 J b i f 9 4 g M e G d n z I Z J 4 Z K s j o U J h y Z C C 2 / j s Z M U W L 4 3 B J M F L N Z E Z l i h Y m x B R W u T B S O p 4 y 8 Z E W V y j x X Q Q x E V r Y F u p t 1 b Z N u q + k 6 T f e x V W + 3 1 l W W w C W 4 A j f A B b e g D R 5 A B 3 i A A A b e w D v 4 g J / w C 3 7 D x W p 0 B 6 5 3 L k A B 8 O c X c d K m E A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " F Q U F B 8 L N x t S L j q O l + X L S 2 P 4 F 8 M M = " > A A A C K H i c b V D N T g I x G G z 9 R R Q F P X p p B B N P Z J e L H k m 8 e M T E B R L Y k G 7 p Q k P b 3 b R d I 2 7 2 G b z q K / g 0 3 g x X n 8 S y c H D B S Z p M Z r 4 v 3 3 S C m D N t H G c B d 3 b 3 9 g 8 O S 0 f l 4 5 P K 6 V m 1 d t 7 V U a I I 9 U j E I 9 U P s K a c S e o Z Z j j t x 4 p i E X D a C 2 b 3 S 7 / 3 T J V m k X w y 8 5 j 6 A k 8 k C x n B x k p e 4 3 U k G q N q 3 W k 6 O d A 2 c d e k D t b o j G q w M h x H J B F U G s K x 1 g P X i Y 2 f Y m U Y 4 T Q r D x N N Y 0 x m e E I H l k o s q P b T P G 2 G r q 0 y R m G k 7 J M G 5 e r f j R Q L r e c i s J M C m 6 n e 9 J b i f 9 4 g M e G d n z I Z J 4 Z K s j o U J h y Z C C 2 / j s Z M U W L 4 3 B J M F L N Z E Z l i h Y m x B R W u T B S O p 4 y 8 Z E W V y j x X Q Q x E V r Y F u p t 1 b Z N u q + k 6 T f e x V W + 3 1 l W W w C W 4 A j f A B b e g D R 5 A B 3 i A A A b e w D v 4 g J / w C 3 7 D x W p 0 B 6 5 3 L k A B 8 O c X c d K m E A = = < / l a t e x i t > p✓(z) = Q n p✓(zn|zn+1) < l a t e x i t s h a 1 _ b a s e 6 4 = " U h Y X z I E 1 9 f 4 U K d U 3 5 c K 5 F + u 1 s v A = " > A A A C b X i c b V D R S h t B F J 2 s r U 1 j t d H S J 0 s Z j K I i h N 2 8 m B c h 4 E s e F R o V s m G Z n d x N h s z O D j N 3 x W T d X / J r + t K H + g v + g p M Y s N E e G D h z z r 3 c e 0 + s p b D o + 3 8 q 3 t q H j + u f q p 9 r G 1 8 2 t 7 7 W t 3 e u b J Y b D j 2 e y c z c x M y C F A p 6 K F D C j T b A 0 l j C d T w 5 n / v X t 2 C s y N Q v n G o Y p G y k R C I 4 Q y d F 9 e 6 + j k I c A z J 6 F K Y M x 3 F S z M p j e k Z D b b J h p O i r X 4 R x Q m d l V K i S 3 t P X 3 0 l Q H u 9 H 9 Y b f 9 B e g 7 0 m w J A 2 y x E W 0 X d k M h x n P U 1 D I J b O 2 H / g a B w U z K L i E s h b m F j T j E z a C v q O K p W A H x e L k k h 4 4 Z U i T z L i n k C 7 U f z s K l l o 7 T W N X O b / K v v X m 4 v + 8 f o 5 J e 1 A I p X M E x V 8 G J b m k m N F 5 f n Q o D H C U U 0 c Y N 8 L t S v m Y G c b R p b w y Z W S Y H g t + V 6 6 q o B Z 7 r Y h x W t Z c g M H b u N 6 T q 1 Y z 8 J v B Z a v R a S + j r J J d s k e O S E B O S Y d 0 y Q X p E U 4 e y G / y l z x W n r z v 3 g / v 5 0 u p V 1 n 2 f C M r 8 A 6 f A b 9 f v N A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U h Y X z I E 1 9 f 4 U K d U 3 5 c K 5 F + u 1 s v A = " > A A A C b X i c b V D R S h t B F J 2 s r U 1 j t d H S J 0 s Z j K I i h N 2 8 m B c h 4 E s e F R o V s m G Z n d x N h s z O D j N 3 x W T d X / J r + t K H + g v + g p M Y s N E e G D h z z r 3 c e 0 + s p b D o + 3 8 q 3 t q H j + u f q p 9 r G 1 8 2 t 7 7 W t 3 e u b J Y b D j 2 e y c z c x M y C F A p 6 K F D C j T b A 0 l j C d T w 5 n / v X t 2 C s y N Q v n G o Y p G y k R C I 4 Q y d F 9 e 6 + j k I c A z J 6 F K Y M x 3 F S z M p j e k Z D b b J h p O i r X 4 R x Q m d l V K i S 3 t P X 3 0 l Q H u 9 H 9 Y b f 9 B e g 7 0 m w J A 2 y x E W 0 X d k M h x n P U 1 D I J b O 2 H / g a B w U z K L i E s h b m F j T j E z a C v q O K p W A H x e L k k h 4 4 Z U i T z L i n k C 7 U f z s K l l o 7 T W N X O b / K v v X m 4 v + 8 f o 5 J e 1 A I p X M E x V 8 G J b m k m N F 5 f n Q o D H C U U 0 c Y N 8 L t S v m Y G c b R p b w y Z W S Y H g t + V 6 6 q o B Z 7 r Y h x W t Z c g M H b u N 6 T q 1 Y z 8 J v B Z a v R a S + j r J J d s k e O S E B O S Y d 0 y Q X p E U 4 e y G / y l z x W n r z v 3 g / v 5 0 u p V 1 n 2 f C M r 8 A 6 f A b 9 f v N A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U h Y X z I E 1 9 f 4 U K d U 3 5 c K 5 F + u 1 s v A = " > A A A C b X i c b V D R S h t B F J 2 s r U 1 j t d H S J 0 s Z j K I i h N 2 8 m B c h 4 E s e F R o V s m G Z n d x N h s z O D j N 3 x W T d X / J r + t K H + g v + g p M Y s N E e G D h z z r 3 c e 0 + s p b D o + 3 8 q 3 t q H j + u f q p 9 r G 1 8 2 t 7 7 W t 3 e u b J Y b D j 2 e y c z c x M y C F A p 6 K F D C j T b A 0 l j C d T w 5 n / v X t 2 C s y N Q v n G o Y p G y k R C I 4 Q y d F 9 e 6 + j k I c A z J 6 F K Y M x 3 F S z M p j e k Z D b b J h p O i r X 4 R x Q m d l V K i S 3 t P X 3 0 l Q H u 9 H 9 Y b f 9 B e g 7 0 m w J A 2 y x E W 0 X d k M h x n P U 1 D I J b O 2 H / g a B w U z K L i E s h b m F j T j E z a C v q O K p W A H x e L k k h 4 4 Z U i T z L i n k C 7 U f z s K l l o 7 T W N X O b / K v v X m 4 v + 8 f o 5 J e 1 A I p X M E x V 8 G J b m k m N F 5 f n Q o D H C U U 0 c Y N 8 L t S v m Y G c b R p b w y Z W S Y H g t + V 6 6 q o B Z 7 r Y h x W t Z c g M H b u N 6 T q 1 Y z 8 J v B Z a v R a S + j r J J d s k e O S E B O S Y d 0 y Q X p E U 4 e y G / y l z x W n r z v 3 g / v 5 0 u p V 1 n 2 f C M r 8 A 6 f A b 9 f v N A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U h Y X z I E 1 9 f 4 U K d U 3 5 c K 5 F + u 1 s v A = " > A A A C b X i c b V D R S h t B F J 2 s r U 1 j t d H S J 0 s Z j K I i h N 2 8 m B c h 4 E s e F R o V s m G Z n d x N h s z O D j N 3 x W T d X / J r + t K H + g v + g p M Y s N E e G D h z z r 3 c e 0 + s p b D o + 3 8 q 3 t q H j + u f q p 9 r G 1 8 2 t 7 7 W t 3 e u b J Y b D j 2 e y c z c x M y C F A p 6 K F D C j T b A 0 l j C d T w 5 n / v X t 2 C s y N Q v n G o Y p G y k R C I 4 Q y d F 9 e 6 + j k I c A z J 6 F K Y M x 3 F S z M p j e k Z D b b J h p O i r X 4 R x Q m d l V K i S 3 t P X 3 0 l Q H u 9 H 9 Y b f 9 B e g 7 0 m w J A 2 y x E W 0 X d k M h x n P U 1 D I J b O 2 H / g a B w U z K L i E s h b m F j T j E z a C v q O K p W A H x e L k k h 4 4 Z U i T z L i n k C 7 U f z s K l l o 7 T W N X O b / K v v X m 4 v + 8 f o 5 J e 1 A I p X M E x V 8 G J b m k m N F 5 f n Q o D H C U U 0 c Y N 8 L t S v m Y G c b R p b w y Z W S Y H g t + V 6 6 q o B Z 7 r Y h x W t Z c g M H b u N 6 T q 1 Y z 8 J v B Z a v R a S + j r J J d s k e O S E B O S Y d 0 y Q X p E U 4 e y G / y l z x W n r z v 3 g / v 5 0 u p V 1 n 2 f C M r 8 A 6 f A b 9 f v N A = < / l a t e x i t > zn < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7 Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < / l a t e x i t > zn+1 < l a t e x i t s h a 1 _ b a s e 6 4 = " r H C O l 6 R F Y o x d c B Q Q I K M J M W S V D 5 s = " > A A A C N H i c b V D L S s N A F J 3 4 r N X a V p d u B l t B E E r S j S 4 L b l x W s A 9 o Q 5 h M J + 3 Q m U m Y m Y g 1 5 E v c 6 i / 4 L 4 I 7 c e s 3 O E 2 z M K 0 H L h z O u Z d 7 O H 7 E q N K 2 / W F t b e / s 7 u 2 X D s q H R 5 X j a q 1 + 0 l d h L D H p 4 Z C F c u g j R R g V p K e p Z m Q Y S Y K 4 z 8 j A n 9 8 u / c E j k Y q G 4 k E v I u J y N B U 0 o B h p I 3 m 1 a j M Z + w F 8 T r 1 E X D l p 0 6 s 1 7 J a d A W 4 S J y c N k K P r 1 a 3 K e B L i m B O h M U N K j R w 7 0 m 6 C p K a Y k b Q 8 j h W J E J 6 j K R k Z K h A n y k 2 y 5 C m 8 M M o E B q E 0 I z T M 1 L 8 X C e J K L b h v N j n S M 7 X u L c X / v F G s g x s 3 o S K K N R F 4 9 S i I G d Q h X N Y A J 1 Q S r N n C E I Q l N V k h n i G J s D Z l F b 5 M J Y p m F D + l R Z W I L F d B 9 H l a N g U 6 6 3 V t k n 6 7 5 d g t 5 7 7 d 6 L T z K k v g D J y D S + C A a 9 A B d 6 A L e g C D G L y A V / B m v V u f 1 p f 1 v V r d s v K b U 1 C A 9 f M L x J G q N g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r H C O l 6 R F Y o x d c B Q Q I K M J M W S V D 5 s = " > A A A C N H i c b V D L S s N A F J 3 4 r N X a V p d u B l t B E E r S j S 4 L b l x W s A 9 o Q 5 h M J + 3 Q m U m Y m Y g 1 5 E v c 6 i / 4 L 4 I 7 c e s 3 O E 2 z M K 0 H L h z O u Z d 7 O H 7 E q N K 2 / W F t b e / s 7 u 2 X D s q H R 5 X j a q 1 + 0 l d h L D H p 4 Z C F c u g j R R g V p K e p Z m Q Y S Y K 4 z 8 j A n 9 8 u / c E j k Y q G 4 k E v I u J y N B U 0 o B h p I 3 m 1 a j M Z + w F 8 T r 1 E X D l p 0 6 s 1 7 J a d A W 4 S J y c N k K P r 1 a 3 K e B L i m B O h M U N K j R w 7 0 m 6 C p K a Y k b Q 8 j h W J E J 6 j K R k Z K h A n y k 2 y 5 C m 8 M M o E B q E 0 I z T M 1 L 8 X C e J K L b h v N j n S M 7 X u L c X / v F G s g x s 3 o S K K N R F 4 9 S i I G d Q h X N Y A J 1 Q S r N n C E I Q l N V k h n i G J s D Z l F b 5 M J Y p m F D + l R Z W I L F d B 9 H l a N g U 6 6 3 V t k n 6 7 5 d g t 5 7 7 d 6 L T z K k v g D J y D S + C A a 9 A B d 6 A L e g C D G L y A V / B m v V u f 1 p f 1 v V r d s v K b U 1 C A 9 f M L x J G q N g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r H C O l 6 R F Y o x d c B Q Q I K M J M W S V D 5 s = " > A A A C N H i c b V D L S s N A F J 3 4 r N X a V p d u B l t B E E r S j S 4 L b l x W s A 9 o Q 5 h M J + 3 Q m U m Y m Y g 1 5 E v c 6 i / 4 L 4 I 7 c e s 3 O E 2 z M K 0 H L h z O u Z d 7 O H 7 E q N K 2 / W F t b e / s 7 u 2 X D s q H R 5 X j a q 1 + 0 l d h L D H p 4 Z C F c u g j R R g V p K e p Z m Q Y S Y K 4 z 8 j A n 9 8 u / c E j k Y q G 4 k E v I u J y N B U 0 o B h p I 3 m 1 a j M Z + w F 8 T r 1 E X D l p 0 6 s 1 7 J a d A W 4 S J y c N k K P r 1 a 3 K e B L i m B O h M U N K j R w 7 0 m 6 C p K a Y k b Q 8 j h W J E J 6 j K R k Z K h A n y k 2 y 5 C m 8 M M o E B q E 0 I z T M 1 L 8 X C e J K L b h v N j n S M 7 X u L c X / v F G s g x s 3 o S K K N R F 4 9 S i I G d Q h X N Y A J 1 Q S r N n C E I Q l N V k h n i G J s D Z l F b 5 M J Y p m F D + l R Z W I L F d B 9 H l a N g U 6 6 3 V t k n 6 7 5 d g t 5 7 7 d 6 L T z K k v g D J y D S + C A a 9 A B d 6 A L e g C D G L y A V / B m v V u f 1 p f 1 v V r d s v K b U 1 C A 9 f M L x J G q N g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r H C O l 6 R F Y o x d c B Q Q I K M J M W S V D 5 s = " > A A A C N H i c b V D L S s N A F J 3 4 r N X a V p d u B l t B E E r S j S 4 L b l x W s A 9 o Q 5 h M J + 3 Q m U m Y m Y g 1 5 E v c 6 i / 4 L 4 I 7 c e s 3 O E 2 z M K 0 H L h z O u Z d 7 O H 7 E q N K 2 / W F t b e / s 7 u 2 X D s q H R 5 X j a q 1 + 0 l d h L D H p 4 Z C F c u g j R R g V p K e p Z m Q Y S Y K 4 z 8 j A n 9 8 u / c E j k Y q G 4 k E v I u J y N B U 0 o B h p I 3 m 1 a j M Z + w F 8 T r 1 E X D l p 0 6 s 1 7 J a d A W 4 S J y c N k K P r 1 a 3 K e B L i m B O h M U N K j R w 7 0 m 6 C p K a Y k b Q 8 j h W J E J 6 j K R k Z K h A n y k 2 y 5 C m 8 M M o E B q E 0 I z T M 1 L 8 X C e J K L b h v N j n S M 7 X u L c X / v F G s g x s 3 o S K K N R F 4 9 S i I G d Q h X N Y A J 1 Q S r N n C E I Q l N V k h n i G J s D Z l F b 5 M J Y p m F D + l R Z W I L F d B 9 H l a N g U 6 6 3 V t k n 6 7 5 d g t 5 7 7 d 6 L T z K k v g D J y D S + C A a 9 A B d 6 A L e g C D G L y A V / B m v V u f 1 p f 1 v V r d s v K b U 1 C A 9 f M L x J G q N g = = < / l a t e x i t > (a) Traditional Latent Variable Models x < l a t e x i t s h a 1 _ b a s e 6 4 = " y M K v c s b Q N k v F r B n V k m 2 K + W 2 6 W + g = " > A A A C G X i c b V C 7 T g J B F L 3 r E / G F W t p M B B M r s k u j J d H G E h N 5 J L A h s 8 N d m D A 7 u 5 m Z N R D C R 9 j K 1 9 g Z W y s / x s R h o R D w J J O c n H N v 7 p k T J I J r 4 7 r f z t b 2 z u 7 e f u 4 g f 3 h 0 f H J a O D t v 6 D h V D O s s F r F q B V S j 4 B L r h h u B r U Q h j Q K B z W D 4 M P e b L 6 g 0 j + W z G S f o R 7 Q v e c g Z N V Z q l j p B S E a l b q H o l t 0 M Z J N 4 S 1 K E J W r d w k + n F 7 M 0 Q m m Y o F q 3 P T c x / o Q q w 5 n A a b 6 T a k w o G 9 I + t i 2 V N E L t T 7 K 4 U 3 J t l R 4 J Y 2 W f N C R T / 2 5 M a K T 1 O A r s Z E T N Q K 9 7 c / E / r 5 2 a 8 M 6 f c J m k B i V b H A p T Q U x M 5 n 8 n P a 6 Q G T G 2 h D L F b V b C B l R R Z m x D K 1 f 6 i i Y D z k b T V R V l l i t v + / L W 2 9 k k j U r Z c 8 v e U 6 V Y v V 8 2 l 4 N L u I I b 8 O A W q v A I N a g D g y G 8 w h v M n J n z 7 n w 4 n 4 v R L W e 5 c w E r c L 5 + A Y 5 0 o P 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " y M K v c s b Q N k v F r B n V k m 2 K + W 2 6 W + g = " > A A A C G X i c b V C 7 T g J B F L 3 r E / G F W t p M B B M r s k u j J d H G E h N 5 J L A h s 8 N d m D A 7 u 5 m Z N R D C R 9 j K 1 9 g Z W y s / x s R h o R D w J J O c n H N v 7 p k T J I J r 4 7 r f z t b 2 z u 7 e f u 4 g f 3 h 0 f H J a O D t v 6 D h V D O s s F r F q B V S j 4 B L r h h u B r U Q h j Q K B z W D 4 M P e b L 6 g 0 j + W z G S f o R 7 Q v e c g Z N V Z q l j p B S E a l b q H o l t 0 M Z J N 4 S 1 K E J W r d w k + n F 7 M 0 Q m m Y o F q 3 P T c x / o Q q w 5 n A a b 6 T a k w o G 9 I + t i 2 V N E L t T 7 K 4 U 3 J t l R 4 J Y 2 W f N C R T / 2 5 M a K T 1 O A r s Z E T N Q K 9 7 c / E / r 5 2 a 8 M 6 f c J m k B i V b H A p T Q U x M 5 n 8 n P a 6 Q G T G 2 h D L F b V b C B l R R Z m x D K 1 f 6 i i Y D z k b T V R V l l i t v + / L W 2 9 k k j U r Z c 8 v e U 6 V Y v V 8 2 l 4 N L u I I b 8 O A W q v A I N a g D g y G 8 w h v M n J n z 7 n w 4 n 4 v R L W e 5 c w E r c L 5 + A Y 5 0 o P 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " y M K v c s b Q N k v F r B n V k m 2 K + W 2 6 W + g = " > A A A C G X i c b V C 7 T g J B F L 3 r E / G F W t p M B B M r s k u j J d H G E h N 5 J L A h s 8 N d m D A 7 u 5 m Z N R D C R 9 j K 1 9 g Z W y s / x s R h o R D w J J O c n H N v 7 p k T J I J r 4 7 r f z t b 2 z u 7 e f u 4 g f 3 h 0 f H J a O D t v 6 D h V D O s s F r F q B V S j 4 B L r h h u B r U Q h j Q K B z W D 4 M P e b L 6 g 0 j + W z G S f o R 7 Q v e c g Z N V Z q l j p B S E a l b q H o l t 0 M Z J N 4 S 1 K E J W r d w k + n F 7 M 0 Q m m Y o F q 3 P T c x / o Q q w 5 n A a b 6 T a k w o G 9 I + t i 2 V N E L t T 7 K 4 U 3 J t l R 4 J Y 2 W f N C R T / 2 5 M a K T 1 O A r s Z E T N Q K 9 7 c / E / r 5 2 a 8 M 6 f c J m k B i V b H A p T Q U x M 5 n 8 n P a 6 Q G T G 2 h D L F b V b C B l R R Z m x D K 1 f 6 i i Y D z k b T V R V l l i t v + / L W 2 9 k k j U r Z c 8 v e U 6 V Y v V 8 2 l 4 N L u I I b 8 O A W q v A I N a g D g y G 8 w h v M n J n z 7 n w 4 n 4 v R L W e 5 c w E r c L 5 + A Y 5 0 o P 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " y M K v c s b Q N k v F r B n V k m 2 K + W 2 6 W + g = " > A A A C G X i c b V C 7 T g J B F L 3 r E / G F W t p M B B M r s k u j J d H G E h N 5 J L A h s 8 N d m D A 7 u 5 m Z N R D C R 9 j K 1 9 g Z W y s / x s R h o R D w J J O c n H N v 7 p k T J I J r 4 7 r f z t b 2 z u 7 e f u 4 g f 3 h 0 f H J a O D t v 6 D h V D O s s F r F q B V S j 4 B L r h h u B r U Q h j Q K B z W D 4 M P e b L 6 g 0 j + W z G S f o R 7 Q v e c g Z N V Z q l j p B S E a l b q H o l t 0 M Z J N 4 S 1 K E J W r d w k + n F 7 M 0 Q m m Y o F q 3 P T c x / o Q q w 5 n A a b 6 T a k w o G 9 I + t i 2 V N E L t T 7 K 4 U 3 J t l R 4 J Y 2 W f N C R T / 2 5 M a K T 1 O A r s Z E T N Q K 9 7 c / E / r 5 2 a 8 M 6 f c J m k B i V b H A p T Q U x M 5 n 8 n P a 6 Q G T G 2 h D L F b V b C B l R R Z m x D K 1 f 6 i i Y D z k b T V R V l l i t v + / L W 2 9 k k j U r Z c 8 v e U 6 V Y v V 8 2 l 4 N L u I I b 8 O A W q v A I N a g D g y G 8 w h v M n J n z 7 n w 4 n 4 v R L W e 5 c w E r c L 5 + A Y 5 0 o P 0 = < / l a t e x i t > c < l a t e x i t s h a 1 _ b a s e 6 4 = " L 5 v s F d f C 3 a d a i A y 6 S N / m h T Z l f f 4 = " > A A A C G X i c b V C 7 T g J B F L 2 L L 8 Q X a m k z E U y s y C 6 N l k Q b S 0 z k k c C G z A 5 3 Y c L s 7 G Z m 1 k g I H 2 E r X 2 N n b K 3 8 G B O H h U L A k 0 x y c s 6 9 u W d O k A i u j e t + O 7 m t 7 Z 3 d v f x + 4 e D w 6 P i k e H r W 1 H G q G D Z Y L G L V D q h G w S U 2 D D c C 2 4 l C G g U C W 8 H o f u 6 3 n l F p H s s n M 0 7 Q j + h A 8 p A z a q z U K n e D k L B y r 1 h y K 2 4 G s k m 8 J S n B E v V e 8 a f b j 1 k a o T R M U K 0 7 n p s Y f 0 K V 4 U z g t N B N N S a U j e g A O 5 Z K G q H 2 J 1 n c K b m y S p + E s b J P G p K p f z c m N N J 6 H A V 2 M q J m q N e 9 u f i f 1 0 l N e O t P u E x S g 5 I t D o W p I C Y m 8 7 + T P l f I j B h b Q p n i N i t h Q 6 o o M 7 a h l S s D R Z M h Z y / T V R V l l q t g + / L W 2 9 k k z W r F c y v e Y 7 V U u 1 s 2 l 4 c L u I R r 8 O A G a v A A d W g A g x G 8 w h v M n J n z 7 n w 4 n 4 v R n L P c O Y c V O F + / a w S g 6 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L 5 v s F d f C 3 a d a i A y 6 S N / m h T Z l f f 4 = " > A A A C G X i c b V C 7 T g J B F L 2 L L 8 Q X a m k z E U y s y C 6 N l k Q b S 0 z k k c C G z A 5 3 Y c L s 7 G Z m 1 k g I H 2 E r X 2 N n b K 3 8 G B O H h U L A k 0 x y c s 6 9 u W d O k A i u j e t + O 7 m t 7 Z 3 d v f x + 4 e D w 6 P i k e H r W 1 H G q G D Z Y L G L V D q h G w S U 2 D D c C 2 4 l C G g U C W 8 H o f u 6 3 n l F p H s s n M 0 7 Q j + h A 8 p A z a q z U K n e D k L B y r 1 h y K 2 4 G s k m 8 J S n B E v V e 8 a f b j 1 k a o T R M U K 0 7 n p s Y f 0 K V 4 U z g t N B N N S a U j e g A O 5 Z K G q H 2 J 1 n c K b m y S p + E s b J P G p K p f z c m N N J 6 H A V 2 M q J m q N e 9 u f i f 1 0 l N e O t P u E x S g 5 I t D o W p I C Y m 8 7 + T P l f I j B h b Q p n i N i t h Q 6 o o M 7 a h l S s D R Z M h Z y / T V R V l l q t g + / L W 2 9 k k z W r F c y v e Y 7 V U u 1 s 2 l 4 c L u I R r 8 O A G a v A A d W g A g x G 8 w h v M n J n z 7 n w 4 n 4 v R n L P c O Y c V O F + / a w S g 6 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L 5 v s F d f C 3 a d a i A y 6 S N / m h T Z l f f 4 = " > A A A C G X i c b V C 7 T g J B F L 2 L L 8 Q X a m k z E U y s y C 6 N l k Q b S 0 z k k c C G z A 5 3 Y c L s 7 G Z m 1 k g I H 2 E r X 2 N n b K 3 8 G B O H h U L A k 0 x y c s 6 9 u W d O k A i u j e t + O 7 m t 7 Z 3 d v f x + 4 e D w 6 P i k e H r W 1 H G q G D Z Y L G L V D q h G w S U 2 D D c C 2 4 l C G g U C W 8 H o f u 6 3 n l F p H s s n M 0 7 Q j + h A 8 p A z a q z U K n e D k L B y r 1 h y K 2 4 G s k m 8 J S n B E v V e 8 a f b j 1 k a o T R M U K 0 7 n p s Y f 0 K V 4 U z g t N B N N S a U j e g A O 5 Z K G q H 2 J 1 n c K b m y S p + E s b J P G p K p f z c m N N J 6 H A V 2 M q J m q N e 9 u f i f 1 0 l N e O t P u E x S g 5 I t D o W p I C Y m 8 7 + T P l f I j B h b Q p n i N i t h Q 6 o o M 7 a h l S s D R Z M h Z y / T V R V l l q t g + / L W 2 9 k k z W r F c y v e Y 7 V U u 1 s 2 l 4 c L u I R r 8 O A G a v A A d W g A g x G 8 w h v M n J n z 7 n w 4 n 4 v R n L P c O Y c V O F + / a w S g 6 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L 5 v s F d f C 3 a d a i A y 6 S N / m h T Z l f f 4 = " > A A A C G X i c b V C 7 T g J B F L 2 L L 8 Q X a m k z E U y s y C 6 N l k Q b S 0 z k k c C G z A 5 3 Y c L s 7 G Z m 1 k g I H 2 E r X 2 N n b K 3 8 G B O H h U L A k 0 x y c s 6 9 u W d O k A i u j e t + O 7 m t 7 Z 3 d v f x + 4 e D w 6 P i k e H r W 1 H G q G D Z Y L G L V D q h G w S U 2 D D c C 2 4 l C G g U C W 8 H o f u 6 3 n l F p H s s n M 0 7 Q j + h A 8 p A z a q z U K n e D k L B y r 1 h y K 2 4 G s k m 8 J S n B E v V e 8 a f b j 1 k a o T R M U K 0 7 n p s Y f 0 K V 4 U z g t N B N N S a U j e g A O 5 Z K G q H 2 J 1 n c K b m y S p + E s b J P G p K p f z c m N N J 6 H A V 2 M q J m q N e 9 u f i f 1 0 l N e O t P u E x S g 5 I t D o W p I C Y m 8 7 + T P l f I j B h b Q p n i N i t h Q 6 o o M 7 a h l S s D R Z M h Z y / T V R V l l q t g + / L W 2 9 k k z W r F c y v e Y 7 V U u 1 s 2 l 4 c L u I R r 8 O A G a v A A d W g A g x G 8 w h v M n J n z 7 n w 4 n 4 v R n L P c O Y c V O F + / a w S g 6 A = = < / l a t e x i t > ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " x W m X r a 5 P 9 0 Y Y t 2 h G o 0 Q W 8 X W b 5 i s = " > A A A C G n i c b V D L T s J A F J 3 i C / G F u n T T C C a u S M t G l 0 Q 3 L j G R R w I N m Q 6 3 d M J 0 2 s z c G g n h J 9 z K 1 7 g z b t 3 4 M S Y O p Q s B T z L J y T n 3 5 p 4 5 f i K 4 R s f 5 t g p b 2 z u 7 e 8 X 9 0 s H h 0 f F J + f S s r e N U M W i x W M S q 6 1 M N g k t o I U c B 3 U Q B j X w B H X 9 8 v / A 7 z 6 A 0 j + U T T h L w I j q S P O C M o p G 6 1 T 6 G g L Q 6 K F e c m p P B 3 i R u T i o k R 3 N Q / u k P Y 5 Z G I J E J q n X P d R L 0 p l Q h Z w J m p X 6 q I a F s T E f Q M 1 T S C L Q 3 z f L O 7 C u j D O 0 g V u Z J t D P 1 7 8 a U R l p P I t 9 M R h R D v e 4 t x P + 8 X o r B r T f l M k k R J F s e C l J h Y 2 w v P m 8 P u Q K G Y m I I Z Y q b r D Y L q a I M T U U r V 0 a K J i F n L 7 N V F W S W q 2 T

6 c t f b 2 S T t e s 1 1 a u 5 j v d K 4 y 5 s r k g t y S a 6 J S 2 5 I g z y Q J m k R R g R 5 J W 9 k b s 2 t d + v D + l y O F q x 8 5 5 y s w P r 6 B e l y o b 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " x W m X r a

5 P 9 0 Y Y t 2 h G o 0 Q W 8 X W b 5 i s = " > A A A C G n i c b V D L T s J A F J 3 i C / G F u n T T C C a u S M t G l 0 Q 3 L j G R R w I N m Q 6 3 d M J 0 2 s z c G g n h J 9 z K 1 7 g z b t 3 4 M S Y O p Q s B T z L J y T n 3 5 p 4 5 f i K 4 R s f 5 t g p b 2 z u 7 e 8 X 9 0 s H h 0 f F J + f S s r e N U M W i x W M S q 6 1 M N g k t o I U c B 3 U Q B j X w B H X 9 8 v / A 7 z 6 A 0 j + U T T h L w I j q S P O C M o p G 6 1 T 6 G g L Q 6 K F e c m p P B 3 i R u T i o k R 3 N Q / u k P Y 5 Z G I J E J q n X P d R L 0 p l Q h Z w J m p X 6 q I a F s T E f Q M 1 T S C L Q 3 z f L O 7 C u j D O 0 g V u Z J t D P 1 7 8 a U R l p P I t 9 M R h R D v e 4 t x P + 8 X o r B r T f l M k k R J F s e C l J h Y 2 w v P m 8 P u Q K G Y m I I Z Y q b r D Y L q a I M T U U r V 0 a K J i F n L 7 N V F W S W q 2 T

6 c t f b 2 S T t e s 1 1 a u 5 j v d K 4 y 5 s r k g t y S a 6 J S 2 5 I g z y Q J m k R R g R 5 J W 9 k b s 2 t d + v D + l y O F q x 8 5 5 y s w P r 6 B e l y o b 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " x W m X r a

5 P 9 0 Y Y t 2 h G o 0 Q W 8 X W b 5 i s = " > A A A C G n i c b V D L T s J A F J 3 i C / G F u n T T C C a u S M t G l 0 Q 3 L j G R R w I N m Q 6 3 d M J 0 2 s z c G g n h J 9 z K 1 7 g z b t 3 4 M S Y O p Q s B T z L J y T n 3 5 p 4 5 f i K 4 R s f 5 t g p b 2 z u 7 e 8 X 9 0 s H h 0 f F J + f S s r e N U M W i x W M S q 6 1 M N g k t o I U c B 3 U Q B j X w B H X 9 8 v / A 7 z 6 A 0 j + U T T h L w I j q S P O C M o p G 6 1 T 6 G g L Q 6 K F e c m p P B 3 i R u T i o k R 3 N Q / u k P Y 5 Z G I J E J q n X P d R L 0 p l Q h Z w J m p X 6 q I a F s T E f Q M 1 T S C L Q 3 z f L O 7 C u j D O 0 g V u Z J t D P 1 7 8 a U R l p P I t 9 M R h R D v e 4 t x P + 8 X o r B r T f l M k k R J F s e C l J h Y 2 w v P m 8 P u Q K G Y m I I Z Y q b r D Y L q a I M T U U r V 0 a K J i F n L 7 N V F W S W q 2 T

6 c t f b 2 S T t e s 1 1 a u 5 j v d K 4 y 5 s r k g t y S a 6 J S 2 5 I g z y Q J m k R R g R 5 J W 9 k b s 2 t d + v D + l y O F q x 8 5 5 y s w P r 6 B e l y o b 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " x W m X r a

5 P 9 0 Y Y t 2 h G o 0 Q W 8 X W b 5 i s = " > A A A C G n i c b V D L T s J A F J 3 i C / G F u n T T C C a u S M t G l 0 Q 3 L j G R R w I N m Q 6 3 d M J 0 2 s z c G g n h J 9 z K 1 7 g z b t 3 4 M S Y O p Q s B T z L J y T n 3 5 p 4 5 f i K 4 R s f 5 t g p b 2 z u 7 e 8 X 9 0 s H h 0 f F J + f S s r e N U M W i x W M S q 6 1 M N g k t o I U c B 3 U Q B j X w B H X 9 8 v / A 7 z 6 A 0 j + U T T h L w I j q S P O C M o p G 6 1 T 6 G g L Q 6 K F e c m p P B 3 i R u T i o k R 3 N Q / u k P Y 5 Z G I J E J q n X P d R L 0 p l Q h Z w J m p X 6 q I a F s T E f Q M 1 T S C L Q 3 z f L O 7 C u j D O 0 g V u Z J t D P 1 7 8 a U R l p P I t 9 M R h R D v e 4 t x P + 8 X o r B r T f l M k k R J F s e C l J h Y 2 w v P m 8 P u Q K G Y m I I Z Y q b r D Y L q a I M T U U r V 0 a K J i F n L 7 N V F W S W q 2 T

6 c t f b 2 S T t e s 1 1 a u 5 j v d K 4 y 5 s r k g t y S a 6 J S 2 5 I g z y Q J m k R R g R 5 J W 9 k b s 2 t d + v D + l y O F q x 8 5 5 y s w P r 6 B e l y o b 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " H 9 c 4 l 

v 8 s c a h Y Y u N 6 N z G l + N K d b D c = " > A A A C G H i c b V D L T s J A F L 3 1 i f h C X b p p B B N X p G W j S 6 I b l 5 j I I 4 G G T I d b G J l O m 5 m p k T T 8 g 1 v 5 G n f G r T s / x s S h d C H g S S Y 5 O e f e 3 D P H j z l T 2 n G + r Y 3 N r e 2 d 3 c J e c f / g 8 O i 4 d H L a U l E i K T Z p x C P Z 8 Y l C z g Q 2 N d M c O 7 F E E v o c 2 / 7 4 b u 6 3 n 1 E q F o l H P Y n R C 8 l Q s I B R o o 3 U q v T i E a v 0 S 2 W n 6 m S w 1 4 m b k z L k a P R L P 7 1 B R J M Q h a a c K N V 1 n V h 7 K Z G a U Y 7 T Y i 9 R G B M 6 J k P s G i p I i M p L s 7 R T + 9 I o A z u I p H l C 2 5 n 6 d y M l o V K T 0 D e T I d E j t e r N x f + 8 b q K D G y 9 l I k 4 0 C r o 4 F C T c 1 p E 9 / 7 o 9 Y B K p 5 h N D C J X M Z L X p i E h C t S l o 6 c p Q E t M L f Z k u q y i y X E X T l 7 v a z j p p 1 a q u U 3 U f a u X 6 b d 5 c A c 7 h A q 7 A h W u o w z 0 0 o A k U n u A V 3 m B m z a x 3 6 8 P 6 X I x u W P n O G S z B + v o F M 8 G g 1 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " H 9 c 4 l v 8 s c a h Y Y u N 6 N z G l + N K d b D c = " > A A A C G H i c b V D L T s J A F L 3 1 i f h C X b p p B B N X p G W j S 6 I b l 5 j I I 4 G G T I d b G J l O m 5 m p k T T 8 g 1 v 5 G n f G r T s / x s S h d C H g S S Y 5 O e f e 3 D P H j z l T 2 n G + r Y 3 N r e 2 d 3 c J e c f / g 8 O i 4 d H L a U l E i K T Z p x C P Z 8 Y l C z g Q 2 N d M c O 7 F E E v o c 2 / 7 4 b u 6 3 n 1 E q F o l H P Y n R C 8 l Q s I B R o o 3 U q v T i E a v 0 S 2 W n 6 m S w 1 4 m b k z L k a P R L P 7 1 B R J M Q h a a c K N V 1 n V h 7 K Z G a U Y 7 T Y i 9 R G B M 6 J k P s G i p I i M p L s 7 R T + 9 I o A z u I p H l C 2 5 n 6 d y M l o V K T 0 D e T I d E j t e r N x f + 8 b q K D G y 9 l I k 4 0 C r o 4 F C T c 1 p E 9 / 7 o 9 Y B K p 5 h N D C J X M Z L X p i E h C t S l o 6 c p Q E t M L f Z k u q y i y X E X T l 7 v a z j p p 1 a q u U 3 U f a u X 6 b d 5 c A c 7 h A q 7 A h W u o w z 0 0 o A k U n u A V 3 m B m z a x 3 6 8 P 6 X I x u W P n O G S z B + v o F M 8 G g 1 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " H 9 c 4 l v 8 s c a h Y Y u N 6 N z G l + N K d b D c = " > A A A C G H i c b V D L T s J A F L 3 1 i f h C X b p p B B N X p G W j S 6 I b l 5 j I I 4 G G T I d b G J l O m 5 m p k T T 8 g 1 v 5 G n f G r T s / x s S h d C H g S S Y 5 O e f e 3 D P H j z l T 2 n G + r Y 3 N r e 2 d 3 c J e c f / g 8 O i 4 d H L a U l E i K T Z p x C P Z 8 Y l C z g Q 2 N d M c O 7 F E E v o c 2 / 7 4 b u 6 3 n 1 E q F o l H P Y n R C 8 l Q s I B R o o 3 U q v T i E a v 0 S 2 W n 6 m S w 1 4 m b k z L k a P R L P 7 1 B R J M Q h a a c K N V 1 n V h 7 K Z G a U Y 7 T Y i 9 R G B M 6 J k P s G i p I i M p L s 7 R T + 9 I o A z u I p H l C 2 5 n 6 d y M l o V K T 0 D e T I d E j t e r N x f + 8 b q K D G y 9 l I k 4 0 C r o 4 F C T c 1 p E 9 / 7 o 9 Y B K p 5 h N D C J X M Z L X p i E h C t S l o 6 c p Q E t M L f Z k u q y i y X E X T l 7 v a z j p p 1 a q u U 3 U f a u X 6 b d 5 c A c 7 h A q 7 A h W u o w z 0 0 o A k U n u A V 3 m B m z a x 3 6 8 P 6 X I x u W P n O G S z B + v o F M 8 G g 1 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " H 9 c 4 l v 8 s c a h Y Y u N 6 N z G l + N K d b D c = " > A A A C G H i c b V D L T s J A F L 3 1 i f h C X b p p B B N X p G W j

O i 4 d H L a U l E i K T Z p x C P Z 8 Y l C z g Q 2 N d M c O 7 F E E v o c 2 / 7 4 b u 6 3 n 1 E q F o l H P Y n R C 8 l Q s I B R o o 3 U q v T i E a v 0 S 2 W n 6 m S w 1 4 m b k z L k a P R L P 7 1 B R J M Q h a a c K N V 1 n V h 7 K Z G a U Y 7 T Y i 9 R G B M 6 J k P s G i p I i M p L s 7 R T + 9 I o A z u I p H l C 2 5 n 6 d y M l o V K T 0 D e T I d E j t e r N x f + 8 b q K D G y 9 l I k 4 0 C r o 4 F C T c 1 p E 9 / 7 o 9 Y B K p 5 h N D C J X M Z L X p i E h C t S l o 6 c p Q E t M L f Z k u q y i y X E X T l 7 v a z j p p 1 a q u U 3 U f a u X 6 b d 5 c A c 7 h A q 7 A h W u o w z 0 0 o A k U n u A V 3 m B m z a x 3 6 8 P 6 X I x u W P n O G S z B + v o F M 8 G g 1 A = = < / l a t e x i t >

N:

< l a t e x i t s h a 1 _ b a s e 6 4 = " P 8 o q S / z s c e I C K 4 9 8 5

T k O 8 r f 6 t p Y = " > A A A C F X i c b V C 7 T s M w F L 0 p r 1 J e B U Y W i x a J q U q 6 w F j B w o S K o A + p j S r H d V K r j h P Z D q K K + g m s 9 G v Y E C s z H 4 O E m 2 a g L U e y d H T O v b r H x 4 s 5 U 9 q 2 v 6 3 C x u b W 9 k 5 x t 7 S 3 f 3 B 4 V D 4 + a a s o k Y S 2 S M Q j 2 f W w o p w J 2 t J M c 9 q N J c W h x 2 n H G 9 / O / c 4 z l Y p F 4 k l P Y u q G O B D M Z w R r I z 1 W 7 6 u D c s W u 2 R n Q O n F y U o E c z U H 5 p z + M S B J S o Q n H S v U c O 9 Z u i q V m h N N p q Z 8 o G m M y x g H t G S p w S J W b Z l G n 6 M I o Q + R H 0 j y h U a b + 3 U h x q N Q k 9 M x k i P V I r X p z 8 T + v l 2 j / 2 k 2 Z i B N N B V k c 8 h O O d I T m / 0 Z D J i n R f G I I J p K Z r I i M s M R E m 3 a W r g Q S x y N G X q b L K h V Z r p L p y 1 l t Z 5 2 0 6 z X H r j k P 9 U r j J m + u C G d w D p f g w B U 0 4 A 6 a 0 A I C A b z C G 8 y s m f V u f V i f i 9 G

C l e + c w h K s r 1 + M P Z 9 n < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " P 8 o q S / z s c e I C K 4 9 8 5

T k O 8 r f 6 t p Y = " > A A A C F X i c b V C 7 T s M w F L 0 p r 1 J e B U Y W i x a J q U q 6 w F j B w o S K o A + p j S r H d V K r j h P Z D q K K + g m s 9 G v Y E C s z H 4 O E m 2 a g L U e y d H T O v b r H x 4 s 5 U 9 q 2 v 6 3 C x u b W 9 k 5 x t 7 S 3 f 3 B 4 V D 4 + a a s o k Y S 2 S M Q j 2 f W w o p w J 2 t J M c 9 q N J c W h x 2 n H G 9 / O / c 4 z l Y p F 4 k l P Y u q G O B D M Z w R r I z 1 W 7 6 u D c s W u 2 R n Q O n F y U o E c z U H 5 p z + M S B J S o Q n H S v U c O 9 Z u i q V m h N N p q Z 8 o G m M y x g H t G S p w S J W b Z l G n 6 M I o Q + R H 0 j y h U a b + 3 U h x q N Q k 9 M x k i P V I r X p z 8 T + v l 2 j / 2 k 2 Z i B N N B V k c 8 h O O d I T m / 0 Z D J i n R f G I I J p K Z r I i M s M R E m 3 a W r g Q S x y N G X q b L K h V Z r p L p y 1 l t Z 5 2 0 6 z X H r j k P 9 U r j J m + u C G d w D p f g w B U 0 4 A 6 a 0 A I C A b z C G 8 y s m f V u f V i f i 9 G

C l e + c w h K s r 1 + M P Z 9 n < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " P 8 o q S / z s c e I C K 4 9 8 5

T k O 8 r f 6 t p Y = " > A A A C F X i c b V C 7 T s M w F L 0 p r 1 J e B U Y W i x a J q U q 6 w F j B w o S K o A + p j S r H d V K r j h P Z D q K K + g m s 9 G v Y E C s z H 4 O E m 2 a g L U e y d H T O v b r H x 4 s 5 U 9 q 2 v 6 3 C x u b W 9 k 5 x t 7 S 3 f 3 B 4 V D 4 + a a s o k Y S 2 S M Q j 2 f W w o p w J 2 t J M c 9 q N J c W h x 2 n H G 9 / O / c 4 z l Y p F 4 k l P Y u q G O B D M Z w R r I z 1 W 7 6 u D c s W u 2 R n Q O n F y U o E c z U H 5 p z + M S B J S o Q n H S v U c O 9 Z u i q V m h N N p q Z 8 o G m M y x g H t G S p w S J W b Z l G n 6 M I o Q + R H 0 j y h U a b + 3 U h x q N Q k 9 M x k i P V I r X p z 8 T + v l 2 j / 2 k 2 Z i B N N B V k c 8 h O O d I T m / 0 Z D J i n R f G I I J p K Z r I i M s M R E m 3 a W r g Q S x y N G X q b L K h V Z r p L p y 1 l t Z 5 2 0 6 z X H r j k P 9 U r j J m + u C G d w D p f g w B U 0 4 A 6 a 0 A I C A b z C G 8 y s m f V u f V i f i 9 G

C l e + c w h K s r 1 + M P Z 9 n < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " P 8 o q S / z s c e I C K 4 9 8 5 

T k O 8 r f 6 t p Y = " > A A A C F X i c b V C 7 T s M w F L 0 p r 1 J e B U Y W i x a J q U q 6 w F j B w o S K o A + p j S r H d V K r j h P Z D q K K + g m s 9 G v Y E C s z H 4 O E m 2 a g L U e y d H T O v b r H x 4 s 5 U 9 q 2 v 6 3 C x u b W 9 k 5 x t 7 S 3 f 3 B 4 V D 4 + a a s o k Y S 2 S M Q j 2 f W w o p w J 2 t J M c 9 q N J c W h x 2 n H G 9 / O / c 4 z l Y p F 4 k l P Y u q G O B D M Z w R r I z 1 W 7 6 u D c s W u 2 R n Q O n F y U o E c z U H 5 p z + M S B J S o Q n H S v U c O 9 Z u i q V m h N N p q Z 8 o G m M y x g H t G S p w S J W b Z l G n 6 M I o Q + R H 0 j y h U a b + 3 U h x q N Q k 9 M x k i P V I r X p z 8 T + v l 2 j / 2 k 2 Z i B N N B V k c 8 h O O d I T m / 0 Z D J i n R f G I I J p K Z r I i M s M R E m 3 a W r g Q S x y N G X q b L K h V Z r p L p y 1 l t Z 5 2 0 6 z X H r j k P 9 U r j J m + u C G d w D p f g w B U 0 4 A 6 a 0 A I C A b z C G 8 y s m f V u f V i f i 9 G C l e + c

= " > A A A C F 3 i c b V A 9 T w J B F H z n J + I X a m m z k Z h Y k T s a r Q y J j S U m H p A A I X v L O 9 i w t 3 f Z 3 T P i h d 9 g K 7 / G z t h a + m N M X A 4 K A S f Z Z D L z X t 7 s B I n g 2 r j u t 7 O x u b W 9 s 1 v Y K + 4 f H B 4 d l 0 5 O G z p O F U O f x S J W r Y B q F F y i b 7 g R 2 E o U 0 i g Q 2 A x G d z O / + Y R K 8 1 g + m n G C 3 Y g O J A 8 5 o 8 Z K f i c I y U u v V H Y r b g 6 y T r w F K c M C 9 V 7 p p 9 O P W R q h N E x Q r d u e m 5 h u R p X h T O C k 2 E k 1 J p S N 6 A D b l k o a o e

5 m e d g J u b R K n 4 S x s k 8 a k q t / N z I a a T 2 O A j s Z U T P U q 9 5 M / M 9 r p y a 8 6 W Z c J q l B y e a H w l 

Q Q E 5 P Z z 0 m f K 2 R G j C 2 h T H G b l b A h V Z Q Z 2 8 / S l Y G i y Z C z 5 8 m y i j L P V b R 9 e a v t

= " > A A A C F 3 i c b V A 9 T w J B F H z n J + I X a m m z k Z h Y k T s a r Q y J j S U m H p A A I X v L O 9 i w t 3 f Z 3 T P i h d 9 g K 7 / G z t h a + m N M X A 4 K A S f Z Z D L z X t 7 s B I n g 2 r j u t 7 O x u b W 9 s 1 v Y K + 4 f H B 4 d l 0 5 O G z p O F U O f x S J W r Y B q F F y i b 7 g R 2 E o U 0 i g Q 2 A x G d z O / + Y R K 8 1 g + m n G C 3 Y g O J A 8 5 o 8 Z K f i c I y U u v V H Y r b g 6 y T r w F K c M C 9 V 7 p p 9 O P W R q h N E x Q r d u e m 5 h u R p X h T O C k 2 E k 1 J p S N 6 A D b l k o a o e

5 m e d g J u b R K n 4 S x s k 8 a k q t / N z I a a T 2 O A j s Z U T P U q 9 5 M / M 9 r p y a 8 6 W Z c J q l B y e a H w l 

Q Q E 5 P Z z 0 m f K 2 R G j C 2 h T H G b l b A h V Z Q Z 2 8 / S l Y G i y Z C z 5 8 m y i j L P V b R 9 e a v t

= " > A A A C F 3 i c b V A 9 T w J B F H z n J + I X a m m z k Z h Y k T s a r Q y J j S U m H p A A I X v L O 9 i w t 3 f Z 3 T P i h d 9 g K 7 / G z t h a + m N M X A 4 K A S f Z Z D L z X t 7 s B I n g 2 r j u t 7 O x u b W 9 s 1 v Y K + 4 f H B 4 d l 0 5 O G z p O F U O f x S J W r Y B q F F y i b 7 g R 2 E o U 0 i g Q 2 A x G d z O / + Y R K 8 1 g + m n G C 3 Y g O J A 8 5 o 8 Z K f i c I y U u v V H Y r b g 6 y T r w F K c M C 9 V 7 p p 9 O P W R q h N E x Q r d u e m 5 h u R p X h T O C k 2 E k 1 J p S N 6 A D b l k o a

o e 5 m e d g J u b R K n 4 S x s k 8 a k q t / N z I a a T 2 O A j s Z U T P U q 9 5 M / M 9 r p y a 8 6 W Z c J q l B y e a H w l Q 

Q E 5 P Z z 0 m f K 2 R G j C 2 h T H G b l b A h V Z Q Z 2 8 / S l Y G i

= " > A A A C F 3 i c b V A 9 T w J B F H z n J + I X a m m z k Z h Y k T s a r Q y J j S U m H p A A I X v L O 9 i w t 3 f Z 3 T P i h d 9 g K 7 / G z t h a + m N M X A 4 K A S f Z Z D L z X t 7 s B I n g 2 r j u t 7 O x u b W 9 s 1 v Y K + 4 f H B 4 d l 0 5 O G z p O F U O f x S J W r Y B q F F y i b 7 g R 2 E o U 0 i g Q 2 A x G d z O / + Y R K 8 1 g + m n G C 3 Y g O J A 8 5 o 8 Z K f i c I y U u v V H Y r b g 6 y T r w F K c M C 9 V 7 p p 9 O P W R q h N E x Q r d u e m 5 h u R p X h T O C k 2 E k 1 J p S N 6 A D b l k o a

o e 5 m e d g J u b R K n 4 S x s k 8 a k q t / N z I a a T 2 O A j s Z U T P U q 9 5 M / M 9 r p y a 8 6 W Z c J q l B y e a H w l Q 

Q E 5 P Z z 0 m f K 2 R G j C 2 h T H G b l b A h V Z Q Z 2 8 / S l Y G i

Graph VAE:

< l a t e x i t s h a 1 _ b a s e 6 4 = " R j

/ F A i J J Y Y u o n I d + c d 7 G H S a D n U c = " > A A A C G 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i T d 6 E o q I r q s Y B / S h j K Z T t q h k 0 m Y m Y g l 9 C v c 2 q 9 x J 2 5 d + D G C k z Q L 2 3 p g 4 H D O v d w z x 4 s 4 U 9 q 2 v 6 3 C 2 v r G 5 l Z x u 7 S z u 7 d / U D 4 8 a q k w l o Q 2 S c h D 2 f G w o p w J 2 t R M c 9 q J J M W B x 2 n b G 9 + k f v u Z S s V C 8 a g n E X U D P B T M Z w R r I z 3 d S R y N U O v 6 t l + u 2 F U 7 A 1 o l T k 4 q k K P R L / / 0 B i G J A y o 0 4 V i p r m N H 2 k 2 w 1 I x w O i 3 1 Y k U j T M Z 4 S L u G C h x Q 5 S Z Z 4 C k 6 M 8 o A + a E 0 T 2 i U q X 8 3 E h w o N Q k 8 M x l g P V L L X i r + 5 3 V j 7 V + 6 C R N R r K k g 8 0 N + z J E O U f p 7 N G C S E s 0 n h m A i m c m K y A h L T L T p a O H K M G 2 G k Z f p o k p F l q t k + n K W 2 1 k l r V r V s a v O Q 6 1 S v 8 q b K 8 I J n M I 5 O H A B d b i H B j S B Q A C v 8 A Y z a 2 a 9 W x / W 5 3 y 0 Y O U 7 x 7 A A 6 + s X a C + h 9 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R j / F A i J J Y Y u o n I d + c d 7 G H S a D n U c = " > A A A C G 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i T d 6 E o q I r q s Y B / S h j K Z T t q h k 0 m Y m Y g l 9 C v c 2 q 9 x J 2 5 d + D G C k z Q L 2 3 p g 4 H D O v d w z x 4 s 4 U 9 q 2 v 6 3 C 2 v r G 5 l Z x u 7 S z u 7 d / U D 4 8 a q k w l o Q 2 S c h D 2 f G w o p w J 2 t R M c 9 q J J M W B x 2 n b G 9 + k f v u Z S s V C 8 a g n E X U D P B T M Z w R r I z 3 d S R y N U O v 6 t l + u 2 F U 7 A 1 o l T k 4 q k K P R L / / 0 B i G J A y o 0 4 V i p r m N H 2 k 2 w 1 I x w O i 3 1 Y k U j T M Z 4 S L u G C h x Q 5 S Z Z 4 C k 6 M 8 o A + a E 0 T 2 i U q X 8 3 E h w o N Q k 8 M x l g P V L L X i r + 5 3 V j 7 V + 6 C R N R r K k g 8 0 N + z J E O U f p 7 N G C S E s 0 n h m A i m c m K y A h L T L T p a O H K M G 2 G k Z f p o k p F l q t k + n K W 2 1 k l r V r V s a v O Q 6 1 S v 8 q b K 8 I J n M I 5 O H A B d b i H B j S B Q A C v 8 A Y z a 2 a 9 W x / W 5 3 y 0 Y O U 7 x 7 A A 6 + s X a C + h 9 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R j / F A i J J Y Y u o n I d + c d 7 G H S a D n U c = " > A A A C G 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i T d 6 E o q I r q s Y B / S h j K Z T t q h k 0 m Y m Y g l 9 C v c 2 q 9 x J 2 5 d + D G C k z Q L 2 3 p g 4 H D O v d w z x 4 s 4 U 9 q 2 v 6 3 C 2 v r G 5 l Z x u 7 S z u 7 d / U D 4 8 a q k w l o Q 2 S c h D 2 f G w o p w J 2 t R M c 9 q J J M W B x 2 n b G 9 + k f v u Z S s V C 8 a g n E X U D P B T M Z w R r I z 3 d S R y N U O v 6 t l + u 2 F U 7 A 1 o l T k 4 q k K P R L / / 0 B i G J A y o 0 4 V i p r m N H 2 k 2 w 1 I x w O i 3 1 Y k U j T M Z 4 S L u G C h x Q 5 S Z Z 4 C k 6 M 8 o A + a E 0 T 2 i U q X 8 3 E h w o N Q k 8 M x l g P V L L X i r + 5 3 V j 7 V + 6 C R N R r K k g 8 0 N + z J E O U f p 7 N G C S E s 0 n h m A i m c m K y A h L T L T p a O H K M G 2 G k Z f p o k p F l q t k + n K W 2 1 k l r V r V s a v O Q 6 1 S v 8 q b K 8 I J n M I 5 O H A B d b i H B j S B Q A C v 8 A Y z a 2 a 9 W x / W 5 3 y 0 Y O U 7 x 7 A A 6 + s X a C + h 9 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R j / F A i J J Y Y u o n I d + c d 7 G H S a D n U c = " > A A A C G 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i T d 6 E o q I r q s Y B / S h j K Z T t q h k 0 m Y m Y g l 9 C v c 2 q 9 x J 2 5 d + D G C k z Q L 2 3 p g 4 H D O v d w z x 4 s 4 U 9 q 2 v 6 3 C 2 v r G 5 l Z x u 7 S z u 7 d / U D 4 8 a q k w l o Q 2 S c h D 2 f G w o p w J 2 t R M c 9 q J J M W B x 2 n b G 9 + k f v u Z S s V C 8 a g n E X U D P B T M Z w R r I z 3 d S R y N U O v 6 t l + u 2 F U 7 A 1 o l T k 4 q k K P R L / / 0 B i G J A y o 0 4 V i p r m N H 2 k 2 w 1 I x w O i 3 1 Y k U j T M Z 4 S L u G C h x Q 5 S Z Z 4 C k 6 M 8 o A + a E 0 T 2 i U q X 8 3 E h w o N Q k 8 M x l g P V L L X i r + 5 3 V j 7 V + 6 C R N R r K k g 8 0 N + z J E O U f p 7 N G C S E s 0 n h m A i m c m K y A h L T L T p a O H K M G 2 G k Z f p o k p F l q t k + n K W 2 1 k l r V r V s a v O Q 6 1 S v 8 q b K 8 I J n M I 5 O H A B d b i H B j S B Q A C v 8 A Y z a 2 a 9 W x / W 5 3 y 0 Y O U 7 x 7 A A 6 + s X a C + h 9 w = = < / l a t e x i t >

zn < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < / l a t e x i t > learned | flexible graph < l a t e x i t s h a 1 _ b a s e 6 4 = " O e u r A 8 L Y F 6 x t H Z M o e t h n d P N Q X W s = " > A A A C P 3 i c b V D L T g I x F O 3 4 R B Q F 3 e m m E U x c k R k 2 s i R x 4 x I T e S Q w I Z 1 y B x o 6 n U n b M Z B x E r / G r f 6 C n + E X u D N u 3 V k G F g K e p M n J O f f k 3 h 4 v 4 k x p 2 / 6 w t r Z 3 d v f 2 c w f 5 w 6 P C 8 U m x d N p W Y S w p t G j I Q 9 n 1 i A L O B L Q 0 0 x y 6 k Q Q S e B w 6 3 u R 2 7 n c e Q S o W i g c 9 i 8 A N y E g w n 1 G i j T Q o n n M g U s A Q V 5 4 q 2 O c w Z S a J R 5 J E 4 0 G x b F f t D H i T O E t S R k s 0 B y W r 0 B + G N A 5 A a M q J U j 3 H j r S b E K k Z 5 Z D m + 7 G C i N A J G U H P U E E C U G 6 S f S L F V 0 Y Z Y j + U 5 g m N M / V v I i G B U r P A M 5 M B 0 W O 1 7 s 3 F / 7 x e r P 2 6 m z A R x R o E X S z y Y 4 5 1 i O e N 4 C G T Q D W f G U K o Z O Z W T M d E E q p N b y t b s l o Y n a a r K o j s r h X R C 9 K 8 K d B Z r 2 u T t G t V x 6 4 6 9 7 V y o 7 6 s M o c u 0 C W 6 R g 6 6 Q Q 1 0 h 5 q o h S h 6 R i / o F b 1 Z 7 9 a n 9 W V 9 L 0 a 3 r G X m D K 3 A + v k F G 3 i u 5 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " O e u r A 8 L Y F 6 x t H Z M o e t h n d P N Q X W s = " > A A A C P 3 i c b V D L T g I x F O 3 4 R B Q F 3 e m m E U x c k R k 2 s i R x 4 x I T e S Q w I Z 1 y B x o 6 n U n b M Z B x E r / G r f 6 C n + E X u D N u 3 V k G F g K e p M n J O f f k 3 h 4 v 4 k x p 2 / 6 w t r Z 3 d v f 2 c w f 5 w 6 P C 8 U m x d N p W Y S w p t G j I Q 9 n 1 i A L O B L Q 0 0 x y 6 k Q Q S e B w 6 3 u R 2 7 n c e Q S o W i g c 9 i 8 A N y E g w n 1 G i j T Q o n n M g U s A Q V 5 4 q 2 O c w Z S a J R 5 J E 4 0 G x b F f t D H i T O E t S R k s 0 B y W r 0 B + G N A 5 A a M q J U j 3 H j r S b E K k Z 5 Z D m + 7 G C i N A J G U H P U E E C U G 6 S f S L F V 0 Y Z Y j + U 5 g m N M / V v I i G B U r P A M 5 M B 0 W O 1 7 s 3 F / 7 x e r P 2 6 m z A R x R o E X S z y Y 4 5 1 i O e N 4 C G T Q D W f G U K o Z O Z W T M d E E q p N b y t b s l o Y n a a r K o j s r h X R C 9 K 8 K d B Z r 2 u T t G t V x 6 4 6 9 7 V y o 7 6 s M o c u 0 C W 6 R g 6 6 Q Q 1 0 h 5 q o h S h 6 R i / o F b 1 Z 7 9 a n 9 W V 9 L 0 a 3 r G X m D K 3 A + v k F G 3 i u 5 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " O e u r A 8 L Y F 6 x t H Z M o e t h n d P N Q X W s = " > A A A C P 3 i c b V D L T g I x F O 3 4 R B Q F 3 e m m E U x c k R k 2 s i R x 4 x I T e S Q w I Z 1 y B x o 6 n U n b M Z B x E r / G r f 6 C n + E X u D N u 3 V k G F g K e p M n J O f f k 3 h 4 v 4 k x p 2 / 6 w t r Z 3 d v f 2 c w f 5 w 6 P C 8 U m x d N p W Y S w p t G j I Q 9 n 1 i A L O B L Q 0 0 x y 6 k Q Q S e B w 6 3 u R 2 7 n c e Q S o W i g c 9 i 8 A N y E g w n 1 G i j T Q o n n M g U s A Q V 5 4 q 2 O c w Z S a J R 5 J E 4 0 G x b F f t D H i T O E t S R k s 0 B y W r 0 B + G N A 5 A a M q J U j 3 H j r S b E K k Z 5 Z D m + 7 G C i N A J G U H P U E E C U G 6 S f S L F V 0 Y Z Y j + U 5 g m N M / V v I i G B U r P A M 5 M B 0 W O 1 7 s 3 F / 7 x e r P 2 6 m z A R x R o E X S z y Y 4 5 1 i O e N 4 C G T Q D W f G U K o Z O Z W T M d E E q p N b y t b s l o Y n a a r K o j s r h X R C 9 K 8 K d B Z r 2 u T t G t V x 6 4 6 9 7 V y o 7 6 s M o c u 0 C W 6 R g 6 6 Q Q 1 0 h 5 q o h S h 6 R i / o F b 1 Z 7 9 a n 9 W V 9 L 0 a 3 r G X m D K 3 A + v k F G 3 i u 5 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " O e u r A 8 L Y F 6 x t H Z M o e t h n d P N Q X W s = " > A A A C P 3 i c b V D L T g I x F O 3 4 R B Q F 3 e m m E U x c k R k 2 s i R x 4 x I T e S Q w I Z 1 y B x o 6 n U n b M Z B x E r / G r f 6 C n + E X u D N u 3 V k G F g K e p M n J O f f k 3 h 4 v 4 k x p 2 / 6 w t r Z 3 d v f 2 c w f 5 w 6 P C 8 U m x d N p W Y S w p t G j I Q 9 n 1 i A L O B L Q 0 0 x y 6 k Q Q S e B w 6 3 u R 2 7 n c e Q S o W i g c 9 i 8 A N y E g w n 1 G i j T Q o n n M g U s A Q V 5 4 q 2 O c w Z S a J R 5 J E 4 0 G x b F f t D H i T O E t S R k s 0 B y W r 0 B + G N A 5 A a M q J U j 3 H j r S b E K k Z 5 Z D m + 7 G C i N A J G U H P U E E C U G 6 S f S L F V 0 Y Z Y j + U 5 g m N M / V v I i G B U r P A M 5 M B 0 W O 1 7 s 3 F / 7 x e r P 2 6 m z A R x R o E X S z y Y 4 5 1 i O e N 4 C G T Q D W f G U K o Z O Z W T M d E E q p N b y t b s l o Y n a a r K o j s r h X R C 9 K 8 K d B Z r 2 u T t G t V x 6 4 6 9 7 V y o 7 6 s M o c u 0 C W 6 R g 6 6 Q Q 1 0 h 5 q o h S h 6 R i / o F b 1 Z 7 9 a n 9 W V 9 L 0 a 3 r G X m D K 3 A + v k F G 3 i u 5 w = = < / l a t e x i t > p✓(z|c) = Q N n=1 p✓(zn|zpa(n), cpa(n),n) < l a t e x i t s h a 1 _ b a s e 6 4 = " L 1 G q 4 r F G V X / t I 7 6 o D 2 w H M o m V 1 h 8 = " > A A A C s X i c b V H N b h M x E P Y u f y V Q S O G C x M U k Q k q k K t q F Q 3 u J F I k L J 1 Q k 0 h Z l w + J 1 Z h M r X q 9 l z 1 Y N i 9 + A h + C 1 e A 2 e A G 9 S p C T t S J Y + f 9 9 8 M 6 O Z T E t h M Y r + B O G 9 + w 8 e P j p 4 3 H r y 9 P D Z 8 / b R i 3 N b V o b D m J e y N J c Z s y C F g j E K l H C p D b A i k 3 C R L T 8 0 + s U V G C t K 9 Q V X G q Y F m y u R C 8 7 Q U 2 n 7 l 0 4 T X A A y 2 k s K h o s s r 3 8 4 + p P + / 3 D X p 0 O a a F P O 0 l o N Y / e t / u T o X S Y v b x s b I k G 4 R l P U m r m e 6 r v j r a r 7 4 j F V r p + 2 u 9 E g W g e 9 D e I b 0 B 1 1 h v o 3 I e Q s P Q o O k 1 n J q w I U c s m s n c S R x m n N D A o u w b W S y o J m f M n m M P F Q s Q L s t F 6 v z d G 3 n p n R v D T + K a R r d t t R s 8 L a V Z H 5 z G Z u u 6 8 1 5 F 3 a p M L 8 d F o L p S s E x T e N 8 k p S L G l z A z o T B j j K l Q e M G + F n p X z B D O P o L 7 X T Z W 6 Y X g h + 7 X Z Z U O u 5 d s i s c C 2 / w H h / X b f B + b t B H A 3 i z 3 F 3 d E o 2 c U B e k w 7 p k Z i c k B H 5 S M 7 I m H D y N 3 g V v A k 6 4 f v w a / g 9 z D a p Y X D j e U l 2 I l z + A 8 z U 2 L E = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f Q p v 7 J Z t p f P 0 X L b E F H y r b C r R / b 0 = " > A A A C s X i c b V H N b h M x E P Y u f y V Q m s I F i Y t J h J R I V b Q L B 3 q J F I k L J 1 Q k 0 h Z l w + J 1 Z h M r X q 9 l z 6 K G x W / A W / B Q i N f g C f A m R U r S j m T p 8 / f N N z O a y b Q U F q P o T x D e u X v v / o O D h 6 1 H j w + f H L W P n 5 7 b s j I c x r y U p b n M m A U p F I x R o I R L b Y A V m Y S L b P m u 0 S + + g b G i V J 9 w p W F a s L k S u e A M P Z W 2 f + o 0 w Q U g o 7 2 k Y L j I 8 v q 7 o z / o / w 9 3 f T q k i T b l L K 3 V M H Z f 6 g + O 3 m b y 8 r a x I R K E K z R F r Z n r q b 4 7 2 a q 6 L 5 5 Q 5 f p p u x s N o n X Q m y C + B t 1 R Z 6 h / / W 4 N z 9 L j 4 D C Z l b w q Q C G X z N p J H G m c 1 s y g 4 B J c K 6 k s a M a X b A 4 T D x U r w E 7 r 9 d o c f e W Z G c 1 L 4 5 9 C u m a 3 H T U r r F 0 V m c 9 s 5 r b 7 W k P e p k 0 q z E + n t V C 6 Q l B 8 0 y i v J M W S N j e g M 2 G A o 1 x 5 w L g R f l b K F 8 w w j v 5 S O 1 3 m h u m F 4 F d u l w W 1 n m u H z A r X 8 g u M 9 9 d 1 E 5 y / H s T R I P 4 Y d 0 e n Z B M H 5 A X p k B 6 J y V s y I u / J G R k T T v 4 G z 4 O X Q S d 8 E 3 4 O v 4 b Z J j U M r j 3 P y E 6 E y 3 + T Z N n S < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f Q p v 7 J Z t p f P 0 X L b E F H y r b C r R / b 0 = " > A A A C s X i c b V H N b h M x E P Y u f y V Q m s I F i Y t J h J R I V b Q L B 3 q J F I k L J 1 Q k 0 h Z l w + J 1 Z h M r X q 9 l z 6 K G x W / A W / B Q i N f g C f A m R U r S j m T p 8 / f N N z O a y b Q U F q P o T x D e u X v v / o O D h 6 1 H j w + f H L W P n 5 7 b s j I c x r y U p b n M m A U p F I x R o I R L b Y A V m Y S L b P m u 0 S + + g b G i V J 9 w p W F a s L k S u e A M P Z W 2 f + o 0 w Q U g o 7 2 k Y L j I 8 v q 7 o z / o / w 9 3 f T q k i T b l L K 3 V M H Z f 6 g + O 3 m b y 8 r a x I R K E K z R F r Z n r q b 4 7 2 a q 6 L 5 5 Q 5 f p p u x s N o n X Q m y C + B t 1 R Z 6 h / / W 4 N z 9 L j 4 D C Z l b w q Q C G X z N p J H G m c 1 s y g 4 B J c K 6 k s a M a X b A 4 T D x U r w E 7 r 9 d o c f e W Z G c 1 L 4 5 9 C u m a 3 H T U r r F 0 V m c 9 s 5 r b 7 W k P e p k 0 q z E + n t V C 6 Q l B 8 0 y i v J M W S N j e g M 2 G A o 1 x 5 w L g R f l b K F 8 w w j v 5 S O 1 3 m h u m F 4 F d u l w W 1 n m u H z A r X 8 g u M 9 9 d 1 E 5 y / H s T R I P 4 Y d 0 e n Z B M H 5 A X p k B 6 J y V s y I u / J G R k T T v 4 G z 4 O X Q S d 8 E 3 4 O v 4 b Z J j U M r j 3 P y E 6 E y 3 + T Z N n S < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 7 b d o C r n T b 7 y Q X t + 8 f 8 4 1 4 j f h 5 7 0 = " > A A A C s X i c b V F N b x M x E P U u X y V Q m s I F i Y t p h J R I V b Q L B 3 q p V I k L J 1 Q k 0 h Z l w + J 1 Z h M r t t e y Z 1 H D 4 n / A H + R v 8 A v w p k F K 0 o 5 k 6 f k 9 P 8 / o T W G k c J g k f 6 L 4 3 v 0 H D x / t P e 4 8 e b r / 7 K B 7 + P z C V b X l M O K V r O x V w R x I o W G E A i V c G Q t M F R I u i 8 W H V r / 8 A d a J S n / B p Y G J Y j M t S s E Z B i r v / j Z 5 h n N A R v u Z Y j g v y u a n p 7 / o / w v 3 A 3 p K M 2 O r a d 7 o 0 9 R / a z 5 5 e p c p y J v G l s g Q r t G q x j D f 1 w N / v P H r r n h M t R / k 3 V 4 y T F Z F b 4 N 0 D X p k X e f 5 Y b S f T S t e K 9 D I J X N u n C Y G J w 2 z K L g E 3 8 l q B 4 b x B Z v B O E D N F L h J s 4 r N 0 z e B m d K y s u F o p C t 2 0 9 E w 5 d x S F e F l O 7 f b 1 V r y L m 1 c Y 3 k y a Y Q 2 N Y L m N 4 3 K W l K s a L s D O h U W O M p l A I x b E W a l f M 4 s 4 x g 2 t d V l Z p m Z C 3 7 t t 1 n Q q 7 m 2 y E L 5 T g g w 3 Y 3 r N r h 4 O 0 y T Y f o 5 7 Z 2 d r K P c I 6 / I E e m T l L w n Z + Q j O S c j w s n f 6 G X 0 O j q K 3 8 V f 4 + 9

x c f M 0 j t a e F 2 S r 4 s U / I j r X A g = = < / l a t e x i t > zpa(n)

< l a t e x i t s h a 1 _ b a s e 6 4 = " B x E D P I 6 F 5 q 5 N y M l o 1

3 J x E G j a F 0 s = " > A A A C P 3 i c b Z D L S g M x F I Y z X m u 9 t b r T T b A K u i k z 3 e h S c O O y g m 2 F t p R M e q Y N J p k h O S P W Y c C n c a u v 4 G P 4 B O 7 E r T v T y 8 K q P w R + v n M O 5 + Q P E y k s + v 6 b t 7 C 4 t L y y W l g r r m 9 s b m 2 X y j t N G 6 e G Q 4 P H M j Y 3 I b M g h Y Y G C p R w k x h g K p T Q C m 8 v x v X W H R g r Y n 2 N o w S 6 i g 2 0 i A R n 6 F C v t H e Y d c K I P u S 9 r I N w j 0 Z l C c u P 9 U l + 2 C t V / K o / E f 1 r g p m p k J n q v b K 3 2 e n H P F W g k U t m b T v w E + x m z K D g E v J i J 7 W Q M H 7 L B t B 2 V j M F t p t N P p H T I 0 f 6 N I q N e x r p h P 6 c y J i y d q R C 1 6 k Y D u 3 v 2 h j + V 2 u n G J 1 1 M 6 G T F E H z 6 a I o l R R j O k 6 E 9 o U B j n L k D O N G u F s p H z L D O L r c 5 r Y M D E u G g t / n 8 x T 0 5 K 4 5 G K q 8 6 A I M f s f 1 1 z R r 1 c C v B l e 1 y n l t F m W B 7 J M D c k w C c k r O y S W p k w b h 5 J E 8 k W f

y 4 r 1 6 7 9 6 H 9 z l t X f B m M 7 t k T t 7 X N w T L r 2 I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B x E D P I 6 F 5 q 5 N y M l o 1

3 J x E G j a F 0 s = " > A A A C P 3 i c b Z D L S g M x F I Y z X m u 9 t b r T T b A K u i k z 3 e h S c O O y g m 2 F t p R M e q Y N J p k h O S P W Y c C n c a u v 4 G P 4 B O 7 E r T v T y 8 K q P w R + v n M O 5 + Q P E y k s + v 6 b t 7 C 4 t L y y W l g r r m 9 s b m 2 X y j t N G 6 e G Q 4 P H M j Y 3 I b M g h Y Y G C p R w k x h g K p T Q C m 8 v x v X W H R g r Y n 2 N o w S 6 i g 2 0 i A R n 6 F C v t H e Y d c K I P u S 9 r I N w j 0 Z l C c u P 9 U l + 2 C t V / K o / E f 1 r g p m p k J n q v b K 3 2 e n H P F W g k U t m b T v w E + x m z K D g E v J i J 7 W Q M H 7 L B t B 2 V j M F t p t N P p H T I 0 f 6 N I q N e x r p h P 6 c y J i y d q R C 1 6 k Y D u 3 v 2 h j + V 2 u n G J 1 1 M 6 G T F E H z 6 a I o l R R j O k 6 E 9 o U B j n L k D O N G u F s p H z L D O L r c 5 r Y M D E u G g t / n 8 x T 0 5 K 4 5 G K q 8 6 A I M f s f 1 1 z R r 1 c C v B l e 1 y n l t F m W B 7 J M D c k w C c k r O y S W p k w b h 5 J E 8 k W f

y 4 r 1 6 7 9 6 H 9 z l t X f B m M 7 t k T t 7 X N w T L r 2 I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B x E D P I 6 F 5 q 5 N y M l o 1

3 J x E G j a F 0 s = " > A A A C P 3 i c b Z D L S g M x F I Y z X m u 9 t b r T T b A K u i k z 3 e h S c O O y g m 2 F t p R M e q Y N J p k h O S P W Y c C n c a u v 4 G P 4 B O 7 E r T v T y 8 K q P w R + v n M O 5 + Q P E y k s + v 6 b t 7 C 4 t L y y W l g r r m 9 s b m 2 X y j t N G 6 e G Q 4 P H M j Y 3 I b M g h Y Y G C p R w k x h g K p T Q C m 8 v x v X W H R g r Y n 2 N o w S 6 i g 2 0 i A R n 6 F C v t H e Y d c K I P u S 9 r I N w j 0 Z l C c u P 9 U l + 2 C t V / K o / E f 1 r g p m p k J n q v b K 3 2 e n H P F W g k U t m b T v w E + x m z K D g E v J i J 7 W Q M H 7 L B t B 2 V j M F t p t N P p H T I 0 f 6 N I q N e x r p h P 6 c y J i y d q R C 1 6 k Y D u 3 v 2 h j + V 2 u n G J 1 1 M 6 G T F E H z 6 a I o l R R j O k 6 E 9 o U B j n L k D O N G u F s p H z L D O L r c 5 r Y M D E u G g t / n 8 x T 0 5 K 4 5 G K q 8 6 A I M f s f 1 1 z R r 1 c C v B l e 1 y n l t F m W B 7 J M D c k w C c k r O y S W p k w b h 5 J E 8 k W f

y 4 r 1 6 7 9 6 H 9 z l t X f B m M 7 t k T t 7 X N w T L r 2 I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B x E D P I 6 F 5 q 5 N y M l o 1 3 J x E G j a F 0 s = " > Rezende et al., 2014; Goyal et al., 2017; Vikram et al., 2018; Webb et al., 2018) . These dependencies provide empirical priors, learned priors that are conditioned on other latent variables. With M latent dimensions, the full prior takes the following auto-regressive form:

M g h Y Y G C p R w k x h g K p T Q C m 8 v x v X W H R g r Y n 2 N o w S 6 i g 2 0 i A R n 6 F C v t H e Y d c K I P u S 9 r I N w j 0 Z l C c u P 9 U l + 2 C t V / K o / E f 1 r g p m p k J n q v b K 3 2 e n H P F W g k U t m b T v w E + x m z K D g E v J i J 7 W Q M H 7 L B t B 2 V j M F t p t N P p H T I 0 f 6 N I q N e x r p h P 6 c y J i y d q R C 1 6 k Y D u 3 v 2 h j + V 2 u n G J 1 1 M 6 G T F E H z 6 a I o l R R j O k 6 E 9 o U B j n L k D O N G u F s p H z L D O L r c 5 r Y M D E u G g t / n 8 x T 0 5 K 4 5 G K q 8 6 A I M f s f 1 1 z R r 1 c C v B l

EQUATION

where z pa(m) denotes the vector of latent variables constituting the parents of z m . Each conditional distribution can be parameterized by deep networks that output the parameters of distributions, e.g. mean and variance of a Gaussian. While these conditional distributions may be relatively simple, the marginal empirical prior, p θ (z m ) = ∫ p θ (z m z pa(m) )p θ (z pa(m) )dz pa(m) , can be arbitrarily complex. By using more flexible priors, models with latent dependency structure are less restrictive in their latent representations, hopefully capturing more information and enabling a better fit to the data.

With the added latent dependencies in the model, the independence assumption in the approximate posterior is even less valid in this setting. While normalizing flows (Rezende & Mohamed, 2015) offers one technique for overcoming the mean field assumption, a separate line of work has investigated the use of structured approximate posteriors, particularly in the context of models with empirical priors (Johnson et al., 2016) . This technique introduces dependencies between the dimensions of the approximate posterior, often mirroring the dependency structure of the latent variable model. An explanation for this was provided by Marino et al. (2018) : optimizing the approximate posterior requires knowledge of the prior, which is especially relevant in models with empirical priors where the prior can vary with the data. Ladder VAE (Sønderby et al., 2016) incorporates these prior dependencies by using a structured approximate posterior of the form

EQUATION

Unlike the mean field approximate posterior, which conditions each dimension only on the data example, x, the distributions in Eq. ( 4) account for latent dependencies by conditioning on samples from the parent variables. Ladder VAE performs this conditioning by reusing the empirical prior during inference, forming the approximate posterior by combining a "bottom-up" recognition distribution and the "top-down" prior.

While Eqs. ( 3) and (4) permit separate latent dependencies for each individual latent dimension, the dimensions are typically partitioned into a set of nodes, with dimensions within each node sharing the same parents. This improves computational efficiency by allowing priors and approximate posteriors within each node to be calculated in parallel. Using z n to denote latent node n of N and z pa(n) to denote the concatenation of its parent nodes, we can write the ELBO (Eq. ( 2)) as

EQUATION

Note that the KL divergence term in the ELBO can no longer be evaluated analytically, now requiring a sampling-based estimate of the expectation (Kingma & Welling, 2014) . While this can lead to higher variance in ELBO estimates and the resulting gradients, models with latent dependencies still tend to empirically outperform models with independence assumptions (Burda et al., 2016; Sønderby et al., 2016) . However, by increasing the number of nodes, the burden of devising a suitable dependency structure falls upon the experimental practitioner. This is non-trivial, as the structure may depend on the data and other model hyperparameters, such as the number of layers in the deep networks, non-linearities, latent distributions, etc. Rather than relying on pre-defined fully-connected structures (Kingma et al., 2016) or chain structures (Sønderby et al., 2016) , we seek to automatically learn the latent dependency structure as part of the variational optimization process. A comparison of these approaches is visualized in Fig. 1 .

3. VARIATIONAL OPTIMIZATION OF LATENT STRUCTURES:

Unlike the model parameters (φ, θ), which are optimized over a continuous domain, the latent dependency structure is discrete, without a clear ordering. The discrete nature of the latent space's topological structure introduces discontinuities in the optimization landscape, complicating the learning process. Fortunately, unlike the related setting of neural architecture search (Zoph & Le, 2016) , there is only a finite number of possible dependency structures over a fixed number of latent dimensions: In a directed graphical model, a fully-connected directed acyclic graph (DAG) models all possible dependencies. In this model, an ordering is induced over the latent nodes, and the parents of node n (of N ) are given as z pa(n) = {z n+1 , . . . , z N }.1 Thus, to learn an appropriate latent dependency structure, we can maintain all dependencies in a fully-connected DAG, modifying their presence or absence during training. This is accomplished by introducing a set of binary dependency gates (Section 3.1). We convert discrete optimization over dependency structures into a continuous optimization problem by parameterizing these gates as samples from Bernoulli distributions, then learning the distribution parameters (Section 3.3). These gating distributions induce an additional lower bound on L, which becomes tight when the distribution converges to a delta function, yielding a single, optimized dependency structure (Section 3.2). Indeed, we observe this process empirically, with the learned dependency structures outperforming their predefined counterparts (Section 4).

3.1. GATED DEPENDENCIES:

To control the dependency structure of the model, we introduce a set of binary global variables, c = {c i,j } i,j , which gate the latent dependencies. The element c i,j denotes the gate variable from z i to z j (i > j), specifying the presence or absence of this latent dependency. Because each element of c takes values in {0, 1}, dependencies can be preserved or removed simply through (broadcasted) element-wise multiplication with the corresponding parent nodes. Removing a dependency entails multiplying the corresponding input parent node by 0. Each possible latent dependency structure can now be expressed through its corresponding value of c.

Each fixed latent dependency structure, c ′ , induces a separate latent variable model p θ (x, z, c) = p θ (x z, c)p θ (z c)δ c,c ′ , where δ ⋅,⋅ is the Kronecker delta, which effectively selects a single structure. Similar to Eq. ( 3), the prior on the latent variables can now be expressed as

EQUATION

where c pa(n),n denotes the gate variables associated with the dependencies between node z n and its parents, z pa(n) . Note that z pa(n) denotes the set of all possible parents of node z n in the fullyconnected DAG, i.e. z pa(n) = {z n+1 , . . . , z N }. To give a concrete example of the gating procedure, Sample c using Eq. ( 9) and determine z pa(n) for each z n based on the sampled structure.

4::

For each node, compute q φ (z n x, z pa(n) ) using Eq. ( 7).

5::

Sample z from q φ (z x) using Eq. ( 6) and compute p θ (x z).

6::

Update θ, φ, µ based on the gradients derived from Eq. ( 8). 7: until Convergence.

3.2. LEARNING STRUCTURE BY INDUCING AN ADDITIONAL LOWER BOUND:

The formulation in Section 3.1 provides the form of the model for a particular configuration of the latent dependency structure. Finding the optimal structure corresponds to a discrete optimization over all values of c, potentially optimizing the model parameters of each possible configuration. To avoid this intractable procedure, we place a distribution over c, then directly optimize the parameters of this distribution to arrive at a single, learned latent dependency structure. Specifically, we treat each c i,j as an independent random variable, sampled from a Bernoulli distribution with mean µ i,j , i.e. c i,j ∼ p(c i,j ) = B(µ i,j ). We denote the set of these Bernoulli means as µ. Introducing this distribution allows us to express the following additional lower bound on L, derived in Appendix A:

EQUATION

where L c is the ELBO for a particular value of dependency gating variables. Thus, L can be interpreted as the expected ELBO under the distribution of dependency structures induced by p(c), which we estimate by sampling c ∼ p(c) and evaluating L c . We note that L is not a proper variational bound, as it is not guaranteed to recover the marginal likelihood if the approximate posterior matches the true posterior. Rather, optimizing L provides a method for learning the latent structure.

For fixed parameters (φ, θ), the optimal L w.r.t. µ is a δ-distribution at the MAP configuration, c * , yielding L = L = L c * . This is because L c * is always greater than or equal to the expected ELBO over all dependency gates, L. In practice, we jointly optimize φ, θ, and µ. While this non-convex optimization procedure may result in local optima, we find this empirically works well, with p(c) converging to a fixed distribution (Fig. 3 ). Thus, by the end of training, we are effectively optimizing the ELBO for a single dependency structure. The training procedure is outlined in Algorithm 1.

3.3. LEARNING THE DISCRETE GATING VARIABLE DISTRIBUTIONS:

For a given latent dependency structure, gradients for the parameters θ and φ can be estimated using Monte Carlo samples and the reparameterization trick (Kingma & Welling, 2014; Rezende et al., 2014) . To obtain gradients for the gate means, µ, we make use of recent advances in differentiating through discrete operations (Maddison et al., 2017; Jang et al., 2017) , allowing us to differentiate through the sampling of the dependency gating variables, c. Specifically, we recast the gating variables using the Gumbel-Softmax estimator from Jang et al. ( 2017), re-expressing c i,j as:

EQUATION

where 1 and 2 are i.i.d samples drawn from a Gumbel(0, 1) distribution and τ is a temperature parameter. The Gumbel-Softmax distribution is differentiable for τ > 0, allowing us to estimate the derivative ∂ci,j ∂µi,j . For large values of τ , we obtain smoothed versions of c, essentially interpolating between different dependency structures. As τ → 0, we recover binary values for c, yielding the desired discrete sampling of dependency structures at the cost of high-variance gradient estimates. Thus, we anneal τ during training to learn the dependency gate means, µ, eventually arriving at the discrete setting. 

4. EVALUATION:

We evaluate the proposed latent dependency learning approach on three benchmark datasets: MNIST (Lecun et al., 1998; Larochelle & Murray, 2011 ), Omniglot (Lake et al., 2013 ), and CIFAR-10 (Krizhevsky, 2009) . After discussing the experimental setup in Section 4.1, we provide a set of qualitative experiments in Section 4.2 to gain insight into the learning process, hyper-parameter selection, and the nature of the inferred structures. In Section 4.3, we provide quantitative comparisons with common predefined latent dependency structures on benchmark datasets. Additional results on training robustness and latent space embeddings can be found in Appendix C.

4.1. EXPERIMENTAL SETUP:

To provide a fair comparison, the encoders of all structured methods use the same MLP architecture with batch normalization (Ioffe & Szegedy, 2015) and ReLU non-linearities (Nair & Hinton, 2010) in all experiments.2 Decoder structures are the reverse of the encoders. Likewise, the number of latent dimensions is the same in all models and experiments (M = 80). As discussed in Section 3.1, all latent dependencies are modeled by non-linear MLPs as well.

For MNIST and Omniglot, we binarize the data and model p θ (x z) as a Bernoulli distribution, using a sigmoid non-linearity on the output layer to produce the mean of this distribution. For CIFAR-10, we model p θ (x z) with a Gaussian density, with mean and log-variance predicted by sigmoid and linear functions, respectively. Further implementation details, including the model architectures and training criteria, can be found in Appendix B.

4.2. QUALITATIVE ANALYSIS:

We first explore the structure learning process. As described in Section 3.2 and Appendix A, optimizing L w.r.t. the dependency gating means, µ, should push this lower bound toward L. Thus, µ should converge to either 0 or 1, yielding a fixed, final latent dependency structure. In Fig. 3 , we visualize this process during training on MNIST. The model has N = 5 nodes and a total latent dimension of M = 80 (i.e., N ′ = M N = 16 dimensions per node). As shown in Fig. 3a , the gating means converge in practice, with 3 out of 10 edges removed and the rest retained. The resulting static dependency structure is visualized in Fig. 3b . We observed that the learned structure is stable across training runs with different seeds for parameter initialization and mini-batch sampling, supporting the hypothesis that the inferred structure indeed depends on the model parameterization and the dataset.

required to understand this behavior at a theoretical level. A full visualization of the training process is provided in Appendix C.1.

5. DISCUSSION:

Performance vs. Speed/Memory. As shown in Fig. 4 , the number of latent nodes can significantly impact the performance of our model. While allowing more complex dependency structures through a low M N -ratio is typically beneficial, it also has an adverse effect on the training time and memory consumption. Fortunately, the ability to freely select this ratio allows a simple adaption to the available processing power, hardware constraints, and application scenarios.

Optimization Order. It is worth noting that the learning process optimizes the model parameters (c, φ, θ) in a clear temporal order. While the latent structure, governed by c, converges during the first ≈ 200 epochs (Fig. 3a ), it takes over 10× as long until the variational and generative parameters (φ, θ) converge. There is no external force enforcing this behaviour, indicating that the loss can initially most easily be decreased by limiting the latent structure to the complexity prescribed by the observed training data.

Performance Improvement over Fully-Connected VAEs. There is an intricate relationship between fully-connected graphs vs. learned graphs along one axis and between prior structures vs. posterior structures along a separate axis: FC-VAEs model all conditional latent dependencies and are thus potentially more expressive and flexible than other latent dependency structures. It is therefore somewhat surprising that the learned latent structures in Graph VAE consistently outperform the FC-VAE baseline. We speculate that this may be due to difficulties in optimization, which is a known problem in hierarchical latent variable models (Bowman et al., 2016; Burda et al., 2016) .

Graph VAE and FC-VAE both contain the same hypothesis space of possible models that can be learned. If all dependencies are needed, Graph VAE could set all dependency gate parameters to 1.

Likewise, if a latent dependency was unnecessary, FC-VAE could set all of the model parameters in that dependency to 0. However, this would require many coordinated steps along multiple parameter dimensions. It is plausible that the benefit of learning the dependency structure may stem from the ability to alter the optimization landscape, using the dependency gates to move through the model parameter space more coarsely and rapidly. The resulting latent dependency structures may thus be less expressive, but easier to optimize. While only an intuition, this hypothesis is also in line with the observations and results in our experiments with fully-connected encoder graphs and learned decoder graphs, where the theoretically more flexible FC-encoder is outperformed by our parameter sharing approach. Follow-up work will be required to test this intuition.

6. CONCLUSION:

We presented a novel method for structure learning in latent variable models, which uses dependency gating variables, together with a modified objective and discrete differentiation techniques, to effectively transform discrete structure learning into a continuous optimization problem. In our experiments, the learned latent dependency structures improve the performance of latent variable models over comparable baselines with predefined dependency structures. The approach presented here provides directions for further research in structure learning for other tasks, including undirected graphical models, time-series models, and discriminative models.

A LOWER BOUND DERIVATION:

Introducing the distribution over dependency gating variables, p(c) = B(µ), modifies the evidence lower bound (ELBO) from Eq. ( 2), as we now have an additional set of random variables over which to marginalize. To see this, we can start by re-expressing Eq. ( 2) as L = E q φ (z x) [log p θ (x, z)] -E q φ (z x) [log q φ (z x)] .

(10) p θ (x, z) can be expressed as a marginalization over the gating variables, effectively averaging over an ensemble of models with a distribution of dependency structures: 

Plugging this into Eq. ( 10):

EQUATION

Using Jensen's inequality, we bring the log inside of the expectation, E p(c) [⋅], yielding L ≥ L = E q φ (z x) E p(c) [log p θ (x, z c)] -E q φ (z x) [log q φ (z x)] ,

where L is a lower bound on L. Swapping the order of expectation, we rewrite L as L = E p(c) E q φ (z x) [log p θ (x, z c)] -E q φ (z x) [log q φ (z x)] ,

and because the second term is independent of p(c), we can include both terms inside of a single outer expectation:

L = E p(c) E q φ (z x) [log p θ (x, z c) -log q φ (z x)]

= E p(c) E q φ (z x) [log p θ (x z, c)] -KL(q φ (z x) p θ (z c))

EQUATION

where we have defined L c as the ELBO for a given dependency structure. Note that L is not a proper variational bound, as it cannot recover the marginal log likelihood. Rather, L allows us to optimize the distribution over gating variables and, thus, the model structure. When we arrive at a fixed structure, we will be optimizing the variational bound for that particular dependency structure.

To see this, note that the bound in Eq. 13 becomes tight when p(c) is any fixed distribution, in which the dependency gating means, µ, are all either 1 or 0. Plugging in a delta distribution for p(c) at a particular configuration, c ′ , i.e. p(c) = δ c,c ′ , we have L = E q φ (z x) log E δ c,c ′ [p θ (x, z c)] -E q φ (z x) [log q φ (z x)]

= E q φ (z x) E δ c,c ′ [log p θ (x, z c)] -E q φ (z x) [log q φ (z x)] = L.

(16)

Intuitively, this is simply the case of a latent variable model with predefined, fixed dependencies. By optimizing L w.r.t. µ, we hope to collapse p(c) to a delta distribution at the optimal (MAP) configuration, c * , because

EQUATION

Effectively, we can search for the dependency structure with the highest ELBO by optimizing the distribution parameters of p(c). Although this optimization procedure may arrive at locally optimal dependency structures, our hope is that these learned structures will still perform better than an arbitrary, predefined dependency structure.

C ADDITIONAL RESULTS:

C.1 ROBUSTNESS OF LOG-LIKELIHOODS In Fig. 5 , we report the averaged test log-likelihoods and associated standard deviations of Graph VAE and our baselines at different epochs. All calculations are based on 5 independent runs. 

:

We follow convention(Dayan et al., 1995;Rezende et al.,

2014; Sønderby et al., 2016), with parent nodes having a larger index than their children.

We implement classic VAEs using a more complex encoder to match the number of parameters of the structured methods. All baselines use the same or more parameters than Graph VAE.

Figures and Tables:

S 6 I b l 5 j I I 4 G G T I d b G J l O m 5 m p k T T 8 g 1 v 5 G n f G r T s / x s S h d C H g S S Y 5 O e f e 3 D P H j z l T 2 n G + r Y 3 N r e 2 d 3 c J e c f / g 8

w h K s r 1 + M P Z 9 n < / l a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " q v / I 6 f m A z e t H s c B r 4 S e D O e e Z + L U

r J N G t e K 5 F e + h W q 7 d L p o r w D l c w B V 4 c A 0 1 u I c 6 + M C A w y u 8 w d S Z O u / O h / M 5 H 9 1 w F j t n s A T n 6 x f K k 6 C f < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q v / I 6 f m A z e t H s c B r 4 S e D O e e Z + L U

r J N G t e K 5 F e + h W q 7 d L p o r w D l c w B V 4 c A 0 1 u I c 6 + M C A w y u 8 w d S Z O u / O h / M 5 H 9 1 w F j t n s A T n 6 x f K k 6 C f < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q v / I 6 f m A z e t H s c B r 4 S e D O e e Z + L U

y Z C z 5 8 m y i j L P V b R 9 e a v t r J N G t e K 5 F e + h W q 7 d L p o r w D l c w B V 4 c A 0 1 u I c 6 + M C A w y u 8 w d S Z O u / O h / M 5 H 9 1 w F j t n s A T n 6 x f K k 6 C f < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q v / I 6 f m A z e t H s c B r 4 S e D O e e Z + L U

y Z C z 5 8 m y i j L P V b R 9 e a v t r J N G t e K 5 F e + h W q 7 d L p o r w D l c w B V 4 c A 0 1 u I c 6 + M C A w y u 8 w d S Z O u / O h / M 5 H 9 1 w F j t n s A T n 6 x f K k 6 C f < / l a t e x i t >

A A A C P 3 i c b Z D L S g M x F I Y z X m u 9 t b r T T b A K u i k z 3 e h S c O O y g m 2 F t p R M e q Y N J p k h O S P W Y c C n c a u v 4 G P 4 B O 7 E r T v T y 8 K q P w R + v n M O 5 + Q P E y k s + v6 b t 7 C 4 t L y y W l g r r m 9 s b m 2 X y j t N G 6 e G Q 4 P H M j Y 3 I b

Figure 1: Overview: Model Comparison. We show the graphical representations of (a) traditional latent variable models (VAE, ladder VAE) and (b) the proposed graph VAE. Solid lines denote generation, dashed lines denote inference, and the dotted area indicates the latent space governed by variational parameters φ and generative parameters θ. Both VAE and ladder VAE use a fixed graph structure with limited expressiveness (VAE: independent; ladder VAE: chain-structured). In contrast, graph VAE jointly optimizes a distribution over latent structures c and model parameters (φ,θ), allowing test-time sampling of a flexible, data-driven latent structure.

Figure 3: Structure Learning. In (a), we show training of the Bernoulli parameters µ i,j governing the distribution over graph structures in architecture space. All edges are color-coded and can be located in (b), where we show a random sample from the resulting steady-state distribution with the same color scheme.

Figure 4: Ablation Study on MNIST. For a fixed latent dimension M (colorcoded), we report the log-likelihood of all possible factorizations of node dimension N ′ (x-axis) and number of nodes N = M N ′ .

p θ (x, z) = p θ (x, z, c)dc = p θ (x, z c)p(c)dc = E p(c) [p θ (x, z c)] .

Figure 5: Comparison of Test-time Log-Likelihoods on MNIST. Models are trained 5 times individually, and test-time performances are evaluated every 300 epochs, with mean log-likelihood and standard deviation indicated by the colored bars.

Figure 6: TSNE-Visualization of Latent Embeddings on MNIST. We visualize embeddings of (a) Graph VAE, (b) VAE, and (c) the data itself.

Algorithm 1 Optimizing VAEs with Latent Dependency Structure Require: Data x, number of latent nodes N , number of dimensions per node N ′ .

Table content: <table/>

Peer review (iclr2019):
 The authors propose to augment the latent space of a Variational AutoEncoder [1] with an auto-regressive structure, to improve the expressiveness of both the inference network and the latent prior, making them into a general DAG of latent variables. This works goes further in the same direction as the Ladder VAE [2]. This paper introduces a mechanism for the latent model to directly learn its DAG structure by first considering the fully-connected DAG of latent variables, and adding Bernoulli variables controlling the presence or absence of each edge. The authors derive a new ELBO taking these variables into account, and use it to train the model. The gradients of the parameters of the Bernoulli variables are computed using the Gumbel-Softmax approach [3] and annealing the temperature. The authors observe with they experiments that the Bernoulli variables converge relatively quickly towards 0 or 1 during the training, fixing the structure of the DAG for the rest of the training. They test their model against a VAE, a Ladder VAE and an alternative to their model were the DAG is fixed to remain fully-connected (FC-VAE), and observe improvements in terms of the ELBO values and log-likelihood estimations. The main addition of this paper is the introduction of the gating mechanism to reduce the latent DAG from its fully-connected state. It is motivated by the tendency of latent models to fall into local optima. However, it is not clear to me what this mechanism as it is now adds to the model: - The reported results shows the improvements of Graph-VAE over FC-VAE to be quite small, making their relevance dubious in the absence of measurement of variance accross different trainings. Additionally, the reported performances for Ladder VAE are inferior to what [2] reports. Actually the performance of Ladder-VAE reported in [2] is better than the one reported for Graph-VAE in this paper, both on the MNIST and Omniglot datasets. - The authors observe that the Bernoulli variables have converged after around ~200 epochs. At this time, according to their reported experimental setup, the Gumbel-Softmax temperature is 0.999^200 ~= 0.82, which is still quite near 1.0, meaning the model is still pretty far from a real Bernoulli-like behavior. And actually, equation 9 is not a proper description of the Gumbel-Softmax as described by [3] : there should be only 2 samples from the Gumbel distribution, not 3. Given these two issues, I can*t believe that the c_ij coefficients behave like Bernoulli variables in this experiment. As such, It seems to me that Graph-VAE is nothing more than a special reparametrization of FC-VAE that tends to favor saturating behavior for the c_ij variables. - On figure 3b, the learned structure is very symmetrical (z2, z3, z4 play an identical role in the final DAG). In my opinion, this begs for the introduction of a regulatory mechanism regarding the gating variable to push the model towards sparsity. I was honestly surprised to see this gating mechanism introduced without anything guiding the convergence of the c_ij variables. I like the idea of learning a latent structure DAG for VAEs, but this paper introduces a rather weak way to try to achieve this, and the experimental results are not convincing. [1] https://arxiv.org/abs/1312.6114 [2] https://arxiv.org/abs/1602.02282 [3] https://arxiv.org/abs/1611.01144