Title: VARIATIONAL AUTOENCODERS WITH JOINTLY OPTIMIZED LATENT DEPENDENCY STRUCTURE

Abstract: We propose a method for learning the dependency structure between latent variables in deep latent variable models. Our general modeling and inference framework combines the complementary strengths of deep generative models and probabilistic graphical models. In particular, we express the latent variable space of a variational autoencoder (VAE) in terms of a Bayesian network with a learned, flexible dependency structure. The network parameters, variational parameters as well as the latent topology are optimized simultaneously with a single objective. Inference is formulated via a sampling procedure that produces expectations over latent variable structures and incorporates top-down and bottom-up reasoning over latent variable values. We validate our framework in extensive experiments on MNIST, Omniglot, and CIFAR-10. Comparisons to state-of-the-art structured variational autoencoder baselines show improvements in terms of the expressiveness of the learned model.

1. INTRODUCTION:

Deep latent variable models offer an effective method for automatically learning structure from data. By explicitly modeling the data distribution using latent variables, these models are capable of learning compressed representations that are then relevant for downstream tasks. Such models have been applied across a wide array of domains, such as images (Gregor et al., 2014; Kingma & Welling, 2014; Rezende et al., 2014; Gregor et al., 2015) , audio (Chung et al., 2015; Fraccaro et al., 2016) , video (He et al., 2018; Yingzhen & Mandt, 2018) , and text (Bowman et al., 2016; Krishnan et al., 2017) . However, despite their success, latent variable models are often formulated with simple (e.g. Gaussian) distributions, making independence assumptions among the latent variables. That is, each latent variable is sampled independently. Ignoring the dependencies between latent variables limits the flexibility of these models, negatively impacting the model's ability to fit the data.

In general, structural dependencies can be incorporated into all phases of a forward process, including inference, latent model, and output space: Normalizing flows (Rezende & Mohamed, 2015) , for instance, accounts for dependencies during inference by learning a mapping from a simple distribution to a more complex distribution that contains these dependencies. Structured output networks (Lehrmann & Sigal, 2017) , on the other hand, directly predict an expressive non-parametric output distribution. On the modeling side, one can add dependencies by constructing a hierarchical latent representation (Dayan et al., 1995) . These structures consist of conditional (empirical) priors, in which one latent variable forms a prior on another latent variable. While this conditional distribution may take a simple form, marginalizing over the parent variable can result in an arbitrarily complex distribution. Models with these more flexible latent dependency structures have been shown to result in improved performance (Sønderby et al., 2016; Burda et al., 2016; Kingma et al., 2016) . However, despite the benefits of including additional structure in these models, their dependency structures have so far been predefined, potentially limiting the performance of this approach.

In this work, we propose a method for learning dependency structures in latent variable models. Structure learning is a difficult task with a long history in the graphical models community (Koller & Friedman, 2009) . Over the years, it has been tackled from several perspectives, including constraint-based approaches (Cheng et al., 2002; Lehmann & Romano, 2008) , optimization of structure scores (Kass & Raftery, 1995; Heckerman et al., 1995; Barron et al., 1998) , Bayesian model averaging (Heckerman et al., 1999; Koivisto & Sood, 2004) , and many more. Unfortunately, the underlying objectives are often limited to graphs of a particular form (e.g., limited tree width), prohibitively expensive, or difficult to integrate with the gradient-based optimization techniques of modern neural networks. Here, we discuss an end-to-end approach for general graph structures introducing minimal complexity overhead. In particular, we introduce a set of binary global variables to gate the latent dependencies. The whole model (including its structure) is jointly optimized with a single stochastic variational inference objective. In our experimental validation, we show that the learned dependency structures result in models that more accurately model the data distribution, outperforming several common predefined latent dependency structures.

2. BACKGROUND 2.1 VARIATIONAL INFERENCE & VARIATIONAL AUTOENCODERS:

A latent variable model, defined by the joint distribution, p θ (x, z) = p θ (x z)p θ (z), models each data example, x, using a local latent variable, z, and global parameters, θ. p θ (x z) denotes the conditional likelihood, and p θ (z) denotes the prior. Latent variable models are capable of capturing the structure present in data, with z forming a compressed representation of each data example. Unfortunately, inferring the posterior, p θ (z x), is typically computationally intractable, prompting the use of approximate inference techniques. Variational inference (Jordan et al., 1999) introduces an approximate posterior, q φ (z x), and optimizes variational parameters, φ, to minimize the KL-divergence to the true posterior, KL(q φ (z x) p θ (z x)). As this quantity cannot be evaluated directly, the following relation is used: log p θ (x) = KL(q φ (z x) p θ (z x)) + L(x; θ, φ),

where L(x; θ, φ) is the evidence lower bound (ELBO), defined as L(x; θ, φ) = E q φ (z x) [log p θ (x z)] -KL(q φ (z x) p θ (z)).

(2)

In Eq. (1), log p θ (x) is independent of φ, so we can minimize the KL divergence term, i.e. perform approximate inference, by maximizing L(x; θ, φ) w.r.t. q φ (z x). Further, because KL divergence is non-negative, L(x; θ, φ) is a lower bound on log p θ (x), meaning we can then learn the model parameters by maximizing L(x; θ, φ) w.r.t. θ.

Variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) amortize inference optimization across data examples by parameterizing q φ (z x) as a separate inference model, then jointly optimizing the model parameters θ and φ. VAEs instantiate both the inference model and latent variable model with deep networks, allowing them to scale to high-dimensional data. However, VAEs are typically implemented with basic graphical structures and simple, unimodal distributions (e.g. Gaussians). For instance, the dimensions of the prior are often assumed to be independent, p θ (z) = ∏ m p θ (z m ), with a common assumption being a fixed standard Gaussian: p θ (z) = N (z; 0, I). Similarly, approximate posteriors often make the mean field assumption, q φ (z x) = ∏ m q φ (z m x). Independence assumptions such as these may be overly restrictive, thereby limiting modeling capabilities.

2.2. EMPIRICAL PRIORS THROUGH LATENT DEPENDENCY STRUCTURE:

One technique for improving the expressive capacity of latent variable models is through incorporating dependency structure among the latent variables, forming a hierarchy (Dayan et al., 1995;  zm < l a t e x i t s h a 1 _ b a s e 6 4 = " F Q U F B 8 L N x t S L j q O l + X L S 2 P 4 F 8 M M = " > A A A C K H i c b V D N T g I x G G z 9 R R Q F P X p p B B N P Z J e L H k m 8 e M T E B R L Y k G 7 p Q k P b 3 b R d I 2 7 2 G b z q K / g 0 3 g x X n 8 S y c H D B S Z p M Z r 4 v 3 3 S C m D N t H G c B d 3 b 3 9 g 8 O S 0 f l 4 5 P K 6 V m 1 d t 7 V U a I I 9 U j E I 9 U P s K a c S e o Z Z j j t x 4 p i E X D a C 2 b 3 S 7 / 3 T J V m k X w y 8 5 j 6 A k 8 k C x n B x k p e 4 3 U k G q N q 3 W k 6 O d A 2 c d e k D t b o j G q w M h x H J B F U G s K x 1 g P X i Y 2 f Y m U Y 4 T Q r D x N N Y 0 x m e E I H l k o s q P b T P G 2 G r q 0 y R m G k 7 J M G 5 e r f j R Q L r e c i s J M C m 6 n e 9 J b i f 9 4 g M e G d n z I Z J 4 Z K s j o U J h y Z C C 2 / j s Z M U W L 4 3 B J M F L N Z E Z l i h Y m x B R W u T B S O p 4 y 8 Z E W V y j x X Q Q x E V r Y F u p t 1 b Z N u q + k 6 T f e x V W + 3 1 l W W w C W 4 A j f A B b e g D R 5 A B 3 i A A A b e w D v 4 g J / w C 3 7 D x W p 0 B 6 5 3 L k A B 8 O c X c d K m E A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " F Q U F B 8 L N x t S L j q O l + X L S 2 P 4 F 8 M M = " > A A A C K H i c b V D N T g I x G G z 9 R R Q F P X p p B B N P Z J e L H k m 8 e M T E B R L Y k G 7 p Q k P b 3 b R d I 2 7 2 G b z q K / g 0 3 g x X n 8 S y c H D B S Z p M Z r 4 v 3 3 S C m D N t H G c B d 3 b 3 9 g 8 O S 0 f l 4 5 P K 6 V m 1 d t 7 V U a I I 9 U j E I 9 U P s K a c S e o Z Z j j t x 4 p i E X D a C 2 b 3 S 7 / 3 T J V m k X w y 8 5 j 6 A k 8 k C x n B x k p e 4 3 U k G q N q 3 W k

6 O d A 2 c d e k D t b o j G q w M h x H J B F U G s K x 1 g P X i Y 2 f Y m U Y 4 T Q r D x N N Y 0 x m e E I H l k o s q P b T P G 2 G r q 0 y R m G k 7 J M G 5 e r f j R Q L r e c i s J M C m 6 n e 9 J b i f 9 4 g M e G d n z I Z J 4 Z K s j o U J h y Z C C 2 / j s Z M U W L 4 3 B J M F L N Z E Z l i h Y m x B R W u T B S O p 4 y 8 Z E W V y j x X Q Q x E V r Y F u p t 1 b Z N u q + k 6 T f e x V W + 3 1 l W W w C W 4 A j f A B b e g D R 5 A B 3 i A A A b e w D v 4 g J / w C 3 7 D x W p 0 B 6 5 3 L k A B 8 O c X c d K m E A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " F Q U F B 8 L N x t S L j q O l + X L S 2 P 4 F 8 M M = " > A A A C K H i c b V D N T g I x G G z 9 R R Q F P X p p B B N P Z J e L H k m 8 e M T E B R L Y k G 7 p Q k P b 3 b R d I 2 7 2 G b z q K / g 0 3 g x X n 8 S y c H D B S Z p M Z r 4 v 3 3 S C m D N t H G c B d 3 b 3 9 g 8 O S 0 f l 4 5 P K 6 V m 1 d t 7 V U a I I 9 U j E I 9 U P s K a c S e o Z Z j j t x 4 p i E X D a C 2 b 3 S 7 / 3 T J V m k X w y 8 5 j 6 A k 8 k C x n B x k p e 4 3 U k G q N q 3 W k 6 O d A 2 c d e k D t b o j G q w M h x H J B F U G s K x 1 g P X i Y 2 f Y m U Y 4 T Q r D x N N Y 0 x m e E I H l k o s q P b T P G 2 G r q 0 y R m G k 7 J M G 5 e r f j R Q L r e c i s J M C m 6 n e 9 J b i f 9 4 g M e G d n z I Z J 4 Z K s j o U J h y Z C C 2 / j s Z M U W L 4 3 B J M F L N Z E Z l i h Y m x B R W u T B S O p 4 y 8 Z E W V y j x X Q Q x E V r Y F u p t 1 b Z N u q + k 6 T f e x V W + 3 1 l W W w C W 4 A j f A B b e g D R 5 A B 3 i A A A b e w D v 4 g J / w C 3 7 D x W p 0 B 6 5 3 L k A B 8 O c X c d K m E A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " F Q U F B 8 L N x t S L j q O l + X L S 2 P 4 F 8 M M = " > A A A C K H i c b V D N T g I x G G z 9 R R Q F P X p p B B N P Z J e L H k m 8 e M T E B R L Y k G 7 p Q k P b 3 b R d I 2 7 2 G b z q K / g 0 3 g x X n 8 S y c H D B S Z p M Z r 4 v 3 3 S C m D N t H G c B d 3 b 3 9 g 8 O S 0 f l 4 5 P K 6 V m 1 d t 7 V U a I I 9 U j E I 9 U P s K a c S e o Z Z j j t x 4 p i E X D a C 2 b 3 S 7 / 3 T J V m k X w y 8 5 j 6 A k 8 k C x n B x k p e 4 3 U k G q N q 3 W k 6 O d A 2 c d e k D t b o j G q w M h x H J B F U G s K x 1 g P X i Y 2 f Y m U Y 4 T Q r D x N N Y 0 x m e E I H l k o s q P b T P G 2 G r q 0 y R m G k 7 J M G 5 e r f j R Q L r e c i s J M C m 6 n e 9 J b i f 9 4 g M e G d n z I Z J 4 Z K s j o U J h y Z C C 2 / j s Z M U W L 4 3 B J M F L N Z E Z l i h Y m x B R W u T B S O p 4 y 8 Z E W V y j x X Q Q x E V r Y F u p t 1 b Z N u q + k 6 T f e x V W + 3 1 l W W w C W 4 A j f A B b e g D R 5 A B 3 i A A A b e w D v 4 g J / w C 3 7 D x W p 0 B 6 5 3 L k A B 8 O c X c d K m E A = = < / l a t e x i t > p✓(z) = Q n p✓(zn|zn+1) < l a t e x i t s h a 1 _ b a s e 6 4 = " U h Y X z I E 1 9 f 4 U K d U 3 5 c K 5 F + u 1 s v A = " > A A A C b X i c b V D R S h t B F J 2 s r U 1 j t d H S J 0 s Z j K I i h N 2 8 m B c h 4 E s e F R o V s m G Z n d x N h s z O D j N 3 x W T d X / J r + t K H + g v + g p M Y s N E e G D h z z r 3 c e 0 + s p b D o + 3 8 q 3 t q H j + u f q p 9 r G 1 8 2 t 7 7 W t 3 e u b J Y b D j 2 e y c z c x M y C F A p 6 K F D C j T b A 0 l j C d T w 5 n / v X t 2 C s y N Q v n G o Y p G y k R C I 4 Q y d F 9 e 6 + j k I c A z J 6 F K Y M x 3 F S z M p j e k Z D b b J h p O i r X 4 R x Q m d l V K i S 3 t P X 3 0 l Q H u 9 H 9 Y b f 9 B e g 7 0 m w J A 2 y x E W 0 X d k M h x n P U 1 D I J b O 2 H / g a B w U z K L i E s h b m F j T j E z a C v q O K p W A H x e L k k h 4 4 Z U i T z L i n k C 7 U f z s K l l o 7 T W N X O b / K v v X m 4 v + 8 f o 5 J e 1 A I p X M E x V 8 G J b m k m N F 5 f n Q o D H C U U 0 c Y N 8 L t S v m Y G c b R p b w y Z W S Y H g t + V 6 6 q o B Z 7 r Y h x W t Z c g M H b u N 6 T q 1 Y z 8 J v B Z a v R a S + j r J J d s k e O S E B O S Y d 0 y Q X p E U 4 e y G / y l z x W n r z v 3 g / v 5 0 u p V 1 n 2 f C M r 8 A 6 f A b 9 f v N A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U h Y X z I E 1 9 f 4 U K d U 3 5 c K 5 F + u 1 s v A = " > A A A C b X i c b V D R S h t B F J 2 s r U 1 j t d H S J 0 s Z j K I i h N 2 8 m B c h 4 E s e F R o V s m G Z n d x N h s z O D j N 3 x W T d X / J r + t K H + g v + g p M Y s N E e G D h z z r 3 c e 0 + s p b D o + 3 8 q 3 t q H j + u f q p 9 r G 1 8 2 t 7 7 W t 3 e u b J Y b D j 2 e y c z c x M y C F A p 6 K F D C j T b A 0 l j C d T w 5 n / v X t 2 C s y N Q v n G o Y p G y k R C I 4 Q y d F 9 e 6 + j k I c A z J 6 F K Y M x 3 F S z M p j e k Z D b b J h p O i r X 4 R x Q m d l V K i S 3 t P X 3 0 l Q H u 9 H 9 Y b f 9 B e g 7 0 m w J A 2 y x E W 0 X d k M h x n P U 1 D I J b O 2 H / g a B w U z K L i E s h b m F j T j E z a C v q O K p W A H x e L k k h 4 4 Z U i T z L i n k C 7 U f z s K l l o 7 T W N X O b / K v v X m 4 v + 8 f o 5 J e 1 A I p X M E x V 8 G J b m k m N F 5 f n Q o D H C U U 0 c Y N 8 L t S v m Y G c b R p b w y Z W S Y H g t + V 6 6 q o B Z 7 r Y h x W t Z c g M H b u N 6 T q 1 Y z 8 J v B Z a v R a S + j r J J d s k e O S E B O S Y d 0 y Q X p E U 4 e y G / y l z x W n r z v 3 g / v 5 0 u p V 1 n 2 f C M r 8 A 6 f A b 9 f v N A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U h Y X z I E 1 9 f 4 U K d U 3 5 c K 5 F + u 1 s v A = " > A A A C b X i c b V D R S h t B F J 2 s r U 1 j t d H S J 0 s Z j K I i h N 2 8 m B c h 4 E s e F R o V s m G Z n d x N h s z O D j N 3 x W T d X / J r + t K H + g v + g p M Y s N E e G D h z z r 3 c e 0 + s p b D o + 3 8 q 3 t q H j + u f q p 9 r G 1 8 2 t 7 7 W t 3 e u b J Y b D j 2 e y c z c x M y C F A p 6 K F D C j T b A 0 l j C d T w 5 n / v X t 2 C s y N Q v n G o Y p G y k R C I 4 Q y d F 9 e 6 + j k I c A z J 6 F K Y M x 3 F S z M p j e k Z D b b J h p O i r X 4 R x Q m d l V K i S 3 t P X 3 0 l Q H u 9 H 9 Y b f 9 B e g 7 0 m w J A 2 y x E W 0 X d k M h x n P U 1 D I J b O 2 H / g a B w U z K L i E s h b m F j T j E z a C v q O K p W A H x e L k k h 4 4 Z U i T z L i n k C 7 U f z s K l l o 7 T W N X O b / K v v X m 4 v + 8 f o 5 J e 1 A I p X M E x V 8 G J b m k m N F 5 f n Q o D H C U U 0 c Y N 8 L t S v m Y G c b R p b w y Z W S Y H g t + V 6 6 q o B Z 7 r Y h x W t Z c g M H b u N 6 T q 1 Y z 8 J v B Z a v R a S + j r J J d s k e O S E B O S Y d 0 y Q X p E U 4 e y G / y l z x W n r z v 3 g / v 5 0 u p V 1 n 2 f C M r 8 A 6 f A b 9 f v N A = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U h Y X z I E 1 9 f 4 U K d U 3 5 c K 5 F + u 1 s v A = " > A A A C b X i c b V D R S h t B F J 2 s r U 1 j t d H S J 0 s Z j K I i h N 2 8 m B c h 4 E s e F R o V s m G Z n d x N h s z O D j N 3 x W T d X / J r + t K H + g v + g p M Y s N E e G D h z z r 3 c e 0 + s p b D o + 3 8 q 3 t q H j + u f q p 9 r G 1 8 2 t 7 7 W t 3 e u b J Y b D j 2 e y c z c x M y C F A p 6 K F D C j T b A 0 l j C d T w 5 n / v X t 2 C s y N Q v n G o Y p G y k R C I 4 Q y d F 9 e 6 + j k I c A z J 6 F K Y M x 3 F S z M p j e k Z D b b J h p O i r X 4 R x Q m d l V K i S 3 t P X 3 0 l Q H u 9 H 9 Y b f 9 B e g 7 0 m w J A 2 y x E W 0 X d k M h x n P U 1 D I J b O 2 H / g a B w U z K L i E s h b m F j T j E z a C v q O K p W A H x e L k k h 4 4 Z U i T z L i n k C 7 U f z s K l l o 7 T W N X O b / K v v X m 4 v + 8 f o 5 J e 1 A I p X M E x V 8 G J b m k m N F 5 f n Q o D H C U U 0 c Y N 8 L t S v m Y G c b R p b w y Z W S Y H g t + V 6 6 q o B Z 7 r Y h x W t Z c g M H b u N 6 T q 1 Y z 8 J v B Z a v R a S + j r J J d s k e O S E B O S Y d 0 y Q X p E U 4 e y G / y l z x W n r z v 3 g / v 5 0 u p V 1 n 2 f C M r 8 A 6 f A b 9 f v N A = < / l a t e x i t > zn < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7 Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < / l a t e x i t > zn+1 < l a t e x i t s h a 1 _ b a s e 6 4 = " r H C O l 6 R F Y o x d c B Q Q I K M J M W S V D 5 s = " > A A A C N H i c b V D L S s N A F J 3 4 r N X a V p d u B l t B E E r S j S 4 L b l x W s A 9 o Q 5 h M J + 3 Q m U m Y m Y g 1 5 E v c 6 i / 4 L 4 I 7 c e s 3 O E 2 z M K 0 H L h z O u Z d 7 O H 7 E q N K 2 / W F t b e / s 7 u 2 X D s q H R 5 X j a q 1 + 0 l d h L D H p 4 Z C F c u g j R R g V p K e p Z m Q Y S Y K 4 z 8 j A n 9 8 u / c E j k Y q G 4 k E v I u J y N B U 0 o B h p I 3 m 1 a j M Z + w F 8 T r 1 E X D l p 0 6 s 1 7 J a d A W 4 S J y c N k K P r 1 a 3 K e B L i m B O h M U N K j R w 7 0 m 6 C p K a Y k b Q 8 j h W J E J 6 j K R k Z K h A n y k 2 y 5 C m 8 M M o E B q E 0 I z T M 1 L 8 X C e J K L b h v N j n S M 7 X u L c X / v F G s g x s 3 o S K K N R F 4 9 S i I G d Q h X N Y A J 1 Q S r N n C E I Q l N V k h n i G J s D Z l F b 5 M J Y p m F D + l R Z W I L F d B 9 H l a N g U 6 6 3 V t k n 6 7 5 d g t 5 7 7 d 6 L T z K k v g D J y D S + C A a 9 A B d 6 A L e g C D G L y A V / B m v V u f 1 p f 1 v V r d s v K b U 1 C A 9 f M L x J G q N g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r H C O l 6 R F Y o x d c B Q Q I K M J M W S V D 5 s = " > A A A C N H i c b V D L S s N A F J 3 4 r N X a V p d u B l t B E E r S j S 4 L b l x W s A 9 o Q 5 h M J + 3 Q m U m Y m Y g 1 5 E v c 6 i / 4 L 4 I 7 c e s 3 O E 2 z M K 0 H L h z O u Z d 7 O H 7 E q N K 2 / W F t b e / s 7 u 2 X D s q H R 5 X j a q 1 + 0 l d h L D H p 4 Z C F c u g j R R g V p K e p Z m Q Y S Y K 4 z 8 j A n 9 8 u / c E j k Y q G 4 k E v I u J y N B U 0 o B h p I 3 m 1 a j M Z + w F 8 T r 1 E X D l p 0 6 s 1 7 J a d A W 4 S J y c N k K P r 1 a 3 K e B L i m B O h M U N K j R w 7 0 m 6 C p K a Y k b Q 8 j h W J E J 6 j K R k Z K h A n y k 2 y 5 C m 8 M M o E B q E 0 I z T M 1 L 8 X C e J K L b h v N j n S M 7 X u L c X / v F G s g x s 3 o S K K N R F 4 9 S i I G d Q h X N Y A J 1 Q S r N n C E I Q l N V k h n i G J s D Z l F b 5 M J Y p m F D + l R Z W I L F d B 9 H l a N g U 6 6 3 V t k n 6 7 5 d g t 5 7 7 d 6 L T z K k v g D J y D S + C A a 9 A B d 6 A L e g C D G L y A V / B m v V u f 1 p f 1 v V r d s v K b U 1 C A 9 f M L x J G q N g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r H C O l 6 R F Y o x d c B Q Q I K M J M W S V D 5 s = " > A A A C N H i c b V D L S s N A F J 3 4 r N X a V p d u B l t B E E r S j S 4 L b l x W s A 9 o Q 5 h M J + 3 Q m U m Y m Y g 1 5 E v c 6 i / 4 L 4 I 7 c e s 3 O E 2 z M K 0 H L h z O u Z d 7 O H 7 E q N K 2 / W F t b e / s 7 u 2 X D s q H R 5 X j a q 1 + 0 l d h L D H p 4 Z C F c u g j R R g V p K e p Z m Q Y S Y K 4 z 8 j A n 9 8 u / c E j k Y q G 4 k E v I u J y N B U 0 o B h p I 3 m 1 a j M Z + w F 8 T r 1 E X D l p 0 6 s 1 7 J a d A W 4 S J y c N k K P r 1 a 3 K e B L i m B O h M U N K j R w 7 0 m 6 C p K a Y k b Q 8 j h W J E J 6 j K R k Z K h A n y k 2 y 5 C m 8 M M o E B q E 0 I z T M 1 L 8 X C e J K L b h v N j n S M 7 X u L c X / v F G s g x s 3 o S K K N R F 4 9 S i I G d Q h X N Y A J 1 Q S r N n C E I Q l N V k h n i G J s D Z l F b 5 M J Y p m F D + l R Z W I L F d B 9 H l a N g U 6 6 3 V t k n 6 7 5 d g t 5 7 7 d 6 L T z K k v g D J y D S + C A a 9 A B d 6 A L e g C D G L y A V / B m v V u f 1 p f 1 v V r d s v K b U 1 C A 9 f M L x J G q N g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r H C O l 6 R F Y o x d c B Q Q I K M J M W S V D 5 s = " > A A A C N H i c b V D L S s N A F J 3 4 r N X a V p d u B l t B E E r S j S 4 L b l x W s A 9 o Q 5 h M J + 3 Q m U m Y m Y g 1 5 E v c 6 i / 4 L 4 I 7 c e s 3 O E 2 z M K 0 H L h z O u Z d 7 O H 7 E q N K 2 / W F t b e / s 7 u 2 X D s q H R 5 X j a q 1 + 0 l d h L D H p 4 Z C F c u g j R R g V p K e p Z m Q Y S Y K 4 z 8 j A n 9 8 u / c E j k Y q G 4 k E v I u J y N B U 0 o B h p I 3 m 1 a j M Z + w F 8 T r 1 E X D l p 0 6 s 1 7 J a d A W 4 S J y c N k K P r 1 a 3 K e B L i m B O h M U N K j R w 7 0 m 6 C p K a Y k b Q 8 j h W J E J 6 j K R k Z K h A n y k 2 y 5 C m 8 M M o E B q E 0 I z T M 1 L 8 X C e J K L b h v N j n S M 7 X u L c X / v F G s g x s 3 o S K K N R F 4 9 S i I G d Q h X N Y A J 1 Q S r N n C E I Q l N V k h n i G J s D Z l F b 5 M J Y p m F D + l R Z W I L F d B 9 H l a N g U 6 6 3 V t k n 6 7 5 d g t 5 7 7 d 6 L T z K k v g D J y D S + C A a 9 A B d 6 A L e g C D G L y A V / B m v V u f 1 p f 1 v V r d s v K b U 1 C A 9 f M L x J G q N g = = < / l a t e x i t > (a) Traditional Latent Variable Models x < l a t e x i t s h a 1 _ b a s e 6 4 = " y M K v c s b Q N k v F r B n V k m 2 K + W 2 6 W + g = " > A A A C G X i c b V C 7 T g J B F L 3 r E / G F W t p M B B M r s k u j J d H G E h N 5 J L A h s 8 N d m D A 7 u 5 m Z N R D C R 9 j K 1 9 g Z W y s / x s R h o R D w J J O c n H N v 7 p k T J I J r 4 7 r f z t b 2 z u 7 e f u 4 g f 3 h 0 f H J a O D t v 6 D h V D O s s F r F q B V S j 4 B L r h h u B r U Q h j Q K B z W D 4 M P e b L 6 g 0 j + W z G S f o R 7 Q v e c g Z N V Z q l j p B S E a l b q H o l t 0 M Z J N 4 S 1 K E J W r d w k + n F 7 M 0 Q m m Y o F q 3 P T c x / o Q q w 5 n A a b 6 T a k w o G 9 I + t i 2 V N E L t T 7 K 4 U 3 J t l R 4 J Y 2 W f N C R T / 2 5 M a K T 1 O A r s Z E T N Q K 9 7 c / E / r 5 2 a 8 M 6 f c J m k B i V b H A p T Q U x M 5 n 8 n P a 6 Q G T G 2 h D L F b V b C B l R R Z m x D K 1 f 6 i i Y D z k b T V R V l l i t v + / L W 2 9 k k j U r Z c 8 v e U 6 V Y v V 8 2 l 4 N L u I I b 8 O A W q v A I N a g D g y G 8 w h v M n J n z 7 n w 4 n 4 v R L W e 5 c w E r c L 5 + A Y 5 0 o P 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " y M K v c s b Q N k v F r B n V k m 2 K + W 2 6 W + g = " > A A A C G X i c b V C 7 T g J B F L 3 r E / G F W t p M B B M r s k u j J d H G E h N 5 J L A h s 8 N d m D A 7 u 5 m Z N R D C R 9 j K 1 9 g Z W y s / x s R h o R D w J J O c n H N v 7 p k T J I J r 4 7 r f z t b 2 z u 7 e f u 4 g f 3 h 0 f H J a O D t v 6 D h V D O s s F r F q B V S j 4 B L r h h u B r U Q h j Q K B z W D 4 M P e b L 6 g 0 j + W z G S f o R 7 Q v e c g Z N V Z q l j p B S E a l b q H o l t 0 M Z J N 4 S 1 K E J W r d w k + n F 7 M 0 Q m m Y o F q 3 P T c x / o Q q w 5 n A a b 6 T a k w o G 9 I + t i 2 V N E L t T 7 K 4 U 3 J t l R 4 J Y 2 W f N C R T / 2 5 M a K T 1 O A r s Z E T N Q K 9 7 c / E / r 5 2 a 8 M 6 f c J m k B i V b H A p T Q U x M 5 n 8 n P a 6 Q G T G 2 h D L F b V b C B l R R Z m x D K 1 f 6 i i Y D z k b T V R V l l i t v + / L W 2 9 k k j U r Z c 8 v e U 6 V Y v V 8 2 l 4 N L u I I b 8 O A W q v A I N a g D g y G 8 w h v M n J n z 7 n w 4 n 4 v R L W e 5 c w E r c L 5 + A Y 5 0 o P 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " y M K v c s b Q N k v F r B n V k m 2 K + W 2 6 W + g = " > A A A C G X i c b V C 7 T g J B F L 3 r E / G F W t p M B B M r s k u j J d H G E h N 5 J L A h s 8 N d m D A 7 u 5 m Z N R D C R 9 j K 1 9 g Z W y s / x s R h o R D w J J O c n H N v 7 p k T J I J r 4 7 r f z t b 2 z u 7 e f u 4 g f 3 h 0 f H J a O D t v 6 D h V D O s s F r F q B V S j 4 B L r h h u B r U Q h j Q K B z W D 4 M P e b L 6 g 0 j + W z G S f o R 7 Q v e c g Z N V Z q l j p B S E a l b q H o l t 0 M Z J N 4 S 1 K E J W r d w k + n F 7 M 0 Q m m Y o F q 3 P T c x / o Q q w 5 n A a b 6 T a k w o G 9 I + t i 2 V N E L t T 7 K 4 U 3 J t l R 4 J Y 2 W f N C R T / 2 5 M a K T 1 O A r s Z E T N Q K 9 7 c / E / r 5 2 a 8 M 6 f c J m k B i V b H A p T Q U x M 5 n 8 n P a 6 Q G T G 2 h D L F b V b C B l R R Z m x D K 1 f 6 i i Y D z k b T V R V l l i t v + / L W 2 9 k k j U r Z c 8 v e U 6 V Y v V 8 2 l 4 N L u I I b 8 O A W q v A I N a g D g y G 8 w h v M n J n z 7 n w 4 n 4 v R L W e 5 c w E r c L 5 + A Y 5 0 o P 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " y M K v c s b Q N k v F r B n V k m 2 K + W 2 6 W + g = " > A A A C G X i c b V C 7 T g J B F L 3 r E / G F W t p M B B M r s k u j J d H G E h N 5 J L A h s 8 N d m D A 7 u 5 m Z N R D C R 9 j K 1 9 g Z W y s / x s R h o R D w J J O c n H N v 7 p k T J I J r 4 7 r f z t b 2 z u 7 e f u 4 g f 3 h 0 f H J a O D t v 6 D h V D O s s F r F q B V S j 4 B L r h h u B r U Q h j Q K B z W D 4 M P e b L 6 g 0 j + W z G S f o R 7 Q v e c g Z N V Z q l j p B S E a l b q H o l t 0 M Z J N 4 S 1 K E J W r d w k + n F 7 M 0 Q m m Y o F q 3 P T c x / o Q q w 5 n A a b 6 T a k w o G 9 I + t i 2 V N E L t T 7 K 4 U 3 J t l R 4 J Y 2 W f N C R T / 2 5 M a K T 1 O A r s Z E T N Q K 9 7 c / E / r 5 2 a 8 M 6 f c J m k B i V b H A p T Q U x M 5 n 8 n P a 6 Q G T G 2 h D L F b V b C B l R R Z m x D K 1 f 6 i i Y D z k b T V R V l l i t v + / L W 2 9 k k j U r Z c 8 v e U 6 V Y v V 8 2 l 4 N L u I I b 8 O A W q v A I N a g D g y G 8 w h v M n J n z 7 n w 4 n 4 v R L W e 5 c w E r c L 5 + A Y 5 0 o P 0 = < / l a t e x i t > c < l a t e x i t s h a 1 _ b a s e 6 4 = " L 5 v s F d f C 3 a d a i A y 6 S N / m h T Z l f f 4 = " > A A A C G X i c b V C 7 T g J B F L 2 L L 8 Q X a m k z E U y s y C 6 N l k Q b S 0 z k k c C G z A 5 3 Y c L s 7 G Z m 1 k g I H 2 E r X 2 N n b K 3 8 G B O H h U L A k 0 x y c s 6 9 u W d O k A i u j e t + O 7 m t 7 Z 3 d v f x + 4 e D w 6 P i k e H r W 1 H G q G D Z Y L G L V D q h G w S U 2 D D c C 2 4 l C G g U C W 8 H o f u 6 3 n l F p H s s n M 0 7 Q j + h A 8 p A z a q z U K n e D k L B y r 1 h y K 2 4 G s k m 8 J S n B E v V e 8 a f b j 1 k a o T R M U K 0 7 n p s Y f 0 K V 4 U z g t N B N N S a U j e g A O 5 Z K G q H 2 J 1 n c K b m y S p + E s b J P G p K p f z c m N N J 6 H A V 2 M q J m q N e 9 u f i f 1 0 l N e O t P u E x S g 5 I t D o W p I C Y m 8 7 + T P l f I j B h b Q p n i N i t h Q 6 o o M 7 a h l S s D R Z M h Z y / T V R V l l q t g + / L W 2 9 k k z W r F c y v e Y 7 V U u 1 s 2 l 4 c L u I R r 8 O A G a v A A d W g A g x G 8 w h v M n J n z 7 n w 4 n 4 v R n L P c O Y c V O F + / a w S g 6 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L 5 v s F d f C 3 a d a i A y 6 S N / m h T Z l f f 4 = " > A A A C G X i c b V C 7 T g J B F L 2 L L 8 Q X a m k z E U y s y C 6 N l k Q b S 0 z k k c C G z A 5 3 Y c L s 7 G Z m 1 k g I H 2 E r X 2 N n b K 3 8 G B O H h U L A k 0 x y c s 6 9 u W d O k A i u j e t + O 7 m t 7 Z 3 d v f x + 4 e D w 6 P i k e H r W 1 H G q G D Z Y L G L V D q h G w S U 2 D D c C 2 4 l C G g U C W 8 H o f u 6 3 n l F p H s s n M 0 7 Q j + h A 8 p A z a q z U K n e D k L B y r 1 h y K 2 4 G s k m 8 J S n B E v V e 8 a f b j 1 k a o T R M U K 0 7 n p s Y f 0 K V 4 U z g t N B N N S a U j e g A O 5 Z K G q H 2 J 1 n c K b m y S p + E s b J P G p K p f z c m N N J 6 H A V 2 M q J m q N e 9 u f i f 1 0 l N e O t P u E x S g 5 I t D o W p I C Y m 8 7 + T P l f I j B h b Q p n i N i t h Q 6 o o M 7 a h l S s D R Z M h Z y / T V R V l l q t g + / L W 2 9 k k z W r F c y v e Y 7 V U u 1 s 2 l 4 c L u I R r 8 O A G a v A A d W g A g x G 8 w h v M n J n z 7 n w 4 n 4 v R n L P c O Y c V O F + / a w S g 6 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L 5 v s F d f C 3 a d a i A y 6 S N / m h T Z l f f 4 = " > A A A C G X i c b V C 7 T g J B F L 2 L L 8 Q X a m k z E U y s y C 6 N l k Q b S 0 z k k c C G z A 5 3 Y c L s 7 G Z m 1 k g I H 2 E r X 2 N n b K 3 8 G B O H h U L A k 0 x y c s 6 9 u W d O k A i u j e t + O 7 m t 7 Z 3 d v f x + 4 e D w 6 P i k e H r W 1 H G q G D Z Y L G L V D q h G w S U 2 D D c C 2 4 l C G g U C W 8 H o f u 6 3 n l F p H s s n M 0 7 Q j + h A 8 p A z a q z U K n e D k L B y r 1 h y K 2 4 G s k m 8 J S n B E v V e 8 a f b j 1 k a o T R M U K 0 7 n p s Y f 0 K V 4 U z g t N B N N S a U j e g A O 5 Z K G q H 2 J 1 n c K b m y S p + E s b J P G p K p f z c m N N J 6 H A V 2 M q J m q N e 9 u f i f 1 0 l N e O t P u E x S g 5 I t D o W p I C Y m 8 7 + T P l f I j B h b Q p n i N i t h Q 6 o o M 7 a h l S s D R Z M h Z y / T V R V l l q t g + / L W 2 9 k k z W r F c y v e Y 7 V U u 1 s 2 l 4 c L u I R r 8 O A G a v A A d W g A g x G 8 w h v M n J n z 7 n w 4 n 4 v R n L P c O Y c V O F + / a w S g 6 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L 5 v s F d f C 3 a d a i A y 6 S N / m h T Z l f f 4 = " > A A A C G X i c b V C 7 T g J B F L 2 L L 8 Q X a m k z E U y s y C 6 N l k Q b S 0 z k k c C G z A 5 3 Y c L s 7 G Z m 1 k g I H 2 E r X 2 N n b K 3 8 G B O H h U L A k 0 x y c s 6 9 u W d O k A i u j e t + O 7 m t 7 Z 3 d v f x + 4 e D w 6 P i k e H r W 1 H G q G D Z Y L G L V D q h G w S U 2 D D c C 2 4 l C G g U C W 8 H o f u 6 3 n l F p H s s n M 0 7 Q j + h A 8 p A z a q z U K n e D k L B y r 1 h y K 2 4 G s k m 8 J S n B E v V e 8 a f b j 1 k a o T R M U K 0 7 n p s Y f 0 K V 4 U z g t N B N N S a U j e g A O 5 Z K G q H 2 J 1 n c K b m y S p + E s b J P G p K p f z c m N N J 6 H A V 2 M q J m q N e 9 u f i f 1 0 l N e O t P u E x S g 5 I t D o W p I C Y m 8 7 + T P l f I j B h b Q p n i N i t h Q 6 o o M 7 a h l S s D R Z M h Z y / T V R V l l q t g + / L W 2 9 k k z W r F c y v e Y 7 V U u 1 s 2 l 4 c L u I R r 8 O A G a v A A d W g A g x G 8 w h v M n J n z 7 n w 4 n 4 v R n L P c O Y c V O F + / a w S g 6 A = = < / l a t e x i t > ✓ < l a t e x i t s h a 1 _ b a s e 6 4 = " x W m X r a 5 P 9 0 Y Y t 2 h G o 0 Q W 8 X W b 5 i s = " > A A A C G n i c b V D L T s J A F J 3 i C / G F u n T T C C a u S M t G l 0 Q 3 L j G R R w I N m Q 6 3 d M J 0 2 s z c G g n h J 9 z K 1 7 g z b t 3 4 M S Y O p Q s B T z L J y T n 3 5 p 4 5 f i K 4 R s f 5 t g p b 2 z u 7 e 8 X 9 0 s H h 0 f F J + f S s r e N U M W i x W M S q 6 1 M N g k t o I U c B 3 U Q B j X w B H X 9 8 v / A 7 z 6 A 0 j + U T T h L w I j q S P O C M o p G 6 1 T 6 G g L Q 6 K F e c m p P B 3 i R u T i o k R 3 N Q / u k P Y 5 Z G I J E J q n X P d R L 0 p l Q h Z w J m p X 6 q I a F s T E f Q M 1 T S C L Q 3 z f L O 7 C u j D O 0 g V u Z J t D P 1 7 8 a U R l p P I t 9 M R h R D v e 4 t x P + 8 X o r B r T f l M k k R J F s e C l J h Y 2 w v P m 8 P u Q K G Y m I I Z Y q b r D Y L q a I M T U U r V 0 a K J i F n L 7 N V F W S W q 2 T

6 c t f b 2 S T t e s 1 1 a u 5 j v d K 4 y 5 s r k g t y S a 6 J S 2 5 I g z y Q J m k R R g R 5 J W 9 k b s 2 t d + v D + l y O F q x 8 5 5 y s w P r 6 B e l y o b 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " x W m X r a

5 P 9 0 Y Y t 2 h G o 0 Q W 8 X W b 5 i s = " > A A A C G n i c b V D L T s J A F J 3 i C / G F u n T T C C a u S M t G l 0 Q 3 L j G R R w I N m Q 6 3 d M J 0 2 s z c G g n h J 9 z K 1 7 g z b t 3 4 M S Y O p Q s B T z L J y T n 3 5 p 4 5 f i K 4 R s f 5 t g p b 2 z u 7 e 8 X 9 0 s H h 0 f F J + f S s r e N U M W i x W M S q 6 1 M N g k t o I U c B 3 U Q B j X w B H X 9 8 v / A 7 z 6 A 0 j + U T T h L w I j q S P O C M o p G 6 1 T 6 G g L Q 6 K F e c m p P B 3 i R u T i o k R 3 N Q / u k P Y 5 Z G I J E J q n X P d R L 0 p l Q h Z w J m p X 6 q I a F s T E f Q M 1 T S C L Q 3 z f L O 7 C u j D O 0 g V u Z J t D P 1 7 8 a U R l p P I t 9 M R h R D v e 4 t x P + 8 X o r B r T f l M k k R J F s e C l J h Y 2 w v P m 8 P u Q K G Y m I I Z Y q b r D Y L q a I M T U U r V 0 a K J i F n L 7 N V F W S W q 2 T

6 c t f b 2 S T t e s 1 1 a u 5 j v d K 4 y 5 s r k g t y S a 6 J S 2 5 I g z y Q J m k R R g R 5 J W 9 k b s 2 t d + v D + l y O F q x 8 5 5 y s w P r 6 B e l y o b 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " x W m X r a

5 P 9 0 Y Y t 2 h G o 0 Q W 8 X W b 5 i s = " > A A A C G n i c b V D L T s J A F J 3 i C / G F u n T T C C a u S M t G l 0 Q 3 L j G R R w I N m Q 6 3 d M J 0 2 s z c G g n h J 9 z K 1 7 g z b t 3 4 M S Y O p Q s B T z L J y T n 3 5 p 4 5 f i K 4 R s f 5 t g p b 2 z u 7 e 8 X 9 0 s H h 0 f F J + f S s r e N U M W i x W M S q 6 1 M N g k t o I U c B 3 U Q B j X w B H X 9 8 v / A 7 z 6 A 0 j + U T T h L w I j q S P O C M o p G 6 1 T 6 G g L Q 6 K F e c m p P B 3 i R u T i o k R 3 N Q / u k P Y 5 Z G I J E J q n X P d R L 0 p l Q h Z w J m p X 6 q I a F s T E f Q M 1 T S C L Q 3 z f L O 7 C u j D O 0 g V u Z J t D P 1 7 8 a U R l p P I t 9 M R h R D v e 4 t x P + 8 X o r B r T f l M k k R J F s e C l J h Y 2 w v P m 8 P u Q K G Y m I I Z Y q b r D Y L q a I M T U U r V 0 a K J i F n L 7 N V F W S W q 2 T

6 c t f b 2 S T t e s 1 1 a u 5 j v d K 4 y 5 s r k g t y S a 6 J S 2 5 I g z y Q J m k R R g R 5 J W 9 k b s 2 t d + v D + l y O F q x 8 5 5 y s w P r 6 B e l y o b 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " x W m X r a

5 P 9 0 Y Y t 2 h G o 0 Q W 8 X W b 5 i s = " > A A A C G n i c b V D L T s J A F J 3 i C / G F u n T T C C a u S M t G l 0 Q 3 L j G R R w I N m Q 6 3 d M J 0 2 s z c G g n h J 9 z K 1 7 g z b t 3 4 M S Y O p Q s B T z L J y T n 3 5 p 4 5 f i K 4 R s f 5 t g p b 2 z u 7 e 8 X 9 0 s H h 0 f F J + f S s r e N U M W i x W M S q 6 1 M N g k t o I U c B 3 U Q B j X w B H X 9 8 v / A 7 z 6 A 0 j + U T T h L w I j q S P O C M o p G 6 1 T 6 G g L Q 6 K F e c m p P B 3 i R u T i o k R 3 N Q / u k P Y 5 Z G I J E J q n X P d R L 0 p l Q h Z w J m p X 6 q I a F s T E f Q M 1 T S C L Q 3 z f L O 7 C u j D O 0 g V u Z J t D P 1 7 8 a U R l p P I t 9 M R h R D v e 4 t x P + 8 X o r B r T f l M k k R J F s e C l J h Y 2 w v P m 8 P u Q K G Y m I I Z Y q b r D Y L q a I M T U U r V 0 a K J i F n L 7 N V F W S W q 2 T

6 c t f b 2 S T t e s 1 1 a u 5 j v d K 4 y 5 s r k g t y S a 6 J S 2 5 I g z y Q J m k R R g R 5 J W 9 k b s 2 t d + v D + l y O F q x 8 5 5 y s w P r 6 B e l y o b 0 = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " H 9 c 4 l 

v 8 s c a h Y Y u N 6 N z G l + N K d b D c = " > A A A C G H i c b V D L T s J A F L 3 1 i f h C X b p p B B N X p G W j S 6 I b l 5 j I I 4 G G T I d b G J l O m 5 m p k T T 8 g 1 v 5 G n f G r T s / x s S h d C H g S S Y 5 O e f e 3 D P H j z l T 2 n G + r Y 3 N r e 2 d 3 c J e c f / g 8 O i 4 d H L a U l E i K T Z p x C P Z 8 Y l C z g Q 2 N d M c O 7 F E E v o c 2 / 7 4 b u 6 3 n 1 E q F o l H P Y n R C 8 l Q s I B R o o 3 U q v T i E a v 0 S 2 W n 6 m S w 1 4 m b k z L k a P R L P 7 1 B R J M Q h a a c K N V 1 n V h 7 K Z G a U Y 7 T Y i 9 R G B M 6 J k P s G i p I i M p L s 7 R T + 9 I o A z u I p H l C 2 5 n 6 d y M l o V K T 0 D e T I d E j t e r N x f + 8 b q K D G y 9 l I k 4 0 C r o 4 F C T c 1 p E 9 / 7 o 9 Y B K p 5 h N D C J X M Z L X p i E h C t S l o 6 c p Q E t M L f Z k u q y i y X E X T l 7 v a z j p p 1 a q u U 3 U f a u X 6 b d 5 c A c 7 h A q 7 A h W u o w z 0 0 o A k U n u A V 3 m B m z a x 3 6 8 P 6 X I x u W P n O G S z B + v o F M 8 G g 1 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " H 9 c 4 l v 8 s c a h Y Y u N 6 N z G l + N K d b D c = " > A A A C G H i c b V D L T s J A F L 3 1 i f h C X b p p B B N X p G W j S 6 I b l 5 j I I 4 G G T I d b G J l O m 5 m p k T T 8 g 1 v 5 G n f G r T s / x s S h d C H g S S Y 5 O e f e 3 D P H j z l T 2 n G + r Y 3 N r e 2 d 3 c J e c f / g 8 O i 4 d H L a U l E i K T Z p x C P Z 8 Y l C z g Q 2 N d M c O 7 F E E v o c 2 / 7 4 b u 6 3 n 1 E q F o l H P Y n R C 8 l Q s I B R o o 3 U q v T i E a v 0 S 2 W n 6 m S w 1 4 m b k z L k a P R L P 7 1 B R J M Q h a a c K N V 1 n V h 7 K Z G a U Y 7 T Y i 9 R G B M 6 J k P s G i p I i M p L s 7 R T + 9 I o A z u I p H l C 2 5 n 6 d y M l o V K T 0 D e T I d E j t e r N x f + 8 b q K D G y 9 l I k 4 0 C r o 4 F C T c 1 p E 9 / 7 o 9 Y B K p 5 h N D C J X M Z L X p i E h C t S l o 6 c p Q E t M L f Z k u q y i y X E X T l 7 v a z j p p 1 a q u U 3 U f a u X 6 b d 5 c A c 7 h A q 7 A h W u o w z 0 0 o A k U n u A V 3 m B m z a x 3 6 8 P 6 X I x u W P n O G S z B + v o F M 8 G g 1 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " H 9 c 4 l v 8 s c a h Y Y u N 6 N z G l + N K d b D c = " > A A A C G H i c b V D L T s J A F L 3 1 i f h C X b p p B B N X p G W j S 6 I b l 5 j I I 4 G G T I d b G J l O m 5 m p k T T 8 g 1 v 5 G n f G r T s / x s S h d C H g S S Y 5 O e f e 3 D P H j z l T 2 n G + r Y 3 N r e 2 d 3 c J e c f / g 8 O i 4 d H L a U l E i K T Z p x C P Z 8 Y l C z g Q 2 N d M c O 7 F E E v o c 2 / 7 4 b u 6 3 n 1 E q F o l H P Y n R C 8 l Q s I B R o o 3 U q v T i E a v 0 S 2 W n 6 m S w 1 4 m b k z L k a P R L P 7 1 B R J M Q h a a c K N V 1 n V h 7 K Z G a U Y 7 T Y i 9 R G B M 6 J k P s G i p I i M p L s 7 R T + 9 I o A z u I p H l C 2 5 n 6 d y M l o V K T 0 D e T I d E j t e r N x f + 8 b q K D G y 9 l I k 4 0 C r o 4 F C T c 1 p E 9 / 7 o 9 Y B K p 5 h N D C J X M Z L X p i E h C t S l o 6 c p Q E t M L f Z k u q y i y X E X T l 7 v a z j p p 1 a q u U 3 U f a u X 6 b d 5 c A c 7 h A q 7 A h W u o w z 0 0 o A k U n u A V 3 m B m z a x 3 6 8 P 6 X I x u W P n O G S z B + v o F M 8 G g 1 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " H 9 c 4 l v 8 s c a h Y Y u N 6 N z G l + N K d b D c = " > A A A C G H i c b V D L T s J A F L 3 1 i f h C X b p p B B N X p G W j

O i 4 d H L a U l E i K T Z p x C P Z 8 Y l C z g Q 2 N d M c O 7 F E E v o c 2 / 7 4 b u 6 3 n 1 E q F o l H P Y n R C 8 l Q s I B R o o 3 U q v T i E a v 0 S 2 W n 6 m S w 1 4 m b k z L k a P R L P 7 1 B R J M Q h a a c K N V 1 n V h 7 K Z G a U Y 7 T Y i 9 R G B M 6 J k P s G i p I i M p L s 7 R T + 9 I o A z u I p H l C 2 5 n 6 d y M l o V K T 0 D e T I d E j t e r N x f + 8 b q K D G y 9 l I k 4 0 C r o 4 F C T c 1 p E 9 / 7 o 9 Y B K p 5 h N D C J X M Z L X p i E h C t S l o 6 c p Q E t M L f Z k u q y i y X E X T l 7 v a z j p p 1 a q u U 3 U f a u X 6 b d 5 c A c 7 h A q 7 A h W u o w z 0 0 o A k U n u A V 3 m B m z a x 3 6 8 P 6 X I x u W P n O G S z B + v o F M 8 G g 1 A = = < / l a t e x i t >

N:

< l a t e x i t s h a 1 _ b a s e 6 4 = " P 8 o q S / z s c e I C K 4 9 8 5

T k O 8 r f 6 t p Y = " > A A A C F X i c b V C 7 T s M w F L 0 p r 1 J e B U Y W i x a J q U q 6 w F j B w o S K o A + p j S r H d V K r j h P Z D q K K + g m s 9 G v Y E C s z H 4 O E m 2 a g L U e y d H T O v b r H x 4 s 5 U 9 q 2 v 6 3 C x u b W 9 k 5 x t 7 S 3 f 3 B 4 V D 4 + a a s o k Y S 2 S M Q j 2 f W w o p w J 2 t J M c 9 q N J c W h x 2 n H G 9 / O / c 4 z l Y p F 4 k l P Y u q G O B D M Z w R r I z 1 W 7 6 u D c s W u 2 R n Q O n F y U o E c z U H 5 p z + M S B J S o Q n H S v U c O 9 Z u i q V m h N N p q Z 8 o G m M y x g H t G S p w S J W b Z l G n 6 M I o Q + R H 0 j y h U a b + 3 U h x q N Q k 9 M x k i P V I r X p z 8 T + v l 2 j / 2 k 2 Z i B N N B V k c 8 h O O d I T m / 0 Z D J i n R f G I I J p K Z r I i M s M R E m 3 a W r g Q S x y N G X q b L K h V Z r p L p y 1 l t Z 5 2 0 6 z X H r j k P 9 U r j J m + u C G d w D p f g w B U 0 4 A 6 a 0 A I C A b z C G 8 y s m f V u f V i f i 9 G

C l e + c w h K s r 1 + M P Z 9 n < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " P 8 o q S / z s c e I C K 4 9 8 5

T k O 8 r f 6 t p Y = " > A A A C F X i c b V C 7 T s M w F L 0 p r 1 J e B U Y W i x a J q U q 6 w F j B w o S K o A + p j S r H d V K r j h P Z D q K K + g m s 9 G v Y E C s z H 4 O E m 2 a g L U e y d H T O v b r H x 4 s 5 U 9 q 2 v 6 3 C x u b W 9 k 5 x t 7 S 3 f 3 B 4 V D 4 + a a s o k Y S 2 S M Q j 2 f W w o p w J 2 t J M c 9 q N J c W h x 2 n H G 9 / O / c 4 z l Y p F 4 k l P Y u q G O B D M Z w R r I z 1 W 7 6 u D c s W u 2 R n Q O n F y U o E c z U H 5 p z + M S B J S o Q n H S v U c O 9 Z u i q V m h N N p q Z 8 o G m M y x g H t G S p w S J W b Z l G n 6 M I o Q + R H 0 j y h U a b + 3 U h x q N Q k 9 M x k i P V I r X p z 8 T + v l 2 j / 2 k 2 Z i B N N B V k c 8 h O O d I T m / 0 Z D J i n R f G I I J p K Z r I i M s M R E m 3 a W r g Q S x y N G X q b L K h V Z r p L p y 1 l t Z 5 2 0 6 z X H r j k P 9 U r j J m + u C G d w D p f g w B U 0 4 A 6 a 0 A I C A b z C G 8 y s m f V u f V i f i 9 G

C l e + c w h K s r 1 + M P Z 9 n < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " P 8 o q S / z s c e I C K 4 9 8 5

T k O 8 r f 6 t p Y = " > A A A C F X i c b V C 7 T s M w F L 0 p r 1 J e B U Y W i x a J q U q 6 w F j B w o S K o A + p j S r H d V K r j h P Z D q K K + g m s 9 G v Y E C s z H 4 O E m 2 a g L U e y d H T O v b r H x 4 s 5 U 9 q 2 v 6 3 C x u b W 9 k 5 x t 7 S 3 f 3 B 4 V D 4 + a a s o k Y S 2 S M Q j 2 f W w o p w J 2 t J M c 9 q N J c W h x 2 n H G 9 / O / c 4 z l Y p F 4 k l P Y u q G O B D M Z w R r I z 1 W 7 6 u D c s W u 2 R n Q O n F y U o E c z U H 5 p z + M S B J S o Q n H S v U c O 9 Z u i q V m h N N p q Z 8 o G m M y x g H t G S p w S J W b Z l G n 6 M I o Q + R H 0 j y h U a b + 3 U h x q N Q k 9 M x k i P V I r X p z 8 T + v l 2 j / 2 k 2 Z i B N N B V k c 8 h O O d I T m / 0 Z D J i n R f G I I J p K Z r I i M s M R E m 3 a W r g Q S x y N G X q b L K h V Z r p L p y 1 l t Z 5 2 0 6 z X H r j k P 9 U r j J m + u C G d w D p f g w B U 0 4 A 6 a 0 A I C A b z C G 8 y s m f V u f V i f i 9 G

C l e + c w h K s r 1 + M P Z 9 n < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " P 8 o q S / z s c e I C K 4 9 8 5 

T k O 8 r f 6 t p Y = " > A A A C F X i c b V C 7 T s M w F L 0 p r 1 J e B U Y W i x a J q U q 6 w F j B w o S K o A + p j S r H d V K r j h P Z D q K K + g m s 9 G v Y E C s z H 4 O E m 2 a g L U e y d H T O v b r H x 4 s 5 U 9 q 2 v 6 3 C x u b W 9 k 5 x t 7 S 3 f 3 B 4 V D 4 + a a s o k Y S 2 S M Q j 2 f W w o p w J 2 t J M c 9 q N J c W h x 2 n H G 9 / O / c 4 z l Y p F 4 k l P Y u q G O B D M Z w R r I z 1 W 7 6 u D c s W u 2 R n Q O n F y U o E c z U H 5 p z + M S B J S o Q n H S v U c O 9 Z u i q V m h N N p q Z 8 o G m M y x g H t G S p w S J W b Z l G n 6 M I o Q + R H 0 j y h U a b + 3 U h x q N Q k 9 M x k i P V I r X p z 8 T + v l 2 j / 2 k 2 Z i B N N B V k c 8 h O O d I T m / 0 Z D J i n R f G I I J p K Z r I i M s M R E m 3 a W r g Q S x y N G X q b L K h V Z r p L p y 1 l t Z 5 2 0 6 z X H r j k P 9 U r j J m + u C G d w D p f g w B U 0 4 A 6 a 0 A I C A b z C G 8 y s m f V u f V i f i 9 G C l e + c

= " > A A A C F 3 i c b V A 9 T w J B F H z n J + I X a m m z k Z h Y k T s a r Q y J j S U m H p A A I X v L O 9 i w t 3 f Z 3 T P i h d 9 g K 7 / G z t h a + m N M X A 4 K A S f Z Z D L z X t 7 s B I n g 2 r j u t 7 O x u b W 9 s 1 v Y K + 4 f H B 4 d l 0 5 O G z p O F U O f x S J W r Y B q F F y i b 7 g R 2 E o U 0 i g Q 2 A x G d z O / + Y R K 8 1 g + m n G C 3 Y g O J A 8 5 o 8 Z K f i c I y U u v V H Y r b g 6 y T r w F K c M C 9 V 7 p p 9 O P W R q h N E x Q r d u e m 5 h u R p X h T O C k 2 E k 1 J p S N 6 A D b l k o a o e

5 m e d g J u b R K n 4 S x s k 8 a k q t / N z I a a T 2 O A j s Z U T P U q 9 5 M / M 9 r p y a 8 6 W Z c J q l B y e a H w l 

Q Q E 5 P Z z 0 m f K 2 R G j C 2 h T H G b l b A h V Z Q Z 2 8 / S l Y G i y Z C z 5 8 m y i j L P V b R 9 e a v t

= " > A A A C F 3 i c b V A 9 T w J B F H z n J + I X a m m z k Z h Y k T s a r Q y J j S U m H p A A I X v L O 9 i w t 3 f Z 3 T P i h d 9 g K 7 / G z t h a + m N M X A 4 K A S f Z Z D L z X t 7 s B I n g 2 r j u t 7 O x u b W 9 s 1 v Y K + 4 f H B 4 d l 0 5 O G z p O F U O f x S J W r Y B q F F y i b 7 g R 2 E o U 0 i g Q 2 A x G d z O / + Y R K 8 1 g + m n G C 3 Y g O J A 8 5 o 8 Z K f i c I y U u v V H Y r b g 6 y T r w F K c M C 9 V 7 p p 9 O P W R q h N E x Q r d u e m 5 h u R p X h T O C k 2 E k 1 J p S N 6 A D b l k o a o e

5 m e d g J u b R K n 4 S x s k 8 a k q t / N z I a a T 2 O A j s Z U T P U q 9 5 M / M 9 r p y a 8 6 W Z c J q l B y e a H w l 

Q Q E 5 P Z z 0 m f K 2 R G j C 2 h T H G b l b A h V Z Q Z 2 8 / S l Y G i y Z C z 5 8 m y i j L P V b R 9 e a v t

= " > A A A C F 3 i c b V A 9 T w J B F H z n J + I X a m m z k Z h Y k T s a r Q y J j S U m H p A A I X v L O 9 i w t 3 f Z 3 T P i h d 9 g K 7 / G z t h a + m N M X A 4 K A S f Z Z D L z X t 7 s B I n g 2 r j u t 7 O x u b W 9 s 1 v Y K + 4 f H B 4 d l 0 5 O G z p O F U O f x S J W r Y B q F F y i b 7 g R 2 E o U 0 i g Q 2 A x G d z O / + Y R K 8 1 g + m n G C 3 Y g O J A 8 5 o 8 Z K f i c I y U u v V H Y r b g 6 y T r w F K c M C 9 V 7 p p 9 O P W R q h N E x Q r d u e m 5 h u R p X h T O C k 2 E k 1 J p S N 6 A D b l k o a

o e 5 m e d g J u b R K n 4 S x s k 8 a k q t / N z I a a T 2 O A j s Z U T P U q 9 5 M / M 9 r p y a 8 6 W Z c J q l B y e a H w l Q 

Q E 5 P Z z 0 m f K 2 R G j C 2 h T H G b l b A h V Z Q Z 2 8 / S l Y G i

= " > A A A C F 3 i c b V A 9 T w J B F H z n J + I X a m m z k Z h Y k T s a r Q y J j S U m H p A A I X v L O 9 i w t 3 f Z 3 T P i h d 9 g K 7 / G z t h a + m N M X A 4 K A S f Z Z D L z X t 7 s B I n g 2 r j u t 7 O x u b W 9 s 1 v Y K + 4 f H B 4 d l 0 5 O G z p O F U O f x S J W r Y B q F F y i b 7 g R 2 E o U 0 i g Q 2 A x G d z O / + Y R K 8 1 g + m n G C 3 Y g O J A 8 5 o 8 Z K f i c I y U u v V H Y r b g 6 y T r w F K c M C 9 V 7 p p 9 O P W R q h N E x Q r d u e m 5 h u R p X h T O C k 2 E k 1 J p S N 6 A D b l k o a

o e 5 m e d g J u b R K n 4 S x s k 8 a k q t / N z I a a T 2 O A j s Z U T P U q 9 5 M / M 9 r p y a 8 6 W Z c J q l B y e a H w l Q 

Q E 5 P Z z 0 m f K 2 R G j C 2 h T H G b l b A h V Z Q Z 2 8 / S l Y G i

Graph VAE:

< l a t e x i t s h a 1 _ b a s e 6 4 = " R j

/ F A i J J Y Y u o n I d + c d 7 G H S a D n U c = " > A A A C G 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i T d 6 E o q I r q s Y B / S h j K Z T t q h k 0 m Y m Y g l 9 C v c 2 q 9 x J 2 5 d + D G C k z Q L 2 3 p g 4 H D O v d w z x 4 s 4 U 9 q 2 v 6 3 C 2 v r G 5 l Z x u 7 S z u 7 d / U D 4 8 a q k w l o Q 2 S c h D 2 f G w o p w J 2 t R M c 9 q J J M W B x 2 n b G 9 + k f v u Z S s V C 8 a g n E X U D P B T M Z w R r I z 3 d S R y N U O v 6 t l + u 2 F U 7 A 1 o l T k 4 q k K P R L / / 0 B i G J A y o 0 4 V i p r m N H 2 k 2 w 1 I x w O i 3 1 Y k U j T M Z 4 S L u G C h x Q 5 S Z Z 4 C k 6 M 8 o A + a E 0 T 2 i U q X 8 3 E h w o N Q k 8 M x l g P V L L X i r + 5 3 V j 7 V + 6 C R N R r K k g 8 0 N + z J E O U f p 7 N G C S E s 0 n h m A i m c m K y A h L T L T p a O H K M G 2 G k Z f p o k p F l q t k + n K W 2 1 k l r V r V s a v O Q 6 1 S v 8 q b K 8 I J n M I 5 O H A B d b i H B j S B Q A C v 8 A Y z a 2 a 9 W x / W 5 3 y 0 Y O U 7 x 7 A A 6 + s X a C + h 9 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R j / F A i J J Y Y u o n I d + c d 7 G H S a D n U c = " > A A A C G 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i T d 6 E o q I r q s Y B / S h j K Z T t q h k 0 m Y m Y g l 9 C v c 2 q 9 x J 2 5 d + D G C k z Q L 2 3 p g 4 H D O v d w z x 4 s 4 U 9 q 2 v 6 3 C 2 v r G 5 l Z x u 7 S z u 7 d / U D 4 8 a q k w l o Q 2 S c h D 2 f G w o p w J 2 t R M c 9 q J J M W B x 2 n b G 9 + k f v u Z S s V C 8 a g n E X U D P B T M Z w R r I z 3 d S R y N U O v 6 t l + u 2 F U 7 A 1 o l T k 4 q k K P R L / / 0 B i G J A y o 0 4 V i p r m N H 2 k 2 w 1 I x w O i 3 1 Y k U j T M Z 4 S L u G C h x Q 5 S Z Z 4 C k 6 M 8 o A + a E 0 T 2 i U q X 8 3 E h w o N Q k 8 M x l g P V L L X i r + 5 3 V j 7 V + 6 C R N R r K k g 8 0 N + z J E O U f p 7 N G C S E s 0 n h m A i m c m K y A h L T L T p a O H K M G 2 G k Z f p o k p F l q t k + n K W 2 1 k l r V r V s a v O Q 6 1 S v 8 q b K 8 I J n M I 5 O H A B d b i H B j S B Q A C v 8 A Y z a 2 a 9 W x / W 5 3 y 0 Y O U 7 x 7 A A 6 + s X a C + h 9 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R j / F A i J J Y Y u o n I d + c d 7 G H S a D n U c = " > A A A C G 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i T d 6 E o q I r q s Y B / S h j K Z T t q h k 0 m Y m Y g l 9 C v c 2 q 9 x J 2 5 d + D G C k z Q L 2 3 p g 4 H D O v d w z x 4 s 4 U 9 q 2 v 6 3 C 2 v r G 5 l Z x u 7 S z u 7 d / U D 4 8 a q k w l o Q 2 S c h D 2 f G w o p w J 2 t R M c 9 q J J M W B x 2 n b G 9 + k f v u Z S s V C 8 a g n E X U D P B T M Z w R r I z 3 d S R y N U O v 6 t l + u 2 F U 7 A 1 o l T k 4 q k K P R L / / 0 B i G J A y o 0 4 V i p r m N H 2 k 2 w 1 I x w O i 3 1 Y k U j T M Z 4 S L u G C h x Q 5 S Z Z 4 C k 6 M 8 o A + a E 0 T 2 i U q X 8 3 E h w o N Q k 8 M x l g P V L L X i r + 5 3 V j 7 V + 6 C R N R r K k g 8 0 N + z J E O U f p 7 N G C S E s 0 n h m A i m c m K y A h L T L T p a O H K M G 2 G k Z f p o k p F l q t k + n K W 2 1 k l r V r V s a v O Q 6 1 S v 8 q b K 8 I J n M I 5 O H A B d b i H B j S B Q A C v 8 A Y z a 2 a 9 W x / W 5 3 y 0 Y O U 7 x 7 A A 6 + s X a C + h 9 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R j / F A i J J Y Y u o n I d + c d 7 G H S a D n U c = " > A A A C G 3 i c b V D L S s N A F L 2 p r 1 p f V Z d u B o v g q i T d 6 E o q I r q s Y B / S h j K Z T t q h k 0 m Y m Y g l 9 C v c 2 q 9 x J 2 5 d + D G C k z Q L 2 3 p g 4 H D O v d w z x 4 s 4 U 9 q 2 v 6 3 C 2 v r G 5 l Z x u 7 S z u 7 d / U D 4 8 a q k w l o Q 2 S c h D 2 f G w o p w J 2 t R M c 9 q J J M W B x 2 n b G 9 + k f v u Z S s V C 8 a g n E X U D P B T M Z w R r I z 3 d S R y N U O v 6 t l + u 2 F U 7 A 1 o l T k 4 q k K P R L / / 0 B i G J A y o 0 4 V i p r m N H 2 k 2 w 1 I x w O i 3 1 Y k U j T M Z 4 S L u G C h x Q 5 S Z Z 4 C k 6 M 8 o A + a E 0 T 2 i U q X 8 3 E h w o N Q k 8 M x l g P V L L X i r + 5 3 V j 7 V + 6 C R N R r K k g 8 0 N + z J E O U f p 7 N G C S E s 0 n h m A i m c m K y A h L T L T p a O H K M G 2 G k Z f p o k p F l q t k + n K W 2 1 k l r V r V s a v O Q 6 1 S v 8 q b K 8 I J n M I 5 O H A B d b i H B j S B Q A C v 8 A Y z a 2 a 9 W x / W 5 3 y 0 Y O U 7 x 7 A A 6 + s X a C + h 9 w = = < / l a t e x i t >

zn < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " C 9 S l w x F R Y j X v y Z M t I q w d E O k D q c 4 = " > A A A C L n i c b V D L S s N A F J 3 4 r N V q q 0 s 3 g 6 3 g q i T d 6 L L g x m U F + 4 A 0 l M l 0 0 g 6 d R 5 i Z i D X k M 9 z q L / g 1 g g t x 6 2 c 4 T b s w r Q c u H M 6 5 l 3 s 4 Y c y o N q 7 7 6 W x t 7 + z u 7

Z c O y o d H l e O T a u 2 0 p 2 W i M O l i y a Q a h E g T R g X p G m o Y G c S K I B 4 y 0 g 9 n t w u / / 0 i U p l I 8 m H l M A o 4 m g k Y U I 2 M l v 5 E O w w g + Z y P R G F X r b t P N A T e J t y J 1 s E J n V H M q w 7 H E C S f C Y I a 0 9 j 0 3 N k G K l K G Y k a w 8 T D S J E Z 6 h C f E t F Y g T H a R 5 5 g x e W m U M I 6 n s C A N z 9 e 9 F i r j W c x 7 a T Y 7 M V K 9 7 C / E / z 0 9 M d B O k V M S J I Q I v H 0 U J g 0 b C R Q F w T B X B h s 0 t Q V h R m x X i K V I I G 1 t T 4 c t E o X h K 8 V N W V I n I c x X E k G d l W 6 C 3 X t c m 6 b W a n t v 0 7 l v 1 d m t V Z Q m c g w t w B T x w D d r g D n R A F 2 A g w Q t 4 B W / O u / P h f D n f y 9 U t Z 3 V z B g p w f n 4 B S 5 q o i Q = = < / l a t e x i t > learned | flexible graph < l a t e x i t s h a 1 _ b a s e 6 4 = " O e u r A 8 L Y F 6 x t H Z M o e t h n d P N Q X W s = " > A A A C P 3 i c b V D L T g I x F O 3 4 R B Q F 3 e m m E U x c k R k 2 s i R x 4 x I T e S Q w I Z 1 y B x o 6 n U n b M Z B x E r / G r f 6 C n + E X u D N u 3 V k G F g K e p M n J O f f k 3 h 4 v 4 k x p 2 / 6 w t r Z 3 d v f 2 c w f 5 w 6 P C 8 U m x d N p W Y S w p t G j I Q 9 n 1 i A L O B L Q 0 0 x y 6 k Q Q S e B w 6 3 u R 2 7 n c e Q S o W i g c 9 i 8 A N y E g w n 1 G i j T Q o n n M g U s A Q V 5 4 q 2 O c w Z S a J R 5 J E 4 0 G x b F f t D H i T O E t S R k s 0 B y W r 0 B + G N A 5 A a M q J U j 3 H j r S b E K k Z 5 Z D m + 7 G C i N A J G U H P U E E C U G 6 S f S L F V 0 Y Z Y j + U 5 g m N M / V v I i G B U r P A M 5 M B 0 W O 1 7 s 3 F / 7 x e r P 2 6 m z A R x R o E X S z y Y 4 5 1 i O e N 4 C G T Q D W f G U K o Z O Z W T M d E E q p N b y t b s l o Y n a a r K o j s r h X R C 9 K 8 K d B Z r 2 u T t G t V x 6 4 6 9 7 V y o 7 6 s M o c u 0 C W 6 R g 6 6 Q Q 1 0 h 5 q o h S h 6 R i / o F b 1 Z 7 9 a n 9 W V 9 L 0 a 3 r G X m D K 3 A + v k F G 3 i u 5 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " O e u r A 8 L Y F 6 x t H Z M o e t h n d P N Q X W s = " > A A A C P 3 i c b V D L T g I x F O 3 4 R B Q F 3 e m m E U x c k R k 2 s i R x 4 x I T e S Q w I Z 1 y B x o 6 n U n b M Z B x E r / G r f 6 C n + E X u D N u 3 V k G F g K e p M n J O f f k 3 h 4 v 4 k x p 2 / 6 w t r Z 3 d v f 2 c w f 5 w 6 P C 8 U m x d N p W Y S w p t G j I Q 9 n 1 i A L O B L Q 0 0 x y 6 k Q Q S e B w 6 3 u R 2 7 n c e Q S o W i g c 9 i 8 A N y E g w n 1 G i j T Q o n n M g U s A Q V 5 4 q 2 O c w Z S a J R 5 J E 4 0 G x b F f t D H i T O E t S R k s 0 B y W r 0 B + G N A 5 A a M q J U j 3 H j r S b E K k Z 5 Z D m + 7 G C i N A J G U H P U E E C U G 6 S f S L F V 0 Y Z Y j + U 5 g m N M / V v I i G B U r P A M 5 M B 0 W O 1 7 s 3 F / 7 x e r P 2 6 m z A R x R o E X S z y Y 4 5 1 i O e N 4 C G T Q D W f G U K o Z O Z W T M d E E q p N b y t b s l o Y n a a r K o j s r h X R C 9 K 8 K d B Z r 2 u T t G t V x 6 4 6 9 7 V y o 7 6 s M o c u 0 C W 6 R g 6 6 Q Q 1 0 h 5 q o h S h 6 R i / o F b 1 Z 7 9 a n 9 W V 9 L 0 a 3 r G X m D K 3 A + v k F G 3 i u 5 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " O e u r A 8 L Y F 6 x t H Z M o e t h n d P N Q X W s = " > A A A C P 3 i c b V D L T g I x F O 3 4 R B Q F 3 e m m E U x c k R k 2 s i R x 4 x I T e S Q w I Z 1 y B x o 6 n U n b M Z B x E r / G r f 6 C n + E X u D N u 3 V k G F g K e p M n J O f f k 3 h 4 v 4 k x p 2 / 6 w t r Z 3 d v f 2 c w f 5 w 6 P C 8 U m x d N p W Y S w p t G j I Q 9 n 1 i A L O B L Q 0 0 x y 6 k Q Q S e B w 6 3 u R 2 7 n c e Q S o W i g c 9 i 8 A N y E g w n 1 G i j T Q o n n M g U s A Q V 5 4 q 2 O c w Z S a J R 5 J E 4 0 G x b F f t D H i T O E t S R k s 0 B y W r 0 B + G N A 5 A a M q J U j 3 H j r S b E K k Z 5 Z D m + 7 G C i N A J G U H P U E E C U G 6 S f S L F V 0 Y Z Y j + U 5 g m N M / V v I i G B U r P A M 5 M B 0 W O 1 7 s 3 F / 7 x e r P 2 6 m z A R x R o E X S z y Y 4 5 1 i O e N 4 C G T Q D W f G U K o Z O Z W T M d E E q p N b y t b s l o Y n a a r K o j s r h X R C 9 K 8 K d B Z r 2 u T t G t V x 6 4 6 9 7 V y o 7 6 s M o c u 0 C W 6 R g 6 6 Q Q 1 0 h 5 q o h S h 6 R i / o F b 1 Z 7 9 a n 9 W V 9 L 0 a 3 r G X m D K 3 A + v k F G 3 i u 5 w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " O e u r A 8 L Y F 6 x t H Z M o e t h n d P N Q X W s = " > A A A C P 3 i c b V D L T g I x F O 3 4 R B Q F 3 e m m E U x c k R k 2 s i R x 4 x I T e S Q w I Z 1 y B x o 6 n U n b M Z B x E r / G r f 6 C n + E X u D N u 3 V k G F g K e p M n J O f f k 3 h 4 v 4 k x p 2 / 6 w t r Z 3 d v f 2 c w f 5 w 6 P C 8 U m x d N p W Y S w p t G j I Q 9 n 1 i A L O B L Q 0 0 x y 6 k Q Q S e B w 6 3 u R 2 7 n c e Q S o W i g c 9 i 8 A N y E g w n 1 G i j T Q o n n M g U s A Q V 5 4 q 2 O c w Z S a J R 5 J E 4 0 G x b F f t D H i T O E t S R k s 0 B y W r 0 B + G N A 5 A a M q J U j 3 H j r S b E K k Z 5 Z D m + 7 G C i N A J G U H P U E E C U G 6 S f S L F V 0 Y Z Y j + U 5 g m N M / V v I i G B U r P A M 5 M B 0 W O 1 7 s 3 F / 7 x e r P 2 6 m z A R x R o E X S z y Y 4 5 1 i O e N 4 C G T Q D W f G U K o Z O Z W T M d E E q p N b y t b s l o Y n a a r K o j s r h X R C 9 K 8 K d B Z r 2 u T t G t V x 6 4 6 9 7 V y o 7 6 s M o c u 0 C W 6 R g 6 6 Q Q 1 0 h 5 q o h S h 6 R i / o F b 1 Z 7 9 a n 9 W V 9 L 0 a 3 r G X m D K 3 A + v k F G 3 i u 5 w = = < / l a t e x i t > p✓(z|c) = Q N n=1 p✓(zn|zpa(n), cpa(n),n) < l a t e x i t s h a 1 _ b a s e 6 4 = " L 1 G q 4 r F G V X / t I 7 6 o D 2 w H M o m V 1 h 8 = " > A A A C s X i c b V H N b h M x E P Y u f y V Q S O G C x M U k Q k q k K t q F Q 3 u J F I k L J 1 Q k 0 h Z l w + J 1 Z h M r X q 9 l z 1 Y N i 9 + A h + C 1 e A 2 e A G 9 S p C T t S J Y + f 9 9 8 M 6 O Z T E t h M Y r + B O G 9 + w 8 e P j p 4 3 H r y 9 P D Z 8 / b R i 3 N b V o b D m J e y N J c Z s y C F g j E K l H C p D b A i k 3 C R L T 8 0 + s U V G C t K 9 Q V X G q Y F m y u R C 8 7 Q U 2 n 7 l 0 4 T X A A y 2 k s K h o s s r 3 8 4 + p P + / 3 D X p 0 O a a F P O 0 l o N Y / e t / u T o X S Y v b x s b I k G 4 R l P U m r m e 6 r v j r a r 7 4 j F V r p + 2 u 9 E g W g e 9 D e I b 0 B 1 1 h v o 3 I e Q s P Q o O k 1 n J q w I U c s m s n c S R x m n N D A o u w b W S y o J m f M n m M P F Q s Q L s t F 6 v z d G 3 n p n R v D T + K a R r d t t R s 8 L a V Z H 5 z G Z u u 6 8 1 5 F 3 a p M L 8 d F o L p S s E x T e N 8 k p S L G l z A z o T B j j K l Q e M G + F n p X z B D O P o L 7 X T Z W 6 Y X g h + 7 X Z Z U O u 5 d s i s c C 2 / w H h / X b f B + b t B H A 3 i z 3 F 3 d E o 2 c U B e k w 7 p k Z i c k B H 5 S M 7 I m H D y N 3 g V v A k 6 4 f v w a / g 9 z D a p Y X D j e U l 2 I l z + A 8 z U 2 L E = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f Q p v 7 J Z t p f P 0 X L b E F H y r b C r R / b 0 = " > A A A C s X i c b V H N b h M x E P Y u f y V Q m s I F i Y t J h J R I V b Q L B 3 q J F I k L J 1 Q k 0 h Z l w + J 1 Z h M r X q 9 l z 6 K G x W / A W / B Q i N f g C f A m R U r S j m T p 8 / f N N z O a y b Q U F q P o T x D e u X v v / o O D h 6 1 H j w + f H L W P n 5 7 b s j I c x r y U p b n M m A U p F I x R o I R L b Y A V m Y S L b P m u 0 S + + g b G i V J 9 w p W F a s L k S u e A M P Z W 2 f + o 0 w Q U g o 7 2 k Y L j I 8 v q 7 o z / o / w 9 3 f T q k i T b l L K 3 V M H Z f 6 g + O 3 m b y 8 r a x I R K E K z R F r Z n r q b 4 7 2 a q 6 L 5 5 Q 5 f p p u x s N o n X Q m y C + B t 1 R Z 6 h / / W 4 N z 9 L j 4 D C Z l b w q Q C G X z N p J H G m c 1 s y g 4 B J c K 6 k s a M a X b A 4 T D x U r w E 7 r 9 d o c f e W Z G c 1 L 4 5 9 C u m a 3 H T U r r F 0 V m c 9 s 5 r b 7 W k P e p k 0 q z E + n t V C 6 Q l B 8 0 y i v J M W S N j e g M 2 G A o 1 x 5 w L g R f l b K F 8 w w j v 5 S O 1 3 m h u m F 4 F d u l w W 1 n m u H z A r X 8 g u M 9 9 d 1 E 5 y / H s T R I P 4 Y d 0 e n Z B M H 5 A X p k B 6 J y V s y I u / J G R k T T v 4 G z 4 O X Q S d 8 E 3 4 O v 4 b Z J j U M r j 3 P y E 6 E y 3 + T Z N n S < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f Q p v 7 J Z t p f P 0 X L b E F H y r b C r R / b 0 = " > A A A C s X i c b V H N b h M x E P Y u f y V Q m s I F i Y t J h J R I V b Q L B 3 q J F I k L J 1 Q k 0 h Z l w + J 1 Z h M r X q 9 l z 6 K G x W / A W / B Q i N f g C f A m R U r S j m T p 8 / f N N z O a y b Q U F q P o T x D e u X v v / o O D h 6 1 H j w + f H L W P n 5 7 b s j I c x r y U p b n M m A U p F I x R o I R L b Y A V m Y S L b P m u 0 S + + g b G i V J 9 w p W F a s L k S u e A M P Z W 2 f + o 0 w Q U g o 7 2 k Y L j I 8 v q 7 o z / o / w 9 3 f T q k i T b l L K 3 V M H Z f 6 g + O 3 m b y 8 r a x I R K E K z R F r Z n r q b 4 7 2 a q 6 L 5 5 Q 5 f p p u x s N o n X Q m y C + B t 1 R Z 6 h / / W 4 N z 9 L j 4 D C Z l b w q Q C G X z N p J H G m c 1 s y g 4 B J c K 6 k s a M a X b A 4 T D x U r w E 7 r 9 d o c f e W Z G c 1 L 4 5 9 C u m a 3 H T U r r F 0 V m c 9 s 5 r b 7 W k P e p k 0 q z E + n t V C 6 Q l B 8 0 y i v J M W S N j e g M 2 G A o 1 x 5 w L g R f l b K F 8 w w j v 5 S O 1 3 m h u m F 4 F d u l w W 1 n m u H z A r X 8 g u M 9 9 d 1 E 5 y / H s T R I P 4 Y d 0 e n Z B M H 5 A X p k B 6 J y V s y I u / J G R k T T v 4 G z 4 O X Q S d 8 E 3 4 O v 4 b Z J j U M r j 3 P y E 6 E y 3 + T Z N n S < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 7 b d o C r n T b 7 y Q X t + 8 f 8 4 1 4 j f h 5 7 0 = " > A A A C s X i c b V F N b x M x E P U u X y V Q m s I F i Y t p h J R I V b Q L B 3 q p V I k L J 1 Q k 0 h Z l w + J 1 Z h M r t t e y Z 1 H D 4 n / A H + R v 8 A v w p k F K 0 o 5 k 6 f k 9 P 8 / o T W G k c J g k f 6 L 4 3 v 0 H D x / t P e 4 8 e b r / 7 K B 7 + P z C V b X l M O K V r O x V w R x I o W G E A i V c G Q t M F R I u i 8 W H V r / 8 A d a J S n / B p Y G J Y j M t S s E Z B i r v / j Z 5 h n N A R v u Z Y j g v y u a n p 7 / o / w v 3 A 3 p K M 2 O r a d 7 o 0 9 R / a z 5 5 e p c p y J v G l s g Q r t G q x j D f 1 w N / v P H r r n h M t R / k 3 V 4 y T F Z F b 4 N 0 D X p k X e f 5 Y b S f T S t e K 9 D I J X N u n C Y G J w 2 z K L g E 3 8 l q B 4 b x B Z v B O E D N F L h J s 4 r N 0 z e B m d K y s u F o p C t 2 0 9 E w 5 d x S F e F l O 7 f b 1 V r y L m 1 c Y 3 k y a Y Q 2 N Y L m N 4 3 K W l K s a L s D O h U W O M p l A I x b E W a l f M 4 s 4 x g 2 t d V l Z p m Z C 3 7 t t 1 n Q q 7 m 2 y E L 5 T g g w 3 Y 3 r N r h 4 O 0 y T Y f o 5 7 Z 2 d r K P c I 6 / I E e m T l L w n Z + Q j O S c j w s n f 6 G X 0 O j q K 3 8 V f 4 + 9

x c f M 0 j t a e F 2 S r 4 s U / I j r X A g = = < / l a t e x i t > zpa(n)

< l a t e x i t s h a 1 _ b a s e 6 4 = " B x E D P I 6 F 5 q 5 N y M l o 1

3 J x E G j a F 0 s = " > A A A C P 3 i c b Z D L S g M x F I Y z X m u 9 t b r T T b A K u i k z 3 e h S c O O y g m 2 F t p R M e q Y N J p k h O S P W Y c C n c a u v 4 G P 4 B O 7 E r T v T y 8 K q P w R + v n M O 5 + Q P E y k s + v 6 b t 7 C 4 t L y y W l g r r m 9 s b m 2 X y j t N G 6 e G Q 4 P H M j Y 3 I b M g h Y Y G C p R w k x h g K p T Q C m 8 v x v X W H R g r Y n 2 N o w S 6 i g 2 0 i A R n 6 F C v t H e Y d c K I P u S 9 r I N w j 0 Z l C c u P 9 U l + 2 C t V / K o / E f 1 r g p m p k J n q v b K 3 2 e n H P F W g k U t m b T v w E + x m z K D g E v J i J 7 W Q M H 7 L B t B 2 V j M F t p t N P p H T I 0 f 6 N I q N e x r p h P 6 c y J i y d q R C 1 6 k Y D u 3 v 2 h j + V 2 u n G J 1 1 M 6 G T F E H z 6 a I o l R R j O k 6 E 9 o U B j n L k D O N G u F s p H z L D O L r c 5 r Y M D E u G g t / n 8 x T 0 5 K 4 5 G K q 8 6 A I M f s f 1 1 z R r 1 c C v B l e 1 y n l t F m W B 7 J M D c k w C c k r O y S W p k w b h 5 J E 8 k W f

y 4 r 1 6 7 9 6 H 9 z l t X f B m M 7 t k T t 7 X N w T L r 2 I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B x E D P I 6 F 5 q 5 N y M l o 1

3 J x E G j a F 0 s = " > A A A C P 3 i c b Z D L S g M x F I Y z X m u 9 t b r T T b A K u i k z 3 e h S c O O y g m 2 F t p R M e q Y N J p k h O S P W Y c C n c a u v 4 G P 4 B O 7 E r T v T y 8 K q P w R + v n M O 5 + Q P E y k s + v 6 b t 7 C 4 t L y y W l g r r m 9 s b m 2 X y j t N G 6 e G Q 4 P H M j Y 3 I b M g h Y Y G C p R w k x h g K p T Q C m 8 v x v X W H R g r Y n 2 N o w S 6 i g 2 0 i A R n 6 F C v t H e Y d c K I P u S 9 r I N w j 0 Z l C c u P 9 U l + 2 C t V / K o / E f 1 r g p m p k J n q v b K 3 2 e n H P F W g k U t m b T v w E + x m z K D g E v J i J 7 W Q M H 7 L B t B 2 V j M F t p t N P p H T I 0 f 6 N I q N e x r p h P 6 c y J i y d q R C 1 6 k Y D u 3 v 2 h j + V 2 u n G J 1 1 M 6 G T F E H z 6 a I o l R R j O k 6 E 9 o U B j n L k D O N G u F s p H z L D O L r c 5 r Y M D E u G g t / n 8 x T 0 5 K 4 5 G K q 8 6 A I M f s f 1 1 z R r 1 c C v B l e 1 y n l t F m W B 7 J M D c k w C c k r O y S W p k w b h 5 J E 8 k W f

y 4 r 1 6 7 9 6 H 9 z l t X f B m M 7 t k T t 7 X N w T L r 2 I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B x E D P I 6 F 5 q 5 N y M l o 1

3 J x E G j a F 0 s = " > A A A C P 3 i c b Z D L S g M x F I Y z X m u 9 t b r T T b A K u i k z 3 e h S c O O y g m 2 F t p R M e q Y N J p k h O S P W Y c C n c a u v 4 G P 4 B O 7 E r T v T y 8 K q P w R + v n M O 5 + Q P E y k s + v 6 b t 7 C 4 t L y y W l g r r m 9 s b m 2 X y j t N G 6 e G Q 4 P H M j Y 3 I b M g h Y Y G C p R w k x h g K p T Q C m 8 v x v X W H R g r Y n 2 N o w S 6 i g 2 0 i A R n 6 F C v t H e Y d c K I P u S 9 r I N w j 0 Z l C c u P 9 U l + 2 C t V / K o / E f 1 r g p m p k J n q v b K 3 2 e n H P F W g k U t m b T v w E + x m z K D g E v J i J 7 W Q M H 7 L B t B 2 V j M F t p t N P p H T I 0 f 6 N I q N e x r p h P 6 c y J i y d q R C 1 6 k Y D u 3 v 2 h j + V 2 u n G J 1 1 M 6 G T F E H z 6 a I o l R R j O k 6 E 9 o U B j n L k D O N G u F s p H z L D O L r c 5 r Y M D E u G g t / n 8 x T 0 5 K 4 5 G K q 8 6 A I M f s f 1 1 z R r 1 c C v B l e 1 y n l t F m W B 7 J M D c k w C c k r O y S W p k w b h 5 J E 8 k W f

y 4 r 1 6 7 9 6 H 9 z l t X f B m M 7 t k T t 7 X N w T L r 2 I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " B x E D P I 6 F 5 q 5 N y M l o 1 3 J x E G j a F 0 s = " > Rezende et al., 2014; Goyal et al., 2017; Vikram et al., 2018; Webb et al., 2018) . These dependencies provide empirical priors, learned priors that are conditioned on other latent variables. With M latent dimensions, the full prior takes the following auto-regressive form:

M g h Y Y G C p R w k x h g K p T Q C m 8 v x v X W H R g r Y n 2 N o w S 6 i g 2 0 i A R n 6 F C v t H e Y d c K I P u S 9 r I N w j 0 Z l C c u P 9 U l + 2 C t V / K o / E f 1 r g p m p k J n q v b K 3 2 e n H P F W g k U t m b T v w E + x m z K D g E v J i J 7 W Q M H 7 L B t B 2 V j M F t p t N P p H T I 0 f 6 N I q N e x r p h P 6 c y J i y d q R C 1 6 k Y D u 3 v 2 h j + V 2 u n G J 1 1 M 6 G T F E H z 6 a I o l R R j O k 6 E 9 o U B j n L k D O N G u F s p H z L D O L r c 5 r Y M D E u G g t / n 8 x T 0 5 K 4 5 G K q 8 6 A I M f s f 1 1 z R r 1 c C v B l

EQUATION

where z pa(m) denotes the vector of latent variables constituting the parents of z m . Each conditional distribution can be parameterized by deep networks that output the parameters of distributions, e.g. mean and variance of a Gaussian. While these conditional distributions may be relatively simple, the marginal empirical prior, p θ (z m ) = ∫ p θ (z m z pa(m) )p θ (z pa(m) )dz pa(m) , can be arbitrarily complex. By using more flexible priors, models with latent dependency structure are less restrictive in their latent representations, hopefully capturing more information and enabling a better fit to the data.

With the added latent dependencies in the model, the independence assumption in the approximate posterior is even less valid in this setting. While normalizing flows (Rezende & Mohamed, 2015) offers one technique for overcoming the mean field assumption, a separate line of work has investigated the use of structured approximate posteriors, particularly in the context of models with empirical priors (Johnson et al., 2016) . This technique introduces dependencies between the dimensions of the approximate posterior, often mirroring the dependency structure of the latent variable model. An explanation for this was provided by Marino et al. (2018) : optimizing the approximate posterior requires knowledge of the prior, which is especially relevant in models with empirical priors where the prior can vary with the data. Ladder VAE (Sønderby et al., 2016) incorporates these prior dependencies by using a structured approximate posterior of the form

EQUATION

Unlike the mean field approximate posterior, which conditions each dimension only on the data example, x, the distributions in Eq. ( 4) account for latent dependencies by conditioning on samples from the parent variables. Ladder VAE performs this conditioning by reusing the empirical prior during inference, forming the approximate posterior by combining a "bottom-up" recognition distribution and the "top-down" prior.

While Eqs. ( 3) and (4) permit separate latent dependencies for each individual latent dimension, the dimensions are typically partitioned into a set of nodes, with dimensions within each node sharing the same parents. This improves computational efficiency by allowing priors and approximate posteriors within each node to be calculated in parallel. Using z n to denote latent node n of N and z pa(n) to denote the concatenation of its parent nodes, we can write the ELBO (Eq. ( 2)) as

EQUATION

Note that the KL divergence term in the ELBO can no longer be evaluated analytically, now requiring a sampling-based estimate of the expectation (Kingma & Welling, 2014) . While this can lead to higher variance in ELBO estimates and the resulting gradients, models with latent dependencies still tend to empirically outperform models with independence assumptions (Burda et al., 2016; Sønderby et al., 2016) . However, by increasing the number of nodes, the burden of devising a suitable dependency structure falls upon the experimental practitioner. This is non-trivial, as the structure may depend on the data and other model hyperparameters, such as the number of layers in the deep networks, non-linearities, latent distributions, etc. Rather than relying on pre-defined fully-connected structures (Kingma et al., 2016) or chain structures (Sønderby et al., 2016) , we seek to automatically learn the latent dependency structure as part of the variational optimization process. A comparison of these approaches is visualized in Fig. 1 .

3. VARIATIONAL OPTIMIZATION OF LATENT STRUCTURES:

Unlike the model parameters (φ, θ), which are optimized over a continuous domain, the latent dependency structure is discrete, without a clear ordering. The discrete nature of the latent space's topological structure introduces discontinuities in the optimization landscape, complicating the learning process. Fortunately, unlike the related setting of neural architecture search (Zoph & Le, 2016) , there is only a finite number of possible dependency structures over a fixed number of latent dimensions: In a directed graphical model, a fully-connected directed acyclic graph (DAG) models all possible dependencies. In this model, an ordering is induced over the latent nodes, and the parents of node n (of N ) are given as z pa(n) = {z n+1 , . . . , z N }.1 Thus, to learn an appropriate latent dependency structure, we can maintain all dependencies in a fully-connected DAG, modifying their presence or absence during training. This is accomplished by introducing a set of binary dependency gates (Section 3.1). We convert discrete optimization over dependency structures into a continuous optimization problem by parameterizing these gates as samples from Bernoulli distributions, then learning the distribution parameters (Section 3.3). These gating distributions induce an additional lower bound on L, which becomes tight when the distribution converges to a delta function, yielding a single, optimized dependency structure (Section 3.2). Indeed, we observe this process empirically, with the learned dependency structures outperforming their predefined counterparts (Section 4).

3.1. GATED DEPENDENCIES:

To control the dependency structure of the model, we introduce a set of binary global variables, c = {c i,j } i,j , which gate the latent dependencies. The element c i,j denotes the gate variable from z i to z j (i > j), specifying the presence or absence of this latent dependency. Because each element of c takes values in {0, 1}, dependencies can be preserved or removed simply through (broadcasted) element-wise multiplication with the corresponding parent nodes. Removing a dependency entails multiplying the corresponding input parent node by 0. Each possible latent dependency structure can now be expressed through its corresponding value of c.

Each fixed latent dependency structure, c ′ , induces a separate latent variable model p θ (x, z, c) = p θ (x z, c)p θ (z c)δ c,c ′ , where δ ⋅,⋅ is the Kronecker delta, which effectively selects a single structure. Similar to Eq. ( 3), the prior on the latent variables can now be expressed as

EQUATION

where c pa(n),n denotes the gate variables associated with the dependencies between node z n and its parents, z pa(n) . Note that z pa(n) denotes the set of all possible parents of node z n in the fullyconnected DAG, i.e. z pa(n) = {z n+1 , . . . , z N }. To give a concrete example of the gating procedure, Sample c using Eq. ( 9) and determine z pa(n) for each z n based on the sampled structure.

4::

For each node, compute q φ (z n x, z pa(n) ) using Eq. ( 7).

5::

Sample z from q φ (z x) using Eq. ( 6) and compute p θ (x z).

6::

Update θ, φ, µ based on the gradients derived from Eq. ( 8). 7: until Convergence.

3.2. LEARNING STRUCTURE BY INDUCING AN ADDITIONAL LOWER BOUND:

The formulation in Section 3.1 provides the form of the model for a particular configuration of the latent dependency structure. Finding the optimal structure corresponds to a discrete optimization over all values of c, potentially optimizing the model parameters of each possible configuration. To avoid this intractable procedure, we place a distribution over c, then directly optimize the parameters of this distribution to arrive at a single, learned latent dependency structure. Specifically, we treat each c i,j as an independent random variable, sampled from a Bernoulli distribution with mean µ i,j , i.e. c i,j ∼ p(c i,j ) = B(µ i,j ). We denote the set of these Bernoulli means as µ. Introducing this distribution allows us to express the following additional lower bound on L, derived in Appendix A:

EQUATION

where L c is the ELBO for a particular value of dependency gating variables. Thus, L can be interpreted as the expected ELBO under the distribution of dependency structures induced by p(c), which we estimate by sampling c ∼ p(c) and evaluating L c . We note that L is not a proper variational bound, as it is not guaranteed to recover the marginal likelihood if the approximate posterior matches the true posterior. Rather, optimizing L provides a method for learning the latent structure.

For fixed parameters (φ, θ), the optimal L w.r.t. µ is a δ-distribution at the MAP configuration, c * , yielding L = L = L c * . This is because L c * is always greater than or equal to the expected ELBO over all dependency gates, L. In practice, we jointly optimize φ, θ, and µ. While this non-convex optimization procedure may result in local optima, we find this empirically works well, with p(c) converging to a fixed distribution (Fig. 3 ). Thus, by the end of training, we are effectively optimizing the ELBO for a single dependency structure. The training procedure is outlined in Algorithm 1.

3.3. LEARNING THE DISCRETE GATING VARIABLE DISTRIBUTIONS:

For a given latent dependency structure, gradients for the parameters θ and φ can be estimated using Monte Carlo samples and the reparameterization trick (Kingma & Welling, 2014; Rezende et al., 2014) . To obtain gradients for the gate means, µ, we make use of recent advances in differentiating through discrete operations (Maddison et al., 2017; Jang et al., 2017) , allowing us to differentiate through the sampling of the dependency gating variables, c. Specifically, we recast the gating variables using the Gumbel-Softmax estimator from Jang et al. ( 2017), re-expressing c i,j as:

EQUATION

where 1 and 2 are i.i.d samples drawn from a Gumbel(0, 1) distribution and τ is a temperature parameter. The Gumbel-Softmax distribution is differentiable for τ > 0, allowing us to estimate the derivative ∂ci,j ∂µi,j . For large values of τ , we obtain smoothed versions of c, essentially interpolating between different dependency structures. As τ → 0, we recover binary values for c, yielding the desired discrete sampling of dependency structures at the cost of high-variance gradient estimates. Thus, we anneal τ during training to learn the dependency gate means, µ, eventually arriving at the discrete setting. 

4. EVALUATION:

We evaluate the proposed latent dependency learning approach on three benchmark datasets: MNIST (Lecun et al., 1998; Larochelle & Murray, 2011 ), Omniglot (Lake et al., 2013 ), and CIFAR-10 (Krizhevsky, 2009) . After discussing the experimental setup in Section 4.1, we provide a set of qualitative experiments in Section 4.2 to gain insight into the learning process, hyper-parameter selection, and the nature of the inferred structures. In Section 4.3, we provide quantitative comparisons with common predefined latent dependency structures on benchmark datasets. Additional results on training robustness and latent space embeddings can be found in Appendix C.

4.1. EXPERIMENTAL SETUP:

To provide a fair comparison, the encoders of all structured methods use the same MLP architecture with batch normalization (Ioffe & Szegedy, 2015) and ReLU non-linearities (Nair & Hinton, 2010) in all experiments.2 Decoder structures are the reverse of the encoders. Likewise, the number of latent dimensions is the same in all models and experiments (M = 80). As discussed in Section 3.1, all latent dependencies are modeled by non-linear MLPs as well.

For MNIST and Omniglot, we binarize the data and model p θ (x z) as a Bernoulli distribution, using a sigmoid non-linearity on the output layer to produce the mean of this distribution. For CIFAR-10, we model p θ (x z) with a Gaussian density, with mean and log-variance predicted by sigmoid and linear functions, respectively. Further implementation details, including the model architectures and training criteria, can be found in Appendix B.

4.2. QUALITATIVE ANALYSIS:

We first explore the structure learning process. As described in Section 3.2 and Appendix A, optimizing L w.r.t. the dependency gating means, µ, should push this lower bound toward L. Thus, µ should converge to either 0 or 1, yielding a fixed, final latent dependency structure. In Fig. 3 , we visualize this process during training on MNIST. The model has N = 5 nodes and a total latent dimension of M = 80 (i.e., N ′ = M N = 16 dimensions per node). As shown in Fig. 3a , the gating means converge in practice, with 3 out of 10 edges removed and the rest retained. The resulting static dependency structure is visualized in Fig. 3b . We observed that the learned structure is stable across training runs with different seeds for parameter initialization and mini-batch sampling, supporting the hypothesis that the inferred structure indeed depends on the model parameterization and the dataset.

required to understand this behavior at a theoretical level. A full visualization of the training process is provided in Appendix C.1.

5. DISCUSSION:

Performance vs. Speed/Memory. As shown in Fig. 4 , the number of latent nodes can significantly impact the performance of our model. While allowing more complex dependency structures through a low M N -ratio is typically beneficial, it also has an adverse effect on the training time and memory consumption. Fortunately, the ability to freely select this ratio allows a simple adaption to the available processing power, hardware constraints, and application scenarios.

Optimization Order. It is worth noting that the learning process optimizes the model parameters (c, φ, θ) in a clear temporal order. While the latent structure, governed by c, converges during the first ≈ 200 epochs (Fig. 3a ), it takes over 10× as long until the variational and generative parameters (φ, θ) converge. There is no external force enforcing this behaviour, indicating that the loss can initially most easily be decreased by limiting the latent structure to the complexity prescribed by the observed training data.

Performance Improvement over Fully-Connected VAEs. There is an intricate relationship between fully-connected graphs vs. learned graphs along one axis and between prior structures vs. posterior structures along a separate axis: FC-VAEs model all conditional latent dependencies and are thus potentially more expressive and flexible than other latent dependency structures. It is therefore somewhat surprising that the learned latent structures in Graph VAE consistently outperform the FC-VAE baseline. We speculate that this may be due to difficulties in optimization, which is a known problem in hierarchical latent variable models (Bowman et al., 2016; Burda et al., 2016) .

Graph VAE and FC-VAE both contain the same hypothesis space of possible models that can be learned. If all dependencies are needed, Graph VAE could set all dependency gate parameters to 1.

Likewise, if a latent dependency was unnecessary, FC-VAE could set all of the model parameters in that dependency to 0. However, this would require many coordinated steps along multiple parameter dimensions. It is plausible that the benefit of learning the dependency structure may stem from the ability to alter the optimization landscape, using the dependency gates to move through the model parameter space more coarsely and rapidly. The resulting latent dependency structures may thus be less expressive, but easier to optimize. While only an intuition, this hypothesis is also in line with the observations and results in our experiments with fully-connected encoder graphs and learned decoder graphs, where the theoretically more flexible FC-encoder is outperformed by our parameter sharing approach. Follow-up work will be required to test this intuition.

6. CONCLUSION:

We presented a novel method for structure learning in latent variable models, which uses dependency gating variables, together with a modified objective and discrete differentiation techniques, to effectively transform discrete structure learning into a continuous optimization problem. In our experiments, the learned latent dependency structures improve the performance of latent variable models over comparable baselines with predefined dependency structures. The approach presented here provides directions for further research in structure learning for other tasks, including undirected graphical models, time-series models, and discriminative models.

A LOWER BOUND DERIVATION:

Introducing the distribution over dependency gating variables, p(c) = B(µ), modifies the evidence lower bound (ELBO) from Eq. ( 2), as we now have an additional set of random variables over which to marginalize. To see this, we can start by re-expressing Eq. ( 2) as L = E q φ (z x) [log p θ (x, z)] -E q φ (z x) [log q φ (z x)] .

(10) p θ (x, z) can be expressed as a marginalization over the gating variables, effectively averaging over an ensemble of models with a distribution of dependency structures: 

Plugging this into Eq. ( 10):

EQUATION

Using Jensen's inequality, we bring the log inside of the expectation, E p(c) [⋅], yielding L ≥ L = E q φ (z x) E p(c) [log p θ (x, z c)] -E q φ (z x) [log q φ (z x)] ,

where L is a lower bound on L. Swapping the order of expectation, we rewrite L as L = E p(c) E q φ (z x) [log p θ (x, z c)] -E q φ (z x) [log q φ (z x)] ,

and because the second term is independent of p(c), we can include both terms inside of a single outer expectation:

L = E p(c) E q φ (z x) [log p θ (x, z c) -log q φ (z x)]

= E p(c) E q φ (z x) [log p θ (x z, c)] -KL(q φ (z x) p θ (z c))

EQUATION

where we have defined L c as the ELBO for a given dependency structure. Note that L is not a proper variational bound, as it cannot recover the marginal log likelihood. Rather, L allows us to optimize the distribution over gating variables and, thus, the model structure. When we arrive at a fixed structure, we will be optimizing the variational bound for that particular dependency structure.

To see this, note that the bound in Eq. 13 becomes tight when p(c) is any fixed distribution, in which the dependency gating means, µ, are all either 1 or 0. Plugging in a delta distribution for p(c) at a particular configuration, c ′ , i.e. p(c) = δ c,c ′ , we have L = E q φ (z x) log E δ c,c ′ [p θ (x, z c)] -E q φ (z x) [log q φ (z x)]

= E q φ (z x) E δ c,c ′ [log p θ (x, z c)] -E q φ (z x) [log q φ (z x)] = L.

(16)

Intuitively, this is simply the case of a latent variable model with predefined, fixed dependencies. By optimizing L w.r.t. µ, we hope to collapse p(c) to a delta distribution at the optimal (MAP) configuration, c * , because

EQUATION

Effectively, we can search for the dependency structure with the highest ELBO by optimizing the distribution parameters of p(c). Although this optimization procedure may arrive at locally optimal dependency structures, our hope is that these learned structures will still perform better than an arbitrary, predefined dependency structure.

C ADDITIONAL RESULTS:

C.1 ROBUSTNESS OF LOG-LIKELIHOODS In Fig. 5 , we report the averaged test log-likelihoods and associated standard deviations of Graph VAE and our baselines at different epochs. All calculations are based on 5 independent runs. 

:

We follow convention(Dayan et al., 1995;Rezende et al.,

2014; Sønderby et al., 2016), with parent nodes having a larger index than their children.

We implement classic VAEs using a more complex encoder to match the number of parameters of the structured methods. All baselines use the same or more parameters than Graph VAE.

Figures and Tables:

S 6 I b l 5 j I I 4 G G T I d b G J l O m 5 m p k T T 8 g 1 v 5 G n f G r T s / x s S h d C H g S S Y 5 O e f e 3 D P H j z l T 2 n G + r Y 3 N r e 2 d 3 c J e c f / g 8

w h K s r 1 + M P Z 9 n < / l a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " q v / I 6 f m A z e t H s c B r 4 S e D O e e Z + L U

r J N G t e K 5 F e + h W q 7 d L p o r w D l c w B V 4 c A 0 1 u I c 6 + M C A w y u 8 w d S Z O u / O h / M 5 H 9 1 w F j t n s A T n 6 x f K k 6 C f < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q v / I 6 f m A z e t H s c B r 4 S e D O e e Z + L U

r J N G t e K 5 F e + h W q 7 d L p o r w D l c w B V 4 c A 0 1 u I c 6 + M C A w y u 8 w d S Z O u / O h / M 5 H 9 1 w F j t n s A T n 6 x f K k 6 C f < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q v / I 6 f m A z e t H s c B r 4 S e D O e e Z + L U

y Z C z 5 8 m y i j L P V b R 9 e a v t r J N G t e K 5 F e + h W q 7 d L p o r w D l c w B V 4 c A 0 1 u I c 6 + M C A w y u 8 w d S Z O u / O h / M 5 H 9 1 w F j t n s A T n 6 x f K k 6 C f < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " q v / I 6 f m A z e t H s c B r 4 S e D O e e Z + L U

y Z C z 5 8 m y i j L P V b R 9 e a v t r J N G t e K 5 F e + h W q 7 d L p o r w D l c w B V 4 c A 0 1 u I c 6 + M C A w y u 8 w d S Z O u / O h / M 5 H 9 1 w F j t n s A T n 6 x f K k 6 C f < / l a t e x i t >

A A A C P 3 i c b Z D L S g M x F I Y z X m u 9 t b r T T b A K u i k z 3 e h S c O O y g m 2 F t p R M e q Y N J p k h O S P W Y c C n c a u v 4 G P 4 B O 7 E r T v T y 8 K q P w R + v n M O 5 + Q P E y k s + v6 b t 7 C 4 t L y y W l g r r m 9 s b m 2 X y j t N G 6 e G Q 4 P H M j Y 3 I b

Figure 1: Overview: Model Comparison. We show the graphical representations of (a) traditional latent variable models (VAE, ladder VAE) and (b) the proposed graph VAE. Solid lines denote generation, dashed lines denote inference, and the dotted area indicates the latent space governed by variational parameters φ and generative parameters θ. Both VAE and ladder VAE use a fixed graph structure with limited expressiveness (VAE: independent; ladder VAE: chain-structured). In contrast, graph VAE jointly optimizes a distribution over latent structures c and model parameters (φ,θ), allowing test-time sampling of a flexible, data-driven latent structure.

Figure 3: Structure Learning. In (a), we show training of the Bernoulli parameters µ i,j governing the distribution over graph structures in architecture space. All edges are color-coded and can be located in (b), where we show a random sample from the resulting steady-state distribution with the same color scheme.

Figure 4: Ablation Study on MNIST. For a fixed latent dimension M (colorcoded), we report the log-likelihood of all possible factorizations of node dimension N ′ (x-axis) and number of nodes N = M N ′ .

p θ (x, z) = p θ (x, z, c)dc = p θ (x, z c)p(c)dc = E p(c) [p θ (x, z c)] .

Figure 5: Comparison of Test-time Log-Likelihoods on MNIST. Models are trained 5 times individually, and test-time performances are evaluated every 300 epochs, with mean log-likelihood and standard deviation indicated by the colored bars.

Figure 6: TSNE-Visualization of Latent Embeddings on MNIST. We visualize embeddings of (a) Graph VAE, (b) VAE, and (c) the data itself.

Algorithm 1 Optimizing VAEs with Latent Dependency Structure Require: Data x, number of latent nodes N , number of dimensions per node N ′ .

Table content: <table/>

Peer review (iclr2019):
 This paper presents a VAE approach in which a dependency structure on the latent variable is learned during training. Specifically, a lower-triangular random binary matrix c is introduced, where c_{i,j} = 1 for i>j, indicates that z_i depends on z_j, where z is the latent vector. Each element of c is separately parametrized by a Bernoulli distribution whose means are optimized for during training, using the target .mathbb{E}_{p(c)}[.mathcal{L}_c] where .mathcal{L}_c indicates the ELBO for a particular instance of c. The resulting "Graph-VAE" scheme is shown to train models with improved marginal likelihood than a number of baselines for MNIST, Omniglot, and CIFAR-10. The core concept for this paper is good, the results are impressive, and the paper is, for the most part, easy to follow. Though I think a lot of people have been thinking about how to learn dependency structures in VAEs, I think this work is the first to clearly lay out a concrete approach for doing so. I thus think that even though this is not the most novel of papers, it is work which will be of significant interest to the ICLR community. However, the paper has a number of technical issues and I do not believe the paper is suitable for publication unless they are addressed, or at the vest least acknowledged. I further have some misgivings with the experiments and the explanations of some key elements of the method. Because of these issues, I think the paper falls below the acceptance threshold in its current form, but I think they could potentially be correctable during the rebuttal period and I will be very happy to substantially increase my score if they are; I feel this has the potential to be a very good paper that I would ultimately like to see published. %%% Lower bound %%% My first major concern is in the justification of the final approach (Eq 8), namely using a lower bound argument to move the p(c) term outside of the log. A target being a lower bound on something we care about is never in itself a justification for that target -- it just says that the resulting estimator is provably negatively biased. The arguments behind the use of lower bounds in conventional ELBOs are based on much more subtle arguments in terms of the bound becoming tight if we have good posterior approximations and implicit assumptions that the bound will behave similarly to the true marginal. The bound derived in A.1 of the current paper is instead almost completely useless and serves little purpose other than adding "mathiness" of the type discussed in https://arxiv.org/abs/1807.0334. Eq 8 is not a variational end-to-end target like you claim. It is never tight and will demonstrably behave very differently to the original target. To see why it will behave very differently, consider how the original and bound would combine two instances of c for the MNIST experiment, one corresponding to the MAP values of c in the final trained system, the other a value of c that has an ELBO which is, say, 10 nats lower. Using Eq 8, these will have similar contributions to the overall expectation and so a good network setup (i.e. theta and phi) is one which produces a decent ELBO for both. Under the original expectation, on the other hand, the MAP value of c corresponds to a setup that has many orders of magnitude higher probability and so the best network setup is the one that does well for the MAP value of c, with the other instance being of little importance. We thus see that the original target and the lower bound behave very differently for a given p(c). Thankfully, the target in Eq 8 is a potentially reasonable thing to do in its own right (maybe actually more so that the original formulation), because the averaging over c is somewhat spurious given you are optimizing its mean parameters anyway. It is easy to show that the "optimum" p(c) for a given (.theta,.phi) is always a delta function on the value of c which has the highest ELBO_c. As Fig 3 shows, the optimization of the parameters of p(c) practically leads to such a collapse. This is effectively desirable behavior given the overall aims and so averaging over values of c is from a modeling perspective actually a complete red herring anyway. It is very much possible that the training procedure represented by Eq 8 is (almost by chance) a good approach in terms of learning the optimal configuration for c, but if this is the case it needs to be presented as such, instead of using the current argument about putting a prior on c and constructing a second lower bound, which is a best dubious and misleading, and at worst complete rubbish. Ideally, the current explanations would be replaced by a more principled justification, but even just saying you tried Eq 8 and it worked well empirically would be a lot better than what is there at the moment. %%% Encoder dependency structure does not match the generative model %%% My second major concern is that the dependency structure used for the encoder is incorrect from the point of view of the generative model. Namely, a dependency structure on the prior does not induce the same dependency structure on the posterior. In general, just because z_1 and z_2 are independent, doesn*t mean that z_1 and z_2 are independent given x (see e.g. Bishop). Consequently, the encoder in your setup will be incapable of correctly representing the posterior implied by the generative model. This has a number of serious practical and theoretical knock-on effects, such as prohibiting the bound becoming tight, causing the encoder to indirectly impact the expressivity of the generative model etc. Note that this problem is not shared with the Ladder VAE, as there the Markovian dependency structure means produces a special case where the posterior and prior dependency structure is shared. As shown in https://arxiv.org/abs/1712.00287 (a critical missing reference more generally), it is actually possible to derive the dependency structure of the posterior from that of the prior. I think in your case their results imply that the encoder needs to be fully connected as the decoder can induce arbitrary dependencies between the latent variables. I am somewhat surprised that this has not had more of an apparent negative impact on the empirical results and I think at the very very least the paper needs to acknowledge this issue. I would recommend the authors run experiments using a fully connected encoder and the Graph-VAE decoder (and potentially also vice verse). Should this approach perform well, it would represent a more principled approach to replace the old on from a generative model perspective. Should it not, it would provide an empirical justification for what is, in essence, a different restriction to that of the learned prior structure: it is conceivably actually the case that these encoder restrictions induce the desired decoder behavior, but this is distinct to learning a particular dependency structure in the generative model. %%% Specifics of model and experiments %%% Though the paper is generally very easy to read, there as some key areas where the explanations are overly terse. In particular, the explanation surrounding the encoding was difficult to follow and it took me a while to establish exactly what was going on; I am still unsure how .tilde{.psi} and .hat{.psi} are combined. I think a more careful explanation here and a section giving more detail in the appendices would both help massively. I was not clear on exactly what was meant by the FC-VAE. I do not completely agree with the assertion that a standard VAE has independent latents. Though the typical choice that the prior is N(0,I) obviously causes the prior to have independent latents, as explained earlier, this does not mean the latents are independent in the posterior. Furthermore, the encoder implicitly incorporates these dependencies through its mean vector, even if it uses a diagonal covariance (which is usually rather small anyway). What is actually changed from this by the FC-VAE? Are you doing some kind of normalizing flow approach here? If so this needs proper explanation. Relatedly, I am also far from convinced by the arguments presented about why the FC-VAE does worse at the end of the experiments. VAEs attempt to maximize a marginal likelihood (through a surrogate target) and a model which makes no structural assumptions will generally have a lower marginal likelihood than one which makes the correct structural assumptions. It is thus perfectly reasonable that when you learn dependency structures, you will get a higher marginal likelihood than if you presume none. I thus find your arguments about local optima somewhat speculative and further investigation is required. %%% Experiments %%% Though certainly not terrible, I felt that the experimental evaluation of the work could have been better. The biggest issue I have is that no error bars are given for the results, so it is difficult to assess the robustness of the Graph-VAE. I think it would be good to add convergence plots with error bars to see how the performance varies with time and provide an idea of variability. More generally, the experiment section overall feels more terse and rushed than the rest of the paper, with some details difficult to find or potentially even straight up missing. Though Fig 3 is very nice, it would be nice to have additional plots seeing qualitatively what happens with the latent space. E.g. on average what proportion of the c tend to zero? Is the same dependency structure always learned? What do the dataset encodings look like? Are there noticeable qualitative changes in samples generated from the learned models? I would be perfectly happy for the paper to extend over the 8 pages to allow more results addressing these questions.