Title: VARIATIONAL HYPER RNN FOR SEQUENCE MODEL-ING

Abstract: In this work, we propose a novel probabilistic sequence model that excels at capturing high variability in time series data, both across sequences and within an individual sequence. Our method uses temporal latent variables to capture information about the underlying data pattern and dynamically decodes the latent information into modifications of weights of the base decoder and recurrent model. The efficacy of the proposed method is demonstrated on a range of synthetic and real-world sequential data that exhibit large scale variations, regime shifts, and complex dynamics.

1. INTRODUCTION:

Recurrent neural networks (RNNs) are the natural architecture for sequential data as they can handle variable-length input and output sequences. Initially invented for natural language processing, long short-term memory (LSTM; Hochreiter & Schmidhuber 1997) , gated recurrent unit (GRU; Cho et al. 2014) as well as the later attention-augmented versions (Vaswani et al., 2017) have found wide-spread successes from language modeling (Mikolov et al., 2010; Kiros et al., 2015; Jozefowicz et al., 2016) and machine translation (Bahdanau et al., 2014) to speech recognition (Graves et al., 2013) and recommendation systems (Wu et al., 2017) . However, RNNs use deterministic hidden states to process input sequences and model the system dynamics using a set of time-invariant weights, and they do not necessarily have the right inductive bias for time series data outside the originally intended domains.

Many natural systems have complex feedback mechanisms and numerous exogenous sources of variabilities. Observations from such systems would contain large variations both across sequences in a dataset as well as within any single sequence; the dynamics could be switching regimes drastically, and the noise process could also be heteroskedastic. To capture all these intricate patterns in RNN with deterministic hidden states and a fixed set of weights requires learning about the patterns, the subtle deviations from the patterns, the conditions under which regime transitions occur which is not always predictable. Outside of the deep learning literature, many time series models have been proposed to capture specific types of high variabilities. For instance, switching linear dynamical models (Ackerson & Fu, 1970; Ghahramani & Hinton, 1996; Murphy, 1998; Fox et al., 2009) aim to model complex dynamical systems with a set of simpler linear patterns. Conditional volatility models (Engle, 1982; Bollerslev, 1986) are introduced to model time series with heteroscedastic noise process whose noise level itself is a part of the dynamics. However, these models usually encode specific inductive biases in a hard way, and cannot learn different behaviors and interpolate among the learned behaviors as deep neural nets.

In this work, we propose a new class of neural recurrent latent variable model, called the variational hyper RNN (VHRNN), which can perform dynamic regime identification and re-identification dynamically at inference time. Our model captures complex time series without encoding a large number of patterns in static weights, but instead only encodes base dynamics that can be selected and adapted based on run time observations. Thus it can easily learn to express a rich set of behaviors including but not limited to the ones mentioned above. Our model can dynamically identify the underlying pattern, express uncertainty due to observation noise, lack of information, or model misspecification. As such, VHRNN can model complex patterns with fewer parameters; and when given lots of parameters, it generalizes better than previous methods.

The VHRNN is built upon the previous variational RNN (VRNN) models (Chung et al., 2015) and hypernetworks (Ha et al., 2016) . The VRNN models introduce stochastic latent variables at every time step, which are inferred using a variational recognition model. The overall model is trained by maximizing the evidence lower bound (ELBO). In VRNN, the latent variables capture the information in the stochastic hidden states and are then fed as input to the RNN and decoding model to produce reconstructed observations. While in our work, the latent variables are decoded to produce the RNN transition weights and observation projection weights in the style of hypernetworks (Ha et al., 2016) , i.e., dynamically generating the scaling and bias vectors to adjust the base weights of the RNN. We demonstrate that the proposed VHRNN model is better at capturing different types of variability on several synthetic as well as real-world time series datasets.

2. BACKGROUND AND RELATED WORK:

Variational Autoencoder Variational autoencoder (VAE) is one of the most popular unsupervised approaches to learning a compact representation from data (Kingma & Welling, 2013) . It uses a variational distribution q(z|x) to approximate the intractable posterior distribution of the latent variable z. With the use of variational approximation, it maximizes the evidence lower bound (ELBO) of the marginal log-likelihood of data L(x) = E q(z|x) [log p(x|z)] -D KL (q(z|x) p(z)) ≤ log p(x),

where p(z) is a prior distribution of z and D KL denotes the Kullback-Leibler (KL) divergence. The approximate posterior q(z|x) is usually formulated as a Gaussian with a diagonal covariance matrix.

Variational RNN for Sequential Data Variational autoencoders have demonstrated impressive performance on non-sequential data like images. Many following works (Bowman et al., 2015; Chung et al., 2015; Fraccaro et al., 2016; Luo et al., 2018) extend the domain of VAE models to sequential data. Among them, variational RNN (VRNN; Chung et al. 2015) further incorporate a latent variable at each time step into their models. A prior distribution conditioned on the contextual information and a variational posterior is proposed at each time step to optimize a step-wise variational lower bound. Sampled latent variables from the variational posterior are decoded into the observation at the current time step. The VHRNN model makes use of the same factorization of sequential data and joint distribution of latent variables as in VRNN. However, in VHRNN model, the latent variables also parameterize the weights for decoding and transition in RNN cell across time steps, giving the model more flexibility to deal with variations within and across sequences.

Importance Weighted Autoencoder and Filtering Variational Objective A parallel stream of work to improve latent variable models with variational inference study tighter bounds of the data's log-probability than ELBO. Importance Weighted Autoencoder (IWAE; Burda et al. 2016 ) estimates a different variational bound of the log-likelihood, which is provably tighter than ELBO. Filtering Variational Objective (FIVO; Maddison et al. 2017 ) exploits the temporal structure of sequential data and uses particle filtering to estimate the data log-likelihood. FIVO still computes a step-wise IWAE bound based on the sampled particles at each time step, but it shows better sampling efficiency and tightness than IWAE. We use FIVO as the objective to train and evaluate our models.

HyperNetworks Our model is motivated by HyperNetworks (Ha et al., 2016) which use one network to generate the parameters of another. The dynamic version of HyperNetworks can be applied to sequence data, but due to lack of latent variables, can only capture uncertainty in the output variables.

For discrete sequence data such as text, categorical output variables can model multi-model outputs very well; but on continuous time series with the typical Gaussian output variables, the model is much less capable at dealing with stochasticity. Furthermore, it does not allow straightforward interpretation of the model behaviour using the time-series of KL divergence as we do in Sec. 4. With the augmentation of latent variables, VHRNN is much more capable of modelling uncertainty.

It is worth noting that Bayesian HyperNetworks (Krueger et al., 2017) also have a latent variable in the context of Hypernetworks. However, the goal of Bayesian HypernNtwork is an improved version of Bayesian neural net to capture model uncertainty. The work of Krueger et al. (2017) has no recurrent structure and cannot be applied to sequential data. Furthermore, the use of normalizing flow dramatically limits the flexibility of the decoder architecture design, unlike in VHRNN. Dezfouli et al. (2019) recently proposed to learn a disentangled low-dimensional latent space such that samples from the latent space are used to generate parameters of an RNN model. Different latent variables could account for the behaviours of different subjects. In spite of similarity in combining RNN < l a t e x i t s h a 1 _ b a s e 6 4 = " Y S T f t 5 5 o k 6 X o L 1 J 2 W i N n I E A e m a I = " > A A A B 7 X i c b V D L S g N B E J y N r x h f U Y 9 e B o P g K e x G Q Y 8 B L z l G M A 9 I l j A 7 6 U 3 G z G O Z m R X C k n / w 4 k E R r / 6 P N / / G S b I H T S x o K K q 6 6 e 6 K E s 6 M 9 f 1 v r 7 C x u b W 9 U 9 w t 7 e 0 f H B 6

V j 0 / a R q W a Q o s q r n Q 3 I g Y 4 k 9 C y z H L o J h q I i D h 0 o s n d 3 O 8 8 g T Z M y Q c 7 T S A U Z C R Z z C i x T m r 3 l Y A R G Z Q r f t V f A K + T I C c V l K M 5 K H / 1 h 4 q m A q S l n B j T C / z E h h n R l l E O s 1 I / N Z A Q O i E j 6 D k q i Q A T Z o t r Z / j C K U M c K + 1 K W r x Q f 0 9 k R B g z F Z H r F M S O z a o 3 F / / z e q m N b 8 O M y S S 1 I O l y U Z x y b B W e v 4 6 H T A O 1 f O o I o Z q 5 W z E d E 0 2 o d Q G V X A j B 6 s v r p F 2 r B l f V 2 v 1 1 p d 7 I 4 y i i M 3 S O L l G A b l A d N V A T t R B F j + g Z v a I 3 T 3 k v 3 r v 3 s W w t e P n M K f o D 7 / M H k 0 2 P J Q = = < / l a t e x i t > zt < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 d W 0 K Q d j r n 4 / g p i S i 6 0 3 A H C Q O E s = " > A A A B + H i c b V D L S s N A F J 3 U V 6 2 P R l 2 6 G S y C q 5 J U Q Z c F N 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M 3 A h t y J e 4 c a G I W z / F n X / j t M 1 C W w 8 M H M 6 5 l 3 v m B I n g G h z n 2 y p t b e / s 7 p X 3 K w e H R 8 d V + + S 0 q + N U U d a h s Y h V P y C a C S 5 Z B z g I 1 k 8 U I 1 E g W C + Y 3 i / 8 3 h N T m s f y E W Y J 8 y I y l j z k l I C R f L s 6 j A h M g j C b 5 3 6 G I f f t m l N 3 l s C b x C 1 I D R V o + / b X c B T T N G I S q C B a D 1 w n A S 8 j C j g V L K 8 M U 8 0 S Q q d k z A a G S h I x 7 W X L 4 D m + N M o I h 7 E y T w J e q r 8 3 M h J p P Y s C M 7 m I q d e 9 h f i f N 0 g h v P M y L p M U m K S r Q 2 E q M M R 4 0 Q I e c c U o i J k h h C p u s m I 6 I Y p Q M F 1 V T A n u + p c 3 S b d R d 6 / r j Y e b W r N V 1 F F G 5 + g C X S E X 3 a I m a q E 2 6 i C K U v S M X t G b N b d e r H f r Y z V a s o q d M / Q H 1 u c P N U 6 T d g = = < / l a t e x i t > zt < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 d W 0 K Q d j r n 4 / g p i S i 6 0 3 A H C Q O E s = " > A A A B + H i c b V D L S s N A F J 3 U V 6 2 P R l 2 6 G S y C q 5 J U Q Z c F N 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M 3 A h t y J e 4 c a G I W z / F n X / j t M 1 C W w 8 M H M 6 5 l 3 v m B I n g G h z n 2 y p t b e / s 7 p X 3 K w e H R 8 d V + + S 0 q + N U U d a h s Y h V P y C a C S 5 Z B z g I 1 k 8 U I 1 E g W C + Y 3 i / 8 3 h N T m s f y E W Y J 8 y I y l j z k l I C R f L s 6 j A h M g j C b 5 3 6 G I f f t m l N 3 l s C b x C 1 I D R V o + / b X c B T T N G I S q C B a D 1 w n A S 8 j C j g V L K 8 M U 8 0 S Q q d k z A a G S h I x 7 W X L 4 D m + N M o I h 7 E y T w J e q r 8 3 M h J p P Y s C M 7 m I q d e 9 h f i f N 0 g h v P M y L p M U m K S r Q 2 E q M M R 4 0 Q I e c c U o i J k h h C p u s m I 6 I Y p Q M F 1 V T A n u + p c 3 S b d R d 6 / r j Y e b W r N V 1 F F G 5 + g C X S E X 3 a I m a q E 2 6 i C K U v S M X t G b N b d e r H f r Y z V a s o q d M / Q H 1 u c P N U 6 T d g = = < / l a t e x i t > xt < l a t e x i t s h a 1 _ b a s e 6 4 = " Y i z I M S T L l o 4 p f X e 6 W F T E F 9 C c E / k = " > A A A B + H i c b V D L S s N A F J 3 U V 6 2 P R l 2 6 G S y C q 5 J U Q Z c F N 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M 3 I g 1 9 E v c u F D E r Z / i z r 9 x 0 m a h r Q c G D u f c y z 1 z g k R w D Y 7 z b Z U 2 N r e 2 d 8 q 7 l b 3 9 g 8 O q f X T c 1 X G q K O v Q W M S q H x D N B J e s A x w E 6 y e K k S g Q r B d M b 3 O / 9 8 C U 5 r G 8 h 1 n C v I i M J Q 8 5 J W A k 3 6 4 O I w K T I M w e 5 3 6 G Y e 7 b N a f u L I D X i V u Q G i r Q 9 u 2 v 4 S i m a c Q k U E G 0 H r h O A l 5 G F H A q 2 L w y T D V L C J 2 S M R s Y K k n E t J c t g s / x u V F G O I y V e R L w Q v 2 9 k Z F I 6 1 k U m M k 8 p l 7 1 c v E / b 5 B C e O N l X C Y p M E m X h 8 J U Y I h x 3 g I e c c U o i J k h h C p u s m I 6 I Y p Q M F 1 V T A n u 6 p f X S b d R d y / r j b u r W r N V 1 F F G p + g M X S A X X a M m a q E 2 6 i C K U v S M X t G b 9 W S 9 W O / W x 3 K 0 Z B U 7 J + g P r M 8 f M j q T d A = = < / l a t e x i t > xt < l a t e x i t s h a 1 _ b a s e 6 4 = " Y i z I M S T L l o 4 p f X e 6 W F T E F 9 C c E / k = " > A A A B + H i c b V D L S s N A F J 3 U V 6 2 P R l 2 6 G S y C q 5 J U Q Z c F N 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M 3 I g 1 9 E v c u F D E r Z / i z r 9 x 0 m a h r Q c G D u f c y z 1 z g k R w D Y 7 z b Z U 2 N r e 2 d 8 q 7 l b 3 9 g 8 O q f X T c 1 X G q K O v Q W M S q H x D N B J e s A x w E 6 y e K k S g Q r B d M b 3 O / 9 8 C U 5 r G 8 h 1 n C v I i M J Q 8 5 J W A k 3 6 4 O I w K T I M w e 5 3 6 G Y e 7 b N a f u L I D X i V u Q G i r Q 9 u 2 v 4 S i m a c Q k U E G 0 H r h O A l 5 G F H A q 2 L w y T D V L C J 2 S M R s Y K k n E t J c t g s / x u V F G O I y V e R L w Q v 2 9 k Z F I 6 1 k U m M k 8 p l 7 1 c v E / b 5 B C e O N l X C Y p M E m X h 8 J U Y I h x 3 g I e c c U o i J k h h C p u s m I 6 I Y p Q M F 1 V T A n u 6 p f X S b d R d y / r j b u r W r N V 1 F F G p + g M X S A X X a M m a q E 2 6 i C K U v S M X t G b 9 W S 9 W O / W x 3 K 0 Z B U 7 J + g P r M 8 f M j q T d A = = < / l a t e x i t > ht < l a t e x i t s h a 1 _ b a s e 6 4 = " y b + z J a V n V x b 7 8 D + u S P 6 s E L i C U y Q = " > A A A B + H i c b V D L S s N A F L 3 x W e u j U Z d u B o v g q i R V 0 G X B T Z c V 7 A P a E C b T S T t 0 M g k z E 6 G G f I k b F 4 q 4 9 V P c + T d O 2 i y 0 9 c D A 4 Z x 7 u W d O k H C m t O N 8 W x u b W 9 s 7 u 5 W 9 6 v 7 B 4 V H N P j 7 p q T i V h H Z J z G M 5 C L C i n A n a 1 U x z O k g k x V H A a T + Y 3 R V + / 5 F K x W L x o O c J 9 S I 8 E S x k B G s j + X Z t F G E 9 D c J s m v s Z 0 r l v 1 5 2 G s w B a J 2 5 J 6 l C i 4 9 t f o 3 F M 0 o g K T T h W a u g 6 i f Y y L D U j n O b V U a p o g s k M T + j Q U I E j q r x s E T x H F 0 Y Z o z C W 5 g m N F u r v j Q x H S s 2 j w E w W M d W q V 4 j / e c N U h 7 d e x k S S a i r I 8 l C Y c q R j V L S A x k x S o v n c E E w k M 1 k R m W K J i T Z d V U 0 J 7 u q X 1 0 m v 2 X C v G s 3 7 6 3 q r X d Z R g T M 4 h 0 t w 4 Q Z a 0 I Y O d I F A C s / w C m / W k / V i v V s f y 9 E N q 9 w 5 h T + w P n 8 A G Z q T Z A = = < / l a t e x i t > ht < l a t e x i t s h a 1 _ b a s e 6 4 = " y b + z J a V n V x b 7 8 D + u S P 6 s E L i C U y Q = " > A A A B + H i c b V D L S s N A F L 3 x W e u j U Z d u B o v g q i R V 0 G X B T Z c V 7 A P a E C b T S T t 0 M g k z E 6 G G f I k b F 4 q 4 9 V P c + T d O 2 i y 0 9 c D A 4 Z x 7 u W d O k H C m t O N 8 W x u b W 9 s 7 u 5 W 9 6 v 7 B 4 V H N P j 7 p q T i V h H Z J z G M 5 C L C i n A n a 1 U x z O k g k x V H A a T + Y 3 R V + / 5 F K x W L x o O c J 9 S I 8 E S x k B G s j + X Z t F G E 9 D c J s m v s Z 0 r l v 1 5 2 G s w B a J 2 5 J 6 l C i 4 9 t f o 3 F M 0 o g K T T h W a u g 6 i f Y y L D U j n O b V U a p o g s k M T + j Q U I E j q r x s E T x H F 0 Y Z o z C W 5 g m N F u r v j Q x H S s 2 j w E w W M d W q V 4 j / e c N U h 7 d e x k S S a i r I 8 l C Y c q R j V L S A x k x S o v n c E E w k M 1 k R m W K J i T Z d V U 0 J 7 u q X 1 0 m v 2 X C v G s 3 7 6 3 q r X d Z R g T M 4 h 0 t w 4 Q Z a 0 I Y O d I F A C s / w C m / W k / V i v V s f y 9 E N q 9 w 5 h T + w P n 8 A G Z q T Z A = = < / l a t e x i t > ht 1

< l a t e x i t s h a 1 _ b a s e 6 4 = " p y H 5 9 v w 3 x 8 S 5 P Y l k O e z m T s y Q 3

8 8 = " > A A A B + n i c b V D L S s N A F L 3 x W e s r 1 a W b w S K 4 s S R V 0 G X B T Z c V 7 A P a U C b T S T t 0 M g k z E 6 X E f I o b F 4 q 4 9 U v c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h X T n q q C i R h L Z J x C P Z 8 7 G i n A n a 1 k x z 2 o s l x a H P a d e f 3 u Z + 9 4 F K x S J x r 2 c x 9 U I 8 F i x g B G s j D e 3 K I M R 6 4 g f p J B u m S F + 4 2 d C u O j V n D r R K 3 I J U o U B r a H 8 N R h F J Q i o 0 4 V i p v u v E 2 k u x 1 I x w m p U H i a I x J l M 8 p n 1 D B Q 6 p 8 t J 5 9 A y d G W W E g k i a J z S a q 7 8 3 U h w q N Q t 9 M 5 k H V c t e L v 7 n 9 R M d 3 H g p E 3 G i q S C L Q 0 H C k Y 5 Q 3 g M a M U m J 5 j N D M J H M Z E V k g i U m 2 r R V N i W 4 y 1 9 e J Z 1 6 z b 2 s 1 e + u q o 1 m U U c J T u A U z s G F a 2 h A E 1 r Q B g K P 8 A y v 8 G Y 9 W S / W u / W x G F 2 z i p 1 j + A P r 8 w f + F Z P W < / l a t e x i t > ht 1

< l a t e x i t s h a 1 _ b a s e 6 4 = " p y H 5 9 v w 3 x 8 S 5 P Y l k O e z m T s y Q 3

8 8 = " > A A A B + n i c b V D L S s N A F L 3 x W e s r 1 a W b w S K 4 s S R V 0 G X B T Z c V 7 A P a U C b T S T t 0 M g k z E 6 X E f I o b F 4 q 4 9 U v c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h X T n q q C i R h L Z J x C P Z 8 7 G i n A n a 1 k x z 2 o s l x a H P a d e f 3 u Z + 9 4 F K x S J x r 2 c x 9 U I 8 F i x g B G s j D e 3 K I M R 6 4 g f p J B u m S F + 4 2 d C u O j V n D r R K 3 I J U o U B r a H 8 N R h F J Q i o 0 4 V i p v u v E 2 k u x 1 I x w m p U H i a I x J l M 8 p n 1 D B Q 6 p 8 t J 5 9 A y d G W W E g k i a J z S a q 7 8 3 U h w q N Q t 9 M 5 k H V c t e L v 7 n 9 R M d 3 H g p E 3 G i q S C L Q 0 H C k Y 5 Q 3 g M a M U m J 5 j N D M J H M Z E V k g i U m 2 r R V N i W 4 y 1 9 e J Z 1 6 z b 2 s 1 e + u q o 1 m U U c J T u A U z s G F a 2 h A E 1 r Q B g K P 8 A y v 8 G Y 9 W S / W u / W x G F 2 z i p 1 j + A P r 8 w f + F Z P W < / l a t e x i t > (c) Generation zt < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 d W 0 K Q d j r n 4 / g p i S i 6 0 3 A H C Q O E s = " > A A A B + H i c b V D L S s N A F J 3 U V 6 2 P R l 2 6 G S y C q 5 J U Q Z c F N 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M 3 A h t y J e 4 c a G I W z / F n X / j t M 1 C W w 8 M H M 6 5 l 3 v m B I n g G h z n 2 y p t b e / s 7 p X 3 K w e H R 8 d V + + S 0 q + N U U d a h s Y h V P y C a C S 5 Z B z g I 1 k 8 U I 1 E g W C + Y 3 i / 8 3 h N T m s f y E W Y J 8 y I y l j z k l I C R f L s 6 j A h M g j C b 5 3 6 G I f f t m l N 3 l s C b x C 1 I D R V o + / b X c B T T N G I S q C B a D 1 w n A S 8 j C j g V L K 8 M U 8 0 S Q q d k z A a G S h I x 7 W X L 4 D m + N M o I h 7 E y T w J e q r 8 3 M h J p P Y s C M 7 m I q d e 9 h f i f N 0 g h v P M y L p M U m K S r Q 2 E q M M R 4 0 Q I e c c U o i J k h h C p u s m I 6 I Y p Q M F 1 V T A n u + p c 3 S b d R d 6 / r j Y e b W r N V 1 F F G 5 + g C X S E X 3 a I m a q E 2 6 i C K U v S M X t G b N b d e r H f r Y z V a s o q d M / Q H 1 u c P N U 6 T d g = = < / l a t e x i t > zt < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 d W 0 K Q d j r n 4 / g p i S i 6 0 3 A H C Q O E s = " > A A A B + H i c b V D L S s N A F J 3 U V 6 2 P R l 2 6 G S y C q 5 J U Q Z c F N 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M 3 A h t y J e 4 c a G I W z / F n X / j t M 1 C W w 8 M H M 6 5 l 3 v m B I n g G h z n 2 y p t b e / s 7 p X 3 K w e H R 8 d V + + S 0 q + N U U d a h s Y h V P y C a C S 5 Z B z g I 1 k 8 U I 1 E g W C + Y 3 i / 8 3 h N T m s f y E W Y J 8 y I y l j z k l I C R f L s 6 j A h M g j C b 5 3 6 G I f f t m l N 3 l s C b x C 1 I D R V o + / b X c B T T N G I S q C B a D 1 w n A S 8 j C j g V L K 8 M U 8 0 S Q q d k z A a G S h I x 7 W X L 4 D m + N M o I h 7 E y T w J e q r 8 3 M h J p P Y s C M 7 m I q d e 9 h f i f N 0 g h v P M y L p M U m K S r Q 2 E q M M R 4 0 Q I e c c U o i J k h h C p u s m I 6 I Y p Q M F 1 V T A n u + p c 3 S b d R d 6 / r j Y e b W r N V 1 F F G 5 + g C X S E X 3 a I m a q E 2 6 i C K U v S M X t G b N b d e r H f r Y z V a s o q d M / Q H 1 u c P N U 6 T d g = = < / l a t e x i t > xt < l a t e x i t s h a 1 _ b a s e 6 4 = " Y i z I M S T L l o 4 p f X e 6 W F T E F 9 C c E / k = " > A A A B + H i c b V D L S s N A F J 3 U V 6 2 P R l 2 6 G S y C q 5 J U Q Z c F N 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M 3 I g 1 9 E v c u F D E r Z / i z r 9 x 0 m a h r Q c G D u f c y z 1 z g k R w D Y 7 z b Z U 2 N r e 2 d 8 q 7 l b 3 9 g 8 O q f X T c 1 X G q K O v Q W M S q H x D N B J e s A x w E 6 y e K k S g Q r B d M b 3 O / 9 8 C U 5 r G 8 h 1 n C v I i M J Q 8 5 J W A k 3 6 4 O I w K T I M w e 5 3 6 G Y e 7 b N a f u L I D X i V u Q G i r Q 9 u 2 v 4 S i m a c Q k U E G 0 H r h O A l 5 G F H A q 2 L w y T D V L C J 2 S M R s Y K k n E t J c t g s / x u V F G O I y V e R L w Q v 2 9 k Z F I 6 1 k U m M k 8 p l 7 1 c v E / b 5 B C e O N l X C Y p M E m X h 8 J U Y I h x 3 g I e c c U o i J k h h C p u s m I 6 I Y p Q M F 1 V T A n u 6 p f X S b d R d y / r j b u r W r N V 1 F F G p + g M X S A X X a M m a q E 2 6 i C K U v S M X t G b 9 W S 9 W O / W x 3 K 0 Z B U 7 J + g P r M 8 f M j q T d A = = < / l a t e x i t > xt < l a t e x i t s h a 1 _ b a s e 6 4 = " Y i z I M S T L l o 4 p f X e 6 W F T E F 9 C c E / k = " > A A A B + H i c b V D L S s N A F J 3 U V 6 2 P R l 2 6 G S y C q 5 J U Q Z c F N 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M 3 I g 1 9 E v c u F D E r Z / i z r 9 x 0 m a h r Q c G D u f c y z 1 z g k R w D Y 7 z b Z U 2 N r e 2 d 8 q 7 l b 3 9 g 8 O q f X T c 1 X G q K O v Q W M S q H x D N B J e s A x w E 6 y e K k S g Q r B d M b 3 O / 9 8 C U 5 r G 8 h 1 n C v I i M J Q 8 5 J W A k 3 6 4 O I w K T I M w e 5 3 6 G Y e 7 b N a f u L I D X i V u Q G i r Q 9 u 2 v 4 S i m a c Q k U E G 0 H r h O A l 5 G F H A q 2 L w y T D V L C J 2 S M R s Y K k n E t J c t g s / x u V F G O I y V e R L w Q v 2 9 k Z F I 6 1 k U m M k 8 p l 7 1 c v E / b 5 B C e O N l X C Y p M E m X h 8 J U Y I h x 3 g I e c c U o i J k h h C p u s m I 6 I Y p Q M F 1 V T A n u 6 p f X S b d R d y / r j b u r W r N V 1 F F G p + g M X S A X X a M m a q E 2 6 i C K U v S M X t G b 9 W S 9 W O / W x 3 K 0 Z B U 7 J + g P r M 8 f M j q T d A = = < / l a t e x i t > ht < l a t e x i t s h a 1 _ b a s e 6 4 = " y b + z J a V n V x b 7 8 D + u S P 6 s E L i C U y Q = " > A A A B + H i c b V D L S s N A F L 3 x W e u j U Z d u B o v g q i R V 0 G X B T Z c V 7 A P a E C b T S T t 0 M g k z E 6 G G f I k b F 4 q 4 9 V P c + T d O 2 i y 0 9 c D A 4 Z x 7 u W d O k H C m t O N 8 W x u b W 9 s 7 u 5 W 9 6 v 7 B 4 V H N P j 7 p q T i V h H Z J z G M 5 C L C i n A n a 1 U x z O k g k x V H A a T + Y 3 R V + / 5 F K x W L x o O c J 9 S I 8 E S x k B G s j + X Z t F G E 9 D c J s m v s Z 0 r l v 1 5 2 G s w B a J 2 5 J 6 l C i 4 9 t f o 3 F M 0 o g K T T h W a u g 6 i f Y y L D U j n O b V U a p o g s k M T + j Q U I E j q r x s E T x H F 0 Y Z o z C W 5 g m N F u r v j Q x H S s 2 j w E w W M d W q V 4 j / e c N U h 7 d e x k S S a i r I 8 l C Y c q R j V L S A x k x S o v n c E E w k M 1 k R m W K J i T Z d V U 0 J 7 u q X 1 0 m v 2 X C v G s 3 7 6 3 q r X d Z R g T M 4 h 0 t w 4 Q Z a 0 I Y O d I F A C s / w C m / W k / V i v V

s f y 9 E N q 9 w 5 h T + w P n 8 A G Z q T Z A = = < / l a t e x i t > ht < l a t e x i t s h a 1 _ b a s e 6 4 = " y b + z J a V n V x b 7 8 D + u S P 6 s E

L i C U y Q = " > A A A B + H i c b V D L S s N A F L 3 x W e u j U Z d u B o v g q i R V 0 G X B T Z c V 7 A P a E C b T S T t 0 M g k z E 6 G G f I k b F 4 q 4 9 V P c + T d O 2 i y 0 9 c D A 4 Z x 7 u W d O k H C m t O N 8 W x u b W 9 s 7 u 5 W 9 6 v 7 B 4 V H N P j 7 p q T i V h H Z J z G M 5 C L C i n A n a 1 U x z O k g k x V H A a T + Y 3 R V + / 5 F K x W L x o O c J 9 S I 8 E S x k B G s j + X Z t F G E 9 D c J s m v s Z 0 r l v 1 5 2 G s w B a J 2 5 J 6 l C i 4 9 t f o 3 F M 0 o g K T T h W a u g 6 i f Y y L D U j n O b V U a p o g s k M T + j Q U I E j q r x s E T x H F 0 Y Z o z C W 5 g m N F u r v j Q x H S s 2 j w E w W M d W q V 4 j / e c N U h 7 d e x k S S a i r I 8 l C Y c q R j V L S A x k x S o v n c E E w k M 1 k R m W K J i T Z d V U 0 J 7 u q X 1 0 m v 2 X C v G s 3 7 6 3 q r X d Z R g T M 4 h 0 t w 4 Q Z a 0 I Y O d I F A C s / w C m / W k / V i v V

s f y 9 E N q 9 w 5 h T + w P n 8 A G Z q T Z A = = < / l a t e x i t > ht 1

< l a t e x i t s h a 1 _ b a s e 6 4 = " p y H 5 9 v w 3 x 8 S 5 P Y l k O e z m T s y Q 3

8 8 = " > A A A B + n i c b V D L S s N A F L 3 x W e s r 1 a W b w S K 4 s S R V 0 G X B T Z c V 7 A P a U C b T S T t 0 M g k z E 6 X E f I o b F 4 q 4 9 U v c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h X T n q q C i R h L Z J x C P Z 8 7 G i n A n a 1 k x z 2 o s l x a H P a d e f 3 u Z + 9 4 F K x S J x r 2 c x 9 U I 8 F i x g B G s j D e 3 K I M R 6 4 g f p J B u m S F + 4 2 d C u O j V n D r R K 3 I J U o U B r a H 8 N R h F J Q i o 0 4 V i p v u v E 2 k u x 1 I x w m p U H i a I x J l M 8 p n 1 D B Q 6 p 8 t J 5 9 A y d G W W E g k i a J z S a q 7 8 3 U h w q N Q t 9 M 5 k H V c t e L v 7 n 9 R M d 3 H g p E 3 G i q S C L Q 0 H C k Y 5 Q 3 g M a M U m J 5 j N D M J H M Z E V k g i U m 2 r R V N i W 4 y 1 9 e J Z 1 6 z b 2 s 1 e + u q o 1 m U U c J T u A U z s G F a 2 h A E 1 r Q B g K P 8 A y v 8 G Y 9 W S / W u / W x

G F 2 z i p 1 j + A P r 8 w f + F Z P W < / l a t e x i t > ht 1

< l a t e x i t s h a 1 _ b a s e 6 4 = " p y H 5 9 v w 3 x 8 S 5 P Y l k O e z m T s y Q 3 The overall computational path. The hypernetwork components are left out.

8 8 = " > A A A B + n i c b V D L S s N A F L 3 x W e s r 1 a W b w S K 4 s S R V 0 G X B T Z c V 7 A P a U C b T S T t 0 M g k z E 6 X E f I o b F 4 q 4 9 U v c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h X T n q q C i R h L Z J x C P Z 8 7 G i n A n a 1 k x z 2 o s l x a H P a d e f 3 u Z + 9 4 F K x S J x r 2 c x 9 U I 8 F i x g B G s j D e 3 K I M R 6 4 g f p J B u m S F + 4 2 d C u O j V n D r R K 3 I J U o U B r a H 8 N R h F J Q i o 0 4 V i p v u v E 2 k u x 1 I x w m p U H i a I x J l M 8 p n 1 D B Q 6 p 8 t J 5 9 A y d G W W E g k i a J z S a q 7 8 3 U h w q N Q t 9 M 5 k H V c t e L v 7 n 9 R M d 3 H g p E 3 G i q S C L Q 0 H C k Y 5 Q 3 g M a M U m J 5 j N D M J H M Z E V k g i U m 2 r R V N i W 4 y 1 9 e J Z 1 6 z b 2 s 1 e + u q o 1 m U U c J T u A U z s G F a 2 h A E 1 r Q B g K P 8 A y v 8 G Y 9 W S / W u / W x G F 2 z i p 1 j + A P r 8 w f + F Z P W < / l a t e x i t > (d) Inference zt < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 d W 0 K Q d j r n 4 / g p i S i 6 0 3 A H C Q O E s = " > A A A B + H i c b V D L S s N A F J 3 U V 6 2 P R l 2 6 G S y C q 5 J U Q Z c F N 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M 3 A h t y J e 4 c a G I W z / F n X / j t M 1 C W w 8 M H M 6 5 l 3 v m B I n g G h z n 2 y p t b e / s 7 p X 3 K w e H R 8 d V + + S 0 q + N U U d a h s Y h V P y C a C S 5 Z B z g I 1 k 8 U I 1 E g W C + Y 3 i / 8 3 h N T m s f y E W Y J 8 y I y l j z k l I C R f L s 6 j A h M g j C b 5 3 6 G I f f t m l N 3 l s C b x C 1 I D R V o + / b X c B T T N G I S q C B a D 1 w n A S 8 j C j g V L K 8 M U 8 0 S Q q d k z A a G S h I x 7 W X L 4 D m + N M o I h 7 E y T w J e q r 8 3 M h J p P Y s C M 7 m I q d e 9 h f i f N 0 g h v P M y L p M U m K S r Q 2 E q M M R 4 0 Q I e c c U o i J k h h C p u s m I 6 I Y p Q M F 1 V T A n u + p c 3 S b d R d 6 / r j Y e b W r N V 1 F F G 5 + g C X S E X 3 a I m a q E 2 6 i C K U v S M X t G b N b d e r H f r Y z V a s o q d M / Q H 1 u c P N U 6 T d g = = < / l a t e x i t > zt < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 d W 0 K Q d j r n 4 / g p i S i 6 0 3 A H C Q O E s = " > A A A B + H i c b V D L S s N A F J 3 U V 6 2 P R l 2 6 G S y C q 5 J U Q Z c F N 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M 3 A h t y J e 4 c a G I W z / F n X / j t M 1 C W w 8 M H M 6 5 l 3 v m B I n g G h z n 2 y p t b e / s 7 p X 3 K w e H R 8 d V + + S 0 q + N U U d a h s Y h V P y C a C S 5 Z B z g I 1 k 8 U I 1 E g W C + Y 3 i / 8 3 h N T m s f y E W Y J 8 y I y l j z k l I C R f L s 6 j A h M g j C b 5 3 6 G I f f t m l N 3 l s C b x C 1 I D R V o + / b X c B T T N G I S q C B a D 1 w n A S 8 j C j g V L K 8 M U 8 0 S Q q d k z A a G S h I x 7 W X L 4 D m + N M o I h 7 E y T w J e q r 8 3 M h J p P Y s C M 7 m I q d e 9 h f i f N 0 g h v P M y L p M U m K S r Q 2 E q M M R 4 0 Q I e c c U o i J k h h C p u s m I 6 I Y p Q M F 1 V T A n u + p c 3 S b d R d 6 / r j Y e b W r N V 1 F F G 5 + g C X S E X 3 a I m a q E 2 6 i C K U v S M X t G b N b d e r H f r Y z V a s o q d M / Q H 1 u c P N U 6 T d g = = < / l a t e x i t > xt < l a t e x i t s h a 1 _ b a s e 6 4 = " Y i z I M S T L l o 4 p f X e 6 W F T E F 9 C c E / k = " > A A A B + H i c b V D L S s N A F J 3 U V 6 2 P R l 2 6 G S y C q 5 J U Q Z c F N 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M 3 I g 1 9 E v c u F D E r Z / i z r 9 x 0 m a h r Q c G D u f c y z 1 z g k R w D Y 7 z b Z U 2 N r e 2 d 8 q 7 l b 3 9 g 8 O q f X T c 1 X G q K O v Q W M S q H x D N B J e s A x w E 6 y e K k S g Q r B d M b 3 O / 9 8 C U 5 r G 8 h 1 n C v I i M J Q 8 5 J W A k 3 6 4 O I w K T I M w e 5 3 6 G Y e 7 b N a f u L I D X i V u Q G i r Q 9 u 2 v 4 S i m a c Q k U E G 0 H r h O A l 5 G F H A q 2 L w y T D V L C J 2 S M R s Y K k n E t J c t g s / x u V F G O I y V e R L w Q v 2 9 k Z F I 6 1 k U m M k 8 p l 7 1 c v E / b 5 B C e O N l X C Y p M E m X h 8 J U Y I h x 3 g I e c c U o i J k h h C p u s m I 6 I Y p Q M F 1 V T A n u 6 p f X S b d R d y / r j b u r W r N V 1 F F G p + g M X S A X X a M m a q E 2 6 i C K U v S M X t G b 9 W S 9 W O / W x 3 K 0 Z B U 7 J + g P r M 8 f M j q T d A = = < / l a t e x i t > xt < l a t e x i t s h a 1 _ b a s e 6 4 = " Y i z I M S T L l o 4 p f X e 6 W F T E F 9 C c E / k = " > A A A B + H i c b V D L S s N A F J 3 U V 6 2 P R l 2 6 G S y C q 5 J U Q Z c F N 1 1 W s A 9 o Q 5 h M J + 3 Q y S T M 3 I g 1 9 E v c u F D E r Z / i z r 9 x 0 m a h r Q c G D u f c y z 1 z g k R w D Y 7 z b Z U 2 N r e 2 d 8 q 7 l b 3 9 g 8 O q f X T c 1 X G q K O v Q W M S q H x D N B J e s A x w E 6 y e K k S g Q r B d M b 3 O / 9 8 C U 5 r G 8 h 1 n C v I i M J Q 8 5 J W A k 3 6 4 O I w K T I M w e 5 3 6 G Y e 7 b N a f u L I D X i V u Q G i r Q 9 u 2 v 4 S i m a c Q k U E G 0 H r h O A l 5 G F H A q 2 L w y T D V L C J 2 S M R s Y K k n E t J c t g s / x u V F G O I y V e R L w Q v 2 9 k Z F I 6 1 k U m M k 8 p l 7 1 c v E / b 5 B C e O N l X C Y p M E m X h 8 J U Y I h x 3 g I e c c U o i J k h h C p u s m I 6 I Y p Q M F 1 V T A n u 6 p f X S b d R d y / r j b u r W r N V 1 F F G p + g M X S A X X a M m a q E 2 6 i C K U v S M X t G b 9 W S 9 W O / W x 3 K 0 Z B U 7 J + g P r M 8 f M j q T d A = = < / l a t e x i t > ht < l a t e x i t s h a 1 _ b a s e 6 4 = " y b + z J a V n V x b 7 8 D + u S P 6 s E L i C U y Q = " > A A A B + H i c b V D L S s N A F L 3 x W e u j U Z d u B o v g q i R V 0 G X B T Z c V 7 A P a E C b T S T t 0 M g k z E 6 G G f I k b F 4 q 4 9 V P c + T d O 2 i y 0 9 c D A 4 Z x 7 u W d O k H C m t O N 8 W x u b W 9 s 7 u 5 W 9 6 v 7 B 4 V H N P j 7 p q T i V h H Z J z G M 5 C L C i n A n a 1 U x z O k g k x V H A a T + Y 3 R V + / 5 F K x W L x o O c J 9 S I 8 E S x k B G s j + X Z t F G E 9 D c J s m v s Z 0 r l v 1 5 2 G s w B a J 2 5 J 6 l C i 4 9 t f o 3 F M 0 o g K T T h W a u g 6 i f Y y L D U j n O b V U a p o g s k M T + j Q U I E j q r x s E T x H F 0 Y Z o z C W 5 g m N F u r v j Q x H S s 2 j w E w W M d W q V 4 j / e c N U h 7 d e x k S S a i r I 8 l C Y c q R j V L S A x k x S o v n c E E w k M 1 k R m W K J i T Z d V U 0 J 7 u q X 1 0 m v 2 X C v G s 3 7 6 3 q r X d Z R g T M 4 h 0 t w 4 Q Z a 0 I Y O d I F A C s / w C m / W k / V i v V s f y 9 E N q 9 w 5 h T + w P n 8 A G Z q T Z A = = < / l a t e x i t > ht < l a t e x i t s h a 1 _ b a s e 6 4 = " y b + z J a V n V x b 7 8 D + u S P 6 s E L i C U y Q = " > A A A B + H i c b V D L S s N A F L 3 x W e u j U Z d u B o v g q i R V 0 G X B T Z c V 7 A P a E C b T S T t 0 M g k z E 6 G G f I k b F 4 q 4 9 V P c + T d O 2 i y 0 9 c D A 4 Z x 7 u W d O k H C m t O N 8 W x u b W 9 s 7 u 5 W 9 6 v 7 B 4 V H N P j 7 p q T i V h H Z J z G M 5 C L C i n A n a 1 U x z O k g k x V H A a T + Y 3 R V + / 5 F K x W L x o O c J 9 S I 8 E S x k B G s j + X Z t F G E 9 D c J s m v s Z 0 r l v 1 5 2 G s w B a J 2 5 J 6 l C i 4 9 t f o 3 F M 0 o g K T T h W a u g 6 i f Y y L D U j n O b V U a p o g s k M T + j Q U I E j q r x s E T x H F 0 Y Z o z C W 5 g m N F u r v j Q x H S s 2 j w E w W M d W q V 4 j / e c N U h 7 d e x k S S a i r I 8 l C Y c q R j V L S A x k x S o v n c E E w k M 1 k R m W K J i T Z d V U 0 J 7 u q X 1 0 m v 2 X C v G s 3 7 6 3 q r X d Z R g T M 4 h 0 t w 4 Q Z a 0 I Y O d I F A C s / w C m / W k / V i v V s f y 9 E N q 9 w 5 h T + w P n 8 A G Z q T Z A = = < / l a t e x i t > ht 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " p y H 5 9 v w 3 x 8 S 5 P Y l k O e z m T s y Q 3 8 8 = " > A A A B + n i c b V D L S s N A F L 3 x W e s r 1 a W b w S K 4 s S R V 0 G X B T Z c V 7 A P a U C b T S T t 0 M g k z E 6 X E f I o b F 4 q 4 9 U v c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h X T n q q C i R h L Z J x C P Z 8 7 G i n A n a 1 k x z 2 o s l x a H P a d e f 3 u Z + 9 4 F K x S J x r 2 c x 9 U I 8 F i x g B G s j D e 3 K I M R 6 4 g f p J B u m S F + 4 2 d C u O j V n D r R K 3 I J U o U B r a H 8 N R h F J Q i o 0 4 V i p v u v E 2 k u x 1 I x w m p U H i a I x J l M 8 p n 1 D B Q 6 p 8 t J 5 9 A y d G W W E g k i a J z S a q 7 8 3 U h w q N Q t 9 M 5 k H V c t e L v 7 n 9 R M d 3 H g p E 3 G i q S C L Q 0 H C k Y 5 Q 3 g M a M U m J 5 j N D M J H M Z E V k g i U m 2 r R V N i W 4 y 1 9 e J Z 1 6 z b 2 s 1 e + u q o 1 m U U c J T u A U z s G F a 2 h A E 1 r Q B g K P 8 A y v 8 G Y 9 W S / W u / W x G F 2 z i p 1 j + A P r 8 w f + F Z P W < / l a t e x i t > ht 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " p y H 5 9 v w 3 x 8 S 5 P Y l k O e z m T s y Q 3 8 8 = " > A A A B + n i c b V D L S s N A F L 3 x W e s r 1 a W b w S K 4 s S R V 0 G X B T Z c V 7 A P a U C b T S T t 0 M g k z E 6 X E f I o b F 4 q 4 9 U v c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h X T n q q C i R h L Z J x C P Z 8 7 G i n A n a 1 k x z 2 o s l x a H P a d e f 3 u Z + 9 4 F K x S J x r 2 c x 9 U I 8 F i x g B G s j D e 3 K I M R 6 4 g f p J B u m S F + 4 2 d C u O j V n D r R K 3 I J U o U B r a H 8 N R h F J Q i o 0 4 V i p v u v E 2 k u x 1 I x w m p U H i a I x J l M 8 p n 1 D B Q 6 p 8 t J 5 9 A y d G W W E g k i a J z S a q 7 8 3 U h w q N Q t 9 M 5 k H V c t e L v 7 n 9 R M d 3 H g p E 3 G i q S C L Q 0 H C k Y 5 Q 3 g M a M U m J 5 j N D M J H M Z E V k g i U m 2 r R V N i W 4 y 1 9 e J Z 1 6 z b 2 s 1 e + u q o 1 m U U c J T u A U z s G F a 2 h A E 1 r Q B g K P 8 A y v 8 G Y 9 W S / W u / W x G F 2 z i p 1 j + A P r 8 w f + F Z P W < / l a t e x i t > (e) Overall

with HyperNetworks, the motivations and architectures are fundamentally different. The work of Dezfouli et al. (2019) intends to learn a low-dimensional interpretable latent space that represents the difference between subjects in decision-making process while our models tries to better handle the variances both within and across sequences in general time-series data modeling. The work of Dezfouli et al. (2019) generate a set of parameters that is shared across time steps for each latent variable. In contrast, our model samples a latent variable and dynamically generates non-shared weights at each time step, which we believe is essential for handling variance of dynamics within sequences.

3. MODEL FORMULATION:

Variational Hyper RNN A recurrent neural network (RNN) can be characterized by h t = g θ (x t , h t-1 ), where x t and h t are the observation and hidden state of the RNN at time step t, and θ is the fixed weights of the RNN model. The hidden state h t is often used to generate the output for other learning tasks, e.g., predicting the observation at the next time step. We augment the RNN with a latent random variable z t , which is also used to output the non-shared parameters of the RNN at time step t.

EQUATION

where θ(z t , h t-1 ) is a hypernetwork that generates the parameters of the RNN at time step t. The latent variable z t can also be used to determine the parameters of the generative model p(x t |z ≤t , x <t ):

EQUATION

We hypothesize that the previous observations and latent variables, characterized by h t-1 , define a prior distribution p(z t |x <t , z <t ) over the latent variable z t ,

EQUATION

Eq. and 3 result in the following generation process of sequential data:

EQUATION

The true posterior distributions of z t conditioned on observations x ≤t and latent variables z <t are intractable, posing a challenge in both sampling and learning. Therefore, we introduce an approximate posterior q(z t |x ≤t , z <t ) such that

EQUATION

This approximate posterior distribution enables the model to be trained by maximizing a variational lower bound, e.g., ELBO (Kingma & Welling, 2013) , IWAE (Burda et al., 2016) and FIVO (Maddison et al., 2017) . We refer to the main components of our model, including g, φ dec , φ enc , φ prior as primary networks and refer to the components responsible for generating parameters, θ and ω, as hyper networks in the following sections.

Implementation Following the practice of VAE, we parametrize the covariance matrices Σ prior t , Σ dec t and Σ enc t as diagonal matrices. Note that Σ prior t in our model is no longer an identity matrix as in a vanilla VAE; it is the output of φ prior and depends on the hidden state h t-1 at the previous time step.

The recurrence model g in Eq. 1 is implemented as an RNN cell, which takes as input x t and z t at each time step t and updates the hidden states h t-1 . The parameters of g are generated by the hyper network θ(z t , h t-1 ), as illustrated in Figure 1b . θ is also implemented using an RNN to capture the history of data dynamics, with z t and h t-1 as input at each time step t. However, it is computationally costly to generate all the parameters of g using θ(z t , h t-1 ). Following the practice of previous works (Ha et al., 2016; Krueger et al., 2017) , the hyper network θ maps z t and h t-1 to bias and scaling vectors. The scaling vectors modify the parameters of g by scaling each row of the weight matrices, routing information in the input and hidden state vectors through different channels.

To better illustrate this mechanism, we exemplify the recurrence model g using an RNN cell with LSTM-style update rules and gates. Let * ∈ {i, f, g, o} denote the one of the four LSTM-style gates in g. W * and U * denote the input and recurrent weights of each gate in LSTM cell respectively. The hyper network θ(z t , h t-1 ) outputs d i * and d h * that are the scaling vectors for the input weights W * and recurrent weights U * of the recurrent model g in Eq. 1. The overall implementation of g in Eq. 1 can be described as follows:

i t = σ (d ii (z t , h t-1 ) • (W i y t ) + d hi (z t , h t-1 ) • (U i h t-1 )) , f t = σ (d if (z t , h t-1 ) • (W f y t ) + d hf (z t , h t-1 ) • (U f h t-1 )) , g t = tanh (d ig (z t , h t-1 ) • (W g y t ) + d hg (z t , h t-1 ) • (U g h t-1 )) , o t = σ (d io (z t , h t-1 ) • (W o y t ) + d ho (z t , h t-1 ) • (U o h t-1 )) , c t = f t • c t-1 + i t • g t , h t = o t • tanh (c t ) ,

where • denotes the Hadamard product and y t is a fusion (e.g., concatenation) of observation x t and latent variable z t . For simplicity of notation, bias terms are ignored from the above equations. This implementation of the recurrence model in VHRNN is further illustrated in the diagram in Fig. 6 in appendix.

Another hyper network ω(z t , h t-1 ) generates the parameters of the generative model in Eq. 2. It is implemented as a multilayer perceptron (MLP). Similar to θ(z t , h t-1 ), the outputs are the bias and scaling vectors that modify the parameters of the decoder φ dec ω(zt,ht-1) .

4. SYSTEMATIC GENERALIZATION ANALYSIS OF VHRNN:

In terms of the general functional form Eq. 1, the recurrence of VRNN and VHRNN both depend on z t and h t-1 , so a sufficiently large VRNN could capture the same behaviour as VHRNN in theory. However, VHRNN's structure better encodes the inductive bias that the underlying dynamics could change, that they could slightly deviate from the typical behaviour in a regime, or there could be drastic switch to a new regime. With finite training data and finite parameters, this inductive bias could lead to qualitatively different learned behaviour, which we demonstrate and analyze now.

In the spirit of Bahdanau et al. (2019) , we perform a systematic generalization study of VHRNN in comparison to the VRNN baseline. We train the models on one synthetic dataset with each sequence generated by fixed linear dynamics and corrupted by heteroskedastic noise process. We demonstrate that VHRNN can disentangle the two contributions of variations and learn the different base patterns of the complex dynamics while doing so with fewer parameters. Furthermore, VHRNN can generalize to a wide range of unseen dynamics, albeit the much simpler training set.

The synthetic dataset is generated by the following recurrence equation:

EQUATION

where t ∈ R 2 is a two-dimensional standard Gaussian noise and x 0 is randomly initialized from a uniform distribution over [-1, 1] 2 . For each sequence, W ∈ R 2×2 is sampled from 10 predefined random matrices {W i } 10 i=1 with equal probability; σ t is the standard deviation of the additive noise at time t and takes value from {0.25, 1, 4}. The noise level shifts twice within a sequence; i.e., there are exactly two t's such that σ t = σ t-1 . We generate 800 sequences for training, 100 sequences for validation, and 100 sequences for test using the same sets of predefined matrices. The models are trained and evaluated using FIVO as the objective. The results on the test set are almost the same as those on the training set for both VRNN and VHRNN. We also find that VHRNN shows better performance than VRNN with fewer parameters, as shown in Tab. 1, column Test. The size of the hidden state in RNN cells is set to be the same as the latent size for both types of models.

We further study the behavior of VRNN and VHRNN under the following systematically varied settings:

• NOISELESS In this setting, sequences are generated using a similar recurrence rule with the same set of predefined weights without the additive noise at each step. That is, σ t = 0 in Eq. 6 for all time step t. The exponential growth of data could happen when the singular values of the underlying weight matrix are greater than 1. • SWITCH In this setting, three NOISELESS sequences are concatenated into one, which contains regime shifts as a result. This setting requires the model to identify and re-identify the underlying pattern after observing changes. • RAND In this setting, the deterministic transition matrix in Eq. 6 is set to the identity matrix Tab. 1 illustrates the experimental results. We can see that VRNN model, depending on model complexity, either underfits the original data generation pattern (Test) or fails to generalize to more complicated settings. In contrast, the VHRNN model does not suffer from such problems and uniformly outperforms VRNN models under all settings. To qualitatively study the behavior of VHRNN and VRNN, we consider a VRNN with a latent dimension of 8 and a VHRNN with a latent dimension of 4 and make the following observations: Dynamic Regime Identification and Re-identification Fig. 2 shows a sample sequence under the NOISELESS setting. VRNN has high KL divergence between the prior and the variational posterior most of the time. In contrast, VHRNN has a decreasing trend of KL divergence while still making accurate mean reconstruction as it observes more data. As the KL divergence measures the discrepancy between prior defined in Eq. 3 and the posterior that has information from the current observation, simultaneous low reconstruction and low KL divergence means that the prior distribution would be able to predict with low errors as well, indicating that the correct underlying dynamics model has likely been utilized. This trend even generalizes to settings with sources of variation unseen in the training data, namely ZEROSHOT and ADD. We speculate that this trend implies the model's ability to identify the underlying data generation pattern in the sequence. The decreasing trend is especially apparent when a sudden and big change in scale happens. We hypothesize that larger changes in scale can better help our model, VHRNN, identify the underlying data generation process because our model is trained on sequential data generated with compound noise. The observation further corroborates our conjecture that the KL divergence would rise again once the sequence switches from one underlying weight to another, as shown in Fig. 3 . It is worth noting that the KL increase happens with some latency after the sequence switches in the SWITCH setting as the model reacts to the change and tries to reconcile with the prior belief of the underlying regime in effect.

Uncertainty Identification Fig. 4 shows that the predicted log-variance of VHRNN can more accurately reflect the change of noise levels under the RAND setting than VRNN. VHRNN can also better handle uncertainty than VRNN in the following two situations. As shown in Fig. 3f , VHRNN can more aggressively adapt its variance prediction based on the scale of the data than VRNN. It keeps its predicted variance at a low level when the data scale is small and increases the value when the scale of data becomes large. VHRNN makes inaccurate mean prediction relatively far from the target value when the switch of underlying generation dynamics happens in the SWITCH setting.

The switch of the weight matrix is another important source of uncertainty. We observe that VHRNN would also make a large log-variance prediction in this situation, even the scale of the observation is small. Aggressively increasing its uncertainty about the prediction when a switch happens avoids VHRNN model from paying high reconstruction cost as shown by the second spike in Fig. 3f . This increase of variance prediction also happens when exponential becomes apparent in setting LONG and the scale of observed data became out of the range of the training data. Given the large scale change of the data, such flexibility to predict large variance is key for VHRNN to avoid paying large reconstruction cost. 

5. EXPERIMENTS ON REAL-WORLD DATA:

We experiment with the VHRNN model on several real-world datasets and compare it against VRNN model. VRNN trained and evaluated using FIVO (Maddison et al., 2017) demonstrates the state-of-the-art performance on various sequence modeling tasks. Our experiments demonstrate the superior parameter-performance efficiency and generalization ability of VHRNN over VRNN. All the models are trained using FIVO (Maddison et al., 2017) and we report FIVO per step when evaluating models. Two polyphonic music dataset are considered: JSB Chorale and Piano-midi.de (Boulanger-Lewandowski et al., 2012) . We also train and test our models on a financial time series data and the HT Sensor dataset (Huerta et al., 2016) , which contains sequences of sensor readings when different types of stimuli are applied in an environment during experiments. We also trained and evaluated the HyperLSTM model without latent variables proposed by Ha et al. (2016) on a few of the datasets above. The results and comparisons are deferred to the appendix.

For the VRNN model, we use a single-layer LSTM and set the dimension of the hidden state to be the same as the latent dimension. For the VHRNN model, θ in Eq. 1 is implemented using a single-layer LSTM to generate weights for the recurrence module in the primary networks. We use an RNN cell with LSTM-style gates and update rules for the recurrence module g in our experiments. The hidden state sizes of both the primary network and hyper network are the same as the latent dimension. A linear transformation directly maps the hyper hidden state to the scaling and bias vectors in the primary network. More details on the architectures of encoder, generation and prior networks are elaborated in the appendix.

Polyphonic Music:

The JSB Chorale and Piano-midi.de are music datasets (Boulanger-Lewandowski et al., 2012) with complex patterns and large variance both within and across sequences. The datasets are split into the standard train, validation, and test sets. More details on data preprocessing, training and evaluation setup are deferred to the appendix. We report the FIVO per time step of VHRNNs and VRNNs and their parameter counts in Fig. 5a and Fig. 5b . The results show that VHRNNs have better performance and parameter efficiency. The number of parameters and FIVO per time step of each model are plotted in the figures, and the latent dimension is also annotated. The parameter-performance plots show that the VHRNN model has uniformly better performance than VRNN with a comparable number of parameters. The best FIVO achieved by VHRNN on JSB dataset is -6.76 (VHRNN-14) compared to -6.92 for VRNN (VRNN-32), which requires close to one third more parameters. This best VRNN model is even worse than the smallest VHRNN model we have evaluated. It is also observed that VHRNN is less prone to overfitting and has better generalization ability than VRNN when the number of parameters keeps growing. Similar trends can be seen on the Piano-midi.de dataset in Fig. 5b . We also find that the better performance of VHRNN over VRNN can generalize to the scenario where we replace LSTM with Gated Recurrent Unit (GRU). Experimental results using GRU implementation are deferred to the appendix.

Stock Financial time series data, such as daily prices of stocks, are highly volatile with large noise. The market volatility is affected by many external factors and can experience tremendous changes in a sudden. To test the models' ability to adapt to different volatility levels and noise patterns, we compare VHRNN and VRNN on stock price data collected in a period when the market went through rapid changes. The data are collected from 445 stocks in the S&P500 index in 2008 when a global financial crisis happened. The dataset contains the opening, closing, highest and lowest prices, and volume on each day. The networks are trained on sequences from the first half of the year and tested on sequences from the second half, during which the market suddenly became significantly more volatile due to the financial crisis.

The evaluation results are shown in Fig. 5c . The plot shows that VHRNN models consistently outperform VRNN models regardless of the latent dimension and number of parameters. The results indicate that VHRNN can have better generalizability to sequential data in which the underlying data generation pattern suddenly shifts even if the new dynamics are not seen in the training data.

HT Sensor:

The comparison is also performed on a dataset with less variation and simpler patterns than the previous datasets. The HT Sensor dataset contains sequences of gas, humidity, and temperature sensor readings in experiments where some stimulus is applied after a period of background activity (Huerta et al., 2016) . There are only two types of stimuli in the experiments: banana and wine. In some sequences, there is no stimulus applied, and they only contain readings under background noise. Experimental results on HT Sensor dataset are shown in Fig. 5d .

It is observed that VHRNN has comparable performance as VRNN on the HT Senor Dataset when using a similar number of parameters. For example, VHRNN achieves a FIVO per time step of 14.41 with 16 latent dimensions and 24200 parameters, while VRNN shows slightly worse performance with 28 latent dimensions and approximately 26000 parameters. When the number of parameters goes slightly beyond 34000, the FIVO of VHRNN decays to 12.45 compared to 12.37 of VRNN.

6. ABLATION STUDY:

We further investigate the effects of hidden state and latent variable on the performance of variational hyper RNN in the following two aspects: the dimension of the latent variable and the contributions by hidden state and latent variable as inputs to hyper networks.

Latent Dimension:

In previous experiments on real-world datasets, the latent dimension and hidden state dimension are set to be the same for each model. This causes VHRNN to have significantly more parameters than a VRNN when using the same latent dimension. To eliminate the effects of the difference in model size, we allow the latent dimension and hidden state dimension to be different. We also reduce the hidden layer size of the hyper network that generates the weight of the decoder. These changes allow us to compare VRNN and VHRNN models with the same latent dimension and a similar number of parameters. The results on JSB Chorale datasets are presented in Tab. 2 in which we denote latent dimension by Z dim. We observe that VHRNNs always have better FIVO with the same latent dimensions than VRNNs. The results show that the superior performance of VHRNN over VRNN does not stem from smaller latent dimension when using the comparable number of parameters.

Inputs to the Hyper Networks We retrain and evaluate the performance of VHRNN models on JSB Chorale dataset and the synthetic sequences when feeding the latent variable only, the hidden state only, or both to the hyper networks. The results are shown in Tab. 3. It is observed that VHRNN has the best performance and generalization ability when it takes the latent variable as its only input.

Relying on the primary network's hidden state only or the combination of latent variable and hidden state leads to worse performance. When the dimension of the hidden state is 32, VHRNN only taking the hidden state as hyper input suffers from over-parameterization and has worse performance than VRNN with the same dimension of the hidden state. On the test set of synthetic data, VHRNN obtains the best performance when it takes both hidden state and latent variable as inputs. We surmise that this difference is due to the fact that historical information is critical to determine the underlying recurrent weights and current noise level for synthetic data. However, the ablation study on both datasets shows the importance of the sampled latent variable as an input to the hyper networks. Therefore, both hidden state and latent variable are used as inputs to hyper networks on other datasets for consistency.

7. CONCLUSION:

In this paper, we introduce the variational hyper RNN (VHRNN) model, which can generate parameters based on the observations and latent variables dynamically. Such flexibility enables VHRNN to better model sequential data with complex patterns and large variations within and across samples than VRNN models that use fixed weights. VHRNN can be trained with the existing off-the-shelf variational objectives. Experiments on synthetic datasets with different generating patterns show that VHRNN can better disentangle and identify the underlying dynamics and uncertainty in data than VRNN. We also demonstrate the superb parameter-performance efficiency and generalization ability of VHRNN on real-world datasets with different levels of variability and complexity. < l a t e x i t s h a 1 _ b a s e 6 4 = " b A F 9 6 + / 4 h K s 0 W A U e N m S i + + T 3

A g I = " > A A A B 7 n i c b Z D L S s N A F I Y n X m u 9 V V 3 q Y r A I b i y J B a s b K b h x W c F e I A 1 l M p 2 0 Q y e T M H M i l N C H E E R Q E b c + j z v f x k l a R K 0 / D H z 8 / z n M O c e P B d d g 2 5 / W w u L S 8 s p q Y a 2 4 v r G 5 t V 3 a 2 W 3 p K F G U N W k k I t X x i W a C S 9 Y E D o J 1 Y s V I 6 A v W 9 k d X W d 6 + Y 0 r z S N 7 C O G Z e S A a S B 5 w S M F a b 9 l I 4 c S a 9 U t m u 2 L n w P D g z K N c P n h 5 c U S O N X u m j 2 4 9 o E j I J V B C t X c e O w U u J A k 4 F m x S 7 i W Y x o S M y Y K 5 B S U K m v T Q f d 4 K P j N P H Q a T M k 4 B z 9 2 d H S k K t x 6 F v K k M C Q / 0 3 y 8 z / M j e B 4 N x L u Y w T Y J J O P w o S g S H C 2 e 6 4 z x W j I M Y G C F X c z I r p k C h C w V y o m B / h I t P Z 9 8 r z 0 D q t O N V K 9 c Y p 1 y / R V A W 0 j w 7 R M X J Q D d X R N W q g J q J o h O 7 R M 3 q x Y u v

R e r X e p q U L 1 q x n D / 2 S 9 f 4 F 8 f i S Z A = = < / l a t e x i t > c t 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " b A F 9 6 + / 4 h K s 0 W A U e N m S i + + T 3

A g I = " > A A A B 7 n i c b Z D L S s N A F I Y n X m u 9 V V 3 q Y r A I b i y J B a s b K b h x W c F e I A 1 l M p 2 0 Q y e T M H M i l N C H E E R Q E b c + j z v f x k l a R K 0 / D H z 8 / z n M O c e P B d d g 2 5 / W w u L S 8 s p q Y a 2 4 v r G 5 t V 3 a 2 W 3 p K F G U N W k k I t X x i W a C S 9 Y E D o J 1 Y s V I 6 A v W 9 k d X W d 6 + Y 0 r z S N 7 C O G Z e S A a S B 5 w S M F a b 9 l I 4 c S a 9 U t m u 2 L n w P D g z K N c P n h 5 c U S O N X u m j 2 4 9 o E j I J V B C t X c e O w U u J A k 4 F m x S 7 i W Y x o S M y Y K 5 B S U K m v T Q f d 4 K P j N P H Q a T M k 4 B z 9 2 d H S k K t x 6 F v K k M C Q / 0 3 y 8 z / M j e B 4 N x L u Y w T Y J J O P w o S g S H C 2 e 6 4 z x W j I M Y G C F X c z I r p k C h C w V y o m B / h I t P Z 9 8 r z 0 D q t O N V K 9 c Y p 1 y / R V A W 0 j w 7 R M X J Q D d X R N W q g J q J o h O 7 R M 3 q x Y u v

R e r X e p q U L 1 q x n D / 2 S 9 f 4 F 8 f i S Z A = = < / l a t e x i t > c t < l a t e x i t s h a 1 _ b a s e 6 4 = " J I T 

x R B h z N S 1 l C D C t j i R C 5 Q d L R T w = " > A A A B 6 n i c b Z D L S g M x F I b P 1 F t t v d S 6 d B O s Q l d l x o K X j R T c u K x o L 9 A O J Z N m 2 t B M Z k g y Q i n 1 C X T j Q i l u f S J 3 v o 2 Z a R G 1 / h D 4 + P 9 z y D n H i z h T 2 r Y / r c z K 6 t r 6 R n Y z l 9 / a 3 t k t 7 B W b K o w l o Q 0 S 8 l C 2 P a w o Z 4 I 2 N N O c t i N J c e B x 2 v J G V 0 n e u q d S s V D c 6 X F E 3 Q A P B P M Z w d p Y t 6 S n e 4 W S X b F T o W V w F l C q F W e P D + W j f L 1 X + O j 2 Q x I H V G j C s V I d x 4 6 0 O 8 F S M 8 L p N N e N F Y 0 w G e E B 7 R g U O K D K n a S j T t G x c f r I D 6 V 5 Q q P U / d k x w Y F S 4 8 A z l Q H W Q / U 3 S 8 z / s k 6 s / X N 3 w k Q U a y r I / C M / 5 k i H K N k b 9 Z m k R P O x A U w k M 7 M i M s Q S E 2 2 u k 0 u P c J H o 9 H v l Z W i e V J x q p X r j l G q X M F c W D u A Q y u D A G d T g G u r Q A A I D e I I X e L W 4 9 W z N r L d 5 a c Z a 9 O z D L 1 n v X 4 C S k E o = < / l a t e x i t > c t < l a t e x i t s h a 1 _ b a s e 6 4 = " J I T x R B h z N S 1 l C D C t j i R C 5 Q d L R T w = " > A A A B 6 n i c b Z D L S g M x F I b P 1 F t t v d S 6 d B O s Q l d l x o K X j R T c u K x o L 9 A O J Z N m 2 t B M Z k g y Q i n 1 C X T j Q i l u f S J 3 v o 2 Z a R G 1 / h D 4 + P 9 z y D n H i z h T 2 r Y / r c z K 6 t r 6 R n Y z l 9 / a 3 t k t 7 B W b K o w l o Q 0 S 8 l C 2 P a w o Z 4 I 2 N N O c t i N J c e B x 2 v J G V 0 n e u q d S s V D c 6 X F E 3 Q A P B P M Z w d p Y t 6 S n e 4 W S X b F T o W V w F l C q F W e P D + W j f L 1 X + O j 2 Q x I H V G j C s V I d x 4 6 0 O 8 F S M 8 L p N N e N F Y 0 w G e E B 7 R g U O K D K n a S j T t G x c f r I D 6 V 5 Q q P U / d k x w Y F S 4 8 A z l Q H W Q / U 3 S 8 z / s k 6 s / X N 3 w k Q U a y r I / C M / 5 k i H K N k b 9 Z m k R P O x A U w k M 7 M i M s Q S E 2 2 u k 0 u P c J H o 9 H v l Z W i e V J x q p X r j l G q X M F c W D u A Q y u D A G d T g G u r Q A A I D e I I X e L W 4 9 W z N r L d 5 a c Z a 9 O z D L 1 n v X 4 C S k E o = < / l a t e x i t > h t 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " C F X T N 7 B V l S g d 9 c n z u L 1 C q p l y / o k = " > A A A B 7 n i c b Z D L S s N A F I Y n X m u 9 V V 3 q Y r A I b i y J B a s b K b h x W c F e I A 1 l M p 2 0 Q y e T M H M i l N C H E E R Q E b c + j z v f x k l a R K 0 / D H z 8 / z n M O c e P B d d g 2 5 / W w u L S 8 s p q Y a 2 4 v r G 5 t V 3 a 2 W 3 p K F G U N W k k I t X x i W a C S 9 Y E D o J 1 Y s V I 6 A v W 9 k d X W d 6 + Y 0 r z S N 7 C O G Z e S A a S B 5 w S M F Z 7 2 E v h x J n 0 S m W 7 Y u f C 8 + D M o F w / e H p w R Y 0 0 e q W P b j + i S c g k U E G 0 d h 0 7 B i 8 l C j g V b F L s J p r F h I 7 I g L k G J Q m Z 9 t J 8 3 A k + M k 4 f B 5 E y T w L O 3 Z 8 d K Q m 1 H o e + q Q w J D P X f L D P / y 9 w E g n M v 5 T J O g E k 6 / S h I B I Y I Z 7 v j P l e M g h g b I F R x M y u m Q 6 I I B X O h Y n 6 E i 0 x n 3 y v P Q + u 0 4 l Q r 1 R u n X L 9 E U x X Q P j p E x 8 h B N V R H 1 6 i B m o i i E b p H z + j F i q 1 H 6 9 V 6 m 5 Y u W L O e P f R L 1 v s X + a q S a Q = = < / l a t e x i t > h t 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " C F X T N 7 B V l S g d 9 c n z u L 1 C q p l y / o k = " > A A A B 7 n i c b Z D L S s N A F I Y n X m u 9 V V 3 q Y r A I b i y J B a s b K b h x W c F e I A 1 l M p 2 0 Q y e T M H M i l N C H E E R Q E b c + j z v f x k l a R K 0 / D H z 8 / z n M O c e P B d d g 2 5 / W w u L S 8 s p q Y a 2 4 v r G 5 t V 3 a 2 W 3 p K F G U N W k k I t X x i W a C S 9 Y E D o J 1 Y s V I 6 A v W 9 k d X W d 6 + Y 0 r z S N 7 C O G Z e S A a S B 5 w S M F Z 7 2 E v h x J n 0 S m W 7 Y u f C 8 + D M o F w / e H p w R Y 0 0 e q W P b j + i S c g k U E G 0 d h 0 7 B i 8 l C j g V b F L s J p r F h I 7 I g L k G J Q m Z 9 t J 8 3 A k + M k 4 f B 5 E y T w L O 3 Z 8 d K Q m 1 H o e + q Q w J D P X f L D P / y 9 w E g n M v 5 T J O g E k 6 / S h I B I Y I Z 7 v j P l e M g h g b I F R x M y u m Q 6 I I B X O h Y n 6 E i 0 x n 3 y v P Q + u 0 4 l Q r 1 R u n X L 9 E U x X Q P j p E x 8 h B N V R H 1 6 i B m o i i E b p H z + j F i q 1 H 6 9 V 6 m 5 Y u W L O e P f R L 1 v s X + a q S a Q = = < / l a t e x i t > h t < l a t e x i t s h a 1 _ b a s e 6 4 = " z Q v 4 n l d b l C n V f 9 R v U u 5 u P j O r 8 J U = " > A A A B 6 n i c b Z D L S g M x F I Y z 9 V Z b L 7 U u 3 Q S r 0 F W Z s e B l I w U 3 L i v a C 7 R D y a S Z N j S T G Z I z Q i n 1 C X T j Q i l u f S J 3 v o 2 Z a R G 1 / h D 4 + P 9 z y D n H i w T X Y N u f V m Z l d W 1 9 I 7 u Z y 2 9 t 7 + w W 9 o p N H c a K s g Y N R a j a H t F M c M k a w E G w d q Q Y C T z B W t 7 o K s l b 9 0 x p H s o 7 G E f M D c h A c p 9 T A s a 6 H f a g V y j Z F T s V X g Z n A a V a c f b 4 U D 7 K 1 3 u F j 2 4 / p H H A J F B B t O 4 4 d g T u h C j g V L B p r h t r F h E 6 I g P W M S h J w L Q 7 S U e d 4 m P j 9 L E f K v M k 4 N T 9 2 T E h g d b j w D O V A Y G h / p s l 5 n 9 Z J w b / 3 J 1 w G c X A J J 1 / 5 M c C Q 4 i T v X G f K 0 Z B j A 0 Q q r i Z F d M h U Y S C u U 4 u P c J F o t P v l Z e h e V J x q p X q j V O q X a K 5 s u g A H a I y c t A Z q q F r V E c N R N E A P a E X 9 G o J 6 9 m a W W / z 0 o y 1 6 N l H v 2 S 9 f w G I M J B P < / l a t e x i t > h t < l a t e x i t s h a 1 _ b a s e 6 4 = " z Q v 4 n l d b l C n V f 9 R v U u 5 u P j O r 8 J U = " > A A A B 6 n i c b Z D L S g M x F I Y z 9 V Z b L 7 U u 3 Q S r 0 F W Z s e B l I w U 3 L i v a C 7 R D y a S Z N j S T G Z I z Q i n 1 C X T j Q i l u f S J 3 v o 2 Z a R G 1 / h D 4 + P 9 z y D n H i w T X Y N u f V m Z l d W 1 9 I 7 u Z y 2 9 t 7 + w W 9 o p N H c a K s g Y N R a j a H t F M c M k a w E G w d q Q Y C T z B W t 7 o K s l b 9 0 x p H s o 7 G E f M D c h A c p 9 T A s a 6 H f a g V y j Z F T s V X g Z n A a V a c f b 4 U D 7 K 1 3 u F j 2 4 / p H H A J F B B t O 4 4 d g T u h C j g V L B p r h t r F h E 6 I g P W M S h J w L Q 7 S U e d 4 m P j 9 L E f K v M k 4 N T 9 2 T E h g d b j w D O V A Y G h / p s l 5 n 9 Z J w b / 3 J 1 w G c X A J J 1 / 5 M c C Q 4 i T v X G f K 0 Z B j A 0 Q q r i Z F d M h U Y S C u U 4 u P c J F o t P v l Z e h e V J x q p X q j V O q X a K 5 s u g A H a I y c t A Z q q F r V E c N R N E A P a E X 9 G o J 6 9 m a W W / z 0 o y 1 6 N l H v 2 S 9 f w G I M J B P < / l a t e x i t > x t < l a t e x i t s h a 1 _ b a s e 6 4 = " D A O 3 R e b F K G c h X A E r h Z k / 4 5 G K Y B 8 = " > A A A B 6 n i c b Z D L S g M x F I Y z X m v r p d a l m 2 A V u i o z F r x s p O D G Z U V 7 g X Y o m T R t Q z O Z I T k j l q E + g W 5 c K M W t T + T O t z E z L a L W H w I f / 3 8 O O e d 4 o e A a b P v T W l p e W V 1 b z 2 x k c 5 t b 2 z v 5 3 U J D B 5 G i r E 4 D E a i W R z Q T X L I 6 c B C s F S p G f E + w p j e 6 T P L m H V O a B / I W x i F z f T K Q v M 8 p A W P d 3 H e h m y / a Z T s V X g R n D s V q Y f r 4 U D r M 1 b r 5 j 0 4 v o J H P J F B B t G 4 7 d g h u T B R w K t g k 2 4 k 0 C w k d k Q F r G 5 T E Z 9 q N 0 1 E n + M g 4 P d w P l H k S c O r + 7 I i J r / X Y 9 0 y l T 2 C o / 2 a J + V / W j q B / 5 s Z c h h E w S W c f 9 S O B I c D J 3 r j H F a M g x g Y I V d z M i u m Q K E L B X C e b H u E 8 0 c n 3 y o v Q O C 4 7 l X L l 2 i l W L 9 B M G b S P D l A J O e g U V d E V q q E 6 o m i A n t A L e r W E 9 W x N r b d Z 6 Z I 1 7 9 l D v 2 S 9 f w G g k J B f < / l a t e x i t > x t < l a t e x i t s h a 1 _ b a s e 6 4 = " D A O 3 R e b F K G c h X A E r h Z k / 4 5 G K Y B 8 = " > A A A B 6 n i c b Z D L S g M x F I Y z X m v r p d a l m 2 A V u i o z F r x s p O D G Z U V 7 g X Y o m T R t Q z O Z I T k j l q E + g W 5 c K M W t T + T O t z E z L a L W H w I f / 3 8 O O e d 4 o e A a b P v T W l p e W V 1 b z 2 x k c 5 t b 2 z v 5 3 U J D B 5 G i r E 4 D E a i W R z Q T X L I 6 c B C s F S p G f E + w p j e 6 T P L m H V O a B / I W x i F z f T K Q v M 8 p A W P d 3 H e h m y / a Z T s V X g R n D s V q Y f r 4 U D r M 1 b r 5 j 0 4 v o J H P J F B B t G 4 7 d g h u T B R w K t g k 2 4 k 0 C w k d k Q F r G 5 T E Z 9 q N 0 1 E n + M g 4 P d w P l H k S c O r + 7 I i J r / X Y 9 0 y l T 2 C o / 2 a J + V / W j q B / 5 s Z c h h E w S W c f 9 S O B I c D J 3 r j H F a M g x g Y I V d z M i u m Q K E L B X C e b H u E 8 0 c n 3 y o v Q O C 4 7 l X L l 2 i l W L 9 B M G b S P D l A J O e g U V d E V q q E 6 o m i A n t A L e r W E 9 W x N r b d Z 6 Z I 1 7 9 l D v 2 S 9 f w G g k J B f < / l a t e x i t > z t < l a t e x i t s h a 1 _ b a s e 6 4 = " H s E Q c 5 Y D D Z / 2 J S E e f l Y 2 l 1 x U M W Y = " > A A A B 6 n i c b Z D L S g M x F I Y z X m v r p d a l m 2 A V u i o z F r x s p O D G Z U V 7 g X Y o m T R t Q z O Z I T k j 1 K E + g W 5 c K M W t T + T O t z E z L a L W H w I f / 3 8 O O e d 4 o e A a b P v T W l p e W V 1 b z 2 x k c 5 t b 2 z v 5 3 U J D B 5 G i r E 4 D E a i W R z Q T X L I 6 c B C s F S p G f E + w p j e 6 T P L m H V O a B / I W x i F z f T K Q v M 8 p A W P d 3 H e h m y / a Z T s V X g R n D s V q Y f r 4 U D r M 1 b r 5 j 0 4 v o J H P J F B B t G 4 7 d g h u T B R w K t g k 2 4 k 0 C w k d k Q F r G 5 T E Z 9 q N 0 1 E n + M g 4 P d w P l H k S c O r + 7 I i J r / X Y 9 0 y l T 2 C o / 2 a J + V / W j q B / 5 s Z c h h E w S W c f 9 S O B I c D J 3 r j H F a M g x g Y I V d z M i u m Q K E L B X C e b H u E 8 0 c n 3 y o v Q O C 4 7 l X L l 2 i l W L 9 B M G b S P D l A J O e g U V d E V q q E 6 o m i A n t A L e r W E 9 W x N r b d Z 6 Z I 1 7 9 l D v 2 S 9 f w G j n J B h < / l a t e x i t > z t < l a t e x i t s h a 1 _ b a s e 6 4 = " H s E Q c 5 Y D D Z / 2 J S E e f l Y 2 l 1 x U M W Y = " > A A A B 6 n i c b Z D L S g M x F I Y z X m v r p d a l m 2 A V u i o z F r x s p O D G Z U V 7 g X Y o m T R t Q z O Z I T k j 1 K E + g W 5 c K M W t T + T O t z E z L a L W H w I f / 3 8 O O e d 4 o e A a b P v T W l p e W V 1 b z 2 x k c 5 t b 2 z v 5 3 U J D B 5 G i r E 4 D E a i W R z Q T X L I 6 c B C s F S p G f E + w p j e 6 T P L m H V O a B / I W x i F z f T K Q v M 8 p A W P d 3 H e h m y / a Z T s V X g R n D s V q Y f r 4 U D r M 1 b r 5 j 0 4 v o J H P J F B B t G 4 7 d g h u T B R w K t g k 2 4 k 0 C w k d k Q F r G 5 T E Z 9 q N 0 1 E n + M g 4 P d w P l H k S c O r + 7 I i J r / X Y 9 0 y l T 2 C o / 2 a J + V / W j q B / 5 s Z c h h E w S W c f 9 S O B I c D J 3 r j H F a M g x g Y I V d z M i u m Q K E L B X C e b H u E 8 0 c n 3 y o v Q O C 4 7 l X L l 2 i l W L 9 B M G b S P D l A J O e g U V d E V q q E 6 o m i A n t A L e r W E 9 W x N r b d Z 6 Z I 1 7 9 l D v 2 S 9 f w G j n J B h < / l a t e x i t > (W o , U o ) < l a t e x i t s h a 1 _ b a s e 6 4 = " M E I p g 4 J R X D t X J E I N e a + V p f 1 w i z Y = " > A A A B 8 X i c b Z D L S s N A F I Y n 9 V b r r d a l m 8 E i V J C S t O B l V 3 D j s o J p i 0 0 I k + m k H T q Z C T M T o Y Q + g x s 3 L h R x 6 1 v 4 C O 5 8 E P c m a R G 1 / j D w 8 f / n M O c c P 2 J U a d P 8 M A p L y y u r a 8 X 1 0 s b m 1 v Z O e b f S U S K W m N h Y M C F 7 P l K E U U 5 s T T U j v U g S F P q M d P 3 x R Z Z 3 b 4 l U V P B r P Y m I G 6 I h p w H F S K f W T a 3 r i W N o e + L I K 1 f N u p k L L o I 1 h 2 q r 4 t Q + 3 + 6 c t l d + d w Y C x y H h G j O k V N 8 y I + 0 m S G q K G Z m W n F i R C O E x G p J + i h y F R L l J P v E U H q b O A A Z C p o 9 r m L s / O x I U K j U J / b Q y R H q k / m a Z + V / W j 3 V w 5 i a U R 7 E m H M 8 + C m I G t Y D Z + n B A J c G a T V J A W N J 0 V o h H S C K s 0 y O V 8 i O c Z z r 5 X n k R O o 2 6 1 a w 3 r 6 x q q w F m K o J 9 c A B q w A K n o A U u Q R v Y A A M O 7 s E j e D K U 8 W A 8 G y + z 0 o I x 7 9 k D v 2 S 8 f g G o l J N f < / l a t e x i t > (W o , U o ) < l a t e x i t s h a 1 _ b a s e 6 4 = " M E I p g 4 J R X D t X J E I N e a + V p f 1 w i z Y = " > A A A B 8 X i c b Z D L S s N A F I Y n 9 V b r r d a l m 8 E i V J C S t O B l V 3 D j s o J p i 0 0 I k + m k H T q Z C T M T o Y Q + g x s 3 L h R x 6 1 v 4 C O 5 8 E P c m a R G 1 / j D w 8 f / n M O c c P 2 J U a d P 8 M A p L y y u r a 8 X 1 0 s b m 1 v Z O e b f S U S K W m N h Y M C F 7 P l K E U U 5 s T T U j v U g S F P q M d P 3 x R Z Z 3 b 4 l U V P B r P Y m I G 6 I h p w H F S K f W T a 3 r i W N o e + L I K 1 f N u p k L L o I 1 h 2 q r 4 t Q + 3 + 6 c t l d + d w Y C x y H h G j O k V N 8 y I + 0 m S G q K G Z m W n F i R C O E x G p J + i h y F R L l J P v E U H q b O A A Z C p o 9 r m L s / O x I U K j U J / b Q y R H q k / m a Z + V / W j 3 V w 5 i a U R 7 E m H M 8 + C m I G t Y D Z + n B A J c G a T V J A W N J 0 V o h H S C K s 0 y O V 8 i O c Z z r 5 X n k R O o 2 6 1 a w 3 r 6 x q q w F m K o J 9 c A B q w A K n o A U u Q R v Y A A M O 7 s E j e D K U 8 W A 8 G y + z 0 o I x 7 9 k D v 2 S 8 f g G o l J N f < / l a t e x i t > (W i , U i ) < l a t e x i t s h a _ b a s e = " X / Y C Y f O Y N N o R b o k G w M b p z a k = " > A A A B X i c b Z D L S s N A F I Y n V b r r d a l m E i V J C S t O B l V D j s o J p i I k + m k H T q Z h J m J U E K f w Y b F q S B H c + i H s n a R G / j D w f / n M O c c P Z U K t P M A p L y y u r a X s b m v Z O e b f S k V E i M L F x x C L R E k j H J i K o Y c W C o N B n p O u P L K e u E p B G / V p O Y u C E a c h p Q j J S b m p d j x D N H X r l q s c c B G s O V R b F a f + X b n t L y u z O I c B I S r j B D U v Y t M Z u i o S i m J F p y U k k i R E e o y H p a + Q o J N J N m n F A A x h E Q j + u Y O + E h R K O U k H V l i N R I / s y + s n j g z E p j x N F O J F C Q M q g h m M B F Q Q r N t G A s K B V o h H S C C s J F K + R H O M r w I n U b d a t a b V a Q A z F c E + O A A Y I F T A K X o A s g A E H + A R P B n S e D C e j Z d Z a c G Y + y B X z J e v w C W O p N T < / l a t e x i t > (W i , U i ) < l a t e x i t s h a _ b a s e = " X / Y C Y f O Y N N o R b o k G w M b p z a k = " > A A A B X i c b Z D L S s N A F I Y n V b r r d a l m E i V J C S t O B l V D j s o J p i I k + m k H T q Z h J m J U E K f w Y b F q S B H c + i H s n a R G / j D w f / n M O c c P Z U K t P M A p L y y u r a X s b m v Z O e b f S k V E i M L F x x C L R E k j H J i K o Y c W C o N B n p O u P L K e u E p B G / V p O Y u C E a c h p Q j J S b m p d j x D N H X r l q s c c B G s O V R b F a f + X b n t L y u z O I c B I S r j B D U v Y t M Z u i o S i m J F p y U k k i R E e o y H p a + Q o J N J N m n F A A x h E Q j + u Y O + E h R K O U k H V l i N R I / s y + s n j g z E p j x N F O J F C Q M q g h m M B F Q Q r N t G A s K B V o h H S C C s J F K + R H O M r w I n U b d a t a b V a Q A z F c E + O A A Y I F T A K X o A

c J K c X V t f W O z t L X d k l E i M L F x x C L R 8 Z A k j H J i K 6 o Y 6 c S C o N B j p O 0 N z 7 O 8 f U O E p B G / U q O Y O C E K O P U p R k p b 1 5 W 2 G x x B 2 w 0 O 3 V L Z r J q 5 4 C x Y U y g 3 d j 9 f 5 c f L o O m W 3 n r 9 C C c h 4 Q o z J G X X M m P l p E g o i h k Z F 3 u J J D H C Q x S Q r k a O Q i K d N J 9 4 D A + 0 0 4 d + J P T j C u b u z 4 4 U h V K O Q k 9 X h k g N 5 N 8 s M / / L u o n y T 5 2 U 8 j h R h O P J R 3 7 C o I p g t j 7 s U 0 G w Y i M N C A u q Z 4 V 4 g A T C S h + p m B / h L N P x 9 8 q z 0 K p V r X q 1 f m m V G z U w U Q H s g X 1 Q A R Y 4 A Q 1 w A Z r A B h h w c A v u w Y M h j T v j 0 X i

c J K c X V t f W O z t L X d k l E i M L F x x C L R 8 Z A k j H J i K 6 o Y 6 c S C o N B j p O 0 N z 7 O 8 f U O E p B G / U q O Y O C E K O P U p R k p b 1 5 W 2 G x x B 2 w 0 O 3 V L Z r J q 5 4 C x Y U y g 3 d j 9 f 5 c f L o O m W 3 n r 9 C C c h 4 Q o z J G X X M m P l p E g o i h k Z F 3 u J J D H C Q x S Q r k a O Q i K d N J 9 4 D A + 0 0 4 d + J P T j C u b u z 4 4 U h V K O Q k 9 X h k g N 5 N 8 s M / / L u o n y T 5 2 U 8 j h R h O P J R 3 7 C o I p g t j 7 s U 0 G w Y i M N C A u q Z 4 V 4 g A T C S h + p m B / h L N P x 9 8 q z 0 K p V r X q 1 f m m V G z U w U Q H s g X 1 Q A R Y 4 A Q 1 w A Z r A B h h w c A v u w Y M h j T v j 0 X i a l M 4 Z 0 5 4 d 8 E v G 8 x f m r J R N < / l a t e x i t > ĥt 1

< l a t e x i t s h a 1 _ b a s e 6 4 = " 9 7 V Y X N Z b g h 6 a 9 9 g K 7 A y B n e l a w f U

= " > A A A B 9 H i c b V D L S s N A F J 3 4 r P V V d a m L w S K 4 s S Q W r G 6 k 4 M Z l B f u A N J T J d N o M n T y c u S m U k L 8 Q R H C h i F s / x p 1 / 4 y Q t o t Y D F w 7 n 3 M u 9 9 7 i R 4 A p M 8 9 N Y W F x a X l k t r B X X N z a 3 t k s 7 u y 0 V x p K y J g 1 F K D s u U U z w g D W B g 2 C d S D L i u 4 K 1 3 d F V 5 r f H T C o e B r c w i Z j j k 2 H A B 5 w S 0 J L T 9 Q g k X t p L 4 M R K e 6 W y W T F z 4 H l i z U i 5 f v B 4 b 4 s a a f R K H 9 1 + S G O f B U A F U c q 2 z A i c h E j g V L C 0 2 I 0 V i w g d k S G z N Q 2 I z 5 S T 5 E e n + E g r f T w I p a 4 A c K 7 + n E i I r 9 T E d 3 W n T 8 B T f 7 1 M / M + z Y x i c O w k P o h h Y Q K e L B r H A E O I s A d z n k l E Q E 0 0 I l V z f i q l H J K G g c y r m I V x k O P t + e Z 6 0 T i t W t V K 9 s c r 1 S z R F A e 2 j Q 3 S M L F R D d X S N G q i J K L p D D + g Z v R h j 4 8 l 4 N d 6 m r Q v G b G Y P / Y L x / g X T S 5 U 2 < / l a t e x i t > ĥt 1

< l a t e x i t s h a 1 _ b a s e 6 4 = " 9 7 V Y X N Z b g h 6 a 9 9 g K 7 A y B n e l a w f U 

= " > A A A B 9 H i c b V D L S s N A F J 3 4 r P V V d a m L w S K 4 s S Q W r G 6 k 4 M Z l B f u A N J T J d N o M n T y c u S m U k L 8 Q R H C h i F s / x p 1 / 4 y Q t o t Y D F w 7 n 3 M u 9 9 7 i R 4 A p M 8 9 N Y W F x a X l k t r B X X N z a 3 t k s 7 u y 0 V x p K y J g 1 F K D s u U U z w g D W B g 2 C d S D L i u 4 K 1 3 d F V 5 r f H T C o e B r c w i Z j j k 2 H A B 5 w S 0 J L T 9 Q g k X t p L 4 M R K e 6 W y W T F z 4 H l i z U i 5 f v B 4 b 4 s a a f R K H 9 1 + S G O f B U A F U c q 2 z A i c h E j g V L C 0 2 I 0 V i w g d k S G z N Q 2 I z 5 S T 5 E e n + E g r f T w I p a 4 A c K 7 + n E i I r 9 T E d 3 W n T 8 B T f 7 1 M / M + z Y x i c O w k P o h h Y Q K e L B r H A E O I s A d z n k l E Q E 0 0 I l V z f i q l H J K G g c y r m I V x k O P t + e Z 6 0 T i t W t V K 9 s c r 1 S z R F A e 2 j Q 3 S M L F R D d X S N G q i J K L p D D + g Z v R h j 4 8 l 4 N d 6 m r Q v G b G Y P / Y L x / g X T S 5 U 2 < / l a t e x i t > ĥt < l a t e x i t s h a 1 _ b a s e 6 4 = " h K f r q k j x y q 2 B O F t x d v Y 3 C H q b Q R I = " > A A A B 8 H i c b Z D L S s N A F I Y n 9 V Z b L 7 U u 3 Q x W o a u S W P C y k Y I b l x X s R d p Q J t N J M 3 Q y C T M n Q g n 1 C d y 5 U V D E

M E O y Y K O B U s G m h H 2 k W E j o m I 9 Z L U B K f a T v O B p 7 i w 8 Q Z Y j d Q y Z O A M / d n R 0 x 8 r S e + k 1 T 6 B D z 9 N 0 v N / 7 J e B O 6 p H X M Z R s A k n X 3 k R g J D g N P t 8 Z A r R k F M E i B U 8 W R W T D 2 i C I X k R o X s C G e p j r 9 X X o T 2 U c 2 q 1 + p X V q V x j m b K o z 2 0 j 6 r I Q i e o g S 5 R E 7 U Q R T 5 6 Q M / o x V D G o / F q v M 1 K c 8 a 8 Z x f 9 k v H + B V d d k x w = < / l a t e x i t > ĥt < l a t e x i t s h a 1 _ b a s e 6 4 = " h K f r q k j x y q 2 B O F t x d v Y 3 C H q b Q R I = " > A A A B 8 H i c b Z D L S s N A F I Y n 9 V Z b L 7 U u 3 Q x W o a u S W P C y k Y I b l x X s R d p Q J t N J M 3 Q y C T M n Q g n 1 C d y 5 U V D E r Y / j z r c x S Y u o 9 Y e B j / 8 / h z n n O K H g G k z z 0 8 g t L a + s r u X X C 8 W N z a 3 t 0 k 6 5 r Y N I U d a i g Q h U 1 y G a C S 5 Z C z g I 1 g 0 V I 7 4 j W M c Z X 6 R 5 5 5 Y p z Q N 5 D Z O Q 2 T 4 Z S e 5 y S i C x b v o e g d i b D m B Q q p g 1 M x N e B G s O l U b 5 6 f 6 u e l B s D k o f / W F A I 5 9 J o I J o 3 b P M E O y Y K O B U s G m h H 2 k W E j o m I 9 Z L U B K f a T v O B p 7 i w 8 Q Z Y j d Q y Z O A M / d n R 0 x 8 r S e + k 1 T 6 B D z 9 N 0 v N / 7 J e B O 6 p H X M Z R s A k n X 3 k R g J D g N P t 8 Z A r R k F M E i B U 8 W R W T D 2 i C I X k R o X s C G e p j r 9 X X o T 2 U c 2 q 1 + p X V q V x j m b K o z 2 0 j 6 r I Q i e o g S 5 R E 7 U Q R T 5 6 Q M / o x V D G o / F q v M 1 K c 8 a 8 Z x f 9 k v H + B V d d k x w = < / l a t

(W ⇤ , U ⇤ ) < l a t e x i t s h a 1 _ b a s e 6 4 = " S k 5 Z s + R c Y U L s 1 Q J O M h N v a 9 A 1 F G k = " > A A A B 8 X i c b Z D L S s N A F I Y n 9 V b r r d a l m 8 E i 1 C I l a c H L r u D G Z Q X T F p s Q J t N p O 3 Q y C T M T o Y Q + g x s 3 L h R x 6 1 v 4 C O 5 8 E P d O 0 i J q / W H g 4 / / P Y c 4 5 f s S o V K b 5 Y e S W l l d W 1 / L r h Y 3 N r e 2 d 4 m 6 p L c N Y Y G L j k I W i 6 y N J G O X E V l Q x 0 o 0 E Q Y H P S M c f X 6 R 5 5 5 Y I S U N + r S Y R c Q M 0 5 H R A M V L a u q l 0 v O o x t L 3 q k V c s m z U z E 1 w E a w 7 l Z s m p f L 7 d O S 2 v + O 7 0 Q x w H h C v M k J Q 9 y 4 y U m y C h K G Z k W n B i S S K E x 2 h I e h o 5 C o h 0 k 2 z i K T z U T h 8 O Q q E f V z B z f 3 Y k K J B y E v i 6 M k B q J P 9 m q f l f 1 o v V 4 M x N K I 9 i R T i e f T S I G V Q h T N e H f S o I V m y i A W F B 9 a w Q j 5 B A W O k j F b I j n K c 6 + V 5 5 E d r 1 m t W o N a 6 s c r M O Z s q D f X A A K s A C p 6 A J L k E L 2 A A D D u 7 B I 3 g y p P F g P B s v s 9 K c M e / Z A 7 9 k v H 4 B 1 X q S 1 Q = = < / l a t e x i t > (W ⇤ , U ⇤ ) < l a t e x i t s h a 1 _ b a s e 6 4 = " S k 5 Z s + R c Y U L s 1 Q J O M h N v a 9 A 1 F G k = " > A A A B 8 X i c b Z D L S s N A F I Y n 9 V b r r d a l m 8 E i 1 C I l a c H L r u D G Z Q X T F p s Q J t N p O 3 Q y C T M T o Y Q + g x s 3 L h R x 6 1 v 4 C O 5 8 E P d O 0 i J q / W H g 4 / / P Y c 4 5 f s S o V K b 5 Y e S W l l d W 1 / L r h Y 3 N r e 2 d 4 m 6 p L c N Y Y G L j k I W i 6 y N J G O X E V l Q x 0 o 0 E Q Y H P S M c f X 6 R 5 5 5 Y I S U N + r S Y R c Q M 0 5 H R A M V L a u q l 0 v O o x t L 3 q k V c s m z U z E 1 w E a w 7 l Z s m p f L 7 d O S 2 v + O 7 0 Q x w H h C v M k J Q 9 y 4 y U m y C h K G Z k W n B i S S K E x 2 h I e h o 5 C o h 0 k 2 z i K T z U T h 8 O Q q E f V z B z f 3 Y k K J B y E v i 6 M k B q J P 9 m q f l f 1 o v V 4 M x N K I 9 i R T i e f T S I G V Q h T N e H f S o I V m y i A W F B 9 a w Q j 5 B A W O k j F b I j n K c 6 + V 5 5 E d r 1 m t W o N a 6 s c r M O Z s q D f X A A K s A C p 6 A J L k E L 2 A A D D u 7 B I 3 g y p P F g P B s v s 9 K c M e / Z A 7 9 k v H 4 B 1 X q S 1 Q = = < / l a t e x i t >

weights in primary network that are dynamically modified Fig. 7 and Fig. 8 show the qualitative study results of VHRNN and VRNN under the ADD and ZERO-SHOT settings. We can see that the KL divergence of VHRNN model decreases as it observes more data. Meanwhile the mean predictions by VHRNN stay relatively close to the actual target value as shown in Fig. 7e and Fig. 8e . The prediction are especially accurate in the ADD setting as Fig. 7e shows. The results demonstrate VHRNN's ability to identify system dynamics can generalize to unseen data generation patterns. By contrast, we does not see any trend of variational RNN that indicates it is capable of doing dynamic regime identification. Fig. 9 illustrates the qualitative study of VHRNN and VRNN under the LONG setting. The magnitude of the data grows rapidly in such setting due to exponential growth and it is well beyond the scale of training data. We can see both VRNN and VHRNN make very inaccurrate mean predictions that are far from the target values as Fig. 9e shows. However, VHRNN pays smaller reconstruction cost than VRNN by also making large predictions of variance. This setting demonstrates a special case in which VHRNN has better ability to handle uncertainty in data than vanilla variational RNN. 

C VRNN AND VHRNN IMPLEMENTATION DETAILS:

Encoder Network and Prior Network The architecture of the encoder in Equation 5 is the same in VHRNN and VRNN. For synthetic datasets, the encoder is implemented by a fully-connected network with two hidden layers; each hidden layer has the same number of units as the latent variable dimension. For other real-world datasets, we use a fully-connected network with one hidden layer.

The number of units is also the same as the latent dimension. The prior network is implemented by a similar architecture as the encoder, with the only difference being the dimension of inputs.

Decoder Network For VHRNN models, we use fully-connected hyper networks with two hidden layers for synthetic data and fully-connected hyper networks with one hidden layer for other datasets as the decoder networks. The number of units in each hidden layer is also the same as the latent variable defined in Equation 2. For each layer of the hyper networks, the weight scaling vector and bias is generated by an two-layer MLP. The hidden layer size of this MLP is 8 for synthetic dataset and 64 for real-world datasets. For VRNN models, we use plain feed-forward networks for decoder.

The number of hidden layers and units in the hidden layer are determined in the same way as VHRNN models.

Observation and Latent Variable Encoding Network For fair comparison with VRNN Chung et al. (2015) , the latent variable and observations are encoded by a network (different from the encoder in Equation 5) before being fed to the recurrence network and encoder. The latent and observation encoding networks have the same architecture except for the input dimension in each experiment setting. For synthetic datasets, the encoding network is implemented by a fully-connected network with two hidden layers. For real-world datasets, we use a fully-connected network with one hidden layer. The number of units in each hidden layer is the same as the dimension of latent variable in that experiment setting. HT Sensor The HT Sensor dataset collects readings from 11 sensors under certain stimulus in an experiment. The readings of the sensors are recorded at a rate of once per second. We segment a sequence of 3000 seconds every 1000 seconds in the dataset and downsample the sequence by a rate of 30. Each sequence we obtained has a fixed length of 100. The types of sequences include pure background noise, stimulus before and after background noise and stimulus between two periods of background noise. The data are normalized to zero mean and unit variance along each dimension. We use 532 sequences for training, 68 sequences for validation and 74 sequences for testing.

E TRAINING AND EVALUATION DETAILS ON REAL-WORLD DATASETS.

For all the real-world data, the models, both VRNN and VHRNN, are trained with batch size of 4 and particle size of 4. When evaluating the models, we use particle size of 128 for polyphonic music datasets and 1024 for Stock and HT Sensor datasets.

F VHRNN AND VRNN HIDDEN UNIT VERSUS PERFORMANCE COMPARISON:

We also compare VHRNN and VRNN by plotting the models' performance against their number of hidden units. The models we considered here are the same as the models presented in Fig. 5 : We use a single-layer LSTM model for the RNN part; the dimension of LSTM's hidden state is the same as the latent dimension. It's worth noting that as VHRNN uses two LSTM models, one primary network and one hyper network. Therefore, the number of hidden units in an VHRNN model is twice the number of latent dimension. We can see that VHRNN also dominates the performance of VRNN with a similar or fewer number of hidden units in most of the settings. Furthermore, the fact that VHRNN almost always outperforms VRNN for all parameter or hidden unit sizes precisely shows the superiority of the new architecture. The results from Fig. 5 and Fig. 10 are consolidated in into Tab. 4. et al. (2016) and delivers promising performance. It also seems like a natural choice as the hidden state of the primary RNN can represent the history of observed data while the hidden state of the hyper RNN can track the history of data generation dynamics. However, it is still intriguing to study other design choices that do not have the recurrence structure in the hyper networks for VHRNN. As an ablation study, we experimented with VHRNN models that replace the RNN with a three-layer feed-forward network as the hyper network θ for the recurrence model g as defined in Equation 6. We keep the other components of VHRNN unchanged on JSB Chorale, Stock and the synthetic dataset. The evaluation results using FIVO are presented in Tab. 6 and systematic generalization study results on the synthetic dataset are shown in Tab. 7. We denote the original VHRNN with recurrence structure in θ as VHRNN-RNN and the variant without the recurrence structure as VHRNN-MLP.

As we can see, given the same latent dimension, VHRNN-MLP models have more parameters than VHRNN-RNN models. VHRNN-MLP can have slightly better performance than VHRNN-RNN in some cases but it performs worse than VHRNN-RNN in more settings. The performance of VHRNN-MLP also degrades faster than VHRNN-RNN on the JSB Chorale dataset as we increase the latent dimension. Moreover, systematic generalization study on the synthetic dataset also shows that VHRNN-MLP has worse performance than VHRNN-RNN no matter in the test setting or in the systematically varied settings. 

:



Figures and Tables:

Figure 1: Diagrams of the variational hyper RNN. Operators are indicated by arrows in different colors, and dashed lines and boxes represent the hypernetwork components. (a) Prior distribution in Eq. 3. (b) Recurrent model in Eq. 1. (c) Generative model in Eq. 2. (d) Inference model in Eq. 5. (e)The overall computational path. The hypernetwork components are left out.

(i.e., W = I), leading to long sequences of pure random walks with switching magnitudes of noise. The standard deviation of the additive noise randomly switches up to 3 times within {0.25, 1, 4} in one sequence. • LONG In this setting, we generate extra-long NOISELESS sequences with twice the total number of steps using the same set of predefined weights. The data scale can exceed well beyond the range of training data when exponential growth happens. • ZERO-SHOT In this setting, NOISELESS sequences are generated such that the training data and test data use different sets of weight matrices. • ADD In this setting, sequences are generated by a different recurrence rule: x t = x t-1 + b, where b and x 0 are uniformly sampled from [0, 1] 2 .

Figure 2: Qualitative study of VRNN and VHRNN under the NOISELESS setting. (a) and (b) show the values of concatenated data at each time step. (c) shows the KL divergence between the variational posterior and the prior of the latent variable at each time step for VHRNN. (d) shows the KL divergence for VRNN. (e) shows L2 distance between the predicted mean values by VHRNN and VRNN and the target. (f) shows the predicted log-variance of the output distribution for VRNN and VHRNN.

Figure 5: VRNN and VHRNN parameter-performance comparison.

s g A E H + A R P B n S e D C e j Z d Z a c G Y + y B X z J e v w C W O p N T < / l a t e x i t > (W f , U f ) < l a t e x i t s h a 1 _ b a s e 6 4 = " D Y t A c J N 2 M T d 9 l T g I w O 6 m q m x 3 5 b 4 = " > AA A B 8 X i c b Z D L S s N A F I Z P v N Z 6 q 5 e d m 8 E i V J C S t O B l V 3 D j s o J p i 2 0 I k + m k H T q Z h J m J U E L f w o 0 L R d y 6 F F / E n a / g 3 r 1 J W 0 S t P w x 8 / P 8 5 z D n H i z h T 2 j T f j b n 5 h c W l 5 d x K f n V t f W O z s L X d U G E s C b V J y E P Z 8 r C i n A l q a 6 Y 5 b U W S 4 s D j t O k N z r O 8 e U O l Y q G 4 0 s O I O g H u C e Y z g n V q X Z e a r n + E b N c / d A t F s 2 y O h W b B m k K x t v v 5 q j 5 e + n W 3 8 N b p h i Q O q N C E Y 6 X a l h l p J 8 F S M 8 L p K N + J F Y 0 w G e A e b a c o c E C V k 4 w n H q G D 1 O k i P 5 T p E x q N 3 Z 8 d C Q 6 U G g Z e W h l g 3 V d / s 8 z 8 L 2 v H 2 j 9 1 E i a i W F N B J h / 5 M U c 6 R N n 6 q M s k J Z o P U 8 B E s n R W R P p Y Y q L T I + X H R z j L d P y 9 8 i w 0 K m W r W q 5 e W s V a B S b K w R 7 s Q w k s O I E a X E A d b C A g 4 B b u 4 c F Q x p 3 x a D x N S u e M a c 8 O / J L x / A X j n Z R L < / l a t e x i t > (W f , U f ) < l a t e x i t s h a 1 _ b a s e 6 4 = " D Y t A c J N 2 M T d 9 l T g I w O 6 m q m x 3 5 b 4 = " > A A A B 8 X i c b Z D L S s N A F I Z P v N Z 6 q 5 e d m 8 E i V J C S t O B l V 3 D j s o J p i 2 0 I k + m k H T q Z h J m J U E L f w o 0 L R d y 6 F F / E n a / g 3 r 1 J W 0 S t P w x 8 / P 8 5 z D n H i z h T 2 j T f j b n 5 h c W l 5 d x K f n V t f W O z s L X d U G E s C b V J y E P Z 8 r C i n A l q a 6 Y 5 b U W S 4 s D j t O k N z r O 8 e U O l Y q G 4 0 s O I O g H u C e Y z g n V q X Z e a r n + E b N c / d A t F s 2 y O h W b B m k K x t v v 5 q j 5 e + n W 3 8 N b p h i Q O q N C E Y 6 X a l h l p J 8 F S M 8 L p K N + J F Y 0 w G e A e b a c o c E C V k 4 w n H q G D 1 O k i P 5 T p E x q N 3 Z 8 d C Q 6 U G g Z e W h l g 3 V d / s 8 z 8 L 2 v H 2 j 9 1 E i a i W F N B J h / 5 M U c 6 R N n 6 q M s k J Z o P U 8 B E s n R W R P p Y Y q L T I + X H R z j L d P y 9 8 i w 0 K m W r W q 5 e W s V a B S b K w R 7 s Q w k s O I E a X E A d b C A g 4 B b u 4 c F Q x p 3 x a D x N S u e M a c 8 O / J L x / A X j n Z R L < / l a t e x i t > (W g , U g )< l a t e x i t s h a 1 _ b a s e 6 4 = " M y C L m v e 5 N 5 + A t h S / e y B t 4 o s n U D g = " > A A A B 8 X i c b Z D L S s N A F I Y n X m u 9 1 c v O z W A R K k h J W v C y K 7 h x W c G 0 x T a E y X S S D p 1 M w s x E K K F v 4 c a F I m 5 d i i / i z l d w 7 9 5 J W k S t P w x 8 / P 8 5 z D n H i x m V y j T f j b n 5 h c W l 5

a l M 4 Z 0 5 4 d 8 E v G 8 x f m r J R N < / l a t e x i t > (W g , U g ) < l a t e x i t s h a 1 _ b a s e 6 4 = " M y C L m v e 5 N 5+ A t h S / e y B t 4 o s n U D g = " > A A A B 8 X i c b Z D L S s N A F I Y n X m u 9 1 c v O z W A R K k h J W v C y K 7 h x W c G 0 x T a E y X S S D p 1 M w s x E K K F v 4 ca F I m 5 d i i / i z l d w 7 9 5 J W k S t P w x 8 / P 8 5 z D n H i x m V y j T f j b n 5 h c W l 5

r Y / j z r c x S Y u o 9 Y e B j / 8 / h z n n O K H g G k z z 0 8 g t L a + s r u X X C 8 W N z a 3 t 0 k 6 5 r Y N I U d a i g Q h U 1 y G a C S 5 Z C z g I 1 g 0 V I 7 4 j W M c Z X 6 R 5 5 5 Y p z Q N 5 D Z O Q 2 T 4 Z S e 5 y S i C x b v o e g d i b D m B Q q p g 1 M x N e B Gs O l U b 5 6 f 6 u e l B s D k o f / W F A I 5 9 J o I J o 3 b P

Figure 6: Implementation of the recurrence model in VHRNN using LSTM cell.

Figure 8: Qualitative study of VRNN and VHRNN under the ZERO-SHOT setting. The layout of subfigures is the same as Fig. 7.

REAL-WORLD DATASETS PREPROCESSING DETAILSPolyphonic Music Each sample in the polyphonic music datasets, JSB Chorale and Piano-midi.de is represented as a sequence of 88-dimensional binary vectors. The data are preprocessed by meancentering along each dimension per dataset.Stock We randomly select 345 companies and use their daily stock price and volume in the first half of 2008 to obtain training data. We another 50 companies' data in the second half of 2008 to generate validation set and get the test set from the remaining 50 companies during the second half of 2008. The sequences are first preprocessed by taking log ratio of the values between consecutive days. Each sequence has a fixed length of 125. The log ratio sequences are normalized using the mean and standard deviation of the training set along each dimension.

Fig.11shows the parameter performance plots of VHRNN and VRNN using GRU implementation on the JSB Chorale dataset. VHRNN models consistently outperform VRNN models under all settings.

Figure 11: VRNN and VHRNN arameter-performance comparison using GRU implementation on JSB Chorale dataset.

Evaluation results on synthetic datasets.

Table content: <table><tr><td>Model</td><td>Z dim.</td><td>Param.</td><td/><td/><td colspan="3">FIVO estimated log likelihood per time step</td><td/><td/></tr><tr><td/><td/><td/><td>Test</td><td>NOISELESS</td><td>SWITCH</td><td>RAND</td><td>LONG</td><td>ZERO-SHOT</td><td>ADD</td></tr><tr><td>VRNN VRNN VRNN VHRNN</td><td>8 6 4 4</td><td>2612 1516 716 1568</td><td>-5.43 -5.80 -7.88 -4.68</td><td>-2.50 -3.66 -5.25 -2.08</td><td>-334173 -19735 -7.85 -4.27</td><td>-5.02 -5.24 -6.81 -3.91</td><td>-1033348 -27200 -5777 -3005</td><td>-3.64 -4.39 -5.35 -2.57</td><td>-3.57 -5.09 -8.67 -2.62</td></tr></table>

VRNN and VHRNN with same latent dimensions.

Table content: <table><tr><td>Model</td><td>Z dim.</td><td>Hidden dim.</td><td>Hyper size</td><td>Param.</td><td>FIVO</td></tr><tr><td>VRNN</td><td>24 28 32</td><td>24 28 32</td><td>---</td><td>23k 31k 39k</td><td>-7.04 -6.99 -6.91</td></tr><tr><td>VHRNN</td><td>24 28 32</td><td>12 14 16</td><td>16 18 20</td><td>24k 31k 39k</td><td>-6.92 -6.73 -6.70</td></tr></table>

Ablation study with different hyper network inputs.

Table content: <table><tr><td>Dataset</td><td>Z dim.</td><td>Hyper Input</td><td>FIVO</td></tr><tr><td>JSB</td><td>14 14 14 32 32 32</td><td>latent only hidden only latent+hidden latent only hidden only latent+hidden</td><td>-6.68 -6.71 -6.76 -6.76 -7.03 -6.82</td></tr><tr><td>Synthetic Test</td><td>4 4 4</td><td>latent only hidden only latent+hidden</td><td>-5.01 -4.79 -4.68</td></tr></table>

e x i t >

Table content: <table><tr><td>Legend</td></tr><tr><td>activation</td></tr><tr><td>pointwise operation</td></tr><tr><td>layers in primary</td></tr><tr><td>network</td></tr><tr><td>layers in hyper</td></tr><tr><td>network</td></tr></table>

VRNN (left)  and VHRNN (right) on Real-World Datasets.

Table content: <table><tr><td>Dataset</td><td>Z dim.</td><td>Hidden Unit</td><td>Param.</td><td>FIVO</td><td>Dataset</td><td>Z dim.</td><td>Hidden Unit</td><td>Param.</td><td>FIVO</td></tr><tr><td>JSB Piano</td><td/><td>64 56 48 32 28 24 80 72 64 56 48</td><td>144k 111k 83k 39k 31k 23k 220k 180k 144k 111k 83k</td><td>-7.25 -7.21 -7.12 -6.92 -6.99 -7.04 -7.85 -7.85 -7.83 -7.86 -7.86</td><td>JSB Piano</td><td>42 40 32 24 16 14 12 48 40 32 24</td><td>84 80 64 48 32 28 24 96 80 64 48</td><td>135k 125k 88k 58k 35k 31k 27k 169k 125k 88k 58k</td><td>-6.90 -6.87 -6.82 -6.86 -6.81 -6.76 -6.80 -7.48 -7.51 -7.33 -7.47</td></tr><tr><td/><td/><td>20</td><td>13k</td><td>9.20</td><td/><td>12</td><td>24</td><td>16k</td><td>9.33</td></tr><tr><td>Stock</td><td/><td>24 28</td><td>20k 27k</td><td>9.21 9.20</td><td>Stock</td><td>14 16</td><td>28 32</td><td>18k 23k</td><td>9.34 9.27</td></tr><tr><td/><td/><td>20</td><td>14k</td><td>6.13</td><td/><td>18</td><td>36</td><td>28k</td><td>9.31</td></tr><tr><td>HT Sensor</td><td/><td>24 28</td><td>20k 26k</td><td>13.70 12.37</td><td>HT Sensor</td><td>12 16</td><td>24 32</td><td>16k 24k</td><td>8.16 14.41</td></tr><tr><td/><td/><td>32</td><td>34k</td><td>12.69</td><td/><td>20</td><td>40</td><td>34k</td><td>12.45</td></tr><tr><td colspan="10">G VHRNN AND VRNN PERFORMANCE-PARAMETER COMPARISON USING</td></tr><tr><td colspan="5">GRU ON JSB CHORALE DATASET</td><td/><td/><td/><td/><td/></tr></table>

HyperLSTM Performance in Log Likelihood Dataset Hidden Units Gauss. Components Param. Log Likelihood

Table content: <table><tr><td/><td>24</td><td>N.A.</td><td>9k</td><td>-9.15</td></tr><tr><td/><td>28</td><td>N.A.</td><td>11k</td><td>-8.91</td></tr><tr><td/><td>32</td><td>N.A.</td><td>14k</td><td>-8.87</td></tr><tr><td/><td>48</td><td>N.A.</td><td>30k</td><td>-8.94</td></tr><tr><td>JSB</td><td>64 80</td><td>N.A. N.A.</td><td>49k 75k</td><td>-8.85 -8.77</td></tr><tr><td/><td>84</td><td>N.A.</td><td>83k</td><td>-8.77</td></tr><tr><td/><td>96</td><td>N.A.</td><td>106k</td><td>-8.74</td></tr><tr><td/><td>104</td><td>N.A.</td><td>124k</td><td>-8.85</td></tr><tr><td/><td>112</td><td>N.A.</td><td>143k</td><td>-8.67</td></tr><tr><td/><td>24</td><td>3</td><td>7k</td><td>8.28</td></tr><tr><td/><td>28</td><td>3</td><td>9k</td><td>8.33</td></tr><tr><td/><td>32</td><td>3</td><td>12k</td><td>8.42</td></tr><tr><td/><td>36</td><td>3</td><td>15k</td><td>8.35</td></tr><tr><td/><td>40</td><td>3</td><td>18k</td><td>8.29</td></tr><tr><td>Stock</td><td>48</td><td>3</td><td>26k</td><td>7.91</td></tr><tr><td/><td>24</td><td>5</td><td>7k</td><td>8.61</td></tr><tr><td/><td>28</td><td>5</td><td>9k</td><td>8.44</td></tr><tr><td/><td>32</td><td>5</td><td>12k</td><td>8.44</td></tr><tr><td/><td>36</td><td>5</td><td>15k</td><td>8.47</td></tr><tr><td/><td>40</td><td>5</td><td>19k</td><td>8.42</td></tr><tr><td/><td>48</td><td>5</td><td>26k</td><td>8.58</td></tr><tr><td colspan="5">I VHRNN WITHOUT TEMPORAL STRUCTURE IN HYPER NETWORKS</td></tr></table>

Parameter-Performance Comparison of VHRNN-RNN and VHRNN-MLP

Table content: <table><tr><td>Dataset</td><td>Z dim.</td><td>VHRNN-MLP Param.</td><td>VHRNN-RNN Param.</td><td>VHRNN-MLP FIVO</td><td>VHRNN-RNN FIVO</td></tr><tr><td>JSB</td><td/><td>146k 135k 94k 62k 37k 32k 28k</td><td>135k 125k 88k 58k 35k 31k 27k</td><td>-7.02 -7.08 -7.13 -6.83 -6.76 -6.73 -6.83</td><td>-6.90 -6.87 -6.82 -6.86 -6.81 -6.76 -6.80</td></tr><tr><td/><td/><td>16k</td><td>16k</td><td>9.47</td><td>9.33</td></tr><tr><td>Stock</td><td/><td>19k 24k</td><td>18k 23k</td><td>9.21 9.31</td><td>9.34 9.27</td></tr><tr><td/><td/><td>29k</td><td>28k</td><td>9.19</td><td>9.31</td></tr><tr><td>Syn-Test</td><td>4</td><td>1676</td><td>1568</td><td>-5.13</td><td>-4.68</td></tr></table>

Systematic Generalization Study on VHRNN-RNN and VHRNN-MLP

Table content: <table><tr><td>Model</td><td>Z dim.</td><td>Param.</td><td/><td/><td colspan="4">FIVO estimated log likelihood per time step</td></tr><tr><td/><td/><td/><td>Test</td><td>NOISELESS</td><td>SWITCH</td><td>RAND</td><td>LONG</td><td>ZERO-SHOT</td><td>ADD</td></tr><tr><td>VHRNN-MLP VHRNN-RNN</td><td/><td>1676 1568</td><td>-5.13 -4.68</td><td>-2.73 -2.08</td><td>-5.22 -4.27</td><td>-4.12 -3.91</td><td>-604937 -3005</td><td>-2.94 -2.57</td><td>-3.84 -2.62</td></tr></table>

Peer review (iclr2020):
This paper proposes a variational hyper recurrent neural network which is a combination of the variational RNN and the hypernetwork. The hypernetwork is an RNN whose output modifies the parameters of the variational RNN dynamically at runtime. Overall, this seems like an extension of the idea of using a hypernetwork with the VRNN (rather than the RNN as done in Ha. et. al). The model is trained via the FIVO objective. The model and learning algorithm are compared to the variational RNN and tested on a variety of synthetic settings where the VHRNN outperforms the VRNN in held-out likelihood. The performance gains are investigated on synthetic datasets where the paper notes that the VHRNN is often quicker to adapt variations that happen within seqences (for example, the paper considers a dataset where multiple patterns are stitched together into a sequence and study the changes in the KL divergence and reconstruction at switch points). On four real-world sequential datasets, the paper finds that the model outperforms the VRNN across many configurations and with a fewer number of parameters. Summary: I don*t think the model presented here is very novel, in that it is a combination of existing ideas; however, the paper does a good job of studying the model in a variety of different configurations on both synthetic and real-world data. The model does appear to consistently outperform the Variational RNN of Chung et. al. Questions and comments: (a) I do not think the word *System Identification* should be used on page 5 to describe the results of Figure 2. Doing so would overload existing notation in the time series literature where the word refers to the identification of parameters under a pre-specified physical system. (b) How well does the Recurrent Hyper network (with no latent variable) do on the tasks considered here? I understand that it may be a less expressive model in general, but it is not clear to me why it would not be a competitive baseline on some of the smaller datasets considered here -- was this baseline tried? (c) Did you experiment with non-temporal architectures for the hypernetwork? Since z_t and h_t-1 (which are conditioned on) contain information about the history of the sequence, one might argue that conditioning on them might suffice to predict the modifications to the parameters of the theta and g.