Variational Hyper RNN for Sequence Modeling
Authors: Ruizhi Deng
Yanshuai Cao
Bo Chang
Leonid Sigal
Greg Mori
Marcus Brubaker
Venue:  (2020)
Keywords: variational autoencoder
hypernetwork
recurrent neural network
time series

Abstract: In this work, we propose a novel probabilistic sequence model that excels at capturing high variability in time series data, both across sequences and within an individual sequence. Our method uses temporal latent variables to capture information about the underlying data pattern and dynamically decodes the latent information into modifications of weights of the base decoder and recurrent model. The efficacy of the proposed method is demonstrated on a range of synthetic and real-world sequential data that exhibit large scale variations, regime shifts, and complex dynamics.

Under review as a conference paper at ICLR 2020
VARIATIONAL HYPER RNN FOR SEQUENCE MODEL- ING
Anonymous authors Paper under double-bling review
ABSTRACT
In this work, we propose a novel probabilistic sequence model that excels at capturing high variability in time series data, both across sequences and within an individual sequence. Our method uses temporal latent variables to capture information about the underlying data pattern and dynamically decodes the latent information into modifications of weights of the base decoder and recurrent model. The efficacy of the proposed method is demonstrated real-world sequential data that exhibit large scale variations, regime shifts, and complex dynamics.
range of synthetic and
INTRODUCTION
Recurrent neural networks (RNNs) are the natural architecture for sequential data they can handle variable-length input and output sequences. Initially invented for natural language processing, long short-term memory (LSTM; Hochreiter & Schmidhuber 1997). gated recurrent unit (GRU; Cho
et al. 2014) as well as the later autention-augmentte versions (Vaswani et al.] 2017) have found wide-spread successes from language modeling (Mikolov et al. 2010 Kiros al. [2015] Jozefowicz et al. 2016) and machine translation (Bahdanau et al./ (0014) to speech recognition (Graves et al. 2013) and recommendation systems (Wu al.] 2017p. However, RNNs use deterministic hidden states to process input sequences and model the system dynamics using a set of inee-invriinnt weights, and they do not necessarily have the right inductive bias for time series data outside the originally intended domains.
Many natural systems have complex feedback mechanisms and numerous exogenous sources of variabilities. Observations from such systems would contain large variations both across sequences in a dataset as well as within any single sequence; the dynamics could be switching regimes drastically, and the noise process could also be heteroskedastic. To capture all these intricate patterns in RNN with deterministic hidden states and a fixed set weights requires learning about the patterns, the subtle deviations from the patterns, the conditions under which regime transitions occur which is not always predictable. Outside of the deep learning literature, many time series models have been proposed to capture specific types of high variabilities. For instance, switching linear dynamical models (Ackerson & Fu 1970 Ghahramani & Hinton| 1996, Murphy 1998| Fox et al. 2009) aim to model complex dynamical systems with set of simpler linear patterns. Conditional volatility models (Engle 1982, Bollerslev 1986) are introduced to model time series with heteroscedastic noise process whose noise level itself is part of the dynamics. However, these models usually encode specific inductive biases in a hard way, and cannot learn different behaviors and interpolate among the learned behaviors as deep neural nets.
In this work, we propose new class of neural recurrent latent variable model, called the variational hyper RNN (VHRNN), which can perform dynamic regime identification and re-identification dynamically at inference time. Our model captures complex time series without encoding large number of patterns in static weights, but instead only encodes base dynamics that can be selected and adapted based run time observations. Thus it can easily learn to express rich set of behaviors including but not limited to the ones mentioned above. Our model can dynamically identify the underlying pattern, express uncertainty due to observation noise, lack of information, model misspecification. As such, VHRNN can model complex patterns with fewer parameters; and when given lots of parameters, it generalizes better than previous methods.

---

Under review as a conference paper at ICLR 2020
The VHRNN is built upon the previous variational RNN (VRNN) models (Chung al. 2013) and hypernetworks (Ha et al. 2016). The VRNN models introduce stochastic latent variables at every time step, which are inferred using variational recognition model. The overall model is trained by maximizing the evidence lower bound (ELBO). In VRNN, the latent variables capture the information in the stochastic hidden states and are then fed input to the RNN and decoding model to produce reconstructed observations. While in our work, the latent variables are decoded to produce the RNN transition weights and observation projection weights the style of hypernetworks (Ha et al.] 2016). i.e., dynamically generating the scaling and bias vectors to adjust the base weights of the RNN. We demonstrate that the proposed VHRNN model is better at capturing different types of variability on several synthetic as well as real-world time series datasets.
2 BACKGROUND AND RELATED WORK
Variational Autoencoder Variational autoencoder (VAE) is one of the most popular unsupervised approaches to learning compact representation from data (Kingma & Welling 2013). It uses a variational distribution ((z|x) to approximate the intractable posterior distribution of the latent variable With the use of variational approximation, it maximizes the evidence lower bound (ELBO) of the marginal log- likeliodl of data
[(x) = (x)x) (og - DRL (g(zx)) |l p(z)) S log)
where p(z) is prior distribution of and DKL denotes the Kullback-Leibler (KL) divergence. The approximate posterior {())x) is usually formulated as a Gaussian with a diagonal covariance matrix.
Variational RNN for Sequential Data Variational autoencoders have demonstrated impressive performance on non-sequential data like images. Many following works (Bowman et al., 2015) Chung et al., 2015 Fraccaro al., 2016 Luo et al. (018)) extend the domain of VAE models to sequential data. Among them, variational RNN (VRNN; (CRNN; et al. 2015) further incorporate a latent variable at each time step into their models. A prior distribution conditioned on the contextual information and a variational posterior is proposed at each time step to optimize a step-wise variational lower bound. Sampled latent variables from the variational posterior at the current time step. The VHRNN model makes use of the same factorization of sequential data and joint distribution of latent variables as in VRNN. However, in VHRNN model, the latent variables also parameterize the weights for decoding and transition in RNN cell across time steps, giving the model more flexibility to deal with variations within and across sequences.
decoded into the observation
Importance Weighted Autoencoder and Filtering Variational Objective A parallel stream of work to improve latent variable models with variational inference study tighter bounds of the data's log-probability than ELBO. Importance Weighted Autoencoder (IWAE: Burda et al. (016) estimates a different variational bound of the ligeliiil,ooo,, which is provably tighter than ELBO. Filtering Variational Objective (FIVO: Maddison et al. (017) exploits the temporal structure of sequential data and uses particle filtering to estimate the data likelikel FIVO still computes a step-wise IWAE bound based on the sampled particles at each time step, but it shows better sampling efficiency and tightness than IWAE. We use FIVO as the objective to train and evaluate our models.
[IWAE:BBuda
Hy Ner Our model is motivated by HyperNetworks (Ha et al. 2016) which use one network to generate the parameters of another. The dynamic version of HyperNetworks can be applied to sequence data, but due to lack of latent variables, can only capture uncertainty in the output variables. For discrete sequence data such as text, categorical output variables can model multi-model outputs very well; but on continuous time series with the typical Gaussian output variables, the model is much less capable at dealing with stochasticity. Furthermore, it does not allow straightforward interpretation of the model behaviour using the time- seneessrieseae of KL divergence as we do in Sec. [ With the augmentation of latent variables, VHRNN is much more capable of modelling uncertainty, is worth noting that Bayesian HyperNetworks (Krueger et al., 2017) also have a latent variable in the context of Hypernetworks. However, the goal of Bayesian Hypern is an improved version of Bayesian neural net to capture model uncertainty. The work of Kruegrr et al. (2017) has no recurrent structure and cannot be applied to sequential data. Furthermore, the use of normalizing flow dramatically limits the flexibility of the decoder architecture design, unlike in VHRNN. Dezfouli et al. (2019) recently proposed to learn disentangled low- dimensiooi latent space such that samples from the latent space are used to generate parameters of an RNN model. Different latent variables could account for the behaviours of different subjects. In spite of similarity in combining RNN
2

---

Under review as a conference paper at ICLR 2020
(a) Prior
(b) Recurrence
(c) Generation
(d) Inference
(e) Overall
Figure 1:
Diagrams of the variational hyper RNN. Operators are indicated by arrows in different
colors, and dashed lines and boxes represent the hypernetorr components. (a) Prior distribution in Eq. 3 (b) Recurrent model in Eq. [I] (c) Generative model in Eq. (d) Inference model in Eq. (c) The overall computational path. The hypernetwork components are left out.
with HyperNetworks, the motivations and architectures fundamentally different. The work of Dezfouli et al. (2019) intends to learn a low-dimensional interpretable latent space that represents the difference between subjects in decision-making process while models tries better handle the variances both within and across sequences in general time-series data modeling. The work of Dezfouli et al. (2019) generate set of parameters that shared across time steps for each latent variable. In contrast, our model samples a latent variable and dynamically generates non-shared weights each time step, which we believe is essential for handling variance of dynamics within sequences.
3 MODEL FORMULATION
Variational Hyper RNN A recurrent neural network (RNN) can be characterized by h, (x,, hı-1), where x, and h, are the observation and hidden state of the RNN at time step I, and @ is the fixed weights of the RNN model. The hidden state h, is often used to generate the output for other learning tasks, e.g., predicting the observation at the next time step. We augment the RNN with latent random variable Z,, which is also used to output the non-shared parameters of the RNN at time step f.
h go(z: by (X,, Ze, hr-1): 1),
where OZz,, h₂-1) 1) is hypernetwork that generates the parameters the RNN time step t. The latent variable z, can also be used to determine the parameters of the generative model p Rgg, x
x1|zge X (Muee >,")) where (ndec Eder)
dec Puiz
(Ze: h--1).
(2)
We hypothesize that the previous observations latent variables, characterized by h define prior distribution p(z: x Z<t)) over the latent variable Zı.
z Xxt, Rett
M( prior sprior), where (44prior Sprior) carior (h+-1).
Eq. and result in the following generation process of sequential data:
T
p(x BgT)
II 2() agee..
The true posterior distributions of z, conditioned on observations x and latent variables Z are intractable, posing a challenge in both sampling and learning. Therefore, we introduce approximate posterior (z, x Zet) such that
24 x Zeet N N(p D.no). where (nr Ee"0) cnc (Xt, h,-1).
(5)
This approximate posterior distribution enables the model to be trained by maximizing variational lower bound, e.g.. ELBO (Kingma & Welling 2013). IWAE (Burda et al.) 2016) and FIVO MMddiison
al. 2017). We refer to the main components of our model, including g, पा dene oprior primary networks and refer to the components responsible for generating parameters, and w, as hyper networks in the following sections.
3

---

Under review as a conference paper at ICLR 2020
Implementation Following the practice of VAE, we parametrize the covariance matrices Ep Eeec and Enne diagonal matrices. Note that Eprior in our model is no longer an identity matrix as in vanilla VAE; it is the output of @erior and depends on the hidden state h at the previous time step.
The recurrence model y in Eq. [ is implemented as an RNN cell, which takes as input x and Z each time step and updates the hidden states h, The parameters of g are generated by the hyper network (zz, h;-1), as illustrated in Figure Tb
O is also implemented using an RNN to capture the history of data dynamics, with 2, and h, as input each time step t. However, is computationally costly to generate all the parameters of using ((z,, h.-1). Following the practice of previous works (Ha et al. 2016, Krueger et al. 2017), the hyper network 0 maps 2₁ and h to bias and scaling vectors. The scaling vectors modily the parameters of g by scaling each row of the weight matrices, routing information in the input and hidden state vectors through different channels. To better illustrate this mechanism,
exemplify the recurrence model g using RNN cell with Ltte-style update rules and gates. Let * E fi.f, g, o) denote the one of the four LSTM-styyle gates in g. W. and U. denote the input and recurrent weights of each gate in LSTM cell respectively. The hyper network O(z,, h2-1) outputs di and d.. that are the scaling vectors for the input weights W. and recurrent weights U, of the recurrent model g in Eq. The overall implementation of g in Eq. can be described as follows:
i, (di h 1) o (Wiy.) + dni h1-1) o (U:h,-1)),
f, (dir(z,, 11 (Wry.) + dpr(ze, h 1) (UFh;-1)), g: tanh (dig (Z,, hy-1) o (Wgyt) + dng (2,, h o (ghh-1)),
O: (dio h,-1) ..) + dna (2,, he-1) (U,hr-1)),
C; f, o c + i, o gr.
h, o, tanh (c:),
where denotes the Hadamard product and y, is a fusion (e.g., concatenation) of observation x; and latent variable 2,. For simplicity of notation, bias terms are ignored from the above equations. This implementation of the recurrence model in VHRNN is further illustrated in the diagram in Fig. in appendix.
Another hyper network (1z, h, generates the parameters of the generative model in Eq. implemented as a multilayer perceptron (MLP). Similar to (Zz,, h;-1), the outputs are the bias and scaling vectors that modify the parameters of the decoder dree
2 It is
w (2.,hh.. h.-1)"
4 SYSTEMATIC GENERALIZATION ANALYSIS OF VHRNN
In terms of the general functional form Eq. [ the recurrence of VRNN and VHRNN both depend on 2; and h,-1, so a sufficiently large VRNN could capture the same behaviour as VHRNN in theory. However, VHRNN's structure better encodes the inductive bias that the underlying dynamics could change, that they could slightly deviate from the typical behaviour in regime, there could be drastic switch to a new regime. With finite training data and finite parameters, this inductive bias could lead to qualitatively different learned behaviour, which demonstrate and analyze now.
In the spirit of Bahdanau et al. (2019), we perform a systematic generalization study of VHRNN in comparison the VRNN baseline. We train the models on one synthetic dataset with each sequence generated by fixed linear dynamics and corrupted by heteroskedastic noise process. We demonstrate that VHRNN can disentangle the two contributions of variations and learn the different base patterns of the complex dynamics while doing so with fewer parameters. Furthermore, VHRNN can generalize wide range of unseen dynamics, albeit the much simpler training set.
The synthetic dataset is generated by the following recurrence equation:
X Wx--1 + o161,
(6)
where E, E R is a two-dimensional standard Gaussian noise and Xy is randomly initialized from a uniform distribution over [-1, 1|2 For each sequence, W R is sampled from 10 predefined random matrices (W with equal probability; o, is the standard deviation of the additive noise at time i and takes value from (0.25, 1,4). The noise level shifts twice within sequence; i.e., there
4

---

Under review as a conference paper at ICLR 2020
exactly two f's such that oi 0t-1- We generate 800 sequences for training, 100 sequences for validation, and 100 sequences for test using the same sets of predefined matrices. The models are trained and evaluated using FIVO as the objective. The results on the test set almost the same as those on the training set for both VRNN and VHRNN. We also find that VHRNN shows better performance than VRNN with fewer parameters, shown in Tab. [ column Test. The size of the hidden state in RNN cells is set to be the same as the latent size for both types of models.
We further study the behavior of VRNN and VHRNN under the following systematically varied settings:
NOISELESS In this setting, sequences are generated using a similar recurrence rule with the same set of predefined weights without the additive noise at each step. That is, 0 in Eq. foro all time step t. The exponential growth of data could happen when the singular values of the underlying weight matrix greater than 1.
SWITCH In this setting, three NOISELESS sequences are concatenated into one, which contains regime shifts as a result. This setting requires the model to identify and re-identify the underlying pattern after observing changes.
RAND In this setting, the deterministic transition matrix in Eq. is set to the identity matrix (i.e., W I). leading long sequences of pure random walks with switching magniudes of noise, The standard deviation of the additive noise randomly switches up to times within (0.25, 1, 4] in one sequence.
LONG In this setting, we generate extra-long NOISELESS sequences with twice the total number of steps using the same set of predefined weights. The data scale can exceed well beyond the range of training data when exponential growth happens.
ZERO-SHOT In this setting, NOISELESS sequences are generated such that the training data and test data use different sets of weight matrices.
ADD In this setting, sequences are generated by different recurrence rule: x, X 1 + b, where b and Xy are uniformly sampled from [0, 1|2.
Tab. [ illustrates the experimental results. We can see that VRNN model, depending on model complexity, either underfits the original data generation pattern (Test) fails to generalize to more complicated settings. In contrast, the VHRNN model does not suffer from such problems and uniformly outperforms VRNN models under all settings. To qualitatively study the behavior of VHRNN and VRNN, we consider a VRNN with a latent dimension of 8 and a VHRNN with a latent dimension of 4 and make the following observations:
Table 1: Evaluation results on synthetic datasets.
FIVO estimated log likelihood per time stop
Model
Z dim.
Param.
Test
NOISELESS
SWITCH
RAND
LONG
ZERO-SHOT
ADD
VRNN
8
2612
5.43
-2.50
-334173
-5.02
-1033348
-3.64
-3.57
VRNN
1516
-3.66
-27200
-4.39
5509
VRNN
4
716
-7. 7.88
-5.25
7.85
-6.81
-5777
-5.35
-8.67
VHRNN
1568
4.68
-2.08
44.77
-3.91
-3005
-2.57
-2.62
Dynamic Regime Identification and Re-identification Fig. 2 shows sample sequence under the NOISELESS setting. VRNN has high KL divergence between the prior and the variational posterior most of the time. In contrast, VHRNN has a decreasing trend of KL divergence while still making accurate mean reconstruction as it observes more data. As the KL divergence measures the discrepancy between prior defined in Eq. 3 and the posterior that has information from the current observation, simultaneous low reconstruction and low KL divergence means that the prior distribution would be able to predict with low errors as well, indicating that the correct underlying dynamics model has likely been utilized. This trend even generalizes to settings with sources of variation unseen in the training data, namely ZEROSHOT and ADD. We speculate that this trend implies the model's ability to identify the underlying data generation pattern in the sequence. The decreasing trend is especially apparent when a sudden and big change in scale happens. We hypothesize that larger changes in scale can better help model, VHRNN, identify the underlying data generation process because our model is trained on sequential data generated with compound noise. The observation further corroborates conjecture that the KI. divergence would rise again once the sequence
5

---

Under review as a conference paper at ICLR 2020
switches from one underlying weight to another, as shown in Fig. It is worth noting that the KL increase happens with some latency after the sequence switches in the SWITCH setting as the model reacts to the change and tries to reconcile with the prior belief of the underlying regime in effeet.
Uncertainty Identification Fig. shows that the predicted log-variance of VHRNN can more accurately reflect the change of noise levels under the RAND setting than VRNN. VHRNN can also better handle uncertainty than VRNN the following two situations. As shown in Fig. 31, VHRNN can more aggressively adapt its variance prediction based on the scale of the data than VRNN. It keeps its predicted variance at low level when the data scale is small and increases the value when the scale of data becomes large. VHRNN makes inaccurate mean prediction relatively far from the target value when the switch of underlying generation dynamics happens in the SWITCH setting. The switch the weight matrix another important source of uncertainty. We observe that VHRNN would also make large log-variance prediction in this situation, even the scale of the observation is small. Aggressively increasing uncertainty about the prediction when switch happens avoids VHRNN model from paying high reconstruction cost shown by the second spike in Fig. 31 This increase of variance prediction also happens when exponential becomes apparent in setting LONG and the scale of observed data became out of the range of the training data. Given the large scale change of the data, such flexibility to predict large variance is key for VHRNN to avoid paying large reconstruction cost.
Russe
0.M
(b)
(c)
24
(d)
(I
Figure 2: Qualitative study of VRNN and VHRNN under the NOISELESS setting. (a) and (b) show the values of concatenated data at each time step. (c) shows the KI. divergence between the variational posterior and the prior of the latent variable at each time step for VHRNN. (d) shows the KL divergence for VRNN. (e) shows L2 distance between the predicted mean values by VHRNN and VRNN and the target. (f) shows the predicted log- variance of the output distribution for VRNN and
VHRNN.
2014
valuess
(a)
(b)
(c)
Log Varfance
(c)
(f)
Figure 3: Qualitative study of VRNN and VHRNN under the SWITCH setting. The layout of
subfigures is the same as Fig. 2 Vertical red lines indicate time steps when regime shift happen.
These two advantages of VHRNN over VRNN not only explain the better performance of VHRNN on the synthetic data but also are critical to RNNs' ability model real-world data with large variations

---

Under review as a conference paper at ICLR 2020
Nobse Magnitude (L2 STDV)
Nolse 12 Noise STDV
VRNN VHRNN
log Variance
(a)
Figure 4: Qualitative study of VRNN and VHRNN under the RAND setting. (a) shows the L2 norm and standard deviation of the additive noise at each time step. (b) shows the log-variance of the output distribution for VRNN and VHRNN.
both across and within sequences. Examples under other settings showing the above properties are deferred to the Appendix.
EXPERIMENTS ON REAL-WORLD DATA
We experiment with the VHRNN model on several real-world datasets and compare it against VRNN model. VRNN trained and evaluated using FIVO (Maddison et al.] 2017) demonstrates the state-of-the-an performance on various sequence modeling tasks. Our experiments demonstrate the superior parameter- performance efficiency and generalization ability of VHRNN over VRNN. All the models are trained using FIVO (Maddison et al. 2017) and we report FIVO per step when evaluating models. Two polyphonic music dataset considered: JSB Chorale and Piano-midi monno-midi.de de (Boulanger Lewandowski et al. 2012). We also train and test our models on a financial time series data and the HT Sensor dataset (Huerta al. 2016), which contains sequences of sensor readings when different types of stimuli are applied in an environment during experiments. We also trained and evaluated the HyperLSTM model without latent variables proposed by Haa et (2016) on few of the datasets above. The results and comparisons are deferred to the appendix.
curramete-performmaceeee
For the VRNN model, we use single-layer LSTM and set the dimension of the hidden state to be the same as the latent dimension. For the VHRNN model, 0 in Eq. Tis implemented using a single-layer LSTM to generate weights for the recurrence module in the primary networks. We use an RNN cell with LSTM-style gates and update rules for the recurrence module y in our experiments. The hidden state sizes of both the primary network and hyper network are the same as the latent dimension. A linear transformation directly maps the hyper hidden state to the scaling and bias vectors in the primary network. More details on the architectures of encoder, generation and prior networks are elaborated in the appendix.
Polyphonic Music The JSB Chorale and Piano-midis de are music datasets et al.] (0122) with complex patterns and large variance both within and across sequences. The datasets are split into the standard train, validation, and test sets. More details on data preprocessing training and evaluation setup deferred to the appendix.
We report the FIVO per time step of VHRNNs and VRNNS and their parameter counts in Fig. 5a and Fig. 5bb The results show that VHRNNs have better performance and parameter efficiency. The number of parameters and FIVO per time step of each model are plotted in the figures, and the latent dimension is also annotated. The has uniformly better performance than VRNN with a comparable number of parameters. The best FIVO achieved by VHRNN on JSB dataset 76.76 (VHRNN-14) compared
plots show that the VHRNN model
-6992 for VRNN (VRNN-32), which requires close to one third more parameters. This best VRNN model is even worse than the smallest VHRNN model we have evaluated. It is also observed that VHRNN less prone to overfitting and has better generalization ability than VRNN when the number of parameters keeps growing. Similar trends can be seen on the Piano-midi.de dataset in Fig. (5b) We also find that the better performance of VHRNN over VRNN can generalize to the scenario where we replace LSTM with Gated Recurrent Unit (GRU). Experimental results using GRU implementation are deferred to the appendix.
Stock Financial time series data, such as daily prices of stocks, are highly volatile with large noise. The market volatility is affected by many external factors and can experience tremendous changes
7

---

Under review as a conference paper at ICLR 2020
14
32
12
VRNN
VHRNN
24
40
FIVO per step
-6.9
FIVO per step
-7.
40
48
-7.
-7.4
VRNN
VHRNN
-7.8
64
64
48
20003
40000
60000
50000
200000
120000
140000
6.6
0.8
1.0
1.2
1.4
2.6
1.9
Parameter Count (a) JSB Choral
Parameter Count
165
(b) Tfano-midi.de
14
9.34
14
24
28
0.52
FIVO Der step
FIVO per step
32
VRNN VHRNN
9:26
9:24
9:22
VRNN VHRNN
28
20
14000
18000
20300
22000 24000
20000
20000
15000
20000
25000
30000
35000
Parameter Count
Parameter Count
(c) Stock
(d) HT Sensor
Figure 5: VRNN and VHRNN parameter--erformmncce comparison.
in sudden. To test the models' ability to adapt different volatility levels and noise patterns, compare VHRNN and VRNN on stock price data collected in period when the market went through rapid changes. The data collected from 445 stocks in the S&P500 index in 2008 when global financial crisis happened. The dataset contains the opening, closing, highest and lowest prices, and volume on each day. The networks are trained on sequences from the first half the year and tested on sequences from the second half, during which the market suddenly became significantly more volatile due the financial crisis.
The evaluation results are shown in Fig. 5c The plot shows that VHRNN models consistently outperform VRNN models regardless of the latent dimension and number parameters. The results indicate that VHRNN can have better generalizability to sequential data in which the underlying data generation pattern suddenly shifts even the new dynamics are not seen in the training data.
HT Sensor The comparison is also performed on a dataset with less variation and simpler patterns than the previous datasets. The HT Sensor dataset contains sequences of gas, humidity, and temper- ature sensor readings in experiments where some stimulus is applied after period of background activity (Huerta et al. 2016). There are only two types of stimuli the experiments: banana and wine. In some sequences, there noise. Experimental results on HT Sensor dataset are shown in Fig. [d
stimulus applied, and they only contain readings under background
It is observed that VHRNN has comparable performance VRNN on the HT Senor Dataset when using a similar number of parameters. For example, VHRNN achieves a FIVO per time step of 14.41 with 16 latent dimensions and 24200 parameters, while VRNN shows slightly worse performance with 28 latent dimensions and approximately 26000 parameters. When the number of parameters goes slightly beyond 34000, the FIVO of VHRNN decays to 12,45 compared to 12.37 of VRNN.
ABLATION STUDY
We further investigate the effects of hidden state and latent variable the performance of variational hyper RNN the following two aspects: the dimension of the latent variable and the contributions by hidden state and latent variable inputs hyper networks.
8

---

Under review as a conference paper at ICLR 2020
Latent Dimension In previous experiments on real-world datasets, the latent dimension and hidden state dimension are set to be the same for each model. This causes VHRNN to have significantly more parameters than VRNN when using the same latent dimension. To eliminate the effects of the difference in model size, we allow the latent dimension and hidden state dimension to be different. We also reduce the hidden layer size of the hyper network that generates the weight of the decoder. These changes allow us to compare VRNN and VHRNN models with the same latent dimension and similar number of parameters. The results on JSB Chorale datasets are presented in Tab. 2 in which we denote latent dimension by Z dim. We observe that VHRNNs always have better FIVO with the same latent dimensions than VRNNs. The results show that the superior performance of VHRNN over VRNN does not stem from smaller latent dimension when using the comparable number of parameters.
Inputs to the Hyper Networks We retrain and evaluate the performance of VHRNN models on JSB Chorale dataset and the synthetic sequences when feeding the latent variable only, the hidden state only, or both to the hyper networks. The results are shown in Tab. 3 It is observed that VHRNN has the best performance and generalization ability when it takes the latent variable as its only input. Relying on the primary network's 's hidden state only the combination of latent variable and hidden state leads to worse performance. When the dimension of the hidden state 32, VHRNN only taking the hidden state as hyper input suffers from ver-pprrameterization and has worse performance than VRNN with the same dimension of the hidden state. On the test set of synthetic data, VHRNN obtains the best performance when it takes both hidden state and latent variable inputs. We surmise that this difference is due to the fact that historical information is critical to determine the underlying recurrent weights and current noise level for synthetic data. However, the ablation study on both datasets shows the importance of the sampled latent variable as an input to the hyper networks. Therefore, both hidden state and latent variable are used as inputs to hyper networks on other datasets for consistency.
7 CONCLUSION
In this paper, we introduce the variational hyper RNN (VHRNN) model, which can generate parame- ters based on the observations and latent variables dynamically. Such flexibility enables VHRNN to better model sequential data with complex patterns and large variations within and across samples than VRNN models that use fixed weights. VHRNN can be trained with the existing off-the- variational objectives. Experiments on synthetic datasets with different generating patterns show that VHRNN can better disentangle and identify the underlying dynamics and uncertainty in data than VRNN. We also demonstrate the superb parameter- perforrene- efficiency and generalization ability of VHRNN on real-world datasets with different levels of variability and complexity.
Table 2: VRNN and VHRNN with same latent dimensions.
Table Ablation study with different hyper
network inputs.
Hidden dim.
Hyper size
Dataset
dim.
Hyper Input
FIVO
Model
Z. dim.
Param.
FIVO
14
latent only
6.68
14
hidden only
-6.7
24
24
23k
7.04
14
latent+hidden
-6.76
VRNN
28
31k
6.99
JSB
latent only
-6.76
6.91
32
hidden only
-7.03
32
Iatent+hidden
-6882
24k
VHINNN
28
14
18
31k
6673
Synthetic Test
4
hadden only
latent only
-4.99
-5001
39k
66700
4
lntent-hidden
-4688
REFERENCES
G Ackerson and K Fu. On state estimation in switching environments. IEEE Transactions on Automatic Control, 15(1):10-17, 1970.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: arXiv:1409.00 000p:://99. 2014.

---

Under review as a conference paper at ICLR 2020
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In International Conference Learning Representations, 2019. URL httpattp///opeeeiiiiie net net/forum " dd=
Tim Bollerslev. Generalized autoregressive conditional heteroskedasticity. Journal of econometrics, 31(3):07--277, 1986.
Nicolas Boulanger-Lewandonn Yoshua Bengio, and Pascal Vincent. Modeling temporal dependen- cies in high-dimensionall sequeeess Application to polyphonic music generation and transcription. arXiv preprint arXiv:1206.63392 6392, 2012.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space, arXiv preprint urXiv:1511.06349, 2015.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In International Conference on Learning Representations, 2016.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decodes for statistical machine translation. In EMNLP, pp. 1724-1734, 2014.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980-2988, 2015.
Amir Dezfouli, Hassan Ashtiani, Omar Ghattas, Richard Nock, Peter Dayan, and Cheng Soon Ong. Disentangled behavioral representations. bioRxiv, pp. 658252, 2019.
Robert F Engle. Autoregressive conditional heteroscedasticity with estimates of the variance of united kingdom inflation. Econometrica: Journal of the Economelric Society, pp. 987-1007, 1982.
Emily Fox, Erik B Sudderth, Michael I Jordan, and Alan S Willsky. Nonparametric bayesian learning of switching linear dynamical systems. In Advances Neural Information Processing Systems, pp.
457-464, 2009.
Marco Fraccaro, Søren Kaae Sønderby, UIrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. In Advances in neural information processing systems, pp. 2199-2199000 22007, 2016.
Zoubin Ghahramani and Geoffrey E Hinton. Switching state-space models. Technical report, Citeseer, 1996.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649. IEEE, 2013.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint 100.:9160, 2016
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()) 1735-1788 1997.
Ramon Huerta, Thiago Mosqueiro, Jordi Fonollosa, Nikolai F Rulkov, and Irene Rodriguez-Lujan. Online decorrelation of humidity and temperature chemical sensors for continuous monitoring. Chemometrics and Intelligent Laboratory Systems, 157:169- 176, 2016.
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling, arXiv preprint arXiv:1602.024100 2016.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiw:1312.6114, 2013.
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In NIPS, pp. 3294-3322 3302, 2015.
10

---

Under review as a conference paper at ICLR 2020
David Krueger, Chin-Wer Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint arXiv: 2017.
Rui Luo, Weinan Zhang, Xiaojun Xu, and Jun Wang. A neural stochastic volatility model. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Chris Maddison, John Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Arnad Doucet, and Yee Teh. Filtering variational objectives. In Advances in Neural Information Processing Systems, pp. 6573-6583, 2017.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2, pp. 3, 2010.
Kevin P Murphy. Switching kalman filters. 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Chao-Yuan Wu, Amr Ahmed, Alex Beutel, Alexander Smola, and How Jing. Recurrent recom- mender networks. In WSDM, pp. 495-503. ACM, 2017.

---

Under review as a conference paper at ICLR 2020
A
VHRNN RECURRENCE MODEL IMPLEMENTATION USING LSTM CELL
Ct
Ct
Legend
X
tanh
activation
O
tanh
pointwise operation
(W.. U.
(W, U (W
h
X
ht
layers in primary network
(W., U.)
layers in hyper network
Et 24
weights in primary (W., network that are dynamically modified
h
Figure 6: Implementation of the recurrence model in VHRNN using LSTM cell.
B QUALITATIVE STUDY OF VRNN AND VHRNN UNDER ADD, ZERO-SHOT, LONG SETTINGS ON SYNTHETIC DATASETS
Divergoncce
20
(b)
(c)
XL Diverrrnnc
VIAN
VAVV
Figure 7: Qualitative study of VRNN and VHRNN under the ADD setting. (a) and (b) show the values of concatenated data at each time step. (c) shows the KI. divergence between the variational posterior and the prior of the latent variable at each time step for VHRNN. (d) shows the KL divergence for VRNN. (c) shows L2 distance between the predicted mean values by VHRNN and VRNN and the target. (f) shows the predicted log- variance of the output distribution for VRNN and VHRNN.
Fig. and Fig. 8 show the qualitative study results of VHRNN and VRNN under the ADD and ZERO-SHOT settings. We see that the KL divergence VHRNN model decreases as it observes more data. Meanwhile the mean predictions by VHRNN stay relatively close to the actual target value shown in Fig. 7eland Fig. &e) The prediction are especially accurate the ADD setting as Fig. 7el shows. The results demonstrate VHRNN's 's ability to identify system dynamics can generalize to unseen data generation patterns. By contrast, we does not see any trend of variational RNN that indicates it is capable of doing dynamic regime identification.
Fig. | Dillustrates the qualitative study of VHRNN and VRNN under the LONG setting. The magnitude of the data grows rapidly in such setting due to exponential growth and it is well beyond the scale of training data. We can see both VRNN and VHRNN make very inaccurrate mean predictions that far from the target values Fig. 9e shows. However, VHRNN pays smaller reconstruction cost than VRNN by also making large predictions of variance. This setting demonstrates special case in which VHRNN has better ability to handle uncertainty in data than vanilla variational RNN.
12

---

Under review as a conference paper at ICLR 2020
Divergance
Xvilues
Vvaluus
(a)
(b)
(C)
81 DNNCri
Log Variance
V-mi
(d)
(e)
Figure 8: Qualitative study of VRNN and VHRNN under the ZERO-SHOT setting. The layout of subfigures is the same as Fig. 7
Divergence
Vallues
(b)
(c)
Log Vorlance
VAKR
VANV
Figure 9: Qualitative study of VRNN and VHRNN under the LONG setting. The layout of subfigures is the same as Fig.
Fig.
9a
9b
9d
9c
use scientific notations for the value of Y axis.
C VRNN AND VHRNN IMPLEMENTATION DETAILS
Encoder Network and Prior Network The architecture of the encoder in Equation 5s the same in VHRNN and VRNN. For synthetic datasets, the encoder is implemented by a fully-connected network with two hidden layers, each hidden layer has the same number of units as the latent variable dimension. For other real-world datasets, we use fully-connected network with one hidden layer. The number of units is also the same as the latent dimension. The prior network is implemented by similar architecture as the encoder, with the only difference being the dimension of inputs.
Decoder Network For VHRNN models, layers for synthetic data and fully-connected hyper networks with one hidden layer for other datasets the decoder networks. The number of units in each hidden layer also the same the latent variable defined in [quation For each layer of the hyper networks, the weight scaling vector and bias generated by an two-layer MLP. The hidden layer size of this MLP is for synthetic dataset and 64 for real-world datasets. For VRNN models, we use plain ferw-forward networks for decoder. The number of hidden layers and units the hidden layer are determined in the same way as VHRNN models.
fully-connected hyper networks with two hidden
Observation and Latent Variable Encoding Network For fair comparison with VRNN Chung et al. (2015), the latent variable and observations are encoded by network (different from the encoder in Equation 5) before being fed to the recurrence network and encoder. The latent and observation encoding networks have the same architecture except for the input dimension in each experiment setting. For synthetic datasets, the encoding network is implemented by a fully-connected network with two hidden layers. For real-world datasets, use fully-connectes network with one hidden layer. The number of units in each hidden layer is the same as the dimension latent variable that experiment setting.
13

---

Under review as a conference paper at ICLR 2020
D REAL-WORLD DATASETS PREPROCESSING DETAILS
Polyphonic Music Each sample in the polyphonic music datasets, JSB Chorale and Piano-midi.dh de represented sequence of 88-dimensional binary vectors. The data are preprocessed by mean- centering along each dimension per dataset.
Stock We randomly select 345 companies and their daily stock price and volume in the first half of 2008 to obtain training data. We another 50 companies' data in the second half of 2008 to generate validation set and get the of 2008. The sequences are first preprocessed by taking log ratio of the values between consecutive days. Each sequence has a fixed length of 125. The log ratio sequences are normalized using the mean and standard deviation of the training set along each dimension.
from the remaining 50 companies during the second half
HT Sensor The HT Sensor dataset collects readings from 11 sensors under certain stimulus in an experiment. The readings of the sensors are recorded at rate of once per second. We segment sequence of 3000 seconds every 1000 seconds the dataset and downsample the sequence by a rate of 30. Each sequence
obtained has fixed length of 100. The types of sequences include pure background noise, stimulus before and after background noise and stimulus between two periods of background noise. The data normalized
mean and unit variance along each dimension. We use 532 sequences for training, 68 sequences for validation and 74 sequences for testing.
E TRAINING AND EVALUATION DETAILS ON BEAL-WORLD DATASETS.
For all the real-word world data, the models, both VRNN and VHRNN, are trained with batch size of and particle size of 4. When evaluating the models, use particle size of 128 for polyphonic music datasets and 1024 for Stock and HT Sensor datasets.
F VHRNN AND VRNN HIDDEN UNIT VERSUS PERFORMANCE COMPARISON
We also compare VHRNN and VRNN by plotting the models' performance against their number of hidden units. The models considered here the a single-layer LSTM model for the RNN part: the dimension of LSTM's hidden state is the same as the latent dimension. It's worth noting that VHRNN uses two LSTM models, one primary network and one hyper network. Therefore, the number of hidden units in an VHRNN model is twice the number of latent dimension. We can see that VHRNN also dominates the performance of VRNN with similar or fewer number of hidden units in most of the settings. Furthermore, the fact that VHRNN almost always outperforms VRNN for all parameter or hidden unit sizes precisely shows the superiority of the new architecture. The results from Fig. 5 and Fig. T0]are consolidated in into Tab.
the models presented in Fig. 5 We use
14

---

Under review as a conference paper at ICLR 2020
82
VRNN VHRNN
40
FIVO per step
-69
-75
40
FIVO
-7.
VRNN
VHRNN
-10
64
64
50
60
70
0.5
0.5
0.7
0.8
Hidden Units
Hidden Units
(a) JSB Choral
(b) Piano-midi.de
16
28
FIVO per step
FIVO per step
32
9:30
9.28
0.26
10-
12
VRNN VHRNN
VRNN VHRNN
24
28
9.0
20
2:00
2.25
2.50
2.75
3.00
3.25
3.50
3.5
4:00
Hidden Units
Hidden Units
1eb
(c) Stock
(d) HT Sensor
Figure 10: VRNN and VHRNN Hidden Units vs Performance comparison.
15

---

Under review as a conference paper at ICLR 2020
Table 4: VRNN (left) and VHRNN (right) on Datasets.
Dataset
dim.
Hiddon Unit
Param.
FIVO
Datasct
Z dim.
Hidden Unit
Param.
FIVO
64
64
144k
7.25
42
84
135k
6.90
56
111k
7.21
40
80
125k
6.87
48
83k
7.12
32
64
88k
6.82
32
32
39k
6.92
JSB
24
48
58k
6.86
28
31k
-6.99
24
24
23k
7.04
14
28
31k
6.76
80
80
220k
7.85
72
180k
7.85
48
169
7.48
Piano
64
144k
7,83
40
80
125k
7.51
Piano
56
56
111k
7.86
32
64
88k
7.33
20
13k
9.20
12
24
16k
9.33
Stock
24
20k
9.21
Stock
14
28
18k
9.34
28
28
27k
9.20
16
32
23k
9.27
18
28k
20
20
14k
6.13
HT Sensor
28
28
26k
12.37
HT Sensor
16
32
24k
14.41
32
34k
12.69
40
34k
12.45
G VHRNN AND VRNN PERFORMANCE-PARAMETE. COMPARISON USING GRU ON JSB CHORALE DATASET
Fig. 11 shows the parameter performance plots of VHRNN and VRNN using GRU implementation the JSB Chorale dataset. VHRNN models consistently outperform VRNN models under all settings.
16
FIVO per step
-6.95
-7.00
28
-7.05
-7.0
VRNN
VHRNN
-7.35
20000
22000
24000
26000
28000
30000
32000
34000
Parameter Count
Figure 11: VRNN and VHRNN arameter-perffrmmnce comparison using GRU implementation on ISB Chorale dataset.
H VHRNN AND HYPERLSTM PERFORMANCE COMPARISON
We compare our VHRNN models using LSTM cell with the HyperL models proposed in HyperNetworks Ha et al. (2016) on JSB Chorale and Stock datasets. Compared with VHRNN, HyperLSTM does not have latent variables. Therefore, it does not have an encoder decoder either. Our implementation of HyperLSTM resembles the recurrence model of VHRNN defined in Equation 6 At each time step, HyperLSTM model predicts the output distribution by mapping the RNN's hidden state to the parameters of binary distributions for JSB Chorale dataset and a mixture of Gaussian for Stock dataset. We consider 3 and 5 as the number of components in the Gaussian
16

---

Under review as a conference paper at ICLR 2020
mixture distribution. HyperLSTM models are trained with the same batch size and learning rate as VHRNN models.
We show the purameter-performatte comparison between VHRNN, VRNN and HyperLSTM models in Fig. 12 The number of components used by HyperL.STM for Stock dataset is in the plot. Since HyperLSS LST models do not have latent variable, the indicator top of each point shows the number of hidden units each model for all the three of them. The number of hidden units for HyperL.TTM model is also twice the dimension of hidden states as HyperLSTM has two RNNs, one primary and one hyper. We report FIVO for VHRNN and VRNN models and exact log likelihood for HyperLSTM models. Even though FIVO is a lower-bount of log likelihood, we can see that the performance of VHRNN completely dominates HyperLSTMM, no matter what number of hidden units we use. Actually, the performance of Hyperl.SS .STM is even worse than VRNN models which do not even have hyper networks. The results indicates the importance of modeling complex time-series data.
We also show the hidden-une wetn---nnnsst-tfnnnne performance comparison between VHRNN and VRNN Fig. The comparison shows similar results.
28
80
32
-.0
24
9.2
28
FIVO/LL per step
FIVO/LL per step
-7.5
VRNN VHRNN HyperLSTM
9.0
VRNN VHRNN
-8.0
HyperLSTM
-8.5
8.6
36
-3.0
20000
40000
50000
50000
100000
320000
140000
36000 18000 22000
20a00
Parameter Count
Parameter Count
(a) JSB
(b) Stock
Figure 12: Parameter vs Performance comparison among VHRNN, VRNN and HyperLSTM. We
report FIVO for VHRNN and VRNN, and exact log likelihood for HyperLSTM. We Gaussian mixture distribution components the HyperLSTM model to estimate log likelihood in Stock dataset. The indicator above each point shows the number of hidden units in the model.
28
24
28
48
-.00
48
20
28
FIVO per step
-7.5
VRNN
VRNN VHRNN
FIVO
VHRNN HyperLSTM
HyperLSTM
-55
24
-3.0
28
8.4
20
22
24
25
30
34
Hidden Units
Hidden Units
(a) JSB
(b) Stock
Figure 13: Hidden units vs Performance comparison among VHRNN, VRNN and HyperL We report FIVO for VHRNN and VRNN, and exact log likelihood for HyperI.STM. We use a Gaussian mixture distribution of 5 components in the HyperLSTM model to estimate log likelihood in Stock dataset. The indicator above each point shows the number of hidden units in the model.
Complete experiment results of Hyperl.STT STM models the datasets shown in Tab.
17

---

Under review as a conference paper at ICLR 2020
Table 5: HyperLSTM Performance in Log Likelihood
Dataset
Hidden Units
Gauss. Components
Param.
Log Likelihood
24
N.A.
-9.15
28
N.A.
11k
-8.1
32
N.A.
14k
-8.87
48
N.A.
30k
-894
64
N.A.
49k
-8.85
JSB
80
N.A.
75k
-8777
84
N.A.
83k
-877
96
N.A.
106k
-874
104
N.A.
124k
88.5
112
N.A.
143k
8.67
24
7k
8.28
28
3
9k
8.33
32
12k
8.42
36
3
15k
8.35
40
18k
8.29
48
3
26k
7.91
Stock
24
5
7k
8.61
28
5
9k
8.44
32
12k
8.44
36
5
15k
8.47
40
19k
8.42
48
5
26k
8.58
I VHRNN WITHOUT TEMPORAL STRUCTURE HYPER NETWORKS
Using RNN to generate the parameters of another RNN has been studied in HyperNetworks Ha ct al. (216) and delivers promising performance. It also seems like a natural choice as the hidden state of the primary RNN represent the history of observed data while the hidden state of the hyper RNN can track the history of data generation dynamics. However, other design choices that do not have the recurrence structure in the hyper networks for VHRNN. As an ablation study, we experimented with VHRNN models that replace the RNN with a three-layer feed-forward network as the hyper network 0 for the recurrence model g as defined in Equation 6 We keep the other components of VHRNN unchanged on JSB Chorale, Stock and the synthetic dataset. The evaluation results using FIVO are presented Tab. and systematic generalization study results on the synthetic dataset are shown in Tab. 7 We denote the original VHRNN with recurrence structure in VHRNN-RNN and the variant without the recurrence structure as VHRNN-MI.P.
still intriguing to study
As we can see, given the same latent dimension, VHRNN-MLP models have more parameters than VHRNN-RNN models. VHRNN-MLP can have slightly better performance than VHRNN-RNN in some cases but it performs worse than VHRNN-RNN in more settings. The performance of VHRNN-MLE also degrades faster than VHRNN-RNE on the JSB Chorale dataset as we increase the latent dimension. Moreover, systematic generalization study on the synthetic dataset also shows that VHRNN-MLP has worse performance than VHRNN-RNN no matter in the test setting or in the systematically varied settings.
18

---

Under review as a conference paper at ICLR 2020
Table 6: Parameter-Performaanc Comparison of VHRNN-RNN and VHRNN-MLP
Dataset
7.dim.
VHRNN-MI. Param.
VHRNN-RNY RENN Param.
PIRNN-MIPFIIO
VHRNN-RNN FIVO
40
135k
125k
7.08
6..87
88k
-7.13
ISB
24
62k
58k
6.83
6.86
16
37k
35k
6.76
-681
32k
31k
6.73
-677
12
28k
27k
6.83
6.80
12
16k
16k
9.47
9.33
Stock
14
19k
18k
9:21
16
24k
23k
9.31
9.27
18
29k
28k
9.19
9.31
Syn Test
1676
1568
-5.13
4.68
Table 7: Systematic Generalization Study VHRNN-RNN and VHRNN-MLP
FIVO estimated log Likelihood per time step
Model
Z. dim.
Param.
Test
NOISELESS
SWTTCH
RAND
LONG
ZERO-SHOT
ADD
VHRNN-MPP VHRNN-RNN
1676
-..13
-2.33
5.22
-4.12
604937
-2.94
-384
-4.8
-2.88
-391
-2.62
19


Reviews:
[{"review_id": "rylIcVsaYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the variational hyper RNN (VHRNN), which extends the previous variational RNN (VRNN) by learning the parameters of RNN using a hyper RNN. VRHNN is tested and compared with VRNN on synthetic and real datasets. The authors report superior performance parameter efficiency over VRNN.\n\nThe performance of VHRNN is promising and certainly better than the previous VRNN for some applications. However, the VHRNN is constructed by a straight-forward combination of existing techniques and hence the technical contribution of this paper is marginal.\n\nAlthough Section 4 is entitled as systematic generalization analysis of VHRNN, the reported results are only for the specific structures of VHRNN and VRNN. Isn’t it useless to present results for the VRNN with a latent dimension of 4, at least as a sanity check? \n\nFig. 2 and the texts referring to it discuss the KL divergence between the prior and the variational posterior. While the FIBO is mainly used as the objective in this paper, is the ELBO enough if the authors care the simultaneous low reconstruction error and low KL divergence?\n\nIt is unclear and explained little if the comparison using parameter count is fair for VHRNN and VRNN since they have different structures.\n\nIt would be nicer to discuss for which kind of time-series VRNN is enough.\n\nMinor comments:\nThe caption of Figure 1 is too close to the main texts.\nEq. (4) is overlapping with texts.\nCan the equations at the bottom of p.3 be explained with an illustration?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SylW2nrRtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a variational hyper recurrent neural network which is a combination of the variational RNN and the hypernetwork. The hypernetwork is an RNN whose output modifies the parameters of the variational RNN dynamically at runtime. Overall, this seems like an extension of the idea of using a hypernetwork with the VRNN (rather than the RNN as done in Ha. et. al). The model is trained via the FIVO objective. The model and learning algorithm are compared to the variational RNN and tested on a variety of synthetic settings where the VHRNN outperforms the VRNN in held-out likelihood. The performance gains are investigated on synthetic datasets where the paper notes that the VHRNN is often quicker to adapt variations that happen within seqences (for example, the paper considers a dataset where multiple patterns are stitched together into a sequence and study the changes in the KL divergence and reconstruction at switch points). On four real-world sequential datasets, the paper finds that the model outperforms the VRNN across many configurations and with a fewer number of parameters.\n\nSummary: I don't think the model presented here is very novel, in that it is a combination of existing ideas; however, the paper does a good job of studying the model in a variety of different configurations on both synthetic and real-world data. The model does appear to consistently outperform the Variational RNN of Chung et. al.\n\nQuestions and comments:\n(a) I do not think the word \"System Identification\" should be used on page 5 to describe the results of Figure 2. Doing so would overload existing notation in the time series literature where the word refers to the identification of parameters under a pre-specified physical system.\n(b) How well does the Recurrent Hyper network (with no latent variable) do on the tasks considered here? I understand that it may be a less expressive model in general, but it is not clear to me why it would not be a competitive baseline on some of the smaller datasets considered here -- was this baseline tried?\n(c) Did you experiment with non-temporal architectures for the hypernetwork? Since z_t and h_t-1 (which are conditioned on) contain information about the history of the sequence, one might argue that conditioning on them might suffice to predict the modifications to the parameters of the theta and g.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkl7IVhBqS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors propose an architecture based on variational autoencoders and hyper-networks. The basic idea is that the weights of the underlying RNN/autoencoder are not fixed, but are coming from another RNN/feed-forward network which captures the underlying dynamics and adjusts the weights accordingly. The experimental results show the benefit of the model compared to a similar method without hypernets.\n \nIn terms of novelty, the combination of auto-encoder RNNs and hyper-networks is not entirely novel and it has previously been developed (https://www.biorxiv.org/content/10.1101/658252v1). However, while I think these previous works should be discussed in the paper (they are not currently), the two architectures are sufficiently different and the current work is novel enough in my opinion. On the other hand, in terms of presentation, I think the paper can be improved. The architecture is not entirely clear from the text. I think a graph showing the architecture of the model would be very helpful here. The notations also seem loosely defined (what is the dimensionality of x_t, z_t, etc.) and sometimes undefined (e.g., x_t in equation 1 is not defined).  \n \nIn terms of model architecture, it wasn’t clear for me why the hyper-network for \\phi is feedforward but the one for \\theta is RNN?\n \nThe experiments seem promising, but I have the following questions before being able to assess the results:\n \n- How many LSTM units are in the model in each experiment? Are they similar in both VHRNN and VRNN?\n \n- What is the structure of encoder/decoder layers? How many units in each layer? Are they the same for VHRNN and VRNN?\n \n-There are four sets of weights in the primary model: weights of RNN, dec, enc, prior. How are these weights generated by the two hyper-networks \\theta and w?\n \n- How is the number of parameters in the experiment calculated. Do they refer to the number of parameters in the hypernetworks only?\n \n \nMinor:\nI suspect the spaces between the equations and captions are also manually changed which has made the paper physically dense and a bit unreadable (see equation 4 for example). Similarly, the space between the caption of Figure 1 and the text after seems too small.\n \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a neural network architecture that uses a hypernetwork (RNN or feedforward) to generate weights for a network (variational RNN), that models sequential data. An empirical comparison of a large number of configurations on synthetic and real world data show the promise of this method.

The authors have been very responsive during the discussion period, and generated many new results to address some reviewer concerns. Apart from one reviewer, the others did not engage in further discussion in response to the authors updating their paper.

The paper provides a tweak to the hypernetwork idea for modeling sequential data. There are many strong submissions at ICLR this year on RNNs, and the submission in its current state unfortunately does not pass the threshold.
Decision: Reject