Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper the authors propose an architecture based on variational autoencoders and hyper-networks. The basic idea is that the weights of the underlying RNN/autoencoder are not fixed, but are coming from another RNN/feed-forward network which captures the underlying dynamics and adjusts the weights accordingly. The experimental results show the benefit of the model compared to a similar method without hypernets.
 
In terms of novelty, the combination of auto-encoder RNNs and hyper-networks is not entirely novel and it has previously been developed (https://www.biorxiv.org/content/10.1101/658252v1). However, while I think these previous works should be discussed in the paper (they are not currently), the two architectures are sufficiently different and the current work is novel enough in my opinion. On the other hand, in terms of presentation, I think the paper can be improved. The architecture is not entirely clear from the text. I think a graph showing the architecture of the model would be very helpful here. The notations also seem loosely defined (what is the dimensionality of x_t, z_t, etc.) and sometimes undefined (e.g., x_t in equation 1 is not defined).  
 
In terms of model architecture, it wasn’t clear for me why the hyper-network for \phi is feedforward but the one for \theta is RNN?
 
The experiments seem promising, but I have the following questions before being able to assess the results:
 
- How many LSTM units are in the model in each experiment? Are they similar in both VHRNN and VRNN?
 
- What is the structure of encoder/decoder layers? How many units in each layer? Are they the same for VHRNN and VRNN?
 
-There are four sets of weights in the primary model: weights of RNN, dec, enc, prior. How are these weights generated by the two hyper-networks \theta and w?
 
- How is the number of parameters in the experiment calculated. Do they refer to the number of parameters in the hypernetworks only?
 
 
Minor:
I suspect the spaces between the equations and captions are also manually changed which has made the paper physically dense and a bit unreadable (see equation 4 for example). Similarly, the space between the caption of Figure 1 and the text after seems too small.
 


review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposes a variational hyper recurrent neural network which is a combination of the variational RNN and the hypernetwork. The hypernetwork is an RNN whose output modifies the parameters of the variational RNN dynamically at runtime. Overall, this seems like an extension of the idea of using a hypernetwork with the VRNN (rather than the RNN as done in Ha. et. al). The model is trained via the FIVO objective. The model and learning algorithm are compared to the variational RNN and tested on a variety of synthetic settings where the VHRNN outperforms the VRNN in held-out likelihood. The performance gains are investigated on synthetic datasets where the paper notes that the VHRNN is often quicker to adapt variations that happen within seqences (for example, the paper considers a dataset where multiple patterns are stitched together into a sequence and study the changes in the KL divergence and reconstruction at switch points). On four real-world sequential datasets, the paper finds that the model outperforms the VRNN across many configurations and with a fewer number of parameters.

Summary: I don't think the model presented here is very novel, in that it is a combination of existing ideas; however, the paper does a good job of studying the model in a variety of different configurations on both synthetic and real-world data. The model does appear to consistently outperform the Variational RNN of Chung et. al.

Questions and comments:
(a) I do not think the word "System Identification" should be used on page 5 to describe the results of Figure 2. Doing so would overload existing notation in the time series literature where the word refers to the identification of parameters under a pre-specified physical system.
(b) How well does the Recurrent Hyper network (with no latent variable) do on the tasks considered here? I understand that it may be a less expressive model in general, but it is not clear to me why it would not be a competitive baseline on some of the smaller datasets considered here -- was this baseline tried?
(c) Did you experiment with non-temporal architectures for the hypernetwork? Since z_t and h_t-1 (which are conditioned on) contain information about the history of the sequence, one might argue that conditioning on them might suffice to predict the modifications to the parameters of the theta and g.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes the variational hyper RNN (VHRNN), which extends the previous variational RNN (VRNN) by learning the parameters of RNN using a hyper RNN. VRHNN is tested and compared with VRNN on synthetic and real datasets. The authors report superior performance parameter efficiency over VRNN.

The performance of VHRNN is promising and certainly better than the previous VRNN for some applications. However, the VHRNN is constructed by a straight-forward combination of existing techniques and hence the technical contribution of this paper is marginal.

Although Section 4 is entitled as systematic generalization analysis of VHRNN, the reported results are only for the specific structures of VHRNN and VRNN. Isn’t it useless to present results for the VRNN with a latent dimension of 4, at least as a sanity check? 

Fig. 2 and the texts referring to it discuss the KL divergence between the prior and the variational posterior. While the FIBO is mainly used as the objective in this paper, is the ELBO enough if the authors care the simultaneous low reconstruction error and low KL divergence?

It is unclear and explained little if the comparison using parameter count is fair for VHRNN and VRNN since they have different structures.

It would be nicer to discuss for which kind of time-series VRNN is enough.

Minor comments:
The caption of Figure 1 is too close to the main texts.
Eq. (4) is overlapping with texts.
Can the equations at the bottom of p.3 be explained with an illustration?

review_title: Official Blind Review #3

Meta-review: The paper proposes a neural network architecture that uses a hypernetwork (RNN or feedforward) to generate weights for a network (variational RNN), that models sequential data. An empirical comparison of a large number of configurations on synthetic and real world data show the promise of this method.

The authors have been very responsive during the discussion period, and generated many new results to address some reviewer concerns. Apart from one reviewer, the others did not engage in further discussion in response to the authors updating their paper.

The paper provides a tweak to the hypernetwork idea for modeling sequential data. There are many strong submissions at ICLR this year on RNNs, and the submission in its current state unfortunately does not pass the threshold.
Decision: Reject