Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This manuscript proposes a novel approach to dynamic modeling of time series data based on a state space model that incorporates a Graph Neural Network to model the relationship between the dynamics of different objects. The (complex) architecture is described in details and to some extent justified before its predictive performance is demonstrated on several simulated and real datasets.
Overall, while the authors provide evidence of a better predictive performance with respect to several baselines, I am left a bit unconvinced by the study for the following reasons:
1.	Lack of relevant baselines: it seems clear that the overall purpose of the approach and the nature of the dataset require fitting a state space model, however, baseline are overall focused on recurrent and autoregressive models which seem underequipped to address these problems. I wonder if the choice of more relevant state space model baseline would convincingly show a true benefit of the proposed approach. In particular, there is likely a large number of variations of the Kalman filter and particle filters that might be relevant. For example, the ensemble Kalman filter has proved accurate for weather forecasting.
2.	Lack of interpretability: given the proposed approach relies on an intuitive representation of the model as representing the dynamics of several object tied by an interaction graph, the purely predictive results are not really matching the expectation of the reader to “see” how well these interactions are captured by the model. Can we check in some way that the latent graphical model is learnt properly, even in a toy dataset?
3.	Writing of the methods section: This is a more vague comment, but while reading the overall description of the approach, one is left wondering how critical are each part of the model and whether some complexity could be spared. Moreover, some descriptions are difficult to follow, perhaps for the reader less familiar with customary design choices in dynamic neural networks, e.g. the objective in section 3.4. In addition, some statements seem at least to lack justification, e.g. stating that AR approaches lead to unimodal distributions at the bottom of page 2.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: * Summary:
The paper presents a relational state-space model that simulates the joint state transitions of correlated objects which are hierarchically coordinated in a graph structure. A structured posterior approximation is developed based on sequential graph neural networks. Two auxiliary contrastive predictive losses are proposed to help circumvent the posterior collapse problem. Graph normalizing flow is further incorporated into the framework to make the joint state transition density more expressive. The proposed R-SSM shows performance gains over state-of-the-arts in three benchmarks.

* Comments:
The paper is generally well written and technically sound. The framework, including formulation of each of its components, is well defined. It is also helpful that the authors included preliminaries of the literature. The number of experiments are adequate. However, there are a few parts that require more extensive clarification and analysis.
1. Different parts of Section 3 appear to be rather disconnected, the reader still has a hard time figuring out how the learning of the whole framework is carried out. It is desirable to include a sketch of learning algorithm.
2. In the formulation GNF, what is the intuition or principle to decouple the state Z_t into two parts Z_a and Z_b? How does the mapping of Z_b into Z'_b help to make the state transition distribution more expressive?
3. In Section 5.3, the authors mention that GNF was not used due to memory cost. Could it be discussed more thoroughly about the complexities of learning R-SSM and GNF?
4. The model keeps track of a global state z^g, but it is not analyzed in experiments. It is strongly recommended that the authors discuss about the (global and individual) states and their transitions. It would provide great insights on how multiple objects interact with each other.

Minor point:
1. Please clarify what is X_{t-h} in Table 3?
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary of what the paper claims and contributes
---
This paper proposes a hierarchical latent variable model of sequential dynamic processes of multiple objects when each object exhibit significant stochasticity. The model leverages Graph Neural Networks to design the architecture of the model. Normalizing flows are used to construct "Graph Normalizing Flows" that model transition densities of the per-object latent random variables, although the full model has more latent variables than tabouthe per-object latents. The approach applies and combines a host of methods to design the objective, the model, and optimize the model under the objective -- for the objective: variational inference, Contrastive Predictive Coding; for the model: normalizing flows, graph neural networks (and their combination, termed by the authors as Graph Normalizing Flows); for the optimization: variational sequential Monte Carlo. From my understanding, the main claims are that the model is a flexible way to incorporate relational information, and that the resulting model outperforms other multi-object time-series forecasting approaches.

Experiments are conducted on a synthetic toy dataset of 1d observations per-object, a real dataset of basketball player movement, corresponding to 2d observations of each object's position (the agent's and the ball's positions), and a real dataset of traffic speed forecasting (1d observations per-object). The experiments demonstrate that the model compares favorably to a baseline and a few other time-series forecasting models from the literature. The experiments show that the method achieves higher test log-likelihood (lower-bounds), and lower MSE than the compared methods. The paper discussed many related works, but it's not clear why the specific methods were chosen for comparisons. More motivation is needed for these comparisons.

Evaluation
---

>Originality:
Are the tasks or methods new?
The method is new.

Is the work a novel combination of well-known techniques?
Yes, albeit a lot of techniques.

Is it clear how this work differs from previous contributions?
Besides the fact that the proposed method is a combination of many techniques, the main / key technical difference was actually unclear to me.

Is related work adequately cited?
From my understanding, yes, mostly; there's just a few places where the paper could be situated better:
- Sec 1: "as they have been shown to be fundamental NN building blocks ...". Citations needed (e.g. include the Graph NN citations later here)
- Sec 1: "R-SSM achieves superior test likelihood and good prediction performance". When compared to what methods (citations to VRNN and GNN-AR/NRI needed here too)?
- The paper contrasted the work to other dynamics modeling works with the statement "R-SSM differs from all these works by introducing structured latent variables to represent the uncertainty on state transition and estimation". The paper also said VRNNs were the most related, because they employ "hidden states of per-agent VRNNs interact through GNNs". The paper (PRECOG, arXiv:1905.01296) appears related as it also constructs a latent variable graphical model of stochastically-transitioning multiple interacting objects with normalizing flows. Like the authors said of Graph VRNNs, it appears that one of the biggest differences to PRECOG looks like the proposed method includes more sources of stochasticity outside of the objects (the global uncertainty) to model the observations.

>Quality:
Is the submission technically sound?
Yes, although not much motivation is given for many of the specific design decisions adopted. The model is a complicated combination of many techniques from the literature, which makes it difficult to understand the importance of each specific piece of the model.
- For instance, why was global stochasticity introduced? Because per-object stochasticity is modeled with normalizing flows, the model would have been trainable with maximum likelihood estimation of a distribution over observations that's efficiently analytically computable, but after the addition of global stochasticity, the joint distribution over the random variables becomes intractable, and training must resort to variational inference. This leads to, as the paper mentioned, RSSM suffering from posterior collapse, which, to my knowledge, does not occur when training direct maximum likelihood estimation procedures (e.g. GLOW, RealNVP).
- As mentioned above, it's not clear what the key technical innovation of this approach is (Graph Normalizing Flows?) It would be good if the claimed innovation was made clear in both the abstract and the introduction.

Are claims well supported by theoretical analysis or experimental results?
The experimental results provide evidence that 1) the model can flexibly incorporate relation information and some evidence that 2) the model outperforms some multi-object time-series forecasting approaches from the literature.

Is this a complete piece of work or work in progress?
The submission appears complete.

Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
Yes, and they included discussion of when the model is under-performing w.r.t. other methods (at longer time horizons).

>Clarity:
Is the submission clearly written?
As mentioned above, many of the numerous design decisions are not motivated well. Essentially, the paper is not very self-contained.

Is it well organized?
Yes.

Does it adequately inform the reader?
No -- the paper assumes a lot of prior knowledge, which most potential readers would likely possess. This issue would be less significant if the main innovation was clearer.

>Significance:
Are the results important?
It's difficult to tell, because it's not very clear why, of the many cited works, the specific ones chosen for comparison were used. More motivation for the comparison is needed.

Are others (researchers or practitioners) likely to use the ideas or build on them?
Possibly, although the model is complicated, which makes it less likely.

Does the submission address a difficult task in a better way than previous work?
It's plausible that the model performs better, but it's unclear how widely-applicable the method is. The model appears to be restricted to lower dimensional settings.

Does it advance the state of the art in a demonstrable way?
It's plausible, but I am not sure, as mentioned above.

Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
No

Additional feedback
---
Sec 5.2 describe what "OOB rate of the rollouts" means.

Sec 3.3 Coupling layer equation is missing a final parenthesis.

3.3 "complicate the learning a lot." -> "significantly complicate the learning." (original is subjectively too informal)

Sec 5.2 "like to dig into it in future work" -> "like to explore it in future work" (original is subjectively too informal)

Recommendation
---
It is unclear what the main key contributions are, outside of usually performing better than some related methods. This lack of clarity, along with a complicated design that is not very well-motivated, make me lean towards rejection. An improved version of this paper would make clear what the precise contributions are, explain how the related work did not achieve them, and why the specific comparisons were adopted.

review_title: Official Blind Review #2

Meta-review: The paper proposed what is termed Relational State Space Model (R-SSM) that can be used for modeling interacting time-series data. The model essentially consists of a set of (nonlinear) state space models whose states are jointly evolved in a way that take into account a known interaction structure between them (the relational part, even though technically it is just a coupling structure -- the term relational structure in the past has been used for models with objects and classes, for example see the difference between "coupled HMM" vs "relational HMM"). The authors also proposed a graph normalizing flow operation to model the joint state evolution. The main weakness of the paper is in the complexity of the model. However, from a modeling point of view, R-SSM seems suitable in situation when the interaction structure is known, and this is demonstrated in the experimental results when comparing against the baselines. 
Decision: Accept (Poster)