Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper models noisy observations from complex dynamical systems, consisting of multiple interacting subsystems, by a set of sparsely interacting recurrent networks. The interactions between the recurrent networks (or modules) are constrained to be spatially localized, this is done by embedding the position of each module in a metric space and scaling the strength of the interactions between modules using this metric.

The overall effect of this constraint is to induce spatial structure in the global dynamics of the interacting modules. The paper argues that this constraint is a good inductive bias for capturing the dynamics of systems that consist of sparsely interacting agents, like cars and traffic lights in a traffic simulation, or characters and actions in a video game.

The paper uses recurrent neural networks for modeling the individual sub-systems, as these are flexible nonlinear dynamical systems that can model the (local) dynamics of individual modules. The interactions between these systems is captured through an attention mechanism. These attention weights are modulated (scaled) by a similarity between embedded positions of pairs of modules.

I found the paper interesting, but I thought the presentation could be clearer, and the paper could better demonstrate the importance of the spatial structure.

First, as far as I can tell, the main difference between this work and previous work (specifically, the recurrent interacting modules (RIM) paper) is the addition of the positional embedding and corresponding weighted interaction terms.

Given that the key novelty is the positional embedding, I think the paper should spend more time introducing and motivating how that is implemented, and experiments testing that implementation.

The positional embedding is introduced in a few lines at the top of section 4, with the motivation that the chosen function is "commonly used" (and cites the original transformer paper). However, the motivation for positional encodings in NLP tasks for the transformer seems to me to be completely different from the motivation for positional or spatial embeddings in this work. This work is (largely) aimed at modeling interacting systems in the physical world, whereas the positional encoding in Vaswani et al was motivated as a simple way to introduce signals to the network that represented position. Given that these are different domains, I think the positional embedding should be better motivate here. In addition, the paper suggests that "other choices might also be viable". What are these other choices? Did the authors experiment with them?

Second, how are the hyperparameters for the similarity metric (epsilon and tau) chosen? These govern how close two subsystems need to be to interact, but as far as I can tell they are simply presented as constants in a table in the appendix. How does performance vary as you change these? At the very least, the main text should state how these were chosen.

Third, from what I can tell, the S2GRU has less capacity than a system that models all of the interactions, such as a global LSTM. If so, then how come the LSTM underperforms on simple tasks such as the bouncing balls? Presumably, the LSTM could just learn the same interactions that are in the S2GRU. Is this underperformance due to issues with trainability (the global LSTM is harder to train?) or perhaps it still has limited capacity (e.g. it does not have enough units or layers to model the dynamics?). If the latter, it would be nice to see experiments with different numbers of units and/or more training data, showing that for large systems the LSTM is as good as (or better) than the S2GRU (since we expect the LSTM to be able to learn any nonlinear dynamics). Then, as you reduce either the number of units or training data, perhaps the S2GRU starts to outperform (as the inductive bias of the S2GRU starts to win). Basically, if the spatial structure is really an inductive bias, I would expect the benefit to go away with a higher capacity model or more training data (where inductive biases are not needed, as we can just learn directly from data).

Fourth, are there cases where the spatial structure is *not* a good inductive bias? For example, situations where modeling the interactions between all pairs of agents or subsystems is necessary to capture the dynamics. If so, I would like to see experiments on these problems that show that the S2GRU does *not* outperform a global model such as an LSTM. This would be a nice control to show that the limits of the inductive biases of the imposed spatial structure.
review_title: Interesting idea, hard to assess the importance of the proposed spatial localization.

final_score: rating: 7: Good paper, accept
review_content: The authors propose a recurrent state space model for partially observed data, where at any given time, only a partial subset of observations are accessible to the model to make future predictions. This is an important and challenging problem in ML as we often only have access to a partial view of temporally evolving data.

The authors propose uses a recurrent model, which models a dynamically evolving partially observed process by introducing spatio-temporal interactions across multiple recurrent neural nets. The key contribution of their work is the use of a shared embedding space, that allows them to employ a mechanism to modulate spatial and temporal interactions between the RNNs.

The proposed method learns a mapping function for both, the embeddings corresponding to the RNNs, and the embedding corresponding to the observations and locations, to a shared metric space. Since the functions used to map the observations and locations, and the embeddings vectors associated with each RNN, are learned, the joint learning objecting corresponds to learning a topological structure on the points in this shared metric space. Additionally, the authors propose a truncated kernel to attenuate effects of observations far apart in time. 

The final learning objective then encourages the shared metric space to be topologically organized in a way such that the embedding space is partitioned into regions, where each region places a varying “responsibility” over each of the different RNNs. Since the observations are also embedded into this space, any given observation can then be mapped into the embedding space, in the embedding space, the observation embedding be close to one or more embedding points corresponding different RNNs (which in turn models the dynamics in that regime).

The idea itself is quite interesting and allows for a “soft” representation of a state space model which through the embedding respace resembles an explicit duration + factorial Markov model. The shared embedding space is an interesting idea and can be extended to other time series models as well. 

The experiments along comparisons to strong baselines demonstrates the validity of their approach. In addition to the interesting core idea, the authors employ a fairly involved attention mechanism. Given the space limitation it is understandable, however, it would have been informative to have some experiments with some ablation studies i.e. what is the simplest model structure they could have used while still incorporating the shared metric space idea to make the model more parameter efficient.


Some minor notational issues:
D_x seems to be defined as a function space in the model, however, it is not a function space. One could think of the forward evolutions of an RNN with a memory cell having an equal representation in function space, however, I feel the function space characterization of dynamical system section 2 obfuscates the model structure. Additionally, s_t^a can be inferred to be P(x_t^a), however, s_t^a is not defined in the text.


A few clarifications from the textual descriptions:
Could the authors elaborate on how exactly does E(.) process all observations in parallel across t and a? 

“Explained by the fact that unlike recurrent models, it does not leverage the temporal dynamics to fill in the missing information due to fewer available observations.” Could you elaborate on why you don’t expect the individual RNNs to leverage the temporal dynamics?

In the experiments, does the reported metric F-1 score account for label switching?

As currently modeled, only the forward rollout of the RNNs are used, could this be extended to having a Bi-directional RNN – one might expect better global estimates as we switch from “filtering” to a “smoothing” estimate

Overall, I believe this an interesting research direction and the approach proposed by the authors can be extended to other useful models. The experiments are well thought out. The paper organization could be clearer but as it stands it is easy to understand after a thorough read.




review_title: Learning shared topological structure for partially observed data: an interesting idea with wider applications

final_score: rating: 7: Good paper, accept
review_content: 
### Summary of the paper

The authors propose a novel architecture for synthesizing information from multiple local observations and making predictions into the future.
Given a set of observations, each with an associated location, the model maps these into an embedding space.
A set of RNNs, each with a learned embedding that corresponds to a vector in the embedding space, is used to process observations that are close in the embedding space.
The activations that the RNNs can receive, exchange between each other and output for a given query location are modulated by the distance in the embedding space.
This is achieved through three separate attention mechanisms with a similar design: A fully non-local (all-to-all) attention layer is followed by a multiplication with a kernel that falls off (and is eventually cut off to 0) with distance.
This type of attention is used once for computing the inputs to the RNNs, once for the hidden state of the LSTM (exchanging information between RNN modules that are close by) and once when computing an output response given a query location.

### Relation to prior work

The paper references previous work that deals with synthesizing information from multiple localized observation and positions itself reasonably with respect to prior work.

### Experimental evaluation

The authors demonstrate results on two domains: A bouncing balls video prediction task which is well-known and useful but could be considered a toy task and predicting sequences from StarCraft 2 battles. In the case of StarCraft 2 the target for the prediction are raw unit stats and actions rather than images.
The model generally outperforms baselines on the bouncing balls task and generalizes better on the StarCraft 2 domain (but does not outperform baselines on the training task itself).
It's difficult for me to judge whether the model will also work on other domains based on these experiments.

I'm curious whether some of the components of the model are actually needed (e.g. attention between RNN modules). In my opinion it would improve the paper significantly if some results on ablation experiments would be provided. Another aspect of the model that could be investigated is the dependence of the performance on the kernel that modulates information exchange by distance in the embedding space. What happens if the modulation is disabled?

### Presentation and clarity

The paper is generally well written. But I did have some difficulties understanding the goal and approach because the paper relies on a lot of wordy exposition (some of which is highly speculative in my opinion). I believe that the paper would be clearer if the introduction was shortened. This could also leave more space to discuss the experiments in more detail or add an ablation study.

### Conclusions

I found the architecture introduced in the paper genuinely interesting and would like to see more work in this direction. I would give the paper a higher rating if it included ablation experiments, or if it was adjusted to simplify and shorten some of the discussion in the introduction.


==========

Edit after author comments:
I've read the author comments and the updated version of the paper.
Although the authors claim that they have shortened the introduction of the paper by 1 page, this doesn't actually seem to be the case in the last version that was uploaded, where the section titled "Introduction" is almost unchanged compared to the original upload (I've used the diff tool between the latest and original version).
Maybe there was a misunderstanding and the authors have shortened a different part of the paper?
Although it would have been nice to shorten and streamline the introduction to make the paper easier to read, it's not critical to my rating.
The added ablation experiments demonstrate that each of the different attention modules proposed in the paper improve results, which I think really improves the paper. I was originally not sure whether the complexity of the model was justified, but the new experiments demonstrate that each of the components seems to be needed.
I've also carefully read the rest of the paper and the author comments explaining details of the tasks under study and now feel that I have a much better understanding of what was done and how the model could be used for other tasks.
I agree with R4 that the novelty of the paper might not be groundbreaking, but I believe the paper could be relevant and interesting for other researchers who want to incorporate attention mechanisms into their architectures, so I recommend accepting the paper.
I've increased my rating from 6 to 7.
review_title: A really interesting architecture. Evaluation tasks could be more challenging. Ablation is missing. Clarity of the paper could be improved.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary

This paper is concerned with making predictions about the (global) state of a dynamical system in a partially observable setting, where only local observations are available. Concretely, this paper studies video prediction from glimpses and world-modeling in a multi-agent setting.

To solve this task the paper proposes Spatially Structured Recurrent Modules (S2RMs), an RNN-based dynamics model based on interacting sub-systems. S2RMs are an extension of RIMs (Goyal et al., 2019) with the main difference being that each module additionally contains an embedding vector that is meant to reflect its location in some learned metric space. This positional information can then be used to limit the interactions between modules, to distribute inputs (that come with an associated location) to the modules, and to query a particular subpart of the global state at a future point in time.  

It is shown how S2RMs outperforms a number of baselines, which are obtained by combining the query mechanism of a GQN with either an LSTM, RIMs, or a Relational RNN to model the dynamics.

### Pro’s / Con’s / Justification

Overall I find that the paper is reasonably well written, although the clarity can be improved significantly. The first _four_ pages essentially only focus on motivation and problem statement, which is excessive, especially since the main contribution is an improved RNN architecture. In contrast, Section 4 that describes the method leaves important details about the adapted input attention mechanisms and inter-cell attention mechanisms to the appendix. Similarly, the experiments on the grid-world are entirely discussed in the Appendix, which I have therefore not considered as part of the contribution. More generally, the problem formulation seems unnecessarily broad given the relatively narrow contribution that is made.

The considered problem setting is interesting and I believe relevant to several real-world settings. The contribution itself is rather incremental since it essentially only involves associating a location with each module in RIMs. The adaptations to the existing attention mechanisms in RIMs that this then necessitates are straightforward, although additionally having the RIM input attention over the local observations is interesting and appears novel (I couldn't figure out if the original RIMs already attend to a set-representation of the input -- i.e. where each element is a different spatial location of the learned CNN representation). However, this paper does not compare alternative ways of implementing these changes or provide an ablation, which leaves it open what the effect is of adapting the attention mechanisms in this way. I also note that requiring the local observations to include an associated spatial location is arguably a more narrow setting than that explored in RIMs (if the input attention in RIMs is additionally directed to attend over the encoded local observations) and it would have been interesting to understand the importance of this. For example, how do S2RMs compare to RIMs when a similar encoder is used for RIMs (except for using the spatial coordinate)?

The experimental evaluation indicates that S2RMs outperforms the considered baselines and most notably RIMs, although there are some concerns. Firstly, for the baselines, the aggregated representations are computed as a sum of the output of the encoder applied to each local observation, which seems like a major bottleneck. While S2RMs can use attention to attend to each patch separately, information is almost certainly lost for the baselines in this way. I can understand why this may be necessary for the LSTM, but why not let RIM attend to the encoded local observations separately? Currently, this makes it difficult to understand in what way S2RMs improve over RIMs, especially since the reported margin already is quite small. Secondly, and related to this, I believe that it is important to include an ablation of the changes proposed to the attention mechanism. For example, what happens if positional information is only used for the recurrence, but not for the input attention (i.e. only using the global term)? Or what if it is only used for input attention, but not for the recurrence? Understanding the effect of these design choices would help strengthen the contribution and make it more significant compared to RIMs.

Given these issues I can not quite yet recommend an accept at this point in time, but I would be willing to increase my score depending on the outcome of some of the experiments I have suggested. More generally, I encourage the authors to revise Section 2-4 to put more emphasis on the actual contribution and discuss the specific design in detail.

### Detailed comments

* Why is it necessary to choose between the product of local and global terms and only the local term for the input attention? It seems to me that the local terms should suffice, and I wonder what the effect is of having this additional term.

* I would appreciate a more detailed discussion of prior work in the related work section. Currently, the paper only makes ‘sweeping claims’ about how the considered setting is different from a bunch of prior approaches. Rather, it would be more interesting to point out specific parallels, or discuss how ideas from prior work may be included in the considered set-up, or in what way those ideas have already been re-used.

* I am also missing a discussion of object-centric approaches to performing physical prediction tasks, like RNEM (Steenkiste et al., 2018), OP3 (Veerapaneni et al., 2020), etc. especially in relation to the considered bouncing balls task. Arguably, on this task these methods provide the best trade-off between locally interacting sub-systems and more global modeling since each RNN specializes to a specific object, which is evidently not achieved by S2RMs. Since each RNN learns to specialize on a single object, it can easily be viewed as modeling the global state of the system through locally interacting subsystems acting on partial observation. On the other hand, it is clear that S2RMs (and RIMs by extension) offer other advantages. For example, an advantage of S2RMs is that they consider modules having their own weights, which can thereby specialize on specific interactions between objects (or break down interactions between objects further). RNEM and the likes are limited to modeling the same set of interactions between interacting subsystems given by objects, although evidently, the notion of an object can also be flexible in this case (i.e. when an object-centric representation corresponds to two physical objects in pixel space).

* The visualization in Figure 5, suggests that individual modules specialize, but there also seems to be quite some redundancy. Would it not be possible to quantify the achieved modularity somehow, eg. by measuring IoU? In that case it would be interesting to see how the system behaves when the number of provided modules is insufficient to model each individual actor in the considered system, and when the number of modules is equal to the number of actors (eg objects on bouncing balls). Additionally, do you have any intuition for what happens if modules are removed at test-time? 

* Why are RIMs not included in Table 1? I would be surprised if it performs that much worse to S2RM, since both models perform similar on the bouncing balls task. If this is the case then please at the very least provide some intuition or insight to explain this difference. 

* Please include a discussion of limitations in the conclusion. 

* Please take a moment to go through the references and correctly cite papers that have been published.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

I have improved my score following the improvements made by the authors. See my reply below for details.
review_title: Interesting problem setting and reasonable approach but important experiments are missing and I have concerns regarding the experimental results

Meta-review: This paper presents a model for dynamical systems with multiple interacting components. Each component is modeled as an RNN, and the interactions between components are functions of their distance in a learned embedding space. It's an interesting idea and well motivated inductive bias. The results were made more compelling with the addition of "ablation" studies during the discussion phase, which showed how various aspects of the model combined to yield the best performance.  Overall, this paper should be of interest to many in the ICLR community working on complex, multi-agent systems.
Decision: Accept (Poster)