Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper draws inspiration from Physical world and considers an independent mechansim among recurrent modules. The authors apply the proposed RIM to several relatively simple tasks and show some advantages.

In general, I like the idea of making recurrent cells operate with nearly independent transition dynamics and interact only sparingly through the attention bottleneck. It is essentially to combine some environment prior into the model design. It makes senses to me that RIMs will work better in environments that objects and background are nearly independent and only interact with each other when collision happens. RIMs share similar to spirits with capsule networks, and its recurrent cells serve somewhat similar role to capsules. Such independent mechanism, selective activation and sparse communication is very inspiring and is indeed a potentially very useful way of modeling the physical world.

For the model itself, I appreciate its simplicity, but I also have some concerns.

1) For the selective activation of RIMs, the number of activated RIMS is a hyperparameter and needs to be pre-defined. According to your experiments, I believe you need tune this hyperparameter a little bit in order to obtain the best performance. First of all, the design of a fixed number of activated RIMs does not seem to be reasonable and is also highly dependent on your task. I believe the framework will be more interesting if the model can determine this number automatically.

2) I find it quite interesting that the top-down attention in selective RIM activation is corresponding to the states of these recurrent cells. I am wondering what if you do not select these top K activation and directly train it using the entire distribution of the soft attention output?

For the experiments, I think they all serve the purpose of showing the advanatges of RIMs quite well, except that thery are relatively easy task. However, it is still interesting to see that RIMs obtain significant gains over some baselines. Some of the details can be made more clear, such as loss function and evaluation metrics in every task. It is sometimes difficult to find what loss function you are using. I suggest the authors make the experiments more self-contained in the main paper, such that authors do not need frequently scroll down to the appendix and check the details.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a neural network architecture consisting of multiple independent recurrent modules that interact sparingly. These independent modules are not all used simultaneously, a subset of them is active at each time step. This subset of active modules is chosen through an attention mechanism. The idea behind this architecture is that it would allow the different modules to specialize in different mechanisms and that would allow compositionality. The empirical results suggest that the proposed approach is able to generalize better than traditional architectures (which all have the implicit assumption that all processes interact).

This paper is well-written and it provides a very thorough empirical analysis of the proposed idea. Because it is not in my area of expertise I’m not confident that I can assess its novelty or its relationship to other existing approaches.

In terms of presentation, I recommend the authors to enlarge some of the figures in the paper (e.g., I can’t read the small box in Figure 1) and to not use citations as nouns (e.g., “The mechanisms of this attention mechanism follow (Vaswani et al., 2017; Santoro et al., 2018), with the …”). I would also like to point out that although fairly different in how they tackle the problem, the work of Arjovsky et al. (2019) seems to be related to this one.

Three questions I believe were not answered in the paper are:

1) How is the performance related to the total number of subsystems (and the number of *active* ones). I can only see results related to that in Table 1, but the variation in the number of modules is pretty small (4-6). The results also don’t give any indication whether we want to have more modules active at each time step, if there’s a sweet spot, etc. It is said that the method seems to be robust to this choice but this claim is made because it performs similarly for the values 5 and 6 if I recall correctly.

2) Is there any incentive in this architecture for a module to not simply “give up”? I mean, the modules are not necessarily incentivized to be used as often as possible, so could it be the case that a module learns to set its weights to zero?

3) Would it make sense to present baseline results for an architecture that uses attention? It seems to me that LSTM was often the baseline of choice but RIMs have two important components: multiple LSTMs and an attention mechanism. Could the attention mechanism be explaining some of the results we are seeing?

Finally, despite the very long appendix, I feel there are important details missing with respect to the empirical setup, at least in the Atari experiments which I’m more familiar with. Was stochasticity used, that is, sticky actions (Machado et al., 2018)? Moreover, for how long was PPO (and RIMs-PPO) trained in terms of number of frames? Finally, I’d recommend the authors to include a table with the actual average (and standard deviation) performance in each Atari games. It is really hard to know how well a method is doing by just squinting at learning curves. It is hard to know if the results are significant without a notion of variance.


References:

Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, David Lopez-Paz: Invariant Risk Minimization. CoRR abs/1907.02893 (2019)

Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, Michael Bowling: Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. J. Artif. Intell. Res. 61: 523-562 (2018)


------


>>> Update after rebuttal: I stand by my score after the rebuttal. 

Unfortunately I'm not an expert in this area and I don't feel confident in having a very strong opinion about this paper, willing to fight for its acceptance. I also agree with concerns raised by other reviewers. As I stated in the discussion with the authors, the clarifications and additional experiment does improve the paper a bit.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This work seems to propose an alternative to general RNN so that  the dynamics of sequential data can be better captured. The work is based on a hypothesis that a causal process can be modeled by "independent" modules and sparse interactions.
 
The paper is written in very fluent English, but the style is less technical. The impression is that the philosophical arguments and the machine learning realization have a gap in between. There is no much rigorous mathematical definition or derivation to back up the entire development. The mathematical symbols are a bit loosely defined. For example, it is unclear to the reviewer if h is a scalar or a vector.

The RIM idea seems to be derived from some ideas from the ``'causality community' . But the authors did not elaborate how significant will this structure change the state of art. In particular, reading the abstract or the introduction does not shed much light on what are the challenges of now the ML community is facing, and how this proposed RIM idea is going to help. This may have been obvious to the authors, but spelling them out may help the reviewer/readers to understand the contribution. The related work section helped a bit, but still unclear.

The reviewer feels that the paper stands at a high level in general, but lacks concrete examples/applications for general readers to appreciate the significance. Perhaps trying to re-organize this part could greatly help the readability.

The mathematical descriptions in 2.2, 2.3 and 2.4 are very hard to follow. There is no cost function (for training) around, but there are discussions of  'gradient'. Gradient of which function? 
review_title: Official Blind Review #1

Meta-review: This paper has, at its core, a potential for constituting a valuable contribution. However, there was a shared belief among reviewers (that I also share) that the paper still has much room for improvement in terms of presentation and justification of the claims. I hope that the authors will be able to address the feedback they received to make this submission get where it should be.

Decision: Reject