Venue: ICLR 2021 Conference

Reviews:

final_score: rating: null
review_content: Glad to see the main problem was just an error in the decription of the algorithm!  It makes much more sense now. We are increasing our score from 4 to 7. 

We'd even increase our score to 8 if the authors added an analysis similar to the one of Figure 3,  just showing which OF is used. As stated in the original review: The experiment could be to add/remove objects, and observe the number of object files, or to corrupt specific files and see which object becomes unpredictable, thus identifying which OF corresponds to which object.
review_title: Glad to see this was just an error in the decription of the algorithm! Score increased from 4 to 7 (8 under condition). 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The motivation and the proposal for splitting the schema from the procedural (representational) block makes sense. This is a good idea. A the authors build on top of RIMs, which have shown reasonable ways to model dynamical systems. However the paper itself needs to be improved and we need to evaluate the model more before publication. 

Firstly, the proposal that SCOFF is a direct alternative for LSTM or GRU and showing that it beats them is not entirely correct. SCOFF comprises of a GRU with a sequence of CNNs operations i.e., its doing more than what a GRU does? What exactly is that? And so when proposing evaluations it is expected that compared to GRU, SCOFF does better (Fig 5). A more valid comparison would be GRU+some standard CNN-style feature learning vs. SCOFF. Its not entirely clear how to do this -- and needs thinking. Secondly, while on Fig 5, the errors cannot be reported as a ratio with respect to GRU because this would miss the true error values; and we cannot know if there is significant difference here (especially with RIMs.vs SCOFF). 
Next, how is question 3 evaluated here? What downstream task is being considered? Am I missing something? 
Next, how doe we interpret the error change here i.e., what does 0.1 change in error mean here for better understanding where things are changing? 

Presentation of the paper needs a lot of improvement: 
Algorithm 1 in the Table needs better clarity. Firstly, a bulk of the notation in the SCOFF presentation is very confusing. Its hard to parse what is going on in each stage. The description in the steps is helpful but the motivation and sequence of operations within each step needs better explanation. 
review_title: Genaralizing RIMs and GRUs by splitting schema memory vs. per-frame feature learning. Good idea. Needs work. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new type of recurrent neural network architecture called schema / object-file factorization (SCOFF). This model contains multiple weight-sharing GRU cells. The input information is fed into each GRU cells through an attention layer. The output information is fetched from these GRU cells and mixed with another attention layer. The model is tested on several intuitive physics benchmarks and basic reinforcement learning environment. This model demonstrates superior performance than other modular RNN architectures such as RIM on specific tasks.
 
+ves: 

+ Overall, the paper is well written. Section 2 clearly explains the proposed model. Section 3 systematically compared the proposed model against other RNN architectures.

+ All experiments results covered in detail including hyperparamters and experimental setting. 

 
Concerns: 

- This paper uses large amount of neuroscience terminology and vague concepts, which are just renaming of existing concepts. This is not novelty or contribution. And it is unnecessary and inappropriate for the conference publication.

 
- The model is simply doing attention + weight sharing RNNs + attention. The only difference from Recurrent Independent Mechanisms (RIMs) is that the modules has shared weights - an inductive bias is be useful for specific tasks. The novelty is weak.

 
- The proposed model on intuitive physics experiments shows slightly better performance than RIM. However, all these experiments have extreme setting, which is for the model's inductive bias that all objects follow exact same rules and contain full input information. It's obvious that the proposed model will perform worse than RIM when not all objects sharing same rules. 


- For these module based RNN models, it is necessary to show that the attention layer is functioning as expected. Therefore, I would suggest that the paper add visualization of the attention layers.

=====POST-REBUTTAL COMMENTS======== 

I thank the authors for the response. All my concerns are addressed. I will increase my score to 6.


review_title: Strong inductive bias for specific tasks

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors propose SCOFF, a novel architectural motif, one with memory, which, as they describe, can serve as a drop-in for an LSTM or GRU within any architecture. It is inspired by the notion that when modeling a structured, dynamic environment (such as one with objects moving around), one must keep track of both declarative knowledge and procedural knowledge. They propose that these two types of knowledge be factored, creating an architecture consisting of "object files" (OF) whose evolution is governed by input, all objects, and  "schemata" which can be selectively applied to each OF.

They evaluate SCOFF along several axes:
(1) Does SCOFF successfully factorize knowledge into OFs and schemata? 
(2) Do the learned schemata have semantically meaningful interpretations?
(3) Is the factorization of knowledge into object files and schemata helpful in downstream tasks?
(4) Does SCOFF outperform state-of-the-art approaches?
To do this, they use several video prediction tasks as well as an RL task.

This is a very interesting paper, with a natural, novel motif. It is well-written -- the motif has several important attributes, and one can quickly come to understand them (though perhaps a more involved diagram, like Figure 2 but showing what parameters come into play where, might be useful). The experiments appear to be carefully done, with considerable effort, and they put forth interesting evidence towards an affirmative in each of the above four questions.

While the evidence put forth is useful, I do think there is considerable follow-up work needed to really demonstrate the efficacy of this system. In the realm of video prediction, the tasks considered are fairly simple, and certainly what is of true interest is video prediction much closer to the real world. With this, there are a wide variety of techniques and benchmarks (https://arxiv.org/abs/1804.01523 and its follow-ons come to mind as useful to quickly try). Much recent work has deferred the task of obtaining a reasonable encoding from/decoding to real-world images and assumed it in order to make progress on predicting the future within a fixed encoding (https://arxiv.org/abs/1612.00222, https://arxiv.org/abs/2002.09405, https://arxiv.org/abs/1806.08047) and have with them benchmarks to which the authors' method could be adapted. Certainly for some of these, SCOFF could be a subcomponent that augments these methods. I think these more complex future predictive tasks could better stress-test the structure of how information passes through SCOFF. Related, many of the experiments rely on the encoding/decoding provided by [Van Steenkiste 2018], which showed considerable success on the environments used, and it would be interesting to see how crucial that is.

I recommend acceptance. The paper is interesting, well-written, and the experiments are useful. While I look forward to more definitive demonstrations of the utility of this approach, I do think that the amount done is considerable and warrants publication, and these important follow-ups would take a great deal more effort and are for future work.

A more minor comment, I think more details could be given for the RL task, both in model implementation and in exactly how the test task is specified -- apologies if I missed but I only see train environment details in the appendix.

Minor, wrong "two"/"to" in "Single object with switching dynamics" experiments description.

review_title: Very interesting work with good experiments.

final_score: rating: 7: Good paper, accept
review_content: Brief summary of your review:

I like the main idea of having an “active memory” where each slot can choose which operation to perform. It is similar to the notion of variables and functions, which should be advantageous for systematic generalization, which is one of the most significant issues of current neural networks.

The paper focuses on the visual domain and is motivated by extracting objects and their behavior. However, the model does not use attention over the input. The only way that the proposed model can use different object files for different instances of the same object type is to have separate schemata (because all other weights are shared), which defeats the purpose.

Thus, sadly, I must reject the paper.

Review:

The authors focus on a very important question of current neural networks: systematic generalization. Their method is interesting: they factorize modeling the scene of objects into object files (memory slots) and schemata (different learned operations). This is related to but different from memory networks, which use a single controller (instead of the multiple schemata), and also to routing networks, which use a single state (instead of multiple object files) and multiple sets of weights describing the different operations.

The authors introduce their method as a way of modeling multiple objects, some of which may share behavior. This can be seen clearly from Figure 1, but also the tone of the whole paper is organized around this goal. However, the way the input image is handled makes this very implausible. Specifically, the problem is in step 2 of Algorithm 1:

The object files compete for the input. In terms of the Pacman example, this implies that multiple moving ghosts, each with its own OF, would have to compete with each other to see the input, which makes no sense: all of them should see the change in their respective object.

The other, perhaps even more significant issue regarding input handling is that there is no attention over the input image. I cannot see how this architecture can focus on a different subset of the input corresponding to different objects. To focus on objects separately, they necessarily have to have different schemata, because the rest of the weights are shared between OF. That is the only way of having a different transformation of the input (assuming the same initial hidden state). This defeats the purpose of schemata, which is to reuse computation when possible. The current setup would make more sense in other input domains, for example for textual input, where different “objects” are not represented by different subsets of a single input z_t.

In Figure 3 the authors analyze which schema the model uses for a single object slot. The result is very nice and it shows that the model learns to use different schema for different types of motion/rooms. However, especially because of the aforementioned issues, it is unclear how the model uses the object files. I would like to see a plot similar to Figure 3, just showing which OF is used. The experiment could be to add/remove objects, and see the number of object files, or to corrupt specific files and see which object becomes unpredictable, thus identifying which OF corresponds to which object. 

I still consider the paper interesting. But the issue above must be fixed, e.g., by changing the tone of the paper and shifting away from focusing on objects or switching the attention in stage 2 to attend to image regions and not over the object files.

More minor issues/questions:

In step 3, the attention is based on comparing the updated state to the old one. Why does this make sense? Is there an intuition behind this? Wouldn’t it make more sense to have fixed keys identifying the schemata?

In step 4, why do the queries use state from the previous time step as opposed to the current?

On page 4, step 2, $\kappa_k$ is mentioned, however, it should be $\kappa_t$, as $\kappa$ is independent of the object file.

On page 4, saying that Gumbel softmax “softens the output” is misleading. The goal is to have a hard, but differentiable output instead of a soft one. 

In the related work section, you compare to CNNs. The same argument of weight sharing can also work for RNNs. However, differently from SCOFF, CNNs can actually “attend” to different parts of the input image, thus in theory focus on different objects differently.

Related work: Routing networks are [1,2,3] are also related. Page 2 and 4: GRU is a variant of vanilla LSTM with forget gates (Gers et al, 2000).

On page 5, the last paragraph, it is described how the output is produced from the OFs. However, it is not clear how this attention works. What is the query? Is it a transformed state? If the transformed stats are concatenated to form the query, then it is not symmetric anymore, so it will not “ensure exchangeability of OFs”. It can just focus on a single OF, based on its index.

In the experiments section, page 6, for the “Single object with switching dynamics” why does the network need markers? Can’t it infer the schema to use just from the motion itself?

Page 6, last paragraph, it should refer to Figure 3 instead of Figure 4.

On page 7, “Multiple objects with multiple dynamics”, the authors use the dataset proposed by “Van Steenkiste et al.”, and claim that they outperform their baselines. But they do not compare to the original method proposed in the same work. How does it compare?

Could the 1.3 lines long appendix E be a footnote instead?

[1] Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning

[2] Kirsch et al: Modular Networks: Learning to Decompose Neural Computation

[3] Chang et al: Automatically Composing Representation Transformations as a Means for Generalization

******************

After rebuttal:

Glad to see this was just an error in the decription of the algorithm! Score increased from 4 to 7. We'd even increase our score to 8 if the authors added an analysis similar to the one of Figure 3, just showing which OF is used. As stated in the original review: The experiment could be to add/remove objects, and observe the number of object files, or to corrupt specific files and see which object becomes unpredictable, thus identifying which OF corresponds to which object.


review_title: Nice but no attention over input, and other issues

Meta-review: This paper proposes a modular RNN architecture called SCOFF. The work was inspired by cognitive science(object file and schema) and was built upon previous work RIMs. The method is validated on tasks having multiple objects of the same type.

Pros:
- It addresses an important problem in DNN -- systematic generalization.
- The proposal makes sense and is more flexible than RIM.
- Experimental results outperform baselines.

Cons before rebuttal:
- The presentation of the algorithm is not very clear due to some confusing notations and missing details of algorithm steps.
- The comparison with baselines might not be fair due to extra parameters.
- The novelty is limited, because the only difference from RIM is weight sharing.

The reviewers raised concerns listed in Cons. The authors successfully addressed concerns: they indicated that the comparison was fair with the same input to both; SCOFF is more flexible than RIM, and there is spatial attention to input.
The authors added the missing details in the revised version.

All reviewers agree that the problem is important and the idea is interesting.  Since the authors' rebuttal was very helpful in clarifying the questions raised, I recommend accept.

Decision: Accept (Poster)