Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a training method based on ideas on neural algorithmic reasoning. As well using a model architecture that is aligned with the underlying algorithmic process this method proposes to pre-train the reasoning module using data from an abstract output stimulator (the work considers problems where this simulator is readily available, and so can generate lots of abstract data cheaply). A key motivation is that the reasoning module $P$ should ultimately be a mapping between two medium dimensional latent spaces (lower dimension than input space, higher dimension than the abstract representation space). Finding this balance allows the module to avoid modeling high dimensional data, whilst also not introducing low-dimensional bottlenecks into the architecture.  This work finds that inserting the pre-trained $P$ into a model that processes raw input data leads to improved performance compared to end-to-end training of a model on raw input data. 
main_review: First, this paper addresses a very important topic. The fusion of deep learning and classical algorithms is a frontier in AI research with significant potential going forwards.

This paper is a somewhat unusual submission. The methodological contribution is minimal (the approach is easily described in words). The work doesn’t follow the usual sorts of patterns e.g., explaining why something works, or shedding new light on some occurrence. Instead, the main contribution of this work is to spell out a philosophy on how to combine deep learning and classical algorithms. It is for this reason that I wouldn’t be surprised if the paper receives some negative reviews. However, I myself find the conceptual contribution to be valuable, with the potential to change the way others think.

With this in mind, here a few high-level strengths and weaknesses of this work:

**Strengths:**
- The paper addresses a fundamental conceptual question, a nice change from many papers I read.
- The approach is simple and general, and could be replicated for other related problems.
- Good empirical performance compared to C-SWM.
- Excellent writing.
- Proper treatment of statistical significance. All too uncommon in our community. 

**Weaknesses:**
- Light on ablations. There is a study of different architectures for $P$, but nothing beyond that. I’ll try to offer some suggestions below, but the authors probably can think of many more. 
- As best I can tell the authors don’t plan on releasing code (please correct me if this is untrue). 
- I wouldn’t call this work reproducible in its current state. E.g., the paper mentions in passing a skip connection in $P$ but I was unable to find further information on this (please point to it if I missed it). Hyperparameters are given, but a lot of the implementation details are left the to imagination. 

**Questions and suggestions:**

- One of the central ideas in this work (also present in a couple of prior works) is that of a dimensionality bottleneck if one were to fix the reasoning module $P$ to have the same low dimensionality as the abstract inputs (size, position etc.). While this is an intuitive point, and somewhat observed already, it is not yet a widely known thing. So I would suggest an additional ablation that shows that RMR performs favorably compared to an architecture with a dimensionality bottleneck. I fully expect RMR to do better, but I would be interested in seeing the delta so I can understand how much of the work is being done by the avoidance of a dimensionality bottleneck.
- Any thoughts on why with Battlezone as the test game the performance is often “no different” from C-SWM (Fig. 4)? Perhaps the reasoning module plays a less important role for this task?
- The fact that $P$ can be transferred effectively between tasks is somewhat surprising to me. It makes me wonder: what is $P$ learning that is helping performance? It seems it isn’t the precise steps of the algorithm in question… 
- The authors mention a greater need for careful hyperparameter tuning. It would be nice to quantify/demonstrate that point somehow. 
- The method doesn't require $(x, \bar{x})$ pairs etc., as you mention. I am curious how close RMR comes to matching the performance of a model trained with this sort of privileged information. 
- One take home from your results is that training $P$ is best done with lower dimensional abstract representations than the high dimensional raw input. It would be interesting to see some further experimentation on the relation between the dimensionality of the data $P$ is trained on, and its performance when inserted into the final model.  
summary_of_the_review: In all, I opt for arguing for acceptance. Since the contribution of this work is somewhat subtle, I am especially open to further discussions. The basis for my current position is this: 

- This works adds further weight behind a school of thought on combining deep learning and classical algorithms: instead of exactly running classical algorithms as subroutines in network architectures, we should instead insert network modules that behave similarly to classical algorithms, without exact execution.
- The key requirement in order to use this method, that there are problems where a cheap abstract simulator is available, is fairly reasonable. For example, any case in which we wish to insert a classical algorithm as a subroutine in a network architecture, we have available a simulator (i.e., the classical algorithm). 
- Given the previous two points, the takeaways from this work have to potential to be of use in various problem settings. 

I opt for a weak accept instead of higher since although I enjoyed reading this paper, I did feel that 1) this paper appears to be somewhat modest extension of existing ideas, and 2) I feel the paper is missing some ablations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for using domain knowledge to learn an interaction model given a groundtruth algorithm that operates over abstract inputs, which can then be used to learn an encoder and decoder for interfacing between this interaction model and naturalistic inputs. The method is evaluated, with favorable results, on both the bouncing balls domain and Atari suite.
main_review: I think this approach could potentially be useful, but I think the current set of experiments don't adequately demonstrate that it poses much of an advantage or will be scalable in a practical manner, for a number of reasons:

- The central assumption of the proposed approach is that we have access to a groundtruth algorithm that governs the interaction of entities in the target domain. This works well in the synthetic domains that are evaluated in this paper, but it's unclear to me whether this approach will be useful in real-world settings. One possibility is that perhaps a simulation engine could be made realistic enough to permit meaningful transfer to real-world data. Is that the sort of approach that the authors envision? If so, it would be helpful to include some kind of proof-of-principle demonstration for this strategy. 
- Given that one has access to such a groundtruth algorithm, wouldn't it be more straightforward to simply use supervised learning to learn representations of the naturalistic inputs of the form that the algorithm expects, and then directly use the algorithm, rather than a learned proxy for it? The paper states that this approach is infeasible because it would require a 'massive dataset of paired $(x,\bar{x})$' for the purposes of training the encoder (and a similar dataset for training the decoder). But why should this dataset need to be any more massive than the one that is necessary to train the encoder/decoder in the context of the learned interaction model? The paper also suggests that this approach may be impractical because of imperfections in the groundtruth algorithm (or the information available to it). Both of these points would be far more convincing if empirically evaluated (e.g. by demonstrating that it requires more training to learn an effective encoder directly via supervised learning vs. using the current approach), and this would help to justify the motivation for the proposed approach.
- The primary contributions emphasized in the conclusion are that the proposed approach protects against bottlenecks, and that it is more data-efficient, but as far as I can tell there is no systematic evaluation of either of these claims. The claim about protection against bottlenecks could be evaluated by systematically manipulating the size of the latent representations $z$ (smaller $z$ should impair performance). The claim about data efficiency could be evaluated by manipulating the size of the training set (the advantage of the proposed approach over the end-to-end model should be particularly pronounced with smaller training sets).
- Most of the comparisons in the paper are between the proposed approach and an end-to-end analog of that approach. This is an important comparison, but the authors should also clarify how these results compare to the current state-of-the-art for the metrics that are evaluated. One exception is the comparison with embeddings from Agent57, but, as the authors note, it's unsurprising that these embeddings don't work very well given that they were learned for the purpose of performing a different task, so that particularly baseline seems a bit contrived.

Other considerations:
- It's not very easy to resolve what's going on in Figure 3. It requires a very close inspection to even detect that the balls are moving, and it isn't easy to compare the predictions to the groundtruth. It might be easier to use an example with fewer balls, or to skip frames to make the movement more salient. If possible, a link to an animation might be even better.
- Can Figure 4 be made quantitative rather than qualitative, e.g. by representing the performance of RMR as a percentage of the baseline performance? It would be nice to see how much of an improvement is yielded for each pair of games.
- The qualitative clustering analyses are kind of cool, but what do we learn about the proposed approach from these analyses? These results seem to pertain primarily to the games themselves, in which case this analysis might be better suited to the supplementary material. 
summary_of_the_review: The proposed approach is favorably evaluated in synthetic domains, but it is unclear from the present set of experiments whether and how this approach will be scalable to more real-world problems. There are also some additional experiments that need to be performed to evaluate some of the paper's primary claims.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a learning paradigm referred to as "Reasoning-Modulated Representation" (RMR). The main argument is that it first learns a latent-space processor for abstract simplification of a problem, then use the processor for any tasks that can be modeled with algorithmic reasoning. The authors then verify RMR in two scenarios, bouncing balls problem (physics simulation) and Atari 2600 games.

For real-world problems, the proposed method alleviates requirements on alignment between abstract and natural inputs, making the learning pipeline more data-efficient, because the pipeline could rely solely on automatically generated, abstract training data. In addition, it is possible that the latent-space processor can be re-used to tasks that do not even directly align with the abstract environment.
main_review: **Strengths**:
\
The paper is well presented, and explores an important and exciting territory. The empirical assessment, though rudimentary, still demonstrates great potential of the proposed approach.

**Weaknesses**:
1. My major concern is that abstract inputs, $\bar{\bf{x}}$, is given, thus substantially simplifies the problem. While in most real-world problems, $\bar{\bf{x}}$ is unknown. (For example, the representation of sentences, speech, or images for object detection)
\
The key difficulty of representation learning is to learn internal representations for natural inputs (images here). However, the authors pre-define the abstract inputs based on their knowledge. In bouncing balls problem, they define $\bar{\bf{x}}$ to be coordinates of balls, and those are the key to predict future trajectories.
\
Given the pre-defined internal representations, it becomes much simpler to supervise a task. And $P$ is learned easily. It is not surprising that $P$ can be directly (re)-used in a natural setting, because the only thing left to do is to learn a mapping between abstract representations and natural representations, i.e., $\tilde{f}$ and $\tilde{g}$.
\
I think a more meaningful setting is that the form of abstract features ($\bar{\bf{x}}$ and $\bar{\bf{y}}$) is unknown. And the goal is to use some external knowledge to learn them instead of defining them for tasks.

2. I also found tasks being evaluated are still toy tasks. I’d suggest test on more realistic tasks, such as visual question answering.

3. There isn’t significant improvement compared to an end-end model (The difference might not be even distinguishable in the reconstruction rollout..); while RMR involves much more complicated modules. I’m not fully convinced to use RMR at least for bouncing balls problem.

4. Not weakness, but a question for style/typo: In Section 5 **natural pipeline**, is there a typo in the function? The right hand side of equation should be $\bf{y}$, not $\tilde{f}(\bf y)$?
summary_of_the_review: The paper tries to tackle a meaningful and intriguing problem. However, I find the pre-assumption of the whole learning paradigm is questionable. And I do not recommend acceptance of the paper in its current form.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper contributes an implementation of the neural algorithmic reasoning blueprint outlined in Velickovic & Blundell (2021). The setting that is considered is one where, for a given complex reasoning task that one is interested in solving, there is access to information about the underlying system that should be incorporated to simplify learning. For example, in the case of making predictions from videos about the future state of a physical system composed of bouncing balls bouncing balls, given access to each ball's radius, position, and velocity it is well known how their future movements follow the laws of physics. 

In order to incorporate this prior knowledge, Velickovic & Blundell (2021) proposed the idea of training a neural processor to capture this prior knowledge between abstract inputs (eg. the 3-dimensional state of each ball) under the assumption that sufficient data obeying the known input-output mapping in this space can be generated (eg. pairs of states at time t and time t+1). An encoder then maps abstract inputs to high dimensional space, the processor transforms it to another high-dimensional space, and a decoder maps it back onto the corresponding abstract output pair. In this way, the processor can distill knowledge about the transformation between abstract inputs (eg. governing the laws of physics in its weights) into its weights. By incorporating this processor in a network that is trained to solve the original task (eg. prediction in pixel spaces) it is expected that this extracted prior knowledge can be leveraged to improve learning and transfer to new tasks.

The implementation considered here (Reasoning-Modulated Representations), is evaluated on two environments: the bouncing balls environment, and various Atari games. On bouncing balls it is shown how RMR can transfer learned representations from trajectories to videos. However, the absence of a stronger baseline makes it difficult to interpret these results. On Atari, RMR is trained on transitions of RAM states to improve predicting latent space transitions from pixels. Compared to C-SWM (Kipf et al., 2019), it is shown how RMR leads to better representations in several games, and it is shown how processors may transfer across games. 

main_review: This paper is concerned with the important problem of incorporating prior knowledge about relevant knowledge for solving a task in neural networks. The paper is well written and fairly accessible and clear. The novelty of the approach is somewhat limited, since to a large extent the idea of pretraining a processor in this way was outlined previously in Velickovic & Blundell (2021). At the same time, the implementation of this idea is by no means trivial and overall the technical contribution is sufficient in my view.

The idea of training a neural processor in the way outlined in this paper is appealing, and the possibility of using it for pixel-level prediction (eg. for bouncing balls) as well as for representation learning (eg. atari) is appealing. At the same time, I don't find that the current experiments sufficiently validate its effectiveness at these tasks, and I would like to see some improvements in this regard. In particular:

* No stronger baseline is provided for the bouncing balls experiment, which leaves it unclear how well RMR performs at this task to alternative methods that also incorporate prior knowledge about the the underlying state of the environment (eg. it being composed of objects, and object interacting mostly pair-wise). From Figure 3 it can be seen that the predictions are quite different after only 5 steps from comparing the position of the dark purple ball. A stronger baseline that also incorporates assumptions about the underlying state of the world is desirable. RIMs seems like a good candidate for this or perhaps VIN.

* On bouncing balls there is a clear expectation for what the processor should learn (i.e. something like how interaction networks are designed for this task) and I would like to see a comparison to using a processor that can be understood in this way. In particular, a comparison to a pre-trained IN as a processor would reveal how much knowledge about the underlying task is distilled. A key difference between using IN and RMR is the lack of an intermediary high-dimensional space on which the processor operates (as opposed to abstract inputs). While it is intuitive that this can be desirable when both the processor and the natural encoder are relatively unstructured, it would be good to ablate this design choice.

* While the results on Atari in Table 1 are promising, it is a little concerning that these were obtained by first pre-training an Agent57 to generate sufficiently diverse RAM state data. Part of the motivation for this work is to alleviate the need to densely sample data in the natural space by incorporating prior knowledge, yet arguably this is exactly what was needed to train agent57 in the first place. As such I would like to see an attempt made at performing this experiment on transitions collected in a way that does not necessitate recording billions of pixel-level transitions. Alternatively, one could compare to a stronger baseline that leverages the data used to train agent57 for representation purposes as well.

Finally some minor comments

* When generating simulation data from abstract inputs as for bouncing balls, how is it ensured that a sufficient range of values is seen during training the processor? In the environments considered, the abstract and natural data always comes from the same simulator, but what if this is not the case? I.e. some real-world phenomena are observed for which no simulator is available, and abstract data can only be provided via a separate simulator (for which ranges must be chosen). It could be useful to comment on this.

* When training the processor for the contrastive learning task, how is it ensured that the transformation of interest is performed by the processor and not the encoder or decoder (which are discarded afterward)? 

* A good reference regarding the third bullet in the intro on the "challenging feat" of learning object representations is "On the Binding Problem in Artificial Neural Networks" by Greff et al. (2020), which could be helpful to include beyond the two specific methods cited there. 

summary_of_the_review: This paper contributes an implementation of an interesting approach to incorporating prior knowledge about a problem in neural networks that was outlined previously. While this constitutes a sufficient contribution, the experimental evaluation should be improved to validate the proposed implementation. I have suggested a number of concrete improvements that can be made, which would have me reconsider the current score.
review_title: null

Meta-review: The paper proposes Reasoning-Modulated Representations (RMR). That is, it incorporates how to incorporate (structure) prior knowledge (such as a law in physics) into a pre-trained reasoning modules, and investigates how doing so shapes the discovered representations in a number of self-supervised learning settings from pixels. The reviews and (short) discussion have presented salient arguments about the suitability of the paper for publication at this stage. One review argues that the "methodological contribution is minimal," another one is asking for "systematic evaluation" of the main claims made. Moreover, while we all agree that the direction is interesting, the RMR approach presented is not shown to "scale well" (yet), as pointed out by one review. This, however, is important since the general idea that prior knowledge shapes the representation learned is common wisdom in the literature. Indeed, one may now argue that the paper is much more about "how best to combine pixel-based deep learning and neural algorithmic reasoning algorithms" as one reviewer puts it. From this perspective the ATARI experiments are more interesting but here the benefit compared to C-SWM seems to be marginal and one should compare to other deep baseline conditions on the RAM; the significance is not looking at the difference in score and degree in freedoms but just the number of wins. Additionally, there should be other baselines that directly make use of more structured models (structure = prior knowledge, e.g., HMMs or some other way to have bit of a memory), other datasets (where no access to RAM exists) as well as a discussion of other approaches that combines (combinatorial) reasoning with pixel-based deep learning. That is, while pushing for a more high-level contributions is fine, this also requires some more illustrations and discussion of the broader context. Therefore, my overall recommendation is reject at this stage of the paper.
Decision: Reject