Venue: LOG 2022 Conference

Reviews:

final_score: rating: 8: Clear accept. A strong paper with novel ideas, flawless evaluation, resources, and reproducibility, and no ethical concerns.
review_content: Summary:

The paper proposes pretraining on abstract algorithms before using it on the natural inputs. They advocate for the encode-process-decode architecture to enable this transfer, where the encoder and decoder deal with the high dimensionality of the natural inputs and the learned abstract process executes the underlying algorithm. The authors validate their method in two primary ways, once on physics simulation of bouncing balls, where the natural inputs are images of what is happening. Secondly, on Atari games, where they pre-train on RAM traces.
Finally, the transfer between Atari games is evaluated.

Strengths:
- Good trade-off between a realistic setting and a feasible one, ie. the choice to not require a paired (\bar{x},x) dataset is good.
- The proposed method is reasonably novel in that I am not aware of this kind of pretraining having been in the context of encode-process-decode architectures for natural inputs. While other works have used self-supervision to learn the underlying dynamics before applying an RL agent, this is quite different.
- The empirical validation is thorough and convincing.
- Writing is excellent and very clear.

Weaknessess:
- Due to the novel setting the baselines of comparisons are quite sparse, however, I think given the objective of the paper this is acceptable, but maybe not ideal.
- The Related work section could be improved, in particular the paper [1] is missing, which also studied the transfer in an algorithmic reasoning setting, but instead did it between different abstract algorithms, rather than going from abstract to natural. However, they reach a different conclusion in the terms of positive transfer and this should be discussed.
- No study is done of the generalisability of the learned algorithm, i.e. could we learn how to predict many more bouncing balls than trainded on. This is a key property for reasoning tasks.

[1] https://proceedings.neurips.cc/paper/2021/file/a2802cade04644083dcde1c8c483ed9a-Paper.pdf

Recommendation:

I recommend to accept the paper as it makes a strong argument for how to improve reasoning on natural input tasks.

Ideas for improvement:
- Acquire a larger variety of baselines.
- Study the systematic generalisability`
review_title: null

final_score: rating: 8: Clear accept. A strong paper with novel ideas, flawless evaluation, resources, and reproducibility, and no ethical concerns.
review_content: The authors propose a novel representation termed 'Reasoning-Modulated Representations' or RMR that is better suited for leveraging the underlying physics (or some other known mechanism) to solve a high dimensional reasoning problem such as trajectory estimation and gameplay. This is in contrast to the traditional approaches that rely on tabula rasa neural networks that need to relearn every time from scratch and require a lot of paired data to work well. The authors demonstrate the efficacy of the proposed method in estimating the trajectories of bouncing balls and ran extensive experiments on different games considered standard benchmarks for this problem domain, showing impressive results. 

I find that the authors present a very innovative idea and something which will surely find application in several domains in computer vision and machine learning and is of particular interest to LOG community. The paper is written very clearly and is suitable for a broader readership. I personally believed that I learned a lot from reading this paper -- surely a great investment of my afternoon! For that, I would like to thank the authors for putting in the work to write this. The authors clearly state the hypothesis and assumptions throughout the paper and provide experiments that clearly support their claims which simply goes on to suggest the effectiveness of using RMRs. 

While I believe the paper is very well written and the authors have done a great job with the experiments, I have a few clarifications to ask. Also, to emphasize that the paper in its current form is already pretty strong and I am merely asking the following to better understand the connections it may have to a wider range of problems. 
(1) It seems that RMR is supposed to be a general approach, however, the authors have presented experiments (which I believe) belonging to a particular class of reasoning tasks namely trajectory prediction (or predicting a sequence of steps in games). I wonder if this technique can also be applied to situations where the Processor isn't really doing any trajectory estimation, but instead performs a sequence of logical deduction steps. These are commonly found in natural language reasoning scenarios. Like if a is better than b and b is better than c, so a is better than c. Here, the quantities a,b, and c can be high dimensional but can be processed in the low dimensional abstract space. Alternatively, say you train the abstract pipeline on solving a system of linear equations, then can the natural pipeline be used to solve similar problems specified in natural language? Like, two apples and one banana cost XX, and 5 apples and 2 banana cost YY, so how much does 1 apple cost? I am curious to know what the authors think about such problems. 
(2) I am also curious to know how the efficacy of RMR changes depending on the complexity of the underlying abstract simulator. For instance, in computer graphics, physically based renderers are very common and have clearly defined 'abstract' quantities in terms of scene lighting, materials, etc. But having a neural processor to simulate all of that is quite challenging owing to expensive ray tracing requirements. So, in such situations when the algorithmic bottleneck is too difficult to be properly modelled, how well is the transfer to the natural pipeline? Do authors have any intuition or thoughts on this?
(3) One recurring assumption I believe the authors have made is the knowledge of the number of objects (such as # balls being =10). I wonder if this is a crucial assumption. What would have happened if the abstract pipeline was trained on data collected for 10 balls but was transferred to the natural pipeline where the data consists of hundreds of balls? As per the claim of the authors, the Processor network should be able to scale to such situations, but I doubt that is the case. 
(4) I am also curious to know if there are particular design choices involved in designing networks f and g? I get that we want them to be 'simple', but what if the task at hand requires them to be more sophisticated? This can happen if the modality of x and z are different. For instance, maybe we want z to have a graph-like structure but x comes as an unordered list. I can think of several situations where it maybe hard to get away from having just a linear network layer for f and g, and therefore I am curious to know what are the repercussions of having more complex architectures there and how severely would the training of the Processor Network suffer (even though it seems to be getting z in an 'ideal' representation!). Would the Processor network still be in a position to learn the correct underlying algorithm?   

review_title: null

final_score: rating: 8: Clear accept. A strong paper with novel ideas, flawless evaluation, resources, and reproducibility, and no ethical concerns.
review_content: This paper proposes the Reasoning-Modulated Representations learning paradigm, containing two aspects: 1. it learns a latent-space processor for abstract simplification; 2. then use the processor for any tasks which can be modeled with algorithmic reasoning. This method is evaluated in two environments: the bouncing balls environment, and various Atari games. 

+ This paper is well presented and the way of training a neural processor is quite interesting and reasonable. Also, incorporating prior knowledge for neural network tasks is important and worthy of discussion. 

- One concern is that all the evaluation tasks here are toy tasks, it will be better to test the method on some real-world tasks since the real tasks will contain biased data or data limitation problems. Another one is for the experimental results. I would like to see the comparison with some stronger SOTA baseline methods, for example, the proposed method only improves over weak baselines for bouncing balls, and there is a limited improvement to the end2end model.

Q: Is there an open-source code for reproduction or will the source code be released?

review_title: null

final_score: rating: 6: Weak accept. A good paper where merits, e.g., significance, slightly weigh over weaknesses, e.g., poor presentation.
review_content: The paper presents an implementation (or demonstration) of a neural algorithmic reasoning (NAR) approach, named Reasoning Modulated Representations (RMR), that can be used in learning a latent space processor module to build reasoning motivated representations. The approach is evaluated on the bouncing balls problem and Atari with positive results in terms of (i) generalization of learned representations on abstract inputs to the natural pipeline. RMR can thus potentially be transferred and eventually, reused across different tasks, as shown with demonstrations in the paper, which is an interesting achievement on the line of knowledge transfer.


Strong points:
1. The paper is a demonstration of an NAR approach and leverages conceptual advancements from prior works.
2. Structure of the paper makes it well readable. Different components of RMR are described as background on the basis of which it shows how such learning paradigm can be used to construct reasoning inspired architectures on abstract datasets which generalize on natural settings.
3. Since a high-dimensional pre-trained processor can be plugged into a natural pipeline, mapped data pairs of natural and abstract inputs may not be necessary for challenging situations where it is infeasible to obtain the pairs. 
4. As shown by the ablation on the dimensions the processor network, there is a consistent advantage of the high-dimensional processor alleviating bottleneck that is otherwise present when directly running a supposed algorithm on raw data.

Comments and recommendation:  
- The paper does not seem to introduce a new method or architecture which could limit its novelty in those terms, looking from one perspective.    
- It dwells largely on the existing works that combine deep learning with algorithmic computation to propose better design of algorithm inspired neural architectures that can replace or augment traditional learning setups, while improving efficiency and re-usability.    
- However, the paper stands as a successful demonstration of reasoning inspired neural learning approach which, to a large extent, delivers on the promise or the reasons of combining neural networks with high dimensional reasoning.  
- I would lean for the acceptance of this paper for the said reasons, even though the ideas presented are not novel. Additionally, a discussion about the transfer of processor networks as questioned as follows would be informative.

Question:   
Is there a probe or current guideline into how processor networks could be transferrerd across tasks? How does the nature of abstract setting affect a pre-trained processor for other tasks? For an instance, if there is a task to refer to an image and answer a question about one of its object locations, what type of abstract setting would be beneficial or useful for such tasks?
review_title: null

final_score: rating: 8: Clear accept. A strong paper with novel ideas, flawless evaluation, resources, and reproducibility, and no ethical concerns.
review_content: **Contributions**

This paper presents Reasoning-Modulated Representations (RMR), an algorithm designed to predict the behaviour of a given system based on the assumption that 
1. The system's observable behaviour is governed by latent (abstract) dynamics that capture the majority of the system's complexity.
2. There is a scarcity of _natural_ training examples in the observable space and there is an abundance of examples in the abstract space.

The main idea of the method is to learn a processor (P) tasked to predict the abstract dynamics. Then the authors train an encoder/decoder pair between the observable space and the abstract space, forcing the prediction to go through the frozen processor. 
This encourages the model to use the abstract knowledge of the system captured by P. 

Through experiments, the authors show that the proposed system achieves better performance than a baseline without the "abstract pre-training" and that the knowledge encoded in the processor seems to be transferable between comparable systems (eg, a processor learned on one Atari game can be used for another game too).

**Strengths**

1. The paper is exceptionally well-written. The notation is clear and consistent, the figures do a good job of explaining the main contributions, and the method and experiments are described clearly. 
2. The idea discussed in the paper is very interesting and could have implications for designing interpretable and universal world models.
3. The experimental results are strong and statistically significant. 


**Weaknesses**

1. While I understand why the idea studied in this paper is interesting, I would argue that the main assumption (few natural observations, many abstract observations) is almost never met by real-world problems. In fact, one of the main drivers for the development of "deep" representation learning models is exactly the hope that the models will uncover these hidden abstract mechanisms on their own, potentially leading to an understanding of the modelled phenomenon (e.g., reference [21]). 
   This is not necessarily a limitation of the paper, but something that is worth discussing. My main takeaway from this work is that the processor P transfers between seemingly different tasks, which could have concrete implications for designing more interpretable and transferable models.  
   Can the authors comment more on the practical implications of the paper?
2. As correctly pointed out by the authors, algorithm A cannot be assumed to represent a perfect transition of the system. I would also add that the processor P is unlikely to learn a perfect approximation of A. 
   If algorithm A could be implemented as a differentiable program (as in the case of the bouncing balls), wouldn't it be better to plug it directly in place of P? This would limit the abstract encoders/decoders to learning maps to the true abstract space, which might be better for interpretability. 
   Again, this might be something worth discussing. Also, I should point out that I'm not too confident about this comment, so please let me know if I misinterpreted something. 


**Recommendation**

I will gladly recommend the acceptance of this paper, which studies an interesting idea in a very thorough way.
The weaknesses I pointed out above are mostly suggestions for an interesting discussion, rather than concrete limitations (I would still like to hear the authors' reply on those two points).

**Additional feedback**

- I don't understand the argument that the encoders/decoders should be simple models while the processor should be more complex. In fact, I would argue for the opposite (e.g., perception trunks are likely to require more parameters and specific inductive biases). 
- I'm surprised to not see any mentions of: 
   Ha, David, and Jürgen Schmidhuber. "World models." _arXiv preprint arXiv:1803.10122_ (2018). 
   This is clearly relevant work that is based on a similar narrative of predicting latent dynamics while relying on separate encoder/decoder networks for the perception.
- The use of "bottleneck" on line 151 is confusing as it seems to contradict the statement on line 139 that the processor is _not_ a bottleneck. The authors might want to rephrase or clarify that passage.
- Lines 198-199: "The pixel-based encoder [...] is concatenation of per-input-image Slot Attention model". This sentence should probably be reworded to make it more clear.
- Typo in the caption of Fig. 3: "The states below [...] is the ground truth"

review_title: null

final_score: rating: 6: Weak accept. A good paper where merits, e.g., significance, slightly weigh over weaknesses, e.g., poor presentation.
review_content: In the real world, bouncing balls’ movements in specific spaces must obey the laws of physics, which motivates this work. This work injects knowledge from a massive abstract dataset of (x ̄, y ̄) pairs, directly through the pre-trained parameters of P. In addition, they build a neural network pipeline to reduce the dependence on the existence of massive abstract datasets.
However, the disadvantage of this paper is obvious:
1. Lack of novelty: injecting knowledge into neural network is not a relatively novel idea to enhance the performance of the NN model. In that way, some clear guidance on how people can utilize the extra information is important.
2. Lack of related work, the reviewer believes that some specific knowledge distillation methods can be related to your work.
3. The experiments need to be improved: an ablation study is needed to show the extra world’s information is more useful than the same model without extra information. 
review_title: null

Decision: Accept (Poster)