Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an experiment for evaluating whether neural models can learn to represent logical reasoning operators purely from observations. The experiments are taken out of literature in developmental and comparative psychology, in which a neural network is trained to represent negation and disjunction operators. The focus is on learning implicit representation, which are domain-specific and may not necessarily generalize to other domains but nonetheless measure the ability of neural models to implicitly perform logical reasoning in some limited fashion.

The experiments reveal that neural networks are indeed not able to learn to represent logical operators without being trained directly on test data. However, the transfer experiments show that neural networks do learn some useful representations from one domain with similar logical task but different features, that enable them to quickly learn to perform well on a test task.
strength_and_weaknesses: Strength

Analyzing the limitations of neural networks is an important contribution to the field, which enables researchers to further enhance the capabilities of neural networks. 
The paper proposes the first set of experiments to characterize the failures of neural networks in tasks that require reasoning. It's well known in the literature that neural network in general lack reasoning capabilities but it's rarely explicitly demonstrated in a set of experiments. This work pushes that boundary.
The proposed work is grounded in theory that aims to understand how logical reasoning "emerges" from human and non-human animals alike.

Weaknesses

Fine-tuning experiments, although in a logical reasoning context, are not really revealing anything new that we don't already know about neural networks.
The experiments are only limited to one type of logical reasoning task and neural network architecture.
It's not clear what it means for a neural network to succeed in solving the Two-Cup task. The ADEPT baseline seems to predict the correct cup 100% of the time but the neural network is considered successful by predicting the correct cup 81% of the time.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and is of good quality. It adds to the list of tasks for evaluating the ability of neural networks on reasoning tasks. It may be difficult to reproduce the experiments if the code is not released as designing the experiments is heavily involved. Some example animations showing the data generation process would certainly help.

There are some minor typos:

"taskk" in 3.1.2 and "to to" in the last paragraph of Section 6.
The caption of Figure 3 refers to "blue ball" but I do not see any blue ball in the figure. 
summary_of_the_review: More research work highlighting the failures of neural networks and drawing insights on how to improve them, is needed. This work adds to the list of benchmarks of highlighting the limitations of neural networks on logical reasoning tasks. While more experimentation on other types of reasoning tasks and for a wide range of neural architectures is still needed, this work will open up more opportunities for other works to follow in the same vein.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents an in-depth study on the ability of neural networks (NN) to learn implicit logic of negation and disjunction. Specifically implicit negation and disjunction here refers to the ability to tell that if a ball fall in one of two recipients, the NN doesn't know which one, but then one is shown empty (the ball is not there) then the agent should know that the ball is in the other recipient. 

Authors do not train the agent directly on the task above, which is used as test task but train in related tasks with the same objects and background. Results in control experiments show that the NN learns to track the ball but it is not able to solve the task.
strength_and_weaknesses: The paper follows a recurrent and very important topic on the ability of NNs to do compositional generalisation of logic reasoning in a novel set up. One of the main strengths of the paper is the careful detail of the experiments and control tests -which rule out plausible concerns about if the model is really failing to solve the task due to the inability of generalisation of logic reasoning or something else-.

My main concern about the paper is that, while authors cover a good amount of related literature, they missed several key papers about compositional generalisation in reinforcement learning. I strongly encourage authors to check those earlier studies because they would explain why the NNs fails to generalise in this work, and what should have been done (or at least tried) for the model to past this test:

* Hill et all, 2020 did an experiment on the ability of DRL agents to generalise negation to unseen instructions. In that work the agent is trained with positive instructions such as "find a plane" or "find a ball" and the agent should find those object in a room and stare at them for a few second to succeed. For some of those objects the agent was trained with negation with instructions such as "find something that is not a plane", and then evaluated to do  find something that is not X, where x was a training object that had been used for positive instructions only. Hill et al. found that for the agents to succeed, they required to had been trained with at least 100 negated instructions to pass this test.

* Also in that line, León et all. did a similar experiment in 2D scenarios to test the ability of DRL agent to achieve compositional generalisation with negation and disjunction and found that with object-based encoders - that this paper under review is using already-  and early stopping, 6 objects are enough to generalise negation and disjunction.

I believe that those two works study a form of implicit logic in a similar fashion that this paper does, but even in the latest they required 6 examples of that form of reasoning before achieving successful results. Thus, when this work shows in Section 4.1 that the agent is not doing that form of generalisation after just one example, it actually aligns with those existing works. Also the result from section 4.2 showing that the agent that has learnt to solve this kind of generalisation once is able to learn to solve the experiment faster also reinforces the hypothesis from those previous works that I cited.

Thus, I believe that it would be key for this paper to be complete to do one further test where they train the NNs with 6 different variants of implicit reasoning and then evaluate the NNs with the two cups test again. Since the setup of this work and context is so different (this is not an RL work for instance) it would be a great contribution to see if the hypothesis from those previous works are confirmed here as well.

I believe then that authors should include those two previous works in their RW section and carry an additional experiment to see if that helps the NNs to pass the test.

Hill, Felix, et al. "Environmental drivers of systematicity and generalization in a situated agent." International Conference on Learning Representations. 2020.

León, Borja G., Murray Shanahan, and Francesco Belardinelli. "Agent, do you see it now? systematic generalisation in deep reinforcement learning." ICLR Workshop on Agent Learning in Open-Endedness. 2022.
clarity,_quality,_novelty_and_reproducibility: The paper is very well written and easy to follow. The only suggestion I would do in this regard is to include further details and a figure about OPNet in the appendix for the paper to be self-contained.

The paper is sound and detailed in their experiments. The methodology to measure implicit negation and disjunction is also novel, which would help to increase the understanding of the research community on the ability of NNs to generalise implicit logic operators if it weren't because it only seems to confirm the previous works.

And I want to remark the "seems" because that is preventing this work from really doing a remarkable contribution. The results suggest that the missing thing could be a higher variety of implicit logic examples as in those previous works but as it is right now we can't know for certain. I know that this additional experiment is not trivial, and probably -and sadly- the rebuttal period won't be enough to carry it out. But without that, the conclusion of this work are only suggestions when built on top of that previous literature.

Once that experiment is included, and we are able to see if that is really a key ingredient in this setting as well, I believe this will be a significant contribution and I strongly encourage the authors to continue this line of work.
summary_of_the_review: Careful and detail experiment in a novel setting for a key ingredient of intelligence. However, authors missed key pieces of literature exploring the very same questions that are the main focus of this work. This leaves this paper incomplete since authors haven't checked if a key ingredient in those earlier studies is what they were missing here.


----------------UPDATE---------------------

After discussing with the authors, I agree with them that from the point of view of demonstrating that physical reasoning has arisen, but it does not mean that (implicit) logical reasoning has arisen too, this work is novel. 

I still believe that the last experiments suggest that if the agent had been trained in at least 6 different contexts implicit logic reasoning might have arisen. Since Hill et al., demonstrated that agents struggled to learn logic without enough examples  and León et al.  pointed that with object-based architectures -as the one in this paper- 6 examples is enough for compositional logic reasoning to arise. I think this should be pointed in the future directions as a possible solution to tackle why the agents are failing in the experiments of this paper.

I believe that the points above need still to be better clarified/incorporated in the discussion conclusion section, but given author's response so far, I am confident that they will and updating my score accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary: The paper mainly explores whether the neural network can learn the implicit representations of negation and disjunction. The results of experiment shows that the neural network lack the ability to generalize to task that requires implicit logic. 
Contributions: 
1)The authors introduce the notion of implicit logic into the repertoire of neural network evaluation. 
2)The authors use a test of logical inference in humans to evaluate the neural network. 
3)The experiments offer some suggestive evidence about the models’ ability to transfer representations between logically equivalent tasks.

strength_and_weaknesses: Strengths: 
A number of training modes were designed and detailed experiments were conducted to comprehensively evaluate the ability of neural network to learn implicit representations of negation and disjunction from data. It has a strong reference significance for works in logical reasoning in neural networks.

Weaknesses: 
There are too few experimental charts, thus the experimental process is not clear and intuitive enough. Another obvious problem with this paper is lack of sufficient explanation of the results. You need to explain your simulation results in detail and why you got such results. In page 9, DISCUSSION, “Thus, we are inclined to conclude that the neural model fails because it does not learn to represent uncertainty over multiple possibilities and/or lacks mechanisms for updating beliefs about possibilities in response to observations. ”, this explanation is ambiguous and not convincing.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow and the idea of the work is interesting.
summary_of_the_review: The research work of this paper mainly focus on neural network’s ability of learning implicit representations. Overall this paper is a comprehensive and complete work.
If the above problems are well-addressed, this reviewer believes that the essential contribution of this paper are important for research of neural models’ ability to learn implicit representations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of whether a neural network model can learn an implicit representation of negation and disjunction in the domain of intuitive physical understanding that predicts how objects might move in space based on visual input. Three experiments are carefully designed to draw two conclusions: on one hand, neural models cannot generalize to a diagnostic task requiring implicit negation and disjunction without observing examples from that diagnostic task; on the other hand, when training on the diagnostic task, pretraining on a similar logical task can enable the models to learn the target task faster than other initialization methods.
strength_and_weaknesses: Strengths:

(1) A new study on the logical reasoning ability of neural models in the domain of intuitive physical understanding.

(2) Carefully designed experiments conducted to prove or refute some interesting hypotheses.

Weaknesses:
The study is restricted on a relatively small domain where the results are entangled with computer vision processing. The findings are hard to adapt to general domains about the reasoning ability for classical logics. In particular, the findings do not tell whether neural models have logical reasoning ability when computer vision processing is not needed.

clarity,_quality,_novelty_and_reproducibility: The quality and clarity of the paper are good. The descriptions about the test hypotheses and experiments are generally clear and easy to follow. The originality is also good by considering that by now (as far as I know) there is no study on the logical reasoning ability of neural models in the domain of intuitive physical understanding, although similar studies exist in other domains. There is no code or data provided in the supplemental material, thus I cannot reproduce the reported experimental results by now.
summary_of_the_review: The paper studies an interesting and important problem on whether a neural model can do perform implicit logical reasoning in the domain of intuitive physical understanding. Some findings are made through carefully designed experiments. However, the findings do not tell whether neural models have logical reasoning ability when computer vision processing is not needed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tries to study if deep neural networks (with a built-in representation of object physics) can learn logical relations such as negation and disjunction. Inspired by cognitive science, the authors propose a task called Two-Cup which requires the agent to have the ability of reason by exclusion. This work has designed extensive experiments and ablation studies, the results show that the tested neural network failed to learn the target concepts.
strength_and_weaknesses: ### Strengths:
- The authors has designed many experiments and ablation datasets, which have covered different aspects in cognition for learning logic reasoning.
- The paper is generally well writen, all the details are covered either in the main text or in the appendix. The authors also provide enough discussion content to explain the observations.
- The paper targets answering an important problem in AI.

### Weaknesses:
- Although the motivation of this paper is intriguing, the methodology adopted by this paper might be weak, because it is difficult to evaluate neural networks' logical reasoning ability by just an empirical study on one type of task. Results on one or two tasks could not prove that they can represent or learn logical calculus theoretically.
- The comparing approach has a built-in Bayesian modelling module, which is too strong in my opinion. The conclusion could be more convincing if the authors can choose a learning system enhanced with explicit symbolic representation, e.g., Neuro-Symbolic Systems.
- It would be more interesting if the authors could provide a set of experiments with preverbal children or animals using the same set of training data. I think the number of training examples could be drastically reduced compared to deep learning models.
- I appreciate that this paper provides many datasets and experiments to make sure the empirical study covers every possible aspect of this problem, the naming of datasets and organisation of experiments are a bit messy. Maybe the reading experience could be improved by adding a graph of the datasets/tasks illustrating their relations, instead of hiding them in the main text.
clarity,_quality,_novelty_and_reproducibility: The paper proposes a novel machine learning task, the overall clarity of this paper is above average. The quality of this paper is good, and the reproducibility could be improved if the dataset and codes can be published.
summary_of_the_review: This paper trys to answer an important problem in AI research, the authors have provided an interesting task and extensive emprical study to answer this question. The experimental methodology could be improved and the result is not strong enough.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors investigate whether neural networks can learn to implicitly perform logical reasoning out of purely perceptual information. For example, by training a network to represent negation and disjunction operators out of image data alone. The contribution is essentially of empirical nature and grounded in the literature of developmental and comparative psychology.  Experiments provide evidence that learning accurate logical operator from data alone is hard, while the network might learn some internal representation that can help transfer from different tasks of the same kind.

The reviewers agreed that addressing what kind of reasoning neural nets can learn is interesting and important. Some concerns raised during the reviewing phase include: limitation of the research question to a single task and set of operators; missing literature comparison and missing experimental details. Authors were responsive and managed to address all the major concerns, improving the quality of the paper and raising one reviewer's score towards full acceptance. The only issue remaining is the scope of the analysis being limited to a single kind of tasks, which limits the generalization of the hardness claims.

I believe that this still provides enough evidence to falsify the generally spread claim that neural nets can learn everything, for all possible tasks, from data alone. Therefore I recommend the paper to be accepted.

justification_for_why_not_higher_score: The study is limited in scope and focuses on a single modality/task as highlighted by reviewers.
Decision: Accept: poster