Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper adopts a dual system theory from cognitive science to check the consistency of the outputs from the system 1 neural generators using the system 2 symbolic reasoners with the aid of facts extracted from the output candidates and the world background knowledge constraints incorporated into the system. The proposed model has been applied to multiple tasks including story generation and question answering over the widely used bAbI task dataset in the domain of people and objects location tracking, story generation in the people and their family relationship domain over CLUTRR dataset, and the grounded instruction task. The results show the improvement in the coherence and accuracy of the neurally-based generation outputs.
main_review: Inspired by the dual systems in cognitive science this paper presents a novel and interesting neuro-symbolic approach for increasing the coherence and consistency of the neural generators where the neural generation candidates serve as system 1 proposals and the symbolic system 2 plays the role of a consistency checker; the neural generator generates the outcome repetitively until the logical consistency checker confirms the quality of its output. 

The paper is very well-written. The idea and the application are very interesting and the general framework introduced in this paper can further get extended in future works in other domains of interest with a lightweight world model/background knowledge/ ontology with different neural generation and semantic parsing approaches for increasing the coherence, accuracy, and consistency in the output of AI systems. The evaluations are acceptable and show the potential of the proposed framework for non-complex generations with a relatively straightforward world model. Hence,  I'd like to see the revised manuscript being accepted by the conference.

Mistakes/Suggestions: 
- In table 1, for the result of story generation over CLUTRR, the result for the dual system has been reported for maximum 20 attempts; the question arises here on how well the dual system will work if you allow lower maximum failure attempts. Please report the system accuracy for different maximum attempts.
- Please provide more discussion on whether/if the proposed model can get applied to other bAbI tasks, how, and why?
- Please provide more information on the number of annotators per trial and their agreement rate.

 
limitations_and_societal_impact: A detailed discussion on the limitations of the proposed approach is provided.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Contribution: The paper builds upon the proposals of  (Goyal & Bengio, 2020; Garcez & Lamb, 2020) to give the neural networks an ability to mediate information between the system 1 and system 2 processing. 

The paper proposes to improve existing sequence models by adding system 2 inspired logical reasoning abilities. 




main_review: Method: The proposed approach proposes to improve the current neural sequence models (system 1) by adding a logical system 2. The basic idea is "system 1" generator suggests various proposals, and a logical system accepts or rejects the proposals generated by system 1. 
The proposed method is  a lightweight, training-free means of improving existing sequence models. 

The pipeline is as follows:
(a) Generate proposals from a neural sequence model.
(b) Fact extraction from the generated proposals.
(c) World model to ensure that the fact extraction is aligned with the set of logical relations and constraints (part of system 2).

Experiments: The paper tests the proposed method on Coherent Language Generation - CLUTRR and grounded language learning tasks. 
The method is tested by its abilities to generate coherent  stories by human participants. 

Pros:
-  The paper is very well written.
-  The relevant work is very well described (especially the  generate-and-rerank literature). 


All the limitations are adequately addressed in the paper. 

*In its current form, our approach is most useful in domains where naturalistic, learned generation is necessary and where a small number of mission-critical logical constraints can be explicitly.*

Suggestions: One suggestion would be, in it's current form the title seems overly generic as compared to what is actually done in the paper as something already hinted in the paper in the limitations section. 

Other work which may be relevant for learning system 2 is the work of Neural Production systems [1] inspired from the work of production systems which consists a set of variables and logical rules, but something entirely end to end differentiable. 

Relevant Literature Work:
- [1] Neural Production Systems, https://arxiv.org/abs/2103.01937.

limitations_and_societal_impact: Limitations are very well addressed. The reviewer appreciates it.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper argues for the importance of logical coherence and consistency in language models such as GPT3, and proposes a neurosymbolic approach whereby logical rules are deployed to validate language sequence generations. This is highly important in the current landscape of research. 

The authors propose to derive logical statements such as "drop(Daniel, apple)" directly from the language sequences generated by the model, and use GPT3 to illustrate. The study of such automatic generation of logical statements is also very relevant at present. 

The logical statements are then applied over time to constrain future GPT3 language derivations, using the symbolic knowledge to validate the derivations with the objective of achieving higher coherence and consistency over time.
main_review: As acknowledged in the paper, a limitation is that the symbols (available commands) are pre-defined when in fact they could have been derived from the text (e.g. the verbs in the sentences).

The terms soundness, consistency and coherence are used in a somewhat interchangeable form, e.g. in: "...while remaining logically sound. In contrast, GPT-3 alone was not able to maintain logical coherence. In a set of 50 generated stories, all stories required at least one sentence to be resampled to maintain coherence, and over half of the generated sentences (53.1%) were rejected by our System 2 model to maintain logical consistency". These terms have precise meaning in logic and ideally, formal definitions of their use in the context of the paper should have been provided.

The human judgement experiment carried out in the paper is sound and its methodology is described in detail. The results are impressive. By evaluating results on two data sets, the authors also contribute potential benchmarks for neurosymbolic AI.   

Quite a good number of references is provided and I am aware that in this multidisciplinary area it is difficult to cover all the relevant work, from the work in language to the theory of neurosymbolic AI and related systems. However, two papers come to mind which seem highly related, since the work is focused highly on adding constraints to neural models [1] and on reasoning in the context of the BaBi data set [2]. Such related work might offer end-to-end alternatives worth comparing with the approach proposed here.

[1] Logic Tensor Networks: Deep Learning and Logical Reasoning from Data and Knowledge. Luciano Serafini and Artur d'Avila Garcez. https://arxiv.org/abs/1606.04422

[2] Learning to Reason with Third-Order Tensor Products. Imanol Schlag and Jürgen Schmidhuber. https://arxiv.org/abs/1811.12143  
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a neural-symbolic system where the symbolic module serves to screen, at testing time, the suggestions of the neural module, allowing only those that obey the constraints of a symbolic theory to be presented as the final predictions of the system. The paper shows that this post-filtering improves the performance against a system comprising only the neural module.

main_review: I found the System 1 / System 2 narrative to be a bit of a stretch. First, the dual process theory is much richer than what this paper would seem to suggest. Second, the use of symbolic constraints on top of a learned model (neural or not) has been explored a lot in the literature (e.g., "Learning and Inference over Constrained Output" by Dan Roth and others, IJCAI 2005), and in fact more advanced integrations have been proposed where the symbolic constraints act not only during the prediction time, but also during the training time to train the learned model to respect the symbolic constraints. This last direction seems to be mentioned at the end of Section 6 as a future direction, ignoring a lot of relevant work (see, e.g., recent work by Efthymia Tsamoura and Loizos Michael on compositional neural-symbolic integration).

Modulo the lack of originality of the idea, the paper also seems to lack sufficient details to appreciate the empirical setting and results. For the most part, the paper offers a high-level summary (revisiting often the System 1 / System 2 narrative), without explaining in detail the key points of the methodology. For example, it is unclear to me how the system was eventually evaluated, and whether the evaluation is meaningful. First: it is obvious, without needing an empirical study, that post-filtering any generating process with sound constraints can only improve performance. This already makes the value of the empirical study mood. Second, the choice of how to compare the no-filtering and the post-filtering approaches seems to raise questions. The authors state: "We selected generations from the neural only baseline that were rejected by the System 2 model in order to maximize the differences between the models’ generations; thus, human judgments pertain to generations that the models disagreed on." Doesn't this inflate the performance in favor of the "paper's favorable" outcome? If, for the sake of discussion, the two approaches only disagreed on 1% of the cases, would focusing on those cases be reasonable? If, on the other hand, the paper is attempting to show results conditioned on disagreement, then isn't it trivial that evaluating two systems in disagreement, with one of them encoding sound constraints, will always end up presenting favorable results for the latter system?

Lastly, the claim (around line 375) that "While System 2 world knowledge can be easily modified by explicit modifications, such knowledge cannot be automatically learned or updated." seems to completely ignore a ton of past and recent work on symbolic learning and machine coaching (e.g, work by Leslie Valiant, Loizos Michael, Brendan Juba, and others on learning and reasoning). 

--- after reading the author response ---

Thank you for your response. I think that the clarifications that were provided in your response could help improve the readability and the impact of the paper. I have increased my score.

limitations_and_societal_impact: Seems fine.
review_title: null

Meta-review: After the author response and the discussion all reviewers agree that the paper should be published at NeurIPS. The authors take inspiration from the dual process theory to the outputs of a neural generation procedure is consistent with a predefined set of world knowledge. The paper is well written and the ideas are clearly explained. The authors test their technique on a number of tasks and evaluate their method using human judgements and show good results.
Decision: Accept (Poster)