Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning
Authors: Maxwell Nye
Michael Henry Tessler
Joshua B. Tenenbaum
Brenden M. Lake
Venue: NeurIPS 2021 Poster (2021)
Keywords: System 2
cognitive science
dual-system
coherence
consistency
dual-process

Abstract: Human reasoning can be understood as an interplay between two systems: the intuitive and associative ("System 1") and the deliberative and logical ("System 2"). Neural sequence models---which have been increasingly successful at performing complex, structured tasks---exhibit the advantages and failure modes of System 1: they are fast and learn patterns from data, but are often inconsistent and incoherent. In this work, we seek a lightweight, training-free means of improving existing System 1-like sequence models by adding System 2-inspired logical reasoning. We explore several variations on this theme in which candidate generations from a neural sequence model are examined for logical consistency by a symbolic reasoning module, which can either accept or reject the generations. Our approach uses neural inference to mediate between the neural System 1 and the logical System 2. Results in robust story generation and grounded instruction-following show that this approach can increase the coherence and accuracy of neurally-based generations.

Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning
Maxwell Nye" MIT
Michael Henry Tessler
Joshua B. Tenenbaum MIT
Brenden M. Lake NYU Facebook Al Research
MIT DeepMind
Abstract
Human reasoning can be understood as an interplay between systems: the intuitive and associative ("System 1'") and the deliberative and logical ("System 2"). Neural sequence models-which have been increasingly successful performing complex, structured exhibt the advantages and failure modes of System I: they are fast and learn patterns from data, but are often inconsistent and incoherent. In this work, we seek lightweight, training-fres free means of improving existing System 1-like sequence models by adding System 2-inspired logical reasoning. We explore several variations this theme in which candidate generations from a neural sequence model are examined for logical consistency by a symbolic reasoning module, which can either accept reject the generations. Our approach uses neural inference to mediate between the neural System I and the logical System 2. Results in robust story generation and grounded instruction -etruuttoo-followinng show that this approach can increase the coherence and accuracy of neurally-base generations.
Introduction
Despite recent success, neural sequence models often fail to produce consistent and coherent genera- tions. When generating stories, language models may forget the attributes of specific characters (such as personality and background information) (Welleck et al. 2018). ignore previously established relationships between characters (such family relationships) (Sinha et al., 2019), otherwise contradict prior statements (Brown et al. 2020). Similarly, neural models can make statements that contradict basic world knowledge or the logical entailment structure of known facts.
Lake & Murphy (2020) illustrated several of these issues with GPT-2 (Radford al. 2019). When given prompts of the form "A dolphin is a ". GPT-2 predicts that the most likely answer is "mammal", "fish", or "bird" depending on small differences in the wording of the prompt. In another example, GPT-2 states that unicorns have "four horns," directly after implying that unicorns only have one horn. Upon diagnosing such issues, it is unclear how to apply a targeted fix to the model, especially if retraining fine-tuning is impractical.
In this work, we draw on insights from cognitive science, especially from "dual process" theories of reasoning (Evans) (003). to explore how neural sequence models can better interface with prior knowledge and made more coherent and consistent. According dual process theories, human cognition can be understood as an interplay between a more intuitive and associative "System 1" and more deliberative and logical "System 2." Within this broad framework, automatic actions are driven by System 1, whereas System engages for more deliberative control: for example, judging validity of a logical argument that requires multiple steps of reasoning (Kahneman) 2013).
Correspondence to mnye @mit edu
Work primarily done during internship at Facebook Al Research.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Story so far
Candidate next sentences
Mary droppet the apple there.
System fast and intuttive
Danict rent to the garden. Mary maveled to the offiie.. Danirl grabhed the apple.
Generate (ex, GPT-3)
Dunist went garden.
Duniet went to the patle.
Final generation
Parse (GPT-3)
Parse (GPT-3)
Daniel went to garden. Mary traveled the office.
Daniel grabied the apple.
Danid soot to the patiis
System slow and logical
World Model
drop(Nary, apple)
Daniel. location - garden
Check (symbolic)
go(Daniel, garden)
apple, holder Daniel
Hary. location office
go(Daniel, patio)
Figure 1: Schematic of dual-system approach to text generation. Conditioned previous text, "System 1" neural generation model produces candidate next sentences. Semantic parses for each candidate are generated via few-shot parsing from GPT-3 and compared minimal world model to check consistency. Only candidates consistent with the world model state are incorporated into the final generation.
The prominent neural language models of today are single systems, with weaknesses akin those exhibited by the human System 1. For example, the cognitive reflection test (CRT) (Frederick) 2005) is classic probe of System 1 vs. System 2 reasoning humans. Participants answer a set of simple questions that have superficially compelling, but logically invalid, answers. These incorrect answers are often generated a first "gut" response (putatively, by System intuitive thinking): upon reflection, however, participants often realize that their responses were not logically or mathematically consistent (via more explicit System 2 reasoning). Consider the CRT problem on the left below:
Total cost in prompt
GPT-3 response
A ball and bat cost $1.10. The but costs one dollar more than the ball. How much does the ball cost?
$1.10
10 cents
$1.20
20 cents
$1.30
$0.30
$1.70
$0.70
Reading quickly, you might be tempted to say the ball costs 10 cents. Most participants give this response, in fact, especially if they are under time pressure or have limited attention (Kahneman 2013). Of course, if the bat is $1.00 more than the ball, and the ball costs 10 cents, then the total cost would be $1.20. The correct answer is that the ball costs 5 cents. Notably, in this and other classic CRT problems, GPT-3 (Brown et al. (2020) predicts the same "gut" response (prediction in red above; the table above shows that adjusting the price in the prompt also leads to similar effects; see Appendix Figure x for more CRT examples). GPT-3 appears vulnerable to the same sort of intuitive, unsystematic pattern recognition errors as humans- in this case, incorrectly subtracting one dollar from $1.10, without confirming that the answer satisfies each of the problem constraints.
Numerous studies have shown that engagement of System 2-style effort can help "override or inhibit default responses emanating from System 1" (Evans 2003), correcting inconsistent or un-systematic intuitive impulses. For example, when System 2 engaged by asking people to take more time to respond. people's 's accuracy improves the CRT task above (Kahneman 2013). It has been argued that integrating System processing could similarly improve AI systems (Goyal & Bengio 2020 Garcez& & Lamb 2020), and here explore this idea applied to neural sequence models.
In this work, we take inspiration from dual process theories to explore a neuro-symbolic generation system, wherein predictions from a neural model are treated as System 1 proposals, and logical, deliberative System 2 filters these proposals for consistency and soundness (see Figure [D. We further take inspiration from the fact that humans often do not need explicit supervision to reason about new problems or domains (e.g., see human evaluation task Section 4.2) and require that the System training, especially on example contradictions or commonsense violations. People can handle novelty by reconfiguring, rather than retraining, their internal models (Lake et al., 2017), and we strive to build machine systems capable of the same. System 2 model can help improve coherence and
2 module not need additional problem-spece
We show how a lightweight, consistency by adding small amount of symbolic reasoning.
We tackle two kinds of domains: text generation and instruction following. In both cases, we construct generative models over sequences by using a neural generation model to propose candidate generations and a symbolic world model that can accept or reject the generations and resample proposals if necessary. We first illustrate the approach by generating short stories based on the bAbl dataset (Weston al. 2015); this pedagogical, synthetic example illustrates how basic commonsense knowledge of objects, agents, and places can inform text generation model. We then test our

---

approach on rich, natural language vignettes based on CLUTRR (inha et al., 2019). focusing on ensuring consistency of family and interpersonal relationships. In both text generation domains, we interface between the explicit logical knowledgerreasonning of System 2 and generations of System I using a few-shot learning approach with state-of-the-an neural language models (GPT-3), which requires no additional training or fine-tuning. Even using off-the-shelff transformers and symbolic solvers, our dual-system model improves the consistency and coherence of text generations as measured by human judges. We test our approach also on instruction following, showing how goal- prediction models and execution models can easily be combined to achieve improved performance in low-data regimes. We show improvements over previous work in the gSCAN grounded compositional challenge (Ruis et al.) 2020); a dual-system model requires much less data to train than previous models, and achieves higher accuracy and stronger generalization. Overall, our findings indicate that neuro-symbolic, dual process models are a promising means of addressing longstanding problems of robustness and consistency in neural sequence models.
2
Related Work
Our approach incorporates semantic parsing (Liang 2016) as a component of a generative process, where neural generation is used in conjunction with parsing techniques. In our text generation experiments, we employ GPT-3 to perform few-shot semantic parsing without fine-tuning. Related work includes few or zero-shot semantic parsing using pre-training techniques and paraphrasing (Su & Yan| 2017: Herzig & Berant 2020). It also includes semantic parsing systems trained either without supervision (Liang et al. 2017: Mou et al. [017]] Muhlgay et al., 2019), or with synthetic language data (Marzoev et al.) 2020 Xu al. 2020b).
One popular technique for improving neural generations is generate-and-reramt wherein one model generates proposals and another reranks them. This broad approach has been used in image
generation (Ramesh et al. 202I), text generation (Holtzman et al. 2018 Shen et 2019 Deng
et al. 2020), dialogue systems (for control, coherence and safety (Welleck et al.] 2018
Smith et al.
2020, Nie et al. 2020 Xu al., 2020a)). and instruction following (Kurita & Cho 2020 Reranking is generally used to improve outputs with respect to relatively broad, holistic criteria. Here, our goal is to make generation robust to particular types of logical errors by pruning with respect to explicit symbolic constraints. Our approach can thus be considered closely related to techniques which employ explicit search to find generations satisfying particular logical constraints. Similar methods, such as guess-and-check or beam search pruning, have had success in neural program synthesis (Devlin et al. 2017 Nye et al. 2020).
Recent work in NLP has used template-based planning, in which a model generates text by first generating plan or skeleton, and filling in the missing words to produce naturalistic text (Xu et al.) 2018, Hua & Wang 2020). To generate stories, Martin et al. (2018) parses previous sentences into events and does planning in event space. Our work extends previous entity/relation/even planning in that the world model is not used for planning, but rather for post-checking candidate generations. Structured parsing of this type is also related to dialog tracking techniques such as slot-filling (Pieraccini et al. [992). In our work, fully compositional logical facts are extracted from utterances. It is therefore more closely related to systems which extract programs from dialogue, such as Andreas et al. (2020).
Recent work has also studied incorporating symbolic constraints into a neural decoding strategy in the context of natural language. Miao et al. (2019) introduce an MCMC-based inference-time propose-and-rejec strategy for satisfying constraints. They test on constraints such as paraphrase and grammatical error correction. Lu et al. (2020) introduces "NeuroLogic decoding," which uses logical constraints on neural language models produce generations which contain (or do not contain) required (or forbidden) keywords. In these works, the constraints are lexical or based on word/sentence similarity (and provided in the problem setup for et al. (2020)), whereas we study logical constraints on the world state decoded directly from observations or generations at test time. Other approaches for solving reasoning tasks end-to-end include Goyal et al. (202T). Serafini & d' ''Avila Garcez (2016), and Schlag & Schmidhuber (2018).
3

---

3 Integrating System and System 2
We introduce our dual dual-system approach using examples from the bAbl domain (Weston et al. 2015). which we also use to perform diagnostic experiments. Consider generating simple story involving people, places and objects, such as (from Figure(]):
Daniel went to the garden. Mary traveled to the office. Daniel grabbed the apple.
A model tasked with generating such stories must juggle several simultaneous demands: staying on topic and maintaining consistency of style and other textural elements (for which people rely on System 1), as well maintaining consistency with previous statements and commonsense knowledge (for which people rely on both systems). Consider continuing the story with one of the following:
(a) Daniel went to the patio.
(b) Mary dropped the apple there.
Sentence (a) is reasonable; sentence (b) is not because it is Daniel, not Mary, who has the apple. During generation, how might model distinguish between these candidates? Perhaps well-trained neural language model could track constraints of these sorts. Neural language models to date, however, often violate these types of commonsense, hard constraints without large high-quality corpus or explicit training on detecting violations of commonsense (Sinha al. 2019).
We address this problem by decomposing text generation into two parts: candidate generation facilitated by deep neural networks and logical pruning process implemented via separate symbolic module. Consider again the example above. To ensure consistency, our model would extract from the text the features of the world that are subject the hard, logical constraints, such as the location of objects and who is holding them. These constraints can then be checked against an explicit representation of current state of the world. For sentences (a) and (b), the system would extract and go (Daniel, patio) and drop (Mary, apple), respectively. A minimal world model would track the state of the apple, such that it maintains apple heple.holfe holder Daniel (or equivalently, Daniel. inventory = [apple]). When such model is given parse of a candidate generation, drop (Mary, apple), the mismatch between the current state and the proposed change would cause a violation, and the candidate generation will be rejected.
The main steps of our general approach are illustrated in Figure [ generate proposals from a System 1 proposal model, extract facts with a f'at extraction model, and filter proposed generations by ensuring that they satisfy the constraints given by the extracted facts and the minimal world model.
System 1: Generation. We use neural sequence models to produce System generations. In text generation domains, we use large, pre-trained model that can be fine-tuned conditioned via short prompt to generate relevant text. Text sampled from the System 1 model will be treated as candidate utterances. which will be parsed and filtered by System 2 (described below). For the bAbl examples, we use GPT-3 as our System proposal model through few-shot prompting with 10 example bAbl stories as context, generating new story one candidate sentence at time.
System 2: Fact extraction. A fact extractor, or parser, is used to mediate between the System I candidate proposals and the minimal world model within System 2. In our text generation domains, we use a pre-trained GPT-3 model without fine-tume mun-ttnnng to perform parsing.
For bAbl, our prompt consist of an initial descriptive sentence "Please parse the following statements into commands. The available commands are pickup, drop, and go." " and a small set (< 10) of representative semantic parsing examples (input sentences; output correct parses, such go (Bob, roof)). The parse of each utterance is produced via few-shot prompting (Brown et al. 2020): the utterance is added to the end of the prompt, and the subsequent GPT-3 generation is interpreted as the target parse. We found that this simple parsing technique works well and could easily be applied to other barsing-based tasks, as in Shin et al. 1202I) The parsing prompts are reproduced in full in the Appendix. As discussed in Section S] for the gSCAN instruction following domain, fact extraction is performed with a learned goal location prediction model.
System 2: Minimal world model. We use a lightweight, incomplete description of the state of the world as a world model in each domain, e.g., commonsense information about the people. objects and locations (Figure []. The goal is not to track and verify all the possible information; instead, we aim for minimalism, capturing just a few commonsense (or application-critical) variables that we want to ensure are correct. The world model facilitates tracking of long-rang logical dependencies and logical consequences, especially those which are not readily decodable from surface forms. The

---

world model also lets us integrate rule- sale- world- horld-knowledge without retraining (and without the need for a large set of labeled examples).
For the bAbl examples, the minimal world model keeps track of the people, locations and objects introduced in the story so far (Figure [T.. This encodes constraints on possible actions related to human core knowledge competencies (objects, agents, places) present early in human development (Spelke & Kinzler 2007); specifically, person or object can only be in one place a time, an object can only be possessed by single person at a time, person cannot "go" to a room they are already in, and a person cannot pick up an object if it is in a different room. See the Appendix for details.
Search. At generation time, the interaction between System generation and System 2 parsing yields neuro-symbolic, guess- wress-and-check search strategy. In a text generation scenario, where text is sampled from the model, our dual-system model improves upon a naive, neural-only sampling method by using the System 2 model to reject candidate utterances which are incompatible with the current state. When a candidate is rejected, a new candidate utterance is sampled from the System 1 model, which is again checked by System 2. This process repeats until a candidate utterance is accepted by System 2 (i.e., the utterance is compatible with the world state). This procedure allows the model to effectively search the space of candidate utterances, guided by the logical constraints from the minimal world model. In this work, we use straightforward probabilistic sampling to illustrate that the approach works with even very simple search mechanism. We imagine that the search procedure could be further optimized by applying, for example, beam search or stochastic beam sampling.
Diagnostic bAbI experiments. We use
GPT-3 only:
GPT-3+ world model:
John went to the bedroom. John picked up the apple there. Mary took the apple there. Mary taaelled to the office.
John went to the bedroom.
Task #2 from bAbl as a diagnostic test for our neuro-symbolic dual-system model. As shown above, this task consists of synthetically-generaate short stories involv- ing people, places and objects, and ques- tions concerning the locations of objects in these stories. We investigate performance on both question answering (QA) tasks and story generation. For the QA tasks, we parse each sentence in the story to encode each fact into the world model and parse the final question to query the world model, returning the answer given by the world model. We compare with two alternative models (Table 3 in the Appendix): GPT-3 by itself and a dual-system baseline that uses a neural Natural Language Inference (NLI) model as its System 2. The NLI-based dual-system model generates 10 candidates from GPT-3 and selects the candidate with the highest predicted probability of entailment under the NLI model given the context. We use the RoBERTa MNLI model as our off-the-shelf neural NLI model (Liu et al. 2019). which operates as a System 2 that does not use additional problem-speciie -speeffccc data or fine-tuming On 200 held-out tasks, our EPT--3-based "fact extractor" achieves 100% QA accuracy, far exceeding the performance of GPT-3 alone (29.0%) or GPT-3 generation with neural NLI scoring (32.5%; also see Table 3 in the Appendix). These results show that GPT-3 can be made to answer questions successfully when used for parsing with a world model, even when GPT-3 alone does not achieve high QA accuracy.
Johm picked up the apple there.
Mary mavelled to the office.
Daniel went back to the garden. Mary went the bedroom.
Daniel went back to the garden. Mary went to the bedroom. John went the hedrom.
Sandra went to the bedroom.
Sandra travelled the office.
Sandra went to the bedroom. Sandra travelled the office. Mary went back to the office. Where is the apple? A: office
Mary went back to the office.
Where is the apple? A: hedrom
Figure 2: Example bAbI stories generated by GPT-3 only (left) and our dual-system model (right). Logi- cally inconsistent lines are written in red text, and are removed from the story-so-far at generation time.
To test story generation, we use our GPT-3-based System I proposal model (few-shot prompted on 10 example stories) to sample new bAbl story, line-by-line. If generated utterance is inconsistent with the current state as indicated by the System world model, a new utterance is sampled from System 1 (repeating until a consistent utterance is sampled). Figure[|showe how the dual dual-system approach generates stories that mimic the statistical structure of bAbl storics, while remaining logically sound In contrast, GPT-3 alone was not able maintain logical coherence. In a set of 50 generated stories, all stories required at least one sentence to be resampled to maintain coherence, and over half of the generated sentences (53.1 1 1%) were rejected by our System 2 model to maintain logical consistency. These results demonstrate that equipping GPT-3 with a minimal world model produces logically coherent stories that mimic the textural structure of the bAbl domain. In the next section, we apply this approach to mimicking human-generated short stories in natural language.
Previous work has used domain specific entailmeet/conttaadiiion data to train reranking models Welleck al. 2018), however, this requires collecting dataset of domain- specific entailment and contradiction data.
5

---

4 Coherent Language Generation - CLUTRR
We apply dual-system approach to dataset of natural language using the CLUTRR dataset. CLUTRR contains human human-written written stories about people and their family relationships (see example in Figure 33. As with bAbl, CLUTRR was originally designed as Question Answering challenge; instead, we use to evaluate coherent language generation by querying models to generate complete CLstTR--ttyle stories
complete partially-generatee stories. Our particular aim is produce stories with coherent and logically consistent family relationships. As above, our language generation setup consists of pre-trained language models acting our System proposer, minimal world model as System 2, and neural semantic parser (implemented via few-shot GPT-3 prediction) as bridge between the systems. We use human judgments to assess whether our neuro-symbolic, dual-system model produces more consistent and coherent stories relative to a baseline.
4.1 Model specification
As our System proposal model, we used pre- trained neural models to produce candidate gen- erations one sentence at time. We experi- mented with GPT-3 as our System model (which we used above for bAbl), but found generations too unreliable, often outputting the empty string. Instead, we used a BART model (Lewis al. 2019) that was fine-tuned on the CLUTRR training corpus. This model also gives us opportunity to compare against best-case neural "single-system" baseline, specifically fine-tuned on story data. To maintain a state of family relations, we constraint solver in our "System 2" to encode family relationships (e.g., child (x, y), spouse (x, z)) and check that the candidate utterances do not contradict the previous state- ments (e.g., person cannot be their own child or married to their sibling). We implemented the world model as a set of logical relations and constraints using the Z3 solver (De Moura & Bjømnrr 2008). For instance, uncle(x, y) = mchild (y, x). To check candidate utterance, we query the solver to determine if the set of constraints is satisfiable or if there is contradiction. The full set of constraints and other details can be found in the Appendix. We again used GPT-3 as our semantic parser, extracting parses for each candidate utterance via few-shot learning. This parsing approach worked well, even for the natural language in this domain. We observed that parsing with GPT-3 was more successful when the target parse was naturalistie, i.e., "Bob is Joe's 's father." rather than "father(Bob Bob, Joe)". The parsing prompt is reproduced in full in the Appendix.
Carol nice Sunday
Carol related Justin Carol the grandmother Justin
attmmoon They want for a movie together a good time.
Figure 3: Sample story from the CLUTRR dataset. Each story consists of sequence of human- generated sentences concerning family relation- ships. Adapted from [inha et al. (2019).
require that the parent of cannot also be the unele of x: For all x, y.
Proportion Dual- Syo--SSttmmm Donerttttonnn Genertion Profemmdd
1.--
Tracy weat to dinner with her daughter Shantel.
Ove Single Systamn Systamm Clanenaten
They then went to the park afterwards.
Tracy loves her Aaron very much. He loves ber. too.
System 2 type
Harold took his grandson Aaron the
0.4
Minimal World Model
Which of the following sentences makes the most sense given the information above?
Harold's is son Aaron didn't 't go because was
afraid of animals.
Harold's daughter, Tracy, went him.
babasst CromitCondiion
Nodd
Figure 4: Example trial from CLUTRR human judgement experiment. Participants were in- structed to select which of two options makes the most sense given the prompt. One op-
Figure 5: CLUTRR human judgment experiment results. Bars denote proportions of dual-systen generations selected as making more sense over single-system generations, in each of four condi- tions. Error-bars denote bootstrapped 95% con- fidence intervals of the item means. The points denote means for each individual item in the exper- iment and are jittered horizontally for clarity.
tion was generated by the System 1 model only ("single-system"), while the other was generated by the dual-system model.

---

Table 1: Statistics from CLUTRR story generation. We report the percentage of generations (on both a per-line and per-story basis) for which the System 2 world model did not detect an error. The dual-system model is able to detect many inconsistencies in the neural single-system generations, and most can be corrected by re-sampling new candidates (up to limit of ten).
% w/out error detected (per line)
% w/out error detected (per story)
single-system
dual-system
singlc-system
dual-system
(neural gen. only)
(neural gon-l model)
(neural gen. only)
(neural wonld model)
prompt from dataset prompt from model
82.8
97.1
60
96.1
71.9
96.3
36.4
93.5
4.2 Human judgments
We test our dual-system neural generation + world model method in its ability to generate stories that are deemed by naive human participants to be more naturalistic and coherent than those generated from the baseline models. Specifically, we asked participants to select which of two continuations made the most sense to them, where one continuation was generated from the neural model alone (single-system) and the other from a dual-system model (either the world model System 2 or the neural NLI System 2).
Participants (N = 101) were recruited on the crowd-sourcing platform Prolific and compensated $2 for the task (~-15 minutes, so roughly $8/hour). Participants gave informed consent, and the study was approved by MIT's s IRB. 21 participants were excluded for failing an instruction quiz, incorrectly answering more than one of five filler questions, or finishing the task too quickly. The data we collected contains no personally identifiable information or offensive content.
Participants.
Procedure. Participants began the experiment by reading a set of instructions and answering comprehension questions. On each main trial, participants were shown a prompt consisting of several sentences and were asked to choose which of two possible continuations made the most sense (an example trial is shown in Figure Participants were instructed that if name appeared multiple times within a trial, then it referred to the same person, whereas if a name appeared across trials, then it was not referring to the same person. For each trial, one continuation option was generated by the neural baseline, while the other was a dual-system generation. We selected generations from the neural only baseline that were rejected by the System 2 model in order to maximize the differences between the models' generations; thus, human judgments pertain to generations that the models disagreed on. Each participant performed between 20 and 26 trials.
only single-systed
Materials. Participants were randomly assigned to one of four between- caawwen-participant conditions, which varied according to the kind of prompt and the kind of dual-system model. The prompt was either generated from the model (up to the point of disagreement between System I and System models; "Prompts from model" condition) or taken completely from the length 4 CLUTRR systematic generalization test dataset ("Prompts from dataset" condition). To generate prompts for the "from model" condition, we took the first sentence of each story from the CLUTRR test dataset and generated subsequent prompt sentences from the dual-system model; sentences were generated until the two systems disagreed (i.e., System generated sentence that System 2 rejected), at which point the "rejected sentence" served as the neural only (single-system) baseline generation and the first resampled sentence that System 2 accepted served as the dual-system generation. Prompts were sampled to a maximum length of four sentences. The dual-system model shown to participants used System 2 based on either our constraint-based "world model" or the neural NLI baseline.
Table [ catalogs critical statistics from the stimulus generation process. We generated vignettes from the System 1 model and report the percentage of System 1 generations which are deemed correct by the System 2 model We also report the percentage of generations corrected by the System model (i.e., if System made an error, could System 2 fix it within 10 attempts??. We report these statistics on both a per-story and per-line basis. According to System 2, the System generation model makes a lot of errors (only 36.4% of stories and 71.9% of lines were error-free, in the "from model" condition). In most instances, re-sampling new generations yields stories that, according to
'For all System 1 generations, we used model temperature of 1.0. For the neural NLI baseline, we used 0.9 probability of contradiction as the cutoff for rejection. Our dual-systent model uses sampling budget of 10 System samples per sentence. Contradictions remaining after 10 samples considered dual-system emror.
7

---

Table 2: Accuracy on gSCAN splits. Models were trained on 5000 examples (only 2.5% of the gSCAN training data). See Appendix Table 4 for additional results.)
Conaliiie sulice sngeetee
gamm
NUX, REGHT,
TURS WALK
KULK TURS LEFT, MAX
Finil actio TURN LEFT, VALK, TURN
Test split:
single-system
dual-system
dev
71.7
83.3
Clock
random
57.2
74.7
yellow squares
68.1
81.3
red squares
64.9
78.1
Figure 6: Schematic of our dual-system approach to gSCAN. We train neural sequence model to predict both a distribution over action sequences, and distribution over target locations. At test time, we decode candidate action sequences from the model, execute them the gridworld, and only accept sequence that brings the agent to the predicted target location (shown in green).
novel direction
0.0
0.01
relativity
41.0
53.6
class inference
68.1
76.2
adverb (k=1)
0.0
0.0
adverb to verb
20.8
21.8
"From Heinze-Deml & Bouchacourt
2020)
System 2, no longer contain logical errors within budget of 10 samples (93.5% of stories and 96.3% of lines were error-free, respectively).
The human evaluation indicates that System 2 is indeed correcting genuine errors in the stories. As summarized in Figure 5 participants strongly preferred the dual-system neural generation + world model continuations in comparison to the neural only single-system continuations (proportion preferring dual-system 0.84; bootstrapped 95% confidence interval [0.77, 0.89] and 0.79 [0.77, 0.89] for the "from dataset" and "from model" prompt conditions, respectively). The dual-system approach, however, did not improve generation quality when the System 2 was based on off-the-shelf neural NLI model (Proportion preferring dual-system 0.51; [0.40, 0.64] for "from dataset"; 0.58 [0.48, 0.68] for "from model"). Thus, when using minimal world model, the dual-system approach dramatically improves logical consistency without any need for additional training or fine-tuning. People clearly prefer suurosyymboli generations from the dual-system model over purely neural generations from a single-system model
Results.
Grounded Instruction Following
The dual-system approach offers general-purpose means of improving upon generative, neural sequence models by incorporating logical constraints. To highlight its generality, we examine how the dual-system perspective can be deployed in very different domain: grounded instruction following. In Heiinze-Dmml & Bouchacourt (2020), a learned target location predictor was used to increase the accuracy of neural action sequence generation model. Here, we show how to increase performance further by enforcing consistency between the target location predictor and the action sequence generator in our dual-system framework.
We use the gSCAN benchmark (Ruis et al. 2020). a recently proposed grounded instruction following dataset designed to measure compositional caatooonnggeeerrai generalization in neural systems. Given an initial gridworld state and an instruction, c.g., "walk to the big square," an agent must predict the sequence of low-level actions which achieve the goal, e.g., "TURN LEFT, WALK, TURN LEFT, WALK" (See Figure 6 The dataset contains several test splits, each testing different aspects of compositional generalization.
Our model builds on Demil & Bouchacourt (2020) by using an LSTM to predict the correct action sequence and target location. Given a command e and an initial gridworld state 8, the neural network defines two distributions: a distribution over action sequences (ala,, s) and a distribution over target grid locations (ooc (I)c, s). FHeinze-Deml & Bouchacourt (2020) showed that when these distributions share parameters, using location prediction an auxiliary loss improves the accuracy of the action sequence prediction model. We can further exploit these two models by noticing that when predicted action sequence is not consistent with predicted target location, then either the action sequence or the target location must be incorrect. Since the target location is much simpler to predict,
*We note that the effect size of our approach depends upon the prevalence disagreements between the single-system and dual-system models. As reported in Table we find that the prevalence of disagreements is at least 14% 57%, depending on the data regime. Therefore, the model predictions do often differ. resulting in large overall effect size.
8

---

and thus much more likely to be correctly predicted, if a predicted action sequence is not consistent with the predicted target location, then the action sequence is most likely incorrect. Our dual-system framework can use this property to increase action sequence prediction accuracy. Consider the initial state and command in Figure Our model predicts candidate action sequences, and also predicts that the most likely target location is the grid containing the bigger yellow square (highlighted in red). The model then executes the candidate action sequences, and only accepts a sequence which results in the agent standing in the target location.
In the language of our dual-system approach, we treat the distribution over actions qa(ale, s) as our System proposal model. The distribution over target locations Qloe (1lc, s) serves as fact extractor model, which extract a location constraint deterministic gridworld execution model [ so) + s,, which takes a state and action and predicts the resulting state. At test time, we first extract the predicted location as | arg maxy qoo (c). We then search through the possible action sequences from qa (-Ic), conditioned on agreement with 1. In our experiments, we use a sample- based search with a maximum budget of 50 samples. We trained models on random subsets of the gSCAN training set of varying sizes: 5000 datapoints, 8000 datapoints, and 20000 datapoints (2.5%, 4% and 10% of the original training set, respectively).
As a minimal world model, we use a
The results show that the System 2 execution model improves performance without the need for any additional training (see Table 2 for results training on 5000 examples). In contrast the single-system model, the dual-system model allows for sampling many candidate action sequences from the neural network, accepting only consistent sequences. This guess-and-chece approach greatly increases the evaluation accuracy, improving upon prior work on gSCAN, particularly in low-data regimes.
Results.
Limitations
In its current form, our approach is most useful in domains where naturalistic, learned generation is necessary and where a small number of mission-critical logical constraints can be explicitly articulated. Our system will be less useful when constraints creative domains such as writing poctry) or when there are many constraints, since the minimal world model must be hand-engineered Enforcing strict constraints may also pose risks: if the constraints are not only logical but cultural, they may be harmful if misapplied. However, these constraints must be articulated explicitly in symbolic model, and are thus easier identify and correct.
more difficult to articulate (c.g..
The current few-shot parsing technique may also suffer from a limited capacity. For more complex domains, the number of examples required to specify the desired parsing behavior may be too large (i.e., they may not fit in the input window) or too complex for model to perform parsing accurately. While some tasks may not be suitable, the complexity of the world model need not necessarily increase hand-in -nddddd with the complexity of the application domain. A dual-system model will be most successful when tracking just few critical variables (e.g., tracking consistency in family relations, as in our experiments, or tracking scheduling constraints when discussing a team plan).
A promising direction for future work is to incorporate learning into the System 2 world model. Currently, the minimal world knowledge that exists in System 2 can be easily modified, but changes must be made by hand. Improvements would come from automatically learning and updating this structured knowledge, possibly by incorporating neuro- neuro-symbolic learning techniques (Ellis et al. 2020 Mao et a (019), or other neuro-symbolic integration work such as Tsamoura et al. (202
Michael & Valiant (0008).
Learning could improve our dual-system approach in other ways, c.g., by training neural module to mimic the actions of a symbolic System 2. The symbolic System 2 judgments could be used as a source of supervision; candidate utterances rejected by the symbolic System 2 model could be used as examples of contradictory sentences, and accepted utterances could be used as examples of non- contradictory statements. This oversight could help train a neural System 2 model capable of more subtleties than its symbolic counterpart, especially in domains where labeled examples are otherwise unavailable. This approach may also help us understand aspects of human learning, where certain tasks that require slower, logical reasoning can be habitualized over time and tackled by faster, more intuitive reasoning.

---

Recent work et al. 2021) has shown that large pre- wre-trained neural models learn to approximately represent certain types of structured semantic information. However, not yet clear how represen- tational fidelity translates to logical coherence during generative tasks. Our current approach allows us to explicitly fix logical errors in generation, which may ultimately be caused by representational errors. Understanding how we might leverage our approach to improve the representation of struc- tured knowledge within neural models a promising direction for future work, which could lead to increased generation consisteney and coherence.
7 Conclusion
Inspired by dual process theories from cognitive science. we combine the respective strengths of neural and symbolic approaches build more robust models that can more effectively incorporate domain knowledge. For language generation, we showed that equipping neural generation with minimal symbolic world model increased language coherence and consistency. For grounded instruction following, we showed that requiring test-time consistency between predicted action sequences and goal locations led to improved performance, especially in low-data regimes. Our neuro-symbolic approach can readily be applied to other domains and types of prior knowledge, as lightweight way of improving the coherence and consistency of powerful neural sequence models.
This paper just scratches the surface of how structured knowledge can make neural systems more robust; we hope inspire further work into neuro-symbolic systems which possess the robustness and commonsense necessary for human-level intelligence.
Acknowledgments
We thank Laura Ruis, Jacob Andreas. Yewen (Evan) Pu, Joe O' O'Connor and Guy Davidson for helpful comments on an earlier version of this manuscript. MN is supported by NSF Graduate Research Fellowship.
References
Andreas, J., Bufe, J., Burkett, D., Chen, C., Clausman, J., Crawford, J., Crim, K., DeLoach, J., Dorner, I.., Fisner, J., et al. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics, 8:556-577 571, 2020.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. arXiv preprini arXiv:2005. 14165, 2020.
De Moura, L. and Björner, N. Z3: An efficient smi solver. In Tools and Algorithms for the Construction and Analysis of Systems, pp. 337-340. Springer, 2008
Deng, Y., Bakhtin, A., Ott, M., Szlam, A., and Marc' Aurelio Ranzato. Residual energy-based models of text generation. In International Conference on Learning Representations (ICLR), pp. 1-18, 2020.
Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed, A.-r,, and Kohli, P. Robustfill: Neural program learning under noisy i/o. ICML, 2017.
Ellis, K., Wong, C., Nye, M., Sable-Meyer, M., Cary, L., Morales, L.,, Hewitt, L., Lolar-Lezama, A., and Tenenbaum, B. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv preprint vxXiv:200608838 2020.
Evans, J. S. B. In two minds: dual-process accounts of reasoning. Trends in cognitive sciences, 7(10): 454-459, 2003.
Frederick, S. Cognitive reflection and decision making. Journal of Economic perspectives, 19(4):
25-42, 2005.
Garcez, d. and Lamb, L. C. Neurosymbolic ai: The 3rd wave. arXiv preprint urXiv:2012.05876, 2020.
10

---

Goyal, A. and Bengio, Y. Inductive biases for deep learning of higher-level cognition. arXiv preprint arXiv:2011.15091, 2020.
Goyal, A., Didolkar, A., Ke, N. R., Blundell, C., Beaudoin, P., Heess, N., Mozer, M., and Bengio, Y. Neural production systems. CoRR, abs/2103.01937, 2021. https//tts//// org/ abs/ 2103. 0193701937
Heinze-Deml. C. and Bouchacourt, D. Think before you act: A simple baseline for compositional generalization. arXiv preprint wrXiv:2009.13960 2020.
Herzig, J. and Berant, J. Span-based semantic parsing for compositional generalization. arXiv preprint arXiv:2009. 2020.
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., and Choi, Y. Learning to write with cooperative discriminators. arXiv preprint arXiv:/805.000 2018.
Hua. X. and Wang, L. Pair: Planning and iterative refinement in pre-trained transformers for long text generation. In EMNLP, 2020.
Kahneman, D. Thinking, fast and slow. kindle ed, 2013.
Kurita, and Cho, K. Generative language- policy in vision-and- and-aauage navigation with bayes' rule. arXiv preprint urXiv:2009.077833 2020.
Lake. B. M. and Murphy, G. L. Word meaning in minds and machines. arXiv preprint MXXi:2008.011600 2020.
Lake, B. M., Ullman, D., Tenenbaum, J. B., and Gershman, S. J. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:E253, 2017.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, I. Bart: Denoising sequence-to-sequencce pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13460 2019.
Li, B. Z., Nye, M., and Andreas, Implicit representations of mcaning in neural language models. arXiv preprint arXiv:2106.0000 00737, 2021.
Liang, C., Berant, J., Le, Q. V., Forbus, K., and Lao, N. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 23 23-33, Vancouver, Canada, coli. wuw.uniiie saar-sarlaaddd.co pttp://pppppppp//ppeeepepptt 10t0///711111......0 pdf
2017. URL
Liang, Learning executable semantic parsers for natural language understanding. Communications of the ACM, 59(9):68-76, 2016.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 2019.
Loshchilov, and Hutter, F. Decoupled weight decay regularization. arXiv preprint arX1v:1711.05101 2017.
Lu, X., West, P., Zellers, R., Bras, L., Bhagavatula, C., and Choi, Y. Neurologic (eco supervised neural text generation with predicate logic constraints, arXiv preprint arXiv:2010.12884, 2020.
Mao, J., Gan, C., Kohli, P., Tenenbaum, J. B., and Wu, J. The neuro-symbolic concept learne: Inter- preting scenes, words, and sentences from natural supervision. arXiv preprint arXiv: 1904. 12584, 2019.
Martin, I.. J., Ammanabrolu, P., Hancock, W., Singh, S., Harrison, B., and Riedl, M. O. Event representations for automated story generation with deep neural nets. In AAAI, 2018.

---

Marzoev, A.. Madden, S., Kaashoek, M. F., Cafarella, M., and Andreas, J. Unnatural language processing: Bridging the gap between synthetic and natural language data. arXiv preprint arXiv:2004. 13645, 2020.
Miao, N., Zhou, H., Mou, L.., Yan, R., and Li, Cgmh: Constrained sentence generation by metropoolis-hattings sampling. In AAAI, 2019.
Michael, L. and Valiant, L. G. A first experimental demonstration of massive knowledge infusion. 2008.
Mou, I.,, Lu, Z.., Li, H., and Jin, Z. Coupling distributed and symbolic execution for natural language queries. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2518-2522 PMLR, 06-11 Aug 2017. URL http: htttt::/proceedingg. mlr wmt..press//////oo//... hhtml. html
Mublgay, D., Herzig, J. and Berant, J. Value-based search in execution space for mapping instructions to programs. pp. 1942-1954, 01 2019. doi: 10. 10.18853/v1/N19-111 1193,
Nie, Y., Williamson, M., Bansal, M., Kiela, D., and Weston, J. I like fish, especially dolphins: Addressing contradictions in dialogue modelling. arXiv preprint arXiv:2012. 13391, 2020.
Nye, M. I., Solar-Lezama, A., Tenenbaum, J. B., and Lake, B. M. Learning compositional rules via neural program synthesis. arXiv preprint arXiv:2003. 05562, 2020.
Pieraccini, R., Tzoukermann, E., Gorelov, Z., Gauvain, J.-L., Levin, E., Lee, C.-H., and Wilpon, J. A speech understanding system based on statistical representation of semantics. In (Proceedings) ICASSP-92: 1992 IEEE International Conference Acoustics, Speech, and Signal Processing, volume 1, pp. 193-196 vol.1, 1992. doi: 10. 10.1099ICASSP.1122225999
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. 2021.
Zero-shot text-to-image generation. arXiv preprint
Ruis, L., Andreas, J., Baroni, M., Bouchacourt, D., and Lake, B. M. A benchmark for systematic generalization in grounded language understanding. arXiv preprint arXiv:2003.05161, 2020.
Schlag. and Schmidhuber, J. Learning to reason with third-order tensor products. CoRR,
abs/181 12111811.12143, 2018. URL http:://axiit hrg. btrgg/bbs/in 1811. 12143
Serafini, and Avila Garcez, A. S. Logic tensor networks: Deep learning and logical reasoning from data and knowledge. CoRR, 0bs/16606004433 2016. http://////rr abs///b////// /166 04422
Shen, S., Fried, D., Andreas, J.. and Klein, D. Pragmatically informative text generation. arXiv preprint #xXiv1904.003300 2019.
Shin, R., Lin, C. H.. Thomson, S., Chen, C., Roy, S., Platanios, E. A., Pauls, A., Klein, D., Eisner, J., and Van Durme, B. Constrained language models yield few-shot semantic parsers. arXiv preprint arXiv:2104.08770 2021.
Sinha, K., Sodhani, S., Dong, J., Pineau, J., and Hamilton, W. L. Clutrr: A diagnostic benchmark for inductive reasoning from text. arXiv preprint arXiv: www.:i90.0000 19177.617.. 2019.
Smith, E. M., Gonzalez-Rine Rico, D., Dinan, E., and Boureau, Y.-L. Controlling style in generated dialogue. arXiv preprint arXiv:2009. 10855, 2020.
Spelke, F. S. and Kinzler, K. D. Core knowledge. Developmental Science, (0(1):89-96 2007.
Su, and Yan, Cross-domain semantic parsing via paraphrasing. arXiv preprint (rXivv1004.05977 2017.
12

---

Tsamoura, E.. Hospedales, T., and Michael, L. Neural-symbolic integration: A compositional perspective. Proceedings of the AAAI Conference on Artificial Intelligence, 35(6):5011-506) May 2021. htt.pttss:///oss. aaai aaai org/ index. phtp:/hAT//articce//ieeee/ wwtew//www/ww/////// 16639
Welleck, S., Weston, J. Szlam, A., and Cho, K. Dialogue natural language inference. arXiv preprint arx0v:181..00671 2018.
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., and Mikolov, T. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05668, 2015.
Xu, J., Ren, X., Zhang, Y., Zeng, Q., Cai, X., and Sun, X. A skeleton-based model for promoting the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4306-4303 Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v 1462........... URL
coherence among sentences in narrative story generation. In Proceedings the
https://ver //www aclweb. org/ wnwhaantoootyooaaaaeee D18- 1462
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., and Dinan, E. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079, 2020a.
Xu, S., Semnani, S. J., Campagna, G., and Lam, M. S. Autoqa: From databases to qa semantic parsers with only synthetic training data, arXiv preprint arXiv:2010.04550 2020b.
13


Reviews:
[{"review_id": "QGJZm4ip8fJ", "reviewer": "Reviewer_Eow4", "summary": "Contribution: The paper builds upon the proposals of  (Goyal & Bengio, 2020; Garcez & Lamb, 2020) to give the neural networks an ability to mediate information between the system 1 and system 2 processing. \n\nThe paper proposes to improve existing sequence models by adding system 2 inspired logical reasoning abilities. \n\n\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Method: The proposed approach proposes to improve the current neural sequence models (system 1) by adding a logical system 2. The basic idea is \"system 1\" generator suggests various proposals, and a logical system accepts or rejects the proposals generated by system 1. \nThe proposed method is  a lightweight, training-free means of improving existing sequence models. \n\nThe pipeline is as follows:\n(a) Generate proposals from a neural sequence model.\n(b) Fact extraction from the generated proposals.\n(c) World model to ensure that the fact extraction is aligned with the set of logical relations and constraints (part of system 2).\n\nExperiments: The paper tests the proposed method on Coherent Language Generation - CLUTRR and grounded language learning tasks. \nThe method is tested by its abilities to generate coherent  stories by human participants. \n\nPros:\n-  The paper is very well written.\n-  The relevant work is very well described (especially the  generate-and-rerank literature). \n\n\nAll the limitations are adequately addressed in the paper. \n\n*In its current form, our approach is most useful in domains where naturalistic, learned generation is necessary and where a small number of mission-critical logical constraints can be explicitly.*\n\nSuggestions: One suggestion would be, in it's current form the title seems overly generic as compared to what is actually done in the paper as something already hinted in the paper in the limitations section. \n\nOther work which may be relevant for learning system 2 is the work of Neural Production systems [1] inspired from the work of production systems which consists a set of variables and logical rules, but something entirely end to end differentiable. \n\nRelevant Literature Work:\n- [1] Neural Production Systems, https://arxiv.org/abs/2103.01937.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are very well addressed. The reviewer appreciates it. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "N8PrfrGPzZq", "reviewer": "Reviewer_wEz8", "summary": "The paper argues for the importance of logical coherence and consistency in language models such as GPT3, and proposes a neurosymbolic approach whereby logical rules are deployed to validate language sequence generations. This is highly important in the current landscape of research. \n\nThe authors propose to derive logical statements such as \"drop(Daniel, apple)\" directly from the language sequences generated by the model, and use GPT3 to illustrate. The study of such automatic generation of logical statements is also very relevant at present. \n\nThe logical statements are then applied over time to constrain future GPT3 language derivations, using the symbolic knowledge to validate the derivations with the objective of achieving higher coherence and consistency over time.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As acknowledged in the paper, a limitation is that the symbols (available commands) are pre-defined when in fact they could have been derived from the text (e.g. the verbs in the sentences).\n\nThe terms soundness, consistency and coherence are used in a somewhat interchangeable form, e.g. in: \"...while remaining logically sound. In contrast, GPT-3 alone was not able to maintain logical coherence. In a set of 50 generated stories, all stories required at least one sentence to be resampled to maintain coherence, and over half of the generated sentences (53.1%) were rejected by our System 2 model to maintain logical consistency\". These terms have precise meaning in logic and ideally, formal definitions of their use in the context of the paper should have been provided.\n\nThe human judgement experiment carried out in the paper is sound and its methodology is described in detail. The results are impressive. By evaluating results on two data sets, the authors also contribute potential benchmarks for neurosymbolic AI.   \n\nQuite a good number of references is provided and I am aware that in this multidisciplinary area it is difficult to cover all the relevant work, from the work in language to the theory of neurosymbolic AI and related systems. However, two papers come to mind which seem highly related, since the work is focused highly on adding constraints to neural models [1] and on reasoning in the context of the BaBi data set [2]. Such related work might offer end-to-end alternatives worth comparing with the approach proposed here.\n\n[1] Logic Tensor Networks: Deep Learning and Logical Reasoning from Data and Knowledge. Luciano Serafini and Artur d'Avila Garcez. https://arxiv.org/abs/1606.04422\n\n[2] Learning to Reason with Third-Order Tensor Products. Imanol Schlag and Jürgen Schmidhuber. https://arxiv.org/abs/1811.12143  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "DocEb2PWwlv", "reviewer": "Reviewer_Tn5q", "summary": "This paper adopts a dual system theory from cognitive science to check the consistency of the outputs from the system 1 neural generators using the system 2 symbolic reasoners with the aid of facts extracted from the output candidates and the world background knowledge constraints incorporated into the system. The proposed model has been applied to multiple tasks including story generation and question answering over the widely used bAbI task dataset in the domain of people and objects location tracking, story generation in the people and their family relationship domain over CLUTRR dataset, and the grounded instruction task. The results show the improvement in the coherence and accuracy of the neurally-based generation outputs.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Inspired by the dual systems in cognitive science this paper presents a novel and interesting neuro-symbolic approach for increasing the coherence and consistency of the neural generators where the neural generation candidates serve as system 1 proposals and the symbolic system 2 plays the role of a consistency checker; the neural generator generates the outcome repetitively until the logical consistency checker confirms the quality of its output. \n\nThe paper is very well-written. The idea and the application are very interesting and the general framework introduced in this paper can further get extended in future works in other domains of interest with a lightweight world model/background knowledge/ ontology with different neural generation and semantic parsing approaches for increasing the coherence, accuracy, and consistency in the output of AI systems. The evaluations are acceptable and show the potential of the proposed framework for non-complex generations with a relatively straightforward world model. Hence,  I'd like to see the revised manuscript being accepted by the conference.\n\nMistakes/Suggestions: \n- In table 1, for the result of story generation over CLUTRR, the result for the dual system has been reported for maximum 20 attempts; the question arises here on how well the dual system will work if you allow lower maximum failure attempts. Please report the system accuracy for different maximum attempts.\n- Please provide more discussion on whether/if the proposed model can get applied to other bAbI tasks, how, and why?\n- Please provide more information on the number of annotators per trial and their agreement rate.\n\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A detailed discussion on the limitations of the proposed approach is provided. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issue. As per the authors' discussion in the \"Participants\" section, the study is approved by IRB.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "130Urvbl4aX", "reviewer": "Reviewer_oMsG", "summary": "The paper proposes a neural-symbolic system where the symbolic module serves to screen, at testing time, the suggestions of the neural module, allowing only those that obey the constraints of a symbolic theory to be presented as the final predictions of the system. The paper shows that this post-filtering improves the performance against a system comprising only the neural module.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the System 1 / System 2 narrative to be a bit of a stretch. First, the dual process theory is much richer than what this paper would seem to suggest. Second, the use of symbolic constraints on top of a learned model (neural or not) has been explored a lot in the literature (e.g., \"Learning and Inference over Constrained Output\" by Dan Roth and others, IJCAI 2005), and in fact more advanced integrations have been proposed where the symbolic constraints act not only during the prediction time, but also during the training time to train the learned model to respect the symbolic constraints. This last direction seems to be mentioned at the end of Section 6 as a future direction, ignoring a lot of relevant work (see, e.g., recent work by Efthymia Tsamoura and Loizos Michael on compositional neural-symbolic integration).\n\nModulo the lack of originality of the idea, the paper also seems to lack sufficient details to appreciate the empirical setting and results. For the most part, the paper offers a high-level summary (revisiting often the System 1 / System 2 narrative), without explaining in detail the key points of the methodology. For example, it is unclear to me how the system was eventually evaluated, and whether the evaluation is meaningful. First: it is obvious, without needing an empirical study, that post-filtering any generating process with sound constraints can only improve performance. This already makes the value of the empirical study mood. Second, the choice of how to compare the no-filtering and the post-filtering approaches seems to raise questions. The authors state: \"We selected generations from the neural only baseline that were rejected by the System 2 model in order to maximize the differences between the models’ generations; thus, human judgments pertain to generations that the models disagreed on.\" Doesn't this inflate the performance in favor of the \"paper's favorable\" outcome? If, for the sake of discussion, the two approaches only disagreed on 1% of the cases, would focusing on those cases be reasonable? If, on the other hand, the paper is attempting to show results conditioned on disagreement, then isn't it trivial that evaluating two systems in disagreement, with one of them encoding sound constraints, will always end up presenting favorable results for the latter system?\n\nLastly, the claim (around line 375) that \"While System 2 world knowledge can be easily modified by explicit modifications, such knowledge cannot be automatically learned or updated.\" seems to completely ignore a ton of past and recent work on symbolic learning and machine coaching (e.g, work by Leslie Valiant, Loizos Michael, Brendan Juba, and others on learning and reasoning). \n\n--- after reading the author response ---\n\nThank you for your response. I think that the clarifications that were provided in your response could help improve the readability and the impact of the paper. I have increased my score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Seems fine.\n", "review_text": "", "extra_scores": {"ethical_concerns": "Not something obvious.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3-4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the author response and the discussion all reviewers agree that the paper should be published at NeurIPS. The authors take inspiration from the dual process theory to the outputs of a neural generation procedure is consistent with a predefined set of world knowledge. The paper is well written and the ideas are clearly explained. The authors test their technique on a number of tasks and evaluate their method using human judgements and show good results.
Decision: Accept (Poster)