Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning
Authors: Maxwell Nye
Michael Henry Tessler
Joshua B. Tenenbaum
Brenden M. Lake
Venue: NeurIPS 2021 Poster (2021)
Keywords: System 2
cognitive science
dual-system
coherence
consistency
dual-process

Abstract: Human reasoning can be understood as an interplay between two systems: the intuitive and associative ("System 1") and the deliberative and logical ("System 2"). Neural sequence models---which have been increasingly successful at performing complex, structured tasks---exhibit the advantages and failure modes of System 1: they are fast and learn patterns from data, but are often inconsistent and incoherent. In this work, we seek a lightweight, training-free means of improving existing System 1-like sequence models by adding System 2-inspired logical reasoning. We explore several variations on this theme in which candidate generations from a neural sequence model are examined for logical consistency by a symbolic reasoning module, which can either accept or reject the generations. Our approach uses neural inference to mediate between the neural System 1 and the logical System 2. Results in robust story generation and grounded instruction-following show that this approach can increase the coherence and accuracy of neurally-based generations.

Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning
Maxwell Nye" MIT
Michael Henry Tessler
Joshua B. Tenenbaum MIT
Brenden M. Lake NYU Facebook Al Research
MIT DeepMind
Abstract
Human reasoning can be understood as an interplay between systems: the intuitive and associative ("System 1'") and the deliberative and logical ("System 2"). Neural sequence models-which have been increasingly successful performing complex, structured exhibt the advantages and failure modes of System I: they are fast and learn patterns from data, but are often inconsistent and incoherent. In this work, we seek lightweight, training-fres free means of improving existing System 1-like sequence models by adding System 2-inspired logical reasoning. We explore several variations this theme in which candidate generations from a neural sequence model are examined for logical consistency by a symbolic reasoning module, which can either accept reject the generations. Our approach uses neural inference to mediate between the neural System I and the logical System 2. Results in robust story generation and grounded instruction -etruuttoo-followinng show that this approach can increase the coherence and accuracy of neurally-base generations.
Introduction
Despite recent success, neural sequence models often fail to produce consistent and coherent genera- tions. When generating stories, language models may forget the attributes of specific characters (such as personality and background information) (Welleck et al. 2018). ignore previously established relationships between characters (such family relationships) (Sinha et al., 2019), otherwise contradict prior statements (Brown et al. 2020). Similarly, neural models can make statements that contradict basic world knowledge or the logical entailment structure of known facts.
Lake & Murphy (2020) illustrated several of these issues with GPT-2 (Radford al. 2019). When given prompts of the form "A dolphin is a ". GPT-2 predicts that the most likely answer is "mammal", "fish", or "bird" depending on small differences in the wording of the prompt. In another example, GPT-2 states that unicorns have "four horns," directly after implying that unicorns only have one horn. Upon diagnosing such issues, it is unclear how to apply a targeted fix to the model, especially if retraining fine-tuning is impractical.
In this work, we draw on insights from cognitive science, especially from "dual process" theories of reasoning (Evans) (003). to explore how neural sequence models can better interface with prior knowledge and made more coherent and consistent. According dual process theories, human cognition can be understood as an interplay between a more intuitive and associative "System 1" and more deliberative and logical "System 2." Within this broad framework, automatic actions are driven by System 1, whereas System engages for more deliberative control: for example, judging validity of a logical argument that requires multiple steps of reasoning (Kahneman) 2013).
Correspondence to mnye @mit edu
Work primarily done during internship at Facebook Al Research.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Story so far
Candidate next sentences
Mary droppet the apple there.
System fast and intuttive
Danict rent to the garden. Mary maveled to the offiie.. Danirl grabhed the apple.
Generate (ex, GPT-3)
Dunist went garden.
Duniet went to the patle.
Final generation
Parse (GPT-3)
Parse (GPT-3)
Daniel went to garden. Mary traveled the office.
Daniel grabied the apple.
Danid soot to the patiis
System slow and logical
World Model
drop(Nary, apple)
Daniel. location - garden
Check (symbolic)
go(Daniel, garden)
apple, holder Daniel
Hary. location office
go(Daniel, patio)
Figure 1: Schematic of dual-system approach to text generation. Conditioned previous text, "System 1" neural generation model produces candidate next sentences. Semantic parses for each candidate are generated via few-shot parsing from GPT-3 and compared minimal world model to check consistency. Only candidates consistent with the world model state are incorporated into the final generation.
The prominent neural language models of today are single systems, with weaknesses akin those exhibited by the human System 1. For example, the cognitive reflection test (CRT) (Frederick) 2005) is classic probe of System 1 vs. System 2 reasoning humans. Participants answer a set of simple questions that have superficially compelling, but logically invalid, answers. These incorrect answers are often generated a first "gut" response (putatively, by System intuitive thinking): upon reflection, however, participants often realize that their responses were not logically or mathematically consistent (via more explicit System 2 reasoning). Consider the CRT problem on the left below:
Total cost in prompt
GPT-3 response
A ball and bat cost $1.10. The but costs one dollar more than the ball. How much does the ball cost?
$1.10
10 cents
$1.20
20 cents
$1.30
$0.30
$1.70
$0.70
Reading quickly, you might be tempted to say the ball costs 10 cents. Most participants give this response, in fact, especially if they are under time pressure or have limited attention (Kahneman 2013). Of course, if the bat is $1.00 more than the ball, and the ball costs 10 cents, then the total cost would be $1.20. The correct answer is that the ball costs 5 cents. Notably, in this and other classic CRT problems, GPT-3 (Brown et al. (2020) predicts the same "gut" response (prediction in red above; the table above shows that adjusting the price in the prompt also leads to similar effects; see Appendix Figure x for more CRT examples). GPT-3 appears vulnerable to the same sort of intuitive, unsystematic pattern recognition errors as humans- in this case, incorrectly subtracting one dollar from $1.10, without confirming that the answer satisfies each of the problem constraints.
Numerous studies have shown that engagement of System 2-style effort can help "override or inhibit default responses emanating from System 1" (Evans 2003), correcting inconsistent or un-systematic intuitive impulses. For example, when System 2 engaged by asking people to take more time to respond. people's 's accuracy improves the CRT task above (Kahneman 2013). It has been argued that integrating System processing could similarly improve AI systems (Goyal & Bengio 2020 Garcez& & Lamb 2020), and here explore this idea applied to neural sequence models.
In this work, we take inspiration from dual process theories to explore a neuro-symbolic generation system, wherein predictions from a neural model are treated as System 1 proposals, and logical, deliberative System 2 filters these proposals for consistency and soundness (see Figure [D. We further take inspiration from the fact that humans often do not need explicit supervision to reason about new problems or domains (e.g., see human evaluation task Section 4.2) and require that the System training, especially on example contradictions or commonsense violations. People can handle novelty by reconfiguring, rather than retraining, their internal models (Lake et al., 2017), and we strive to build machine systems capable of the same. System 2 model can help improve coherence and
2 module not need additional problem-spece
We show how a lightweight, consistency by adding small amount of symbolic reasoning.
We tackle two kinds of domains: text generation and instruction following. In both cases, we construct generative models over sequences by using a neural generation model to propose candidate generations and a symbolic world model that can accept or reject the generations and resample proposals if necessary. We first illustrate the approach by generating short stories based on the bAbl dataset (Weston al. 2015); this pedagogical, synthetic example illustrates how basic commonsense knowledge of objects, agents, and places can inform text generation model. We then test our

---

approach on rich, natural language vignettes based on CLUTRR (inha et al., 2019). focusing on ensuring consistency of family and interpersonal relationships. In both text generation domains, we interface between the explicit logical knowledgerreasonning of System 2 and generations of System I using a few-shot learning approach with state-of-the-an neural language models (GPT-3), which requires no additional training or fine-tuning. Even using off-the-shelff transformers and symbolic solvers, our dual-system model improves the consistency and coherence of text generations as measured by human judges. We test our approach also on instruction following, showing how goal- prediction models and execution models can easily be combined to achieve improved performance in low-data regimes. We show improvements over previous work in the gSCAN grounded compositional challenge (Ruis et al.) 2020); a dual-system model requires much less data to train than previous models, and achieves higher accuracy and stronger generalization. Overall, our findings indicate that neuro-symbolic, dual process models are a promising means of addressing longstanding problems of robustness and consistency in neural sequence models.
2
Related Work
Our approach incorporates semantic parsing (Liang 2016) as a component of a generative process, where neural generation is used in conjunction with parsing techniques. In our text generation experiments, we employ GPT-3 to perform few-shot semantic parsing without fine-tuning. Related work includes few or zero-shot semantic parsing using pre-training techniques and paraphrasing (Su & Yan| 2017: Herzig & Berant 2020). It also includes semantic parsing systems trained either without supervision (Liang et al. 2017: Mou et al. [017]] Muhlgay et al., 2019), or with synthetic language data (Marzoev et al.) 2020 Xu al. 2020b).
One popular technique for improving neural generations is generate-and-reramt wherein one model generates proposals and another reranks them. This broad approach has been used in image
generation (Ramesh et al. 202I), text generation (Holtzman et al. 2018 Shen et 2019 Deng
et al. 2020), dialogue systems (for control, coherence and safety (Welleck et al.] 2018
Smith et al.
2020, Nie et al. 2020 Xu al., 2020a)). and instruction following (Kurita & Cho 2020 Reranking is generally used to improve outputs with respect to relatively broad, holistic criteria. Here, our goal is to make generation robust to particular types of logical errors by pruning with respect to explicit symbolic constraints. Our approach can thus be considered closely related to techniques which employ explicit search to find generations satisfying particular logical constraints. Similar methods, such as guess-and-check or beam search pruning, have had success in neural program synthesis (Devlin et al. 2017 Nye et al. 2020).
Recent work in NLP has used template-based planning, in which a model generates text by first generating plan or skeleton, and filling in the missing words to produce naturalistic text (Xu et al.) 2018, Hua & Wang 2020). To generate stories, Martin et al. (2018) parses previous sentences into events and does planning in event space. Our work extends previous entity/relation/even planning in that the world model is not used for planning, but rather for post-checking candidate generations. Structured parsing of this type is also related to dialog tracking techniques such as slot-filling (Pieraccini et al. [992). In our work, fully compositional logical facts are extracted from utterances. It is therefore more closely related to systems which extract programs from dialogue, such as Andreas et al. (2020).
Recent work has also studied incorporating symbolic constraints into a neural decoding strategy in the context of natural language. Miao et al. (2019) introduce an MCMC-based inference-time propose-and-rejec strategy for satisfying constraints. They test on constraints such as paraphrase and grammatical error correction. Lu et al. (2020) introduces "NeuroLogic decoding," which uses logical constraints on neural language models produce generations which contain (or do not contain) required (or forbidden) keywords. In these works, the constraints are lexical or based on word/sentence similarity (and provided in the problem setup for et al. (2020)), whereas we study logical constraints on the world state decoded directly from observations or generations at test time. Other approaches for solving reasoning tasks end-to-end include Goyal et al. (202T). Serafini & d' ''Avila Garcez (2016), and Schlag & Schmidhuber (2018).
3

---

3 Integrating System and System 2
We introduce our dual dual-system approach using examples from the bAbl domain (Weston et al. 2015). which we also use to perform diagnostic experiments. Consider generating simple story involving people, places and objects, such as (from Figure(])
Daniel went to the garden. Mary traveled to the office. Daniel grabbed the apple.
A model tasked with generating such stories must juggle several simultaneous demands: staying on topic and maintaining consistency of style and other textural elements (for which people rely on System 1), as well maintaining consistency with previous statements and commonsense knowledge (for which people rely on both systems). Consider continuing the story with one of the following:
(a) Daniel went to the patio.
(b) Mary dropped the apple there.
Sentence (a) is reasonable; sentence (b) is not because it is Daniel, not Mary, who has the apple. During generation, how might model distinguish between these candidates? Perhaps well-trained neural language model could track constraints of these sorts. Neural language models to date, however, often violate these types of commonsense, hard constraints without large high-quality corpus or explicit training on detecting violations of commonsense (Sinha al. 2019).
We address this problem by decomposing text generation into two parts: candidate generation facilitated by deep neural networks and logical pruning process implemented via separate symbolic module. Consider again the example above. To ensure consistency, our model would extract from the text the features of the world that are subject the hard, logical constraints, such as the location of objects and who is holding them. These constraints can then be checked against an explicit representation of current state of the world. For sentences (a) and (b), the system would extract and go (Daniel, patio) and drop (Mary, apple), respectively. A minimal world model would track the state of the apple, such that it maintains apple heple.holfe holder Daniel (or equivalently, Daniel. inventory = [apple]). When such model is given parse of a candidate generation, drop (Mary, apple), the mismatch between the current state and the proposed change would cause a violation, and the candidate generation will be rejected.
The main steps of our general approach are illustrated in Figure [ generate proposals from a System 1 proposal model, extract facts with a f'at extraction model, and filter proposed generations by ensuring that they satisfy the constraints given by the extracted facts and the minimal world model.
System 1: Generation. We use neural sequence models to produce System generations. In text generation domains, we use large, pre-trained model that can be fine-tuned conditioned via short prompt to generate relevant text. Text sampled from the System 1 model will be treated as candidate utterances. which will be parsed and filtered by System 2 (described below). For the bAbl examples, we use GPT-3 as our System proposal model through few-shot prompting with 10 example bAbl stories as context, generating new story one candidate sentence at time.
System 2: Fact extraction. A fact extractor, or parser, is used to mediate between the System I candidate proposals and the minimal world model within System 2. In our text generation domains, we use a pre-trained GPT-3 model without fine-tume mun-ttnnng to perform parsing.
For bAbl, our prompt consist of an initial descriptive sentence "Please parse the following statements into commands. The available commands are pickup, drop, and go." " and a small set (< 10) of representative semantic parsing examples (input sentences; output correct parses, such go (Bob, roof)). The parse of each utterance is produced via few-shot prompting (Brown et al. 2020): the utterance is added to the end of the prompt, and the subsequent GPT-3 generation is interpreted as the target parse. We found that this simple parsing technique works well and could easily be applied to other barsing-based tasks, as in Shin et al. 1202I) The parsing prompts are reproduced in full in the Appendix. As discussed in Section S] for the gSCAN instruction following domain, fact extraction is performed with a learned goal location prediction model.
System 2: Minimal world model. We use a lightweight, incomplete description of the state of the world as a world model in each domain, e.g., commonsense information about the people. objects and locations (Figure []. The goal is not to track and verify all the possible information; instead, we aim for minimalism, capturing just a few commonsense (or application-critical) variables that we want to ensure are correct. The world model facilitates tracking of long-rang logical dependencies and logical consequences, especially those which are not readily decodable from surface forms. The

---

world model also lets us integrate rule- sale- world- horld-knowledge without retraining (and without the need for a large set of labeled examples).
For the bAbl examples, the minimal world model keeps track of the people, locations and objects introduced in the story so far (Figure [T.. This encodes constraints on possible actions related to human core knowledge competencies (objects, agents, places) present early in human development (Spelke & Kinzler 2007); specifically, person or object can only be in one place a time, an object can only be possessed by single person at a time, person cannot "go" to a room they are already in, and a person cannot pick up an object if it is in a different room. See the Appendix for details.
Search. At generation time, the interaction between System generation and System 2 parsing yields neuro-symbolic, guess- wress-and-check search strategy. In a text generation scenario, where text is sampled from the model, our dual-system model improves upon a naive, neural-only sampling method by using the System 2 model to reject candidate utterances which are incompatible with the current state. When a candidate is rejected, a new candidate utterance is sampled from the System 1 model, which is again checked by System 2. This process repeats until a candidate utterance is accepted by System 2 (i.e., the utterance is compatible with the world state). This procedure allows the model to effectively search the space of candidate utterances, guided by the logical constraints from the minimal world model. In this work, we use straightforward probabilistic sampling to illustrate that the approach works with even very simple search mechanism. We imagine that the search procedure could be further optimized by applying, for example, beam search or stochastic beam sampling.
Diagnostic bAbI experiments. We use
GPT-3 only:
GPT-3+ world model:
John went to the bedroom. John picked up the apple there. Mary took the apple there. Mary taaelled to the office.
John went to the bedroom.
Task #2 from bAbl as a diagnostic test for our neuro-symbolic dual-system model. As shown above, this task consists of synthetically-generaate short stories involv- ing people, places and objects, and ques- tions concerning the locations of objects in these stories. We investigate performance on both question answering (QA) tasks and story generation. For the QA tasks, we parse each sentence in the story to encode each fact into the world model and parse the final question to query the world model, returning the answer given by the world model. We compare with two alternative models (Table 3 in the Appendix): GPT-3 by itself and a dual-system baseline that uses a neural Natural Language Inference (NLI) model as its System 2. The NLI-based dual-system model generates 10 candidates from GPT-3 and selects the candidate with the highest predicted probability of entailment under the NLI model given the context. We use the RoBERTa MNLI model as our off-the-shelf neural NLI model (Liu et al. 2019). which operates as a System 2 that does not use additional problem-speciie -specffccc data or fine-tuming On 200 held-out tasks, our aPT--3-based "fact extractor" achieves 100% QA accuracy, far exceeding the performance of GPT-3 alone (29.0%) or GPT-3 generation with neural NLI scoring (32.5%; also see Table 3 in the Appendix). These results show that GPT-3 can be made to answer questions successfully when used for parsing with a world model, even when GPT-3 alone does not achieve high QA accuracy.
Johm picked up the apple there.
Mary mavelled to the office.
Daniel went back to the garden. Mary went the bedroom.
Daniel went back to the garden. Mary went to the bedroom. John went the hedrom.
Sandra went to the bedroom.
Sandra travelled the office.
Sandra went to the bedroom. Sandra travelled the office. Mary went back to the office. Where is the apple? A: office
Mary went back to the office.
Where is the apple? A: hedrom
Figure 2: Example bAbI stories generated by GPT-3 only (left) and our dual-system model (right). Logi- cally inconsistent lines are written in red text, and are removed from the story-so-far at generation time.
To test story generation, we use our GPT-3-based System I proposal model (few-shot prompted on 10 example stories) to sample new bAbl story, line-by-line. If generated utterance is inconsistent with the current state as indicated by the System world model, a new utterance is sampled from System 1 (repeating until a consistent utterance is sampled). Figure[|showe how the dual dual-system approach generates stories that mimic the statistical structure of bAbl storics, while remaining logically sound In contrast, GPT-3 alone was not able maintain logical coherence. In a set of 50 generated stories, all stories required at least one sentence to be resampled to maintain coherence, and over half of the generated sentences (53.1 1 1%) were rejected by our System 2 model to maintain logical consistency. These results demonstrate that equipping GPT-3 with a minimal world model produces logically coherent stories that mimic the textural structure of the bAbl domain. In the next section, we apply this approach to mimicking human-generated short stories in natural language.
Previous work has used domain specific entailmeet/conttaadiiion data to train reranking models Welleck al. 2018), however, this requires collecting dataset of domain- specific entailment and contradiction data.
5

---

4 Coherent Language Generation - CLUTRR
We apply dual-system approach to dataset of natural language using the CLUTRR dataset. CLUTRR contains human human-written written stories about people and their family relationships (see example in Figure 33. As with bAbl, CLUTRR was originally designed as Question Answering challenge; instead, we use to evaluate coherent language generation by querying models to generate complete CLstTRR-ttyle stories
complete partially-generatee stories. Our particular aim is produce stories with coherent and logically consistent family relationships. As above, our language generation setup consists of pre-trained language models acting our System proposer, minimal world model as System 2, and neural semantic parser (implemented via few-shot GPT-3 prediction) as bridge between the systems. We use human judgments to assess whether our neuro-symbolic, dual-system model produces more consistent and coherent stories relative to a baseline.
4.1 Model specification
As our System proposal model, we used pre- trained neural models to produce candidate gen- erations one sentence at time. We experi- mented with GPT-3 as our System model (which we used above for bAbl), but found generations too unreliable, often outputting the empty string. Instead, we used a BART model (Lewis al. 2019) that was fine-tuned on the CLUTRR training corpus. This model also gives us opportunity to compare against best-case neural "single-system" baseline, specifically fine-tuned on story data. To maintain a state of family relations, we constraint solver in our "System 2" to encode family relationships (e.g., child (x, y), spouse (x, z)) and check that the candidate utterances do not contradict the previous state- ments (e.g., person cannot be their own child or married to their sibling). We implemented the world model as a set of logical relations and constraints using the Z3 solver (De Moura & Bjømnrr 2008). For instance, uncle(x, y) = mchild (y, x). To check candidate utterance, we query the solver to determine if the set of constraints is satisfiable or if there is contradiction. The full set of constraints and other details can be found in the Appendix. We again used GPT-3 as our semantic parser, extracting parses for each candidate utterance via few-shot learning. This parsing approach worked well, even for the natural language in this domain. We observed that parsing with GPT-3 was more successful when the target parse was naturalistie, i.e., "Bob is Joe's 's father." rather than "father(Bob Bob, Joe)". The parsing prompt is reproduced in full in the Appendix.
Carol nice Sunday attmmoon They want a movie together a good time.
Carol related Justin Carol the grandmother Justin
Figure 3: Sample story from the CLUTRR dataset. Each story consists of sequence of human- generated sentences concerning family relation- ships. Adapted from [inha et al. (2019).
require that the parent of cannot also be the unele of x: For all x, y.
Proportion Dual- Syo--SSttmmm Donerttttonnn Genertion Profemmdd
1.--
Tracy weat to dinner with her daughter Shantel.
Ove Single Systain Systamm Clanenaten
They then went to the park afterwards.
Tracy loves her Aaron very much. He loves ber. too.
System 2 type
Harold took his grandson Aaron the
0.4
Minimal World Model
Which of the following sentences makes the most sense given the information above?
Harold's is son Aaron didn't 't go because was
afraid of animals.
Harold's daughter, Tracy, went him.
babasst CromitCondiion
Nodd
Figure 4: Example trial from CLUTRR human judgement experiment. Participants were in- structed to select which of two options makes the most sense given the prompt. One op-
Figure 5: CLUTRR human judgment experiment results. Bars denote proportions of dual-systen generations selected as making more sense over single-system generations, in each of four condi- tions. Error-bars denote bootstrapped 95% con- fidence intervals of the item means. The points denote means for each individual item in the exper- iment and are jittered horizontally for clarity.
tion was generated by the System 1 model only ("single-system"), while the other was generated by the dual-system model.

---

Table Statistics from CLUTRR story generation. We report the percentage of generations (on both a per-line and per-story basis) for which the System 2 world model did not detect an error. The dual-system model is able to detect many inconsistencies in the neural single-system generations, and most can be corrected by re-sampling new candidates (up to limit of ten).
% w/out error detected (per line)
% w/out error detected (per story)
single-system
dual-system
singlc-system
dual-system
(neural gen. only)
(neural gon-l model)
(neural gen. only)
(neural wonld model)
prompt from dataset prompt from model
82.8
97.1
60
96.1
71.9
96.3
36.4
93.5
4.2 Human judgments
We test our dual-system neural generation + world model method in its ability to generate stories that are deemed by naive human participants to be more naturalistic and coherent than those generated from the baseline models. Specifically, we asked participants to select which of two continuations made the most sense to them, where one continuation was generated from the neural model alone (single-system) and the other from a dual-system model (either the world model System 2 or the neural NLI System 2).
Participants (N = 101) were recruited on the crowd-sourcing platform Prolific and compensated $2 for the task (~-15 minutes, so roughly $8/hour). Participants gave informed consent, and the study was approved by MIT's s IRB. 21 participants were excluded for failing an instruction quiz, incorrectly answering more than one of five filler questions, or finishing the task too quickly. The data we collected contains no personally identifiable information or offensive content.
Participants.
Procedure. Participants began the experiment by reading a set of instructions and answering comprehension questions. On each main trial, participants were shown a prompt consisting of several sentences and were asked to choose which of two possible continuations made the most sense (an example trial is shown in Figure Participants were instructed that if name appeared multiple times within a trial, then it referred to the same person, whereas if a name appeared across trials, then it was not referring to the same person. For each trial, one continuation option was generated by the neural baseline, while the other was a dual-system generation. We selected generations from the neural only baseline that were rejected by the System 2 model in order to maximize the differences between the models' generations; thus, human judgments pertain to generations that the models disagreed on. Each participant performed between 20 and 26 trials.
only single-systed
Materials. Participants were randomly assigned to one of four between- caawwen-participant conditions, which varied according to the kind of prompt and the kind of dual-system model. The prompt was either generated from the model (up to the point of disagreement between System I and System models; "Prompts from model" condition) or taken completely from the length 4 CLUTRR systematic generalization test dataset ("Prompts from dataset" condition). To generate prompts for the "from model" condition, we took the first sentence of each story from the CLUTRR test dataset and generated subsequent prompt sentences from the dual-system model; sentences were generated until the two systems disagreed (i.e., System generated sentence that System 2 rejected), at which point the "rejected sentence" served as the neural only (single-system) baseline generation and the first resampled sentence that System 2 accepted served as the dual-system generation. Prompts were sampled to a maximum length of four sentences. The dual-system model shown to participants used System 2 based on either our constraint-based "world model" or the neural NLI baseline.
Table [ catalogs critical statistics from the stimulus generation process. We generated vignettes from the System 1 model and report the percentage of System 1 generations which are deemed correct by the System 2 model We also report the percentage of generations corrected by the System model (i.e., if System made an error, could System 2 fix it within 10 attempts??. We report these statistics on both a per-story and per-line basis. According to System 2, the System generation model makes a lot of errors (only 36.4% of stories and 71.9% of lines were error-free, in the "from model" condition). In most instances, re-sampling new generations yields stories that, according to
'For all System 1 generations, we used model temperature of 1.0. For the neural NLI baseline, we used 0.9 probability of contradiction as the cutoff for rejection. Our dual-systent model uses sampling budget of 10 System samples per sentence. Contradictions remaining after 10 samples considered dual-system emror.
7

---

Table 2: Accuracy on gSCAN splits. Models were trained on 5000 examples (only 2.5% of the gSCAN training data). See Appendix Table 4 for additional results.)
Conaliiie sulice sngeetee
gamm
NUX, REGHT,
TURS VALK
KULK TURS LEFT, MAX
Finil antio TURN LEFT, VALK, TURN
Test split:
single-system
dual-system
dev
71.7
83.3
Clock
random
57.2
74.7
yellow squares
68.1
81.3
red squares
64.9
78.1
Figure 6: Schematic of our dual-system approach to gSCAN. We train neural sequence model to predict both a distribution over action sequences, and distribution over target locations. At test time, we decode candidate action sequences from the model, execute them the gridworld, and only accept sequence that brings the agent to the predicted target location (shown in green).
novel direction
0.0
0.01
relativity
41.0
53.6
class inference
68.1
76.2
adverb (k=1)
0.0
0.0
adverb to verb
20.8
21.8
"From Heinze-Deml & Bouchacourt
2020)
System 2, no longer contain logical errors within budget of 10 samples (93.5% of stories and 96.3% of lines were error-free, respectively).
The human evaluation indicates that System 2 is indeed correcting genuine errors in the stories. As summarized in Figure 5 participants strongly preferred the dual-system neural generation + world model continuations in comparison to the neural only single-system continuations (proportion preferring dual-system 0.84; bootstrapped 95% confidence interval [0.77, 0.89] and 0.79 [0.77, 0.89] for the "from dataset" and "from model" prompt conditions, respectively). The dual-system approach, however, did not improve generation quality when the System 2 was based on off-the-shelf neural NLI model (Proportion preferring dual-system 0.51; [0.40, 0.64] for "from dataset"; 0.58 [0.48, 0.68] for "from model"). Thus, when using minimal world model, the dual-system approach dramatically improves logical consistency without any need for additional training or fine-tuning. People clearly prefer suurosyymboli generations from the dual-system model over purely neural generations from a single-system model
Results.
Grounded Instruction Following
The dual-system approach offers general-purpose means of improving upon generative, neural sequence models by incorporating logical constraints. To highlight its generality, we examine how the dual-system perspective can be deployed in very different domain: grounded instruction following. In Heiinze-Dmml & Bouchacourt (2020), a learned target location predictor was used to increase the accuracy of neural action sequence generation model. Here, we show how to increase performance further by enforcing consistency between the target location predictor and the action sequence generator in our dual-system framework.
We use the gSCAN benchmark (Ruis et al. 2020). a recently proposed grounded instruction following dataset designed to measure compositional caatoonnnggeeerrai generalization in neural systems. Given an initial gridworld state and an instruction, c.g., "walk to the big square," an agent must predict the sequence of low-level actions which achieve the goal, e.g., "TURN LEFT, WALK, TURN LEFT, WALK" (See Figure 6 The dataset contains several test splits, each testing different aspects of compositional generalization.
Our model builds on Demil & Bouchacourt (2020) by using an LSTM to predict the correct action sequence and target location. Given a command e and an initial gridworld state 8, the neural network defines two distributions: a distribution over action sequences (ala,, s) and a distribution over target grid locations (ooc (I)c, s). FHeinze-Deml & Bouchacourt (2020) showed that when these distributions share parameters, using location prediction an auxiliary loss improves the accuracy of the action sequence prediction model. We can further exploit these two models by noticing that when predicted action sequence is not consistent with predicted target location, then either the action sequence or the target location must be incorrect. Since the target location is much simpler to predict,
*We note that the effect size of our approach depends upon the prevalence disagreements between the single-system and dual-system models. As reported in Table we find that the prevalence of disagreements is at least 14% 57%, depending on the data regime. Therefore, the model predictions do often differ. resulting in large overall effect size.
8

---

and thus much more likely to be correctly predicted, if a predicted action sequence is not consistent with the predicted target location, then the action sequence is most likely incorrect. Our dual-system framework can use this property to increase action sequence prediction accuracy. Consider the initial state and command in Figure Our model predicts candidate action sequences, and also predicts that the most likely target location is the grid containing the bigger yellow square (highlighted in red). The model then executes the candidate action sequences, and only accepts a sequence which results in the agent standing in the target location.
In the language of our dual-system approach, we treat the distribution over actions qa(ale, s) as our System proposal model. The distribution over target locations Qloe (1lc, s) serves as fact extractor model, which extract a location constraint deterministic gridworld execution model [ so) + s,, which takes a state and action and predicts the resulting state. At test time, we first extract the predicted location as | arg maxy qoo (c). We then search through the possible action sequences from qa (-Ic), conditioned on agreement with 1. In our experiments, we use a sample- based search with a maximum budget of 50 samples. We trained models on random subsets of the gSCAN training set of varying sizes: 5000 datapoints, 8000 datapoints, and 20000 datapoints (2.5%, 4% and 10% of the original training set, respectively).
As a minimal world model, we use a
The results show that the System 2 execution model improves performance without the need for any additional training (see Table 2 for results training on 5000 examples). In contrast the single-system model, the dual-system model allows for sampling many candidate action sequences from the neural network, accepting only consistent sequences. This guess-and-chece approach greatly increases the evaluation accuracy, improving upon prior work on gSCAN, particularly in low-data regimes.
Results.
Limitations
In its current form, our approach is most useful in domains where naturalistic, learned generation is necessary and where a small number of mission-critical logical constraints can be explicitly articulated. Our system will be less useful when constraints creative domains such as writing poctry) or when there are many constraints, since the minimal world model must be hand-engineered Enforcing strict constraints may also pose risks: if the constraints are not only logical but cultural, they may be harmful if misapplied. However, these constraints must be articulated explicitly in symbolic model, and are thus easier identify and correct.
more difficult to articulate (c.g..
The current few-shot parsing technique may also suffer from a limited capacity. For more complex domains, the number of examples required to specify the desired parsing behavior may be too large (i.e., they may not fit in the input window) or too complex for model to perform parsing accurately. While some tasks may not be suitable, the complexity of the world model need not necessarily increase hand-in -nddddd with the complexity of the application domain. A dual-system model will be most successful when tracking just few critical variables (e.g., tracking consistency in family relations, as in our experiments, or tracking scheduling constraints when discussing a team plan).
A promising direction for future work is to incorporate learning into the System 2 world model. Currently, the minimal world knowledge that exists in System 2 can be easily modified, but changes must be made by hand. Improvements would come from automatically learning and updating this structured knowledge, possibly by incorporating neuro- neuro-symbolic learning techniques (Ellis et al. 2020 Mao et a (019), or other neuro-symbolic integration work such as Tsamoura et al. (202
Michael & Valiant (0008).
Learning could improve our dual-system approach in other ways, c.g., by training neural module to mimic the actions of a symbolic System 2. The symbolic System 2 judgments could be used as a source of supervision; candidate utterances rejected by the symbolic System 2 model could be used as examples of contradictory sentences, and accepted utterances could be used as examples of non- contradictory statements. This oversight could help train a neural System 2 model capable of more subtleties than its symbolic counterpart, especially in domains where labeled examples are otherwise unavailable. This approach may also help us understand aspects of human learning, where certain tasks that require slower, logical reasoning can be habitualized over time and tackled by faster, more intuitive reasoning.

---

Recent work et al. 2021) has shown that large pre- wre-trained neural models learn to approximately represent certain types of structured semantic information. However, not yet clear how represen- tational fidelity translates to logical coherence during generative tasks. Our current approach allows us to explicitly fix logical errors in generation, which may ultimately be caused by representational errors. Understanding how we might leverage our approach to improve the representation of struc- tured knowledge within neural models a promising direction for future work, which could lead to increased generation consisteney and coherence.
7 Conclusion
Inspired by dual process theories from cognitive science. we combine the respective strengths of neural and symbolic approaches build more robust models that can more effectively incorporate domain knowledge. For language generation, we showed that equipping neural generation with minimal symbolic world model increased language coherence and consistency. For grounded instruction following, we showed that requiring test-time consistency between predicted action sequences and goal locations led to improved performance, especially in low-data regimes. Our neuro-symbolic approach can readily be applied to other domains and types of prior knowledge, as lightweight way of improving the coherence and consistency of powerful neural sequence models.
This paper just scratches the surface of how structured knowledge can make neural systems more robust; we hope inspire further work into neuro-symbolic systems which possess the robustness and commonsense necessary for human-level intelligence.
Acknowledgments
We thank Laura Ruis, Jacob Andreas. Yewen (Evan) Pu, Joe O' O'Connor and Guy Davidson for helpful comments on an earlier version of this manuscript. MN is supported by NSF Graduate Research Fellowship.
References
Andreas, J., Bufe, J., Burkett, D., Chen, C., Clausman, J., Crawford, J., Crim, K., DeLoach, J., Dorner, I.., Fisner, J., et al. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics, 8:556-577 571, 2020.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. arXiv preprini arXiv:2005. 14165, 2020.
De Moura, L. and Björner, N. Z3: An efficient smi solver. In Tools and Algorithms for the Construction and Analysis of Systems, pp. 337-340. Springer, 2008
Deng, Y., Bakhtin, A., Ott, M., Szlam, A., and Marc' Aurelio Ranzato. Residual energy-based models of text generation. In International Conference on Learning Representations (ICLR), pp. 1-18, 2020.
Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed, A.-r,, and Kohli, P. Robustfill: Neural program learning under noisy i/o. ICML, 2017.
Ellis, K., Wong, C., Nye, M., Sable-Meyer, M., Cary, L., Morales, L.,, Hewitt, L., Lolar-Lezama, A., and Tenenbaum, B. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv preprint vxXiv:200608838 2020.
Evans, J. S. B. In two minds: dual-process accounts of reasoning. Trends in cognitive sciences, 7(10): 454-459, 2003.
Frederick, S. Cognitive reflection and decision making. Journal of Economic perspectives, 19(4):
25-42, 2005.
Garcez, d. and Lamb, L. C. Neurosymbolic ai: The 3rd wave. arXiv preprint urXiv:2012.05876, 2020.
10

---

Goyal, A. and Bengio, Y. Inductive biases for deep learning of higher-level cognition. arXiv preprint arXiv:2011.15091, 2020.
Goyal, A., Didolkar, A., Ke, N. R., Blundell, C., Beaudoin, P., Heess, N., Mozer, M., and Bengio, Y. Neural production systems. CoRR, abs/2103.01937, 2021. https//tts//// org/ abs/ 2103. 0193701937
Heinze-Deml. C. and Bouchacourt, D. Think before you act: A simple baseline for compositional generalization. arXiv preprint wrXiv:2009.13960 2020.
Herzig, J. and Berant, J. Span-based semantic parsing for compositional generalization. arXiv preprint arXiv:2009. 2020.
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., and Choi, Y. Learning to write with cooperative discriminators. arXiv preprint arXiv:/805.000 2018.
Hua. X. and Wang, L. Pair: Planning and iterative refinement in pre-trained transformers for long text generation. In EMNLP, 2020.
Kahneman, D. Thinking, fast and slow. kindle ed, 2013.
Kurita, and Cho, K. Generative language- policy in vision-and- and-aauage navigation with bayes' rule. arXiv preprint urXiv:2009.077833 2020.
Lake. B. M. and Murphy, G. L. Word meaning in minds and machines. arXiv preprint MXXi:2008.011600 2020.
Lake, B. M., Ullman, D., Tenenbaum, J. B., and Gershman, S. J. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:E253, 2017.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, I. Bart: Denoising sequence-to-sequencce pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13460 2019.
Li, B. Z., Nye, M., and Andreas, Implicit representations of mcaning in neural language models. arXiv preprint arXiv:2106.0000 00737, 2021.
Liang, C., Berant, J., Le, Q. V., Forbus, K., and Lao, N. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 23 23-33, Vancouver, Canada, coli. wuw.uniiie saar-sarlaaddd.co pttp://pppppppp//ppeeepepptt 10t0///711111......0 pdf
2017. URL
Liang, Learning executable semantic parsers for natural language understanding. Communications of the ACM, 59(9):68-76, 2016.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 2019.
Loshchilov, and Hutter, F. Decoupled weight decay regularization. arXiv preprint arX1v:1711.05101 2017.
Lu, X., West, P., Zellers, R., Bras, L., Bhagavatula, C., and Choi, Y. Neurologic (eco supervised neural text generation with predicate logic constraints, arXiv preprint arXiv:2010.12884, 2020.
Mao, J., Gan, C., Kohli, P., Tenenbaum, J. B., and Wu, J. The neuro-symbolic concept learne: Inter- preting scenes, words, and sentences from natural supervision. arXiv preprint arXiv: 1904. 12584, 2019.
Martin, I.. J., Ammanabrolu, P., Hancock, W., Singh, S., Harrison, B., and Riedl, M. O. Event representations for automated story generation with deep neural nets. In AAAI, 2018.

---

Marzoev, A.. Madden, S., Kaashoek, M. F., Cafarella, M., and Andreas, J. Unnatural language processing: Bridging the gap between synthetic and natural language data. arXiv preprint arXiv:2004. 13645, 2020.
Miao, N., Zhou, H., Mou, L.., Yan, R., and Li, Cgmh: Constrained sentence generation by metropoolis-hattings sampling. In AAAI, 2019.
Michael, L. and Valiant, L. G. A first experimental demonstration of massive knowledge infusion. 2008.
Mou, I.,, Lu, Z.., Li, H., and Jin, Z. Coupling distributed and symbolic execution for natural language queries. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2518-2522 PMLR, 06-11 Aug 2017. URL http: htttt::/proceedinng. mlr wmt..press/////noo//... hhtml. html
Mublgay, D., Herzig, J. and Berant, J. Value-based search in execution space for mapping instructions to programs. pp. 1942-1954, 01 2019. doi: 10. 10.18853/v1/N19-111 1193,
Nie, Y., Williamson, M., Bansal, M., Kiela, D., and Weston, J. I like fish, especially dolphins: Addressing contradictions in dialogue modelling. arXiv preprint arXiv:2012. 13391, 2020.
Nye, M. I., Solar-Lezama, A., Tenenbaum, J. B., and Lake, B. M. Learning compositional rules via neural program synthesis. arXiv preprint arXiv:2003. 05562, 2020.
Pieraccini, R., Tzoukermann, E., Gorelov, Z., Gauvain, J.-L., Levin, E., Lee, C.-H., and Wilpon, J. A speech understanding system based on statistical representation of semantics. In (Proceedings) ICASSP-92: 1992 IEEE International Conference Acoustics, Speech, and Signal Processing, volume 1, pp. 193-196 vol.1, 1992. doi: 10. 10.1099ICASSP.1122225999
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. 2021.
Zero-shot text-to-image generation. arXiv preprint
Ruis, L., Andreas, J., Baroni, M., Bouchacourt, D., and Lake, B. M. A benchmark for systematic generalization in grounded language understanding. arXiv preprint arXiv:2003.05161, 2020.
Schlag. and Schmidhuber, J. Learning to reason with third-order tensor products. CoRR,
abs/181 1211181,.12143, 2018. URL http:://axiit hrg. btrgg/bbs/in 1811. 12143
Serafini, and Avila Garcez, A. S. Logic tensor networks: Deep learning and logical reasoning from data and knowledge. CoRR, 0bs/16606004433 2016. http://////rr abs///b////// /166 04422
Shen, S., Fried, D., Andreas, J.. and Klein, D. Pragmatically informative text generation. arXiv preprint #xXiv1904.003300 2019.
Shin, R., Lin, C. H.. Thomson, S., Chen, C., Roy, S., Platanios, E. A., Pauls, A., Klein, D., Eisner, J., and Van Durme, B. Constrained language models yield few-shot semantic parsers. arXiv preprint arXiv:2104.08770 2021.
Sinha, K., Sodhani, S., Dong, J., Pineau, J., and Hamilton, W. L. Clutrr: A diagnostic benchmark for inductive reasoning from text. arXiv preprint arXiv: www.:i90.0000 19177.617.. 2019.
Smith, E. M., Gonzalez-Rine Rico, D., Dinan, E., and Boureau, Y.-L. Controlling style in generated dialogue. arXiv preprint arXiv:2009. 10855, 2020.
Spelke, F. S. and Kinzler, K. D. Core knowledge. Developmental Science, (0(1):89-96 2007.
Su, and Yan, Cross-domain semantic parsing via paraphrasing. arXiv preprint (rXivv1004.05977 2017.
12

---

Tsamoura, E.. Hospedales, T., and Michael, L. Neural-symbolic integration: A compositional perspective. Proceedings of the AAAI Conference on Artificial Intelligence, 35(6):5011-506) May 2021. htt.pttss:///oss. aaai aaai org/ index. phtp:/hAT//articce//ieeee/ hwtew///ww/ww/////// 16639
Welleck, S., Weston, J., Szlam, A., and Cho, K. Dialogue natural language inference. arXiv preprint arx0v:181..00671 2018.
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., and Mikolov, T. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05668, 2015.
Xu, J., Ren, X., Zhang, Y., Zeng, Q., Cai, X., and Sun, X. A skeleton-based model for promoting the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4306-4303 Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v 1462.......... URL
coherence among sentences in narrative story generation. In Proceedings the
https://ver //www aclweb. org/ wnwhaantoootyooaaaaeee D18- 1462
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., and Dinan, E. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079, 2020a.
Xu, S., Semnani, S. J., Campagna, G., and Lam, M. S. Autoqa: From databases to qa semantic parsers with only synthetic training data, arXiv preprint arXiv:2010.04550 2020b.
13


Reviews:
[{"review_id": "P-tl6lJp_hW", "reviewer": "Reviewer_Ms3t", "summary": "The authors take some natural language text `X`, and use GPT-3 to output a machine-readable piece of text representing the world state `S`. Next, they get a model to output a set of candidate next sentences `Yᵢ` for `X`. Another GPT-3 model is used to output a machine-readable state-change `Δᵢ`. Symbolic methods are used (e.g., Z3 with hard-coded rules) to determine whether `S` and `Δᵢ` are consistent, and finally the `Yᵢ` that are consistent are put forward as states. The claim is that this will produce better, logically coherent text.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is appropriate for NeurIPS. The paper is organised and presented well.\n\n# Validity: \n\nOverall, there are not that many technical parts of the paper in which I could check for validity. On a less formal level, I don't quite buy that 'using Z3' is the same as what Kahneman means by 'System 2’. \nIn the human-judgment experiment, I take issue with the 'single-system' sample being a sample which was _rejected_ by 'System 2'. This is not comparing 'System 1 + 2' with 'System 1'; it is comparing 'System 1 + 2' with 'System 1 - 2'.\n\nI also do not think the experiment in section 5 is valid:  why does the LSTM need to produce the sequence of command-words if it is producing a heat map on where the goal is and a symbolic check that the sequence of words finishes on the goal? Once you are doing the symbolic check and know the goal position you can easily generate the command-word list symbolically. This is not a valid experiment for testing the claim that a system 2 helps with the prediction, if system 2 doesn't need system 1.\n\n# References: \n\nI think that including all the cognitive science is a distraction from the core of the paper.\nThe results of using a 2-system approach should stand on their own.\nIt also reminds me of the general idea of 'autoformalization' as described, e.g., in Szegedy at CICM 2020.\n\n# Contribution: \n\nThe contribution is to use generate-and-rerank, but with the reranking performed by composing hybrid language model + symbolic checker. This is not a significant enough contribution and the problems tackled are easy. The symbolic checker's rules had to be manually coded and so I am not convinced that this approach is going to scale.\n\nThe notion of having a two-system-style model is also very common.\n\nThere has also already been a lot of work on using GPT to generate machine-readable text.\n\nThere has also already been work on using language-model-generated, machine-readable text to guide a search, for example, for lemma selection in automated theorem proving. So to me it is not a great leap to join these pieces together. The challenge in this space is in how to remove the human dictating the rules of system 2.\n\nThe experiments do not compare the performance of their system to any other techniques, e.g., Welleck 2018. It's only compared to a baseline (which I argue is unfair for both experiments).\n\n# Details\n\n- This sounds exactly like generate-and-rerank, so how is yours different to the related work?\n\n- Related work should include a few sentences on using language models for logical reasoning; e.g., original GPT3 paper, Lample ICLR 2020.\n\n- _Thinking Fast and Slow_ is not an appropriate citation. Surely Kahneman has a specific cogsci paper or citation on the same specific point you are trying to make? For example, the ball+bat study is an actual study that can be cited.\n\n- Lots of these citations are arxiv preprints. I know that the field is moving so fast that you need to cite preprints. But in some cases there is a published version of the work, e.g., the preprint (Lake, Murphy 2020) says \"In press at Psychological Review\". Maybe at the time of writing, this hadn't been published yet, but really you should be citing published, peer-reviewed papers if you are using the citation to support an argument.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are addressed in the paper:\n\n- the constraints must be constructed manually (can’t be learned) and so you can’t scale or change domains easily;\n\n- because the formal translation is done with a language model, there is limited capacity. So it might only work with a few variables.\n\nSocietal impact is addressed: enforcing the constraints on ’System 2’ might include cultural constraints rather than logical constraints (e.g., forbids poetic language).\nThe authors also claim that having an explicit set of rules in ’system 2’ makes it easier to find these harmful assumptions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {"ethical_concerns": "The only place where an ethical issue may arise is in the use of humans to assess the output. But the authors state in the paper that the participants were renumerated and gave informed consent so there are no ethical issues."}}, {"review_id": "D67dcWRlXP", "reviewer": "Reviewer_jmVT", "summary": "Inspired by the philosophical literature, this paper proposes to view human reasoning as consisting of two Systems: System 1 represents the \"reflexive\", everyday part of our reasoning, which makes sure we can do most of our tasks without having to deliberate about them. System 2 represents out \"deliberative\" part, which we activate when something surprising happens and we have to react in a different way than we are used to, and often involves some sort of logical reasoning. This view is popularized by Kahneman in his book \"Thinking, fast and slow\", and it provides an appealing way of separating the two main schools of thought in AI: the classical symbolic approach (System 2), and the currently very popular ML approach (System 1).\n\nThe authors of this paper observe that pre-trained models such as GPT-3 appear to lack System 2 abilities, and they propose an approach where they build a \"minimal world model\" for a specific domain using a logic. They use GPT-3 to rewrite problems to this logical language, and use a solver to verify whether the output from the pre-trained model is consistent. If not, the output is rejected and the next best one is tried.\n\nThey evaluate their approach on text question answering and generation (bAbI) and instruction generation (gScan), and show that their approach can increase the coherence and accuracy.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written, and the proposed approach is explained well. I could understand all details and the experiments make sense to me. I also think the topic is of high interest to the AI community. It is clear that pre-trained models have limitations, and the current paper addresses an important one, namely that they often make mistakes that we humans wouldn't make.\n\nMy main issue with this paper is that I am not convinced about the significance of the results and the usability of the framework in more general settings. I see the following problems:\n1) Inference speed. Since the consistency check is not differentiable, it seems that one is required to run this at each inference step. When the complexity of the domain grows, computing whether an output is consistent according to the underlying logic can become a bottleneck, and I would have liked to see some more details on this.\n2) Developing the logic. It seems to me that this approach will only work if the minimal world model is specified correctly. This works for the simple toy domains discussed in this paper, but for any realistic domain, developing a consistent logic is a major undertaking, and I feel the current approach sidesteps this problem by taking very simple domains. I therefore don't see how this approach can be useful in a more general setting.\n3) Empirical results. I feel that the main reason the authors managed to get good empirical results is that they developed domain-specific logics and it is quite simple to rewrite sentence from the dataset to this logic (I guess one could almost handcode them). Therefore I wasn't particularly impressed by the results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see problems here", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ArJ-WIPQXEW", "reviewer": "Reviewer_CnYk", "summary": "This paper tries to improve neural network reasoning by combing the deep-learning-based System 1 and logical-reasoning-based System 2. The proposed approach uses an attached logical reasoning module to check the consistency of the predicted text by GPT-3. In order to connect neural NLP with logical reasoning, it pre-trains a GPT-3/BART model to extract ground atoms from both the input and predicted natural language sentences, and then a domain-specific language (DSL) based reasoning module searches in the pre-defined knowledge base to check whether the extracted groundings are logically consistent. The top-ranked consistent result is returned as the final output.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Separating end-to-end neural reasoning into multiple-stage processing that involves both neural computing and logical calculi seems to be a promising direction for AI. This work presents a straightforward and effective way to address natural language reasoning tasks. The main idea in this paper is to extract logical atoms from text and then uses a pre-defined knowledge base to constrain the predicted result. The advantage of this work is that the pure deep learning-based prediction can now be filtered by the symbolic constraints. The symbols extracted by the pre-trained GPT-3 model has shown their effectiveness empirically in the presented experiments.\n\nThere are few points that are not clear to me. The symbol extraction requires both supervised pre-training and a well-defined DSL knowledge base. The paper, at least in its main text, does not include any information about how much-supervised data is required and how to design such a  DSL knowledge base in general, e.g., for tasks other than the ones in this paper. Thus, I cannot tell whether the proposed approach is scalable and general enough in practice. Another problem is that this work heavily depends on GPT-3, which is not an open AI model. Even for GPT-3, its performance on the CLUTRR dataset is still questionable, so I wonder how applicable is the proposed approach. The logical reasoning module is only used for consistency checking, while the System 2 for \"logical reasoning\" is actually done by GPT-3, which is not \"logical\" at all. Furthermore, the \"reasoning\" cannot be improved as the System 2 GPT-3 is not updated anymore.\n\nThis paper has clearly missed many references, especially those neuro-symbolic systems rooted in the classic symbolic AI. Many works in this area have proposed to improve deep learning via symbolic consistency checked by a systematic logical inference engine. For example:\n- R. Manhaeve, et al., Deepproblog: Neural probabilistic logic programming, NeurIPS. 2018.\n- W. Dai, et al., Bridging machine learning and logical reasoning by abductive learning, NeurIPS. 2019.\n- R. Evans, et al., Making sense of raw input. Artificial Intelligence, 299:103521, 2021.\n\nThe above related works aim at training neural networks/symbolic models by including a symbolic System 2 module without (strong & data hungry) pre-training, which I personally think are stronger than this work. Meanwhile, similar ideas (i.e., using symbolic rules to constrain the machine learning models' outputs to get reasonable predictions) have been explored even earlier. For example:\n- I. Tsochantaridis, et al.,. Support Vector Learning for Interdependent and Structured Output Spaces, ICML, 2004.\n- S. Mei, et al., Robust RegBayes: Selectively Incorporating First-Order Logic Domain Knowledge into Bayesian Models, ICML, 2014.\n\nDetailed questions:\n- Line 77, \"few-shot learning approach\" and \"requires no additional training\", which one is correct? From what I can tell, the paper does require some supervision for pre-training, which is still \"training\".\n- Does the GPT-3 training data contain bAbI text?\n- How to ensure GPT-3 outputs the predicate logic format defined by the DSL in this paper?\n- How many pre-train data for semantic parsing?\n- Line 166: \"could easily be applied to other tasks\". How easy is it? does it need pre-training or the model learned here can be directly adapted to other domains?\n- The compared NLI does not uses external logic constraints and pre-training.\n- Why not fine-tune GPT-3 on CLUTRR directly instead of using another BART model?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The use of GPT-3 limits the applicability of the presented approach.\n2. The proposed system can only make inferences and does not learn.\n3. The paper is missing some very related works.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "5QlOsX64dH", "reviewer": "Reviewer_2tE8", "summary": "The paper proposes a neuro-symbolic model to improve coherence and consistency in neural language generation models. In particular, the authors use a neural language model to generate samples, and reject invalid ones using a world model built on context sentences and the parsed results on the generated sentences, and resample if necessary. In a set of experiments (bAbI, CLUTRR, and gSCAN), the authors show that the proposed system could improve coherence and consistency in the generated sequences. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* I actually quite like the idea of combining neural models and logic systems to improve performance, especially in achieving greater human-likeness. The proposed idea, *i.e.*, using a symbolic workflow to reject and resample, is quite intuitive, and the reported results are encouraging.\n* I also like how the authors present the experiments, detailing the experimental protocol, the human study procedure, and a lot of other small design choices.\n\nCons:\n* The way to combine the two systems is too simple: without further integration, success of the separate processes is completely dependent on luck. You could neither revise / correct the errors, nor bound the number of trials. If you are very unlucky, you might get trapped in a region with no valid generations, or you need a very large number of resamples to get a valid one.\n* The world model requires ad-hoc design. This could be a potential issue in scalability. When a problem domain becomes relatively large, it would be really hard to track everything. \n* The parsing process (and potentially the world model building process) involves additional neural processing. How to ensure coherence and consistency in these parts? Will it be a chicken-and-egg problem? What if the parsing / world-model-building process gives you wrong answers? \n\nIn summary, I like the idea and the general methodology presented in this work. However, it also bears some limitations from Good-Old-Fashion-AI, most of them discussed in the Limitation section. Despite of that, I think the general methodology should be encouraged but I would also love to sync with other reviewers just in case I miss some points / issues.\n\nPost rebuttal: I have read the authors' rebuttal and other reviews. The authors basically agree that the limitations exist and they don't have a very nice way to solve them for now. Therefore, I keep my initial rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper borrows insights from neuroscience and psychology, exploring how symbolic approaches can be combined with a large-scale neural language models to improve responses because systems like GPT3 make mistakes that humans would not. This work can be viewed in several different lights: evidence that the system1/system2 insight is useful for ML systems and should be explored more; further evidence that hybrid ML and symbolic approaches are promising; a way to improve (increasing coherence and accuracy) black-box ML systems for deployment. Indeed the paper speaks to many of these views, but the reviewers felt that the paper does none of them convincingly.

The reviewers flagged several issues including: speed of the inference step, that this is not really a compelling instantiation & evaluation of the sys1/sys2 concept, and the consistency in the parsing process. One reviewer was concerned that the work builds on and uses a closed source entry (GPT3) for both practical and ethical reasons. Several reviewers flagged useful related works whose discussion will strengthen the work. In the end all four reviewers---all working in related areas---all agreed on rejection. The main concern being scalability and generality of the proposed approach due to the need to hand-design design world models for each domain. The reviewers found the lack of general principles & advice offered for this critical step unconvincing. The experiments presented did not do enough to convince on this critical question. Further, the discussion revealed the reviewers were not satisfied with this as a first step---that too much was being left to future work---and that non-trivial revisions are needed. Besides these concerns the work is polished and he author response comprehensive. On balance, this work was close but just below the bar.

 
Decision: Accept (Poster)