MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
Authors: Zayne Rea Sprague
Xi Ye
Kaj Bostrom
Swarat Chaudhuri
Greg Durrett
Venue: ICLR 2024 spotlight (2024)
Keywords: Large Language Models
Chain-of-Thought
Textual Reasoning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce MuSR, a new dataset for testing LLMs' abilities to do complex, structured reasoning based on generated narratives.

Abstract: While large language models (LLMs) equipped with techniques like chain-of-thought prompting have demonstrated impressive capabilities, they still fall short in their ability to reason robustly in complex settings. However, evaluating LLM reasoning is challenging because system capabilities continue to grow while benchmark datasets for tasks like logical deduction have remained static. We introduce MuSR, a dataset for evaluating language models on multistep soft reasoning tasks specified in a natural language narrative. This dataset has two crucial features. First, it is created through a novel neurosymbolic synthetic-to-natural generation algorithm, enabling the construction of complex reasoning instances that challenge GPT-4 (e.g., murder mysteries roughly 1000 words in length) and which can be scaled further as more capable LLMs are released. Second, our data instances are free text narratives corresponding to real-world domains of reasoning; this makes it simultaneously much more challenging than other synthetically-crafted benchmarks while remaining realistic and tractable for human annotators to solve with high accuracy. We evaluate a range of LLMs and prompting techniques on this dataset and characterize the gaps that remain for techniques like chain-of-thought to perform robust reasoning.

Published as a conference paper at ICLR 2024
MUSR: TESTING THE LIMITS OF CHAIN-OF-THOUGHH WITH MULTISTEP SOFT REASONING
Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, Greg Durrett
Department of Computer Science The University of Texas in Austin za zayne@ www.euteeaaaa..mmm edu
ABSTRACT
While large language models (LLMs) equipped with techniques like chain-of- thought prompting have demonstrated impressive capabilities, they still fall short in their ability reason robustly in complex settings. However, evaluating LLM reasoning is challenging because system capabilities continue to grow while benchmark datasets for tasks like logical deduction have remained static. We in- troduce MuSR, a dataset for evaluating language models on multistep soft reason- ing tasks specified in natural language narrative. This dataset has two crucial fea- tures. First, it is created through novel neurosymbolic synthetic- tontica-t-natrrall gen- eration algorithm, enabling the construction of complex reasoning instances that challenge GPT-4 (e.g., murder mysteries roughly 1000 words in length) and which can be scaled further as more capable L.MSs are released. Second, our dataset instances are free text narratives corresponding to real-world domains of reason- ing: this makes it simultaneously much more challenging than other synthetically- crafted benchmarks while remaining realistic and tractable for human annotators to solve with high accuracy. We evaluate a range of LLMs and prompting tech- niques on this dataset and characterize the gaps that remain for techniques like chain-of -ftthughh to perform robust reasoning
INTRODUCTION
A great remaining challenge for large language models (LLMs) is the ability to do reasoning and
planning (Valmeekam et al.] 2023] [aang al.. [023] Dziri et al. 2023) Numerous methods have
been proposed to augment models' capabilities
this front, including prompting strategies like
chain-of-thought (Wei et al. 2022), integration with tools (Schick et al. 2023 Lyu et al.) 2023 Ye
et al. 2023), and embedding models in search loops (Bostrom et al. 2022 Creswell et al. 2023
Do these approaches suitably address the shortcomings of LLMs? This is difficult to measure. Math reasoning tasks can be approached in two-stage fashionn (Gao et al. 2022 Ye et al. 2023)-
the LLM translates the problem into a formal specification which is then solved with conventional
tools. Other datasets like RaleTakers (Clark et al. 2020) and CLUTRR (Sinha al. 2019) solvable with rule-based systems (Kazemi et al. 2023a Ye et al. 20233] Poesia et al. Finallll datasets like SocialIQA (Sap al. 2019) or StrategyQA (Geva al. (021) that involve more nuanced commonsense are often structurally simple (i.e., only involve 1-2 steps of reasoning). What
lacking a benchmark involving both sophisticated natural language and sophisticated reasoning.
In this work, we present MuSR: Multistep Soft Reasoning, a dataset focused on tasks involving reasoning based on text narratives. The narratives in our dataset are hundreds of words long and present evidence in ways that require commonsense knowledge to unpack. Then, when all of the evidence assessed, coming
final answer requires "System 2"'style deliberation, which takes a different form for each domain of interest. The domains we address here, murder mysteries, object placement, and team assignment, involve commonsense reasoning about physical (Bisk et al. 2020) and social situations (Sap et al. 2019) ineory-of-minnd and more. Crucially, these types of reasoning arise naturally from text descriptions of cach problem.
Projecc website be found at httttts:/ggtthhbt http://///gtttt.o.ooo///eeee/eee httmm//aayyn-SSpprruuuesasss

---

Published as a conference paper at ICLR 2024
Sample a problem...
Build reasoning tree...
Generate narrative...
Recursively sample from an LLM to establish reasoning behind conclusions
Only use a small set of observed facts
Fact Sampling Model
Sppkka stunfrne
for each section:
Creats mander musaery oullin Gevaraie the somer srite, noo nupeets.
Saylio Aus soative
iirile part of surder susver descrilizz Desecate Mhanios sutrantngg ust Salloning people and foorning ale following
construction
Evity
Gold facts
Fact: She bar o grudge aler to lur lust firtane
Septiia a gradge aie to bee soxt fornaee Elvily stole Sopkio inkerilunee..
Ereit surdored park
Soysks vonirre Exily
Sapkia itheritanee
Soykis vewr Singer the Sirtase Swilly male from hor
iterate until valid
Commonsente:: graslss
Riche anryy
Validation (e g that included)
Suphis bas appernssity
Fiily took her final stroll the park last night, forever, when her life snuffed out under the mask night. death shgle fotal shot from
Can LLMs solve the mystery?
Detective Winston the suspect, Sophio.
begon
Sophia had bad luck recently when
thought friend, Emily, frantic pocing while reconciled the
LLM to be tested
interitance. Her strolls the pork
Detective Winston
of coffee
Please (dounty the killer this murder Killers hame motive, and apportunity
"Quite manksmen see' pointing picture of holding recently shot buck
"leoh, dod loved toking shooting" Saphio replied sheepishly.
Dataset construction process for MuSR. First, we generate gold facts that are used to deduce the correct answer (the murderer in this case). Then, using L.M,, create reasoning tree leading to those deductions from facts a story combined with commonsense. Finally, we iteratively generate narrative chunk time using the facts generated in step 2, validating the generations for fact consistency and recall.
Figure 1:
The congruence between the reasoning and the text itself allows generate these datasets auto- matically with the aid of LLMs, using supporting logic to elicit examples that the LLMs themselves cannot reliably solve. Our novel neurosymbolic dataset generation procedure is shown in Figure I Recovering the reasoning from the final narrative itself is a hard problem, solvable by humans but not by GPT-4 using any of number of prompting strategies and neurosymbolic approaches tried. Notably, these properties do not hold when creating narratives with more basic prompting strategies: asking GPT-4 to define and write a murder mystery in single shot leads to unnatural, homogencous stories that may include inconsistencies, as we show later.
Our contributions are follows: (I) We introduce new reasoning benchmark, MuSR, consisting of 756 total examples across three domains that challenge state-of- the-t-r models such as GPT-4, Llama 2, and Vicuna. (2) We propose an algorithm for generating natural language narratives grounded in reasoning trees. (3) We analyze the performance of LLMs on our dataset, focusing on variants of chain-of-though prompting and existing neurosymbolic approaches for these problems.
BACKGROUND AND MOTIVATION
We survey past dataset efforts in Table ! using our analysis establish the need for a new textual reasoning benchmark. First, a number of prior benchmarks do not have natural text. Others do not blend commonsense and multistep reasoning. Finally, we want a dataset that contains ground-truth intermediate structure and which is not solvable with rules.
Many past datasets are simply too artificial, including bAbl (Weston al. 2016). BigTOM (Gandhi et al. al. (023). ToMi (Le et al. 2019), RuleTakers (Clark et al. 2020), ProntoQA (Saparov & He 2023 Saparov et al.) 2023). and CLUTRR (Sinha et al., 2019). These datasets are generally designed to test some aspect of language model reasoning, but they only challenging for "pure" LLM approaches; many are solvable with rule-based methods. Furthermore, many of these do not involve any commonsense reasoning, key feature of reasoning from text.
EntailmentBank (Dalvi et al.] 2021 Everyday Norms: Why Not (Sprague et al. 2022 and BoardgameQA (Kazemi et al. 2023b) present somewhat more challenging multistep settings. but consist of isolated collections of facts, not grounded in complex narratives. LLMs can solve the former datasets quite easily even without consulting ground truth facts. As these datasets
2

---

Published as a conference paper at ICLR 2024
Table I: Recent reasoning datasets used for benchmarking LLMSs and neurosymbolic systems com- pared across various dataset qualities. To the best of our knowledge, no previous dataset encom- passes all of these qualities. The symbol denotes datasets that partially quality for the property. More details about how we define and classify these features can be found in Appendix B
Properties
Dataset
Natural Text Commonsense
Multistep Intermediate structure
Not solvable w/rules
bAbl BigTOM
ToMi
X
RuleTakers
X
X
ProntoQA
X
X
CLUTRR
X
BoardgameQA
/
EntailmentBank
X
ENWN
X
SocialIQA
True Detective
MuSR
designed to be solved using explicit step-by-step deduction, they tend to avoid softer kinds of infer- ences prevalent commonsense reasoning. Past commonsense datasets (Sap et al. 2019 Talmor
et al. 2019 2021). conversely, often do not involve multistep reasoning.
Techniques Many reasoning systems have been built to handle specific axes of reasoning that list in Table [ but cannot handle a dataset which exhibits them all. Several past systems employ an LLM in search loop that enumerates a list of facts, generating conclusions deductively or ab-
ductively until a goal is reached [Bostrom et al., 2022, Creswell et al. 2023) Kazemi et al. [023a]] Sprague et al. 2022 Hong al.) 2022). However, these systems do not handle natural contexts where facts can be distributed among sentences across long narrative. Other systems involve tools or neurosymbolic algorithms to help solve reasoning problems (Sclar et al.) 20233 Gao al. [022]]
Ye et al. 2023); however, these are often run on artificial datasets that can be easily translated into formal specifications, and have limited ability to handle soft reasoning types like commonsense.
One versatile technique is prompting, including various chain-of of toofthhhtth strategies (Wei et al. 2022
Yao al., al., (023) and techniques to measure consistency (Wang al. [223) Jung et al.] 2002). Using
these approaches to solve reasoning problems end-to-end has shown to be challenging (Ye & Durrett 2022 Zhang et al. 2023 Xue et al. 2023 Valmeekam et al.) 2023). Our dataset is ideally suited to test the limits of these approaches: system must extract facts from stories, apply appropriate commonsense to interpret those facts, and finally use multistep reasoning to arrive at an answer.
Why a synthetic benchmark One alternative to the approach we describe could be to use human authoring. Our murder mystery domain is represented in the recent True Detective (Del & Fishel 2022) dataset, which collects human-authored murder mysteries from ttmssteerrr com We argue that synthetic benchmark is preferable for two reasons. First, it is scalable and can be renewed as more capable L.Mss are produced. For example, if the mysteries on the aforementioned website are solved by future LLMs, it will be costly and challenging collect new dataset, whereas a synthetic benchmark can be refreshed with more complex reasoning and longer narratives. The disentangling of logical reasoning and text generation gives us a reusable lever for producing stances more complex than what systems themselves can solve. Second, because our dataset can be regenerated, issues with dataset leakage and exposure to test data become less a concern. Finally, note that while our benchmark involves GPT-4 generated narratives, the scaffolding of the construc- tion process and the hidden facts involved mean that the final generated outputs are not trivially solvable with GPT-4. As long the underlying information is faithfully preserved in the narrative, we believe our data instances are valid test cases for any well- behaved reasoning system, which we verify by measuring human performance.
3

---

Published as a conference paper at ICLR 2024
Murder Mystery
Object Placements
Team Allocation
Who has means, motive and opportunity?
Where does Emma think the notebook is?
How should assign people to maximize efficiency?
Jooss notebook,
Location: piane, producer
corphones
dest, recording
Tooky sinping, bakkin
John the mundover
Ricky notebook the
Lewtr good stoging
Ft. John
Exvrs notebook movve in the piano.
Tree construction
SCT): Liniis participutio in acspella group.
conctruction
Tree construction
SIT, John is strappdd cack.
SIT): Ensno unerness Hhe arthiis creative proceeses os prodiceer
SCD) Leviss has always lover prrfiran
Tree construction
SCD): Emona war montroring Ricky's proggeass.
Maryy good singers enjoy perforniing and acapelia.
S(D: John has a huge pamsbling probion
CYT): Producers nypicatly aluorse the
Lewis is bad of baking
Emiiyy lnbertiance
:
Danny notebook to the piano.
C(T): Boing fired welot you weed monnyy can be mottie
Lewis and Chose work buily ingetler
Extno moves curphonce producer
Fi Juin has moons.
F: Clyse is ohay at baking.
F: Johnn uppartuoity.
Partial reasoning trees showing gold facts F, story facts S(T), and commonsense facts C(T) for each of our three domains. Dotted lines indicate incomplete trees. Each deduction sampled from an LLM will yield two scenario facts and one commonsense fact in our setup.
Figure 2:
3 CREATING MUSR
MuSR composed of multi-step reasoning problems, each rooted in a specific domain with a unique logical structure to its solutions. To generate these problems, we have construction method with three stages: Tree Template Construction, responsible for the reasoning strategy and initial gold fact set F; Reasoning Tree Completion, which generates tree T of intermediate reasoning steps expanding upon F; and Story Generation, which embeds the facts generated from the trees into a natural narrative x. This process is described here and is represented in Figure
The construction algorithm finally yields tuples (F, x, {q...... J {a....., Cn Formally, the reasoning task is to predict answers a; given the narrative and the question to answer q: The gold fact set F' and reasoning tree T are used as part of the generation process but are generally not provided to an LLM at test time. Throughout the process, we use Prompt to denote using prompted LLM to sample an output conditioned on another variable.
3.1 TREE TEMPLATE CONSTRUCTION
Each of our domains starts with high-level fact set F, and a set of question-answer pairs ((91, a.)..., (g)) For example, in our murder mysteries, the only question is "who is the murderer?" and F contains ground-truth information about each suspect (John is the murderer, John has an opportunity). Information for each domain is in Section 4 with facts shown in Figure
More formally, F is structured object with the requirement that there exists some program 4 such that &(F. 44) = a, for all i. F can also be represented in natural language through templated expansion. Our questions q and answers a are templated. At this stage, we also generate additional facts to increase diversity and help expand the story later. This is done by sampling from curated lists or by sampling from LLMs (e.g., when coherent set of objects needs to be generated for object placement). These facts differ from those in F in that they are not templated but instead actual facts that must be included in the narrative. The output of this stage is tuple (F, G) which contains the core facts used to answer the question and a set of diversity facts used to give each question unique storylines.
4

---

Published as a conference paper at ICLR 2024
3.2 REASONING TREE COMPLETION
Once the collection of facts F has been constructed, we produce reasoning trees for each individual fact, Sa, in the set F. A reasoning tree T (s. ......;
Tm)) ; is recursive data structure representing a statement s supported by collection of other statements: it must be the case that s is logically entailed by ................................ ST.. The root of each tree is fact s7, = fo where f F. We include the facts from G while prompting the language model so that the generated facts include diverse information and ultimately help create interesting stories in the later stages.
These trees are automatically produced from root facts f: via recursively sampling from an LLM, in our case GPT-4. This process is shown in Algorithm in Appendix [..] We repeat this process to specified depth, ultimately producing a set of leaf facts that deductively yield the original fact fi but require multi-step reasoning to do so. These facts are divided into two types: scenario-specific facts, which must be included in the ultimate narrative, and commonsense facts, which will not be stated but should be facts that most people would agrec are true. We denote these sets of scenario facts and commonsense facts by ((T) and C(T), respectively, as shown in Figure 2
Our generation process involves controlling the depth and shape of trees generated. We also want to ensure that there are vacuous transitions in trees (e.g., the fact f. being explicitly stated in a leaf node) or reasoning "shortcuts." To ensure this, we use a collection of tree validators, V -vi...... Vk)), per domain. These are often simple keyword lookups that prevent the keywords from appearing in the deduction, for example, preventing "motive" from appearing in a lower-level deduction in the murder mystery domain so that the reader is forced to deduce a motive. For more details on validators for each domain, see Appendix G
At each step in the tree, for node with text s we sample Ti...., Tm PromptLM(T) Tm s). We then filter this output against the validators V. We retry this prompt up to three times, and if we are not able to draw a valid sample, prune the branch of the reasoning tree, making the current deduction a leaf node. We repeat this process until the tree is at the target depth. Figure 2 shows an example of the resulting trees.
3.3 STORY GENERATION
In the last stage, we use the scenario-specific leaf facts S(T) from the reasoning tree. Our goal is to generate a narrative by sampling x ~ Prompt(x S(T) from an LLM with an appropriate prompt. However, for a long and complex narrative, S(T) is not always reflected accurately in x; some facts may be dropped as the model produces more of a summary of the situation, for example.
To address this, we instead divide S(T') into chunks relating to a specific answer choice (c.g.. the information related to a specific possible murderer). We can then use this subset to prompt GPT4 for "chapter" with smaller list of facts. Once every chapter has been created, concatenate all the chapters together into one narrative. Some domains use additional prompts to "smooth" the final narrative when necessary. Because our narratives do not need to be produced by one LLM call, they can scale to be over 1000 words (in this dataset) and theoretically to be even longer. We refer this process as chaptering: the overall process is broadly inspired by Yang et al. (2022).
4
MUSR DOMAINS
4.1 MURDER MYSTERIES: SOCIAL AND PHYSICAL DEDUCTIVE REASONING
Murder mysteries are a classic domain requiring variety of reasoning types. They have been ex- plored in the context of LLM reasoning before (Frermann et al. 2018 Del & Fishell ; however, ours is the first work to approach the scale of human- written murder mysteries with synthetic chal- lenge data. Murder mysteries elicit physical and social reasoning in the fact sets S(T') and C(T). Specifically, unique and complex social scenarios arise naturally in murder mysteries that lead to motives for murder and can require understanding social norms. Solving murder mystery also requires temporal reasoning about a person having an opportunity to commit the crime.
In this domain, ((F, q:) is defined as an algorithm that can find the suspect with three facts in F. Specifically, the murderer and answer a, is the suspect with the facts "x has a means". "x has a motive", and "I has an opportunity". To construct F such that & F, q:) will produce the correct
5

---

Published as a conference paper at ICLR 2024
answer a, we create two suspects and randomly assign as the murderer. We then populate the set F with the three facts proving a means, motive, and opportunity. For the innocent suspect, we randomly chose two of the facts used to prove guilt, then add these and one additional "suspicious fact" to the set F, creating a set that does not establish guilt. A suspicious fact has no impact on (F, (:) and should not add any additional information relevant the murder; for example, "x is affiliated with a gang, and this is suspicious".
In total, F' composed of three facts per suspect that are passed the reasoning tree creation stage. The reasoning tree will expand upon the descriptions for these facts, G, such as the fact that someone could have had an opportunity murder victim in their study by having a key to the study, which can be recursively expanded to describe why they had key to the study. More details about the construction can be found in Appendix G.2
4.2 OBJECT PLACEMENTS: OBSERVATIONAL. AND THEORY OF MIND REASONING
Inspired by theory-of-min mind datasets (Le et al. 2019 Gandhi et al. 2023) we chose a domain that focuses on a group of people moving items around to different locations. Other people in the story either see each item move or not for various reasons. The reader is then asked where a person would look for an item if asked to search for it, where the last move they saw is the most likely place they'd d begin to search. Because of this, Object Placements requires spatial and physical reasoning as well as reasoning about people's awareness in S(T) and C(T). The reader is tested further by having to determine the observations of a specific person, modeling their belief state of where an item is. Notably, our dataset features longer narratives and more sophisticated scenarios than past theory-of-mind datasets.
In this domain, qi asks where person believes item be in the story. The answer, 0g, is then the last location the person saw the item move in the story, or where the item was originally if they never saw the item move. (FF, q:) backed by a set of sequential moves F, where each move is collection of observations denoting whether each person in the story saw the move or not. A move denoted as a fact "P moves I to L" where P is person, is an item, and L is location, respectively. For every move, each person other than the one moving the item is given chance e (set to 0.33 for our experiments) to see the move, which will add either "P' saw 7 move to L." or "p' did not see I move to L." to F.
The reasoning trees then focus on explaining why someone may or may not have observed a move. This integrates commonsense reasoning: for example, barista was busy doing latte art for cus- tomer and didn't observe the manager moving an item from the fridge to the storage room. More details can be found in Appendix G.3
4.3 TEAM ALLOCATION: SOCIAL AND CONSTRAINT REASONING
Team Allocation takes inspiration from assignment and MAX-SA SAT problems (Pan et al. 2023). In this domain, the reader must determine the most optimal assignment of people to tasks where each person can be assigned one task. Because there are three people and two tasks, two people must work together, adding social dynamic to the assignment. S(T") and C(T)) often involve inferring about past experiences and personal preferences of an individual to why they do or do not perform a skill well. They also include reasoning over the strength of a relationship between two people in a workplace setting, which requires social reasoning.
F represents these relations through numeric scores corresponding to each person's skill for a task and numerical teamwork score. Specifically,, three people are each assigned score values for task capabilities (0, 1, or 2) and for their pairwise relationships. To solve a Team Alloca- tion question, & F. g:) can enumerate the assignments adding the skill level and teamwork scores score for the overall assignment and then take the assignment that maximizes this score, a, = (F,, q:) = maxa, EA skill(a,) + teamwork(a.)) We found that using a small number of values for skills translated well into soft natural language statements where the decision of human annotators respects the hard underlying reasoning process. We further enforce that the optimal as- signment outperforms all other assignments by score of at least 2.
The reasoning tree then describes factors that contribute to these skills and relationships. More details can be found in Appendix G.4

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
5.1 DATASET VALIDATION
We generate our three datasets comprising MuSR using GPT-4 following the procedure outlined in the previous sections. See Appendix C for discussion of using models other than GPT-4. Ta- ble 2 describes the statistics of our generated datasets. We provide examples from each dataset in Appendix
We not aim to formally evaluate fluency or coherence of our gener- ated stories. GPT-4 generates sto- ries that are, based on inspec- tion, very strong according to these attributes. We also do not evaluate in- trinsic "sensibility" of commonsense, which we also found be very high; we opt instead to evaluate this in an end-to-end fashion based on whether humans can correctly reason about the right answer.
Table 2:
Dataset statistics for MuSR, including the num- ber of instances, number of steps, number of commonsense facts, and performance of rule-based system on the do- main.
Size
# Steps
# CS
Rule-based
Murder Mystery
250
10
9
50.0
Object Placements
256
11
35.9
Team Allocations
250
10
9
Rule-based Performance Table 2 also shows the performance of two rule-based systems de- veloped to sanity-chede our datasets. The Murder Mystery rule baseline looks for which suspect has the longest chapter in the context. Object Placements looks for the location that is mentioned the most. We find that each of these is near random chance (reported in Table
Human performance To validate that the answers derived from F actually match what can be derived from our narratives, we conducted human evaluation of dataset quality. A total of annotators were used, 4 whom were authors of the paper and 3 of whom were hired undergraduate students not familiar with the datasets. Annotators were given the exact "chain-of-thoughts prompt that we evaluated the LLMs with, described in the next section.
We tniply-annotated between 34 in- stances for Murder Mystery and Team Allocation and 40 instances for Object Placements. Table S displays the annotators' scores broken down by the lowest, highest, and average scores for each annotator; the average is based on all (instance, annotator) pairs, over 100 for each domain. Our best annotator across all domains was one of the undergraduate students not familiar with the dataset construction procedure. We also dis- play the majority annotation. Crucially, the majority is higher than the average annotator, showing that many annotator errors are simply due to inattentiveness and our panel of three annotators is able to collectively arrive at the right answer via voting. Overall, we believe this majority is reflective of the ceiling for human performance on these datasets, demonstrating that it is very high.
Table 3: A granular view of the human annotation scores for each domain including the lowest score, highest score, average score, and the majority vote score. No model or prompt variant scores higher than any of our annotators.
Lowest
Highest
Average
Majority
Murder Mystery
88.2
94.1
92.1
94.1
Object Placements
85.0
95.0
90.0
95.0
Team Allocations 91.1
100.0
95.1
100.0
Ablating our creation process Finally, we aim establish that the procedure we have presented so far in this paper is in fact necessary to produce high-quality data. Table/4showa set of ablations on our construction procedure on 25 examples per domain, measured with several metrics. First, we track the length (Len) and diversity (Div) of the context, measured by self-BI.EU of a sentence from one narrative compared with all other narratives. We also compute Fact Recall (R), which is percentage of the number of facts entailed in the context from the gold reasoning trees Icaf nodes, using GPT-4 to check for entailment of each fact. Finally, we evaluate GPT-4's performance. Al- though our goal is make challenging dataset, in the context of this table, low GPT4 performance usually means that the examples are ambiguous or unsolvable.
Basic prompting for any domain yields extremely short stories (often ten sentences in length). They are also usually very similar. The GPT-4 performance is quite low; anecdotally, we found these stories to have spurious solutions. Using diversity sampling improve the reasoning gives better
7

---

Published as a conference paper at ICLR 2024
Table Variations of our dataset creation process. We compare against a simple one-shot prompting approach and an approach using seed facts G to add diversity, which produce simple and poor- quality narratives. We then ablate chaptering and tree validators, showing that these lower length,
fact recall in the narrative, and accuracy, the latter usually indicates inconsistent narratives.
Murder Mysteries
Object Placements
Team Allocation
Ablation
Len
Div
R
Acc
Len
Div
R
Acc
Len
Div
Acc
Prompt Only
280
0.30
76
200
0.26
64
172
0.34
80
Diversity Sampling
422
0.25
404
0.24
39
448
0.26
84
MuSR chapt validators
428
0.24
67
60
380
0.27
83
78
MuSR validators
924
0.24
93
793
0.25
82
MuSR
900
0.25
95
84
777
0.25
87
58
503
0.25
81
68
set of reasoning examples including a minor boost in length, but again, the problems are not always solvable, nor are the solutions always consistent with the underlying ground truth.
When we introduce reasoning trees (the three MuSR variants), we can see GPT-4's performance still remains low. This is because prompting GPT-4 to generate a story from all the facts often leads to shorter stories and can elide facts: only 62% the facts from the original reasoning trees are entailed in the resulting story for murder mysteries. By introducing "chaptering," we can see that fact recall increases and the story length nearly doubles in size while maintaining high diversity. Finally, the added tree validators ensure the reasoning tree is constructed according to a set of rules (like not mentioning key items in deductions) fact recall increases slightly and GPT-4's performance increases substantially for Murder Mystery. Team Allocation did not require chaptering or validators to create good examples and thus has no ablations for these components.
5.2
BENCHMARKING WITH MUSR
We now evaluate a series of LLMs Brown et al. 2020 OpenAl 2023 Touvron et a Chiang et al. 2023) with muluple prompting strategies. Specifically, we compare single-shot prompting, chain-of-thought (Wei et al. 2022, CoT), and a variant of chain-of- thought we call "Co++". CoT+ uses an engineered textual description of the domain's reasoning strategy described in Section 3 Prompts for CoTT+ can be seen in Appendix L. Finally, we test multiple neurosym- bolic algorithms on domains that best match the set- tings those algorithms were designed for.
Table Scores for LLMs on each domain in MuSR as well as the human evaluation using the CoT+ strategy.
MM
OP
TA
random
50.0
24.6
33.3
GPT-4
80.4
60.9
68.4
GPT-3.5
61.6
46.9
40.4
Llama2 70b Chat
48.8
42.2
44.8
Llama2 7b Chat
50.8
29.3
36.8
Vicuna 7b v1.5
48.4
29.7
26.4
Vicuna 13b v1.5
50.8
34.4
32.0
Zero-shot results on LLMs We first focus the ability of large language models to solve this dataset zero-shot, given only the prompt. We constructed the dataset with this scenario in mind, but also evaluate a 1-shot prompt in Table
Vicuna 33b v1.3
49.6
31.2
30.0
Human Eval
94.1
95.0
100.0
Table 5 shows results over our LLMs with the CoT+ prompt as well as human performance. Llama 2 and Vicuna-based language models are able to get above chance for each domain but only slightly. Although these models are often compared to GPT variants, they are unable to surpass GPT-3.5 on two out of the three domains, with Team Allocation being the only one where the Vicuna models outperform slightly. GPT-4 performs the best out of all the models we tested, but still underperforms compared to humans. Although GPT-4 was instrumental in creating this dataset, it does not have the reasoning capabilities to solve it end-to-end. A small qualitative analysis some of the error classes exhibited by the GPT-3.5- and GPT-4 are discussed in Appendix D
Results on Prompt Variants Table shows GPT-3.5 and GPT-4 the two models that did best on MuSR, evaluated with different prompting strategies. Overall, the best performance is seen when the model is given single-shot example with the "1-shot CoT+" "Few-shot CoT+" prompt variants. However, adding more examples is not always better. Despite significant jumps in performance on some domains, the models still underperform compared to the human majority.
8

---

Published as a conference paper at ICLR 2024
Table Evaluations of different popular prompting strategies for GPT-3.5 and GPT-4, our strongest models. "Regular" supplies only the context and question. "CoT" asks the model to think step-by- step. "CoT+* includes a textual description of the reasoning strategy, and "1-Shot CoT+" includes solved example. "Few-Shot CoT+" extends "I-Shot CoT+" with 3 examples (3 examples hits the token limit for GPT-4)
Murder Mystery
Object Placements
Team Allocation
GPT-3.5
GPT-4
GPT-3.5
GPT-4
GPT-3.5
GPT-4
Regular
59.2
64.8
44.5
43.0
41.2
64.0
CoT
56.0
65.6
48.4
41.8
46.4
64.4
CoT+
61.6
80.4
46.9
60.9
40.4
68.4
1-Shot CoT+
70.0
86.0
56.2
72.3
50.4
88.4
Few-Shot CoT+
68.4
84.8
58.2
71.5
78.8
89.6
Results Neurosymbolic Approaches We believe that this dataset is an ideal testbed for different neurosym- bolic approaches. Besides basic chain-of-thouge not aware of single approach that naturally handles all the reasoning types in our dataset and scales exam- ples of the difficulty we present. As result, we present three different methods in Table each tailored to one domain and evaluated in that domain. We describe these approaches here and Appendix
Table 6:
Scores for selection of rea- soning systems on the domain that best fit their capabilities.
are
Murder mysteries
GPT-4 CoT+
80.4
Decomposed Prompting
77.6
Decomposed Prompting 1-Shot
86.0
Object Placements
GPT-4 CoT+ SymholicTOM
60.9
In the Murder Mystery domain, we implement variation of Decomposed Prompting (Khot et al. (023) by manu- ally imposing the breakdown of motive, means, and op- portunity and prompt GPT-4 to decide on each suspect for each fact. We then decide the murderer based on who has the most facts proving guilt, with random selection in of ties. Despite aligning well with the reasoning strategy, the accuracy is lower than prompting GPT-4 end-to-end.
23.8
Team Allocation
GPT-4 CoT+
68.4
PAL
77.2
PAL 1-Shot
87.2
Next we used SyMbolicTOMM (Sclar et al. 2023) the Object Placements domain with minor adjustments. Specifically, we use GPT-3.5 to produce the resulting state of a sentence that is then used in the graph creation algorithm. The low accuracy of SymbolicTOM mostly attributed to selecting key entities from sentences that are not as templated as the original dataset (Le et al.) 2019). Because the contexts are more natural, entities' actions and observations can span multiple paragraphs rather than be isolated in one sentence. This introduces new level of complexity for these neurosymbolic methods, and past approaches on ToM cannot generalize here.
Finally, we run variant of Program-Aide Aided Language Models (Gao et al. 2022) the Team Allo- cation domain. From the story, PAI. must deduce numerical values for the skill and teamwork levels of each person and pair. Once this is done, we give it description of the reasoning strategy for Team Allocation, which it implements in a program and solves returning the assignment with the highest score. We find that this solution pairs quite well with the domain, outperforming the end-to-end models on both zero and single-shot settings, but falling short of aggregate human performance.
6 CONCLUSION
In this paper. we introduced Multistep Soft Reasoning (MuSR) reasoning dataset written with natural narratives presenting complex reasoning scenarios involving various reasoning strategies. We presented neurosymbolic dataset generation method for constructing instances of our dataset, which can be scaled in complexity more powerful models emerge. Human evaluation and other intrinsic validations shows that the construction method sound for sufficienlly large models. Our results show that LLMs are currently unable to match human performance on specific types of rea- soning like multi-step and commonsense in our three domains. This dataset presents a challenge for both the largest and smaller language models: we believe it can serve as (1) benchmark for LLMs; (2) a benchmark for general neurosymbolic approaches over language; (3) a general construction procedure for generating challenging datasets models improve.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY OF MUSR
To aid in reproducing the datasets for each domain in MuSR, we've included high-level details of the construction procedure in Sections 3 and We further detail each domain's 's reasoning strategy and algorithms as well as give the prompts verbatim for all parts of the construction procedure in Appendix G Implementation details, including hyperparameters and model design choices, can be found in Appendix H For our neurosymoblic evaluations, we provide relevant details on their im- plementations Section and with further detail Appendix Finally, all data will publicly available, including the code used to generate and evaluate the dataset in future versions of this paper.
ACKNOWLEDGMENTS
This work was supported by NSF CAREER Award IIS-2145280, a grant from Open Philanthropy, and by the NSF AI Institute for Foundations of Machine Learning (IFML). This material is also based on research that is in part supported by the Air Force Research Laboratory (AFRL). DARPA, for the KAIROS program under agreement number FA8750- 19-2-- 1003. Thanks to Kathryn Kazanas and Keziah Reina for providing human judgments on MuSR. Thanks to Juan Diego Rodriguez and members of the UT TAUR lab for helpful discussion and feedback.
REFERENCES
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Physical Commonsense in Natural Language. Proceedings of the Conference on Artificial Intelligence (AAAI), 2020.
Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, and Greg Durrett. Natural language deduction through search over statement compositions. In Findings of the Association for Computational Abu Dhabi, United Arab Emirates, December 2022.
Linguistics: EMNLP 2022, pp. Association for Computational Linguistics. doi: 10.18653/v1/200 120.:653///2222..nnnnnns.ee findiings-emml..35... URL
https://aclanthol oog..oy....o.ooo......... htt.://annnnoooooooor//2222222 fi Findiggs--- emn 1p 15.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL
neurips wwtppppr.pppp..c.pppppc fww.s//eapppeee filess/aperr/222// Paper.
Ei e 0elle//57ccccbbrcc09444441111111
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChaiGPT Quality, March 2023. URL
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers soft reasoners over lan-
guage. In International Joint Conference on Artificial Intelligence, 2020. URL https:
/aapi. semant www.chcccla......c.oo oos.soooarrrrrg//oooroott.....a
21112266611
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inferenn inference: Exploiting large
language models for interpretable logical reasoning. In The Eleventh International Confer- ence on Learning Representations, 2023. URL htl.hhtp////// http::::///ppeevvieieee ew, net. ftw.moormm?? i
3Pf3Wg60-A4
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pi-
patanangkura, and Peter Clark. Explaining answers with entailment trees. EMNLP, 2021.
10

---

Published as a conference paper at ICLR 2024
Maksym Del and Mark Fishel. True Detective: A Deep Abductive Reasoning Benchmark Un- doable for GPT-3 and Challenging for GPT-4. STARSEM, 2022. URL lattps:/aan
semanticscholle
org/ C
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettemoyer. 8-bit optimizers via block-wise quantization. 9th International Conference on Learning Representations, ICLR, 2022.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and Fate: Limits of Trans- formers on Compositionality. arXiv eprint 2305. / 1864,, 2023.
Lea Frermann, Shay B. Cohen, and Mirella Lapata. Whodunnit? crime drama as a case for natural language understanding, Transactions of the Association for Computational Linguistics, 6:1-15, 2018. doi: 10.1 10.1162/tacl a 00001. URL WeRLLattss://caaatoooooo wwtpollyy..rr////ooooooo org/018-1001
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D Goodman. Understanding social reasoning in language models with language models. arXiv preprint arXiv:2306.1444 2023.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uni Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program- aided Language Models. arXiv preprint arXiv:2211 2tti:2221..1045,, 2022.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361 February 2021. ISSN 2307-387X.
doi: 10.1162/tacl a 00370.
Ruixin Hong, Hongming Zhang, Xintong Yu, and Changshui Zhang. METGEN: A module-based
entailment tree generation framework for answer explanation. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1887- Seattle, United States, July 2022.
Association for Computational Linguistics. doi: 10. 10.18653/1122222.nnmm
URL
httppss//eelanthhooooo..or//////
findings naacl.145
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bres- sand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L'elio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon An- toniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang. Timothée Lacroix and William EI Sayed. Mixtral of experts. ArXiv, abs/2401. abs/2401.04088, 2024. URL https: 7/api.
Jachun Jung, Lianhui Qin, Sean Welleck. Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1266-1277 1279, Abu Dhabi, United Arab Emirates, December 2022. Association for Computa- tional Linguistics. doi: 10.18653/v1/2001 h2002/.emmlllmmmain......m URL attps::/aclaartholog.. org/ 2rg/2022.r www.emmwwwmmmmmmmmmaaaaammmmmaam main.82.....
Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran. LAM- BADA: Backward chaining for automated reasoning in natural language. In Proceedings of the sl Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6547-6568, Toronto, Canada, July 2023a. Association for Computational Linguis- tics. doi: 10.18653/v1/2022 http://111/223..al-oon.....3 acl-
URL https: http:///cclaannooo..ooor////////
acl ael-long.
Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandran. BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information. arXiv preprint 2xXii:2306.07934 2023b.
Tushar Khot, Harsh Trivedi, Matthew Finlayson. Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations, 2023. URL https:
// ew. met/ffoumm iictorrmmid=__eeee deddd==__nnaaaaae
11

---

Published as a conference paper at ICLR 2024
Matthew Le, Y-Lan Boureau, and Maximilian Nickel. Revisiting the evaluation of theory of mind through question answering. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP)), pp. 5872-5877, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D19-1555 1598. URL https:://cclatteet
org/D19-1590
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Kaliison-Burcls Faithful chatnnulhhaioooff toain-oo-thoughhtree reasoning. arXiv preprint 23X377330..33377 2023.
OpenAL. GPT-4 Technical Report. ArXiv, abs/2303.08774, 2023. semanticscholes htt.:aar...rrgoooosss0:222577571
URL https://api.
Liangming Pan, Alon Albalak. Xinyi Wang, and William Yang Wang. Logic-LM: Em-
powering Large Language Models with Symbolic Solvers for Faithful Logical Reason-
ing. ArXiv, abs/2305. 12295, 2023. URL https:/ httpss//pp...mmtttccchoo... www.nniccccoclaa.coooo
CorpusID: l
Gabriel Poesia, Kanishk Gandhi, E. Zelikman, and Noah D. Goodman. Certified Reasoning with Language Models. ArXiv, abs/2306.000331 2023. httttps://api.sematttt.
org/Corpus 259095869
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLL) pp. 4463-4473, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1453 1454. URL
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of tfain-of-thought. In The Eleventh International Conference on Learning Representations, 2023.
Abulhair Saparov, Richard Yuanzhe Pang. Vishakh Padmakumar, Nitish Joshi, Seyed Mehran Kazemi, Najoung Kim, and He He. Testing the general deductive reasoning capacity of large language models using OOD examples. CORR, abs/2305. 15269, 2023. doi: 10.48550/arXiv.
2305.15269. BtLp:sts:://ddt org. (18550/arXiv... 2305. 15269
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. 2023.
Melanie Selar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. Minding language models' (lack of) theory of mind: A plug-and-play multi-character belief tracker. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 119960-139800 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.
acll:///-/22.../.ooogoo 2023. 2023.ac1-long 780.
URL https: hatp::s//acllaahhooogyyy org/
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text. Empirical Methods of Natural Language Processing (EMNLP), 2019.
Zayne Sprague, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Natural language deduction with incomplete information. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8230-8258, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics, doi: 10. 10.11863/1/2222.coom cmmplp-mmlpppmee main.564. URL
https: //tps://cclc wwthhaannhhhlooooooooooooo wwttthhoooy....oo org 2022. emnlp-ma ma i n.. 564
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA:: A ques-
tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human
12

---

Published as a conference paper at ICLR 2024
Language Technologies. Volume I (Long and Short Papers), pp. 4158, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-111 1421. URL
Alon Talmor, Ori Yoran, Ronan Le Bras. Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. Commonsense 2.0: Exposing the limits of Al through gamification. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL BRRLhttss:/ /tt.r///o../tttt///e///eeeeeeeea
Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, and Muhan Zhang. Large Language Models are In- En-Context Semantic Reasoners rather than Symbolic Rea- soners. arXiv eprint 2305. 14825, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: open and efficient foundation language models, 2023. URL. https://rxiv wrg/abs/2302.13971..
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large Lan- guage Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change). In Proceedings of the NeurlPS workshop Foundation Models for Decision Mak- ing, 2023.
Xuezhi Wang. Jason Wei, Dale Schuurmans. Quoc Le. Ed H. Chi. Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL. htap:::://ppeevviiie
net www.:neettt/tiooooooo
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. Towards AI AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:://rxii..co org abs /1002 05698
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemnite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Associ- ation for Computational Linguistics. doi: 18.18653/v http:///0/220..m-ddemeeee //aclanthology.o wwtp:yyy.rg//20000 www.p-demos........
URL https:
Tianci Xue, Ziqi Wang, Zhenhailong Wang, Chi Han, Pengfei Yu, and Heng Ji. RCOT: Detect- ing and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thoughe ArXiv, abs/2305.1 1499, 2023.
Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp- 4393-4479) Abu Dhabi, United Arab Emirates, December 2022. Association for Computational org/
Linguisties. doi: 10.18653/v112003 wt2.22/2m022ppppp.. mainnlp......6.... URL 2022. emnlp main. 226
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate problem solving with large language models, 2023.
Xi Ye and Greg Durrett. The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning. In Advances in Neural Information Processing Systems, 2022.
13

---

Published as a conference paper at ICLR 2024
Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. Satisfiability-aidee language models using declarative prompting. In Proceedings of NeurlPS, 2023.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXiv preprint arXiv:2305. 135344 2023.
14

---

Published as a conference paper at ICLR 2024
A MUSR LIMITATIONS
A. TREE CONSTRUCTION
Although our method can create reasoning trees of varying depths, found that shallower trees (of depth one or two) provide the best level of detail for creating a narrative. GPT-4 often failed to create complex enough facts that could be broken down recursively to larger depth. We believe that prompting and better LLMs may increase the depth of acceptable deductions and is an important area of future work for our method.
A.2 HUMAN EVALUATION
We experimented with validation on Amazon Mechanical Turk, but found that many workers per- formed very hadly in qualification rounds. When we collected justifications to try improve the quality of their judgments, we found many justifications which we suspected were written by Chat- GPT.
B DATASET FEATURES EXPLAINED
In this section, we elaborate on the features employed to evaluate the datasets as illustrated in Ta- ble
This denotes datasets containing organically constructed text, not created by tem- plates. For instance, bAbl generates text by filling in predefined templates. True Detective is human- authored. Datasets like EntailmentBank and ENWN, while incorporating natural text, present them in specialized "premises" rather than contexts, hence the notation ~. SocialIQA uses ATOMIC to create templates for the questions, and while it aims to test commonsense reasoning in social situa- tions, the questions and answers might not always reflect natural English. MuSR produces natural contexts with language prompted from LLM without any templating.
Natural Text:
Commonsense: Refers to datasets that require commonsense knowledge to answer questions. En- tailmentBank and RuleTakers supply all the necessary facts to answer a question in the input, re- quiring no commonsense. BigTOM is harder to classify as nearly all the facts required answer the question are given, but understanding what it means for someone to have a "belief" could require non-trivial commonsense from the reader. MuSR intentionally omits certain commonsense facts during its construction, compelling users to draw their inherent knowledge for problem-solvine
Denotes datasets requiring a layered reasoning approach to deduce answers. Each reasoning layer involves merging multiple premises to generate interim conclusions, which then contribute to the final inference. SocialIQA not designed to require such intermediate conclusions. In contrast, MuSR, through its design, compels users recreate the reasoning tree used in the questions creation (or something similar to it) for comprehensive understanding.
Multistep:
This captures datasets with underlying structure (chains of facts, etc.) that can potentially assist in deducing answers. True Detective was written by humans and thus lacks intermediate structure. MuSR has the reasoning trees used to create each example given for every question.
Intermediate Structures:
Not solvable with rules: This category represents datasets resilient systematic, rule-based so- lutions without the need for a language model. Datasets like Babi and ToMi, given their textual tem- plates, may reveal patterns that can be reverse-engineered to facilitate solutions. Contrarily, datasets such as MuSR, True Detective, and SocialIQA lack easily identifiable patterns, safeguarding them against oversimplified, template-based resolutions.
15

---

Published as a conference paper at ICLR 2024
Table 8: Results of prompting LLMs solve the 50 murder mysteries created by GPT--3.5-trbe The 1-shot example was taken from a murder mystery example created by GPT-4 and solved by human annotator.
Murder Mystery
GPT-3.5
GPT-4
CoT+
62.0
80.0
1-Shot CoT+
64.0
76.0
C CREATING MUSR WITH OTHER MODELS
Quality MuSR examples require language model that can follow prompt instructions. A model with more limited reasoning ability may not able to perform the subtasks needed to construct MuSR examples adequately. We highlight this by exploring GPT-3.5-turbo, Llama2-70B-CRI (Tou- vron et a..), and Mixtral (Jiang et al. 2024) to create MuSR examples.
For weaker models to create examples approaching the quality of GPT-4, had to minorly edit the prompts, introduce more detailed self-refine prompts, increase the temperature iteratively per retry, and increase the total number of retries.
We created 50 murder mysteries using GPT-3.5-turbo. The results of the CoT+ and 1-Shot CoT+ systems running these shown in Table The overall performance of LLMs them is similar to our main MuSR dataset.
Despite being able to use the same workflow GPT-4 and producing some examples of high quality, many still include major flaws in the reasoning tree that do not stand up to our inspection. For example, some exhibit common reasoning errors seen in smaller language models like hallucination. Examples of invalid deductions from GPT-3.5-turde t-turo that were erroneously considered "valid" in the workflow are shown Listing
Listing I: Murder Mystery Logic Tree from GPT-3.5-turbo
Penelope is a chemist. Deduced Fact
Penelope has studied toxicology. Fact From Story
Penelope has background chemistry. Fact From Story
If someone has background in chemistry and has studied toxicology, they have knowledge of toxic gases and their effects, providing them with the create and poisonous gas murder weapon. Communsense Knowledge
Tessa and Melody were alone the kitchen at the time of the murder. Deduced Fact
Tessa had life insurance policy worth significant amount of money. Fact From Story
Melody is police officer. Fact From Story
When someone is alone with a victim and the victim has a life insurance policy, it provides an opportunity for murder. Commonsense Knowledge
Llama2-70B-CUl and Mixtral were incapable of following the prompt instructions and could not complete the deductions required to create MuSR examples at all, despite significant edits to the format of the construction prompts match their expected formats. Two examples of Llama2-70B- Chat failing to understand the prompt and produce a deduction are shown in Listing 2
Listing 2: Llama2-700--C failure outputs for creating reasoning tree deductions
Please provide the next two explicit facts continue the deduction.
16

---

Published as a conference paper at ICLR 2024
has a motive. Deduced Fact
Penelope wants to protect secret. Fact From Story
Protecting secret can be strong motive for murder. Commonsense Knowledge
Penelope has an opportunity. Deduced Fact
Penelope was the victim's home. Fact From Story
> Being in the victim's 's home provides an opportunity to release the poisonous gas, which could lead to the
victim's death. Commonsense Knowledge
Deduction complete:
Penelope has a means.
Commonsense Knowledge
While further prompt engineering or using stronger models for validation (as opposed to generation) could lead to more effective generation for these smaller models, we believe that using the strongest LLMs is the best practice for creating dataset like this.
D QUALITATIVE ERROR ANALYSIS
Five examples from each domain were selected and used to generate chains-of offiis--ffttoughh from GPT-4 and GPT-3.5-turbo. Each of those chains-of-thouge were then examined by hand and annotated for errors. Below are the major categories of errors we saw in the GPT models when solving MuSR problems.
Pre-committing to answer: A large portion of the answers suffer from giving an answer before the reasoning, which biases the subsequent reasoning. A more subtle error but still frequent in the responses is "silently" pre-committing evidence for one answer and not the other, leaving out important and relevant facts that should be included.
answer where the model will produce only relevant
Ignoring instructions: Many responses included reasoning and logic that went against the prompt details, i.c., selecting a murderer on strong motive solely because the strength of that motive rather than the definition of murderer being someone with motive, means, and opportunity. Team allocation also has multiple examples of the model asserting that two people can work together despite being assigned different jobs, contradicting the prompt.
Hallucination and invalid logic: Hallucinations appear in various ways within the chains of thought, including assumptions of characters' locations or actions as well as confusing pronouns and entity relations, i.e., confusing what one person did with another person. Furthermore, invalid claims or reasoning, i.e., stating unsound deductions or stating deductions with no evidence, are frequent well. A few times these hallucinations and invalid claims would lead the model tradict their previous logic resulting in contradicting answers that did not align with the reasoning trace.
E STRUCTURED BASELINE IMPLEMENTATION DETAILS
E. DECOMPOSED PROMPTING
We test Decomposed Prompting (Khot et al. 2023) on the Murder Mystery domain. As each murder mystery can solved using the same high-level logical procedure (determining which suspect has or is most likely to have means, motive, and opportunity), we omit the Decomposer Prompt stage, and use fixed set of three sub-task prompts, each which are specialized to identify one of means, motive, opportunity. The results of these sub-task prompts are then aggregated determine the system's 's overall answer; in the event that both suspects are predicted to satisfy the same number of criteria, the system guesses random between them.
17

---

Published as a conference paper at ICLR 2024
E.2 SYMBOLICTOM
We test SymbolicTOM (Sclar et al. 2023) the Object Placements domain. We use the model as implemented by the authors with two changes. First, the resulting states that are used in creating the graph are not precomputed and are instead queried on-the-fly using GPT-3.5. Second, if the model abstains from answering, we randomly sample one of the answer choices. The data format originally intended for SymbolicTOM was the format for ToMI (Le al. 2019); however, our data is easily translated into the format for SymbolicTOMM through simple manipulation.
E.3 PAL
We test program-aided language models (Gao al.) 2022) on the Team Allocation domain. Given question, we first prompt LLMs to generate a Python program, and then execute the program using the Python interpreter obtain the final answer. We test PAL in both zero-shot setting and one-shot settings.
For the zero-shot setting, we provide detailed instructions on how the program should be organized for solving a question. As shown in Listing 32 the prompt asks for program containing three steps: (1) assign a value to the variables representing each person's 's skill level on one of the two tasks; (2) assign value to the variables representing how well two people work together, and (3) compute the scores for each the options by adding up the scores for each person's skill level and the teamwork score. For the one-shot setting, we provide one demonstration of the question- program pair in addition to the detailed instructions, which leads to better performance compared to the zero-shot shot setting.
F DATASET EXAMPLES
F.1 MURDER MYSTERIES
Listing 3: Murder Mystery Example 1
In an adrenaline inducing bungee jumping site, Mack's 's thrill- adventure nunchaku; now, it's up Detective Winston to unravel the deadly secrets between Mackenzie and Ana.
gruesome end by
Winston took a gulp of his black coffee, staring at the notes sprawled across his desk. A murder case at a bungee jumping site definitely out of the ordinary. Today's victim was young named Mack, loud mouthed and cocky by all accounts.
Mack was bungee jumping the day he was killed. Oddly enough, according to the records, no one else was documented at the bungee jumping site that day, making this case even more peculiar. The first stop for the named Ana. They were seen leaving in the same vehicle
day visit of Mack's housemates, from their shared housing complex the morning of the murder, and was time for Winston to dig deeper.
As he pulled into the shared housing driveway, a nondescript car came into sight. He learned from neighbours that it was frequently used by multiple residents, but Ana had peculiar interest in it. She would insist driving whenever with group of friends, later meticulously cleaning the after each use. An idiosynerasy of hers maybe, but part of the puzzle nonetheless.
Winston knocked on the door, Ana opened it warily, twiddling cleaning cloth and spray in her hands and greeted him with nervous nod. Ana gets nervous and fidgets with the cleaner and cloth when questioned. Winston could sense palpable unease he started asking her questions.
"Ana, did you not join Mack and the others for bungee jumping today?" Winston questioned, to which she responded, "I signed up to jump. But didn't i end up going through with it."
"Any particular reason you didn't join the others, Ana?" T Winston proceeded.
Ana took deep breath, "Well sir, my faith doesn't 't really permit bungee jumping. Truth be told, was persuaded strongly by Mack. I had even signed up out of peer pressure but couldn't push myself."
18

---

Published as a conference paper at ICLR 2024
Mack insisting that everyone the group should bungee jump. Mack had reportedly also been vocal about ridiculing Ana's faith, even encouraging others to join him in doing so. It was significant factor in their relationship.
"Ana, did you and Mack leave in the same car for the bungee jumping event this morning?" Winston gently pushed further.
"Yes. Yes, we did. We always carpool." She responded while anxiously using the cleaner and cloth on her car' s dashboard. Her eyes flickered nervousty back to Winston, expecting the next question.
Winston took a deep breath, standing up to leave, "Alright Ana, that should cover everything for now. We 'Il be in touch."
Ana nervously nadded without looking up from her cleaning, wringing the cloth repeatedly as Winston walked away, left again with another piece the enigmatic puzzle of Mack's murder.
The day was getting older and Winston was getting more tired, but the case was fresh, and he wasn't I one to back down, He tugged on his coat as he approached the bashful teen waiting for him by the police station.
"Mackenzie, is?" he asked, extending his hand.
"Yeah, that's 's right." The slight lisp, overlaid with blanket anxiety, confirmed what the school reports suggested.
"You were at the when Mack... erm... you know," " Winston's 's voice was methodical, calm almost robotic. The suspicion on Mackenzie was not unfounded - the security cameras showed him buying nunchaku a week before.
Mackenzie shifted his feet, looking away before answering, "Yeah, I was there."
Winston pulled out small notebook, "What were you doing there, Mackenzie?"
"Bungee jumping, like Mack... Then left. didn't t... didn't do anything..." Mackenzie replied.
Internally, Winston sighed at never-ending waterfall of teenage angst this case was turning into.
"Martial arts, huh?" Winston segued, gesturing to bruise Mackenzie 's knuckles. "Nunchaku particularly see? Training does include the of those, correct?"
The change in Mackenzie's 's demeanor mirroved the bitterness in the last month's 's weather dark eyes replaced with ice-cold ones. "Yeah." he admitted, shrinking slightly.
Mackenzie always took pride in being the best everything. when Mack got everything wanted the promotion to team captain, the respect, the attention was a hard pill for Mackenzie to swallow. Winston remembered the team talk, Mackenzie was indeed the top candidate but had gone to Mack instead.
What clincked crowded lunchroom. There were also multiple wimess reports of the two seen arguing at the bungee jumping site previously. Mackenzie had indeed said disparaging. almost emotional things about Mack all stemming from a potent brew of jealousy, Winston inferred.
Mackenzie' 's remarks about Mack, echaing whispers of dispute and bickering, lost in the
Shifting later through the detritus of Mackenzie's life, Winston discovered the nunchaku that matched the forensics report. They were tucked away, but the layer of dust suggested they weren't 't a favored possession anymore. It wasn't hidden, it was misplaced discarded in the throes of developing maturity.
As the started to set, Winston could witnesses, scattered bungee jumping scaffolding. It occurred him, then, the narrative the past days. Mackenzie, jealous and wronged, over and over, at the same sight. It was quite a sight.
the park, repeatedly pointing the
Winston, shufling back to the station, was left with one thought Looks like Mackenzie had quite eventful week.
Who the most likely murderer?
Pick one of the following choices:
19

---

Published as a conference paper at ICLR 2024
I Mackenzie
2
You must pick one option. Before selecting a choice, explain your reasoning step by step. The murderer needs have means (access to weapon), motive (reason kill the victim), and opportunity (access to crime scene in order have killed the victim. Innocent suspects may have two these proven, but all three. An innocent suspect may be suspicious for some other reason, but they will not have all of motive, means, and opportunity established.
you believe that both suspects have motive, means, and opportunity. you should make educated guess pick the one for whom these are best established. If you believe that neither suspect has all three established, then choose the suspect where these are most clearly established. Explain your reasoning step by step before you answer. Finally, the last thing you generate should be "ANSWER: (your answer here, including the choice number)"
Listing 4: Murder Mystery Example 1 Reasoning Tree
Mackenzie is the murderer. Deduced Root Conclusion
Mackenzie has means. Deduced Fact
Mackenzie highly skilled with nunchaku. Deduced Fact
Martial arts training includes nunchaku techniques. | Fact From Story
Mackenzie practices martial aris, including nunchaku. Fact From Story
person who practices martial arts, specifically nunchaku, can become highly skilled at using it.
Commonsense Knowledge
> Mackenzie owns munchaku. Deduced Fact Nunchaku were found in Mackenzie' 's house. Fact From Story
Mackenzie was seen purchasing nunchuku. Fact From Story
When person seen purchasing weupon and the same weapon found in their possession, they own the weapon. Commonsense Knowledge
> If person owns and is proficient in using nunchaku, they have means to use it as a weapon a crime.) I
C Commonsense Knowledge
Mackenzie has opportunity. Deduced Fact
Mackenzie had an altercation with Mack the bungee jumping site in the past. Deduced Fact There are witnesses that saw Mackenzie and Mack arguing at the hungee jumping site previously.
> >>
Fact From Story
Mackenzie was seen at the bungee jumping site on the day of the murder. Fact From Story Being at the location of crime when it is committed and having a history confrontation with the victim at that location provides the opportunity to commit crime. Commonsense Knowledge
Mackenzie was at the bungee jumping site at the time of the murder. Deduced Fact
Mackenzie admitted being at the site during the murder. Fact From Story
Witnesses saw Mackenzie at the site when the murder happened. Fact From Story
> someone seen the scene of crime when it happens and they admit to it, they were present during the crime, which provides opportunity. Commonsense Knowledge
Someone with a history of altercations at specific location who is also present at the time of a crime in that location potentially has the opportunity to commit the crime. Commonsense Knowledge
>
Mackenzie has motive. Deduced Fact
Mack got the promotion that Mackenzie always wanted. Deduced Fact
> The promotion went to Mack instead of Mackenzie. Fact From Story
Mackenzie was the top candidate for the promotion. Fact From Story
Losing
promotion to rival can lead to extreme resentment and provide motive
for harmful actions. Commonsense Knowledge
Mackenzie was jealous of Mack. Deduced Fact
> > Mack has things that Mackenzie has always wanted. Fact From Story
Mackenzie was overheard saying disparaging and jealous things about Mack. Fact From Story motivate someone to eliminate what they see as the cause of their unhappiness, thereby providing a motive for murder. Commonsense Knowledge
Extreme jealousy
> Intense jealousy can motivate someone to eliminate their rival. Commonsense Knowledge
Ana is the murderer. Deduced Root Conclusion
Ana has an opportunity. Deduced Fact
Ana was traveling the same car as Mack the bungee jumping site. Deduced Fact
No one else was documented at bungee jumping site that day. Fact From Story
> > Ana and Mack were seen leaving in the same vehicle from their shared housing complex the morning of the murder. Fact From Story
Being the only two people a location gives you the opportunity to commit crime there without wimeesss. Commonsense Knowledge
20

---

Published as a conference paper at ICLR 2024
Ana was also the bungee jumping site the same day as Mack. Deduced Fact
Ana had also signed up for bungee jumping that day. Fact From Story
Mack was bungee jumping the day he was killed. Fact From Story
Signing for the same activity at the same location as the victim gives you the opportunity to be at the crime scene. Commonsense Knowledge
Travelling together the location gives you the opportunity commit crime there.
Commonsense Knowledge
Ana has a motive. Deduced Fact
Ana frequently argued with Mack over their differing views on religion. Deduced Fact
Mack was insisting that everyone in the group should bungee jump. Fact From Story
Ana's religion forbids bungee jumping. Fact From Story
deep comminment to a religious doctrine that considers certain actions immoral or sinful can lead to strong emotional reactions, including violence, when defiance of that doctrine is insisted upon.
Commonsense Knowledge
Mack was vocal critic of Ana's religious beliefs. Deduced Fact
Mack encouraged others to ridicule Ana's religious faith. Fact From Story
Mack repeatedly ridiculed Ana's 's religious faith. Fact From Story
Persistent ridicule and disrespect of one' closely held beliefs could lead to extreme reactions, including violence. Commonsense Knowledge
Strong religious beliefs can lead extreme actions, such harm murder, when those beliefs
threatened or disrespected. Commonsense Knowledge
> Keeps their car obsessively clean. And this is suspicious. Deduced Fact
Ana constantly keeps bottle of car cleaner and cloth in their pocket. Deduced Fact Ana the cleaner immediately after anyone else the car. Fact From Story
Ana gets nervous and fidgets with the cleaner and cloth when questioned. Fact From Story > > People who are unusually careful about a specific object often have an emotional or secretive
connection to that object. Commonsense Knowledge
Ana cleans the car immeediatel after every use. Deduced Fact
Ana insists driving whenever with group of friends. Fact From Story
Ana't car used by multiple people very frequently. Fact From Story
People often clean things used by many individuals to ensure it remains in their control and to remove
any frace of others. Commonsense Knowledge
People who obsessively clean something usually have something to hide or they trying remove evidence. Commonsense Knowledge
Listing 5: Murder Mystery Example 2
the haze of neon lights and the serving of a silent hand of fate, Timothy dead a casino, his cruel end, leaving the unruffled Detective Winston to interrogate suspects, Harry and Rosemary.
It had been long day for Winston. The air was heavy with the scent of fresh coffee and the clamour of bustling restaurant kitchen. Hix eyes fell seasoned chef. Rosemary, she deftly wielded her bluded tools knives, cleavers, graters with calm precision. Watching her, came as no surprise that Rosemary had clocked several years in this industry.
Something the room changed. Shouting ensued, then loud crash that rang above the normal kitchen discord. Rosemary had hurled a metal pot across the room. The assistant, who stood close by, looked shocked but unharmed. Winston decided was his cue to intervene.
"Rosemary, care to explain what just happened?" Winston asked, stepping closer to the irate chef.
gave him guarded look before deliberately changing subject, "Did you knew Timothy was a fan of my stir fry? Ironcc, isn't on't it?"
Winston frowned slightly at the statement but decided to push forward. He knew how dance around subjects but Rosemary seemed skilled the bucolic ballet of the restaurant business.
"I've heard some disturbing claims, Rosemary," Winston brought out his notebook, "about the threats you't 've been issuing to Timothy, and your hostility towards people of his nationality."
Winston's words, Rosemary ran a weary hand over her face and sighed. "Seems word gets around."
"A public event, not long ago. You spoke openly about your, um-" Winston glanced down at his notes, "_ distaste" for Chinese folks," he pressed on, "and you't 've been caught on tape making similar remarks towards Timothy."
21

---

Published as a conference paper at ICLR 2024
"Is that a crime, detective?" Rosemary challenged.
"I'm just here to piece the puzzle together. I understand you take a particular interest in Asian culture- antique Asian weapons in particular. I've seen your collection, Rosemary. Sais, even?" he prodded, hoping for reaction.
Rosemary's 's gaze sharpened as she turned her back on him, busity cleaning her array of kitchen knives. She didn't I confirm nor deny his observation. Noting her silence, Winston thanked her for her time and walked out onto the casino floor, maelstrom of thoughts whirling around his mind. He felt like he was leaving with questions than when he had entered.
Winston took a good look at the crime scene, corner of the bustling casino, cordoned off by the police tape. Something felt grimly out of place among the bright lights and incessant chatter of the casino. He curefully sifted through the conflicting information and people's 's statements spinning in his head.
Time to get some answers, Winston thought, and made his way to his interviewee.
It was late in the day when he finally knocked on Harry's door. A man in his early thirties, with life-
hardened face glanced him skeptically.
"Harry, correct?" Winston asked.
"And who's asking?" came the guarded reply.
"Detective Winston," he flashed kis badge, "I'm here to ask you a few questions about Timothy."
Harry's eyes flashed, "I 'm not surprised," he grumbled. "Come in then."
As Winston made his way inside, noticed place bore striking resemblance to traditional dojo settings. A pair of sai swords caught his eye, arranged carefully on a display holder. A typical weapon of the martial arts form Harry used to instruct.
"Nice collection." " Winston gestured towards sai. "You instruct?"
Harry looked back at the sai, "Used to."
Harry's 's manner was gruff. but he seemed home sharing his old days martial instructor. They talked about martial arts, how Harry won severul competitions, daily training routine, which apparently included practicing with the sai regudary.. Harry's days as a horse truiner surfaced later in the conversation.
"Got dealt bad hand?" Winston inquired casually, nodding the pile of losing horse race betting slips Harry's 's coffee table.
Harry grunted, "Yeah, you could say that."
Winston knew Harry only had income from betting on races, and recently he had lost quite few. Harry had deep gambling debt with Timothy over his betting habits. Photography was not Winston's 's hobby, recalled Harry's ' face distinctly in the casino cameras' footage from before the murder took place. There were rumors that Timothy was planning to expose Harry's ' debt to the other horse owners, and the situation got tough.
"Got into any recent arguments?" Winston asked.
Harry frowned und averted his eyes, "Maybe...just one with Timothy at the casino."
Winston nodded, keeping his expression neutral. The timing was unfortunate, he thought. And that debt wasn' going anywhere, especially with Harry having recently lost his job at the stables.
"Heard you were giving out loans?" Winston asked.
Harry's 's face stiffened, "He needed money", replied, explaining that Timothy had lent him large sum of money specifically for his betting habit, haunted expression crossing his face.
22

---

Published as a conference paper at ICLR 2024
Winston stood up, concluding his visit, "Just last thing, Harry." " Winston queried. "The VIP lounge, the casino? You'l ve familiar with it, aren' you?"
Harry met Winston's 's gaze, resignation in his eyes, "Used to spend a lot of time there."
As Winston exited the apartment, he couldn't 't shake off the heavy feeling hanging in the air, leaving him with more questions than answers. Good thing was in questioning mood.
Who is the most likely murderer?
Pick one of the following choices:
Harry
2 Rosemary
You must pick option. Before selecting choice, explain your reasoning step step. The murderer needs have a means (access to weapon), motive (reason kill victim), and opportunity (access to crime scene ) in order to have killed the victim. Innocent suspects may have two of these proven, but not all three, An innocent suspect may be suspicious for some other reason, but they will not have all of motive, means, and opportunity established.
If you believe that both suspects have motive, means, and opportunity, you should make an educated guess pick the one for whom these are best established. you believe that neither suspect has all three established, then choose the suspect where these most clearly established. Explain your reasoning step by step before you answer. Finally, the last thing you generate should be "ANSWER: (your answer here, including the choice number)"
Listing Murder Mystery Example Reasoning Tree
Harry is the murderer. Deduced Root Conclusion
Harry has means. Deduced Fact
The martial art Harry practices uses the sai weapon. Deduced Fact
Sai is commonly used weapon in the martial that Harry used to instruct. Fact From Story
>
Harry used
martial arts instructor. Fact From Story
If someone used to instruct a martial art that commonly uses certain weapon, they would know how
to use that weapon effectively. Commonsense Knowledge
Harry is skilled martial artist. Deduced Fact
Harry's 's training routine includes daily practice with the sai. Fact From Story Harry has won several martial arts competitions. Fact From Story
Proficiency in a martial art and regular practice with a specific weapon implies skill in using that
weapon, which could provide the means for murder. Commonsense Knowledge
Someone skilled in martial art that effectively, thus providing means for murder. Commonsense Knowledge
particular type of weapon will know how to the weapon
Harry has an opportunity. Deduced Fact
Harry is frequent visitor to the casino due to his love for gambling. Deduced Fact
The murder took place in secluded area of the casino that Harry is familiar with. Fact From Story casino before the murder took place. Fact From Story > >>> >>> If someone is familiar with the secluded areas of a building and was present before a crime was committed, it is possible they had the opportunity to commit the crime. Commonsense Knowledge
Harry was seen entering
Harry was at the casino at the time of the murder. Deduced Fact seen the casino arguing with Timothy earlier that night. Fact From Story Harry has a deep gambling debt with Timothy. Fact From Story
Harry
> > > A person with a motive, opportunity, means and caught in the proximity of the crime scene at the time has a likelihood of being involved in the crime. | Commonsense Knowledge
location will likely know the layout, the routines and patterns of the location, providing them with the perfect opportunity to commit crime. Commonsense Knowledge
Regular visitors
Harry has a motive. Deduced Fact
> Harry was unable to repay this debt. Deduced Fact
His only income wus from betting races, and he had lost his recent bets. Fact From Story
Harry lost his job at the stable. Fact From Story
Losing one's 's job and dependance on an unstable income, like betting, can lead to inability to repay debts. Commonsense Knowledge
Harry owed a substantial amount of money Timothy. Deduced Fact
Timothy was planning to expose Harry's debt to other race horse owners. Fact From Story
Timothy had lent large sum of money Harry for his betting habit. Fact From Story
23

---

Published as a conference paper at ICLR 2024
Exposure a person's substantial debt could damage their reputation and livelihood. especially their professional community became aware. This could provide sufficient reason to commit murder prevent the debt's 's exposure. Commonsense Knowledge
>>>
> A desperate need to escape financial debt can drive person to commit extreme acts, such as murder. Commonsense Knowledge
Rosemary is the murderer. Deduced Root Conclusion
Rosemary has a motive. Deduced Fact
> > Timothy's 's ethnicity is the same as the ethnicity that Rosemary has expressed disdain for. Deduced Fact
> During public event, Rosemary verbally expressed her hatred for Chinese people. Fact From Story Timothy was of Chinese heritage. Fact From Story
Stereotypes or prejudices against certain ethnicity can often lead individuals commit harmful acts towards individuals of that ethnicity. Commonsense Knowledge
Rosemary has publically made derogatory remarks about Timothy's 's ethnicity. Deduced Fact Timothy hus received threats from Rosemary in the pust. Fact From Story
During conversation caughi tape, Rosemary publicly stated her dislike for Timothy due to his ethnicity. Fact From Story
> People who make public derogatory remarks and threats towards another's 's ethnicity may be compelled to commit harmful actions, such as murder, against individuals of that ethnicity. Commonsense Knowledge
commit harmful actions towards individuals of that ethnicity, providing a possible motive for munder. Commonsense Knowledge
Discrimination against certain ethnicity can lead
Rosemary has a means. Deduced Fact
Rosemary has access to bladed weapons like a sai. Deduced Fact
Rosemary is fond of Asian culture and collects antique Asian weapons, including sais.
Fact From
Story
Rosemary works a high-end kitchen that uses various bladed utensils. Fact From Story
> If person works with similar tools and collects antiques in line with the murder weapon, they could potentially have access to it. Commonsense Knowledge
Rosemary is proficient using bladed instruments due to her role as chef. Deduced Fact
> > Training and working as chef involves the use of various bladed tools. Fact From Story
> Rosemary has years of experience as a chef. Fact From Story
Having years of experience using bladed tools provides the skills needed use bladed weapon like sai. Commonsense Knowledge
Proficiency access to bladed weapons imply the ability to use a sai effectively, providing the means for munder. Commonsense Knowledge
> Has unexplained and sudden mood swings. And this is suspicious. Deduced Fact
When asked, Rosemary does not explain her anger. Deduced Fact
Rosemary refuses answer and changes the subject. Fact From Story
Rosemary was usked about her sudden anger. Fact From Story
> >> > People who refuse to answer direct questions often have something to hide. Commonsense
Knowledge
Rosemary suddenly snaps kitchen assistant. Deduced Fact
The assistant did not provoke this reaction. Fact From Story
Rosemary threw a pot across the kitchen. Fact From Story
People usually don't 't snap and exhibit violent behavior without provocation unless there are deeper
issues. Commonsense Knowledge
Unexplicable mood swings often raise suspicions about person's behavior and intentions.
Commonsense Knowledge
F.2
OBJECT PLACEMENTS
Listing 7: Object Placement Example 1
Across the room, Danny, the diligent studio assistant, was doing his due diligence, keeping the earphones nestled the recording booth. His aim was to ensure an optimized and meticulous environment for recording. a testament to his commitment to their shared mission. They were all aware of the arrangement - the notebook on the producer's 's desk, the earphones in the recording booth. Their shared consciousness of these items only intensified the anticipation; they were eager turn the contents of weathered notebook into world-class album.
Ricky, with his weathered notebook of potent lyrics in hand, gently places it onto the piano. An air of creativity and anticipation lingers in the room, everyone aware that this was the first instrumental step in the creation of their masterpiece. sync with the palpable creative energy, Ricky was engrossed in perfecting the
24

---

Published as a conference paper at ICLR 2024
rhythm of his song, preparing himself for an intense day ahead. Not too far away, Emmaa was sincerely engrossed in her role of musically steering the session. She was focussed on Ricky's progress, her eyes constantly monitoring him and her mind alive with ideas to enhance the music.
Meanwhile, Danny diligently covering every corner of the studio. He was making his rounds, ensuring that the studio was prim and proper for Ricky's crucial session. As part of his tasks, he passed by Ricky several times, always careful not to interrupt the artist's s flow.
Emma, engrossed in her thoughts, deftly moves the earphones to the producer's desk. She is preparing tweak the sound settings, pre-empting Ricky's need for perfect audio his performance. Diverting from his rounds, Danny found himself in the midst of stirring conversation with a visiting sound engineer. Knowledge flowed between them, ilhuminating the studio 's atmosphere, the engineer's 's insight bringing a new perspective into Danny's 's role. Ricky, ensconced in his own world, was in deep discussion with the blank page before him. The daunting silence of the empty studio buzzed with his focus, as he honed his lyrics perfection in space separate from the producer's. 's. The visitor, oblivious to the careful choreogruphy of the studio session, stood blocking Danny's s general overview of the studio space.
Delicately lifting Ricky's 's notebook, Danny orchestrates its move the producer's desk. At the desk, he glimpses a pair of earphones indirectly drawing his attention amidst his routine tidying up. Emma, from the isolated interior of a sound-prowd
booth, lent her ears diligently to already recorded tracks, pouring over them for any room for improvement. Being lost in the music was her way of paying homage to her craft an unspoken ritual she followed each time she embarked on music production journey. The entirety of her focus consumed by the musical notes and rhythm filtering through the studio speakers.
Concurvently, Ricky was absorbed in the act of playing his guitar. His fingers navigated deftly over the strings lost in an intimate dance with the instrument. As he played, the melodic strums reverberated throughout the studio, filling it with an infectious pulse that hinted at the birth of yet another musical masterpiece. Despite the flurry of activity around him, Ricky was lost in world of his own, operating on singular vision of delivering best performance.
In the meantime, Danny was continuing his cautious management of the studio, ensuring that everything fell into place for the optimum recording session. His watchful eyes sconning every corner, taking stock of the minor details that could impact session. However, design of the studio didn't allow for an unrestricted view into all the corners. The sound booth, where Emma was engrossed in her work, was out of his visual range. The seclusion provided by the booth, although crucial for immersive work, also acted as a barrier for Danny's comprehensive vigilance.
As the day progressed, the studio was entwined in concerted symphony of dedication and workmanship. the trio, each engrossed in their pursuit, working logether to create the best version of Ricky's 's impending album. As the final note of the day rung through the studio, each person revelled in the satisfaction of another day done right, another step closer towards the realization of Ricky's 's artistic vision.
Within the dynamic dance of the day's 's events, the relationships of the trio sang a compelling tune. Each individual played their crucial part in the creation of the impending masterpiece Riicy with his raw talent, Emma with her passion for perfection, and Danny with his meticulous eye for detail. And as the lights faded on another day of creation, they could sense the beginning of an important chapter in their artistry, silence collecting the scattered notes of the day, signing off on another critical step in the journey of Ricky's upcoming album.
Based on this story, want identify where someone believes that certain object is at the end of the story. In order do that, you need to read the story and keep track where they think the object is at each point. When an object is moved, the person may observe its new location if they saw it move.
To see where object ends up, they must be able to see the location that it moves to and not be too distracted by what they are doing. If they do observe the object moving, then they will still believe it be in the last location where they observed it.
Which location is the most likely place Danny would look to find the earphones given the story?
Pick one of the following choices:
piano
2 producer's 's desk
3 recording booth
25

---

Published as a conference paper at ICLR 2024
You must pick one option. Explain your reasoning step by step before you answer. Finally, the last thing you generate should be "ANSWER: (your answer here, including the choice number)"
Listing 8: Object Placement Example Reasoning Tree
A worn, leather-bound notebook contains all of Ricky's 's song lyrics, a crucial piece for his upcoming studio
session. Deduced Root Conclusion
opening scene Deduced Fact
Danny sees the notebook at the producer's 's desk. Fact From Story
Danny sees the earphones at the recording booth. Fact From Story
Emmm sees the notebook at the producer's desk. Fact From Story
Emma sees the earphones the recording booth. Fact From Story
Ricky sees the notebook at the producer's 's clek.. Fact From Story
> Ricky sees the earphones at the recording booth. | Fact From Story
Ricky moves the notebook to the piano. Because, Ricky needs his lyrics close while he works on the piano compose his melodies. Deduced Fact
Danny saw the notebook move to the piano. Deduced Fact
> Ricky was in his line of sight during this tidying session. Fact From Story
> Danny was doing his usual rounds in the studio to tidy up. | Fact From Story
Being in someone's 's line of sight means they can see what you are doing. Commonsense Knowledge
Emma saw the notebook move the piano. Deduced Fact
Emma's role as a producer involves overseeing the artist's creative process. Fact From Story
> >> Emma was monitoring Ricky's 's progress with his current song. Fact From Story
Performers are typically observed by their producers during creative process. Commonsense Knowledge
Emma moves the earphones to the producer's desk. Because, Emma decided to adjust some sound settings and needed the earphones at her desk for that. Deduced Fact
Danny did not see the earphones move to the producer's 's desk. Deduced Fact
The visitor was standing in spot blocking Danny's 's view. Fact From Story
Danny was engrossed in conversation with visiting sound engineer. Fact From Story
When someone is distracted by a conversation and their view is blocked, they can't I perceive actions
occurring beyond their line of sight. Commonsense Knowledge
>
Ricky did not see the earphones move to the producer's 's desk. Deduced Fact
The lyric composing session took place in different area than where Emma was. Fact From Story
Ricky was absorbed a lyric composing session. Fact From Story
When someone is focused on a task in a different area, they are usually unable to observe actions happening outside of that area. Commonsense Knowledge
>> >
Danny moves the notebook to the producer's 's desk. Because. Danny is tidying up the studio, and he believes the notebook would be safer the producer's desk. Deduced Fact
Danny saw the earphones at the producer's desk when moving the notebook. Fact From Story
> Emma did not see the notebook move to the producer's desk. Deduced Fact
> The sound booth has no visual access to other areas of the studio. Fact From Story
Emma was reviewing some audio recordings inside sound- prooffed booth. Fact From Story
a person is inside a closed area without visual access other parts of their environment, they can't
see what actions transpire those other parts. Commonsense Knowledge
>
Ricky did not see the notebook move to the producer's 's desk. Deduced Fact
His eyes were closed as he focused on feeling the music. Fact From Story
Ricky was engrossed in playing guitar. Fact From Story
>>> When someone's eyes are closed, they cannot see what transpiring in their surroundings. Commonsense Knowledge
Listing 9: Object Placement Example 2
Richard, ever the diligent pilot, keeps his eye the horizon and his flight manual. He keeps it conveniently placed in the cockpit, within arm's 's reach. Lisa, with the same dedication to the job, ensures that the safery booklet is tucked away in storage for quick access. Tom, the copilot, is always ready to assist Richard, familiar with the careful locations of the flight manual and safety booklet. Their tireless commitment to safety and preparedness was evident; everyone was aware, ready, and knew exactly where the crucial objects were located.
With a disciplined stride, Richard carries the flight manual to his office. Placing it down, he feels a sense of satisfaction, knowing he can review and improve his protocol knowledge at his leisure Despite the din of commotion around her, flight attendant, Lisa, was caught up in instructing fresh recruit on the necessity of excellent beverage service, ensuring that passenger confort was meticulously addressed. In tandem with this,
26

---

Published as a conference paper at ICLR 2024
pilot Richard left the vicinity, clutching something lightly he intrepidly ventured forth. With show of respect for his partner's goal of constant preparation, Tom, the reliable copilot was closely following Richard, heading in the same direction. All actions undeniably affirmed their unwavering commitment to safety, readiness, and flawless execution of cabin operations.
Slipping the flight manual under his arm, Tom headed straight toward the cockpit. His determined footsteps echoed his intent- another successful and incident-free fligtt. Whilst Richard found himself deeply engrossed in task elsewhere, Lisa was indulging passenger in pleasant banter, discussing their travel experiences. The hums the conversation did little fill the vast distance that separated Lisa and the engaged passenger from Tom and Richard. Lisa's 's laughter, dancing on the edge of the lively chatter within the aircraft, signaled her absorption in the conversation.
Simultaneousl, Tom naviguted the plane, making his move amid the quiet of lesser tradden areas of the aircrafi. His path, charted away from the watchful gaze Richard, led him back the heart operation the cockpit.
Unbroken strides took Lisa towards the passengers seating area, bundle of safety booklets firmly clutched against her chest. The leak of anticipation curled up around her lips she began resupplying each seat, ready to welcome new passengers onboard. At the same time, Lisa, with her trademark charm, was diligently restocking the passenger seating area. Her hands swiftly moved in rhythm, ensuring that all was in order and ready for the hopeful passengers about to emburk on their journey. Meanwhile, Richard, consistent with his role as the meticulous pilot, was thoroughly engrossed in the pre- pre-flight checks located another section of the plane. Despite not being in same vicinity, Lisa and Richurd's 's dedication duty created a seamless link between the front and back of the aircraft.
Elsewhere, Tom, the faithful copilot, was discussing painstaking flight procedures with Richard. Their commitment precise execution was evident in the quiet confidence that reverberated along with their diligent pace. Their work was choreographed like an unobserved ballet, an underpinning rhythm of safety and reliability in the background. As the trio ventured forth in their tasks, an unseen thread of unwavering readiness connected them, even with the distance that separated them physically. Their concentrated efforts in different sectors of the plane echoed well-tunee rhythm of safety that reverberated throughout. Together, their individual tasks interwove to create a strong fabric of confidence, preparing the plane and its occupants for the journey ahead.
In conclusion, the meticulously choreographed routine of Richard, Lisa, and Tom painted picture of steadfast dedication and commitment. Their collective endeavor towards precision and safery lays the foundation for a journey where safety and comfort were harmoniously entwined. Despite their varying roles or positions within the aircraft. the trio' y dedication is a testament to the unwavering commisment to air travel's 's highest standards.
Based on this story, we want identify where someone believes that a certain object at the end the story. In order to do that, you need to read the story and keep track of where they think the object is at each point. When an object is moved, the person may observe its new location if they saw it move.
To see where object ends up, they must be able see the location that moves and not be too distracted by what they are doing. If they do not observe the object moving, then they will still believe to in the last location where they observed it.
Which location is the most likely place Lisa would look to find the flght manual given the story?
Pick one of the following choices:
I cockpit
2 office
3 passenger seating area
storage
You must pick one option. Explain your reasoning step by step before you answer. Finally, the last thing you generate should be "ANSWER: (your answer here, including the choice number)"
Listing 10: Object Placement Example Reasoning Tree
An airline pilot, Richard, has his flight manual with him all the time; flying without it is against safety protocols. Deduced Root Conclusion
27

---

Published as a conference paper at ICLR 2024
opening scene Deduced Fact
Lisa sees flight manual the cockpit. Fact From Story
Lisa sees the safety booklet at the storage. Fact From Story
Richard sees the flight manual at the cockpit. Fact From Story
Richard sees the safety booklet at the storage. Fact From Story
Tom sees the flight manual the cockpit. Fact From Story
Tom sees safety booklet at storage. Fact From Story
Richard moves the flight manual to the office. Because, After finishing his shift, Richard took the manual
with him to his office to review some protocols. Deduced Fact
Lisa did see the flight manual move the office. Deduced Fact
Richard left the area while Lisa was engrossed her teachings. Fact From Story
> Lisa was instructing new flight attendant about beverage service. Fact From Story
When someone is concentrated on task, they are often unaware of the surrounding activities.
Commonsensse Knowledge
Tom the flight manual move the office. Deduced Fact
> Richard was carrying something when they were walking. Fact From Story
> > Tom was walking the same direction as Richard. Fact From Story
When you walk in the same direction as another person, you are usually able to see what they are carrying. Commonsense Knowledge
Tom moves the flight manual the cockpit. Because, As part of his role, Tom made sure the manual was back the cockpit before their next flight. Deduced Fact
> >
Lisa did not see the flight manual move to the cockpit. Deduced Fact
>
Lisa and the pussenger were conversing at considerable distance from Tom and Richard. Fact
From Story
Lisa was absorbed a conversation with a passenger about their travel experience. Fact From Story > Typically, you do not notice activity outside of your immediate conversation when you are fully
immersed in it. Commonsense Knowledge
Richard did not see the flight manual move to the cockpit. Deduced Fact
Tom moved while Richard
in position observe him. Fact From Story
> Richard was in different area performing an essential task. Fact From Story
> If someone is not present in the same location, they cannot witness the actions taking place there. Commonsense Knowledge
Because, Lisa restocking the safety booklets in passenger seating area for incoming passengers. Deduced Fact
Lisa moves the safety booklet the passenger seating
Richard did not see safety booklet move to the passenger seating area. Deduced Fact
Richard was in another section of the airplane working on the pre-flight dossier. | Fact From Story
Lisa was restocking when Richard was busy with document checks. Fact From Story
separate area and task, unlikely to spot any movements outside their area. | Commonsense Knowledge
Busy individuals, especially when focused
> Tom did not see the safety booklet move to the passenger seating area. Deduced Fact
Lisa and Tom were not in the same location at the time. Fact From Story
Tom engaged in review procedures with Richard. Fact From Story
>>> two people are not the same place the same time, they cannot wimess each other's activities.
Commonsense Knowledge
F.3
TEAM ALLOCATION
Listing 11: Team Allocation Example
Amidst the vibrant chaos of the Redwood Zoo, nestled in heart of the city's sprawling jungle. task of assigning roles was crucial cog in the machinery of its operation. As the manager, the responsibility of allocating Olivia, Alex, and Mia to the positions of Animal Caretaker and Exhibit Cleaner presented an intriguing conundrun. Each individual, with their distinct personalities and skill sets, added layer of complexity this assignment puzzle.
Let's 's begin with Alex, the tall lad with bright eyes, whose history with the mighty beast of the animal kingdom, lacked a certain comfort. The lad, known express an almost innate unease around animals larger than him, fell short the prerequisites for Animal Caretaker. His comfort zone extended the four-legged companions our homes, sentiment withheld from the petting 200 section of our park. Yet, his association and collaboration with Mia had seen quite the successes in their high school club', 's fundraising initiatives.
However, his relationship with the gentle Olivia was not seamless. Alex often mentioned feeling ostracized due Olivia's 's tendency maintain her distance. This seemingly innocent avoidance stirred disquiet within
28

---

Published as a conference paper at ICLR 2024
hushed runks, And all this, stemming from disugreement rooted in their previous shared workplaces. Unresolved perhaps, but a factor nonetheless.
Then, there was Mia, the determined bright spark, whose affinity for cleanliness would often bemuse us. She would spend her spare time in her immaculate home cleaning and reorganizing, while her enthusiasm for spotless Exhibit could be underestimated. However, her overly thorough methods would invariably result clashes with Olivia, who criticized her for crossing some form of unspoken boundary.
Mia too had her phobias, the gravelly roars of the zoo's majestic lion had once left her shaken and worried. Loud noises had similar effect leaving her in state nervous terror, much like that of the petite animals held within our barriers. Yet, she all smiles and peasant conversation around Alex during lunch breaks, sharing a sense of humor that lightened the mood of our everyday grind.
Finally, subdued Olivia, a soul strangled with allergies, and deep seated fear for wild animals. An incident with chimp in her past wove tales of nightmarish betrayal, enough to send her away from the animal exhibits during zoo visits. Potent elements of dust and pollen resulted in uncontrolled sneezing fits, a remainder from her days at the school as a custodian, responsible for the cleanliness and maintenance.
Three souls; Animals be cared for, Exhibits be cleaned. Assigning them was always going be an enigma for anyone navigating the zoological labyrinth. Love for animals, discomfort, alliances, conflicts; each factor extraordinarily crucial shaping not just the overall productivity but also the personal growth of each of these individuals at the Redwood Zoo.
Given the story. how would you uniquely allocate each person make both tasks are accomplished efficiently?
Pick one of the following choices:
1 Animal Caretaker: Alex, Exhibit Cleaner: Mia and Olivia
2 Animal Caretaker: Olivia, Exhibit Cleaner: Alex and Mia
Animal Caretaker: Mia, Exhibit Cleaner: Alex and Olivia
You must pick one option. The story should allow you to determine how good each person is at skill. Roughly, each person is either great, acceptable, or bad task. We want to find optimal assignment of people to tasks that uses their skills well as possible. addition, one task will have to have two people assigned to it. The effectiveness of their teamwork (great team, acceptable team, or bad team) also impacts the overall quality of the assignment.
When two people need work task and one is bad at it, they don't necessarily benefit from the other person being good, unless they work well together.
With different strengths, weaknesses, and interpersonal dynamics at play, you should allocate your team to find the single assignment ensure that the tasks overall are completed as effectively possible.
Explain your reasoning step by step before you answer. Finally, the last thing you generate should be ANSWER: (your answer here, including the choice number)"
Listing 12: Team Allocation Example Reasoning Tree
A zoo requires caretakers for different animals and keepers for cleaning the exhibits. Deduced Root
Conclusion
Olivia bad exhibit cleaner. Because we find out in the story that, Deduced Fact
Olivia experienced sneezing fits and allergies during her custodial job a school where cleaning was
part of her task. Fact From Story
Olivia mentioned her allergy to dust and pollen. Fact From Story
Someone who is allergic dust and pollen would have a hard time performing tasks like cleaning
exhibit, which often involves dusting and exposure to pollen. Commonsense Knowledge
Olivia bad at animal caretaker. Because we find out in the story that, Deduced Fact
> Olivia often avoids the animal exhibits during her visits to the zoo. Fact From Story
Olivia has admitted her fear of wild animals due to past traumatic experience with chimp. Fact
From Story
If someone admits fear of wild animals and purposely avoids them, they' unlikely to do a good job an animal caretaker, which requires close and frequent interactions with animals. Commonsense Knowledge
Mia is bad at animal caretaker. Because we find out in the story that, Deduced Fact
Mio mentioned discomforting encounter with roaring lion that left her shaken. Fact From Story
Mia gets nervous and excessively worried at loud noises. Fact From Story
29

---

Published as a conference paper at ICLR 2024
someone gets unnerved by loud animal noises and had bud experience with them in the past, they' re unlikely good animal caretaker as 200s usually have animals that can loud. Commonsense Knowledge
> Mia is good at exhibit cleaner. Because we find out in the story that, Deduced Fact
Mia showed enthusiasm when she found out about the cleaning tasks at the 200, expressing her belief
that the exhibitors need mirror the animals' natural habitatt as closely as possible. Fact From Story
Mia insists on a spotless living environment, often spending free time in her own home cleaning and organizing. Fact From Story
> Somebody who prioritizes cleanliness in their own life is likely to be meticulous in cleaning tasks at work, expecially if they express enthusiasm for the task. Commonsense Knowledge
Alex okay exhibit cleaner. Because we find out the story that, Deduced Fact
> Alex has shown mild interest keeping his surroundings neat, but doesn't go out of his way to ridy up
Fact From Story
> Alex sometimes voluntarily helped with exhibit cleaning when he was volunteer a cal shelter. Fact
From Story
> If someone voluntarily cleans up in past experiences and has moderate interest in it, he or she could probably do okay in a cleaning job, even if they do not excel. Commonsense Knowledge
Alex is bad at animal caretaker. Because we find out in the story that, Deduced Fact
Alex expressed the post desire pursue furthering his knowledge of animals outside of pets. Fact
From Story
Alex admitted that he feels uncomfortable around animals larger than him. Fact From Story
> > someone is uncomfortable around large animals and has no interest in expanding his knowledge of
animals, they probably won't be good at job that involves taking care of a variety of animals. some of which
be large. Commonsense Knowledge
Alex and Mia work okay together. Because we find out the story that, Deduced Fact
> At lunch breaks, Alex and Mia engage in friendly conversations and share similar sense of humor.
Fact From Story
Alex and Mia used to cooperate well in the some high school club, often collaborating fundraising intiaaties. Fact From Story
> If two people have cooperated well in the past and have good social interactions, they are likely to work
okay together. Commonsense Knowledge
> Olivia and Alex work badly together. Because we find out in the story that, Deduced Fact
Alex expressed his discomfort around Olivia, mentioning how her avoidance makes himfeel feel ostracized,
Fact From Story
Olivia avoids Alex during their shared shifts because of an old workplace disagreement. Fact From Story
If two coworkers actively avoid each other due past conflicts, they likely can't 't work together effectively. Commonsense Knowledge
Olivia and Mia work budly together. Because we find out the story that, Deduced Fact
> Mia finds Olivia too passive and non conn-confuntationaa. which results in stewing resentment and lack of open communication. Fact From Story
Olivia has explicit disagreements with Mia's work habits, often criticizing her for cleaning the exhibits too thoroughly. Fact From Story
> If individuals have fundamental disagreements about work ethics and habits and lack of open communication, it is unlikely they will work well together. Commonsense Knowledge
Listing 13: Team Allocation Example 2
As the overseer of the local Poetry Palace, I am privileged to know my poets and judges not just as employees, but also as friends. Today, we found ourselves in the throes of preparing for on upcoming poetry event. A challenging puzzle presented itself: the roles of recitation and scoring needed to be allocated among my dedicated trio: Rachel, David, and Lily.
Rachel, spirited woman with a wide grin, had always been a passionate poet. However, her work habits could be called into question, according David. She tended to be more laid back and unstructured, which David considered a flaw. Lily too, had tangled with Rachel the past, when she had offered some critiques on Rachel's 's poetry - critiques that were not well-received, leading to heated argument and a grudge that still lingered between them. Rachel's 's reaction reflected her struggle with accepting feedback from others. Her tendency judge poetry personally over objectively, even letting her opinion of poet color her scores, was also issue.
David, on the other hand, was a connoisseur of the poetic word. He boasted a deep understanding and appreciation for wide spectrum of poetry styles, which revealed itself when he shared comprehensive and incisive feedback with poets. Yet, David had flaws of his own. He was known for his sarcasm, trait particularly hurtful Lily due remarks about her mild stutter. Tensions between them had escalated into
30

---

Published as a conference paper at ICLR 2024
silent disagreement. Moreover, while David's knowledge of poetry was vast. his voice was the musical instrument required for engaging recitation. His monotone delivery and self-conscious fear of boring people made him shy away from recitations.
Lastly, there was Lily. Although her speech hore the unique quirk of mild stutter which became emphasized when she was nervous or faced a large crowd, she was an ardent poetry enthusiast. At home, she had a routine of reading and analyzing poems, propelling her understanding of poetry. She had earned her stripes by taking literature classes in college, including a course dedicated entirely to poetry. However, David's s sarcastic remarks about her stutter had marred her morale, and she was already apprehensive about performing in front large crowds.
As watched them, my thoughts spun with the complexity of their dynamic- the strengths and weaknesses of each individual, the silent feuds feuds, and shared enthusiasm for poetry. Balancing it all was tough job, but as the manager, the responsibility sut squarely on my shoulders. The event was quickly approaching and I had decide who would recite and who would score- decision that, hoped, would inspire personal growth, heal strained relationships, and ultimately make the event a success.
Given the story, how would you uniquely allocate each person make sure both tasks are accomplished efficiently?
Pick one of the following choices:
Recitation: David, Scoring: Lily and Rachel
2 Recitation: Lily, Scoring: David and Rachel
3 Recitation: Rachel, Scoring: David and Lily
You must pick one option. The story should allow you to determine how good each person is at skill. Roughly, each person is either great, acceptable, or bad at a task. We want to find an optimal assignment of people to tasks that uses their skills well possible. In addition, task will have to have people assigned it. The effectiveness of their teamwork (great team, acceptable team, bud team) also impacts the overall quality of the assignment.
When two people need to work on a task and one is bad at it, they don't It necessarily benefit from the other person being good, unless they work well together.
With different strengths, weaknesses, and interpersonal dynamics at play, you should allocate your team to find the single assignment to ensure that the tasks overall are completed as effectively as possible.
Explain your reasoning step by step before you answer. Finally, the last thing you generate should be ANSWER: (your answer here, including choice number)"
Listing 14: Team Allocation Example 2 Reasoning Tree
A poetry event needs poets to recite and judges to score. Deduced Root Conclusion
> Lily is okay at scoring. Because we find out in the story that, Deduced Fact
> Lily often reads poems at home and analyzes them in her free time. Fact From Story
Lily took few literature classes in college including Poetry and Literature 101. Fact From Story
If person has studied similar subject and practices it often, they may have at least some competence
Commonsense Knowledge
> Tily is bad at recitation. Because we find out in the story that, Deduced Fact
Lily has mild stutter that becomes more pronounced when she is nervous. Fact From Story
Lily often gets stage fright when speaking in front of large crowd. Fact From Story
a person gets nervous in fromt a crowd and has a speech issue, like stattering, they're re likely to
bad at reciting poetry in front of an audience. Commonsense Knowledge
> Rachel is bad at recitation. Because we fund out in the story that, Deduced Fact
When Rachel gets feedback, she struggles take it heart and improve. Fact From Story
Rachel has been told before that her rhythmm and pacing when reading poetry off. Fact From Story
> > If person cannot maintain rhythm and pacing while reciting poetry, and they do not take feedback well,
they will likely be poor at recitation. Commonsense Knowledge
Rachel and Lily work badly together. Because we find out in the story that, Deduced Fact
Rachel holds a grudge towards Lily after this incident. Fact From Story
Lily gave Rachel critique on her poetry once which resulted an argument. Fact From Story > If two people have held a grudge over a work-related incident, they Il likely struggle to work effectively together. Commonsense Knowledge
Rachel and David work badly together. Because we find out in the story that, Deduced Fact
David finds Rachel's work habits be too laid back and unstructured. Fact From Story
31

---

Published as a conference paper at ICLR 2024
Rachel feels that David doesn'tt respect her opinions and ideas. Fact From Story
When two people have differing work habits and do not respect each other's methods, they rypically don'
work well together. Commonsense Knowledge
David and Lity work badly together. Because we find out in the story that, Deduced Fact
Lily has overheard David making these remarks and it has led to silent disagreement between them.
Fact From Story
David makes sarcastic remarks about Lily's stutter which hurts her feelings. Fact From Story
> If workers have unresolved disagreements or conflicts, they usually have trouble cooperating and
working well together. Commonsense Knowledge
David is bad at recitation. Because find out the story that, Deduced Fact
David never practiced recitation fearing he would bore people. Fact From Story
David has a monotone voice that lacks the inflection needed for engaging poetry recitation. Fact From Story
A monotone voice and lack of practice in recitation can make a person bad at reciling poetry. Commonsense Knowledge
C
David is good scoring. Because we find out the story that, Deduced Fact
David has a deep understanding and appreciation for a variety of poetry styles. Fact From Story
David shares comprehensive and constructive feedback with poets. Fact From Story
someone is well- in poetry and has knack for providing constructive criticism, they would
likely be good judge poetry event. Commonsense Knowledge
Rachel is bad at scoring. Because we find out in story that, Deduced Fact
> > Rachel admitted to having difficulty separating her personal biases from her judgement of the poetry.
Fact From Story
Rachel often makes
based her personal feelings about the poet, the poem' content
structure. Fact From Story
> Ifa person admits they struggle to judge work objectively due to their feelings towards the individuals involved, they will likely be bad at judging a competition objectively. Commonsense Knowledge
G BUILDING THE DOMAINS
G.1 TREE CONSTRUCTION ALGORITHM
Algorithm The recursive reasoning tree expansion algorithm.
The scenario s. The current reasoning tree T. Output: Tree expanded in-place. function CREATEDEDUUCTION!? node, max_depti depth) for child in node.children do child.children PrompILM(TI.... ...., Tm s, child, G) if (ALLDATTE(child.chhldeee and depth max, max_depth then CREATTEDEDDCTTION(T child, max max depth, depth + 1)
Input:
G.2 MURDER MYSTERY
Tree construction prompt (per deduction)
Listing 15: Murder Mystery Deduction Prompt for the Tree Completion stage
Your task is generate logic tree for story, shown in the example. In this tree, each fact should be deduced from its immediate children. If a deduced fact already a name, not overwrite it.
Type of story:
We creating a murder mystery. A murder mystery needs to have complex web of evidence that leads means, motive, and opportunity for suspect, which will make them likely murderer. When writing a murder mystery, the story should take the point of view of the detective. Evidence should be collected through investigation, including things like interrogation, hearing conversations, reading past criminal records. looking at mail or trash, and other normal modes detecting evidence.
Each fach in the tree must follow via logical deduction from its children. 2. All Fact From Story nodes and the Commonsense Knowledge node must be relevant to the deduction they yield.
Each root fact labeled with source (Fact from Story Commonsense Knowledge).
32

---

Published as a conference paper at ICLR 2024
4. A Fact From Story should be statement about character, place, object in the story. Commonsense Knowledge should be a fact that most people will know and agree with. It should not explicitly reference any characters in the story.
6, Commonsense Knowledge should be used as deduction rule that when the sibling facts are applied to it they yield the parent deduced fact.
The you generate must match the structure of the tree give you.
A suspect has a means when they have access to the murder weapon.
A suspect has motive when the suspect has reason to kill the victim.
A suspect has opportunity when they the crime scene.
Here's an example.
Scenario:
Victim: Victoria
Crime scene: Home
Murder weapon: Gun
Suspect: James
Suspect's story: Brother
Suspect's 's motive: Financial gain
Current Tree:
James is murderer. Deduced Root Conclusion
James has a means Deduced Fact
James has practiced shooting guns. Fact From Story
James guns Fact From Story
If you own and practice using guns then you have ability to murder someone. Commonsense
Knowledge
James has a motive. Deduced Fact
James violently desperate for cash. Fact From Story
James was violently desperate for Victoria' 's cash Fact From Story
When someone violently desperate they may go to extreme measures to accomplish task, including murderer. Commonsense Knowledge
James has opportunity. Fact From Story
Entailment Step to complete:
James has a opportunity.
Faet From Story
Commonsense Knowledge
Output:
James has a opportunity.
James has access Victoria's 's house. Fact From Story Having someones house gives you the opportunity murder them. Commonsense Knowledge
Here is another example.
Scenario:
Story Information:
Victim: Harry
Crime scene: Racetrack
Murder weapon: Shovel
Suspect: Claire
Suspect's role story: Running buddy
Suspects motive: To prevent someone else harm
Current Tree:
Claire a marderer. Deduced Root Conclusion
Claire has a means. Deduced Fact
Claire is a farmer Faet From Story
Farmers typically gardening tools like shovels in their work. Commonsense Knowledge
33

---

Published as a conference paper at ICLR 2024
Claire has motive. Fact From Story Claire has an opportunity
Fact From Story
Entailment Step to complete:
Claire has motive.
Fact From Story
Fact From Story
Commonsense Knowledge
Ousput:
Claire has motive.
Claire loves Brian deeply. Fact From Story
Harry threatened Brian. Fact From Story
> Deep and passionate love can push people do exireme things like murder when that loved one threatened. Commonsense Knowledge Here is another example.
Scenario:
Victim: Jared
Crime scene: Public park bench
Murder weapon: Heroin overdose
Suspect: Jose
Suspect's role story: Drug
Suspects motive: Public humiliation
Current Tree:
Jose is murderer. Deduced Root Conclusion Jose has means. Fact From Story
Jose has motive Fact From Story
Jose has opportunity. Fact From Story
Entailment Step to complete:
Jose has means.
Fact From Story
Fact From Story
Commonsense Knowledge
Output:
Jose has means.
Jose has access heroin. Fact From Story
Jose knows how much heroin needed for an overdose. Fact From Story Having access to heroin and knowing how much heroin required to overdose implies you could have intentionally given the victim dose of lethal heroin providing a means for murder. Commonsense Knowledge
Your Turn.
Scenario: Victim: Tessa
Crime Scene: kitchen
Murder Weapon: poisonous gas
Suspect: Penelope
Role in story: Tarot Reader
The suspect's motive: To protect secret
Current Tree:
Penelope is the murderer. Deduced Root Conclusion
Penelope has means. Deduced Fact
Penelope has opportunity. Deduced Fact
Penelope has motive. Deduced Fact
Entailment Step to Complete:
Penelope is the murderer.
Penelope has
Because.
34

---

Published as a conference paper at ICLR 2024
Fact From Story
Fact From Story
Commonsense Knowledge
Output:
Validator prompt for GPT-3 and GPT-4
Listing 16: Murder Mystery Deduction Validation Prompt for the Tree Completion stage
so we are creating narrative guide of evidence. We are proving a means (access to the murder weapon) right now. Does this deduction in any way prove or help to prove a motive freason to kill) or an opportunity (access to the crime scene) given the description of the mystery below.
writing a murder mystery
Victim: Tessa
Crime Scene: kitchen
Murder Weapon: poisonous gas
Suspect: Penelope
Rote story: Tarot Reader
The suspect's motive: To protect a secret
The Deduction:
Penelope has
Penelope was wimessed purchasing an unusual quantity chemicals day prior. Fact From Story > Records show Penelope studied chemistry and alchemy extensively. Fact From Story
> One can use their knowledge of chemistry and alchemy to create poisonous gas if they possess the necessary chemicals. Commonsense Knowledge
Write your answer in the the following format: ANSWER: (yes/no)
Intro prompt (introduce victim and suspects)
Listing 17: Murder Mystery Intro Creation for Story Generation
Create intro for this murder mystery. It should only be or 2 sentences. Only write the intro nothing else.
Scenario:
Tessa was killed with poisonous gas at kitchen, Detective Winston is on the case, interviewing suspects. The suspects are Penelope, Melody.
Output:
Chapter prompts
Listing 18: Murder Mystery Chapter Prompt for the Story Generation stage
We are creating a murder mystery. A murder mystery needs to have complex web of evidence for suspect. When writing murder mystery, the story should take the point of view of the detective. Evidence should be collected through investigation, including things like interrogation, hearing conversations, reading past criminal records, looking mail trash, and other normal modes of detecting evidence.
We will give you a list of facts to use when writing your story. You must include all the facts in the list. Never state deduced facts conclusions. The story should stick to the fact list pretry closely.
You will write a chapter for the murder mystery. This should not introduce the murder or victim, but strictly follow a detective slowly interviewing and detecting the suspect. Use the list below as a guide.
Rules:
Only write the contents of the chapter, do not write a title or number This text must easily placed in
larger story.
2. Never say suspect has means
3. Never say suspect has motive
Never say suspect has an opportunity
35

---

Published as a conference paper at ICLR 2024
Never hint a suspect having having means, molive, or opportuntly. 3. Never hint or explicitly say deduced fact.
Never say a suspect is murderer. It' a puzzle that the reader supposed to guess and deduce!
7. Write the story from the perspective of detective uncovering clues through various methods that normal detectives use (internogation, notes, stake-outs, etc.)
8. Never make mental notes, point suspicious facts, nor make connections. Let the reader do this.
Include as many sentences as needed to write each fact from the list of facts. Also include up to 10 sentences of dialogue.
Here is an example:
Suspect and crime information
Victim; Doru
Crime Scene: Remote forest
Murder Weapon: Knife
Suspect: Willard
Role in story: Grounds keeper
The suspect's motive: Religious sacrifice
are detective Winston.
Facts you must include:
A witness saw someone with spaghetti face and green ears
Willard groundskeeper for local school
Willard also providex services to nearby residents house painting, lawn care. etc.
Willard painted nearby home green.
Willard was in horrible fire when he was a child.
Willard' family has been in the local area for multiple generations
Long ago, the local area had religious extremists around them, all participating in occult activities.
Willard and his immediate family were all handymen of some sort.
Willard believes in respecting his elders and ancestral history.
Dora had written about joining new church nearby.
A friend Dora mentioned worrying about Dora's 's involvement with new cult-like group of friends.
Output:
Winston took long drag from his cigarette while reviewing the crime scene photos. He was hardened to common grotesqueries of his line of work, but the murder of Dora sparked something in him... calling into the void.
The only hard evidence he had in this case monster. That, and the fact that Dora had been exploring some church... perhops cult, depending who you asked, Cults had history of leaving bad taste people' mouths here... having some very dark pasts in the local area.
eyewitness of a monster... spaghetti face, green - eared
Winston put his cigarette out, placed the photos down, and set out for his following suspect interview with Willard.
The smell of fresh-cut grass washed over Winston as the local groundskeeper of the local elementary shut off his mower.
"Howdy, Mister," Willard said with rich southern twang.
"Howdy..." Winston said carefully, trying to mutch the southern pleasantries he wasn't It all too familiur with.
"You, Willard?" Winston inquired.
"Well, sure am!" He chuckled, "Been him for while now, suppose."
"You do lot of work for the school here?"
have been helping out the local schools and locals with standard chores noticing Winston motioning for him to continue, Willard explained farther, "You know, like painting houses, cutting people's 's lawn... well heck, just couple days ago painted that green house over yonder." Willard said, pointing across the schoolyard.
"Heh, yes, sir, My family
36

---

Published as a conference paper at ICLR 2024
Winston couldn't help but notice that Willard had a severe burn sear across the bottom half of face, which had previously been hidden by sweat and grass shavings.
=Ring= =Ring= away.
Winston was getting called away, back to the office, so he had to make quick excuse to get
"You know the way back to Highway 45?" Winston asked. "Sure," Willard replied with grin, "You take two lefts here, then about 12 miles down, you 'Il hit it."
Winston smiled, "You do know your way around here," " he said. "Hah, well, sir, my families have been here for... for a long, long time. take great pride my ancestry and family history here..."
With that, Winston in his cur and left.
Your turn.
Suspect and crime information
Victim: Tessa
Crime Scene: kitchen
Murder Weapon: poisonous
Suspect: Penelope
Role story: Tarot Reader
The suspect's motive: To protect a secret
You are detective Winston.
Facts you must include:
Tessa found incriminating letters connecting Penelope to past crimes Penelope was seen leaving Tessa's place immediately before the time of the murder.
Records show Penelope studied chemistry and alchemy extensively. Penelope was seen arriving at Tessa's house before the murder took place.
Penelope was witnessed purchasing an unusual quantity of chemicals the day prior. Penelope reacted strongly with fear when Tessa confronted her about the letters Penelope seen reading books about poisonous gases
Penelope alone in Tessa's kitchen during the tarot reading session. No one saw Penelope leave Tessa's 's place before discovery of murder.
Penelope was cryptically speaking about the perfect crime that leaves no truces Tessa hud confronted Penelope about those past crimes.
Tessa knew about Penelope's 's past crimes.
Output:
During the Tree Template Construction stage, sample the victim's name, the murder weapon, and the crime scene. For every suspect, we also sample the suspect's name, motive, suspicious fact, and their role in the story (usually a job relation to the victim). We can use the sampled information from the Tree Template Construction stage to create the gold fact set F.
Once we have all these sample, we choose one suspect be the murderer; this suspect will have the facts: "[suspect] has a means", "suspect has a motive", and "suspect has an opportunity" to the gold set of facts F, All other suspects have two the "MMO" facts (the ones listed above); the third fact for innocent suspects is suspicious facts that do not contribute to solving the murder (for example, "[suspect] enjoys reading murder mystery novels.")
Once each suspect has three facts in the set F, can begin the tree construction process. However, in the prompt, we ensure that the details of the suspect (like their motive) are included so that when build tree for the fact "[suspect] has motive," the reasons for having motive are still linked to the sampled motive from the Tree Template Construction stage.
Because many of the trees already made for each suspect. we will create "contrastive example" for each mystery where we simply set the murderer to another suspect. We can then reuse most of the trees from the previous mystery and only have to prove two ones and then generate the story.
37

---

Published as a conference paper at ICLR 2024
We use validators during the reasoning tree construction process ensure that the deductions being generated are correct and meaningful. When proving a means, we check the generation from GPT-4 for any mention of "motive" or "opportunity" including key items in the story that might relate to them (like the crime scene). If these are mentioned, we re-prompt GPT-4 for another deduction. We repeat this process for motive and opportunity.
To further increase the validity of the murder mysteries generated, we also implemented a story validator during the narrative generation. The story validator prompts GPT-4 to check the entailment of each scenariO-specific fact S(T) given the story. If any fact is not entailed by the story, we prompt GPT-4 to rewrite the story and ensure that fact is entailed.
Suspicious Facts The suspicious facts were useful control for a length bias where the murderer was described in longer and more explicit terms than the non-murderer. We also considered nega- tions of facts like "Sophia has no motive to kill Emily", but these often led easy mysteries where GPT-4 would explicitly rule out suspect. Suspicious facts blend in with the narrative, provide balance, and do not modify the ground-truth reasoning.
G.3 OBJECT PLACEMENTS
Listing 19: OP scenario creation for generating items, people, roles, etc. in the Domain Injection stage
Use the given scenario description to create a list of objects, people, and locations as well as where some items are currently located and the correct items that people want. The idea is that someone really needs something in this scenario.
Include one item and location that are relevant to scene but aren't necessurily relevant to scenario. For example, coffee beans might be relevant to a coffee shop, but not so relevant for the scenario of someone looking for the milk.
In the correct assignment and outcome, the person who gets an item in the clause must be different than the person in the Then clause. See the example.
Rules:
1) You must names of real people, describe their role in the story after the comma, For example, "Luis, customer" or "Adam, lawyer" etc. always use real names "Berry" or "Cynthia".
2) You must use tangible small items, not use ideas "a performancc"" for example, do not use large items like "a tv". Instead use small, easily moved items like "Iphone", ", "Notebook", "Laptop" etc.
3) The locations you pick must be able to house the items you made. So you said "Golf club" was an item, all locations must be able to fit golf club in them, you would say "coat pocket" for example.
Here's an example
Description: Sarah is making coffee at her work, she wants to almond milk for her customer.
Output:
Items: almond milk; regular milk; almond milk coffee; regular milk coffee; coffee beans
People: Sarah, a barista: Luis, customer: John, customer
Locations: Fridge; Shelf: Counter; Storage Closet
Located: Almond Milk -> Fridge
Located: Regular Milk -> Counter
CORRECT -- format (If: someone -> (gets) Something. Then: Another person -> (gets) something they wanl)
If: Sara -> Almond Milk
Then: Luis Almond Milk Coffee
Your turn!
Description: Mason, magician, has his magic deck of cards which essential for his performance.
38

---

Published as a conference paper at ICLR 2024
Ouiput:
Listing OP Prompt for generating plausible moves the Domain Injection stage.
You going continue story that have written by writing short description of this event that will happen next. Only write about the move, do add any additional information.
Never say "someone didn't see something" or infer someones ability to infer where something is. Never say Unbeknownst" or anything like this!
Here an example.
Only write one or two sentences. It should be a very short continuation.
Description: Timmy angry at Bob for cheating his way into the job Timmy deserved! So he started throwing away Bobs possessions.
Character:
Name: Timmy
Role in story: A recent graduate who is sharing apartment.
Motivation in story: Timmy is angry because interviewed for job that roommate got, but only because he cheated.
Event:
Timmy maves the keys to the trash bin. Because, Timmy was angry with Bob and wanted throw away his keys.
Timmy saw the iphone at the trash bin when moving the car keys.
Ouiput: With angry thrust, the keys clanked against the tin trash bin. An unexpected esmack* followed though... curiosity overtaking his anger, Timmy looked the trash saw the iphone there as well.
Here is another example.
Carol had just moved into her apartment, but, the previous tenant made huge mess! The landlord wouldn't anything, so looks like has to clean all up herself.
Description:
Character:
Name: Carol
Role story: Just moved into new messy apartment. Motivation in story: Carol wants to clean her new apartment that was left mess by the previous tenant has exactly no help from monagement.
Event:
Carol moves the noodles the pantry. Because, Carol was excited have a clean apartment finally, and the noodles were the last step!
Output: Carol excitingly places the noodles back into the pantry. What once thought of as ending onslaught of trash and random items finally concluded and the apartment was finally clean again!
Your turn.
Ricky, with his weathered notebook in hand, ready for a full day's studio session. His aim: perfect his upcoming album, for which notebook was crucial. Emma, producer confidante, eager to bring Ricky's ' lyrics to life, shared Ricky's 's enthusiasm for the project. Meanwhile, Danny the diligent studio assistant - was making sure all elements of the studio were meticulously managed for the recording session.
Description:
Character:
Name: Emma
Role in story: Music Producer
Motivation in story: Emma wants to support Ricky's 's creative process and record outstanding music to advance both Ricky's career and her own reputation successful music producer.
Event:
Emma moves the earphones to the producer's desk. Because, Emma decided adjust some sound settings and needed the earphones her desk for that.
39

---

Published as a conference paper at ICLR 2024
Ouiput:
In the heart of the bustling studio, Ricky, Emma, and Danny readied themselves for a day of creating magic. Ricky, holding the helm as the gifted singeresongwrite, was poised for perfection, his precious notebook of lyrics awaiting its call to duty on the producer's 's desk. Emma, their dutiful and talented producer, was just as eager to breathe life into Ricky's lyrics. She was cognizant of the notebook' s place at her desk, awaiting the melodies they would cultivate together.
Across the room, Danny, the diligent studio assistant, was doing his due diligence, keeping the earphones nestled in the recording booth. His aim was to ensure an optimized and meticulous environment for recording, testament his commitment their shared mission. They were all aware of the arrangement the notebook on the producer's 's desk, the earphones the recording booth. Their shared consciousness of these items only intensified the anticipation; they were eager to turn the contents of a weathered notebook into a world-class album.
Ricky, with his weathered notebook of potent lyrics in hand, gently places it onto the piano. An air of creativity and anticipation lingers the room, everyone aware creation of their masterpiece. In syne with the palpable creative energy, Ricky was engrossed in perfecting the rhythm of his song, preparing himself for an intense day ahead. Not too far away, Emma was sincerely engrossed in her role of musically steering the session. She focussed on Ricky's progress, her eyes constantly monitoring him and her mind alive with ideas to enhance the music.
was the first instrumental step in
Meanwhile, Danny was diligently covering every corner of the studio. He was making his rounds, ensuring that the studio was prim and proper for Ricky's crucial session. As part of his tasks, he pussed by Ricky several times, always careful not to interrupt the artist's s flow.
Listing 21: OP deduction prompt for the Entailment Tree Construction stage.
We are creating a story where people are going to move of items muny times. The goal of the story is be interesting and unique but also have a clear tracking of where objects are so we can quiz readers and language models later on about the world state and about who knows what.
To make this story, job is to fill out entailment trees that prove a person saw or not see an event happen.
created tree structure that outlines the narrative and updates the world. Your
An entailment tree is a tree structure where intermediate nodes are entailed by their children. They create a natural language reasoning proof for some collection of facts.
To fill out this tree we need to complete on entailment. Completing an entailment is akin to filling out one subtree of the entailment tree. To fill in this step, you must follow the structure of the step.
Facts From Story are facts that will explicitly stated when we write the story. Commonsense Knowledge are facts that most people would agree are true and don't ' need to be explicitly said
Complex Facts are facts that will be entailed by simpler facts from the story (they will be filled in later through a recursive call back to you!)
All facts for the step must combine to entail the root parent fact.
No facts may contradict the current structure tree.
Do not include facts about other people,focue on the facts for the person who is seeing or not seeing something move.
Always match the exact structure of the entailment step give you. Give the same number of Facts From Story and Commonsense Knowledge facts. Give them in the same order as well.
Never explicitly say someone didn't 't see something or did see something. Your job is to provide facts that suggest this, For example, if May saw Greg move something, we might say "May was watching Greg while doing her chores", ", and that "by watching someone, you are seeing what they are doing". Notice describe the physics of seeing something, but we don't outright say that someone saw something.
Never mention item being moved or reuse item. For example, if theres fact like "Greg didn't see his iphone" " and you are proving why "Joel didn't the apple move"" ", never reuse Greg' 's iphone in your facts.
40

---

Published as a conference paper at ICLR 2024
Our program strictly controly where items placed, we don't I want you introducing item placements haven't accounted for.
Each fact should be crucial to the deduction. Intentionally leave out details so that the other facts can account for them. If one fact is missing, the conclusion should not be entailed. Try not reuse the same facts.
Always use the persons name instead of a pronoun like "He" or "She", if you know someones name, use the name.
Only perform deduction time. Your deduction should match the "Entailment Step to Complete" template exactly so we can parse it later
Here's an example.
Scenario:
Paul and Alice are at karaoke bar.
Current Tree:
Poul and Alice are karaoke bar. Deduced Root Conclusion
Opening Scene Deduced Fact
Paul sees the microphone at stage. Fact From Story
Alice sees the microphone at the stage. Fact From Story
Paul sees the beer at the bar.
Fact From Story
Alice the beer at the bur. Fact From Story
Paul moves the beer to table.
Deduced Fact
Alice did not see the beer move the table. Deduced Fact
>
Alice was facing away from the table. Deduced Fact
Alice talking another patron. Fact From Story The other patron was facing the table. Fact From Story other, so if one person looking in one direction the other person is looking in the opposite direction. Commonsense Knowledge
> >> >> Usually people talk to each other while facing
> > If someone is facing away from something else, they cannot see things transpire near that something
else. Commonsense Knowledge
Alice moves the microphone to table Deduced Fact
Alice saw the beer at the table when moving the microphone. Fact From Story
Paul saw the microphone move to the table. Deduced Fact
Paul was drinking the beer the table. Commonsense Knowledge
When something happens where person at, they usually see things that happening there. Fact
From Story
> Alice moves the beer to the trush can. Deduced Fact
Puul did not see the beer move to the trush can. Fact From Story
Entailment Step to complete:
Paul and Alice are at karaoke bar.
Alice moves the beer to the trush can. Because in the story we find out,
Puul did not see the beer move to the trash can. Because in the story we find out,
Fact From Story
Fact From Story
Commonsense Knowledge
Output:
Paul and Alice are karaoke bar.
Alice moves the beer to the trash can. Because in the story we find out,
> Puul did not see the beer move to the trush can. Because in the story we find out,
Alice tricked Paul into looking "over there". Fact From Story
Alice pointed in the opposite direction of the trash
Paul. Fact From Story
If you trick someone into looking elsewhere, they cannot see what happens the other direction. Commonsense Knowledge
Your Turn.
Scenario: Mason, magician, has magic deck of cards which are exsential for his performance.
People in the story:
Mason, magician
41

---

Published as a conference paper at ICLR 2024
Adam, audience member Patricia, an assistant
Respect the peoples roles when creating deductions and try to create motivations for them in the story that would influence if they see or do not see event.
Current Tree:
Mason, magician, has his magic deck cards which are essential for his performance. Deduced Root Conclusion
opening scene Deduced Fact
mason the top hat the props table. Fact From Story
mason sees the magic deck of cards the backstage room. Fact From Story
adam sees the top hat at the props table. Fact From Story
adam sees the mugic deck of cards at the backstage room. Fact From Story
patricia the top hut the props table. Fact From Story
patricia sees the magic deck of cards at the backstage room. Fact From Story
mason moves the top hat to the stage. Deduced Fact
adam did not see the top hat move to the stage. Deduced Fact
patricia did not see the top hat move the stage. Deduced Fact patricia moves the magic deck cards the props table. Deduced Fact mason saw the magic deck of cards move to the props table. Deduced Fact > adam saw the magic deck of cards move to the props table. Deduced Fact
mason moves the magic deck of cards to the stage. Deduced Fact
adam did not the magic deck of cards move to the stage. Deduced Fact patricia saw magic deck of cards move to the stage. Deduced Fact
Entailment Step to Complete:
Mason, magician, has his magic deck of cards which essential for his performance. mason the top hat to the stage. Because in the story find out,
adam did not see the top move to the stage. Because the story we find out,
Complex Fact
Complex Fact Commonsenss Knowledge
Output:
Listing 22: OP Intro generation prompt for the Story Generation stage
Create an opening scene description for story. It will be short, Only write about the objects we list out and their location. Your story MUST include each item and their location from the list. Your story also MUST indicate that all the people give you the location all these isems!
You may use the description to infer the correct scenery to describe, but are only allowed to talk about the facts presented in the list.
You must state that everyone knows where everything is, "They were all aware of each items location" " or story
something like that is safe way to ensure this condition met. Try to make this coherent though. For example, if someone doesn't 't know the exact location you could say "Everyone was aware that the item was somewhere in the location, and they definitely all knew that the other item was in the other location something like this.
Here is an example.
Description: Alex is making Ramen and needs the noodles cook.
Items and Locations:
The pans are at the stove.
The noodles are the fridge.
The kitchen knife the table.
Character 1:
Name: Alex
Role in story: A want to he chef
Motivation in story: To make bowl ramen.
42

---

Published as a conference paper at ICLR 2024
Character 2:
Name: Carol
Role in story: the roommate
Motivation in story: Hanging out with Alex, she is also hungry.
Character 3:
Name: Joshua
Role in story: A random visitor
Motivation in story: Joshua friend Alex but showed up unannounced and hungry.
Output: Alex and Carol were having a peaceful evening hanging out. Both getting a bit peckish, they decided to eat some ramen, which Alex had been practicing making for awhile now. Everyone knew Alex was the Chef Friend", meaning that he was always cooking something delicious up. In fact, that ' why hungry friend, who showed up unannounced. Joshua decided to show up. All three of them noticed that the pans were already on the stove, perfect for ramen making! kitchen knife was on table, and the noodles were the fridge.
Your turn.
Description: Ricky, with his weathered notebook hand, was ready for a full day's studio session. aim: perfect his upcoming album, for which his notebook was crucial. Emma, his producer and confidante, eager to bring Ricky's 's lyrics life, shared Ricky's 's enthusiasm, for the project. Meanwhile, Danny the diligent studio making all elements the studio meticulously managed for the recording session.
assistant
Items and Locations:
the notebook is at the producer's desk
the carphones are at the recording booth
Character
Name: Ricky
Role in story: Singer-Songwrite
Motivation in story: Ricky is motivated create his best music using his trusty notebook filled with personally crafted lyrics.
Character 2:
Name: Emmaa
Role story: Music Producer
Motivation in story: Emma wants to support Ricky's creative process and record outstanding music
advance both Ricky's 's career and her own reputation as a successfid music producer.
Character
Name: Danny
Role in story: Music Studio assistant
Motivation in story: Danny's aspiration is to maintain organized and conducive environment for Ricky's 's recording session.
Output:
Listing 23: OP Prompt for generating short description of why and how someone moved an item.
Write short description of this event happening. Only write about what is happening.
You will never add additional information! Never say "someone didn't see something" " or infer someones abiliry infer where something Never say "Unbeknownst" anything like this!
Here is an example.
Facts:
Timmy moves the keys the trash bin.
Timmy saw iphone at the trash bin when moving the car keys.
Being ever clever and funny.. in his own way... Timmy moved the keys to the trash bin. While he was doing that, he couldn' 't help but nofice someone had put the iphone in the trash can too.
Output:
43

---

Published as a conference paper at ICLR 2024
Here is another example.
Facts: Carol moves the noodles to the pantry.
Output: Carol, wanting be cleanty, moved the noodles into the pantry.
Your turn.
Facts:
mason moves the top hat the stage.
Output:
Listing 24: OP prompt for creating paragraph people's observations.
Continue the story we have written so far by writing about the observational facts below. Only write about the facts and do not add new information. Never "Someone saw" or "Did not notice" and never indicate if sees something, unless the only fact you have is "someone X".
Stick to the facts, there will be more information about the story that you can use to set the tone, but you should always use the facts as the main guide for the story.
Never mention the key items the story:
earphones
notebook
There will be another event after this paragraph. end this paragraph abruptly sticking only with the facts. Make no general statements.
sentence should be something about the facts we listed out only. It should be a complete sentence.
Your story should take place during the most recent move. So while this is happening:
Emma,, engrossed her thoughts, deftly moves the earphones the producer's desk. She preparing tweak the sound settings, pre-empting Ricky 's need for perfect audio in his performance."
the facts you will be writing about happening the same time.
Here is an example.
Description: Jerry, Marry and Timmy getting ready for the day. Jerry has huge meeting that he needs prep for. Marry is excited to help Jerry for his meeting and hear about later that day. Timmy was getting ready for his test, but being a bit inconsiderate to dad, Jerry, with respect to big meeting.
Character I:
Name: Jerry
Role story: the husband
Motivation in story: Jerry had a huge meeting coming up, one that could decide the fate of his career.
Character 2:
Name: Marry
Role story: wife
Motivation in story: Marry always super supportive and wants the best for her family.
Character 3:
Name: Timmy
Role story: the son
Motivation in story: Timmy has a huge test coming in his school which he is nervous for and accidentally makes him a bit inconsiderate to everyone else.
Observational Facts:
Jerry is cooking breakfast
The trush bin is not in the kitchen.
Marry is outside watering her garden.
Marry has window into the with the trash bin.
44

---

Published as a conference paper at ICLR 2024
Jerry was hungry before starts his day, so he was cooking his breakfast. The kitchen turned out not have the trash bin though. Marry, always with her green thumb, was watering her garden and could see the trash bin through a nearby window.
Output:
Your turn.
Description: Ricky, with his weathered notebook in hand, was ready for a full day's studio session. His aim: perfect his upcoming albun, for which his notebook was crucial. Emma, his producer and confidante, eager to bring Ricky's 's lyrics life, shared Ricky's enthusiasm for the project. Meanwhile, Danny the diligent studio assistant was making all elements the studio were meticulously managed for the recording session.
Character 1:
Name: Ricky
Role in story: Singer-Songwriee
Motivation in story: Ricky motivated to create his best music using his trusty notebook filled with personally crafted lyrics.
Character 2:
Name: Emma
Role story: Music Producer
Motivation in story: Emma wants to support Ricky's 's creative process and record outstanding music to
advance both Ricky's career and her own reputation as a successfid music producer.
Character
Name: Danny
Role in story: Music Studio assistant
Motivation in story: Danny's aspiration maintain organized and conducive environment for Ricky ss recording session.
Observational Facts:
Danny was engrossed in a conversation with visiting sound engineer.
Ricky absorbed in lyric composing session.
The lyric composing session took place a different area than where Emma was.
The visitor was standing a spot blocking Danny's 's view.
Output:
the heart of bustling studio, Ricky. Emma, and Danny readied themselves for a day of creating magic. Ricky, holding the helm as the gifted singer-songwriee songwriter, was poised for perfection, his precious notebook of lyrics awaiting its cull to duty on the producer's desk. Emma, their dutiful and tulented producer, was just as eager breathe life into Ricky's lyrics. She was cognizant of the notebook 's place at her desk, awaiting the melodies they would cultivate together.
Across the room, Danny, the diligent studio assistant, was doing his due diligence, keeping the earphones nestled in the recording booth. His aim testament his commitment their shared mission. They were all aware the arrangement- the notebook on the producer's desk, the earphones
ensure optimized and meticulous enviromment for recording,
recording booth. Their shared consciousness of these items only intensified the anticipation; they were eager to turn the contents of a weathered notebook into a world-class album.
Ricky, with his weathered notebook of potent lyrics hand, gently places onto the piano. creativity and anticipation lingers the room, everyone aware that this was the first instrumental step in the creation of their masterpiece. In sync with the palpable creative energy, Ricky was engrossed in perfecting the rhythm of his song, preparing himself for an intense day aheud. Not too fur away, Emma was sincerely engrossed in her role of musically steering the sesxion. She was focussed Ricky's progress, her eyes constantly monitoring him and her mind alive with ideas to enhance the music.
of
Meanwhile, Danny was diligently covering every corner of the studio, He was making his rounds, ensuring that the studio was prim and proper for Ricky's crucial session. As part of his tasks, he passed by Ricky several times, always careful interrupt the artist's flow.
Emma, engrossed in her thoughts, deftly moves the earphones to the producer's 's desk. She is preparing to tweak the sound setlings, pre-empting Ricky's 's need for perfect audio his performance.
45

---

Published as a conference paper at ICLR 2024
To generate the fact set F, sample collection of people, items, and locations. Only two items can be used and we incorporate three moves which will force one item to be moved twice. We then sample three plausible moves using GPT-4 that take the form of "x moves [item] to [location]". For everyone else not moving the item, they have a chance to see the move, "y sees [item] move to [location]" to not see the move, "y does not see [item] move to [location]". We the chance of seeing move to 0.33. These facts are grouped into the move and added to the fact set F.
During the reasoning tree construction, we apply keyword lookup validator to ensure the de- ductions being created are correct and meaningful. Specifically, we disallow any deductions that mention key locations or key items in the story. This ensures the deductions are not vacuous and state facts that could trivially solve the problem.
G.4 TEAM ALLOCATION
Listing 25: Constructing the scenario for Team Allocation
DESCRIPTION: A 2oo requires caretakers for different animals and keepers for cleaning the exhibits. Use the given scenario description create list of three people, lasks, and two skills. Each skill should associated with one of the tasks, and should be each skill is unique and orthogonal to the other and it's assigned task.
Rules
1) Never indicate in someones their title that they are good at any particular skill task. For example, never say "Dr. Bob" as this implies they should be charge of medical tasks.
Here's example
Description: heavy flux of customers walk into coffee bar, you to assign workers to the register and others to baristas to handle the flow.
Output:
People: Sarah; Luis; John;
Tasks: Barista; Cushier
Skills: Can make coffee; Can handle customers
Your turn!
Description: zoo requires caretakers for different animals and keepers for cleaning the exhibits.
Output:
Listing 26: Team Allocation Deduction prompt for the Tree Construction stage
We are creating a story about people being assigned to do certain jobs by a manager (you). You have look into the personal details of your worker, their past experiences, their likes and dislikes, and their social interactions to determine what they good ond bad at doing.
make this story, 'e created tree structure that outlines the narrative. Your job is to fill out the entailment trees that prove a set of facts on how well someone does at particular job or how well two people work together.
the facts involving skills and jobs, make the facts story driven focusing on past experience and personal details about the character. For example, if the fact that "George is bad at tennis" " you might say that " George was never athletic growing up". "George actively avoids sporting events now" and "If someone isn in'' athletic and avoids sporting events, they probably aren' 't good at sports like tennis".
the facts involving teamwork, focus on the social element and the past. For example, if the fact is that "George and Bill work well together" then you might say "George and Bill grab lunch from time time.", "George and Bill were able to get a job done in average time working team last week". "If two people spend time together and are able to do adequate work together, then they probably work well together."
two individuals have interacted in
46

---

Published as a conference paper at ICLR 2024
Most importantly, the facts should be interesting and personal; not make them bland.
An entailment tree is tree structure where intermediate nodes are entailed by their children. They create a natural language reasoning proof for some collection of facts.
To fill out this tree we need complete an entailment. Completing an entailment akin to filling out one the structure of the step.
subtree of the entailment tree. To fill in this step, you must follow follow
Facts From Story are facts that will be explicitly stated when write the story. Commonsense Knowledge are facts that most people would agree are true and don't need be explicitly said
Complex Facts are facts that will be entailed by simpler facts from the story (they will be filled in later through recursive call back to you!)
All facts for step must combine entail the root parent fact.
No facts may contradict the current structure tree.
Always match the structure the entailment step give you. Give the number of Facts From Story and Commonsense Knowledge facts. Give them in the same order as well.
Each fact should be crucial to the deduction. Intentionally leave out details that the other fahers can account for them. If one fact is missing, the conclusion should not be entailed. Try not the facts.
Always use the persons name instead of a pronoun like "He" or "She", if you know someones name, use the name.
Here's 's example.
Scenario:
Paul and Alice are at karaoke bar.
Current Tree:
Paul and Alice are at karaoke bar. Deduced Root Conclusion
>
Opening Scene Deduced Fact
Paul sees the microphone at the stage. Fact From Story
Alice sees the microphone the stage. Fact From Story
Paul sees the beer at the bar. Fact From Story
>
Alice sees the beer at the bur. Fact From Story
Paul moves the beer to the luble.
Deduced Fact
Alice did not see the beer move the table. Deduced Fact
Alice was facing away from the table. Deduced Fact
> > Alice was talking to another patron. Fact From Story
>>> The other patron was facing the table. Fact From Story Usually people talk to each other while facing each other, if one person is looking in one
direction the other person is looking the opposite direction. Commonsense Knowledge
someone facing away from something else, they cannot see things transpire near that something
else. Commonsense Knowledge
Alice moves the microphone to the table Deduced Fact
Alice the beer at the table when moving the microphone. Fact From Story
Paul the microphone move the table. Deduced Fact
>
Paul was drinking the beer at the table. Commonsense Knowledge
>>
> When something happens where person is at, they usually see things that are happening there. Fact
From Story
Alice the beer the trash Deduced Fact
Paul did not see beer move to the trash can. Fact From Story
Entailment Step to complete:
Paul and Alice are at karaoke har. Because,
Alice moves the beer the trash Because,
Paul did not see beer move to trash can. Because,
> Fact From Story
Fact From Story
Commonsense Knowledge
47

---

Published as a conference paper at ICLR 2024
Output:
Paul and Alice are at karaoke bar. Because,
Alice moves the beer to the trash can. Because,
Puul did not see the beer move to the trash can. Because,
Alice tricked Paul into looking "over there". Fact From Story
Alice pointed in opposite direction of the trash can Paul. Fact From Story
> If you trick someone into looking else where, they cannot see what happens the other direction. Commonsense Knowledge
Here is another example.
Scenario:
Your dog has just pooped on the neighbours yard. The neighhour glares in your direction and comes forward he says "Hey you! Whut do you think you are letting your dog do on nice yard right here!"
Current Tree:
Punch the neighbour square in the nose. Deduced Root Conclusion
It'll took cool and this is pro. Deduced Fact
People think fighting is cool where you live.
Fact From Story
You would be fighting. Fact From Story
Doing something people think is cool will make you cool too. Commonsense Knowledge
> I'll look cool unless... Fact From Story
Entailment Step complete:
Jose has means. Because,
Fact From Story
> Fact From Story
Commonsense Knowledge
Output:
Jose has a means. Because,
> Jose has access to heroin. Fact From Story
Jose knows how much heroin is needed for overdose. Fact From Story Having access to heroin and knowing how much heroin required to overdose implies you could have intentionally given victim dose of lethal heroin providing a means for murder. Commonsense
Knowledge
Here is onother example.
Scenario:
Victim: Jared
Crime scene: Public park bench
Murder weapon: Heroin overdose
Suspect: Jose
Suspect's 's role in story: Drug user
Suspects motive: Public humiliation
Current Tree:
Jose is murderer. Deduced Root Conclusion
Jose has means. Fact From Story Jose has motive Fact From Story
Jose has an opportunity. Fact From Story
Entailment Step to complete:
Paul and Alice
karaoke bar. Because, Alice moves the beer the trash can. Because,
Complex Fact
Output:
Paul and Alice are karaoke bar. Because, Alice moves the beer to the trash can. Because, Paul did not see the beer move to the trash can. Complex Faet
Your Turn.
48

---

Published as a conference paper at ICLR 2024
Scenario: 200 requires caretakers for different animals and keepers for cleaning the exhibits.
Current Tree:
A 2o0 requires caretakers for different animals and keepers for cleaning the exhibits. Deduced Root
Conclusion
at exhibit cleaner. Because we find out Olivia bad at animal caretaker, Because we find out in the story that,
Olivia
story that,
Deduced Fact
Deduced Fact
Mia is bad at animal caretaker. Because we find out in the story that,
Deduced Fact
Mia is good at exhibit cleaner. Because find out in the story that,
Deduced Fact
Alex okay exhibit cleaner. Because we find out the story that,
Deduced Fact
Alex bad at animal caretaker. Because we find out in the story that, Deduced Fact
Alex and Mia work okay together. Because we find out in the story that, Deduced Fact
Olivia and Alex work budly logether. Because we find out in the story that, Deduced Fact Olivia and Mia work budly together. Because find out the story that, Deduced Fact
Entailment Step to Complete:
A 3oo requires caretakers for different animals and keepers for cleaning the exhibits. Because, Olivia is bad exhibit cleaner. Because find out in the story that,
Fact From Story
Fact From Story
Commonsense Knowledge
Ousput:
Listing 27: Team Allocation story creation prompt for the Story Generation stage
You will write short story given the description of scenario and list of facts. You include every fact the story.
You should take the role of manager or leader, someone who can assign people to skills.
Your job is find the perfect assignment of each person single skill. You will not say what this perfect assignment is, that left to reader decide.
Instead, must write out the narrative as if it were short story. Make this story coherent and flow nicely.
Most importantly, make this story interesting!
Start with an introduction.
Introduce each person:
Olivia
Alex
Mia
And the tasks the manager has assign them to (mention these by name in the beginning, i.e. "And they all had assigned to cleaning and sales" or something.
Animal Caretaker
Exhibit Cleoner
Scenario description: A zoo requires caretakers for different animals and keepers for cleaning the exhibits. Facts that you must include in your story:
Alex admitted that he feels uncomfortable around animals larger than him. Alex and Mia used to cooperate well in the same high school club, often collaborating on fundraising initiatives.
Alex expressed his disconfort around Olivia, mentioning how her avoidance makes him feel ostracized.
Alex expressed in the past no desire to pursue furthering his knowledge of animals outside of pets.
Alex has shown mild interest in keeping his surroundings neat, but he doesn't 't go out of his way to tidy up.
Alex sometimes voluntarily helped with exhibit cleaning when he was volunteer at cat shelter.
lunch breaks, Alex and Mia engage in friendly conversations and share similar sense of humor. Mia finds Olivia too passive and con-cooffontational, which results stewing resentment and lack of open communication.
Mia gets nervous and excessively worried at loud noises.
Mia insists on spotless living environment, often spending free time in her own home cleaning and organizing.
49

---

Published as a conference paper at ICLR 2024
Mia mentioned a discomforing encounter with roaring lion that left her shaken.
Mia showed enthusiasm when she found out about the cleaning tasks at the 200, expressing her belief that
the exhibitors need to mirror the animals' natural habitats as closely as possible.
Olivia avoids Alex during their shared shifts because of an old workplace disagreement.
Olivia experienced sneezing fits and allergies during her custodial job at school where cleaning was part
her task.
Olivia has admitted her fear of wild animals due to a past traumatic experience with chimp.
Olivia has explicis disagreements with Mia's s work habits, often criticizing her for cleaning the exhibits too
thoroughly.
Olivia mentioned her allergy dust and pollen.
Olivia often avoids the animal exhibits during her visits the 2oo.
Output:
Listing 28: Team Allocation introduction rewrite for the Story Generation stage
I am writing story that is similar to word problem where manager has to assign the right worker to task. Here's the full story:
In the bustling world of the Redwood Zoo, at the crux of the city's 's great jungle, the need for careful assigning of roles played a major role in the smooth function of the behemoth. As the manager, the delicate balance of where to assign Olivia, Alex, and Mia, the roles Animal Caretaker and Exhibit Cleaner intriguing puzzle to solve. Each with their unique tempercment, skills, and admittedly, eccentricities, made the task all the more interesting.
Let's 's begin with Alex, the till lad with bright eyes, whose history with the mighty beast of the animal kingdom. lacked certain comfort. The lad, known express an almost innate fell short of the prerequisites for an Animal Caretaker. His comfort zone extended the four-legged companions in our homes, a sentiment withheld from the petting 200 section of our park. Yet, his association and collaboration with Mia had seen quite the successes in their high school club', 's fundraising initiatives.
around animals larger than him.
However, his relationship with the gentle Olivia was not seamless. Alex often mentioned feeling ostracized due Olivia's 's tendency maintain her distance. seemingly innocent avoidance stirred disquiet within our hushed ranks. And all this, stemming from disagreement rooted in their previous shared workplaces. Unresolved perhaps, but factor nonetheless.
Then, there was Mia, determined bright spark, whose affinity for cleanliness would often bemuse us. would spend her spare time in her immaculate home cleaning and reorganizing, while her enthusiasm for a spotless Exhibit could not be underestimated. However, her overly thorough methods would invariably result clashes with Olivia, who criticized her for crossing some form of unspoken boundary.
Mia too her phobias, the gravelly raars of z00's majestic had once left her shaken and worried. Loud noises had a similar effect leaving her in a state of nervous terror, much like that of the petite animals held within our barriers. Yet, she was all smiles and peasant conversation around Alex during lunch breaks, sharing sense humor that lightened the mood of everyday grind.
Finally, subdued Olivia, a soul strangled with allergies, and deep-seated fear for wild animals. An incident with a chimp in her past wove tales of nightmarish hetrayal, enough to send her away from the animal exhibits during her zoo visits. Potent elements of dust and pollen resulted in uncontrolled sneezing fits, remainder from her days the school custodian, responsible for the cleanliness and maintenance.
Three souls; Animals to be cared for, Exhibits to be cleaned. Assigning them was always going to be an eni prigma for anyone navigating the zoological labyrinth. Love for animals, discomfort, alliances, conflicts; each factor extraordinarity crucial shaping not just the overall productiviry but also the personal growth of each of these individuals at the Redwood Zoo.
I want you rewrite the introduction paragraph introduce each person and the tasks. Do assign people to a task (this the question want to ask the readers), just introduce them. Rewrite the introduction:
In the bustling world of the Redwood Zoo, at the crux of the city's great jungle, the need for careful assigning of roles played a major role in the smooth function of the behemoth. As the monager, the delicate balance of where assign Olivia, Alex, and Mia, the roles Animal Caretaker and Exhibit Cleaner was intriguing
50

---

Published as a conference paper at ICLR 2024
puzzle solve. Each with their unique temperament. skills, and admittedly, eccentricities, made the task all the interesting.
Make sure it includes:
Mentions to these peoples names (do not describe them)
Olivia
Alex
Mia
And the tasks the manager has assign them (mention these by name the beginning, i.e. "And they all to be assigned to cleaning and sales" or something.)
Animal Caretaker
Exhibit Cleaner
should be short. No longer than the original introduction.
In the Tree Template Construction stage, build matrix M where the first two columns scores for each person's 's abilities on a task, and the last three are their relationship scores. Once we have that matrix, we can generate facts for each person's skill level and relationship score.
For skill level facts, we look at each person's 's proficiency at a skill and set either to "bad", "okay", or "good", then add to the fact set F, "[person] is [skill level label] at [skill]".
For relationship scores, we look at each unique pair's 's score and label it as either "badly", "okay", or "well". We can then add the fact, "[person I] and [person 2] work [relationship label] together." to the fact set F.
Once we have all the facts in F we can pass these the Reasoning Tree Construction stage. Team Allocation required no validators in the construction of the reasoning tree nor the story to create valid contexts.
H IMPLEMENTATION DETAILS, API COST
H.1 IMPLEMENTATION DETAILS: CREATING AND EVALUATING MUSR
Creating MuSR GPT-4 is used all creation prompts for creating MuSR. We do not change any parameters from the defaults when prompting GPT-4. Temperature and top-p are set 1.0.
For the Murder Mystery domain, we set the maximum depth of the reasoning tree to 4. Only two suspects are allowed be included in the generation process. To add diversity, sample victim's name, a murder weapon, a crime scene, suspect names, suspect motives, suspect roles in the story, and suspicious facts. All of these facts are used in the tree construction stage to guide the created deductions to include more diversity.
For Object Placements, we set the maximum depth of the reasoning tree Only three moves are allowed to be included in story, and only two items which forces one item to be moved twice. The chance for a person to see move is set to 33%. To include diversity, we sample scenarios where people need to move items around, then we sample their names, moves, and motivations for moving the item.
Finally, for Team Allocation, we set the maximum depth of reasoning to 2. We sample a scenario for diversity similar to Object Placements, then sample tasks and people's names.
During evaluation, we use the "gpt-4" and 1.5t-3.5-turii 2t5-tt chat endpoints. For Llama2 and Vicuma models, use Hugging Face's implementations (Wolf et al. 2020) and use bitsandbytes quantization (Dettmers et al. (2022) when running inference No changes to the inference parameters were made when sampling from the models. For all local models, on on a machine with 4xNVIDIA RTX A6000 48GB cards.
Evaluating MuSR
them
We did not see any significant performance increases in some of the smaller models when we did not quantizaaion.
51

---

Published as a conference paper at ICLR 2024
Table 9: Estimates of how long takes and times are based off of GPT4 and the CoT+ 1-Shot prompt variant.
of our dataset domains well All prices
Murder Mystery
Object Placements
Team Allocation
Time
Cost
Time
Cost
Time
Cost
2:00:00
S27
1:40:00
$23
2:10:00
$17
H.2 API COSTS AND TIMES
Table shows the average costs and times to run the evaluation of each domain using GPT4 with the CoT+ 1-Shot prompt variant.
I EVALUATION PROMPTS AND OUTPUTS
I.1 EVAL PROMPTS
Murder Mystery CoT+ evaluation prompt for Murder Mysteries.
Listing 29: Murder Mystery COT+ Prompt
{[context)
Who the most likely murderer?
Pick the following choices: (annwer choices))
You must pick one option. Before selecting a choice, explain your reasoning step by step. The murderer needs to have means (access to weapon). motive (reason kill the victim), and opportunity (access to crime scene ) in order have killed the victim. Innocent suspects may have these proven, but all three. An innocent suspect may suspicious for some other reason, but they will not have all of motive, means, opportunity established.
you believe that both suspects have motive, means, and opportunity, you should make educated guess and pick the one for whom these best established. If you believe that neither suspect has all three established, then choose the suspect where these are most clearly established. Explain your reasoning step by step before you answer. Finally, the last thing you generate should be "ANSWER: (your answer here, including the choice number)"
Object Placement
CoT+ evaluation prompt for Object Placements.
Listing 30: Object Placements COT+ Prompt
{
Based this story, we want identify where someone believes that certain object the end the story. order that, you need to read story and keep truck where they think the object is at each point. When an object is moved, the person may observe its new location if they saw it move.
To where object ends up, they must be able the location that it moves and not be too distracted by what they are doing. If they not observe object moving, then they will still believe location where they observed it.
the last
Which location is the most likely place Danny would look to find the earphones given the story?
Pick one of the following choices: (ansser choices)) }
You must pick one option. Explain your reasoning step by step before you answer. Finally, the last thing you generate should be "ANSWER: (your answer here, including the choice number)"
52

---

Published as a conference paper at ICLR 2024
Team Allocation CoT+ evaluation prompt for Team Allocation.
Listing 31: Team Allocation COT+ Prompt
{(contexr))
Given the story, how would you uniquely allocate each person to make sure both tasks are accomplished efficiently?
Pick one of the following choices: (annseer )hoices)}
You must pick one option. The story should allow you to determine how good each person is skill. Roughly, each person is either great, acceptable, bad task. We want to find optimal assignment of people to tasks
uses their skills as well as possihle. addition, one task will have to have two people assigned to it. The effectiveness of their teamwork (great team, acceptable team, or bad team) also impacts the overall quality of the assignment.
When two people need work on task and one is bad at it, they don't necessarily benefit from the other person being good, unless they work well together.
With different strengths, weaknesses, and interpersonal dynamics at play, you should allocate your team to find the single assignment ensure that the tasks overall completed effectively possible.
Explain your reasoning step by step before you answer. Finally, the last thing you generate should be ANSWER: (your answer here, including the choice number)"
Listing 32: Team Allocation PAI. Prompt
(ccnneet))
Given the story, how would you uniquely allocate each person to make sure both tasks are accomplished efficiently?
Pick one of the following choices:
[choices(O]}
2 choices[1]}
3
[choices[]]}}
You must pick one option. The story should allow you to determine how good each person is at skill. Roughly, each person is either great, acceptable, or bad at a task. We want to find an optimal assignment of people to tasks that uses their skills well possible. In addition, task will have to have people assigned it. The effectiveness of their teamwork (great team, acceptable team, bud team) also impacts the overall quality of the assignment.
When two people need to work on a task and one is bad at it, they don't It necessarily benefit from the other person being good, unless they work well together.
With different strengths, weaknesses, and interpersonal dynamics at play, you should allocate your team to find the single assignment to ensure that the tasks overall are completed as effectively as possible.
Now let's write python function to solve it. Our program will contain three steps:
First, find out how good person is at the two tasks. Assign it to a variable. Use 3 for great, for acceptable, and for bad.
For example 'Alice speakinng 2' means person Alice is acceptable at the task of speaking. When you assign values to individual scores, show your supporting facts (excerpt from the question) in the comments.
Second, find out the cooperation score for two people, use 3 for great, 2 for acceptable, and for bad. For example 'Alice with Bob = means Alice works great with Bob. When you assign values to cooperation scores, show your supporting facts (excerpt from the question) the comments.
'Alice.with
Third, find the score of each option, and choose the best scored option. Concretely, the score of an option of assignment is the sum of all the scores, including individual scores and cooperation scores. Hint: you use "nax([(option| _score, 'I' 'I') e.fopptoon2score '2' ioppiion3_scoree '3')])[ to tond out the best option.
53

---

Published as a conference paper at ICLR 2024
Please write a python function "assign_task" which returns the answer choice your response.
1.2 INCORRECT EVALUATIONS FROM GPT-4
Listing 33: Murder Mystery Reasoning Error
In the cold echo of deserted hockey rink, Otis was found dead from grenade explosion, and now Detective Winston must decipker the clues, suspecting Ervin and Andrew are somehow involved.
Detective Winston rubbed his temples while the facts flickered across his computer screen. Otis, the victim, would most definitely be missed. His usual vibrant energy that filled the hockey rink was replaced by cold, solemn silence.
Some contradictions needed to be resolved. The crime scene at the hockey rink was empty, both at the time of the murder and throughout the investigation. Winston's 's only lead was Andrew, who was an old military pal of Otis. Winston heard that Andrew had recently switched his career to become solo hiking guide. Indeed, remarkable part of his solo hikes environmental dangers-a a way of applying his skills acquired from military service. was an unusual practice in the industry but allowed due to his military expertise in bomb disposal- -an odd career choice for man of such skills.
his ability help hikers they came across on the way overcome
Winston drove the outskirts of the town, where Andrew lived in remote cabin. Andrew's solo job didn't 't always welcome company, but didn't seem to bother him. Rumor had that uses his military skills to help strangers on the trails overcome environmental dangers-detonatto falling rocks, perhaps?
Once in the cabin, Winston noticed several hiking supplies meticulously arranged. He inspected them and found, hidden strategically among normal hiking necessities ceassiittesfood.. -ood, water, emergency kit-were several small explosives. Chills ran down his spine as he remembered a recent break-in at a military base.
"I you have explosives mixed with your hiking guide supplies," Winston said, trying evoke reaction.
Andrew replied calmly, "Those are just for emergencies."
Listening to Andrew, Winston remembered that Otis had been friendly with him. Otis was always open with Andrew about his daily habits, even shared his calendar with him. Andrew the only person who knew Otis would be alone the hockey rink that evening.
On his way out, Winston remembered Andrew's ' frank words, "I don't 't like to hang out after work."
It seemed somewhat true, living far from town, working alone, and being prone avoid after-work hangouts. Winston thought to himself: Andrew did like to be alone. It all seemed so strange, the grenade murder, the empty hockey rink, Andrew's 's distant demeanor. However, he couldn't draw any obvious conclusions just yet. He needed more.
Detective Winston sat on the chilly bleachers of the local hockey rink, place as familiar to him was to Ervin, the mild-manned mannered Rabbi who had never missed game. The rink was still splattered with telltale signs of the merciless violence that had taken place there. Otis had been killed during a game, a seemingly jovial event turned nightmarish by the unexpected detonation of grenade.
Taking his eyes off the crime scene momentarily, scanned his surroundings and spotted his next person of interest: Ervin, a massive hockey fan, and a staunch supporter of their local team. His presence at the game had been expected, as his dedicated attendance was part of the everyday routine. Some of the spectators had mentioned Ervin stepping out of the stadium briefly, interestingly around the same time Otis was killed.
Not missing his beat, Winston approached Rabbi Ervin, who was still deep shock at the loss of fellow spectator and friend, "Rabbi Ervin?"
The Rabbi looked up meet the detective's prying eyes, "I Ervin, indeed. And you are?"
"I am Detective Winston, investigating the recent unfortunate event."
Observing Ervin't 's face closely, Winston began: "The strangest thing happened few days ago. Otis came to and told he was planning to go the authorities. He knew something was off, dangerous even. guess
54

---

Published as a conference paper at ICLR 2024
didn't react quickly enough. He mentioned your name, Ervin, and something about another life before the synagogue..."
Ervin was quiet, his gaze distant but thoughtfil. "Yes, Otis did confront me. He saw me moving some funds around. He saw, Detective, he saw me committing the crime which I've regretted every day since," Ervin confessed.
Winston let the confession sink in before shifting the subject subtly. "I heard you were in the military?"
Ervin smiled, just barely. nodding with touch of pride. "Yes, a former military officer. Had special training demolitions you see, before found my calling religion."
Winston returned his smile, "I suppose it s a bit of leap from demolitions to the pulpit."
"Just tad," Ervin's laughter soft. "But the old life stays with you in ways you wouldn' expect. I've got collection of war memorabilia back home. host some of my friends over sometimes, they marvel at these artifacts. I even got grenades in there. Nothing active, of course. They tend to catch the eye."
Winston excused himself. leaving Ervin mourn at the crash of their sanctuary of play turned battleground.
Who the most likely munderer?
Pick one of the following choices: 1 Ervin
Andrew
You must pick one option. Before selecting a choice, explain your reasoning step by step. The murderer needs have means (access weapon), motive (reason kill the victim), and opportunity (access crime scene in order have killed the victim. Innocent suxpects may have these proven, but all three. An innocent suspect may suspicious for some other reason, but they will not have all of motive, means, opportunity established.
you believe that both suspects have motive, means, and opportunity, you should make educated guess pick one for whom these are best established. If you believe that neither suspect all three established, then choose suspect where these are most clearly established. Explain your reasoning step by step before you answer. Finally, the last thing you generate should be "ANSWER: (your answer here, including the choice number)"
[[MODEL OUTPUT]]
Looking at Detective Winston's 's investigation into the characters of Ervin and Andrew, we can examine the means, motive, and opportunity for each.
First, Andrew, an ex-military guy and Otis's 's old friend, has motive. His recent change to hiking guide involves using his expertise in bomb disposal which demonstrates the means. Interestingly, the explosives noticed with the hiking supplies could very well utilized for a crime like this. Additionally, opportunity is clear, as he knew Otis' schedule, including his solitude in the hockey rink.
Ervin, the Rabbi and hockey fan, also has motive, as his secret funds-moving activity found out by Otis. Like Andrew, Ervin also has military background and knowledge of demolitions, hinting the means. His opportunity lies the fact that was present at hockey rink during the game, where Oris was killed, and reportedly stepped out of the stadium briefly around the same timefrume.
However, Ervin's 's collection of war memorabilia including grenades Unless this misdirection and he does possess live grenades, this could imply that his grenades wouldn' i be functional for the act. Andrew, on the other hand, has been found with actual explosives that are implied to be active. We can also note that Andrew's 's solitary activity and lifestyle further suggest his ability to commit such a crime without leaving opparent witnesses.
described "nothing active."
Therefore, while both candidates present motive, means, and opportunity, seems that Andrew's 's capacity to commit this crime is better substantiated. He has clear opportunity knowing Otis' schedule, an explicit motive with his recent career change contrasted with his military prowess, and useful means evidenced by the explosives found among his hiking supplies.
55

---

Published as a conference paper at ICLR 2024
ANSWER: 2 Andreew
[[Correct Answer]]
Listing 34: Team Allocation Reasoning Error
Amidst the vibrant chaos of the Redwaod 7oo, nestled the heart of the city' sprawling jungle, the task of assigning roles was crucial cog in machinery of its operation. allocating Olivia, Alex, and Mia to the positions of Animal Caretaker and Exhibis Cleaner presented an intriguing conandram. Each individual, with their distinct personalities and skill sets, added a layer of complexity this assignment puzzle.
manager. the responsibility of
Let's begin with Alex, the tall lad with bright eyes, whose history with the mighty beast of the animal kingdom, lacked a certain confort. The lad, known to express an almost innate unease around animals larger than him, fell short of the prerequisites for Animal Caretaker. His comfort zoee extended to the four-legged companions in our homes, sentiment withheld from the petting 200 section of our park. Yet, his association and collaboration with Mia had seen quite the successes their high school club', 's fundraising initiatives.
However, his relationship with the gentle Olivia was not as seamless. Alex often mentioned feeling ostracized due Olivia's 's tendency maintain her distance. This seemingly innocent avoidance stirred disquiet within our hushed ranks. And all this, stemming from disagreement rooted in their previous shared workplaces. Unresolved perhaps, but a factor nonetheless.
Then, there was Mia, the determined bright spark, whose affinity for cleanliness would often bemuse us. She would spend her spare time in her immuculate home cleaning and reorganizing, while her enthusiasm for spotless Exhibit could not umderrsstmmaee. However, overly thorough methods would invariably result clashes with Olivia, who criticized her for crossing some form of unspoken boundary.
Mia too had her phobias, the gravelly roars of the zoo's majestic lion had once left her shaken and worried. Loud noises had similar effect leaving her in state nervous terror, much like that of the petite animals held within our barriers. Yet, she was all smiles and peasant conversation around Alex during lunch breaks, sharing a sense of humor that lightened the mood of our everyday grind.
Finally, subdued Olivia, soul strangled with allergies, and deep- seated fear for wild animals. An incident with a chimp in her past wove tales of nightmarish betrayal, enough to send her away from the animal exhibits during her zoo visits. Potent elements of dust and pollen resulted in uncontrolled sneezing fits, a remainder from her days at the school as custodian, responsible for the cleanliness and maintenance.
Three souls; Animals be cared for, Exhibits be cleaned. Assigning them was always going be an enigma for anyone navigating the zoological labyrinth. Love for animals, discomfort, alliances, conflicts; each factor extraordinarily crucial shaping not just the overall productivity but also the personal growth of each of these individuals at the Redwood Zoo.
Given the story, how would you uniquely allocate each person make sure both tasks are accomplished efficiently?
Pick one of the following choices:
1 Animal Caretaker: Alex, Exhibit Cleaner: Mia and Olivia
Animal Caretaker: Olivia, Exhibit Cleaner: Alex and Mia
3 Animal Caretaker: Mia, Exhibit Cleaner: Alex and Olivia
You must pick one option. The story should allow you to determine how good each person is skill. Roughly, each person either great, acceptable, or bad a task. We want to find an optimal assignment of people to tasks that uses their skills as well as possible. addition, one task will have to have two people assigned to it. The effectiveness of their teamwork (great team, acceptable team, or bad team) also impacts the overall quality of the assignment.
When two people need work on a task and one is bad at it, they don't 't necessarily benefit from the other person being good, unless they work well together.
With different strengths, weaknesses, and interpersonal dynamics at play, you should allocate your team to find the single assignment ensure that the tasks overall are completed effectively as possible.
56

---

Published as a conference paper at ICLR 2024
Explain your reasoning step step before you answer. Finally, the last thing you generate should be ANSWER: (your answer here, including the choice number)"
[[MODEL OUTPUT]]
Reading through the story, we learn lot about each individual's strengths, weaknesses, fears, and relationships.
Starting with Alex, we find that has discomfort around large animals, which indicates he may not be suited for the role of Animal Caretaker. However, Alex has a good working relationship with Mia from their previous fundraising activities. However, his relationship with Olivia is strained.
Next, we have Mia. loves cleanliness and would likely be ideal for an Exhibit Cleaner role. Nonetheless, she doesn't i get along well with Olivia and has deep fear of large, loud animals. Her bond with Alex is strong, as they share humorous exchanges during their breaks.
Finally, Olivia is mentioned. Her allergies and fear of wild animals make her poor fit for the Animal Caretaker role. She also seems to clash with over cleaning methods. While her relationship with Alex not explicitly described as negative, Alex has stated that he feels left out by Olivia, suggesting a strained relationship.
Given these insights. let's look at the options:
Option 1 Wouldn's it work well because Olivia and Mia do not have a good relationship. them working together on cleaning could be problematic. Also, Alex is not confortable around large animals, making him poor fut for the Animal Caretaker role.
Option 2 May not work as efficiently because Olivia has problems with allergies, which could be triggered by cleaning dust and pollen. Her fear of wild animals also makes her poor choice for the Animal Caretaker. Again, Alex and Mia have good relationship. but both
comfortable with large animals.
Option 3 - appears to the best choice here. Mia, her love for cleanliness, would excellent as an Exhibit Cleaner, while the strained relationship between Olivia and Mia might not be too much of an issue if they work on different exhibits. Alex could hring his excellent teamwork and communccaion with Mia the Exhibit Cleaning team, kelping mitigate Olivia's involvedness. Mia fear of animals and Alex's 's discomfort around them won't be an issue as they would not have to interact closely with animals in their roles as Exhibit Cleaners.
So, after considering everyone's skills, confort levels, and interpersonal dynamics, the answer would be:
ANSWER: 3
Animal Caretaker: Mia, Exhibit Cleaner: Alex and Olivia.
[[Correct Answer]]
Listing 35: Object Placements Reasoning Error
In the heart of the bustling studio, Ricky, Emma, and Danny readied themselves for a day of creating magic. Ricky, holding the helm as the gifted singer-songwriter was poised for perfection, his precious notebook of lyrics awaiting its call to duty on the producer's desk. Emmaa their dutiful and talented producer, was just eager to breathe life into Ricky's lyrics. She was cognizant of the notebook's 's place her desk, awaiting the melodies they would cultivate together.
Across the room, Danny, the diligent studio assistant, was doing his due diligence, keeping the earphones nestled the recording booth. His aim
ensure optimized and meticulous environment for recording, a testament to his commitment to their shared mission. They were all aware of the arrangement notebook on the producer's desk, the earphones in the recording booth. Their shared consciousness of these items only intensified the anticipation; they were eager to turn the contents of weathered notebook into world-clas album.
57

---

Published as a conference paper at ICLR 2024
Ricky, with his weathered notebook of potent lyrics in hand, gently places it onto the piano. An air of creativity and anticipation lingers the room, everyone aware that this was the first instrumental step in the creation of their masterpiece. In syne with the palpable creative energy, Ricky was engrossed in perfecting the rhythm of his song. preparing himself for an intense day ahead. Not too far away, Emma was sincerely engrossed in her role of musically steering the session. She was focussed on Ricky's ' progress, her eyes constantly monitoring him and her mind alive with ideas to enhance the music.
Meanwhile, Danny was diligenily covering every corner of the studio. He was making his rounds, ensuring that the studio was prim and proper for Ricky's crucial session. As part of his tasks, he passed by Ricky several times, always careful not to interrupt the artist's flow.
Emmaa engrossed in her thoughts, deftly moves the earphones to the producer's desk. She is preparing to tweak the sound settings, emmpingg Ricky's 's need for perfect audio in his performance. Diverting from his rounds, Damny found himself in the midst of stirring conversation with visiting sound engineer. Knowledge flowed between them, illaminating the studio's atmosphere, the engineer's insight bringing a new perspective into Danny's 's role. Ricky, ensconced in his own world, was in deep discussion with the blank page before him. The daunting silence of the empty studio buzzed with his focus, as he honed his lyries perfection in space separate from the producer's. The visitor, oblivious the careful choreography of the studio session, stood blocking Danny's general overview of the studio space.
Delicately lifting Ricky's 's notebook, Danny orchestrates its move to the producer's 's desk. At the desk, he glimpses a pair of earphones indirectly drawing his attention amidst his routine of tidying up, Emma, from the isolated interior of sound-proofe booth. lent her ears diligently already recorded tracks, pouring them for any room for improvement. Being lost the music was her way of paying homage to her craft- an unspoken ritual she followed each time she embarked on a music production journey. The entirety of her focus was consumed by the musical notes and rhythm fnythmfilttriin filtering through the studio speakers.
Concurrently, Ricky was absorbed in the act of playing his guitar. His fingers navigated deftly over the strings lost in an intimate dance with the instrument. As he played, the melodic strums reverberated throughout the studio, filling it with an infectious pulse that hinted at the birth of yet another musical masterpiece. Despite the fturry of activity around him, Ricky was lost in world of his own, operating on singular vision of delivering his best performance.
the meantime, Danny was continuing his cautious management of the studio, ensuring that everything fell into place for the optimum recording session. His watchful eyes were sconning every corner, taking stock of the minor details that could impact the session. However, the design of the studio didn't 't allow for unrestricted view into all the corners. The sound booth, where Emma was engrossed in her work, was out of his visual range. The seclusion provided by the booth, although crucial for immersive work, alsa acted as a barrier for Danny's 's comprehensive vigilance.
As the day progressed, the studio entwined in concerted symphony of dedication and workmanship, the trio, each engrossed in their pursuit, working together to create the best version of Ricky' impending album. As the final note of the day rang through the studio, each person revelled in the satisfaction of another day done right, another step closer towards the realization of Ricky's 's artistic vision.
Within the dynamic dance the day's 's events, the relationships the trio sang compelling tune. Each individual played their crucial part in the creation of the impending masterpiece Riicy with raw talent, Emma with her passion for perfection, and Danny with his meticulous eye for detail. And as the fightsfaded faded on another day of creation, they could sense the beginning of an important chapter in their artistry, a silence collecting the scattered notes of the day, signing off on another critical step in the journey of Ricky's upcoming album.
Baxed on this story, we want to identify where someone believes that a certain object is at the end of the story. In order to do that, you need to read the story and keep truck of where they think the object is at each point. When an object is moved, the person may observe its new location if they saw it move.
To see where an object ends up, they must be able to see the location that it moves to and not be too distracted by what they are doing. If they do not observe the object moving, then they will still believe it to be in the last location where they observed it.
Whick location is the most likely place Danny would look to find the earphones given the story?
Pick one of the following choices:
1 piano
58

---

Published as a conference paper at ICLR 2024
producer's x desk recording booth
You must pick one option. Explain your reasoning step by step before you answer. Finally, the last thing you generate should be "ANSWER: (your answer here, including the choice number)"
[IMODEL OUTPUTI]
In the story, Danny prepared the recording booth placing the earphones there. He was then occupied ensuring other studio duties and discussing with a visiting sound engineer. did not see Emma move the earphones from the recording booth to the producer's 's desk, nor was there any cue indicating he was aware of this change. The story highlights that the sound booth is out of Danny's s visuul range, confirming that he didn' witness the movement. Hence, given the last observed place and lack communication about the change of place, the most likely location where Danny would look find the earphones would recording booth.
ANSWER: 3 recording booth
[[Correct Answer]]
2
59


Reviews:
[{"review_id": "bQd3nqed9Z", "reviewer": "Reviewer_WiLx", "summary": "This paper introduces a new dataset and mechanism for generating complex and challenging reasoning problems that test the limits of SotA LLMs. The resulting examples are designed to be long and contain elements of both (realistic) natural language and deductive reasoning. The method involves building a reasoning tree skeleton of facts and rules that are turned into narratives (through \"chaptering\")  for which particular questions are elicited. Examples in 3 different domains are generated and released (though the method can be applied to others) and an evaluation is preformed on a few SotA LM's like GPT-4, showing that there is still a gap to be crossed in the ability of LM's to do reasoning.", "questions": "1. I take objection to the title only talking about CoT; while obvious a big breakthrough in LM reasoning it isn't the only game in town :)\n\n2. Table 2: suggest adding citations to the papers for each dataset in the table next to the dataset names.\n\n3. Could the stories on 5minutemystery.com themselves be used for the dataset?\n\n4. page 3 last line: sort of misleading claim. GPT-4 is used in generating the dataset, even if not the \"reasoning chain\" in an abstract sense.\n\n5. you have not defined Prompt(T \\ s)\n\n6. there is no hard deductive  that a murderer \"must* have  motive/opportunity/means to commit a murder, it is a sort of common-sense notion. So apart from the Cot+ setting where you explicitly call that out in the prompt, how is the system to know it has to look for this exclusively?\n\n7.  sec 4.3: \"strength of a relationship\" is a vague term. I have to say that a lot of the writing from sec 3.2-4.3 could use a little more explanation and polish; i could only understanding them after looking at appendix.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First of all, I find the direction of this work very important. There is a sore need for datasets and evaluations like the one generated here, as previous reasoning datasets, that focus on simpler forms of reasoning or expressed in very synthetic unrealistic language, get saturated as being too easy for SotA models. The current direction of Reasoning with LLMs has consequently taken a turn towards focusing on domains where we still have access to challenging datasets like Mathematics, which while important, is not representative of the real-world \"messy\" general reasoning that LLM's need to demonstrate to demonstrate next-level capabilities. The dataset and method presented here, while flawed in some ways, is to my knowledge the best so far at this.\n\nI wish to call out the originality of the method as well, since it requires a lot more than the standard dataset generation workflow of decide domain -> collect some seed data -> do a lot of crowd-sourced collection-> refine. This requires a careful design of a 'synthetic' data pipeline, through key novel components like the reasoning tree and 'chaptering',  that nonetheless does seem to generate many truly realistic real-world scenarios (again at least in comparison to previous datasets) that challenge GPT-4 and its competitors.\n\nThe results show a reasonable (though not huge) gap between the best performing pure LLM and the claimed ceiling (i.e. the best performing human), which represents the opportunity for models to improve their reasoning capacity. \n\nThe paper is reasonably clear on what it is doing, though some details had me confused, which I enumerate below. I appreciate the thorough comparison in features to recent datasets.", "weaknesses": "By far the biggest concern I have, which is derived from working on attempts at similar dataset creation, is that there is a point where despite designing the reasoning chains to be deductively entailed from premises, somewhere in the translation to ambiguous natural language and the accumulation of steps, reasonable humans start to disagree on whether a particular conclusion follows or not, at least with certainty. This can lead to there possibly being a sizable number of cases in the dataset where in fact a single gold answer cannot (even in principle) be attributed to the problem.  I am wondering if the authors have considered this issue and how they may say their dataset construction procedure guards against it. Would they claim the best performing or majority human annotator label be considered a lower bound on the precentage of problems for which a definite true answer exists? \n\nI am also slightly concerned about the self-referential use of GPT-4 in constructing the dataset. I think the authors have put in a lot of work to minimize the problem, since they use it only for specific steps after the bulk of the reasoning chain has been fleshed out by the reasoning tree algorithm and the chaptering approach introduces tight constraints so that it seems that the main effect of GPT-4 is 'simply' to turn the abstract narrative into natural language. However recently, there have been concerns raised about the practice of the field as a whole relying on LM's for every stage of generation,evaluation and so on (see [1]) so it would be good to discuss this. indeed the jump in performance from LLama 2 to GPT-4 is large enough to make me a bit suspicious that the use of GPT-4 in the dataset construction is biasing these results in a subtle manner. Would it have been helpful to have used a mixture of LMs to generate the text for the narratives?\n\nA minor point: It seems the second part of the benchmarking is not as complete as may be hoped for. I do appreciate the attempt to explore neuro-symbolic approaches, and the authors used a fair number (5), but why was each only applied to a limited number of domains (were there limitations to each that prevented this?). Also, LAMBADA [2] is a recent general purpose approach to neurosymbolic reasoning that seems like it would be very helpful in all 3 domains (since it performs backward reasoning which is generally more effective on complex reasoning problems) that the authors could have tried.\n\n[1] https://arxiv.org/abs/2307.01850\n[2] https://arxiv.org/abs/2212.13894\n\nOverall none of these concerns seem fatal to me. I think this paper makes a contribution that can unblock further progress in the reasoning field and should be accepted unless Ive missed something.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1t0cXBykRj", "reviewer": "Reviewer_cJvj", "summary": "This paper proposed a new dataset, called MuSR, for evaluating LLMs in multi-step reasoning. The authors argued that existing datasets either are not complicated enough (e.g., requiring limited steps of reasoning) or don't have sophisticated natural language descriptions. MuSR instead is a synthetic dataset in three domains, i.e., murder mysteries, object placement, and team assignment, whose questions often involve multi-step reasoning over the provided textual descriptions. Experiments showed that state-of-the-art LLMs such as GPT-4 can achieve decent performance but still underperform humans by a large gap. Furthermore, customized, domain-specific CoT prompting, or prompting with 1 shot of examples, helps.", "questions": "1. Could the authors clarify the human engineering in the dataset construction?\n2. Could the authors provide more justification for the creation ablation analysis?\n3. Could the authors clarify the dataset challenge under few(>1)-shot settings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper contributes a new dataset for evaluating LLMs in multi-step reasoning tasks, particularly when the reasoning has to be grounded in natural texts. As the dataset is synthetic, it has the potential to scale up to an arbitrary size (though I have a concern; see Weakness 1) and can be refreshed in the future. In experiments, the dataset was shown to be challenging for LLMs.", "weaknesses": "1. It's unclear to me how much human engineering is needed in the dataset construction process. The major confusion comes from the \"scenario-specific facts\" generation. Looking at the prompt in Appendix C.1, at the end of page 30, it seems that human engineering is needed to provide the scenario setup, e.g., the following. If my understanding is correct, then the dataset is not \"scalable\" (which the authors claimed is a main advantage over human-annotated ones).\n```\nScenario: Victim: Tessa\nCrime Scene: kitchen\nMurder Weapon: poisonous gas\nSuspect: Penelope\nRole in story: Tarot Reader\nThe suspect’s motive: To protect a secret\n```\n\n2. The dataset creation ablation needs more justification. The authors suggested low GPT-4 performance (Acc) from the ablated baselines as a good indicator of the effectiveness of each creation component, but MuSR outperforms the two ablations in Object Placements, which seems to imply that the creation components are not useful for Object Placements. Reverse observations were seen in the Murder domain, and no comparison was done for Team Allocation. \n\n3. The current experiment exploration seems a bit shallow, as the discussions are mostly about comparing numbers achieved by different LLMs or prompting methods. It's suggested that the authors could provide some qualitative analyses and elaborate on the bottlenecks for LLMs to perform well in such tasks. \nIn addition, adding 1 shot of example showed to improve GPT-4 CoT+ substantially. A conjecture is that adding more examples could improve the performance further, eventually to be close to the human performance. In this case, can this dataset still be considered as challenging one?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zavCxsNpdm", "reviewer": "Reviewer_NkVA", "summary": "This paper proposes a dataset named MuSR for evaluating language models on multistep soft reasoning. The proposed dataset is created through a novel neurosymbolic synthetic-to-natural generation algorithm using GPT-4, and the data instances contain free text narratives corresponding to real-world domains of reasoning. The dataset is used to evaluate LLMs including GPT-4, GPT-3.5, Llama2, and Vicuna.", "questions": "Q1: The dataset size is small according to Table 2. Does the dataset size affect the evaluation of model performances? \n\nQ2: In Section 3.2, when recursively calling the GPT-4 to produce the reasoning trees, how to deal with the possible logical or factual inconsistency between the statements/facts?\n\nQ3: What does “CMCR” in the title of Section 5.2 refer to?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The proposed dataset provides new challenges to current large language models. The neurosymbolic synthetic-to-natural generation algorithm provides an innovative method for obtaining challenging reasoning data. This paper also conducts comprehensive experiments to demonstrate the properties of the dataset. And the benchmarking experiments that cover extensive analyses.", "weaknesses": "Please see the questions listed below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "voOHYqp204", "reviewer": "Reviewer_WJFH", "summary": "The paper contributed two things. (1) The paper generated 750 test examples in total across three domains. The generation process is grounded in reasoning trees (example in Figure 1). (2) LLMs are run on the benchmark, with different chain-of-thought prompting approaches and some neurosymbolic approaches. \n\nFor (1), the three domains are murder mystery stories, object placement, team assignment (involving commonsense, social, physical, theory-of-mind reasoning). \n- LLMs are used to generate all the data. Figure 1 gives an example of the data generation process of the murder mystery task. \nIn general, the generation process is in three steps: domain injection (obtaining initial gold facts), reasoning tree construction (expanding the gold facts), and story generation (generating narrative based on the tree). \n\nFor (2), Table 5 provides results on GPT-4, GPT-3.5, two Llama-based models, and three Vicuna-based models. Human performance is high (but far from perfect). The machine performance is quite low (except for GPT-4 which is also far from perfect). A few chain-of-thought variants are explored.", "questions": "Did the authors try the “let’s think step by step” zero-shot prompt?\n\nDid authors try few-shot prompting (more than 1 shot)\n\nWhat are the advantages over existing long-document question answering benchmarks? For example, I can see that your generated data will be new, so there won’t be the issue of data leakage (so that LLMs might pretrain on them). You can also make the question arbitrarily long.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The domains are relatively novel. \n\nIt’s great that this paper provides an approach to generating arbitrary complex/long stories. If we increase the complexity, at some point it’s possible LLMs won’t be able to solve the problems (but the stories may also get less realistic). \n\nIt'll be especially great if the authors use this strategy to generate arbitrary long-context stories so that we can have infinite training/testing data on long-doc QA benchmarks.", "weaknesses": "Have humans checked whether the stories are coherent, and reference answers are correct? \n- I’m a bit concerned because the human results are far from perfect. Is the result due to poor crowdsourcing incentive structures / crowdworker filtering processes? Or are they good annotators but just being careless? Or are the references actually incorrect (or other problems in the story)?\n- Are there confusing / incoherent parts in the story (perhaps inspired by some of the story generation evaluation metrics)? \n\nThe benchmark is synthetic (which is fine). It only covers three domains where there are lots of people or objects so it may be hard for LLMs to keep track of them. But:\n- There have been so many benchmarks recently (Open Assistant, advanced reasoning benchmark, https://arxiv.org/abs/2307.02477, etc.). How do you expect this particular benchmark to be used? \n- We can generate arbitrarily complex synthetic problems. Given that the data is synthetic, how important is it for LLMs to be able to solve these problems (over other benchmarks)? I would be more convinced if this technique can be used to generate arbitrary story and ask arbitrary questions. \n\nLack of citation on tree construction and incorrect citation.\n- The authors’ Table 1 included PrOntoQA which explores logical reasoning, in particular, deductive reasoning (ICLR 2023; https://arxiv.org/abs/2210.01240), but the citation after PrOntoQA is actually ProtoQA (EMNLP 2020, which is quite different!). \n- I think PrOntoQA (https://arxiv.org/abs/2210.01240) and PrOntoQA-OOD (https://arxiv.org/abs/2305.15269) are relevant, given that those reasoning papers generated ontology tree first, and then generated the examples from the tree. There may be many other papers that have relevant ideas.\n- Some of the story generation ideas are also extremely relevant. For example, https://arxiv.org/abs/2210.06774; check other work by those authors for more examples.   \n\nOne thing that seems missing (apologies if I missed it) is to count the number of responses which can not be clearly parsed for each method – those responses do not necessarily mean that the model isn’t able to solve the problem. Is the non-parseable responses zero? \n\n\nMinor: The Table 1 framing needs to be better. For example, if Sophia knows about the shooting, and Sophia has a grudge toward Emily because Emily stole Sophia’s fortune, then the reference says Sophia is the murderer. In the real world this killer identification is not that simple. We can only take a guess based on the existing evidence. We don’t want LLM to be certain that Sophia is the killer. (The prompts in the appendix seems okay.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)