Disentangling Extraction and Reasoning in Multi-hop Spatial Reasoning
Authors: Roshanak Mirzaee
Parisa Kordjamshidi
Venue: EMNLP 2023 Findings (2023)
Keywords: Spatial Reasoning
Spatial Role Labeling
Disentangling Extraction and Reasoning
Pretrained Language Models
Large Language Models

Abstract: Spatial reasoning over text is challenging as the models not only need to extract the direct spatial information from the text but also reason over those and infer implicit spatial relations. Recent studies highlight the struggles even large language models encounter when it comes to performing spatial reasoning over text. In this paper, we explore the potential benefits of disentangling the processes of information extraction and reasoning in models to address this challenge. To explore this, we design various models that disentangle extraction and reasoning(either symbolic or neural) and compare them with state-of-the-art(SOTA) baselines with no explicit design for these parts. Our experimental results consistently demonstrate the efficacy of disentangling, showcasing its ability to enhance models' generalizability within realistic data domains.

Disentangling Extraction and Reasoning in Multi-hop Spatial Reasoning
Roshanak Mirzaee Michigan State university
Parisa Kordjamshidi Michigan State university
mirzaeememsu. edu
kordjamsêmsu. edu
Abstract
grey car is parking in front grey house with brown window frames and plants on the balcony.
Spatial reasoning over text is challenging as the models not only need extract direct spatial information from the text but also son over those and infer implicit spatial rela- tions. Recent studies highlight the struggles even large language models encounter when it comes to performing spatial reasoning over text. In this paper, we explore the potential benefits of disentangling the processes of in- formation extraction and reasoning in models address this challenge. To explore this, design various models that disentangle extrac- tion and reasoning (either symbolic or neural) and compare them with tate-offthe--a (SOTA) baselines with no explicit design for these parts. Our experimental results consistently demon- strate the efficacy of disentangling, showcasing its ability to enhance models' generalizability within realistic data domains.
Explicit
FRONT( a grey car, a grey house)
Extraction
triplcts
NTPPI grey house, plants)
Q: Are the plants behind the car?
Explicit
BEHLNND the plants, cur)
triplets
Reasoning
START Plants
2
INVERSE NTPP( plants, a grey house)
INVERSE BEHIND(a grey house, a grey car)
4
COMBINE BEHIND(plants, grey car)
Coreference (a grcy the car)
5.6
BEHIND the plants, the car)
Answer = Yes
Figure I: An example of steps of spatial reasoning on RESQ dataset. We begin by searching for the plants from the question triplct within the text, enabling us to extract explicit triplets (1,2). Next, we apply rules such as INVERSE to deduce implicit triplets (3,4.5). Then, utilizing triplets and we determine the final answer. 'Yes". NTPP: Non-Tann -on-Tangettia Proper Part (Table 1).
1 Introduction
Despite the high performance of recent pretrained language models on question-answering (QA)
tasks, solving questions that require multi-hop rea- soning is still challenging (Mavi et al., 2022). In this paper, focus on spatial reasoning over text which can be described inferring the implicit! spatial relations from explicit relations? described in the text. Spatial reasoning plays a crucial role in diverse domains, including language ground- ing (Liu et al., 2022), navigation (Zhang et al., 2021), and human-robot interaction (Venkatesh et al.. 2021). By studying this task, we can an- alyze both the reading comprehension and logical reasoning capabilities of models.
spatial question answering (SQA). PLMs show rea- sonable performance on the SQA problem and can implicitly learn spatial rules from large set of training examples. However, the black-box nature of PLMs makes it unclear whether these models are making the abstractions necessary for spatial reasoning or their decisions are based solely on patterns observed in the data.
As a solution for better multi-hop reasoning, re- cent research has investigated the impact of using fine- grained information extraction modules such as Named Entity Recognition (NER) (Mollá al., 2006; Mendes et al., 2010), gated Entity/Relation (Zheng and Kordjamshidi, 2021) 1) or semantic role labels (SRL) (Shen and Lapata, 2007; Faghihi et al., 2023) on the performance of models.
Previous work has investigated the use of general end-to-end deep neural models such as pretrained language models (PLM) (Mirzaee et al., 2021) in
By implicit. we mean indirect relations, not metaphoric usages or implicit meaning for relations. "relationships hetween objects and entities in the environ- ment, such as location, distance, and relative position.

---

Formalism
Spatial Type
Expressions (e.g.)
end-to-end model with explicit layers for extraction and reasoning performs better.
(General Type)
(disconnected)
disjoint
EC (Exterualyy Connected)
touching
PO (Partially Overlapped)
overlapped
These results show that disentangling extraction and reasoning benefits deterministic spatial rea- soning and improves generalization in realistic do- mains despite the coverage limitations and sensitiv- ity to noises in symbolic reasoning. These findings highlight the potential of leveraging language mod- els for information extraction tasks and emphasize the importance of explicit reasoning modules rather than solely depending on black-box neural models for reasoning.
Topological
EQ (Fiqual)
equal
(RCC8)
coveree hy in, inside
NTPP
TPPI
covers
NTPPI
has
LAFT, RIGHT
left of, right
Directional (Relative)
BELOW, ABOVE
under, over behind, in front far, close
BEHIND, FRONT
Distance
Far, Near
Table I: List of spatial relation formalism and types.
On different thread, cognitive studies (Stenning and Van Lambalgen, 2012; Dietz et al., 2015) show when the given information is shorter, humans also find spatial abstraction and use spatial rules to infer implicit information. Figure 1 shows an example of such extractions. Building upon these findings, we aim to address the limitations of end-to-end models and capitalize on the advantages of fine-graines information extraction in solving SQA. Thus, propose models which disentangle the language understanding and spatial reasoning computations as two separate components. Specifically, we first design a pipeline model that includes trained neural modules for extracting direct fine-grained spatial information from the text and performing symbolic spatial reasoning over them.
Related Research
End-to-end model on SQA: To solve SQA tasks, recent research evaluates the performance of dif- ferent deep neural models such as Memory net- works (Shi et al., 2022; Sukhbaatar et al., 2015), Self-attentive Associative Memory (Le et al., 2020), subsymbolic fully connected neural network (Zhu et al., 2022), and Recurrent Relational Network (RRN) (Palm al., 2017). Mirzaee and Kord- jamshidi; Mirzaee et al. use transfer learning and provide large synthetic supervision that enhances the performance of PLMs on spatial question an- swering. However, the results show a large gap be- tween models and human performance on human- generated data. Besides, none of these models use explicit spatial semantics to solve the task. The only attempt towards integrating spatial seman- tics into spatial QA task is a baseline model in- troduced in (Mirzaee et al., 2021), which uses rule- based spatial semantics extraction for reasoning on bAbI (task 17) which achieves 100% accuracy without using any training data.
The second model is simply end-to-end PLM that uses annotations used in extraction modules of pipeline model in the format of extra QA su- pervision. This model aims to demonstrate the advantages of using separate extraction modules compared to a QA-based approach while utilizing the same amount of supervision. Ultimately, the third model is an end-to-end PLM-basee model on relation extraction tasks that has explicit latent layers to disentangle the extraction and reasoning inside the model. This model incorporates neu- ral spatial reasoner, which trained to identify all spatial relations between each pair of entities.
Extraction and Reasoning: While prior research has extensively explored the use of end-to-end mod- els for learning the reasoning rules (Minervini al., 2020; Qu et al., 2021), there limited discussion on separating the extraction and reasoning tasks. Nye et al. utilizes LMs to generate new sentences and extract facts while using some symbolic rules to ensure consistency between generated sentences. Similarly, ThinkSum (Ozturkler al., 2022) uses LMs for knowledge extraction (Think) and sepa- rate probabilistic reasoning (Sum), which sums the probabilities of the extracted information. How- ever, none of these works are on multi-step or spa- tial Reasoning.
We evaluate the proposed models on multi- ple SQA datasets, demonstrating the effectiveness of the disentangling extraction and reasoning ap- proach in controlled and realistic environments. Our pipeline outperforms existing SOTA models by a significant margin on benchmarks with con- trolled environment (toy tasks) while utilizing the same or fewer training data. However, in real- world scenarios with higher ambiguity of natural language for extraction and more rules to cover, our

---

3 Proposed Models
A parkitg Frant of grey heuse
Extraction
Ressoning
Story procossing
tépiets
Facts
LEFTI Filse
To understand the effectiveness of disentangling the extraction and reasoning modules, we provide three groups of models. The first model is pipeline of extraction and symbolic reasoning ($3.1), the second model an end-to-end PLM that uses the same spatial information supervision but in a QA format ($3.2), and the third model is an end-to- end neural model with explicit layers of extraction and reasoning ($3.3). We elaborate each of these models in the subsequent sections.
Rols Exiraction
FACNR 1,2)
XXPPPI 2, 3)
Syrert Medial Ravserne (ou
RIGHT: Thue
Carof Rosolutor
Corefemce info
fgrey carl [a grey hosse!
BEHEND: The Flese
Are the plants bokind
re-Cowf
tke sar?
Queekkn proceseng
iplants)
PROMICCA MINNE ANMONNE PROVIC
ARANPI
Rioe Esiraction
Question iriplots
Answer Ganaration
SEHIND(A
Figure 2: PISTAQ pipeline based on disentangled ex- traction and reasoning. In this model, facts, egg. FRONT(grey car, grey house), extracted from the story and linked by coreference modules, The R- R-Coref equates 'the car' from the question with 'a grey car' in the story and forms query. This query, along with facts, is sent to the spatial reasoner: Finally, the spatial reasoner employs FRONT and BEHIND rules and re- turns True as the answer.
Task The target task is spatial question answer- ing (SQA), which assesses models' ability to com- prehend spatial language and reason over it. Each example includes textual story describing enti- ties and their spatial relations, along with questions asking an implicit relation between entities (e.g.. Figure 1). SQA benchmarks provide two types of questions: YN (Yes/No) queries about the exis- tence of a relation between two groups of entities, and FR (Find Relation) seeks to identify all possi- ble (direct/indirect) relations between them. The answer to these questions is chosen from pro- vided candidate list. For instance, the candidate list for FR questions can be sublist of all relation types in Table 1.
entity, l-entity, B-indicator, and I-indicator) tags. Finally, a softmax layer on the BIO tagger out- put selects the spatial entities' (e.g., 'grey car' or 'plants' in Figure 2) and spatial indicators (c.g., 'in front of" in Figure 2).
Given the output of the spatial role ex- traction module, for each combination of (Trajector, Spatial indicator, Landmark) in each sentence, create a textual input and pass it to BERT model. To indicate the position of each spatial role in the sentence, we use segment embeddings and add if it is a role position and otherwise. The [CLS] output of BERT will be passed to a one-layer MLP that provides the prob- ability for each triplet. To apply the logical rules on the triplets, we need to assign a relation type to each triplet. To this aim, we use another multi- classification layer on the same [CT.S] token identify the spatial types of the triplet. The classes are relation types in Table I alongside class NaN for triplet with no spatial meaning. For instance, in Figure 2, (grey car, in front of, grey house) is a triplet with F RONT as its relation type while (grey house, in front of, grey car) is not a triplet and its relation type is Na.. We use a joint loss function for triplet and relation type classification to train the model.
3.1 Pipeline of Extraction and Reasoning
Here, we describe our suggested pipeline model designed for spatial question answering task, re- ferred as P As shown in the extraction part of Figure 2, the spatial information is extracted first and forms a set of triplets for a story (Facts) and a question (Query). Then a coreference reso- lution module is used to connect these triplets to each other. Given the facts and queries, the spatial reasoner infers all implicit relations. The answer generator next conducts the final answer. Below we describe each module in more detail.
Spatial Role Labeling (SPRL) is the task of iden- tifying and classifying the sputial roles of phrases within a text (including the Trajector, Landmark, and Spatial Indicator) and formalizing their rela- tions (Kordjamshidi et al., 2010). Here, we use the same SPRL modules as in (Mirzaee and Ko- rdjamshidi, 2022). This model first computes the token representation of story and its question us- ing a BERT model. Then BIO tagging layer is applied on the tokens representations using (O, B-
Coreference Resolution Linking the extracted triplets from the stories is another important step required in this task, as different phrases or pro- nouns may refer to same entity. To make such connections, implement coreference resolu-
'Trajector/Landmurtt
PPipeline model for SpaTiAl Question answering
SCCLS, traj. SEP, indic, SEP, lond, SEP, sentence, SEP

---

Not
V[X, Y) Entities
R E (Dir PP)
IFR(X, IF X(Y,) IFY
NOT reeeeree
Inverse
V[X. Y) Entilice
R E (Dir PP}
R (ereerae((X)))
Symmetry
VLX,Y) Entitice
R E (Dia (RCC-PP) -PP]]
-RXX
Transitivity
V(X, Y. Entities
R V
RRX.Y)
Combination
V(X. Y, 2, Entities
R Dir,
M(X.Y)
Table 2: Designed spatial rules (Mirzaee and Kordjamshidi, 2022). Dir: Directional relations (c.g., LEFT), Dis: Distance relations (e.g.. FAR), PP: all Proper parts relations (NTPP, NTPPI, TPPI, TPP), RCC PP: All RCCS relation except proper parts relations. +PP: one of TPP or NTPP. +PPi: one of NTPPi or TPPi.
tion model based on (Lee et al.. 2017) and ex- tract all antecedents for each entity and assign unique id to them. In contrast to previous work, we have extended the model to support plural an- tecedents (e.g., two circles). More details about this model can be found in Appendix C.2. To find the mentions of the question entities in the story and create the queries, we use a Rule-based Coref- erence (R-Coref) based on exact/partial matching. In Figure 2, 'the car' in the question has the same id as 'the grey car' from the story's triplets.
triplet relation type. To have equal positive and neg- ative questions, we reverse the relation in half of the questions. We train BERT-EQ using both original and extra questions by passing the "question ceasstion-ston story" into BERT with answers classification layers.
3.3 PLMs with Explicit Extractions
As another approach, we aim to explore a model that disentangles the extraction and reasoning parts inside a neural model. Here, rather than directly predicting the answer from the output of PLMs (as typically done in the QA task), we introduce ex- plicit layers on top of PLM outputs. These layers are designed to generate representations for enti- ties and pairs of entities, which are then passed to neural layers to identify all relations. We call this model SREQA 8 which is an end-to-end spatial re- lation extraction model designed for QA. Figure 3 illustrates the structure of this model.
Logic-based Spatial Reasoner To do sym- bolic spatial reasoning. we use the reasoner from (Mirzaee and Kordjamshidi, 2022). This rea- soner is implemented in Prolog and utilizes a set of rules on various relation types, as illustrated Ta- ble 2. Given the facts and queries in Prolog format, the spatial reasoner can carry out the reasoning pro- cess and provide answer to any given query. The reasoner matches variables in the program with con- crete values and backtracking search to explore different possibilities for each rule until a solution is found. As shown in Figure 2, the reasoner uses a FRONT and BEHIND rules over the facts and generates the True response for the query.
In this model, we first select the entity men- tions (M)(E1) from the BERT tokens represen- tation and pass it the extraction part shown in Figure 3a. Next, the model computes entity repre- sentation (M (M))) by summing the BERT token representations of all entity's mentions and passing it to an MLP layer. Then for each pair of entities, triplet is created by concatenating the pair's en- tities representations and the BERT [CLS] token representation. This triplet is passed through an MLP layer compute the final pair representations. Next, the reasoning part in Figure 3a, for each re- lation type in Table 1, we use a binary 2-layer MLP classifier to predict the probability of each relation between the pairs. We remove the inconsistent re- lations by selecting one with a higher probability at inference time, e.g., LEFT and RIGHT cannot be true at the same time. The final output is list of all possible relations for each pair. This model is trained using the summation of Focal loss (Lin et al., 2017) of all relation classifiers.
3.2 PLMs Using SPRL Annotations
To have a fair comparison between the QA base- lines and models trained on SrRL supervision, we design BERT-EQ". We convert the SPRL anno- tation into extra YN questions? asking about ex- plicit relations between pair of entities. To gen- erate extra questions, replace triplets from the SPRL annotation into the "Is [Trajector] [Rela- tion*] [Landmark]?" template. The [Trajector] and [Landmark] are the entity phrases in the main sen- tence ignoring pronouns and general names (e.g.. "an object/shape"). The [Relation*] is a relation expression (examples presented in Table 1) for the
Extra Question
'This augmentation does not apply to FR type since it inquires about all relations between the two asked entities.
"Spatial Relation Extraction for QA

---

FoLe) (iure) sno or sunn +/bucakort Arn plante behree the cer 1:00
Extraction
Ressoning
relation is converted to label based on the Yes/No answer. For example, in Figure 3a, the question relation is 'BEHIND" and the answer is Yes, so the label for the BEHIND classifier 1.
Polie:: Hoge www.attaaa....o wor.ioo Son
Select Erdly Mension
Mr (B,)
MLL (Entay tep)
(C_S]
MLP (Triplet rep)
Coontarr
Answe Goneration
(
RIGHT
BERT
M-
pair
(Rg)
BEHIND Conceter
We evaluate the SREQA model's performance in predicting the accurate answers of the test set's questions same as training step 2.
M- M-
Chesaber
(a) Model structure. First, entity mentions such as 'plants' and grey car' are selected from the BERT output and the entity representation is formed. Next, triplets like ('plants', 'car'.
4
Experiments
4.1 Datasets
(CLS)) are generated and fed into the reasoning component. The collective output of all relation classifiers determines the relationships hetween each pair. * *ll hatched parts are trained end-to-end. The rest of the data is obtained from annotations she-the-seell modules.
SPARTQA is SQA dataset in which examples contain story and multiple YNº and FR ques- tions that require multi-hop spatial reasoning to be answered. The stories in this dataset describe rela- tions between entities in a controlled (toy task) envi- ronment. This dataset contains a large synthesized part, SPARTQA- APARTQA-AUTO, AUTO, and more complex small human-generane subset, SPARTQA-HUMAN All stories and questions in this dataset also contain SPRL annotations.
Source of
Entity Mentions
Relation Supervision
All story relations
Entiy Coraf annolition
SpRL Coref annotation Spatial Reasoner
(Training stsp#1)
Question relation
Trained antity extraction
QA Answor
(Training slep!/2)
-R-Coref
(b) The source of supervision in each step of training. In steph1, we train the model on story relations, and stepl12, we only train it on question relations. These modules and data are the same as the ones used in PISTAQ.
SPARTUN is an extension of SPARTQA with YN and FR questions containing SPRL annotations. Compared to SPARTQA, the vocabulary and rela- tion types in this dataset are extended, and it covers more relation types, rules, and spatial expressions to describe relations between entities.
Figure 3: The SREQA model with explicit neural layers to disentangle extraction and reasoning part.
We train SREQA in two separate steps. In the first step, the model is trained on a relation ex- traction task which extracts all direct and indirect relations between each pair of entities only from stories. The top row of Figure 3b shows the anno- tation and modules employed in this step to gather the necessary supervision. We use the entity and coreference annotation to select the entity mentions from the BERT output. To compute the relations supervision for each pair of entities, we employ the spatial reasoner from PISTAQ and apply it to the direct relations (triplets) from the SPRL annotation, which are connected to each other by coreference annotations. This step of training is only feasible for datasets with available SPRL and coreference annotations.
RESQ is SQA dataset with Yes/No questions over the human-generated text describing spatial relations in real-world settings. The texts of this dataset are collected from MSPRL dataset (Kord- jamshidi et al., 2017) (See Figure 1), which de- scribe some spatial relations in pictures of Im- ageCLEF (Grubinger et al., 2006). Also, the MSPRL dataset already contains some SPRL an- notations. To answer some of the questions in this dataset, extra spatial-commonsenss needed (e.g., a roof is on the top of buildings).
information is
4.2 Model Configurations & Baselines
We compare the models described in section with the following baselines.
In the next step, we further train SREQA on extracting questions relation using QA supervision. As shown in the bottom row of Figure 3b, we em- ploy the trained spatial role extraction model used in PISTAQ to identify the entities in the question and use R-Coref to find the mentions of these enti- ties in the story. To obtain the relation supervision, we convert the question answers to relation labels. In FR, the label is similar to the actual answer, which is a list of all relations. In YN, the question
Majority Baseline: This baseline selects the most frequent answer(s) in each dataset.
GT-PISTAQ: This model uses ground truth (GT) values of all involved modules in PISTAQ elimi- nate the effect of error propagation in the pipeline. This baseline is used to evaluate the alignments between the questions and story entities and the reasoning module in solving the QA task. It also
"We ignore 'Dont know" answers in YN question and change them to No

---

Model
Supervisions
Rule-based Modules
their supervision as well as extra off-the-shelf rule-based modules employed in them.
BERT
QA
GPT3 GPT3,5/u Afo
ex]
Since the performance of extraction modules, Spatial Role Labeling (SPRL) and Coreference Resolution (Coref), directly contribute to the final accuracy of the designed models, we have evalu- ated these modules and reported the results in Ta- ble 4. We choose the best modules on each dataset for experiments. For detailed discussion on the performance of these modules, see Appendix C.
GPT3.5/ f13.5/ CoT
QAI8 ex) CoT
BERT-EQ
QA
SpRL(S)
SREQA
QA SpRL(all)+Crre
Reasoner, R-Corel
SREQA*
SpRL(Q)
R-Coref
PISTAQ
SpRizal) + Coret
Reasoner, R Coref
PISTAQ
Reasoner, R Coref
Table 3: The list of annotations from the target bench- marks and rule-hased modules employed each model. We quarter of SPRL annotations to train the mod- ules on auto-generated benchmarks. S: Stories, Q: Ques- tions, All: Stories Questions.
Dataset MSPRL MSPRI.*
Coref
SRole
SRel
SType
88.59
69.12
19.79
88.03
71.23
23.65
gives upper bound for the performance of the pipeline model, as the extraction part is perfect. BERT: We select BERT as a candidate PLM that entangles the extraction and reasoning steps. In this model, the input of the "question+story" is passed to the BERT, and the [CLSS] representation is used to do the answer classification.
HUMAN
82.16
55.8
S: 57.43
43.79
Q: 52.55
39.34
S: 60.24
48.74 74
HUMAN
81,51
72.53
Q: 61.53
48.07
S: 99.72
99.05
SPARTQA
99.83
99.92
Q: 98.36
98.62
39.19.8
98.57
SPARTUN
99.35
99.96
Q: 97.68
98.11
GPT3.5: GPT3.5 (Brown et al., 2020) base- lines (GPT3.5 text-davinci-003) is selected candidate of generative larger language models which already passes many SOTAs in reasoning tasks (Bang et al., 2023; Kojima et al., 2022). We use Zero_shol and Few_shol (In-context learn- ing with few examples) settings to evaluate this benchmarks. We
Table 4: Performance of the extraction modules. Q:
question. S: stories. HUMAN: SPARTQA-HUMAN SPARTQA: SPARTQA-
"Further pretraining modules on SPARTUN. We report macro Fl for SPRL. and the accuracy of the Coref modules.
5.1 Result Controlled Environment
model on the human-generan also evaluate the Chain-of-Thoughh (CoT)) prompt- ing method (Wei al., 2022) to extend the prompts with marttlly-written reasoning steps. The format of the input and some prompt examples are pre- sented in Appendix E.
Table 5 shows the performance of models on two auto-generated benchmarks, SPARTUN and SPARTQA-AUTO We can observe that PISTAQ outperforms all PLM baselines and SREQA. This outcome first highlights the effectiveness of the extraction and symbolic reasoning pipeline com- pared to PLMs addressing deterministic reason- ing within controlled environment. Sccond, it shows that disentangling extraction and reasoning as a pipeline works better than explicit neural lay- ers in SQA with a controlled environment. The complexity of these environments is more related to conducting several reasoning steps and demands accurate logical computations where rule-based reasoner excels. Thus, the result of PISTAQ with a rule-based reasoner module is also higher than SREQA with a neural reasoner.
Transfer learning has already demonstrated sig- nificant enhancements in numerous deep learning tasks (Soroushmojdehi et al., 2022; Rajaby Faghihi and Kordjamshidi, 2021). Thus, when applica- ble, further train models on SPARTUN syn- thetic data shown by *** The datasets' examples and statistics and more details of the experimen- tal setups and configurations are provided in Ap- pendix A and B. All codes are publicly available at https:/githbbb.omm/Raik333P/P SASEEA...AP.......QQ.....
5 Results and Discussion
The superior performance of PISTAQ over BERT suggests that SPRL annotations are more effee- tive in the PISTAQ pipeline than when utilized in BERT-EQ in the form of QA supervision. Note that the extraction modules of PISTAQ achieve per- feet results on auto geterreedetted benchmarks while trained only on a quarter the SPRL annotations
Here, we discuss the influence of disentangling extraction and reasoning manifested in PISTAQ and SREQA models compared to various end-to-end models with no explicit design for these modules, such as BERT, BERT-EQ, and GPT3.5. Table shows the list these models with the sources of

---

SPARTUN
SPARTQA-
#
Models
#
Models
YN
FR
YN
FR
Acc
P
FI
1
Majority baseline
53.62
14.23
51.82
44.35
1
Majority baseline
52.44
29.87
14.28
6.57
2
GT-PISTAQ
99.07
99.43
99.51
98.99
2
GT-PISTAQ
79.72
96.38
66.04
75.16
3
BERT
91.80
91.80
84.88
94.17
3
BERT
51.74
30.74
30.13
28.17
4
BERT-FQ
90.71
N/A
85.60
N/A
4
BERT*
48.95
60.96
49.10
50.56
5
SREQA
88.21
83.31
85.11
86.88
GPT3,5 Zerua.ze
45.45
40.13
16.51
Feu
45.20
44.28
6
PISTAQ
96.37
94.52
98.02
7
GPT3.5 Foucahok CCoT
62.93
57.18
37.92
38.47
8
BERT-EQ
50.34
Table 5: Results auto seoo-generated datasets. We use the accuracy metric for both YN and FR questions.
9
BERT-FQ*
45.45
10
SREQA
53.23
15.68
13.85
13.70
11
SREQA
46.96
18.70
25.79
24.61
PISTAQ
46.80
as shown Table 5. However, BERT-EQ uses all
the original dataset questions and extra questions created from the full SPRL annotations.
Table 6: Results on SPARTQA-HUMAN We use ac-
curacy on questions and average Precision (P), Re- call (R), and Macro-F1 FR question types. *Using SPARTUN supervision for further training.
Table 6 demonstrates the results of models on SPARTQA-HUMAN with controlled environment setting. As can be seen, our proposed pipeline, PISTAQ, outperforms the PLMs by a margin of 15% on YN questions, even though the extraction modules, shown in Table 4, perform low. This low performance is due to the ambiguity of human lan- guage and smaller training data. We also evaluate PISTAQ SPARTQA-HUMAN FR questions us- ing Macro_ff score on all relation types. PISTAQ outperforms all other baselines on FR questions, except for B
Models
Accuracy
Majority baseline
50.21
2
BERT
57.37
3
BERT Zero_shot
49.18
BERT*
63.60
GPT3.5 2 Zero_shosss
5
60.32
GPT3.5 Few_shot
65,90
GPT3.5 F eew_whooo +CoT
7
67.05
BERT-EQ
56.55
BERT-EQ Zero_shot
51.96
10
BERT-EQ
61.47
11
SREQA
53.15
There are two main reasons behind the inconsis- tency in performance between YN and FR question types. The first reason is the complexity of the YN questions, which goes beyond the basics of spa- tial reasoning and is due to using quantifiers (e.g., all circles, any object). While previous studies have demonstrated that PLMs struggle with quanti- fiers (Mirzaee et al., 2021), the reasoning module in PISTAQ can adeptly handle them without any per- formance loss. Second, further analysis indicates that PISTAQ predicts 'No' when relationship is not extracted, which can be correct when the swer is 'No'. However, in FR, missed extraction causes false negative which decreases FI score.
12
SREQA Zeroos=errohhos
53.32
13
SREQA*
69.50
PISTAQ MSPRL
41.96
PISTAQ SPARTUN+MSPRL
47.21
16
Human
90.38
Table 7: Result RESQ. Further training SPAR- TUN. The Zero_sha shot refers to evaluation without fur- ther training RESQ MSPRL training data.
notations, employ the model trained on SPAR- TUN for this purpose. Rows 14 and 15 in Table 7 show the performance of the PISTAQ on RESQ is inferior compared other baselines. To find the reason, we analyze the first 25 questions from the RESQ test set. We find that 18 out of 25 questions required spatial commonsense information and can- not be answered solely based on the given relations in the stories. From the remaining 7 questions, only 2 can be answered using the SpRL annotations pro- vided in the MSPRL dataset. Some examples of this analysis are provided in Appendix D. Hence, the low performance PISTAQ attributed to first the absence integrating commonsense informa- tion in this model and, second, the errors in the
5.2 Results on Real-world Setting
We select RESQ as an SQA dataset with realistic settings and present the result of models on this dataset in Table 7.
To evaluate PISTAQ RESQ, we begin by adapting its extraction modules through training on the corresponding dataset. We train the SPRL modules on both MSPRL and SPARTUN, and the performance of these models is presented in Ta- ble 4. As the L dataset lacks coreference an-

---

SpaRTDA SARTT04-Hummm Itumnn
Story: There block called A. Inside there's red oval that is above blue triangle ... There block called which is to right touching Inside there are triangles, red one,
one, large yellow one, brown one, and orange one The large yellow triangle is touching the top edge B. The green triangle is touching the right edge B...
Quection t: the green thing above the yollow thing? Answer: No
Quextion 2: Does have all of the triangles? Answer: No
OPT3.5 (Prodiction)
01: The green triangle the right of the large yellow triangle and the large yellow triangle touching the top edge of B. Using inverse rule, the green
triangle is below the yellow triangle. The onswer is
02: There are triançles mentioned in block B. red one, graen one. large yellow one, in block B, the answer Yes.
and orange one. All these triangles
DP14 (Prediction)
01: The green triangle is touching the right edge and the large yellow triangle is touching the top edge of it not mentioned the green
triangle is above or below the yellow triangle. The answer is Don't Know.
02: There a blue triangle in black A, and there are 5 triangles black B. Since there is triangle block A, not all triangles are block B. The answer
PaLM2 iPrediction)
01: The green triangle is touching the right edge of and the large yellow triangle touching the top edge of 8. The right edge of below the top idge of fB, So the green triangle is below the yellow triangle. The answer No.
Q2: There are triangles in B. There is a red triangle in B. There is a green triangle B. There is large yellow triangle in B. There is brown is Yes.
Iriangle There an orange triangle B. The
02: Extracted Facts (0PT3, Turbol:
NIPPlblue triangle, Al TPPtred triangle, left edge of B), IPPtorange triangle, left edge of B)... NTPPUTPPIIB, blue triangle), NTTpUU NTPFU/PPI [a, red triangle). NTPPU/TPPI(E, orange triangle)....
PISTAO
Extracted Queries Serries/GPPT, (GPT3, Turbo}
Symbolic Reasoning (Rale-based Reasoner):
True &
Answer
An example of Large Language Models (LLMs) prediction on SQA task evaluated with CoT prompting. The last row shows an example of using Gurboo-Turbo for information extraction in PISTAQ. See Appendix E for seto_shot examples.
Figure 4:
extraction modules, which are propagated to the reasoning modules.
perior performance in deterministic spatial reason- ing within controlled settings. Moreover, explicitly training the extraction module proves advantageous in leveraging SPRL annotation more effectively compared using this annotation in QA format in the end-to-end training. Comparison between dis- entangling extraction and reasoning as a pipeline and incorporating them within an end-to-end model demonstrates that the end-to-end model performs better in realistic domains even better than PLMs. The end-to-end architecture of this model effec- tively enhances the generalization in the real-world setting and addresses some of the limitations of rule coverage and commonsense knowledge.
As shown in Table 7, the best result on RESQ is achieved by SREQA* model. Compared to SREQA, SREQA* is trained on SPARTUN in- stead of MSPRI. 10 in the first step of the training. MSPRL lacks some SPRL and coreference annota- tions to answer RESQ questions. In the absence of this information, collecting the supervision for the first phase of training results in a significant number of missed relations. Therefore, as shown in row 11 of Table 7, employing MSPRL in the first training phase decreases the performance while replacing it with SPARTUN in SREQA* significantly en- hances the results.
5.3 LLMs on Spatial Reasoning
SREQA* * surpasses the PLMs trained on QA and QA+SPRI annotation, showcasing the advan- tage of the design of this model in utilizing QA and SpRL data within explicit extraction layers and the data preprocessing. Also, the better performance of this model compared to PISTAQ demonstrates how the end-to-end structure of SREQA can han- dle the errors from the extraction part and also can capture some rules and commonsense knowledge from RESQ training data that are not explicitly supported in the symbolic reasoner.
Recent research shows the high performance of LLMs with zerol few_show shoth setting many tasks (Chowdhery et al., 2022; Brown et al., 2020). However, (Bang et al., 2023) shows that Chat- GPT (GPT3.5-Turbo) with zero_shoe scoo evaluation cannot perform well on SQA task using SPARTQA- HUMAN test cases. Similarly, our experiments, as shown in Tables 6 and 7, show the lower perfor- mance of GPT3.5 (davinci) with zerol fewl few_seesssl shot settings compared to human and our models PIS- TAQ and SREQA. Figure 4, shows an example of three LLMs, GPT3.5, GPT4 and PaLM2 on SPARTQA-HUMAN example"' (complete figure
In conclusion, compared to PLMs, disentangling extraction and reasoning pipeline indicates su-
10 As mentioned, we use the MSPRL. annotation for RESQ
Due to the limited resources, we only use GPT4 and PaLM2 on few examples to evaluate their performance
datasct.

---

Bory:
tetween and picture the well above the bods.
a photo of room with white walls, two single beds with right toots
neural models for reasoning tasks.
Useerio:
Ara the beds below the picture? Answor les
[ba [Taapicte picture, the bocrl) 2fx] pictune the war]]
Conclusion and Future Works
Sury Facts:
Facts: right22 belowi 0) nean2: 01
[two bodd hodr] pictune",, (the the becs]
5P13
We investigate the benefits of disentangling the processes of extracting spatial information and rea- soning over them. To this end, we devised a series of experiments utilizing PLMs for spatial informa- tion extraction coupled with symbolic reasoner for inferring indirect relations. The outcomes of our experiments provide noteworthy insights: (1) Our observations in controlled experimental condi- tions demonstrate that disentangling extraction and symbolic reasoning, compared to PLMs, enhances the models' reasoning capabilities, even with com- parable reduced supervision. (2) Despite the acknowledged fragillty of symbolic reasoning in real-world domains, our experiments highlight that employing explicit extraction layers and utilizing the same symbolic reasoner in data preprocessing enhances the reasoning capabilities of models. (3) Despite the limitations of LLMs in spatial reason- ing, harnessing their potential for information ex- traction within disentangled structure of Extrac- tion and Reasoning can yield significant benefits. All of these results emphasize the advantage of dis- entangling the extraction and reasoning in spatial language understanding.
Facts: shovlls sbooal,
Darriss
DERY
belext,, selow/o '1
3P13
BERT
belevill, 0]- Falso, bolowtl, Fase Anower No
Reasoning:
SP13
belevl3 Oj= Trus, belowks, G]- Faiee knower a Yes
Figure 5: An example of using BERT-based SPRL and GPT3.5 as information extraction in PISTAQ on a RESQ example.
including zero_shot examples is presented in Ap- pendix E). Although Wei al. shows that using CoT prompting improves the performance PaLM on multi-step reasoning task, its spatial reasoning capabilities still does not meet the expectation.
5.3.1 LLMs as Extraction Module in PISTAQ
A recent study (Shen et al.. 2023) shows that LLMs have a promising performance in information re- trieval. Building upon this, we employ L.M,, GPT3.5-Turbo with few_shot prompting to ex- tract information from a set of SPARTQA-HUMAN and RESQ examples that do not necessitate com- monsense reasoning for answering. The extracted information is subsequently utilized within the framework of PISTAQ.
In future research, an intriguing direction is in- corporating spatial commonsense knowledge us- ing LLMs as an extraction module in the pipeline of extraction and reasoning. Additionally, the model's applicability extends beyond spatial rea- soning, making it suitable for various reasoning tasks involving logical rules, such as temporal or arithmetic reasoning.
The results, illustrated in the last row of Fig- ure 4, highlight how the combination LLM ex- traction and symbolic reasoning enables answering questions that LLMs struggle to address. Further- more, Figure 5 provides a comparison between the trained BERT- SPRL extraction modules and GPT3.5 with few__whwweeee shotssthttth prompting in PISTAQ. It is evident that GPT3.5 extracts more accurate in- formation, leading to correct answers. As we men- tioned before, out of 25 sampled questions from RESQ, only can be solved without relying on spa- tial commonsense information. Our experimental result shows that PISTAQ using LLM as extraction modules can solve all of these questions.
Acknowledgements
This project is partially supported by the Na- tional Science Foundation (NSF) CAREER award 202826. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. We thank all reviewers for their helpful comments and suggestions. We would like to express our grati- tude to Hossein Rajaby Faghihi for his valuable discussions and input to this paper.
Based on these findings, leveraging LLMs in PISTAQ to mitigate errors stemming from the SPRL extraction modules rather than relying solely on LLMs for reasoning can be an interesting future research direction. This insight emphasizes the importance of considering new approaches for in- corporating explicit reasoning modules whenever possible instead of counting solely on black-box
Limitations
Our model is evaluated on a Spatial Reasoning task using specifically designed spatial logical rules.
SQA tasks.

---

However, this methodology can be readily extended other reasoning tasks that involve limited set of logical rules, which can be implemented us- ing logic programming techniques. The extraction modules provided in this paper are task-specific and do not perform well on other domains, but they can be fine-tuned on other tasks easily. Using LLM in the extraction phase can also deal with this issue. Also, using MSPRL. annotation ReSQ(which this data is provided on) decreases the performance of our models. This annotation does not contain the whole existing relations in the context. The evaluation of the reasoning module is based on the existing datasets. However, we cannot guarantee that they cover all possible combinations between spatial rules and relation types. Many questions in RESQ need spatial commonsense to be answered. As result, due the limitation of our symbolic spatial reasoner, the performance of the pipeline model much lower than what we expected. Due to the high cost of training GPT3.5 on large syn- thetic data, we cannot fine-tune the whole GPT3.5 and only provide the GPT3.5 with Few_ shru_ssotttie learn- ing on small human- gennrratd benchmarks. Also, due to the limited access, we can only test PaLM2 and GPT4 on few examples.
semantic parsing in understanding procedural text. arXiv preprint arXiv:2302.06822
Michael Grubinger, Paul Clough, Henning Müller, and
Thomas Deselaers. 2006. The iapr tc-12 benchmark: A new evaluation resource for visual information systems. In International workshop ontolmage, vol- ume 2.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processing systems, 35:22199- 22213.
Parisa Kordjamshidi. Marie-Francing Moens, and Mar- tijn van Otterlo. 2010. Spatial Role Labeling: Task definition and annotation scheme. In Proceedings of the Seventh conference on International Lan- guage Resources and Evaluation (LREC' 10), pages 413-420. European Language Resources Association (ELRA).
Parisa Kordjamshidi, Taher Rahgooy, Marne-Fra Moens, James Pustejovsky, Umar Manzoor, and Kirk Roberts. 2017. Clef 2017: Multimodal spatial role labeling (msprl) task overview. In International Con- ference of the Cross-Languch Evaluation Forum for European Languages, pages 367-376. Springer.
Hung Le, Truyen Tran, and Svetha Venkatesh. 2020. Self-attentive associative memory. International Conference on Machine Learning, pages 5682-5691. PMLR
References
Kenton Lee, Luheng He, Mike I Lewis, and Luke Zettle- moyer. 2017. End-to-end neural coreference reso- lution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188 197, Copenhagen, Denmark. Association for Computational Linguistics.
Yejin Bang. Samuel Cahyawijaya, Naycon Lee, Wen- liang Dai, Dan Su, Bryan Wille, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilin- gual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1877--919
Xiao Liu, Da Yin, Yansong Feng, and Dongyan 7hao 2022. Things not written in text: Exploring spatial commonsense from visual signals. of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 2365 2276, Dublin, Ireland, Association for Computational Linguistics.
Proceedings
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint 17x.v.1711.05111.
Emmanuelle-Anne Dietz, Steffen Hölldobler, and Raphael Höps. 2015. A computational logic ap- proach to human spatial reasoning. In 2015 IEEE Symposium Series on Computational Intelligence, pages 1627-1634. IEEE.
Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt, 2022. A survey on multi-hop question answering and gen- eration. arXiv preprint arXiv:2204.09140.
Ana Cristina Mendes, Luísa Coheur, and Paula Vaz Lobo. 2010. Named entity recognition in questions: Towards golden collection. In LRECC.
Hossein Rajaby Faghihi, Parisa Kordjamshidi,
Choh Man Teng, and James Allen. 2023. The role of

---

Pasquale Minervini, Sebastian Riedel, Pontus Stenetorp. Edward Grefenstette, and Tim Rocktäschel. 2020. Learning reasoning strategies in end-to-end differ- entiable proving. In International Conference on Machine Learning, pages 6938-6949. PMLR.
Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Tianyi Zhou, and Daxin Jiang. 2023. Large language models are strong zero-shot retriever. retrivee.. arXiv preprint arXiv:2304. 14233.
Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. 2022. Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts. In Proceedings of the Asso- ciation for the Advancement of Artificial Intelligence. AAAI '22.
Roshanak Mirzaee and Parisa Kordjamshidi. 2022. Transfer learning with synthetic corpora for spatial role labeling and reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natu- Abu
ral Language Processing, pages Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Rahil Soroushmojdehi, Sina Javadzadeh, Alessandra Pedrocchi, Marta Gandolla, et al. 2022. Transfer learning hand movement intention detection based on surface electromyography signals. Frontiers in Neuroscience, 16:1-15
Roshanak Mirzaee, Hosscin Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. 2021. SPARTQA: A textual question answering benchmark for spatial reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. pages 4582-4598, Online. Association for Computational Linguistics.
Keith Stenning and Michiel Van Lambalgen. 2012. Hu- man reasoning and cognitive science. MIT Press.
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. arXiv preprint 103ix:1533008895
Diego Mollá, Menno Van Zaanen, Daniel Smith, et al. 2006. Named entity recognition for question answer- ing.
Sagar Gubbi Venkatesh, Anirban Biswas, Raviteja Upadrashta, Vikram Srinivasan, Partha Talukdar, and Bharadwaj Amrutur. 2021. Spatial reasoning from natural language instructions for robot manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11196-111 11202. IEBE,
Maxwell Nye. Michael Tessler, Josh Tenenbaum, and Brenden M Lake. 2021. Improving coherence and consistency in neural sequence models with dual- system, neuro-symbolic reasoning. Advances in Neural Information Processing Systems, 34:25192- 25204.
Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. 36th Conference on Neural Infor- mation Processing Systems (NeurIPS 2022).
Batu Ozturkler, Nikolay Malkin, Zhen Wang, and Neho- jsa Jojic. 2022. Thinksum: Probabilistic reasoning over sets using large language models. arXiv preprint arXiv:2210.01293.
Yue Zhang. Quan Guo, and Parisa Kordjamshidi. 2021. Towards navigation by reasoning over spatial con- figurations. In P Proceedings of Second International Combined Workshop on Spatial Language Under- standing and Grounded Communication for Robotics. pages 42-52, Online. Association for Computational Linguisties.
Rasmus Berg Palm, Uirich Paquet, and Ole Winther. 2017. Recurrent relational networks. arXiv preprint arXiv./71..0002..
Meng Qu, Junkun Chen, Louis-Pascal Xhonneux. Yoshua Bengio, and Jian Tang. 2021. Rnnlogic: Learning logic rules for reasoning on knowledge graphs. In International Conference on Learning Representations.
Chen Zheng and Parisa Kordjamshidi. 2021. Rela- tional gating for" what if" reasoning. arXiv preprint N1i5:2105.13444
Rui Zhu, Krzysztof Janowicz, Ling Cai. and Gengehen Mai. 2022. Reasoning over higher-order qualitative spatial relations via spatially explicit neural networks. International Journal of Geographical Information Science, pages 1-32.
Hossein Rajaby Faghihi and Parisa Kordjamshidi. 2021. Time-stamped language model: Teaching language models to understand the flow of events. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4560 Online. Association for Computational Linguisties.
Dan Shen and Mirella Lapata. 2007. Using semantic roles to improve question answering. In Proceedings of the 2007 Joint Conference on Empirical Meth- ods in Natural Language Processing and Computa- tional Natural Language Learning
pages 12-21, Prague, Czech Republic. Association for Computational Linguistics.

---

A Statistic Information
Train
Test
All
Sentences
600
613
1213
This section presents statistical information regard- ing dataset sizes and additional analyses conducted on the evaluation sets of human -ennann-eeertedd datasets.
Trajectors
716
874
1590
Landmarks
612
573
1185
Spatial Indicators
666
795
1461
Table 8 provides the number of questions in the training and evaluation sets of the SQA bench- marks. Tables 9 and 10 present the sentence and number of relation triplets within the SPRI. an- notation for each dataset, respectively. Table 11 illustrates a comprehensive breakdown of the size of Role and Relation sets in the MSPRL datasct.
Spatial Triplets
761
939
1700
Table 11: MSPRL size (Kordjamshidi et al., 2017).
Three boxes called one, two and three exist in an image. Box one contains big yellow melon and small orange watermelon. Box two has a small yellow apple. A small orange apple is inside and touching this box. Box one in box three. Box two is to the south of, far from and the west of box three. A small yellow watermelon is inside box three.
Dataset
Train
Dev
Test
SPARTQA- AUTO (YN)
26152
3860
3896
SPARTQA- AUTO (FR)
25744
3780
3797
SPARTQA HUMAN HUMTTA
(YN)
162
51
143
SPARTQA HUMAN (FR)
149
SPARTUN (YN)
20334
3152
3193
SPARTUN (FR)
18400
2818
2830
RESQ(YN)
1008
333
610
Q: Is the yellow apple to the west the yellow watermelon? Yes
Table 8: Number of questions in training and evaluation sets of SQA benchmarks.
Q: Where is box two relative to the yellow watermelon? Left, Below, Far
Dataset
Train
Dev
Test
Figure 6: An example of SPARTUN dataset from (Mirzaee and Kordjamshidi, 2022).
SPARTQA- AUTOO (story)
36420
16214
16336
SPARTQA AUTOO (question)
53488
15092
15216
SPARTQA-HUMA (story)
389
213
584
HUMANN (question)
623
190
549
information is not available, resulting in more No and fewer Yes predictions compared to other mod- cls, presented in Table 12. The number of truc positive predictions for PISTAQ more than other baselines, and result, it achieves higher accuracy.
SPARTUN (story)
68048
9720
10013
SPARTUN (question)
41177
6355
6340
MSPRI.
600
Table 9: Number of sentences in SPRL annotations of each benchmarks. train models on the auto-generated benchmarks, only the quarter of training exam- ples from SPARTUN and SPARTQA- QUToO..
Predictions/ Answer
Yes
No
No prediction
Ground Truth
74
69
BERT
131
12
Dataset
Train
Dev
Test
BERT
54
SPARTQA-AUS (stury)
159712
22029
21957
PISTAQ
43
97
SPARTQA-AUTE (question)
232187
34903
35011
SPARTQA-HUMAN
HUMAN (story)
176
99
272
Table 12: Detailed information about the prediction of
SPARTQA HUMAN (question)
155
127
367
PISTAQ and BERT on SPARTQA- -UUMAN YN ques- tions. "No prediction" is related to the PISTAQ model when no correct SPRL extraction was made for the text of the question, and result, have no answer pre- diction.
SPARTUN (story)
48368
7031
7191
SPARTUN (question)
38734
5970
6023
MSPRL.
761
939
Table 10: Number of triplets in SPRL annotations of each benchmarks.
A.2 SPRL Annotations in MSPRL
A.1 Analyzing SPARTQA-HUMAN YN
RESQ is built the human-written context of MSPRL dataset which includes SPRL annotations. However, using this annotation in BERT-EQ and SREQA models causes lower performance (Check the result on Tablel 14). Our analysis shows that the
We conducted additional evaluations on the supe- rior performance of PISTAQ over other baseline models on SPARTQA -PARTQA-HUMAN YN questions. As explained before, PISTAQ tends to predict No when

---

3.30 GHz) CPU and Titan RTX with NVLink GPU.
Story: behind a bar with chairs and two people. and bench with one person lying on it.. Upper level with doors and a blue rail.
mSpRL annotation:
Triplet: Behind (a bar (id: 11), behind, it (d::
For GPT3.3, we use Instruct-GPT, davinci- 00313 The cost for running GPT3.5 on the human- generated benchmarks was 0.002S per 1K tokens. For GPT3.5 as information extraction, we use GPT3.5 turbo (a.k.a ChatGPT) with cost of 0.0001$ per 1k tokens. We also use the GPT4 playground in OpenAl and PaLM2 playground find the prediction of examples in Figure 11.
Triplet: Bahind bench (id: 12), behind, (id: Triplet: EC (one person (id: 13), on, (id: (2))
Question 1:
people bahind bar?
Answer: Yes
Predicted answer besed on mSpRL annotations: No
Question 2:
the doer above the bar?
Answer: Yes
Predicted answer on
Predicted answer besed on the commonsense (upper lovel is ahove the levelj:: Yes
Figure 7: An example of the limitation of MSPRL and coreference annotation to answer RESQ question. The answer of the questions predicted wrongly due two main reasons. First, the missed commonsense knowl- edge in question and second, the limited coverage of ground truth annotation in MSPRL. in question 2.
C Extraction and Reasoning Modules
Here, we discuss each module used in PISTAQ and their performance including the Spatial Role Label- ing (SPRL), Coreference Resolution, and Spatial reasoner.
SPRL annotations of MSPRL are not fully practical in our work due to two main reasons:
triplet dss (frpped
Spatial Type Cls (fope)
1. Missed annotations: As shown in Figure there are many missed annotations for each text, e.g., NTPP(bar, with, chair).
Spatial Relation (triplet) Extraction
CLS
Thim
0
2. No coreference
The coreference is not sup- ported in this dataset, c.g., "L2: it" and "T2: bench" are the same entity with different mentions, but they are mentioned with differ- ent ids. These missed coreferences result in fewer connections between entities and fewer inferred relations.
entity: (Trajector)
Indicatorz
entity,
Sentence
(spatial indicatar) (Landmark)
List of all spatial entities
List of all spotial Indicators
SP
E
O
Spatial Role (entity and spatial, indicator) extraction Extracting entiy spanil indicators
B Models and modules configuration
CLS T
T7
SEP
We use the huggingFace implementation of pre- trained BERT base models, which have 768 hid- den dimensions. All models are trained on the training set, evaluated the dev set, and reported the result on the test set. For training, we train the model until no changes happen on the dev set and then store and use the best model on the dev set. use A (Loshchilov and Hutter, 2017)), and learning rates from 2 10 6 (depends on the task and datasets) on all models and modules. For the extraction modules and the baselines, we used the same configuration and set- ting previous works (Mirzaee and Kordjamshidi, 2022). For SREQA models of 2 1 10-6 for SREQA(story and SREQ (question) respectively. To run the models we use machine with Intel Core i9-9820X (10 cores,
A car parking in front of house Segment Embedding: (01 1110010111111
Figure & Spatial role labeling model includes two sepa- rately trained modules. E: entity, SP: spatial indicators. As an example, triplet (a grey house, front A grey car) correct and the 'spatial_type (A grey car, front, a grey house) is incorrect, and the "spatial_ _ypletype== = NaN". Image from (Mirzaee and Kord- jamshidi, 2022)
FRONT", and
5
C.1 Spatial Role Labeling (SPRL)
The SPRL module, shown in Figure 8 is divided into three sub-modules, namely, spatial role extrac- tion (SRole), spatial relation extraction (SRel) and spatial type classification (SType). We only
learning rates
13 from ht /btta.bb/aaaae/eeeeeeee https///eea...onii...... openai.comm
14 "Sinnee the questions(Q) and stories(S) have different anno- tations (questions have missing roles,, we separately train and test the SRel and SType modules
12 httpps: http///uuggnngaaaccce att.a//aaaaaaassrrrrrr//////// bodel_doc/bers html

---

use these modules on sentences that convey spa- tial information in each benchmark. To measure the performance of SPRL modules, we use the macro average of Fl measure for each label. These modules are evaluated on three datasets that pro- vide SPRL annotations, MSPRL, SPARTQA, and SPARTUN. When training the SPRL module on nuto-generated benchmarks, we achieved perfor- mance of 100% using only quarter of the training data, therefore we stopped further training.
Datasets
Q-TYPE
Total
A
C
R
YN
18
7
10
1
SPARTQA-AUTO
FR
38
5
20
13
YN
4
9
0
SPARTUN
FR
35
0
35
0
SPARTQA-HUMAN
YN
20
6
3
Table 13: Analyzing wrong predictions in GT-
PISTAQ. A: Missing/errors Annotation, C: rule-based Coreference issues in connecting extracted information, R: Shortcomings of the Reasoner.
As shown in Table 4, SPRL sub-modules achieve a high performance on synthetic datasets, SPARTQA and SPARTUN. This good performance is because these datasets may contain less ambigu- ity in the natural language expressions. Therefore, the BERT-base models can easily capture the syn- tactic patterns needed for extracting the roles and direct relations from the large training set.
grey car), Front(the car, church), id 1= cat, a grey car, the car, and a church. So we create new triplets in the form of BELOW(I, 2) and Front(2, 3).
To train the model, we pair each mention with its previous antecedent and use cross-entropy loss to penalize the model if the correct pair is not chosen. For singletons and starting mention of objects, the model should return class O, which is the [CLS] token. Since the previous model does not support the plural antecedent (e.g., two circles), we include that by considering shared entity in pairs like both (two circles, the black circle) and (two circles, the blue circle) are true pairs.
C.2 Coreference Resolution (Coref) in Spatial Reasoning
Softmax (P(s: P))
(the company. <)
(th company, General Electric)
*(the company the Postal Service)
As an instance of the importance of coreference resolution in spatial reasoning, consider this con- text "block A has one black and one green circle. The black circle above a yellow square. The yel- low square is to the right of the green circle. Which object in block A is to the left of yellow square" The reasoner must know that the NTPPP(block A, green circle) and RIGHT the yellow square, the green circle) are talking about the same object to connect them via transitivity and find the answer. To evaluate the coreference resolution mod- ule (Coref in Table 4), we compute the accuracy of the pairs predicted as Corefs. The Coref model achieves high performance on all datasets. The performance is slightly lower on the SPARTQA- HUMAN dataset when SPARTUN is employed for additional pre-training. However, we observed that there are many errors the annotations in SPARTQA AUMMAN and the pre-trained model is, in fact, making more accurate predictions than what is reflected in the evaluation.
Coreference scure (e)
Antecedent score (sw)
Mention score (Aa)
Span
representation (g)
General Electric the Postal Service the company
(a) The coreference resolution model structure.
Sm (i) Wm FFNN (g:)
9, (1, WU FFNN, [9: 9j. ((,
(b) The formula for computing the coreference scores
Figure 9: The coreference resolution model (Lee et al., 2017).
We implement a coreference resolution model based on (Lee et al., 2017) to extract all antecedents for each entity (check Figure 9a). Compared to the previous works, we have the entities (phrase) an- notations. Hence, we ignore the phrase scoring modules and this annotation instead. We first collect all mentions of each predicted entity from spatial role extraction or role annotations, then as- sign an "id" to the same mentions and include that in each triplet. For example, for BELOW(a cat,
C.3 Logic-based Spatial Reasoner
To solely evaluate the performance of the logic- based reasoner, we use the "GT-PISTAQ". We look into the errors of this model and categorize them based on the source of errors. The categories are

---

missing/wrong ground truth direct annotations (A), rule-based Coreference Error (C) in connecting the extracted information before passing to the rea- soner, and the low coverage of spatial concepts in the reasoner (R). As is shown in Table 13, spatial Reasoner causes no errors for SPARTUN since the same reasoner has been used to generate it. How- ever, the reasoner does not cover spatial properties of entities (e.g., right edge in "touching right edge") in SPARTQA and causes wrong predictions in those cases.
E Large Language Models (LLMs)
Figure 11 presents examples showcasing predic- tions made by three Large Language Models (LLMs):
GPT4, and PaLM2, on a story from the SPARTQA-HUL -UMMAN dataset. These examples demonstrate that while these mod- els, specifically GPT4 and PaLM2, excel in multi- hop reasoning tasks, solving spatial question an- swering remains a challenging endeavor.
To evaluate the LLMs' performance spatial reasoning, we use Zero_shon Few_shof and Few_shot CoT.. In the Zero_sho setting, the prompt given as input to the model formatted as "Context: story. Question: question?" and the model returns the answer to the question.
D SREQA on All Story Relations
Datasets
F1 on SREQA
In the Few_shol setting, we add two random examples from the training data with story, all its questions, and their answers. Figure 12 depicts a prompt example for SPARTQA-HUMAN YN ques- tions, passed GPT3.5.
SPARTUN
96.37
SPARTQA-AUT
97.78
SPARTQA -UUMAN
23.79
MSPRL (Used RESQ)
16.59
For Fre_ssool+t CCT, we use the same idea as (Wei et al., 2022) and manually write the reason- ing steps for eight questions (from two random stories). The input then is formatted as "Context: story. Question: COT. Answer. Asked Context: story. Question: question?". Figure 13 shows example of these reasoning steps RESQ.
Table The result SREQA model only trained on all story relations of the SQA datasets.
Table 14 displays the results of the SREQA model trained and tested solely on all the story's s relation extraction parts (step l). During the evalu- ation, we also possess the same data preprocessing and gather annotations of all relations between sto- ries' entities and select the best model based on performance on the development set.
E.1 LLMs for Information Extraction
As discussed in Section 5.3.1, we utilize LLM, GPT3.5-Turbo, for information extraction from human-generated texts. The extraction process en- compasses Entity, Relation, Relation Type, and coreference extractions from the story, as well as entity and relation extraction from the question. Additionally, LLM is employed to identify men- tions of question entities within the text.
Notably, the performance on the human- generated datasets, SPARTQA-HUMAN and RESQ, is significantly lower compared to the auto- generated datasets. As discussed in the MSPRL datasets contain missed annotations, resulting in the omission of several relations from the stories' entities and incomplete training data for this phase. SrRL annota-
We construct multiple manually crafted prompt examples for each extraction task, depicted in Figure 14. Subsequently, the extracted information inputted into the reasoner module of PISTAQ to compute the answers.
Similarly, the SPARIQA-HUL tion, as discussed Appendix C, exhibits some noise, particularly in coreference annotation, lead- ing to similar issues as observed in MSPRL regard- ing annotation of all story relations.
In addition to our experiment, we attempted to incorporate LLMs as neural spatial reasoners but in pipeline structure of extraction and reasoning. To do so, illustrated in Figure 10, add the ex- tracted information of LLM with the written CoTs based on this extracted information the prompt of G-T3..5-aainee The results, however, become even lower (62.62%) compared to GPT3.5-CoT with the main text (67.05%) when evaluated on
Consequently, this reduced performance in all story relation extraction impacts the overall perfor- mance of the main SREQA model trained using steps; however, as illustrated in the results of SREQA* in Table7, which utilizes Spartun instead of MSPRL. for training on all story's relations, the performance substantially improves on the RESQ dataset.

---

Staryz-a a man a weoden chair and tilknng on the phone on the right a weeden bed with white bedcovers the left ( before the man a woodon desk and vase with flowees. there as black hetef-ean in front of the cheir and there is also a picture hanging the wall above bed.
Black jicker, grey pants
siiting
Relation in story: (C"a man". "in", "whsee sharf), ("a man" "In". "blzck jucket") man", "ia", "grey pante") ("A mun". "in", "black shoer"), ("A man", "siting
plane") wooden chai"") man", "uiking hodkowms" "M's "wit"", "belbre"
bed", foe", "the night""ss
). vise". "the left" M'' vase". "betore", "the man"M's MM's vaset, "witt", "Blowes" uf's black hrief-case", fin fiont of", "the chair" MM" picture", "hanging ou", "the J'"A picture". "above", "the bed"))
lett ("a
the man lonching the chair? (the mun, touching, the chair), Clain of reasonings ("A man". "sitting un", wooden chiir") which is externally connected and above. The inseeris Yes
Question:
Pith
picture),
Chain rensouling: wooden bed", "with, "white bedcovers") which the postion bedkover, picture". "sbove". J, (the picture above, buloover), (the bedowvrr below, the picturo). The answer is Yes.
We employ LLMs in both extraction and reasoning tasks, but in disentangled manner. Initially, we extract information using LLMs, and subsequently incorporate this extracted information into the prompt alongside written CoTs based on the extracted data.
Figure 10:
the RESQ dataset. This outcome highlights the superior ability of LLMs to capture information from natural language compared to structured data without fine-tuning.

---

There block called Inside there's red oval that is above blue triangle. The blue triangle is touching right mige of and the tup large black square. Touching the bottom edge of A small orange square. The is block called Which is the right and tuuching Inside orange one. The left udge of and the orange triangle is abeve the red triangle. The large yellow triangle is touching the top edge of B. The green triangle is toushiin the right edge
Story
there are triangles, red green
large yellow
brown one,
orange triangles
touching
SpaRT
brown triangla below all of other trianglas touching bottom
Question
What the relation between triangles orange triangles? Answer: left, near
Question 2:
the greann thing abave the yellow thing? Answer: No
Question
Does B have the triangles? Arewwrr: No
Dl: Don't Know
Zero shot
02: No, there is no green thing mentioned the description.
03: No
0P1SS (Prediction
triangle touching right edge of block and orange triangle touching the left edge of block Block B the right and touching block A so the orange triangle right of the blue triangle. The enswer is [Right].
T2e green triangle is to the right the large yellow triangle and the large yellow triangle is touching the top edge of B. Using inverse rule, the green triangle below yellow triangle. answer No.
DoT
03: There five triangles mentioned block red one, triangles are block so the answer Yes.
large yellow brown one. and an orange of these
Ol: Don't
description does not provide enough information to determine if the green triangle above yallow triangle. states that green triangle touching the right edge of block B and the lange yellow triangle touching the top edge of B. The relative vertical positions of green and yellow triangles
Zéro shot
specified.
(Pradiction)
03: Yes. has all of the triangles mentioned the description. Inside block B, there triangles: red green large yellow brown one, and on orange one.
OheThe triangle the orange trianqle. The answer is [Laft].
block A, and the orange triangle block Block to the touching block blue triangle left of
02: The green triançle touching the right edge of B, and the large yellow triangle touching the top edge of It not mentioned if the green triangle above or below the yellow triangle. The answer la Don't Know.
DoT
03: There is blue triangle in block A and there are 5 triangles in block B. Since there is triangle block A, not all triangles are in block B. The enswer
0t: The answer mear. The blue triangle and the orange triangle are both the block The blue triangle is above the orange triangle.
02: Yes. the green thing is above the yellow thing. The green thing a triangle that is touching the right edge of block B. and the yellow thing is triangle that touching the top edge block
Zero shot
03: Yes. has all of the triangles. The question as has all of the triangles in the block. The answer is yas because the passage says that
Pal (Precdiction
Inside there 5 triangles, red green large yellow brown and orange This means that has all the triangles in the block.
Ol: The blue triangle is block and the orange triangle block Block is of block B sp the blue triangle is of the orange triangle. [Left]
yellow 02: The green triangle touching the right edge of B and the yellow triangle touching the top edge of B. The right edge below the top adge So the green triangle below the yallow triangle. The answer No.
DoT
03: There are 5 triangles B. There is red triangle in B. There a green triangie in There is large yellow triangle in B. There is brown
triangla There an orange triangle The answer
Figure 11: example of three Large Language Models (LLMs), GPT3.5 (0)))))) GPT4, and PaLM2 prediction on SQA task evaluated with zero_ shot and CoT prompting.

---

There are three blocks called B, and C. A to the right B and is the right of In there large yellow square that touching the right edge of A. Near and to the left of the square is a small black circle. There is large black circle circle. There is a large black triangle is below the medium blue circle. In B, there is a medium yellow square is touching the left corner of small blue triangle.
Context:
and the right of the small black circle. There medium blue circle below the large black
there is amall black circle touching the bottom of There large yellow aquare near and above the small black circle. There is large black triangle far above the large yellow square. There small yellow triangle near and the left the large black triangle. Near and the left of the small yellow triangle is small blue triangle.
Question: the small black thing below large black thing Answer: Yes
Question: Is the small blue thing in to the right of medium blue thing? Answer: No
Question: the medium yellow thing below blue thing? Answer:
Question: Is the amall black in near and to the right of large yellow thing? Answer: Yes
Answer below questions:
Context: There block called A there red triangle Which above black circle. The black circle touching the bottom of A. A yellow square is touching the top edge of A and below that there is red oval is touching the right edge of A. There is second block one, and red one. The green square is above the red square and touching the left edge of B. The brown square is touching both the bottom edge the right edge of The square touching the edge of the brown square. Question: Is the oval to the right of the brown thing? Answer:
Which is the left of Inside there 3 squares,
green
Looking for ChatGPT?
Try now t
X
Figure 12:
Example of the input for (FF3.5(FFe_sshir) The shot setting is the just do not have the first training example.
Chain-of-Thoughts Reasoning Steps.
Consider Below examples:
There three blocks called and C. the of and B the right of A, there small blue right of medium black there large black cirele below yellow triangle near helow blue squre, In C; large yellow square near above medium square. for the left of the large yollow square.
for below black triangl: medium medium yellow square
Is the modium thing near below a large yellow thing?
The clain of ressaning to anserr this quesiion is: The large yellows square now and above medium biue square. Using the inverse nile, the medium blue and below the large yollow
The answer Yes.
small thing The chain of rensoning to answer this question is: he small blue cirole in B ahove the black triungle which above the medium yellow cinele So the small blue thing is above medium yellew thing. The answer is No.
yellow thing?
small thing The chain of reasoning to answer this question is: block right of block black is right of block C, so block A right of block The modiuum bine thing is block C, All abjects in blok A are right of objedt in black C, the small blue thing in right of the mediom blue thing in The Yes.
right
thing?
medium thing in The chain of rensoning to anser this question is: There is small blue thing block A and block B. It's not mentoond black H below or above block A noc. Alsa is not memtoned if medium black thing A above the small bise thing A. So the answer Don't Know(No).
thing?
Asswer below quections
Figure 13: Example of the input for GPT3.5(Fes (Few Bertseewssssttooeeeeesee with human written Chain-of of Thoughts.

---

Extraction
Prompt Example
Context there are two social workers in the foreground one wearing red pullover and brown pants is bending over to access the blue paint. the other one red tee-shirt and black jeuns is watching her at the back the room there another worker wearing white teo-shirt and blue jeans, acutally painting. there are many newspapers the ground protect the kindergarten floor all three workers wearing mask
Eatity
entities ['wwo social workers', 'the foreground', 'one', ' red pullover', 'brown pants', 'the blue paint, 'the other one', 'a red tec-shirt' 'black jeans', 'her', 'the back', 'the room', 'another worker', white toe-shirt', 'blue jeans', 'many newspapers', 'the ground', kindergarien floor, tthree workers', mask"]
a man in white shirt. black jacket, grey pants and black shoes sitting on a weeden chair and talking on the phone. relation triplets:
Relation
[C"a man", "in", "white shirt"), ("a man", "in", "black jacket"), ("a man", "in". "grey pants"). ("a man", "in", "black shoes"), ("A man". "sitting on", "a wooden chair"), ("a man", "talking on", "the phone")] on the right a wooden bod with white bodeovers. relation triplets: (("a wooden bed", "on", "the right"), ), ("a wooden bed", "with", "white bedcowers"))
If the relation set is:
Story
Relation Type
Relation type set:
LEFT left another object,
DC= Discomneteed, disconnected from other object,
(the wall, behind, the tourists), relation type is: ['BEHIND'] (Lots of locals, front of, a blue building) , relation type is: [FRRON]]
(pictures, on, the wall), relation type is: ('FRONT, 'EC" (a clock, above, the blackboard), relation type is[[ABOVE]
Context 1:
Three women sitting weoden bench in front of an about metre high red brick wall. they are all wearing skirts and jumpers...
If the list of all entities is:
Coreference
of noun plarases = ["throe women", "they", "two them", "a wooden bench", "an about one metre high red brick wall", "the wall", "skirts".... ...]
The below list shows which noun phrases the "list of noun phrases" refers which same phrase: ("Three women": ["Three women", "they", "two of them"). they: ["Three women", "they", "two of them"]. two them: ("two of them"),
wooden bench: ("a wooden bench".]. ...."........ .....
Relation and Type
If the relation set is:
Question
Are the lamps behind the building?: list, of dictionary [["triplet": ("the lamps", "behind", "the building") ) "relation type": [BEEINN]]]]]
Is the camera in front of the all kids?: list of dictionary - [["triplet": ("the camera", "in oro"t "the all kids"), "relation
type": ['FRONT']]
Is a flag the left of the stairs?: list of dictionary = [[ [["triplet:: ("a flag". "to the left of", "the stairs"), "relation type": [[EF]]]]]
Question entity to Story Meutions
This should consider the exact partially matching based on the phrase root. For examples." "0: "small window", 2: "large window". S: "three windows")" " all can be matched with "the window" since the root is window here. Also the output should be in the form only a python dictionary like ("the window": [0,2.5]].
Figure 14: The example of prompts used for LLMs (GPT3.5-Turbe) in information extraction.


Reviews:
[{"review_id": "7uLKGE3apI", "reviewer": "Reviewer_K1u1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable suggestions you've provided for improving our paper. We will certainly take these into account as we work on revising the paper.\n\nIt's important to note that the suggestions regarding section organization and formatting can be helpful for enhancing the paper's structure and readability. Still, we acknowledge that these aspects alone do not constitute grounds for rejection. \nRegarding soundness scoring, “ Some of the main claims/arguments are not sufficiently supported, there are major technical/methodological problems.”, we would greatly appreciate further clarification on which specific claims may not have been adequately supported or which technical/methodological aspects are of concern. This information would be tremendously helpful as we work towards addressing any potential shortcomings and ensuring the overall quality of the paper.\n\nPlease find below our responses addressing the points and questions you have raised.\n\n - Regarding \"dilemma on which model, PISTA-Q or SREQA, to use for any use- case\":\n\n   As we state in several places in the paper, such as in the introduction (lines 87 - 104) and conclusion(lines 560-577), our experiments show that PISTAQ is better than other models in controlled environments.  The complexity of these datasets is more related to conducting several reasoning steps and demands accurate, logical computations where a rule-based reasoner excels compared to black-box PLMs and results in higher performance. \n\n   However, on real-world data, these implemented logical rules has some limitation(not considering spatial commonsense), and also, the extraction module is weaker due to the fewer annotations, so using SREQA works better. The end-to-end structure of SREQA also helps to capture some spatial commonsense knowledge from the training data alongside using the benefit of the rule-based reasoner in the data preprocessing.\n\n   Compared to these, if we use LLMs for extraction, we can have more accurate extracted information from even real-world datasets which are more complex. In our case study (section 5.3), our model effectively answered all seven out of twenty-five sampled ResQA questions, which can be solved without relying on spatial commonsense information.\n\n- Reason to Reject1: (a,c,e,d) We prefer to have a short description of the model in the subsections of section 3 as the models' names are not described before. (b) The first time we mention the name of BERT-EQ is in subsection 3.2 when it is discussed. (f) Line 525 is part of the 5.3 section on LLM on spatial reasoning and is a title for the following paragraph (the “:” is missed after that). We can add another subsection there to clear that.\n\n- Reason to Reject2: Although your suggestion might be another way to list the models, we made a table to clarify the source of their supervision (not a glance of all models), as this is the most important feature. The rest of the information is available in the text when we describe the models. Also, as we described models in Subsection 4.2, the Majority baseline and GT-PistaQ doesn’t have any supervision, while as the caption state, Table 2 is a training supervision of baselines in this paper. \n\n- Reason to Reject3: Thank you for your suggestion. However, we chose the current structure to represent our results. The result section includes the report numbers and proper discussions and further information on the rationale behind each model’s performance. We respectfully disagree that the current section does not provide interesting discussions.\n\n- Reason to Reject4: All the acronyms are described when they are mentioned for the first time. Also, we consider most of them to be well-known acronyms in the NLP. Please let us know if there is a specific acronym that we did not describe, and we will ensure to include that given the extra page. Regarding the specified dataset name, we have only used the dataset name which was proposed in prior research and is not our own invention. \n\n- Typo 1: MSPRL is the backbone of the ReSQ dataset, and we described it in lines 329-337.\n\n- Typo 3: We use Figure in the text, but in the parentheses, we use Fig as a short form.\n\n- Typo 4.a: We don’t have the coreference resolution in question as we describe it in the text (lines 217-221). To connect question entities to the stories, we use Rule-Coref, which we also have in Figure 2. We make this clear in the text.\n\n- Typo 4.b: All of the computation of spatial reasoner happens inside the prolog considering all the implemented rules shown in Table 12. Overall, Prolog would get the question as a query and the relations as facts; it will return an answer to our query directly. We will ensure to extend the information related to the spatial reasoner in text and within the figure, enabling a more coherent visual representation of its processes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis paper demonstrates the effectiveness of separating the process of information extraction from reasoning while performing spatial question answering.\n2.\tAlthough the quantitative and qualitative are strong against the baseline models, yet, they may lead a reader to a dilemma on which model, PISTA-Q or SREQA, to use for any use- case.", "reasons_to_reject": "1.\tThe structure of the paper is not good. For instance, \n      a.\tText below section 3 should briefly introduce all the 3 models (with names of models) and should refer to subsections 3.1, 3.2 and 3.3 to provide the reader an idea of what to expect.\n      b.\tNames of the models should be easy to remember. For example, what is the purpose of EQ in BERT-EQ. As a reader, I don’t understand it until I read section 3.2.\n      c.\tHeading of Section 3.1 should be PISTAQ: ….. to make it evident that the section talks about first model.\n      d.\tHeading of Section 3.2 should be BERT-EQ: ….. to make it evident that the section talks about second model.\n      e.\tHeading of Section 3.3 should be SREQA: ….. to make it evident that the section talks about second model.\n      f.\tIs the section starting at Line 525 a subsection or something different?\n\n2.\tTable 2 does not help in dissecting the attributes of proposed models among each other and with the baselines. The authors are requested to make it easy for the readers to get an understanding of the various models. For instance, a suggestion is to make a table where every row is a model name and columns are 4 or 5 attributes which the authors feel critical to distinguish the various models. This will help in the ablation analysis aswell. Moreover, current Table 2 does not contain baselines: Majority Baseline and GT-PISTAQ. The purpose of this table is to give readers a quick glance about all the models discussed in the paper with their attributes.\n\n3.\tSections discussing results simply report the results without any interesting insights. My suggestion will be to discuss the results with respect to the new Table 2 mentioned in point 2. This will keep readers interested in reading the results with some insights about attributes of the proposed models.\n\n4.\tThe paper uses too many acronyms, and some acronyms have an unusual mention. For eg. SPARTUN where size of “S” is smaller than “RTUN” but bigger than “PA”.\n\n5.\tThe authors are requested to create separate sections for quantitative results which can have Table 3-6 at one place and a qualitative section which can have Figures 4 and 5.", "questions_to_authors": "", "paper_topic_and_main_contributions": "To show benefits of segregating the processes of information extraction and reasoning in the task of spatial question answering, the paper proposes 3 models: PistaQ, BERT-EQ and SREQA. In brief, PistaQ is a model based on extracting relations and then perform symbolic reasoning. BERT-EQ is an end-to-end pre-trained language model that uses the same spatial information supervision but in question-answer format. Lastly, SREQA model is again an end-to-end neural model with explicit layers of extraction and reasoning. Various experiments are conducted over 3 datasets: SPARTQA, SPARTUN and RESQ, which show the efficacy of separating the processes information extraction and reasoning.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1.\tIn Table 3, dataset MSPRL is mentioned. However, I don’t see MSPRL described in section 4.1\n\n2.\tLine140: Reasoning -> reasoning.\n\n3.\tInconsistent usage of reference to artifacts. For instance, “Figure 1” in Line 058 and “Fig 1” in Line 156.\n\n4.\tFigure 2 and the text in section 3.1 should match to make the reader understand the model easily. For instance:\n        a.\tIn Figure 2, why is “Coref Resolution” missing in Question Processing?\n        b.\tWhat process does Spatial Reasoner follow to generate the answer? Currently, its not clear either from the Figure 2 or paragraph in Line222-229."}}, {"review_id": "9GwC7fonMJ", "reviewer": "Reviewer_WRQX", "summary": "", "questions": "Refer to the above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable points and comments, which will significantly enhance the paper's readability and clarity. Please find below our responses addressing the specific points and questions you have raised.\n\n- Reason to reject 1: Thank you for raising this concern. We have indeed incorporated running examples in Figures 2 and 3 with the main models. However, considering the extra page in the camera-ready version, we will further elaborate on these examples in the context of the paper to enhance clarity, especially for the SREQA model. \n\n- Reason to reject 2-a (Logical rules): The logic rules employed for reasoning are consistent with those outlined in the referenced paper (line 223), and these rules are also documented in Table 12. We will ensure to clear this in the text with reference to Table 12.\n\n- Reason to reject 2-b (BERT-EQ): Indeed. In lines 74- 75 and 234, we mention extra questions, meaning extra than the primary question. This also is illustrated in Table 2 with QA(main questions)+SPRL(augmented questions). We also used similar naming with extra EQ in the end for the same reason. We will further address this in section 3.2 to enhance clarity.\n\n- Reason to reject 2-c (SREQA): In response to your question about entity selection and final answer production in SREQA, please note that Figure 3.b and lines 286-287 show that the entity selection data comes directly from the annotation in the first training phase. In the next phase, we employ the same trained extraction models as utilized in PistaQ(lines 296-297). We make sure to clear this in the text. The final answers come from the output of each relation classifier, which identifies the relations between entity pairs. These details are more elaborated in lines 301-306 of the paper.\n\n- Reason to reject 3: We do not have the empirical result on the whole ReSQ datasets since structuring the extracted information needed engineering and was time-consuming. So we only showed some case-studies of this idea in the paper in subsection 5.3. However, we have the result on a subset of ReSQ. Of 25 sampled questions only 7 of them can be solved without relying on spatial commonsense information (more details lines 451-459). The PISTAQ model using LLM as extraction modules can solve all 7 questions. We will add this result in the next revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper raises an interesting and valuable problem of whether disentangling the extraction and reasoning process leads to improved spatial reasoning.\n2. Comprehensive model designs and comparisons are given to ablate specific functionalities with empirical results, including direct black-box inference, neural disentanglement, and symbolic disentanglement.\n3. The results indicate the potential of using specific reasoning models to replace LLMs in reasoning-intensive tasks.", "reasons_to_reject": "1. The descriptions are intensive, but lack clearer organizations which makes it a bit challenging to follow. A running example could be added to make description much easier. \n2. The settings of different proposed models are not so clear, as well as some of the illustrations. For example:\n- What are logic rules used in the prolog for the reasoning part? \n- Is BERT-EQ essentially similar to BERT but with data augmentations using SPRL labels? \n- For SREQA, how is entity selection  (via BIO, similar to PISTAQ?) and the final answer produced given the predicted pairs?\n3. The proposed models still require SPRL labels for training the extraction module. The authors mention that it is beneficial to use LLMs to produce zero-shot or few-shot extractions, but is there any empirical results for the final performance?", "questions_to_authors": "Refer to the above.", "paper_topic_and_main_contributions": "This paper proposes to disentangle the information extraction process with reasoning process for multihop spatial reasoning dataset, and design both symbolic pipeline disentangling and end-to-end neural disentangling to achieve the objective. To showcase the effect of the disentanglement, the authors compare with PLMs such as BERT and GPT-3 which produce direct inference given the story and question.\n\nExtensive experiments are conducted over different datasets, automatic or human-authored, with or without SPRL labels, to empirically verify the effect of different components, and analyze the models' generalization towards unseen datasets during training.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5oKNfku6S9", "reviewer": "Reviewer_6kdH", "summary": "", "questions": "1)  PISTAQ is better than SREQA on SPART datasets, However, it performs worse than SREQA on RESQ. Why?\n\n2) The performance of entity coreference is still poor and most models are less than 70 in F1-score. Why can the accuracy of the proposed Coref module acheive 99? How about the Precision and Recall?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback and recognizing and acknowledging the contributions of our paper. We believe that your comments and questions can effectively improve the quality and understanding of our paper. \n\nPlease find below our responses addressing the points and questions you have raised.\n\n- Reason to reject 1: The main idea of this paper is to propose the different ways of disentangling extraction and rule-based reasoning as a pipeline and an end-to-end solution. These models are proposed for the multi-hop reasoning task, which is more challenging than a simple IE task, which extracts direct information and needs complex logical computation.\nIn the related work section 2, we mentioned the previous research in this area and their differences. Several end-to-end models for SQA tasks exist, and PLMs are the SOTA in this area. Even compared to more complex models (lines 108-117). As a result, we select PLMs as the primary end-to-end baselines to compare our model results.  \nOn the other hand, the previous pipeline models, with some separate extraction and reasoning, mainly use logical rules to keep the consistency of the extraction prediction (lines 132-135). Also, other works use simple summation over the computed probability of extraction modules instead of logical rules (lines 135-138) for the reasoning module. So, using the implemented crisp logic in the reasoning part to solve the multi-hop reasoning in both forms that we have, PistaQ as a separate module or as in SREQA in data preprocessing was not discussed before.\n\n- Reason to reject 2: Just to confirm again, we do not propose methods for IE but for multi-hop question-answering task. As we state in several places in the paper, such as in the introduction (lines 87 - 104) and conclusion(lines 560-577), our experiments show that PISTAQ (pipeline) is better than other models in controlled environments.  The complexity of these datasets is more related to conducting several reasoning steps and demands accurate, logical computations where a rule-based reasoner excels and results in high performance. However, in real-world data, these implemented logical rules have some limitations. The extraction module is weaker due to the fewer annotations, so using SREQA (End-to-End modeling + logical pre-processing) works better. The end-to-end structure of SREQA also helps to capture some spatial commonsense knowledge from the training data alongside utilizing the benefit of the rule-based reasoner in the data preprocessing. Compared to these, if we use LLMs for extraction in PistaQ, we can have more accurate extracted information from even real-world datasets, which are more complex. Of 25 sampled questions from RESQ, only 7 can be solved without relying on spatial commonsense information (lines 451-459). The PISTAQ model using LLM as extraction modules can solve all 7 questions.\n\n   We have conducted comparisons against recent SOTA, especially LLMs, on specific datasets relevant to our research scope. The current LLMs surpass all previous unified, joint, and multitask SOTAs[1,2,3] and as shown in [4], using SPARTUN as extra supervision in BERT already surpassed the models with more complex structures. We have some related research in section 2, which shows how our model captures some ideas from prior work or how it differs from theirs. If you are aware of any relevant work that could serve as additional baselines for our study, we would greatly appreciate your insights to help enhance the comprehensiveness of our analysis.\n\n\n- Reason to reject 3: The baselines we compared our proposal against are the state-of-the-art performing models published recently in top NLP venues. Consider that this is a relatively new task and does not have many baselines. Besides, the current LLMs, specifically LLMs with Chain-of-out, already surpassed other models[1], specifically in the reasoning area. Therefore, the baselines are strong, and the SOTA on the task.\n\n- Reason to reject 4 and Question 1: Thank you for the good question. We make sure to include this discussion in the paper. The SREQA performance does not surpass PISTAQ’s in a controlled environment. The complexity of these datasets is more related to conducting several reasoning steps and demands accurate, logical computations areas where a rule-based reasoner excels. So, in a controlled environment, the result of PistaQ with a reasoner module is higher than any other model. The reason for the lower performance of  PISTAQ on ReSQ is discussed in lines 451-464 and the better result of SREQA on ReSQ is discussed in lines 483-489.  \n\n\n- Reason to reject 5: Throughout our work, we emphasize our motivation, which centers on evaluating the efficacy of disentangling extraction and reasoning within the context of multi-hop spatial reasoning. Our motivation stems from the limitations of existing black-box PLMs in lines 35-45 (the performance of PLMs is not consistent and mostly captures the pattern[5]), the benefits of using extracted information (lines 46-53), and human performance(line 54).\n\n   Many works, including the cited paper in our work and our experiments, show that the black-box NN models, even the powerful LLMs, have limitations in multi-step reasoning. Hence, in our disentangled model, we utilize NN models for extraction instead of the whole reasoning process. Subsequently, we perform rule-based reasoning over them to prevent erroneous reasoning computation. As a result, our disentangled model performs better than joint models.\n\n- Question 2:  Firstly, the coreference challenges in the synthetic datasets (Spartun and SpartQA-Auto) are not as challenging as datasets on the Coreference resolution task. Also, these datasets contain large training sets, which helps the learning process to effectively capture the coreferences within this context. Secondly, to consider the plural coreference, we evaluate the models based on the accuracy of correctly predicting coreference pairs (as detailed in Appendix C2, Line 911). Moreover, ignoring the plural coreferences, we additionally evaluated the model by the B3 score, which still indicated the high performance of ~99% B3 on synthetic benchmarks. As an example, below, you can see the result of our model ignoring plural coreference on SPARTUN:\nTest Precision B3:  0.99005\nTest Recall B3:  0.99217\nTest Macro-F1 B3:  0.99107\nTest accuracy:  0.98741\n\n\nReferences:\n\n[1]Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. \"Chain-of-thought prompting elicits reasoning in large language models.\" Advances in Neural Information Processing Systems 35 (2022): 24824-24837.\n\n[2]Ozturkler, Batu, Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. \"ThinkSum: Probabilistic reasoning over sets using large language models.\" arXiv preprint arXiv:2210.01293 (2022).\n\n[3]Bang, Yejin, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia et al. \"A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.\" arXiv preprint arXiv:2302.04023 (2023).\n\n[4]Roshanak Mirzaee and Parisa Kordjamshidi. 2022. Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6148–6165, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n\n[5]Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. 2021. SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4582–4598, Online. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This paper proposed three different model on spatial relation extraction.\n\n2) The proposed models achieve good performance on multiple datasets.\n\n3) This paper is well-written.", "reasons_to_reject": "1) I mainly concern its novelty. The three proposed models are not new, because pipeline models and QA-style models are widely used in IE field.\n\n2) Which model is the best for a IE task? Pipeline model? Unified model? Joint model? or multitask model? I think this paper should compare   the proposed models with other framework, e.g., joint or multitask framework.\n\n3) The baselines are weak. This paper only compared the proposed models with some basic models (e.g., Majority, BERT,GPT3) and did not compare them with the SOTA models.\n\n4) PISTAQ is better than SREQA on SPART datasets, However, it performs worse than SREQA on RESQ. I would like to see the reason.\n\n5) The motivation is not clear. This paper did not answer the question: Why is disentangling extraction and reasoning better than joint  extraction and reasoning in multi-hop spatial reasoning?", "questions_to_authors": "1)  PISTAQ is better than SREQA on SPART datasets, However, it performs worse than SREQA on RESQ. Why?\n\n2) The performance of entity coreference is still poor and most models are less than 70 in F1-score. Why can the accuracy of the proposed Coref module acheive 99? How about the Precision and Recall?", "paper_topic_and_main_contributions": "This paper focuses on spatial role extraction and spatial relation extraction. Its main contribution is to disentange the processes of spatial relation extraction to different stages. In details, it proposes three different models: a pipeline of extraction and symbolic reasoning, an end-to-end PLM in a QA format, and  an end-to-end neural model with explicit layers of extraction and reasoning. The experimental results on multiple datasets show the effectiveness of the proposed models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an approach for spatial reasoning over text to tackle Spatial Question Answering. The proposed solution disentangles the processes of information extraction and reasoning, and this is demonstrated to be effective on multiple Spatial QA datasets. 
The paper has some merits, especially the experimental section which is very exhaustive. Still the paper can be largely improved, in particular it can be better organized and made more clear. 

Furthermore, the authors proposed three different solutions (a pipeline of linguistic modules, end two end-to-end systems based on pre-trained Language Models) which are tested only on 3 different datasets resulting in no clear best system (depending on the dataset the best solution changes). This is very problematic since it is hard to draw conclusions. What model should a practitioner use? I recommend the authors to try to take the best from each proposed solution and verify whether they can come up with individual model that works well in every setting.
Decision: Accept-Findings