Extracting and Following Paths for Robust Relational Reasoning with Large Language Models
Authors: Ge Zhang
Mohammad Ali Alomrani
Hongjian Gu
Jiaming Zhou
Yaochen Hu
Bin Wang
Qun Liu
Mark Coates
Yingxue Zhang
Jianye HAO
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) possess vast semantic knowledge but often struggle with complex reasoning tasks, particularly in relational reasoning problems such as kinship or spatial reasoning. In this paper, we present Path-of-Thoughts (PoT), a novel framework for solving relation reasoning that decomposes the task into three key stages: graph extraction, path identification, and reasoning. Unlike previous approaches, PoT efficiently extracts a reasoning graph that identifies crucial entities, relations, and attributes within the context. Subsequently, PoT identifies query-relevant reasoning paths within the graph, facilitating downstream reasoning of potential answers. Experimental evaluations across four datasets of relational reasoning demonstrate that PoT surpasses state-of-the-art baselines by a significant margin (up to 21.3%) without requiring fine-tuning or extensive LLM calls. Furthermore, unlike prior neuro-symbolic methods, PoT exhibits improved resilience against LLM extraction errors and input ambiguity by leveraging the compositional nature of graphs.

Published in Transactions on Machine Learning Research (01/2026)
Extracting and Following Paths for Robust Relational Rea- soning with Large Language Models
Ge Zhang Huowei Technologies Canada
ge.shangl @huowei. com
Mohammad Ali Alomrani
mohammad ali, clomrrani@houuues.cco
Huawei Technologies Canada
Hongjian Gu Huawei Technologies Canada
hongjian.gu gu com
Jiaming Zhou Huawei Technologies Canada
jiaming.zhoou@huuuvii.ccom
Yaochen Hu
jaochen.hrrShuawei.coo
Huowei Technologies Canada
Bin Wang Huawei Technologies
wangbin1158@hhaawvcccomm
Qun Liu Huowes Technologies
gun.in@@haawei.com
Mark Coates
mark. mark.coutess@magil..c
McGill University
Mila Québec Al Institute
Yingxue Zhang Huowei Technologies Canada
singmue.hhngghhhauei.coo
Jianye Hao
hauo Chwawoii..aooooooomooo com
Huowei Technologies
Reviewed on OpenReview: https: openrevvee... net/ forum? id= EoKL
Abstract
Large language models (L.LMs) possess vast semantic knowledge but often struggle with complex reasoning tasks, particularly in relational reasoning problems such as kinship or spatial reasoning. In this paper, we present Path-of-Thoughtt (PoT), a novel framework for solving relation reasoning that decomposes the task inlo three key stages: graph ex- traction, path identification, and reasoning. Unlike previous approaches, PoT efficiently extracts reasoning graph that identifies crucial entities, relations, and attributes within the context. Subsequently, PoT identifies query-relevant reasoning paths within the graph, facilitating downstream reasoning of potential answers. Experimental evaluations across four datasets of relational reasoning demonstrate that PoT surpasses state-of-the-art base- lines by a significant margin (up to 21.3%) without requiring fine-tuning or extensive LLM calls. Furthermore, unlike prior neuro-symbolic methods, PoT exhibits improved resilience against LIM extraction errors and input ambiguity by leveraging the compositional nature of graphs.

---

Published in Transactions on Machine Learning Research (01/2026)
1 Introduction
Large language models (LLMs) have shown remarkable generalization abilities in natural language (NL) lasks (Wei el al. 2022a Kojima et al.j 2022). such as generaling useful code (Chen el al. 2021) and fluently engaging in dialogue (Thoppilan et al. 2022j. Their success can be attributed to pre-training on large
human language datascts, which express real-world concepts, and thereby allow LLMs to implicitly learn about the entities and relations that exist in the physical world (Patel & Pavlick 2022]. Nonetheless,
some argue that the underlying meaning of language cannot be learned from text alone without appropriate grounding to the (non-text) real-world experiences (Bisk al. 2020 Cohn & Blackwell 2024). Prior studies (Tolman 1948; Whittington et al., 2022 Garvert et al. 2017) have shown that humans can create "cognitive maps" while navigating and experiencing their environments. Cognitive maps represent the latent relational structure of a /esvirinirnn and are particularly helpful for multi-hop relational reasoning tasks
such as planning/ /naiigaiiinnnnn (aaaada et al. 2024 Momennejad et al. 2023
While LLMs do exhibit some competence basic planning tasks (Momennejad et al. 2023 Valmeekam
ct al. 2023). they are known to perform shallow reasoning and suffer multi-hop relational reasoning tasks (e.g-, kinship inference (Sinha et al.) 2019), or spatial reasoning (Shi et. al.) 2022)) ). In contrast, symbolic solvers (e.g., Answer Set Programs (ASP) (Lifschitz
2008) can faithfully perform reasoning using well- defined symbolic rules written by domain experts. Consequently, there has been surge of neuro-symbolic
works (Yang et al.) 2023 Mirzaee & Kordjamshidi 2023 Silver et al. 2024 Pan et al. 2023 which combine
the rich LLM natural language abilities with interpretable symbolic solver modules. These works typically leverage LLMs to transfer any natural language (NL) based problem formulation to the appropriate symbolic language. This is then exccuted by the solver, honce maintaining the ficxibility of LLMs while transferring the burden of complex reasoning to the symbolic reasoning module. This disentanglement of language understanding and reasoning displays significant performance improvements over prompt-based baselines (e.g,, Chain-of-thought (CoT) (Wei et al.] (022b)). This idea of task decomposition also demonstrates the benefits in other research domains (Wang et al. 2023a). Nonetheless, prior works suffer from several shortcomings, such as task-specific and highly specialized translation and reasoning modules, brittleness to LLM errors, or requiring many LLM calls.
In this work, we introduce a novel framework, Path-of- Thohhghssshhuhss (PoT), that decomposes a relational reason- ing problem into three stages: graph extraction, path identification, and reasoning. During the first stage, single LLM call extracts the key entities, relations, and their corresponding attributes in the problem, constructing graph (akin to a cognitive map), which serves reasoning tasks (e.g., kinship or spatial). Subsequently, the path identification module identifies the key reasoning paths in the graph that are associated with the question. Finally, an LLM or symbolic reasoner is used to infer probable answers based on the input and identified paths. Our evaluations across several well-established relational reasoning datasets indicate a maximum improvement of 21.3% in accuracy and superior robustness to LLM extraction errors and potential controversial descriptions. To the best of our knowledge, PoT is the first work that enhances relational reasoning tasks through identifying query-relevant subgraphs before reasoning. Our contributions can be summarized as follows:
foundation for downstream relational
We present a prompting-base based approach to efficiently extract graphs and queries in a single LLM call.
We propose a path identification stage that can identify multiple independent reasoning paths in- volving the queried entities to infer all possible answers.
We benchmark on several kinship and spatial reasoning datasets to validate the performance gain of proposed framework against basclines.
2 Problem Definition
In relational reasoning, a sample (s, a) consists of a textual story s and a target relation a E R, where R is the overall set of pre-defined relations. A story consists of context e and question q. The context
2

---

Published in Transactions on Machine Learning Research (01/2026)
An orange rectangle O is inside and touching box named D. Another box named E with a midsize green rectangle G behind box D. This box covers midsize white thing W which to the south the green thing. Where is the green thing regarding the orange thing?
(front, behind}
Graph Extraction
Path Identification
Reasoning
Triplets:
(0, covered by, box D)
LLM Reasoner
(G, behind, box D)
below
behind
(box D, behind, box E)
(box E, cover, W]
W
D
(W, below,
OR
cover
Query:
covered by
(G,?,0)
E
Symbolic Reasoner
O
behind
Graph
below
W
behind
cover
behind
covered_
Domain Knowledge
covered by
The PoT framework with example featuring robustness against LLM extraction errors high- lighted in red (flipped relation). The LLM is employed to extract the graph representing the story's relational structure. Path identification isolates the reasoning paths relevant to the query entities. PoT reasons over each path independently to alleviate cascading errors due to extract ion and infers all possible answers.
Figure 1:
describes relations among set of entities N in natural language (e.g., A is the son of B), where the relation set mentioned context с is a subset of R. The question asks about the relation between 2 entities, ne ny N in natural language (e.g., How should address C?), whose relation not directly provided the context c. Note that many prior works (Yang al.) 2023 Mirzaee & Kordjamshidi (023) assumee story only consists of context. e without question q, and queried entities n, and n2 are provided by the dataset instead of being extracted from the question q. We target not annotate 2 queried entities and must extract them from the question for the story s.
general setting in which the dataset does
For some datasels, there could be multiple possible answers due to ambiguities or errors in the story (See Appendix A.& for examples). Therefore, a method is allowed to output multiple possible relations.
To complete the task, the algorithm must understand how relations combine (c.g., A to the west of B and is north of A, then C is north-west. of B). In the problems address, assume that these compositions either common-sense (and thus implicitly encoded in an LLM) that domain-specific rule set is provided, specified as logical rules a set of examples.
3 Related Work
Multi-hop Relational Reasoning: Before the advent of LLMs, several neural network architectures were proposed to solve the relational reasoning problem. The introduction of benchmark datasets often accompanies these. Shi et al. (2022) introduces the StepGame dataset, which tests for multi-hop spatial reasoning. That is, given a story describing the spatial relations (on top of, down, right, etc.) between entities, the task is to infer the implicit relation between two entities in the story. The authors introduce the Tensor-Proddie based Memory-Augment networks (Schlag et al.] (021) and specialized for spatial reasoning tasks. [alm al. (2018) design a relational recurrent network, which treats the input relational problem as fully connected graph with nodes representing the facts. Message passing iteratively performed before the answer is predicted. Recent
Neural Network (TP-MANN), which is based on memory
3

---

Published in Transactions on Machine Learning Research (01/2026)
methods Mirzaee et al. 2021 Mirzace & Kordjamshidi 2022 2023 fine-tune pre-trained language models (PLMs) (e.g.; BERT (Devlin et al.; 2019)) to extract richer textual features and cast the problem into a sequence classification task. Waan et al. (2023b) provides another synthesized dataset called SPARTUN for testing spatial reasoning problems. Compared to the StepGame datasct, it includes a larger variety of spatial relation types and expressions. The authors fine-tune a PL.M-based model with a classification layer on top of to predict the final relation between two queried entities. [Sinha et al. (2019) introduces the CLUTRR dataset to benchmark the kinship reasoning abilities of NLP models. Experimental results show that large gap exists between PLMs that reason directly on the textual imput and graph neural network models (Veličković et al. 2018) that work directly on the underlying symbolic graph manifested by the story.
Prompting-based Reasoning Methods: With the emergence of powerful LLMs like GPT-4 (Achiam et al.) 2024) and GPT-4o, many approaches leverage the natural language understanding and reasoning capabilities of these models to tackle multi-hop relational reasoning problems. Wei et al. (2022b) intro- duces Chain-of- Thhought (CoT) prompting, which instructs LLMs to reason step-by-step before arriving at conclusions. A follow-up work, Chain--ffThoought with Self Consistency (CoT-SC) (Wang et al. 2023b).
seeks to improve CoT by performing multiple independent reasoning iterations followed by majority vote. Subsequent frameworks, such as Tree-of- Thoouhhs (ToT) Yao et al.) 2024) and Graphhof-Thoughts (GoT) (Besta et al. 2024), further enhance LLMs' reasoning capabilities on specific downstream tasks by utilizing sophisticated search strategies and task-specific heuristics (e.g., scoring functions). However, these tailored search strategies limit their adaptability to other downstream tasks (e.g., relational reasoning). To address these challenges, Zhou et al. (2024) proposed using graph- based synthetic data to fine-tune LLMs, along with Extraact-then-Answe prompting strategy. This approach showed improved performance in inductive and spatial reasoning tasks by leveraging structured reasoning representations. Additionally, Hu et al. (2024) proposed Chain-of-Symbol (CoS) prompting to address spatial reasoning problems by presenting LLMs with in-context examples that include stories and corresponding symbolic chains. However, CoS still relies on LLMs to not only translate natural language into symbolic notations but also to implicitly construct rea- soning paths, which makes it vuluerable to interfering or disordered relations within the input relational reasoning problem.
The interprotability (Singh al.) 2024 and hallucination (Huang et al. 2025) issues of LLMs have led many works to complement them with symbolic modules (Pan et al, 2021 Wong et al. 2023b Yu et al.] 2023). Such neuro-symbolic
Extraction and Symbolic Reasoning:
2023 Olausson et al. 2023 Nye et al.
systems have been successfully applied to visual question answering (Ding et al. 2021 and robot planning
(Wong et al. 2023b Silver et al., 2024 Yang et al., 2023 DSR-LM (Zhang et al., 2023) presents a differ- entiable symbolic reasoning [ramework that uses pre-trained language models for fact extraction alongside a differentiable symbolic module for deductive reasoning using learned rules. The method displays good per- formance on kinship reasoning but requires significant finetuning and can fail due to fact extraction errors. LINC OOlussoo et al.) 2023) introduces framework for first-order logic (FOL) reasoning that employs LLMs as semantic parsers to translate natural language premises and conclusions into first-order logic expressions. Subsequently, external theorem provers are used for deductive inference. This approach leads to significant performance improvements over pure prompting-bese based methods. However, it is limited to first-order logic problems that are expressed in relatively short statements, which makes the semantic parsing task tractable. Logic-LM (Pan et al. (2023) also employs LLMs for semantic parsing but tackles more logic-oriented prob- lems, such as logic programming and constraint satisfaction. LLM-ASP (Yang et al., 2023) uses answer set programs [Lifschitz 2008) as generic symbolic solvers, resulting in versatile system capable of achieving state-of-the-are performance across various problems. Our framework, PoT, embraces the neuro-symbolic paradigm, but has important, distinct features. Rather than converting the input problem into task-specific symbolic language, we opt to extract the fundamental entities and relations, constructing a versatile graph that can be utilized by a variety of downstream reasoners (e.g., LLM, symbolic solver, etc). Unlike other complex symbolic formats, graphs offer support for compositional and easily interpretable reasoning, making them particularly suitable for tasks rooted in relationships, such as spatial reasoning. Moreover, we refrain the input text problem. Rather, we efficiently extract all
from imposing any specific format assumptions relations and queries simultaneously within a single LLM call. Lastly, while traditional symbolic solvers may fail if given contradictory facts, the inherent compositional nature of graphs in PoT enables us to mitigate the impact of conflicting information due to LLM extraction errors or ambiguities in the input problem.
4

---

Published in Transactions on Machine Learning Research (01/2026)
By exploring multiple reasoning paths between queried entities, approach offers resilience against such challenges (See Figure 1 for an example). To the best of our knowledge, our approach is the first that directly mitigates the effect of LLM extraction errors on the reasoning module.
4 Methodology
The proposed framework, PoT, consists of 3 modules: graph extraction, path identification, and reasoning. The graph extraction module extracts all mentioned entities and relations with corresponding attributes from the input story with LLMs, and later converts them into graph. Subsequently, the path identification module identifies all reasoning paths between the two queried entities on the graph. Lastly, the reasoning module infers the answer given for each reasoning path independently.
Figure I shows the overall diagram of the proposed framework. Section elaborates on how to prompt LLMs to extract the graph effectively. Scction 4.2 describes the process of finding the relevant reasoning paths between the queried entities on the graph. Section 1.3 introduces how we employ either an LLM or a symbolic solver to infer the final answers given the reasoning paths.
4.1 Graph Extraction
Given a textual input story s, the objective of the graph extraction module is to convert the context of the story s into a graph 9 = (WE), where the node set N = (m1, ni,..... "......,,k...................... mk..... represents entities the context, with their associated attributes, and the edge set & {e1, eg...., ,k.... contains triplets represented (Mhead; F; Ptaa), where head node Theead and tail node Nsai represent entities, and denotes the relation from Rhead to Attil Note the relation belongs to the pre-defined relation set R. For example, in the kinship domain, node includes attributes like 'identity' and 'gender', representing person's 's name and gender, respectively. This section details our approach to constructing effective prompts for graph extraction using large language models (LLMs).
Despite the effort of few-shot prompting (Brown et al. 2020). a significant challenge in graph extraction lies in the potential for the LLM to misinterpret the lextual input, leading to missing nodes incorrect relations. These inaccuracies can compromise the reliability of the graph G, ultimately affecting the reasoning tasks that depend on it.
To address these challenges, we designed prompts that explicitly guide the LLM toward accurate relation identification and triplet extraction. Our approach builds on principles of structured guidance and decom- position, adapting strategies from prior works while introducing specific enhancements tailored to the graph extraction task. Key components of our methodology include: (i) Sectional markup for logical structure, (ii) Syntactic delimiters for output consistency, (iii) Predefined categories for standardized outputs, and (iv) A decomposed approach to task simplification. Examples of the prompts developed for our experiments are detailed in the Appendix A.5
Structured Prompts with Sectional Markup. Inspired by previous work (Zhng et al. 2022 that
organizes prompts into logically segmented sections to improve interpretability, we structure our prompts with distinct sections marked by special characters (for example, #). This logical organization provides the LLMs with a clear and navigable framework, reducing ambiguity during task processing.
Structured Output with Syntactic Delimiters. Inspired by the method proposed by Zhong et al. (2022) that uses logical segmentation for clarity, we systematically organize prompts into distinct sections, marked with special characters (e.g., #). This structure allows the LLM to navigate the task more effectively and minimizes ambiguity in interpreting the input.
Predefined Output Categorics. Following principles of consistent formatting, we use syntactic markers such as brackets or parentheses to enforee a standardized output format. This approach ensures precision in the extracted data, reducing the likelihood of parsing errors during downstream processing.
5

---

Published in Transactions on Machine Learning Research (01/2026)
Decomposition of the Extraction Task. Similar to the prompt ideas outlined by Li et al. (2023) and Wu et al. (202),, we decompose the graph extraction process into smaller subtasks. For example, the prompt separates the generation of relational triplets from the queries identifying the two nodes. This explicit task dccomposition reduces the cognitive load on the LLM, enabling it to focus on individual subtasks and improving overall performance.
By integrating these strategies, we tailored the LLM prompts balance clarity, consistency, and task-specific adaptability, enabling effective graph extraction across diverse domains.
The output of the LLM is parsed into in set of triplets & which is the edge set of the graph g. The queried entities from the question are also extracted as nodes on graph G, represented as Mrre and Reary respectively.
4.2 Path Identification
The path identification module is responsible for identifying all reasoning paths on the graph g. A reasoning path p is a sequence of edges on that connects the query nodes nsrc and ntar- Specifically, p = [ex....;e]l where, esy ej E E. Note that the direction of the edge e, between any adjacent nodes T2 E N can be either forward as e; (Na, n&) backward as 6 (nu: n4), depending on which of them exists in edge set E. We apply depth-first search (Sedgewick 2001 to identify such reasoning paths the graph G between the given queried entities.
Intuitively, a single reasoning path is all that is needed to infer the implicit relation between nsre and Btar (See Figure [ for example). However, there could be multiple possible reasoning paths from nare to near In cases where there are LLM extraction crrors or ambiguitics in the story, each reasoning path can infer a different possible answer (i.e., relation).
4.3 Reasoning
For each reasoning path p, call an external reasoner (e.g., symbolic solver) to obtain the target relation E R. The choice of reasoner depends on whether domain-specific rules (e.g., logic rules) are available and other user considerations (e.g., speed, robustness, optimality). In this work, we explore both LLM and symbolic reasoners.
The LLM directly infers the answer given the input problem, query, and extracted reasoning path (expressed in natural language). This assumes that the LLM has common-sense knowledge of the problem at hand (e.g. spatial rules). Unlike Chain-of- Thoought prompting (Wei et al. 2022b), which asks the LLM to perform step-by-step reasoning before answering, we explicitly extract the reasoning path relevant to the query in the path identification stage, before fceding alleviates common issues suffered by LLMs due to irrelevant context, (Shi et al. 2023).
LLM Reasoner:
the LLM for reasoning. This
Symbolic Reasoner: We use the CLINGO solver [Lisschitz] 2019) which based on answer set program- ming (ASP) (Lifschitz, 2008). ASP is a logic programming paradigm that is effective for various knowledge- intensive reasoning tasks, particularly difficult (NP-Hard) search problems. Using CLINGO requires defining ASP knowledge modules which outline the rules needed to solve the problem at hand (e.g., grandson (a, b) A sister cb,c) = granddaughter c)).)). Each edge in the extted reasoning path is translated to a fact represented in ASP language (e.g., (John, brother, Jack) + brother (Jack, John)) The solver infers the answer given the facts and rules (i.c., problem-specific knowledge modulc). See Appondix A.9 for details.
5 Experiments
5.1
Experimental Setup
We evaluate all methods on four relational reasoning datasots: (i) StepGame (Shi et al. 2022 QA benchmark aiming to evaluate spatial reasoning abilities. This dataset contains a controllable parameter k which specifies the possible length of reasoning hops. We use 1000 samples for each à {3, 4, 10). (i)
6

---

Published in Transactions on Machine Learning Research (01/2026)
CLUTTR (Sinha et al. 2019 A banchmark for evaluating the English kinship reasoning abilities. We used the test set provided by the author at huggingface D Within this test set, the number of reasoning hops required to infer the answer ranges from 2 to 10. The final test dataset has 1049 samples. Each sample consists of the context, query, and label. The context describes relationships among persons within a family in a natural lone. The query provides the names of the two persons whose relation we need to deduce. The label contains the answers to the query. Moreover, the context has names tagged within 'II' and the queried entities are known and not part of the story, which is inconsistent with our problem definition (See Section 2 Therefore,
modified the test set by canceling name tagging and merging the query into the story as natural language question (e.g., How should x address y?). (Mi)) SPARTUN (Mirzaee & Kordjamshidi ) A synthesized dataset created for spatial question answering. It has a broad coverage of various types of spatial relations, spatial language expressions, and utterances. It has 2 types of questions: Find Relation and YES/NO. We run the experiments on the Find Relation type of questions as they return the actual relationships, while YES/NO questions do not. We use the first 1000 Find Relation questions from the test set in all experiments. The original dataset has its label in one of 15 concepts: 'FAR', 'NTPP", 'EC', 'NTPPI, 'TPP', etc. To make it easier for LLMs to understand the label of the questions, map the concepts back to their meaning in plain text. (iv) Chinese Kinship: We employed annotators to manually compile 73 annwwer pairs specifically focused on evaluating LLMs' ability for Chinese kinship reasoning. Chinese kinship is known for its complex relationships and presents a significant challenge for LLMs in conducting relational reasoning. Compared to English, Chinese kinship reasoning is more challenging for reasons: 1) the ages of people affect the final kinship title; 2) there are over 500 possible titles; and 3) there are aliases for kinship titles due to regional customs.
Baseline Methods: We benchmark our method against range of prompting baaspting--aeedd and neuro-yymbllic methods. We use standard Input-Output prompting (Zero-shot), Few Shot prompting (Brown et al. 2020). Chain-of-Thouuhtt (CoT) prompting (Wei et al.) 2022b), and CoT with self consistency (CoT-SC) (Wang ct al. 2023b) as prompting baselines. Zero-shot prompts the LLM to generate the answer directly, given an instruction and the input story. Few-shot prompting provides a few question-answer pairs as examples. CoT encourages LLMs to outline detailed reasoning steps before outputting the answer. Finally, CoT-SC repeatedly calls the LLM with the same prompt and outputs the most frequent answer. Both CoT and CoT-SC are with few-shot examples. To represent newro-ymmoolic methods, we benchmark LLM-ASP (Yang et al. 2023). which first extracts symbolic facts from the story using LLMs and then uses ASP (Lifschitz, 2008 | solvers for inferring answers. We choose LLM-ASP since it displays good performance on a variety of relational reasoning lasks and requires no finetuning. More details on LLM-ASP experiments can be found in Appendix A.9 We do not benchmark neuro-symbolic methods (e.g., LLM-ASP, PoT w/ symbolic reasoner (PoT-Symbolic)) on the Chinese kinship and SPARTUN datascts as the complexity of their possible relations (e.g., >50 possible Chinese kinship relations) makes it difficult to write symbolic knowledge module (See Appendix A.11 for details).
We benchmark all methods using GPT-3.5-turbo (0125) (Ouyang et al. 2022). GPT-4- turbo (2024-04-09) (Achiam et al. 2024), GPTT-o (2024-05-13) and
Backbone LLMs:
(Grattafiori et al. 2024h. Besides the genoral-purpose LLMs, we also benchmark all methods on the CLUTTR (Sinha et al. 2019) and StepGame (Shi el al. (022)) dataset using reasoning-focuse LLMs, namely GPT-ol-mini (Jaech) et al.) 2024h, Claude-3.7-sonnee
and GPT-5 7 All experiments were conducted with a sampling temper- ature of 0.3 and maximum output token length of 4096, except for GPT-5, where we set the sampling temperature and output token length to their default values.
Performance Metrics: We follow prior works to measure accuracy between predicted relations and ground truth relations, where it checks whether at least. one target relation exists within the predicted relations.
http::///wugggngggaacooo//aasaas /t/we////////////////////nn/ 3rairr/22n_tai2234 train234 tost2to10/tcst.
3 httpe///oomma.coommmmmeeeetttttt

---

Published in Transactions on Machine Learning Research (01/2026)
5.2 Experimental Results
Table 1: Single-trial accuracy results. Prompting-bascd methods LLMs directly predict the Neuro-symbolic methods (extraction + symbolic reasoning) use LLMs for semantic parsing and symbolic solvers for reasoning. PoT-LLM and PoT-Symbolic represent using an LLM or symbolic solver as the reasoner, respectively. The for StepGame represents the number reasoning hops required to infer the answer. The bold and underline fonts represent the best and second-best results, respectively. means the highest accuracy statistical significantly higher than the second highest accuracy measured by the paired McNemar test.
Stepgame k-4
Chinese Kinship
LLM
Method
CLUTRR SPARTUN
k-3
k-10
Prompting- based
Zero-shot
24.0
22.5
17.0
31.2
44.3
20.5
GPT-3.5-ture 5-turo
Few-shot
21.3
20.8
16.8
33.4
35.1
23.3
CoT
31.1
26.7
19.3
35.6
21.9
CoT-SC
30.7
28.0
21.7
37.1
47.6
24.7
PoT-LLM (Ours)
50.9*
44.8"
28.8"
35.1
52.7
27.4
Extraction + Symbolic Reasoing
LLM-ASP
76.4"
83.7"
72.6'
32.8
PoT-Symbolic (Ours)
72.4
75.9
66.0
54.1"
Prompling-buse based
Zero-shot
59.0
52.5
32.8
45.9
72.8
45.2
Few-shot
55.3
50.7
29.8
12.0
76.9
37.0
GPT-4-turr 4-turbo
CoT
58.3
51.3
34.2
53.0
79.7
39.7
CoT-SC
57.4
51.7
34.4
54.6
78.1
46.6
PoT-LLM (Ours)
67.4*
59.8
40.1*
57.6"
75.5
53.4
Extraction + Symbolic Reasoing
LLM-ASP
83.7
89.4
81.1
48.1
PoT-Symbolic (Ours)
88.2
92.6*
85.6
66.1
Prompting-basee
Zero-shot
68.6
60.1
37.7
45.5
81.6
67.1
Few-shot
36.6
36.4
26.6
36.5
80.5
65.8
CoT
69.4
61.0
40.0
57.6
81.4
68.5
GPT-40 40
CoT-SC
70.0
63.2
40.4
59.4
78.9
68.5
PoT-LLM (Ours)
73.4"
68.0
48.7"
61.9"
83.1
71.2
Extraction + Symbolic Reasoing
LLM-ASP
85.3
84.7
71.6
56.7
PoT-Symbolic (Ours)
88.2*
92.9*
86.3*
67.7*
Prompting- based
Zero-shot
21.4
20.0
10.1
21.2
65.0
31.5
Feu-shot
41.2
38.2
25.7
18.6
70.0
35.6
Llama3 70B
CoT
60.0
50.6
31.6
52.8
73.8
27.4
CoT-SC
61.3
50.2
32.9
54.7
72.6
27.4
PoT-LLM (Ours)
69.1*
61.6"
41.8"
56.3
72.8
35.6
Extraction + Symbolic Reasoing
LLM-ASP
85.7
83.2
PoT-Symbolic (Ours)
86.4
92.6
84.5
65.9
8

---

Published in Transactions on Machine Learning Research (01/2026)
Full Pipeline Performance: In Table 1 we compare the full pipeline performance of all baselines us- ing different general-purpose backbone LLMss. For computational cost. reasons, all results are single-trial. POT-LLM and PoT-Symbolic represent using an LLM or a symbolic solver as the reasoner, respectively. For fair comparison, we compare PoT-LLM to prompting-based pipelines. On the other hand, PoT-Symbolic is compared to the LLM-ASP as extra domain knowledge (i.e., symbolic rules) is required for reasoning in both methods. The prompts we use can be found in Appendix A.3
The results show clear improvement of PoT over the basclines. Among the prompting-basse based methods, POT-LLM outperforms all baselines in most scenarios, except the CLUTRR on GPT-3.5-turbo, and the SPAR- TUN on GPT-4-turbo and Llama3-70B. Mcanwhile, for the extraction plus symbolic reasoning methods, results show a clear improvement of PoT-Symbolic over the LLM-ASP, with exceptions of the StepGame on GPT-3.5-turbo.
Interestingly, COT and Few-shot prompting have only a minor improvement with powerful models (c.g.. GPT-4) compared to Zero-shot prompling, as observed previously (Yang et al., 2023]. This suggests that linear chain of thought reasoning may already exist in larger models, and imposing it externally is not always helpful for complex reasoning tasks. Moreover, observe that the performance of most methods steadily degrades as the number of possible relations increases. Consequently, the Stepgame dataset shows the largest gap with prompting baselines, because directly solving this complex and high-order reasoning problem too challenging for LLMs with just in-context learning examples.
More specifically, for CPP--35-turrke with prompting-base based methods, PoT-LLM exceeds all baselines with a large margin except for the CLUTRR dataset. As for the extraction plus symbolic reasoning methods, the rosults show an improvement of PoT-Symbolic over the LLM-ASP the CLUTRR datasct, but not the Stepgame dataset. We believe the issue stems from the weaker instruction-following ability of GPT-3.5- turbo, which leads to poor triplet extraction outcomes. Better-optimized prompts could possibly improve performance.
For GPT-40, employing prompting-basee based methods (Zero-shot, Few-shot, CoT, CoT-SC), shows significant improvement over GPT-4-turbo. This suggests an enhancement in its fundamental reasoning abilities, po- tentially due to training on larger and more recent data corpus. Performance stcadily decreases from k = 3 to k = 10 for Stepgame, except for neuro-symbolic methods, where k has the highest performance. We also observe that the improvements of our methods compared to the second-best. methods increase when the number of reasoning hops increases. This observation holds for both prompting-basee based and extraction + symbolic reasoning, indicating that our method, with access to a powerful LLM, can outperform harder questions that require reasoning over longer reasoning chains.
As for 70b-isttbb,,,,trutt, our methods outperform or equal baselines in the majority of scenarios except for prompting-based based on the SPARTAN dataset.
Moreover, we further evaluate our approach and the baseline methods on the CLUTRR and Stepgame dataset using rcasoning- ceasoning-oocused focused LLMs, GPT-ol-mini, Claude-3.T-somnet, and GPT-5. The results are presented in Table 2 For GPT-ol-mini and Claude-3.7-sonmee lines in both prompting- based and extraction-plus-symbol settings, with a large margin in most scenarios, demonstrating the compatibility of the proposed method with these rcasoning LLMs. As for GPT-5, we notice that its reasoning effort adjustable, which could affect the reasoning task we are trying to solve. Therefore, we try GPT-5 under two reasoning efforts: minimal and medium. Although our method performs par with baselines under medium reasoning effort, it shows improvements when the reasoning effort is minimal, requiring much fewer tokens overhead than in medium reasoning effort. This demonstrates that our design effectively preserves reasoning accuracy even with reduced model reasoning effort.
results show that our approach outperforms the base-
To evaluate the cost of experiments in detail, we record the average times and tokens spent on each module over all samples from the k=3 subset of the Stepgame dataset. The results are shown in Table 3 and Table respectively.
Computational Cost:
Despite requiring two LLM calls, POT-LLM achieves computational cost and latency comparable to 10 and CoT. We attribute this efficiency to the simplicity of our extraction prompt and the modular design of
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Results on the Stepgame and CLUTRR datasets with reasoning-focuse focused LLMs. means the highest accuracy is statistical significantly higher than the second highest accuracy mcasured by the paired McNemar test
Stepgame km
CLUTRR
LLM
Method
3
k-10
Prompling-baee based
Zero-shot
70.6
63.5
37.8
47.9
Chaade-3. I-somnet
Few-shot
71.4
64.1
40.9
44.2
CoT
76.0
70.2
44.5
37.4
CoT-SC
75.3
70.6
46.0
61.7
PoT-LLM (Ours)
81.0*
76.9"
57.2"
67.7'
Estraction Symbolic Reasoing
LLM-ASP
84.9
89.8
83.7
67.3
PoT-Symbolic (Ours)
88.0
92.6*
87.8"
67.6
Prompting-basei
Zero-shot
67.5
63.5
42.0
58.4
GPT-o1 OPT-ol-mint -mink
Few-shot
67.5
57.9
38.3
61.3
CoT
68.5
61.8
37.5
61.0
CoT-SC
69.3
59.8
39.8
64.6
PoT-LLM (Ours)
72.9*
65.8"
49.3*
58.5
Extraction + Symbolic Reasoing
LLM-ASP
81.1
87.0
63.5
20.2
PoT-Symbolic (Ours)
82.8
88.3
80.3
65.9*
Prompting-
Zero-shot
89.1
88.2
81.4
73.2
GPT-5-Medamm
Few-shot
89.2
89.3
82.2
67.4
CoT
89.4
89.0
82.0
70.2
CoT-SC
89.3
89.7
83.0
71.1
PoT-LLM (0urs)
89.5
89.5
83.7
71.5
Extraction + Symbolic Reasoing
LLM-ASP
89.2
93.8
88.1
44.0
PoT-Symbolic (Ours)
89.2
93.3
72.3*
Prompting-buut
Zero-shot
44.7
40.9
27.5
28.6
GPT-5-Miniie Minnimal
Few-shot
45.1
40.1
29.6
31.9
CoT
69.3
76.9
60.5
55.2
CoT-SC
73.7
82.7"
67.4
60.7
PoT-LLM (Ours)
75.6
75.5
68.9
65.0*
Extraction Symbolic Reasoing
LLM-ASP
82.4
87.6
82.1
2.3
PoT-Symbolic (Ours)
88.8*
93.2"
87.8*
65.4"
our pipeline. Specifically, the reasoning module can operate on the reasoning path identified by the path identification module, rather than processing the entire input story. This targeted approach significantly reduces computational overhead while preserving strong performance.
To evaluate the impact of prompts on the performance of the relation extraction, we construct a synthetic set consisting of stories of multiple sentences and their corresponding triplets as labels. To balance the trade-off between manual labeling and data quantity for accurate results,
Graph Extraction Performance:
10

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Detailed statistics of average time, in seconds, spent on each module. The results are calculated on all samples from the k=3 subset of the Stepgame datasct with GPT-4o backbone LLM. mcans actual time is less than L.
Method
Modules
Overall
Graph Extraction
Path Identification
Reasoning
Zero-shot
2.925
CoT
3.029
PoT-LIM (Ours)
1.363
0.001
2.049
3.412
PoT-Syabolic (Ours)
1.337
0.001
0.727
2.064
Table 4: Detailed statistics of average LLM tokens spent on each module. The results are calculated over all samples from the k=3 subset of the Stepgame dataset with GPT-4o backbone LLM.
Mothod
Graph Extraction Path Identification
Reasoning
Overall
In Token Out Token In Token Out Token In Token Out Token
In Token Out Token
Zero-ahot CoT
249.5
196.6
2251.5
235.2
PoT-LLM (Ours)
1359.8
56.4
1285.6
147.8
2645.4
204.2
PoT-Symboicc (Ours)
1359.8
55.8
1359.8
55.8
manually labeled a pool of 100 sentences from the Stepgame dataset (Shi et al. 2022) with their corresponding triplets as the sentence pool. Each test story consists of 20 sentences uniformly sampled from the sentence
pool and query sentence asking about the spatial relation between 2 mentioned entities. The results of
testing different prompts using GPT4-turo on 1,000 such stories are presented in Table 5 The results demonstrate that the prompt design strategy employ, which is explicitly tailored to extract relations (as detailed in Section |.1), can accurately extract triplets from unstructured text. The introduced method significantly outperforms non-customized, in-context learning-base methods, such as zero-shot and CoT.
To investigate the effectiveness of the path identi- fication module, we conducted an ablation study by removing the path identification (P.I.) from both the prompting-based and symbolic-based variants of our framework. As shown in Table 6 eliminating P.I. leads to a notable performance drop for the prompting based PoT-LLM pipeline, particularly on the Stepgame dataset, where accuracy decreases from 73.4 to 69.2 (k=3), 68.0 to 61.7 (k=4), and 48.7 to 38.8 (k=10), and on CLUTRR (61.9 to 50.7). This confirms that P.I. effectively guides the LLM to locate the correct reasoning chain and miligates distraction from irrelevant paths. In contrast, the symnolic reasoning variant (PoT-Symbolic) shows only marginal changes (less than 0.5%), indicating that once explicit logical structures are extracted, path identification bocomes less critical on the Stepgame and the CLUTRR datasct.
Ablation Study on Path Identification Module:
However, errors may arise from LLM graph extractions or roverrvll descriptions the story. To better investigate the effectiveness of the path identification module, we built synthetic dataset to simulate situations in which extraction errors occur during graph extraction, including but not exclusive to LLM errors controversial descriptions in the story.
Based on observations of common LLM extraction errors (see Table 25 we design 7 possible noise types. Details can be found in Appendix A.I
We build synthetic datasets based on the graph structural information underlying the stories that come with the CLUTRR datasct (Sinha et al. 2019). Details can be found in Appendix A.7 For cach noise type, wo generate 100 noisy samples. The results are shown in Figure 2 Symbolic reasoner w/ P.I. beats or ties the symbolic reasoner w/o P.I. for all noise types. Symbolic reasoner w/o P.I. is particularly sensi- tive to noise type like "adding irrelevant edge" and "adding main edge". Note that in the synthetic dataset, it is possible to introduce conflicting information since relations are chosen randomly. It is observed that symbolic reasoner w/o P.I. can struggle to resolve contradictions and is perturbed even if the conflicts are irrelevant to answer the question.
We also evaluate how symbolic reasoner w/ P.I. and symbolic reasoner w/o P.I. fare as more noise elements introduced. We observe that symbolic reasoner w/ P.I. remains robust under various
11

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Extraction performance of different prompt techniques with GPT-4-turbo. 'Acc. Triplets' and 'Acc. Query' represents the percentage of correctly extracted triplets and queries among all stories, respectively. correctly extracted. The bold and underline fonts represent the best and second-best results within the group, respectively.
"Acc. All' denoles the percentage of stories where the triplets and query
Prompt Method
Acc. Triplets
Acc. Query
Acc. All
Zero-shot CoT Zero-shot
74.1
99.3
74.0
70.3
93.8
70.3
Few-shot
87.4
99.8
87.4
CoT Few-shot
91.9
99.9
91.9
ours
95.9
100.0
95.9
Table 6: Ablation study over the path identification module. The performance drops for the PoT-LLM pipeline and remains for PoT- Symbolic pipeline over Stepgame and the CLUTRR dataset.
Stepgame k=4
CLUTRR
LLM
Method
k=3
k=10
Prompting-boat baaeed
PoT-LLM PoT-LLM w/o P.I.
73.4
68.0
48.7
61.9
GPT-10
69.2
61.7
38.8
50.7
Extraction + Symbolic Reasoing
PoT-Symbolic
88.2
92.9
86.3
67.7
PoT-Symbolic w/o P.I.
88.2
93.1
86.6
67.2
amounts of noise interference. contrast, the performance of the symbolic reasoner w/o P.I. solver declines significantly the amount of noise elements increases.
1.0
1.0
0.8
0.8
accuracy
accuracy
0.6
0.4
0.4
0.2
Symbolic Reasoner P.I.
0.2
Symbolic Reasoner w/o
Symbolic Reasoner w/ P.I.
Symbolic Reasoner w/ P.I.
0.0
type type_B B type C type type_E type_F type_G lype of noise
0
6 number of noise
7 8
9
10
Figure 2: Left: Accuracy of symbolic reasoner w/ P.I. and symbolic reasoner w/o P.I. w.r.t noise Types. A: flip - irrelevant edge, add new_nodw one new edge, C: add - new new_nodw conflict edge, add - no node irrelevant edge, E: add no node main edge, F: replace irrelevant edge, and G: disconnccted edges. Right: Accuracy of symbolic reasoner w/ P.I. and symbolic reasoner w/o P.I. w.r.t the number of noises.
6 Limitations
Although the proposed framework demonstrates promising performance, it has several important limitations.
First, PoT has been tested only on problems that satisfy the definition of relational reasoning, which cach relationship can be expressed as a composition or a chain of other relationships. The generality of PoT on non-relational reasoning tasks has not yet. been systematically evaluated and remains open question.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Second, the experimental setup is constrained by dataset quality. During our case studies, we identified multiple instances of incorrect or incomplete ground-truth labels that could confound evaluation results. Addressing these data-quality issues would likely further improve the reliability of our conclusions.
Third, constructing symbolic rules remains a challenge for PoT-Symbolic. This process requires domain expertise and significant manual effort. Recent work (e.g., (Lee & Hwang 2025) suggests that L.MSs may help automate rule generation, an avenue we leave for future exploration.
Fourth, while PoT-LLM removes the need for manual rules, it introduces higher inference latency and a small accuracy drop due to its reliance on LLM-based reasoning. Advances in model architectures (e.g., MoE) and LLM training methods are likely to mitigate these issues, potentially enabling PoT-LLM to achieve both lower lateney and stronger reasoning performance.
Given these complementary trade-offs between the PoT variants, we recommend PoT-Symbolic for domains with small, well-defined relations, where manual rules provide higher performance. In contrast, PoT-LLM is preferable in settings that require greater scalability and generality, despite its higher computational cost.
7
Conclusion
We introduce Path-of- Thoughts (PoT), a novel framework that decomposes a relational reasoning task into three stages: graph extraction, path identification, and reasoning. Our experiments demonstrate that PoT outperforms basclines across four benchmark datasets, without the need for finc-tuning or extensive large language model (LLM) calls. Unlike previous approaches, PoT exhibits strong resilience to noise relations by leveraging the compositional nature of graphs. Additionally, we conduct analysis experiments to demonstrate the contributions of cach module of the PoT, and to highlight the importance of identifying key relations and the order of reasoning path in effective relational reasoning tasks.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Bal- aji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, et al. GPT-4 technical report. arXiv preprint wxXi:2303.087777 2024.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michat Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of Thoughts: Solving elaborate problems with large language models. Proceedings of the AAAI Confer- 2024.
ence on Artificial Intelligence, 78616)
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. Experience grounds language. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, et al. Language models are shw-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and H. Lin (ods.), Aduances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretehon Krucger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, et al. Evaluating large language models trained on code. orXiv preprint orXiv:2107.0337/ 2021.
Anthony G Cohn and Robert Blackwell. Evaluating the ability of large language models to reason about cardinal directions. In Benjamin Adams, Amy L. Griffin, Simon Scheider, and Grant McKenzie (eds.),
13

---

Published in Transactions on Machine Learning Research (01/2026)
16th International Conference on Spatial Information Theory (COSIT 2024), volume 315 of Leibniz In- ternalional Proceedings in Informalics (I.PPcs), pp. 28:1-28:9 Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the Conference of the North American Chapler of the Association for Compulational Linguislics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.
Mingyu Ding, Zhenlang Chen, Tao Du, Ping Luo, Josh Tenenbaum, and Chuang Gan. Dynamic visual reasoning by learning differentiable physics models from video and language. In Advances in Neural Information Processing Systems, 2021.
Mona M Garvert, Raymond J Dolan, and Timothy EJ Behrens. A map of abstract relational knowledge in the human hippocampal entorhinal cortex. eLife, 6:e17086, 2017. doi:
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, et al. The llama 3 herd of models. arXiv preprint ar 2xXi:2407.21186 2024.
Hanxu Hu, Hongyuan Lu, Huajian Zhang, Yun-Ze Song, Wai Lam, and Yue Zhang. Chain-of-symbol prompt- ing for spatial reasoning in large language models. In First Conference on Language Modeling, 2024.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2): 1-55, 2025.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Ei-Kishky, Aiden Low, Alec Helyar, Alck- sander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, et al. Openai ol system card. arxiv preprint or Xrv: 16720, 2024.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, 2022.
Jinu Lee and Wonscok Hwang. Symba: Symbolic backward chaining for structured natural language rea- soning, In Proceedings of the 2025 Conference of the Nations of the Americas Chapler of the Association for Compulational Linguistics: Human Language Technologies ( (Volume 1: Long Papers), pp. 2468 2484, 2025.
Guozheng Li, Peng Wang, and Wenjun Ke. Revisiting large language models as zero-shot relation extractors. In Findings of the Association for Computational Linguistics: EMNLP, 2023.
Vladimir Lifschitz. What is answer set programming? In Proceedings of the National Conference on Artificial Intelligence - Volume 3, 2008.
Vladimir Lifschitz. Answer Set Programming. Springer Publishing Company, Incorporated, 1st edition, 2019.
Roshanak Mirzaee and Parisa Kordjamshidi. Transfer learning with synthetic corpora for spatial role labeling and reasoning. In Proceedings of the Conference on Empirieal Methods in Natural Language Processing, 2022.
Roshanak Mirzaee and Parisa Kordjamshidi. Disentangling extraction and reasoning in multi-hop spatial reasoning. In Findings of the Association for Compulational Lònguistics: EMNLP, 2023.
Roshanak Mirzace, Hosscin Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. SPARTQA: A tex- tual question answering benchmark for spatial reasoning. In Proceedings of the Conference of the North American Chapter of the Association for Compulational Linguisties: Human Language Technologies, 2021.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Ida Momennejad, Hoscin Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson. Evaluating cognitive maps and planning in large language models with cogeval. In Advances in Neural Information Processing Systems, 2023.
Maxwell Nye, Michael Henry Tessler, Joshua B. Tenenbaum, and Brenden M. Lake. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. In Advances in Newral Information Processing Systems, 2021.
Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua Tenenbaum, and Roger Levy. LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Rasmus Berg Palm, Ulrich Paquet, and Ole Winther. Recurrent relational networks. In Advances Neural Information Processing Systems, 2018.
Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Findings of the Association for Computational Linguistics: EMNLP, 2023.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. Proceedings of the International Conference on Learning Representations, 2022.
Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning associative inference using fast
weight memory. In Proceedings of the International Conference on Leurning Represenlations, 2021.
R. Sedgewick. Algorithms in C, Part 5: Graph Algorithms. Addison Wesley Professional, 2001.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models be easily distracted by irrelevant context. In Proceedings of the 40th International Conference Machine Learning, ICML'22 JMLR.org, 2023.
Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts. In Proceedings of the AAAI conference on arlificial intelligence, volume 36, pp. 11321 11329, 2022.
Tom Silver, Soham Dan, Kavitha Srinivas, Josh Tenenbaum, Leslie Kaelbling, and Michael Katz, Generalized planning in pddl domains with pretrained large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethinking inter- pretability in the era of large language models. orXiv preprint S.kk::2402.17666 2024.
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William Hamilton. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joinl Conference on Natural Longuage Processing, 2019.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslic Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, et al. LaMDA: Language models for dialog applications. arXio preprint ar arx:::2001.00 .08239, 2022.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Edward C. Tolman. 10. 10.1037/h006160
Cognitive maps in rats and men. Psychological Review, 55(4))189-2088 1948. doi:
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models- critical investigation. In Advacnes in Neural Information Processing Systems, 2023.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In Proceedings of the International Conference on Learning Representations, 2018.
Mengke Wang, Shaorong Xie, Xiangfeng Luo, Yang Li, Han Zhang, and Hang Yu. HCTA: Hierarchical coop- erative task allocation in multi-agent reinforcement learning. In 2023 IEEE 35th International Conference on Tools with Artificial Intelligence (ICTAI), pp. 934-941, 2023a. doi:
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Fleventh International Conference on Learning Representations, 2023b.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022a.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning large language models. In Advonces in Newral Information Processing Systems, 2022b.
James Whittington, David McCaffary, Jacob Bakermans, and Timothy Behrens. How to build cognitive map. Nature Newroscience, 25:1 16, 2022. doi: 111133//411193-22---010
Lionel Wong, Gabriel Grand, Alexander K. Lew, Noah D. Goodman, Vikash K. Mansinghka, Jacob Andreas, and Joshua B. Tenenbaum. From word models to world models: Translating from natural language to the probabilistic language of thought. arXiv preprint (tl::2306112674. 2023a.
Lionel Wong, Jiayuan Mao, Pratyusha Sharma, Zachary S. Siegel, Jiahai Feng, Noa Korneev, Joshua B. Tenenbaum, and Jacob Andreas. Learning adaptive planning representations with natural language guid- ance. arXiv preprint arXin:2312 08566, 2023b.
Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carric J Cai. PromptChainer: Chaining large language model prompts through visual programming. In CHI Conference Human Factors in Computing Systems Fatended Abstmacts, 2022.
Yutaro Yamada, Yihan Bao, Andrew Kyle Lampinen, Jungo Kasai, and llker Yildirim. Evaluating spatial understanding of large language models. Transactions on Machine Learning Research, 2024.
Zhun Yang, Adam Ishay, and Joohyung Lee. Coupling large language models with logic programming for robust and general reasoning from text. In Findings of the Association for Computational Linguistics: ACL, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree Thoughts: Deliberate problem solving with large language models. Advances in Newral Information Processing Systems, 36, 2024.
Dongran Yu, Bo Yang. Dayou Liu, Hui Wang, and Shirui Pan. A survey on neural-symbolic learning systems. Newral Networks, 166:105-126. 2023. doi: http:///abborrr/..o0////.eetttt 202.006.88.. URL
16

---

Published in Transactions on Machine Learning Research (01/2026)
Hanlin Zhang, Jiani Huang, Ziyang Li, Mayur Naik, and Eric Xing. Improved logical reasoning of lan- guage models via differentiable symbolic programming. In Findings of the Associalion for Compulational Linguistics: ACL, 2023.
Wanjum Zhong, Yifan Gao, Ning Ding, Yujia Qin, Zhiyuan Liu, Ming Zhou, Jiahai Wang, Jian Yin, and Nan Duan. ProQA: Structural prompt-based pre-training for unified question answering. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022.
Jiaming Zhou, Abbas Ghaddar, Ge Zhang, Liheng Ma, Yaochen Hu, Soumyasundar Pal, Mark Coates, Bin Wang, Yingxue Zhang, and Jianye Hao. Enhancing logical reasoning in large language models through graph-based synthetic data, 2024.
A Appendix
A.1 Noise Types
We consider the graph to have two parts: main chain, which is the primary reasoning path connecting the source node to the target node; and an irrelevant part, which consists of all nodes and edges that not part of the main chain. When introducing noise, we do not corrupt the main chain, we do not want to change the ground-truth answers. The 7 noise types are shown in Figure 3 Specifically, (A) Flip an irrelevant edge: Flip the dircction of an irrelevant edge connecting 2 irrelevant nodes. (B) Add a new node with a new edge: Add a new node and a new edge that connects the new node to either the main chain or lo a node in the irrelevant part of the graph. (C) Add conflict edges: Add new node and connect it to either the main chain or the irrelevant part with 2 new edges. Noted that new edges may contain conflict attributes. (D) Add an irrelevant edge: Add edge connecting 2 irrelevant nodes. (E) Add main edge: Add an edge between two nodes on the main chain. (F) Modify the relation of an irrelevant edge: Change the relation on an irrelevant edge without changing its direction. (G) Add disconnected
edge and nodes:
Add 2 new connected nodes that are both disconnected from the original graph.
A.2 Path Counts Analysis
To better understand how multiple reasoning paths affect framework accuracy, we analyzed the distribution of path counts in the PoT-symbolic experiment on the CLUTRR dataset, using GPT-4o as the backbone LLM. The first two rows of Table 7 summarize both the path count distribution and the corresponding average accuracy. Most stories (697 out of 1048) produce only a single path, achieving an average accuracy of 68.9%. Accuracy increases as more paths are extracted, peaking at 84.6% when the path count reaches three, and then gradually declines the path count continues to rise, reflecting the increasing difficulty of those samples.
We further conducted an experiment in which only the shortest path was identified and passed to the downstream symbolic solver. As shown in the last row of Table 7 this sctting led to a notable drop in average accuracy across multiple path counts, reaching 14.3% drops for samples containing five or more paths. These results highlight the importance of capturing multiple reasoning paths in boosting final reasoning accuracy.
Table 7: Number of samples and average accuracy with respect to the number of chains extracted from the sample for the experiment of PoT Sym-oyyco over the CLUTRR dataset. The GPT-4o is applied as the backbone LLM.
# of chains
0
2
3
4
5+
# of samples
39
697
175
13
75
49
Acc. of Identifying Multiple Paths (default)
0.0
68.9
73.1
84.6
74.7
71.4
Acc. of Identifying single Shortest Path
0.0
68.9
63.4
84.6
66.7
57.1
17

---

Published in Transactions on Machine Learning Research (01/2026)
Flip
Irrelevant edge (type_A) A):
Toy example:
With an new edge (type_B):
or
question:
New node
With conflict edges (type_
Add
noises
Irrelevant edge (type_D):
Main edge (type_E):
Replace
Irrelevant edge (type_F):
8
Disconnected edges (type_G):
Figure 3: Illustrations of 7 noise types in the synthetic noise dataset. A toy graph with 5 nodes and 4 edges is shown. The nodes and edges that are relevant or irrelevant to answering the question are marked in red and green, respectively. The noisy nodes/edees /edges are marked in purple.
A.3 Implementation Details
Compute: All experiments were conducted using the OpenAl AP on an Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz.
ASP Solver For the neuro-symbolic methods, the Clingo ASP solver (Lifschitz) 2019). We utilize the ASP knowledge modules from LLM-ASP (Yang et al. 2023 which were written by human domain experts.
A.4 Prompts for Baselines
In this section, we show the prompt templates we use for baselines in Table [ The prompts are identical for each baseline across all backbone LLMs.
A.4.1 Prompt Templates for 10
The prompt templates of 10 for StepGame, CLUTRR, SPARTUN, and Chinese kinship datasets can be found in Tables and respectively.
A.4.2 Prompt templates for Few-Shot
The prompt templates of Pew-Shot for StepGame, CLUTRR, SPARTUN, and Chinese kinship datasets can found Tables 12
13 14 and respectively.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Table 8: I0 prompt template for StepGame.
Given story about spatial relations among objects, answer the relation between two queried objects. The answer could only be one of following: [top, bottom_ left, top left, bottom, bottom right, vop_right, right, left, overlapl.
If a sentence the story is describing clock-wise information, then 12 denotes above, and denote upper- right, 3 denotes right, 4 and 5 denote lower-right, 6 denotes below, 7 and 8 denote lower-left, 9 denote left, 10 and 11 denote upper-left. If the sentence is describing cardinal directions, then north denotes above, east denotes right, south denotes below, and west denotes left. Wrap your final answer in brackets. Example: [Lop]-
Story: {story]
Answer:
Table 9: TO prompt template for CLUTRR.
Given story about kinship relations among persons, answer the relation between two queried persons. The answer could only be one of following: [son, grandmother, daughter-in-law, grandson, greatgrandson, grandfather, mother-in-law, greatgranddanghter, uncle, son-in-law, wife, greatgrandfather, brother, husband, daughter, father-in-law, sister, greatgrandmother, granddaughter, aunt, nephew, niece, mother, father]. Wrap your final answer in brackets. Example: [grandfather]
Story: {story}
Answer:
A.4.3 Prompt Templates for CoT and CoT-SC
The prompt templates of CoT and CoT-SC for StepGame, CLUTRR, SPARTUN, and Chinese kinship datasets can be found in Tables 16
18
19
20 and 21 respectively.
A.4.4 Prompt Templates for LLM-ASP
The prompt templates of LLM-ASP for StepGame and CLUTRR (prompts for extracting relations and genders.) datasets can be found in Tables 22
23
and 24
A.5 Prompt Templates for Graph extraction
Tables 27
29 and 30 showcase the prompt templates used for extracting the instance graph via in-context learning for the StepGame, CLUTRR, and Chinese kinship datasets, respectively.
28
We find GPT-4 models to be powerful information extractors. However, we do observe some common errors such as wrong gender, missing relations, or wrong entity. See Table 25 for examples from the CLUTRR dataset.
Common Extraction Errors:
A.6 Prompt Templates for LLM Reasoner
Tables 31 33 34 and 35 showcase the prompt templates used for LLM reasoner for the StepGame, CLUTRR, SPARTUN, and Chinese kinship datasets respectively. Note that for the Stepgame and CLUTRR datasets, we replace the original stories with the extracted instance paths, whose format can be found in the corre- sponding in-context learning examples. On the other hand, for the SPARTUN and Chinese kinship datasets, we append the extracted instance path together with the original story.
A.7 Noise Datasets
We build synthetic noise datasets based on the clean samples in the CLUTRR dataset (Sinha al. 2019)) which we call the root sample set. When generating a noisy sample for specific type of noise we randomly select sample from the root sample set and then add the noise to the sample random location. For each
19

---

Published in Transactions on Machine Learning Research (01/2026)
Table 10: 10 prompt template for SPARTUN.
Given a story about spatial relations among objects, answer the relation between two queried objects step by step. The answer could only one following: [far, in, touch, has, covered by, right, overlap, front, behind, cover, left, disconnected. from, below, above, near]. 'inside and touching" refers 'covered__y 'inside' and 'within' and 'inside' refers 'in'. 'contain' refers 'has'. If the sentence is describing clock-wise information, then denotes right, 6 denotes below, 9 denotes left, and 12 denotes above. If the sentence is describing cardinal directions, then north denotes above, east denotes right, south denotes below, and west denotes left. There could be multiple answers. Wrap all your answers in brackets. Example: (above, behiud].
Story: {story]
Answer:
Table 11: IO prompt template for Chinese kinship.
You are given question about chinese kinship relations, please the question step step. ansaers include but not limited to chinese kinship titles: (M/# your final answer in square brackets Tf more than relation is correct, separate the relations by comma, like:
Wrap
(
Question: {story)
Answer:
noise type, we generate 100 noisy samples. In all of these noise types, it is possible to introduce conflicting information (relations are chosen randomly).
To generate sample containing n noise elements, first randomly select types of noise (with replace- ment), and then we introduce them into randomly chosen sample from the root sample set. We generate 100 noisy samples for each number category. The findings are presented in Figure 2
A.8 CLUTRR Dataset Ambiguities
After manual checking, we have found several cases where the story has more than one possible answer. See Table 36 for examples in the CLUTRR dataset.
A.9 Baselines
LLM-ASH We use the same solver (CLINGO v5.6.0) and knowledge modules used in LLM-ASP Yang
et al. 2023) for the CLUTRR and Stepgame datasets. Contrary to our problem definition (see Section 2 LLM-ASP Yang et al. 2023 assumes that the query is given and need not be extracted for the CLUTRR dataset. Moreover, facts extracted one sentence at time for the stepgame dataset. Therefore, modify the prompt so that the LLM extracts all triplets and queries with one LLM call.
CoT-SC: We use the same prompt as CoT for CoT-SC. We call LLMs 5 times and take the top result of a majority vote as the final output of CoT-SC.
A.10
More Experiments
A.11
Chinese Kinship Complexity
The Chinese kinship reasoning is much more challenging than English kinship reasoning in the following 2 dimensions:
More kinship titles: The English kinship reasoning benchmark, CLUTRR, contains 24 kinship titles, while our internal Chinese kinship datasot has over 500 possible kinship titles.
20

---

Published in Transactions on Machine Learning Research (01/2026)
Table 12: Few shot prompt template for StepGame.
Given a story about spatial relations among objects, answer the relation between two queried objects. The anawer could only be one of following: [top, bottom left, top_ top_leff,, bottom, bottom bottom_right, vop_right, right, left, overlap]- If sentence in the story is describing clock-wise information, then 12 denotes above, 1 and denote upper-right, 3 denotes right, 4 and denote lower-right, denotes below, 7 and denote lower-left, 9 denote left, 10 and 11 denote upper-left. If the sentence is describing cardinal directions, then north denotes above, east denotes right, south denotes below, and west denotes left. Wrap your final answer in brackets. Example: [top).
Story: J is over there and D is on the top of it. S is upper right to W. J is directly south west of S. M is below P and to the right of C is sitting at the 3:00 position D. A is diagonally above D) to the left at a 45 degree angle. C is sitting at the 9:00 position of Y. S presents left: to Y. J is on the right side to V. What is the relation of the agent to the agent S?
Answer: [top left]
Story: Object Y is below object X and to the left of it, too. H is to the right of M. Y is placed at the bottom of U. is over there and T is on the right. J is directly below V. is over there and A is on the right of it. U is over there and H is on the right. F is sitting in the left direction of H. M is positioned below Y. What is the relation of the agent to the agent U? Answer: [right]
Story: B is to the right of L and is on the same horizontal plane. M and L are next to each other with L on the right and M on the left. B is at the bottom and D is on the top. J is to the top of W vertically. A is to the bottom-left of I. J is sitting the top position to M. H is above J with small gap between them. B is on the same horizontal plane directly right to E. E is on the right and W is on the left. What is the relation of the agent L to the agent H?
Auswer:
[bottom right]
Story: H and K are side by side with K at the bottom and H on the top. P is below K with small gap between them. U is there and Z is at the 10 position of a clock face. Object A is above object M and to the right of it, too. D is to the right of H horizontally. P and C are parallel, and P is to the right of C. G and C are vertical and G is above C. Q and E are next to each other with Q on the left and on the right. The object is positioned below and to the right of the object J. E is above S at 2 o o'clock. F and J are both there with the object F is to the right of object J. Z is over there and N is on the left. is diagonally left and below L. If U is the center of a clock face, G is located between 10 and 11. F is directly above W. P is directly north west of V. S is there and L is at the 10 position of a clock face. Q is positioned below D. N is to the bottom left of D. Tf A is the center of a clock face, located between 4 and 5. What is the relation of the agent E the agent. 7? Answer: [right]
Story: (story)
Answer:
Complex deduction rules: English kinship treats the maternal family and paternal family in the same way, while Chinese kinship distinguishes between them. The simplest example would be that in English, one's father's brother and mother's brother are both addressed as 'uncle', while in Chinese, they would be titled 'BoBo' and 'JiuJin', respectively.
It is nearly infeasible to build an ASP program that completely covers such a large label set with these complex rules. Therefore, we did not test ASP in our Chinese kinship experiments.
21

---

Published in Transactions on Machine Learning Research (01/2026)
Table 13: Few shot prompt template for CLUTRR.
Given story about kinship relations among persons, answer the relation between two queried persons. The answer could only be one following: (son, grandmother, daughter-ine -iughter-in-law, grandson, greatgrandson, grandfather, mother-in-laww, grealgranddaughher,, uncle, son-in-law, wife, grealgrandfather, brother, lusband, daughter, father- in-law, sister, grealgrandmother, granddaughter, aunt, nephew, niece, mother, father].
Story: Edd took his sister Marion out lunch after learning that she got accepted into her first choice for university. Washington bought to dress for his father Edd Washington and his uncle Bird went to the movies Sunday after church and got popcorn and candy while they were there. What should Marion address Bird? Answer: [brother]
Story: Ottilia asked her husband Friend if he could chop up some vegetables for dinner. Christine's mother Ottilia was teaching her how teach when Christine's husband Rollie arrived home. What should Friend address Rollie?
Answer: [son-in-law]
Story: May joined her husband Young, her son Miles and daughter-in-las Abbie for brunch last Sunday. May fixed her husband Young dinner and then they watched movie they rented. What should Young address Abbie?
Answer: (daughter-in-law]
Story: Leonard and his wife, Ella, went over to Genevieve's house for the weekend. Genevieve told her mother, Ella, that Rose would be over later. Leonard, Rose's father, was happy to hear this. Leila brought her grandmother, Genevieve, some muffins. What should Rose address Genevieve? Answer: [sister]
Story:
(story)
Answer:
Table 14:
Few shot prompt template for SPARTUN.
Given a story about spatial relations among objects, answer the relation between two queried objects step by step. The answer could only be one of following: [far, in, touch, has, covered_ by, right, overlap, front, behind, cover, left, disconnected from, below, above, near]. 'inside and touching' refers 'covered__by 'inside' and 'within' and 'inside' refers 'in'. 'contain' refers 'has?. If the sentence is describing clock-wise information, then 3 denotes right, denotes below, 9 denotes left, and 12 denotes above. If the sentence is describing cardinal directions, then north denotes above, east denotes right, south denotes below, and west denotes left. There could be multiple answers. Wrap all your answers in brackets. Example: [above, behind].
Story: A box called one covers a medium green apple. Covered by another box called two there is this box. Box two has medium orange apple which touches a yellow apple. Box two covers the yellow fruit. Where box regarding box one? Auswer: [cover]
Story: A midsize orange rectangle inside and touching a box named DDD. Above and in front of box DDD is another box named EEE. Box DDD is disconnected from and near to this box. A midsize orange rectangle is over and touches another midsize orange rectangle. Midsize orange rectangle number one within box EEE. Box EEE covers midsize orange rectangle number two. Where is DDD relative to midsize orange rectangle number two? Answer: [behind, below].
Story: A medium triangle, a big black square and big circle are in a block called AAA. The big black square is behind the big circle and in front of the medium triangle. In front of and touches a small black triangle there is this thing. Block AAA has the small black triangle. This block has a small blue square. Behind the medium triangle there is the small black triangle. Behind the big circle is the medium object. The small blue square is in front of the object which was in front of the medium thing. Under the big circle is this shape. What is the position the medium object regarding the small blue square? Answer: [behind].
Story: (story)
Answer:
22

---

Published in Transactions on Machine Learning Research (01/2026)
Table 15: Few shot prompt template for Chinese kinship.
You are given question about chinese kinship relations, please answer the question step by step. ansaers include but not limited to chinese kinship titles: (H (. /, & E. 2,, Wrap your final answer in square brackets []. If more than one relation is correct, separate the relations by comma, like: #111,
Question: A
MT /tw.:///wSS.......SSAAAAAAAAEEES
Answer:
||
T - 3 MB ( Uパ 20 ER, XrA
Question:
SI カ * M : X 75
Answer:
TI.
Question: 1
- - 1
Answer:
Question: {story)
Answer:
23

---

Published in Transactions on Machine Learning Research (01/2026)
Table 16: CoT and CoT-SC prompt template for StepGame.
Given story about spatial relations among objects, answer the relation between two queried objects step by step. The answer could only one of following: [top, bottom Mettom__eft, top_left, bottom, bottom siatto____hhh,,, top_ right, right, left, overlap). If sentence in the story is describing clock-wise information, then 12 denotes above, and 2 denote upper-right, denotes right, and 5 denote lower-right, denotes below, 7 and 8 denote lower-left, 9 denote left, 10 and 11 denote upper-left. If the sentence is describing cardinal directions, then north denotes above, cast denotes right, south denotes below, and west denotes left. Wrap your final answer in brackets. Example: [top]- A 'Reasoning chain' with respect to the question is provided help solve the question. Please take it as the important reference. If the Reasoning chain' is None, then extact the reasoning chain yourself.
Story: J is over there and D is on the top of it. S is upper right to W. J is directly south west of S. M is below P and to the right of C is sitting at the 3:00 position to D. A is diagonally above D to the left at a 45 degree angle. C is sitting al the 9:00 position of Y. S presents left to Y. J is on the right side to V. What is the relation of the agent to the agent S?
Answer: To determine the relation of agent A to agent S, let's analyze the story step by step:
1. **J is over there and D is on the top of it.**
** D is top of J.
2. **$ is upper right to W. W.**_, S is top right of W.
3. **J is directly south west of S.** - is bottom left of S.
4. **M is below P and to the right of p.** M is bottom right of P.
5. **C is sitting at the 3:00 position to D. ** C is right of D.
6. **A is diagonally above D to the left at 45 degree angle.** ** A is top_left left of D.
7. **C is sitting at the 9:00 position of Y.**.. C is left of Y.
8. **S presents left to Y.**.. S is left of Y.
9. **J is on the right side to V.** J is right of V.
Now, we need to find the relation between agent A and agent S:
From step 3, we know J is bottom left of S. From step 1, we know D is top of J. Therefore, D is top of J, and is bottom left S, so is to the left of S. - From step 6, we know top_ left D. left of D. we can determine that A is top left of S.
Since D is to the left of S, and A is Final answer: (top left]
Story: Object Y is below object X and to the left of it, too. H is to the right of M. Y is placed at the bottom of U. H is there and T is the right. J is directly below V. U is there and A is the right of it. U is over there and H is on the right. F is sitting in the left direction of H. M is positioned below Y. What is the relation of the agent X to the agent U?
Answer: To determine the relation of the agent X to the agent U, we need to analyze the spatial relations described in the story step by step:
1. 'OOject Y is below object X and to the left of it, too.' Y is bottom left of X.
2. 'H is to the right of M. H is right M.
3. "Y is placed at the bottom of U." Y is bottom of U.
4. 'H is over there and T is on the right." T is right of H.
5. "J is directly below V. - J is bottom of V.
6. "U is over there and A is on the right of it." A is right of U.
7. "U is over there and H is the right.' H is right of U.
8. "F' is sitting in the left direction of H.' F is left of H.
9. "M is positioned below Y.' M is bottom of Y.
Now, let's determine the relation of X to U:
From step 1, bottom left of X. From step 3, Y bottom of U. Since Y is bottom of U and bottom left of X, X must be top_ top_right of U. Therefore, the relation of the agent X to the agent U is [right].
Story: B is to the right of I. and on the same horizontal plane. M and I. are next to each other with on the right and M on the left. B is at the bottom and D is the top. J is to the top of W vertically. A is to the bottom-left of I. J is sitting at the top position to M. above J with a small gap between them. on the same horizontal plane directly right to E. E is on the right and W is the left. What is the relation of the agent L to the agent H?
Auswer: To determine the relation of the agent L to the agent H, let's break down the spatial relations step by step:
24

---

Published in Transactions on Machine Learning Research (01/2026)
Table 17: CoT and CoT-SC prompt template for StepGame. (Continue) 1. **B is to the right of L and is on the same horizontal plane. L is to the left of B. the right and M the left. ** M is the left of L. 3. **B is at the bottom and D is on the top.** B is below D.
2. **M and L are next to each other with
4. **J is the top of W vertically. ** J above W.
5. **A is to the bottom-left of I,** A is bottom left of I.
6. **J is sitting the top position to M.** ** J above M.
7. **H is above J with small gap between them. H is above J. to the left of 9. **E is on the right and W is on the left. W is to the left of E.
8. **B on the same horizontal plane directly right E.**
Now, let's find the relation between and
From steps 1 and 2, know that M is to the left of L, and L is to the left of B. From step 6, J is above M.
From step 7,
above J.
Since M is to the left of L, and J is above M, is also to the left of L but higher up. Since H is above J, H is even higher up and still to the left of L. Therefore, the relation of L to H is [bottom_righh
Story: H and K side by side with K the bottom and on the top. P is below K with small gap between them. U is there and 7 is at the 10 position of a clock face. Object A is above object M and to the right of it, horizontally. P and C are parallel, and G is above C. Q and E are next to each other with Q on the left and E on the right. The object O is positioned below and the right of the object J. the right of object J. Z is over there and N is on the left. Y is diagonally left and below L. If U is the center of clock face,
too.
the right
to the right of C. G and C are vertical and
above at o'clock. F and are both there with the object F to
located between 10 and 11. F is directly above W. P is directly north west of V. S is there and L is at the 10 position of a clock face. Q is positioned below D. N is to the bottom left of D. If A is the center of clock face, Q is located between 4 and 5. What is the relation of the agent E to the agent Z? Answer: To determine the spatial relation between agent E and agent Z, let's break down the information provided in the story step by step:
1. **Position of **** "Z is at the 10 position of clock face." If U is the center of clock face, 'G located between 10 and 11." 'Z is over there and N on the left. "S is there and L is at the 10 position of clock face.'
From this, we can infer that Z is at the 10 o'clock position relative to U. 2. **Position of E.*# - "Q and are next to each other with Q on the left and E on the right." 'E above
2 o'clock.' 'If A is the center of clock face, Q is located between 4 and 5."
From this, we can infer that at the 2 o'clock position relative to
3. **Relative Positions:** We know 7. is at the 10 o'clock position relative to U. We know E. is at the o ''clock position relative to
4. **Combining Information: Since Z is at the 10 o o'clock position relative to U, Z is in the upper-left direction. Since E is at the 2 o o'clock position relative to S, E is in the upper-right direction.
To find the relation of to Z, need to consider their relative positions: Z is at the 10 o'clock position (upper-left). E is at the 2 o'clock position (upper-right). Therefore, E is the **right** of Z. Final answer: [right]
Story: {input}
Answer:
25

---

Published in Transactions on Machine Learning Research (01/2026)
Table 18: CoT and CoT-SC prompt template for CLUTRR.
Given story about kinship relations among persons, answer the relation between two queried persons step by step. The answer could only be one of following: (aunt, son-in-law, grandfather, brother, sister, father, mother, grandmother, uncle, daughter-in-law, grandson, granddauhher, father-in-law, mother-in-law, nephew, daughter, miece, husband, wife, sister-in-law]. Wrap your final answer in brackets. Example: [grandfather].
Story: Edd took his sister Marion out to lunch after learning that she got accepted into her first choice for university. Washington bought to dress for his father Edd. Washington and his uncle Bird went to the movies Sunday after church and got popcorn and candy while they were there. How should Marion address Bird? Answer: To determine how Marion should address Bird, we need to carefully analyze the relationships mentioned in the story.
1. Edd look his sister Marion out to lunch. This tells us that Marion is Edd's sister.
2. Washinglon bought dress for his father Edd. This tells that Washington is Edd's son, making Edd Washington's father.
3. Washington and his uncle Bird went the movies. This tells us that Bird is Washington's uncle.
Since Bird is Washington's uncle and Washington is Edd's son, Bird must be Edd's brother. Therefore, Bird is Marion's brother, because Marion is Edd's sister.
So, Marion should address Bird as her brother. Final answer: [brother]
Story: Ottilia asked her husband Friend if he could chop up some vegetables for dinner. Christine's mother Ottilia was teaching her how to teach when Christine's hushand Rollie arrived home. How should Friend address Rollie?
Answer: To determine how Friend should address Rollie, we need to understand the relationships between the characters in the story.
1. Ottilia asked her husband Friend to chop vegetables. - Ottilia and Friend are married.
2. Christine's mother is Ottilia. Christine is Ottilia's daughter.
3. Christine's husband is Rollie. Rollie is Christine's husband.
Since Ottilia and Friend are married, and Ottilia is Christine's mother, Friend is Christine's father. Therefore,
Rollie, being Christine's s husband, is Friend's son-in-law.
So, Friend should address Rollie as his son-in-law. Final answer: [son-in-law]
Story: May joined her hushand Young, her Miles and daughter-in-law Abbie for brunch last Sunday. May fixed her husband Young dinner and then they watched a movie they rented. How should Young address Abbie? Answer: To determine how Young should address Abbie, need to establish their relationship based on the information given in the story.
1.
Identify the relationshhis: May is Young's wife. Miles is May and Young's son. Abbie is Miles' wife,
making her May and Young's daughter-in-law.
2. Since Abbie is married Miles, Young should address Abbie his danghter-in-law. Therefore, the appropriate way for Young to address Abbie is: [daughter-in-law]
wife, Ella, went over to Genevieve's house the weekend. Genevieve told mother, Ella, that Rose would be over later. Leonard, Rose's father, was happy to hear this. Leila brought her grandmother, Genevieve, some muffins. How should Rose address Genevieve?
Story: Leonard
Answer: To determine how Rose should address Genevieve, let's break down the relationships step by step: 1. Leonard and Ella are married. Genevieve is Ella's daughter, making Genevieve Leonard's daughter as well. 3. Rose is Leonard's daughter, which means Rose is also Ella's daughter. 4. Since Genevieve is Ella and Leonard's daughter, Genevieve is Rose's sister.
Given these relationships, Rose should address Genevieve as her sister.
Final answer: [sister]
Story: {story}
Answer:
26

---

Published in Transactions on Machine Learning Research (01/2026)
Table 19: CoT and CoT-SC prompt templates for SPARTUN.
Given a story about spatial relations among objects, answer the relation between two queried objects step by step. The answer could only be one of following: [far, in, touch, has, covered_ by, right, overlap, front, behind, cover, left, disconnected from, below, above, near]. 'inside and touching' refers 'covered by! 'inside' and 'within' and 'inside' refers 'in'. 'contain' refers 'has', If the sentence is describing clock-wise information, then 3 denotes right, denotes below, 9 denotes left, and denotes above. If the sentence describing cardinal directions, then north denotes above, east denotes right, south denotes below, and west denotes left. There could be multiple answers. Wrap your answers in brackets. Example: [above, behind].
Story: A box called one covers a medium green apple. Covered by another box called two there is this box. Box two has medium orange apple which touches yellow apple. Box two covers the yellow fruit. Where is box two regarding box one? Auswer: To determine the spatial relation between box two and box one, we need to analyze the given story step by step.
1. **First Sentence:** "A box called one covers medium green apple." - Box one is covering the green apple.
2. **Second Sentence:** 'Covered by another box called two there is this box." Box is covered by box two.
From these two sentences, can infer the following: Box one is covered by box two.
Therefore, the spatial relation between box two and box one that box two covering box one.
Final answer: [cover]
Story: A midsize orange rectangle is inside and touching a box named DDD. Above and in front of box DDD is another box named EEE. Box DDD is disconnected from and near to this box. A midsize orange rectangle is over and touches another midsize orange rectangle. Midsize orange rectangle number one is within box EEF. Box EEE covers midsize orange rectangle number two. Where is DDD relative to midsize orange rectangle number two? Answer: To determine the spatial relation between box DDD and midsize orange rectangle number two, let's break down the given story step by step:
1. 'A midsize orange rectangle inside and touching box named DDD.' -his indicates that there is midsize orange rectangle within box DDD and touching its sides.
2. 'Above and in front of box DDD is another box named EEE." Box EEE is positioned above and in front of
box DDD.
3. 'Box DDD disconeectdd from and near to this box." * Box DDD near but not touching box EEE.
4. "A midsize orange rectangle is over and touches another midsize orange rectangle." There are two midsize orange rectangles, and one positioned over and touching the other.
5. 'Midsize orange rectangle number one is within box EEF." The first midsize orange rectangle is inside box
EEE.
6. 'Box EEE covers midsize orange rectangle number two.' Box EEE is covering the second midsize orange rectangle.
From these statements, midsize orange rectangle is covered by box EEE, meaning it is inside or directly beneath box EEE. Since box FEF is above and in front of box DDD, and box EEE covers the second midsize orange rectangle, it implies that the second midsize orange rectangle is also above and in front of box DDD. Therefore, the relation between box DDD and midsize orange rectangle number two that DDD is behind and below the second midsize orange rectangle. Final answer: [behind, below].
infer the following: Box EEE is above and in front of box DDD. The second
Story: A medium triangle, a big black square and big circle are in a block called AAA. The big black square is behind the big circle and is in front of the medium triangle. In front of and touches small black triangle there is this thing. Block AAA has the small black triangle. This block has a small blue square. Behind the medium triangle there is the small black triangle. Behind the big circle is the medium object. The small blue square is in front of the object which was in front of the medium thing. Under the big circle is this shape. What is the position of the medium object regarding the small blue square?
Answer: To determine the position of the medium object regarding the small blue square, let's break down the story step by step:
1. **Ojjects in Block * Medium triangle Big black square Big circle Small black triangle Small
blue square
27

---

Published in Transactions on Machine Learning Research (01/2026)
Table 20: CoT and CoT-SC prompt templates for SPARTUN. (Continue)
2. **Relations** The big black square behind the big circle. The big black square front the medium triangle. The small black triangle is touched by something in front of it. The small black triangle is in Block AAA. The small blue square is Block AAA. The small black triangle is behind the medium triangle. The medium object (medium triangle) is behind the big circle. The small blue square is in front of the object which was in front of the medium object (medium triangle). Under the big circle is another shape (not directly relevant to query).
3. **eyy Relations for Query** The medium triangle is behind the big circle. The big black square is behind the big circle and in front of the medium triangle. The small blue square is in front of the object which in front of the medium triangle.
4. **Position of Medium Object** The medium triangle behind the big circle. The small blue square front of the object which was in front of the medium triangle. Since the big black square is in front of the medium triangle, the small blue square must be in front of the big black square. Given these relations, the medium triangle (medium object) is behind both the big black square and the small blue square.
Therefore, the position of the medium object regarding the small blue square is [behind].
Story: {story)
Answer:
28

---

Published in Transactions on Machine Learning Research (01/2026)
Table 21: CoT and CoT-SC prompt templates for Chinese kinship.
You are given a question about chinese kinship relations, please answer the question step by step. ansaers include but limited to chinese kinship titles: / ff LL, R 16 your final answer in square brackels [] If more than one relation is correct, separale the relations by comma, like: [ [
Wrap
Question:
mi W/ HN / H 5 - - FOR SER
# 10 E
/
Answer: # E #4 B
1. / 2. 21 3. W5 4.
M. - - DA /4
T :0 -17881355 1E
E, []T00/ Nh: ANT BE MEC
BR /
Question: # H 1 -
A
$11 T Æ. ク
TMKe 11 # - T
Answer: To determine how 1 might refer to 113, we need to establish the relationships step by step:
1. 1 has daughter named 1 2. 1 invited her 1² (sister-in-law) 1|B to the birthday party. 3. 1
husband is ///lt, 4. 1NI0p is 1>*k's son because NNH is referred to as 1 1 T. meaning 1NDit is / brother.
5. 1i is xJxllt'ss mother.
Since NNllp / son, 1/ must be Iek's wife. Therefore, 1 would refer to 1xf6 T..
Question: R
MATNE - 2 T
H + ( 22
Answer: determine what H| (Xiao Yu) would call //A (Xiao Min), we need understand the relationships described in the question:
1. ASZ (Xiao Meng) the mother MM (Xiao Yu). M husband 1 (Xiao Hua), making sJ the father of /MBB. 3. 1.4 mentioned his father </ (Xiao Min).
Thus, 1|18] (Xiao Min) is sJM (Xiao Yu's) paternal grandfather.
In Chinese kinship terms, paternal grandfather is called #1 or $9
Therefore, 1-B would call 1
[
Question: {story}
Answer:
29

---

Published in Transactions on Machine Learning Research (01/2026)
Table 22: LLM-ASP extraction prompt. template for StepGame. The prompt are a slightly modified version of the original so that all triplets and queries extracted at once.
Given a story, please parse each sentence into a fact. If the sentence is describing clock-wise information, then 12 denotes top, and 2 denote denote down__elt 9 denote left, 10 and 11 denote top_ left. If the sentence is describing cardinal directions, then north denotes top, east denotes right, south denotes down, and west denotes left. If the sentence is question, the fact starts with query. Otherwise, the fact starts with one of top, down, left, right, top left, gightthh, down Jeft and down right.
right, 3 denotes right, 4 and denote down right, 6 denotes down, 7 and 8
down_right,
top_left.
Story: If H is the center of a clock face, X is located between 4 and 5. V is directly north east of D. H and E are next to each other with on the left and on the right. What the relation of the agent H to the agent E? Semantic Parse: top leH' 'X'). top_riig rrghhihh"""" 'D"). left("H', 'E'). query("H" 'E").
Story: and P are parallel, and on the right of P. K is above I and to the right of I. B and P are parallel, and B is to the right of P. P is below J with small gap between them. T is below A at 7 o'clock. What is the relation of the agent to the agent B? Semantic Parse: right("I, 'P'). top right ("K', 'T'). right("B', 'P'). down("P? 'J"). down left("T', 'A"). query("T", 'B").
Story: Z is below S with small gap between them. The object M is positioned directly below the object J. A is the left side of and below M. Y presents upper right to N. B is positioned down and to the left of M. N is over there and C is on the right. W and A are parallel, and W on the left of A. S and D) are both there with the object is to the right of object D. W is at the bottom of D. Z at W's 9 o'clock. What is the relation of the agent A to the agent M? Semantic Parse: down("Z', 'S'). down("M) "MM, , "J"). down left("A", "M'). top right("Y", 'N"). down IeBt 'M'). right("C', 'N'). left 'A'). right("S", 'D"). down("W", 'D'). left 'W"). query("A", 'M').
Story: H and Y are in a horizontal line with H on the left. V is at the o'clock position relative to X. The object U is positioned below and to the right of the object W. R is diagonally left and below D. Z presents below I. Z is diagonally above P to the right at a 45 degree. Object P is above object and to the left of it, too. T is placed the top of V. N is positioned up and to the right of D. X is at Z'ss 6 o'clock. Y is over there and V is at the bottom it. What is the relation of the agent N to the agent X? Semantic Parse: left("H', 'Y"). down('V,, 'X"). down right("U', 'W"). down left("R", 'D'). down( 'z", 'T'). top right(ZZ) 'P"). top_ leftee", 'R"). top("T' 'V'). top_rri__ hop_right("") right("N', 'D"). down(" X", 'Z"). down ('V", 'Y"). query("N", 'X").
Story: H and K are side by side with K the bottom and on the top. P is below K with small gap between them. U is there and Z is at the 10 position of a clock face. Object A is above object M and to the right of it, too. D to the right of H horizontally. P and C are parallel, and P to the right of C. G and C are vertical and G is above C. Q and E are next to each other with Q on the left and E on the right. The object O is positioned below and the right of the object J. the right of object J. 7. is over there and N is on the left. Y is diagonally left and below L. If U is the center of clock face, is located between 10 and 11.
above S at 2 o'clock. F and are both there with the object F is to
there and L is at the 10 position of clock face. Q is positioned below D. N is to the bottom left of D. If A is the center of a clock face, Q is located between 4 and 5. What is the relation of the agent E to the agent Z? Semantic Parse: down("K' 'K", "H"). down 'K'). top left("Z', 'U"). top rightttt'''' 'M"). right("D', 'H"). right("P", "C"). lop("G", "C"). lef''"Q', 'E'). down righu("'' 'J"). top top_riggg_ right("E', "S'). right("F", 'J'). len''"N' 'Z"). down leYi("Y", 'L"). top_ left("",, 'U"). top("F", 'W"). top Je_eee'''' "V'). top lef'""), "S'). down("Q', 'D"). down Nown_leett"N leNt("NN, 'D"). down sown_right(""" 'A'). query("F', '7").
directly above W. P is directly north west of V.
Story: {story}
30

---

Published in Transactions on Machine Learning Research (01/2026)
Table 23: LLM-ASP relation extraction prompt template for CLUTRR. The prompt are a slightly modified version of the original so that all triplets and queries are extracted at once.
Given a story, extract alomic facts of the form relation("Person", "Person"). ). Example relations are: father, mother, parent, son, daughter, child, grandfather, grandmother, grandson, granddaughter, wife, husband, spouse, sibling, nephew, nicce, uncle, aunt, child in _a_,aw, and parent in law. Do not answer the query. Story: Edd took his sister Marion out to lunch after learning that she got accepted into her first choice for university. Washington bought. dress for his father Edd. Washington and his uncle Bird went to the movies Sunday after church and got popcorn and candy while they were there. What should Marion ad- dress Bird? Semantic Parse: sister( ("Edd,, 'Marion'). father("Washinng "Washington', 'Edd"). modee"Waahington" "Bird"). query("Marion)
'Bird").
Story: Michelle was excited for today, its her daughter's, Theresa, spring break. She will finally get to see her. Michael was busy and sent his wife, Marlene, instead. Kristen loved to care for her newborn child Ronald. Eric's son is Arthur. What should Theresa address Michelle? Semantic Parse: daugh- ter(" ter("Michelle', "Theresa'). MiichelMl,,el"', "Marlene"). child( ("risten""""" 'Ronald"). son(" 'Arthur'). query("Thereaa,, 'Michelle").
Story: Vernon was present in the delivery room when his daughter Raquel was born, but when his daughter Constance was born he was too sick. Vernon and his daughter Margaret went to the movies. Constance, Margaret's sister, had to stay home as she was sick. What should Raquel address Margaret? Semantic Parse: daughter(" ('Vento',, "Raquel"). dnughter("Vernon" 'Constance"). daughter("Vernon) 'Margaret"). sister(" (Maaerrrr""""""' 'Coustance"). query( ("Rqqel"", 'Margaret").
Story: Eric who is Carl's father grounded Carl after finding out what Carl had done at school. Ronald was busy planning a 90th birthday party for his aunt, Theresa. Eric and his son Carl went to the park and saw Eric's father Kyle there with his dog. What should Carl address Kylc? Semantic Parse: father ('Carl,, 'Eric"). aunt("Ronallle
"Theresa") )) son("Eric", 'Carl'). father("Eriee "Kyle). query query("Carll, ('Carl", 'Kyle"). Story: Shirley and Edward are siblings and best friends. They do everything together. Henry walked his daughters Amanda and Michelle to school. Kyle enjoys watching movies with his son's 's daugh- ter. Her name is Amanda. What should Kyle address Michelle? Semantic Parse: sibling("Shinn 'Shirley', 'Edward"). daughter(" ("Herry", "Amanda"). daughterr"Heurry. 'Michelle"). granckdaugherr(Kyyr ("Kyle,, "Amanda"), query("Kyle', "Michelle"). Story: Michael is taking his wife Henry out to dinner for their date tonight. Avis went with her grand- mother, Henry, to the grocery store to help her while she shopped. Alberta, who is the sister of Avis, is a lovely girl. What should Michael address Alberta? Semantic Parse: wife("Michael", 'Henry"). grand- mother("Aryi Avis", 'Henry"). sister(" ("Avis", 'Alberta"). query("Michael', "Alberta").
Story: Allen's father, Eric, bought him some ice cream. Karen was baking cookies for her grandson, Allen. Allen's brother Arthur came home from school, so she baked some extra for him, too. Eric's son, Arthur, was ill and needed to be picked up at school. Eric hurried to his side. What should Karen address Arthur? Semantic Parse: father(" Allen', 'Eric"). grandson(" "raarssnn"Karen', 'Allen'). brother(" Allen", 'Arthur'). (rriic, "Arthur'). query(" ('aree", "Arthur)..
Story: Karen was spending the weekend with her grandson, Eddie. Eddie's sister Michelle was supposed to come too, but she was busy and could n't make it. Theresa took her daughter, Michelle, out to High Tea yesterday afternoon. Eddie's mother Theresa baked brownies for dessert after they had dinner. What should Karen address Michelle? Semantic Parse: grandson( Grandoo("Kareen,, 'Karen', "Eddie"). ('Edde'',,,'e, "Michelle"). daughter("Thenn ("Theresa', "Michelle"). mother("Eddis "Theresa"). query Query("Karen) ("Karen,, "Michelle").
Story: (story)
31

---

Published in Transactions on Machine Learning Research (01/2026)
Table 24: LLM-ASP gender extraction prompt template for CLUTRR. The prompl template are a slightly modified version of the original so that all genders are extracted at once.
Given a story, extract atomic facts of the form male("Person) or demale("Person) )) for every person that appears in the sentences.
Story: Edd took his sister Marion out to lunch after learning that she got accepted into her first choice for university. Washington bought to dress for his father Edd. Washington and his uncle Bird went the movics Sunday after church and got popcorn and candy while they were there. What should Marion female("Marion")
address Bird? Semantic Parse: male("Edd)) Story: Michelle was excited for today, its her daughter's, Theresa, spring break. She will finally get to see her. Michael was busy and sent his wife, Marlene, instead. Kristen loved to care for her newborn child Ronald. Eric's son is Arthur. What should Theresa address Michelle? Semantic Parse: female(" "emale("Michenn Michelle'".
male("Washington).
"emale(MMarlenel)
male("Ronald") male("Fric"). male(" AAtthrr").
Story: Vernon was present in the delivery room when his daughter Raquel was born, but when his daughter Constance was born he was too sick. Vernon and his daughter Margaret went to the movies. Constance, Margaret's sister, had to stay home as she was sick. What should Raquel address Margaret? Semantic Parse: (veeee"Vrnnn").. female( female("Raquel",,
("osstl("o.t,..ee) female( "arrgartt).. Story: Eric who is Carl's father grounded Carl after finding out what Carl had done at school. Ronald was busy planning a 90th birthday party for his aunt, Theresa, Eric and his son Carl went to the park and saw Eric's father Kyle there with his dog. What should Carl address Kyle? Semantic Parse: male("Eric"). malo("Carl'). malc( "Ronald"). female("Theresa)) ). male( ("Kyle").
Story: Shirley and Edward are siblings and best friends. They do everything together. Henry walked his daughters Amanda and Michelle to school. Kyle enjoys watching movies with his son's 's daughter. Her name is Amanda. What should Kyle address Michelle? Semantic Parse: malo("Henry"), fomale(" Amanda"). female("Micnn Michelle'". malo("Kyle"). Story: Michael is taking his wife Henry out to dinner for their date tonight. Avis went with her grand- mother, Henry, to the grocery store to help her while she shopped. Alberta, who is the sister of Avis, is lovely girl. What should Michael address Alberta? Semantic Parse: male("Michaee female(" ""Hery""). femalo("Avis") female(' Alberta").
Story: Allen's father, Erie, bought him some ice cream. Karen was baking cookies for her grandson, Allen. Allen's brother Arthur came home from school, so she baked some extra for him, too. Fric's son, Arthur, was ill and needed to be picked up at school. Eric hurried to his side. What should Karen address Arthur? Semantic Parse: male(" ('Allenn).. male( ("Eric"). Gemale("Karen)
male(" Arthur').
Story: Karen was spending the weekend with her grandson, Eddie. Eddie's 's sister Michelle was supposed to come too, but she was busy and could n't make it. Theresa took her daughter, Michelle, out to High Tea yesterday afternoon. Eddie's mother Theresa baked brownies for dessert after they had dinner. What Domale("Michelly
should Karen address Michelle? Semantic Parse: femeee(Kaarnn'. male("Eddio) Gemale("Theresse
Story: {story}
Table 25: Extraction mistakes for CLUTRR dataset.
Type
Sentesse
Extracted 1960
Woing Rolation Wiing Souree
Elsi and Laruis disl the Fa lie dasghtter thomar at the maiiaad and his wilh Dellie one su lagpy sle cried
EaineGimmles, Insiaan. madic) [Dauecuaaees,, mother, Clrmae-lommno> Praud-ccalmoen., Ottilia clonaler) Cllennc-cunknora, children, miales
Magaard and häs son Dunn wan. to his mother Curine's Lome Daun rextied novel for Claistmas from his aunt Lou.
Whong Gender
Frsend's mother Ottila had to help him with his homework because having test Clatence has childrea, and one grandoon. The Grandions nome Toay
Wrong Belation
Table 26: Reasoning errors on CLUTRR dataset.
Ronsening
Whone senaoning
Nicolas is Clareson's spandau. Nivsin is June's soteer Sinoe Nocolas i Canuue's graddonn this meae Tune is Clecure's is danglter. Janob Loctaine's beand uow beiy loother, Claarece Roid oddros foos to his jaunsdild)
Rhone rddttin deductoo
Fai to specily the gradsoon Given informiio that locoh male
32

---

Published in Transactions on Machine Learning Research (01/2026)
Table 27: The full relation extraction prompt tion that demonstrates our prompting methods: Structured Prompts with Sectional Markup
template used for the StepGame extrac
Structured Output with Syntactic Delimiters
Predefined Output Categories
Decomposition of the Extraction Task
# Background information
Given story about spatial relations among agents and finally query asking about the relation between two agents. Please extract triplets encoding the relations between two agents as well as the query.
#
Predefined relations
Possible relations are: lop, down, left, right, top left, top right, down left, and down right.
#
Hints
If sentence the story is describing clock-wise information, then 12 denotes above, and denote top_ right, denotes right, 4 and 5 denote down sow__right, denotes down, and 8 denote
9 denote left, 10 and 11
denote top_let
If the sentence is describing cardinal directions, then north denotes top, east denotes right, south denotes down, and west denotes left.
Note that front means top; above and right means cop_right, below and left means down left, etc.
# Output format
Write each triplet
line. The triplet should be in the format: [A.ellatoon the query should be
in the format: A.B
i.e. you should use nothing but single letter represent an agent. Do not output
thinking process.
# EXAMPLE
STORY: H and K are side by side with the bottom and on the top.
P is below K with small gap between them.
U is there and
the 10 position
clock face.
What is the relation the agent E to the agent Z?
RELATIONSHIP
H,top.K
[
P,down,K
Z,top l
QUERY
E,Z
Please Gill in RELATIONSHIP and QUERY.
STORY:inpul
RELATIONSHIP FILL IN
QUERY FILL IN
33

---

Published in Transactions on Machine Learning Research (01/2026)
Table 28: The full relation extraction prompt tion that demonstrates our prompting methods:
template used for the CLUTRR extrac Structured Prompts with Sectional Markup
Structured Output with Syntactic Delimiters
Predefined Output Categories
Decomposition of the Extraction Task
# Placeholders in the triplets:
relation_ query kinship in question of the input.
label the gender of person by:
male
female
and
unknown
Explanation of sections
STORY: contains kinship keywords between the characters.
RELATIONSHIP: summarize the kinship relations with triplets, with every triplet represent the kinship of 2 female means "Terry's 's daughter is Mozella",
characters. For example,
Terry <
male ,daughter,Mozella.
"Mozella the daughter of Terry". You should label every character with male female
unknown if the gender uncertain.
QUERY: the final question about kinship, also represented by triplets. For example, if
the question asks about how should A male addresses female
the triplet
should
male rrelation _uerri__u_______ B < female >)
# Examples
#
Example
STORY: 'Edd took his sister Marion lunch after learning that she got accepted into her first choice for university. Washington bought to dress for his father Edd. Washington and his uncle Bird went to the movies Sunday after church and got popcorn and candy while they were there. What should Marion address Bird?'
RELATIONSHIP
Edd male sister, Marion female
Washington male
father, Edd male >)
Washington
male
Luncle, Bird male
QUERY
Marion female
relation query,Bird male
)]
#
#
Example 2
Please fill in the sections: RELATIONSHIP and QUERY of Example 3 below
# #
Example 3:
STORY:iinput'
RELATIONSHIP FILL IN
QUERY FILL IN
34

---

Published in Transactions on Machine Learning Research (01/2026)
Table 29: The relation extraction
prompt methods:
template
used for the SPARTUN dataset,
which demonstrates our prompting Structured Output with Syntactic Delimiters
Structured Prompts with Sectional Markup
Predefined Output Categories
Decomposition of the Extraction Task
A problem consists of story and question.
For story, please parse relations between entities list of triplets the format:
A.relation
For question, please parse the pair of entities asked the format: [[ A,
Possible relations are: [far, in, touch, has, covered by, right, overlap, front, behind, cover, left, disconnected from,
below, above, near]
If the sentence is describing clock-wise information, then 3 denotes right, 6 denotes below, denotes left, and 12 denotes above.
If the sentence is describing cardinal directions, then north denotes above, east denotes right, south denotes below, and west denotes left.
Write each triplet new line.
# EXAMPLE
STORY: A medium triangle, big black square and big circle are in block called AAA. The big black square is behind the big circle and is in front of the medium triangle.
RELATIONSHIP
medium triangle,in block AAA
big black square,in, AAA
big circle,in, block blockk AAA
big black qquare,behinai big circle
freont big black square, square,front, medium triangle
QUERY
medium srial blue square
Please fill in RELATIONSHIP and QUERY. 'input"
STORY:
RELATIONSHIP FILL IN
QUERY FILL IN
35

---

Published in Transactions on Machine Learning Research (01/2026)
The relation extraction prompt template used for the Chinese kinship dataset, which demonstrates our prompting methods:
Table 30:
Structured Prompts with Scctional Markup
Structured Output with Syntactic Delimiters
Predefined Output Categories
Decomposition of the Extraction Task
#
Placeholders in the triplets:
P1, P2, P3 and on: alias for the person appeared in the original imput.
person query: person in question of the input.
relation query: kinship in question of the input.
label the gender of person by:
male
female
and
unknown
label the age by:
older
younger
and
unknown
#
Examples
#
#
Example 1
ORIGINAL INPUT:
E. A 22 BAL NO AO ममा i
2 BS
RELATIONSHIP:
unknown
younger
female
>)
P1
unknown
younger
.P3
female
>)
unknown
2
younger .P4 male
QUERY:
P4
male
relation
query unknown
PP2
female
Example 2
Structure of examples
ORIGINAL INPUT:
contains information of the kinships between the people mentioned RELATIONSHIP summarize the kinships with triplets, with every triplet represent the kinship of 2 middle kinship element.
people. Include the relative seniority
example, P2 female
1 younger .P3 female
> means P3 33 P2, or P2 P2 younger than P3.
QUERY the final question about a kinship or a person, also represented by triplets. For example, if the
question asks about how should P2 male addresses P4 female and P2 is older than P4, the triplet
should be P2 male relatioon query older P4 female
You should label the relationships in sections RELATIONSHIP and QUERY with the relative age: older
means "is older than' and younger means "is younger than'. Note that there might be descriptions about the ages of the people their relative seniority, like (b meaning "/lV is younger than "(2".
Please the sections: RELATIONSHIP, and QUERY Example # Example 3:
ORIGINAL INPUT: 'input'
RELATIONSHIP FILL IN
QUERY FILL IN
36

---

Published in Transactions on Machine Learning Research (01/2026)
Table 31: PoT-LLM prompt template for StepGame.
Given a story about spatial relations among objects, answer the relation between two queried objects step by step. The answer could only be one of following: [top, bottom left, top_let bottom, bottom right, top top_right, right, left, overlap]. If sentence in the story describing clock-wise information, then 12 denotes above, 1 and denote upper-right, 3 denotes right, 4 and denote lower-right, 6 denotes below, 7 and 8 denote lower-left, 9 denote left, 10 and 11 denote upper-left. If the sentence is describing cardinal directions, then north denotes above, east denotes right, south denotes below, and west denotes left. Wrap your final answer in brackets. Example: [top]-
Story: A is the lop left of D, D is the top of is the bottom left of S. What is the relation of the A the S?
Answer: Let's break down the spatial relations step by step:
1. A is at the top left of D. - This means A is positioned at the top left relative to D.
2. D is at the top of J. This means is positioned above J.
3. J is at the bottom left of S. This means J is positioned at the bottom left relative to S.
Now, let's determine the relation of A S:
Since A is at the top left of D, and D is at the top of J, A is also above J but slightly to the left. Since J is at above (and slightly to the left), A will be positioned at the top left relative to Thus, the relation of the A to the S is lef]le.t].
the bottom left of S, and
Story: Y is at the bottom left of X, Y is at the bottom of U. What is the relation of the X to the U? Answer: To determine the relation of X to U, we need to follow the spatial relations step by step as described in the story.
1. Y is at the bottom left of X. This means X is at the top right of Y.
2. Y is at the bottom of U. This means U is at the top of Y.
Now, combining these two pieces of information: Since X is at the top right of Y and U is at the top of Y, we
need to determine the relation of X U.
From the spatial relations: If X is at the top right of Y and U is at the top of Y, then X is to the right of U.
Therefore, the relation X to is [right].
Story: I, is at the right of M, at the top of M, at the top of J. What the relation of the I. the H?
Answer: Let's break down the spatial relations step by step according to the story:
1. **L is at the right M**; This means is positioned to the right of M.
2. **J is at the top of M** This means J is positioned above M.
3. **H is at the top of J*x;- This means H is positioned above J.
Now, let's visualize the positions of each object:
M is our reference point.- L is to the right of M. - J is above M. H. is above J.
Since is directly above M, and H is directly above H is also directly above M.
Now, considering the position of T. (which is to the right of M) and H (which is above M), we can determine the relation between and
L is to the right of M. H. is above M. Therefore, L is positioned to the bottom-right. of H. The final answer is: [bottom__rde right).
Story: E is the right of Q, Q is at the bottom of D, N is at the bottom left of D, N is at the left of Z. What is the relation of the to the Z?
Answer: To determine the spatial relation of E to Z, let's analyze the story step by step:
1. **E is at the right of Q++ This means E is to the right of Q.
2. **Q is at the bottom of D** This means Q is below D.
3. *"N is at the bottom left of *** This means N is below and to the left of D.
4. **N at the left of z**; This means N is the left of Z.
37

---

Published in Transactions on Machine Learning Research (01/2026)
Table 32: PoT-LLM prompt template for StepGarne. (Continue)
Now let's combine these relations to find the position of E relative to 7:
Since Q is at the bottom of D, and E is to the right of Q, E is to the right and slightly below D. - N is at the is the left of Z, which means Z is the right of N. Since E is to the right of Q and Q is below D, E is to the right and slightly below D. Given that N is to left of Z, and is at bottom left D, must Combining all these, E is to the right of Q which is below D, and Z is to the right of N, which is at the bottom and slightly above N.
bottom left of D, which means
below and to the left of D.
the right and slightly above N.
left D. Therefore, E is to the right and slightly below D, while is to the right Thus, E is to the right and slightly below D, and Z is to the right of N. Since N is at the bottom left of D, is the right N, making Z to the right and slightly above N. Therefore, E is to the right of Z. Final answer: [right].
Story:
(extacted instance path]
Answer:
38

---

Published in Transactions on Machine Learning Research (01/2026)
Table 33: PoT-LLM prompt template for CLUTRR.
Given a story about kinship relations among persons, answer the relation between two queried persons step by step. The answer could only be of following: [aunt, son-in-law, grandfather, brother, sister, father, mother, grandmother, uncle, daughter-in-laww grandson, granddaughter, father-in-law, mother-in-law, nephew, son, danghter, niece, husband, wife, sister-in-law]. Wrap your final answer brackets. Example: [grandfather]. Story: Marion is Edd's sister, Edd is Washington's 's father, Bird is Washington's 's uncle. Marion is female, Edd is male, Washington is male, Bird is male. How should Marion address Bird?
Answer: To determine how Marion should address Bird, let's break down the relationships step by step:
1. Marion is Edd's sister. Edd Washington's father. 3. Washington is male. Bird is Washington's uncle. 5. Marion is female. 6. Edd is male. 7. Bird is male.
From the information given: Edd is Washinglon's father, making Washington Edd's son. Bird Washington's uncle, meaning Bird is either Edd's brother or Edd's brother-in-law.
Since Marion Edd's sister, and Bird is Washington's 's uncle, Bird must be Edd's brother (because uncles are typically brothers of child's parent). Therefore, Bird is Marion's brother as well. Marion should address Bird her brother. Final Answer: [brother]
Story: Friend is Ortilia's husband, Ortilia is Christine's mother, Rollie is Christine's husband. Friend is male, Ottilia female, Christine female, Rollie is male. How should Friend address Rollie?
Answer: To determine how Friend should address Rollie, let's break down the relationships step by step:
1. Friend is Ottilia's husband. Therefore, Friend is Outilia's spouse. Ottllia is Christine's mother. Therefore, Christine is the daughter of Ottilia and Friend. 3. Rollie is Christine's husband. Therefore, Rollie is Christine's spouse.
Now, need to find the relationship between Friend and Rollie: Since Christine is the daughter of Friend and Ottilia, and Rollie is Christine's husband, Rollie the son-in-law of Friend. Therefore, Friend should address Rollie his (son-in-law).
Story: Young May's husband, Abbie is May's daughter-in-law. May is female, Young is male, Abbic is female. How should Young address Abbie?
Answer: To determine how Young should address Abbie, let's break down the relationships step by step:
1. Young is May's husband. 2. May is female. 3. Young is male. 4. Abbie is May's daughter-in-law.
Since Abbie is May's daughter-in-law this means Abbie is married to May's son. Therefore, Abbie is also Young's daughter-ing law because Young is May's husband.
So, Young should address Abbie as his daughter-in-laww
Final answer: [daughter-in-law]
Story: Leonard is Rose's 's father, Ella is Leonard's wife, Ella is Genevieve's mother. Rose is female, Leonard is male, Ella is female, Genevieve is female. How should Rose address Genevieve? Answer: To determine the relationship between Rose and Genevieve, let's break down the information given in the story step by step:
1. Leonard is Rose's father. Ella is Leonard's wife. 3. Ella Genevieve's mother. 4. Rose is female. 5. Leonard is male. 6. Ella is female. 7. Genevieve is female.
From the above points, can deduce the following relationships: Since Leonard is Rose's father and Ella is Leonard's wife, Ella is Rose's mother. Ella is also Genevieve's mother, which means Rose and Genevieve share the same mother. Therefore, Rose and Genevieve are siblings.
Since both Rose and Genevieve are female, Rose should address Genevieve as her sister.
Final answer: [sister]
Story: (extracted instance path) Answer:
39

---

Published in Transactions on Machine Learning Research (01/2026)
Table 34: PoT-LLM prompt template for SPARTUN.
Given a story about spatial relations among objects, answer the relation between two queried objects step by step. The answer could only be one of following: [far, in, touch, has, covered by, right, overlap, front, behind, cover, left, disconnected from, below, above, nearl. 'inside and touching' refers 'covered by: 'inside' and 'within' and 'inside' refers 'in'. 'contain' refers 'hass. If the sentence is describing clock-wise information, then denotes right, 6 denotes below, 9 denotes left, and 12 denotes above. If the sentence is describing cardinal directions, then north denotes above, east denotes right, south denotes below, and west denotes left. There could be multiple answers. Wrap all your answers in brackets. Example: [above, behind]. A 'Reasoning chain' with respect the question is provided to help solve the question. Please take it as the important reference. If the 'Reasoning chain' is None, then extact the reasoning chain yourself.
Story: A box called one covers a medium green apple. Covered by another box called two there is this box. Box two has medium orange apple which touches a yellow apple. Box two covers the yellow fruit. Where is box two regarding box one?
Reasoning chain: box one is covered by box two.
Answer: determine the spatial relation between box two and box one, we need to analyze the given reasoning chain step by step.
1. Box one is covered by another box called two.
Therefore, box two covers box one.
The final answer: [cover]
Story: A midsize orange rectangle is inside and touching a box named DDD. Above and in front of box DDD is another box named EEE. Box DDD is disconnected from and near to this box. A midsize orange rectangle over and touches another midsize orange rectangle. Midsize orange rectangle number one is within box EEE, Box EEE covers midsize orange rectangle number two. Where is DDD relative midsize orange rectangle number two? Reasoning chain: box EEE is above and in front of box DDD, box EEE covers midsize orange rectangle number two.
Answer: To determine the relation between DDD and midsize orange rectangle number two, let's follow the reasoning chain step by step:
1. **Identify the position of box EEE relative to box DDD, The story states: "Above and in front of box DDD is another box named EEE." This means box EEE is positioned above and in front of box DDD.
2. **Identify the position of midsize orange rectangle number two relative box EEE;** The story states: 'Box REE covers midsize orange rectangle number two." This means midsize orange rectangle number two is within under box FEF.
3. **Determine the position of DDD relative to midsize orange rectangle number two:** - Since box EEE is above and front of box DDD, and box EEE covers midsize orange rectangle number two, it implies that midsize orange rectangle number two is above and in front of box DDD.
Therefore, the relation between DDD and midsize orange rectangle number two is that DDD is behind and below midsize orange rectangle number two. Final answer: [behind, below]
Story: A medium triangle, a big black square and a big circle are in a block called AAA. The big black square is behind the big circle and is in front of the medium triangle. In front of and touches a small black triangle there is this thing. Block AAA has the small black triangle. This block has a small blue square. Behind the medium triangle there is the small black triangle. Behind the big circle is the medium object. The small blue square is in front of the object which was in front of the medium thing. Under the big circle is this shape. What is the position of the medium object regarding the small blue square?
Reasoning chain: medium triangle is behind the big circle, small blue square is in front of big circle. Answer: To determine the position of the medium object regarding the small blue square, let's follow the reasoning chain step by step: 1.
The medium triangle is behind the big circle. 2. The small blue square is in front of the big circle. 3. Since the medium triangle is behind the big circle and the small blue square is front the big circle, the medium triangle must be behind the small blue square. Answer: [behind]
Story: {story)
Reasoning chain: (extracted instance path)
Answer:
40

---

Published in Transactions on Machine Learning Research (01/2026)
Table 35: PoT-LLM prompt template for Chinese kinship.
You are given a question about chinese kinship relations, please answer the question step by step. ansaers include but not limited to chinese kinship titles: /M//2//f / /E ,, Nt, L
2, R 2,, Wrap your final answer in square brackets []. If more than one relation is correct, separate the relations by comma, like: [
A "Resooiig chain' with respect to the question is provided to help solve the question. Please take it as the important reference. If the Reasoning chain' is None, then extact the reasoning chain yourself.
Question: A F aed B3 BETT x + IN ME J RT / JC in (33 -i E D & M KTT 1
1i / / fi,
Ste #
# Hes Li ... /ND HE Ef
Reasoning chain: y)
1 # 1 NIKE
Answer: To determine how A&E would address Nol let's break down the relationships step by step: 1. 1 has grandson named 1|8B 2. </^B]] has aunt ($H13) named 1-7E. 3. <J grandmother is A1. 4. M is a member of A>Jb's ancestral generation. 5. 1 has brother named //
Given that /0 is 1/t''s s grandmother, h must be 1tIt's mother mother-in-law. Since ANB is M paternal grendmother, 1s8 is A<l''s mother.
Thus, 1/00 being Ail's 's brother, would be s/yt's maternal uncle ()))). Therefore, 1/tt would refer to Nliiii as M]]
Question: K 2 E. #AIIE Mt, AT - A UI E AT t HR. DI T X4, AT M/, NNE
A, 1
Reasoning chain:
/ A
MM.
Answer: determine how 1 would address 144 let's verify step by step: 1
m 3.
11100 4.
I 144 5. 1
A AL iT HE
t.
- T|
Question:
A
/L *
m+ - 7
1e
Reasoning chhin: <
/ - #
Answer: To determine how 1 would address 1 let's analysis step by step: 1L N バ
/ 3. / 4. E1)
# KR 14 /
Answer: [
Question: {story} Reasoning chain: (extracted reasoning path)
Answer:
41

---

Published in Transactions on Machine Learning Research (01/2026)
Table 36: Target Ambiguities found in CLUTRR.
Story
Answers
Ellsworth played chess with his brother Nick. Ellsworth took his Tony to the park to feed the squirrels. Tony and his grandmother Daisic went to the science museum. They both had fun, and learned some things, dress Daisie?
mother-in-law, mother
What should Nick ad-
Hampton bought to dress for his father Chester Hampton and his sister Serena went out for ice cream. Serena bought her grandfather, Orville, tie for his birthday. Travis likes to visit his sister. Her name is Rachael. What should Chester address Orville?
father, father-in-law
Hessie's daughter Maymie went to grab dinner. Hessie's husband, Nicholas, it. Maymie made a cake for her grandfather, Eliza- beth. Nicholas went to lunch with his wife Hessie.
father,fatter-inn-le
not happy about
What should Nicholas address Elizabeth?
42


Reviews:
[{"review_id": "H5aLbykTIU", "reviewer": "Reviewer_1SGm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes path of thought a three-stage framework for relational reasoning that decomposes reasoning tasks into 1) graph extraction from natural language 2) path identification between query entities using depth-first search, and 3) reasoning over extracted paths. The authors evaluate PoT on four relational reasoning benchmarks using multiple LLMs.\n\n# Strengths\n- The explicit path identification stage is an interesting architectural contribution to reasoning\n- Comprehensive empirical evaluation across 4 datasets and multiple LLM backbones \n- Thorough prompt engineering documentation in appendix\n- Practical efficiency analysis (time and token usage)\n\n# Weaknesses\n- Some overclaims (see below)\n- Missing discussion and critical analysis\n- No ablation study to back up some claims\n- Single-trial runs means no confidence intervals or error bars", "claims_and_evidence_yn": "No", "claims_explanation": "1. The paper repeatedly claims PoT is \"task-agnostic\" and doesn't require domain-specific knowledge (e.g., Abstract: \"task-agnostic graph extraction\" Page 2: \"does not rely on any predefined rules or domain-specific symbolic formalization\"). However, when looking at the actual prompts (Tables 24-27, Appendix A.3) reveals that they actually have predefined (hard-coded) relations. These prompts encode substantial domain expertise and essentially the same inductive biases as symbolic rules, just expressed in natural language. I find the \"task-agnostic\" claim to be misleading. Note that here I am not talking about PoT vs. baseline prompts, I understand the baselines also receive similar domain knowledge, the concern here the \"task agnostic\" claim.\n2. It is not clear if path identification improves robustness. There would be a few ways to demonstrate it, perhaps the most straightforward is a simple ablation study of PoT with and without the path identification module. In the current results, the robustness experiments compare PoT-Symbolic vs LLM-ASP, which differ in both graph structure and path identification so we cannot isolate which component drives improvements\n3. I understand that cost is high so single runs are often the only option. However, they do weaken the significance of the results and some performance improvements seem marginal (possibly within noise). This might be an acceptable trade-off but there is no discussion of limitations or a discussion section at all, so the authors have not addressed this.\n4. The claim that this is the \"First work to deal with LLM hallucinations through path identification\" requires some nuance. Multiple prior work addresses LLM errors (self-consistency, self-correction, uncertainty quantification) and uses multiple reasoning paths (CoT-SC, Tree-of-Thoughts). What seems to be novel is the explicit path extraction via DFS on relational graphs for multi-hop reasoning. The paper should do more to identify what is genuinely new vs. what is building on prior work.", "audience_yn": "Yes", "audience_explanation": "The paper addresses an important problem (robust relational reasoning with LLMs) that is relevant to the TMLR community.", "requested_changes": "Critical \n1. Reframe \"task-agnostic\" claims in the paper i.e. the claims PoT doesn't require domain-specific knowledge, but prompts encode substantial domain expertise (predefined vocabularies, mapping rules, attribute schemas). \n2. Add a Discussion section (or provide critical analysis somewhere in the paper). Conclusion is only 11 lines and just restates contributions. There needs to analysis of things like: when does PoT not work? What are critical assumptions? What are the limitation of the current experimental set up? Boundary conditions? Why does path identification help? Under what conditions is it beneficial vs. harmful? How to apply to new domains? What engineering effort is required? When to choose PoT-LLM vs. PoT-Symbolic? \n3. Add ablation study for path identification to isolate the contribution of path identification vs. graph extraction alone\n\nStrengthen\n1. Repeat (at least some) of the results so we see variance and error bounds.\n2. Report distribution of path counts across datasets, correlation between path availability and performance, analysis of path redundancy\n3. Release implementation, code and non-public data", "broader_impact_concerns": "No significant concerns.", "additional_comments": "1. Page 1 states: \"Prior studies (Tolman, 1948; Whittington et al., 2022; Garvert et al., 2017) have shown that humans, unlike LLMs, can create 'cognitive maps'...\" None of these papers discuss LLMs. So I'm not sure where the \"unlike LLMs\" comment comes from, if it is the authors' addition it should be separated out to not appear as if this is a claim from those papers. \n2. Figure 3 appears within references section, this is strange placement. A simple latex formatting fix should address it.", "extra_scores": {}, "extra_text": {}}, {"review_id": "X6lKRRxv4c", "reviewer": "Reviewer_6UKo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Path-of-Thoughts, a framework designed to improve the multi-hop relational reasoning capabilities of LLMs. The core idea is to decompose the reasoning process into three distinct stages. First, a graph extraction module uses a single LLM call to parse a natural language story into a graph of entities and their relationships. Second, a path identification module finds the specific chain(s) of relationships in the graph that connect the entities mentioned in the query. Finally, a reasoning module takes this isolated path and infers the final answer. The authors propose two versions of this final stage: one using an LLM and another using a symbolic solver. The method is evaluated on four relational reasoning datasets and shows significant performance improvements over various prompting baselines and a neuro-symbolic baseline.\n\nStrengths:\n- The three-stage decomposition is logical and intuitive. The idea of separating the extraction of all facts from the identification of the relevant reasoning path is a intuitive way to handle the problem.\n- The robustness analysis is a strength of the paper. The experiments where noise is deliberately injected into the extracted graph provide evidence that PoT's approach of isolating the reasoning path makes the symbolic reasoner more resilient than the baseline LLM-ASP method.\n\nWeaknesses:\n- My main concern lies on the longer-term relevance of the research question studied. The paper's own results show a clear trend: the performance gap between the proposed method and simpler baselines narrows as the underlying LLM becomes more capable. Given that the experiments rely on models that are now over a year old (e.g., GPT-4o, released May 2024), it is difficult to assess whether the significant performance gains are a fundamental advantage of the PoT framework or an effective scaffold for a previous generation of models. It remains an open question whether this explicit decomposition is still required for the latest state-of-the-art LLMs, which may have overcome the specific reasoning limitations PoT aims to solve.\n- The novelty feels somewhat incremental to me. The general approach of using an LLM to extract a knowledge graph which is then passed to a symbolic reasoner is a well-established pattern in neuro-symbolic AI. The key innovation here is the intermediate path identification step.\n- The problem definition in Section 2 is purely descriptive and lacks formal, mathematical notation. This makes the task setup feel less rigorous and can create ambiguity. A formalized definition would provide a stronger theoretical grounding for the work.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper's central claim, PoT surpasses baselines, is well-supported by the results presented in the tables. However, the convincingness of this evidence is tempered by the timeliness of the models used.\n\nWhile the results are internally valid for the models tested, they do not provide fully convincing evidence that PoT offers an enduring advantage over simpler prompting strategies when applied to the current state-of-the-art. The claims would be substantially more compelling if the performance gap were shown to persist on more recent models.", "audience_yn": "Yes", "audience_explanation": "The proposed approach is simple to understand and implement, and the results are strong on established benchmarks, which makes it a solid contribution to the field. Researchers on multi-hop reasoning would be interested in this paper.", "requested_changes": "The most critical change is to evaluate on State-of-the-Art LLMs. The biggest question hanging over this work is its relevance in the face of ever-improving models. To demonstrate the continued need for this framework, I believe it is critical for the authors to re-run their key experiments using one or two of the most powerful, recently released LLMs like GPT-5 and Qwen3 family. If the performance gap between PoT and simpler baselines like CoT remains significant, it would strongly validate the paper's contribution. If the gap shrinks considerably, this is an important finding that should be discussed in the paper's limitations.\n\nThe following changes are needed to position the paper's contribution and relevance to the field:\n1. I strongly suggest that the authors revise Section 2 to include a more formal definition of the relational reasoning task. This would involve using notation to define the input space (stories, entities, relations), the output space, and the objective. This would improve the scientific rigor of the paper.\n2. Revise the novelty claim. The claim of being the \"first work to deal with LLM hallucinations and input ambiguities through path identification\" feels too narrow. \"Path identification\" is a term defined within the paper itself. I would recommend rephrasing this to situate the contribution more clearly within the existing literature on neuro-symbolic reasoning, focusing on how isolating query-relevant subgraphs improves robustness over prior methods that might feed the entire set of extracted facts to a solver.\n3. The current cost analysis is based on only 50 samples from a single, complex subset of one dataset. To make these results more generalizable, I would like to see this analysis expanded to cover at least one full test set. This is required to provide greater confidence in the efficiency claims.\n4. For clarity, I suggest replacing the term \"I/O prompting\" with a more standard name like \"Zero-shot\" or \"Vanilla Prompting\".\n5. The authors correctly note that writing the symbolic rule set for complex domains like Chinese kinship is infeasible. This is a very important practical limitation of the PoT-Symbolic approach. I think this point deserves more prominent discussion in the main body of the paper (perhaps in a dedicated limitations section) as it highlights the key trade-off between the PoT-LLM and PoT-Symbolic variants.", "broader_impact_concerns": "I do not have any broader impact concerns regarding this work. The research is foundational, aimed at improving the reasoning capabilities of AI models on abstract, synthetic tasks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y2SUU70yjw", "reviewer": "Reviewer_1wRc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Path-of-Thoughts (PoT), a novel neuro-symbolic framework designed to improve relational reasoning in large language models (LLMs). PoT decomposes complex reasoning tasks into three stages:\\\nGraph Extraction: A single LLM call extracts entities, relations, and attributes from natural language text to construct a task-agnostic graph.\\\nPath Identification: Relevant reasoning paths between queried entities are identified within the graph.\\\nReasoning: An external reasoner (LLM or symbolic solver) infers answers based on the identified paths.\\\nKey Strengths:\\\nNovelty: First work to explicitly mitigate LLM extraction errors via multi-path reasoning.\\\nEffectiveness: Strong empirical results across diverse datasets and model backbones.\\\nEfficiency: Low computational overhead despite two-stage LLM usage.\\\nGeneralizability: Applicable to both spatial and kinship reasoning, including challenging non-English contexts.\\\nKey Weaknesses:\\\nDependence on Symbolic Rules: For symbolic reasoning, performance depends on manually crafted domain rules, which may not scale to very large or complex relation sets.\\\nLimited Error Analysis: While robustness is demonstrated, deeper analysis of failure modes (e.g., path selection errors) is lacking.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. Extensive experiments on four diverse datasets with varying complexity.\\\n2. Comparisons against strong baselines (IO, Few-Shot, CoT, CoT-SC, LLM-ASP).\\\n3. Use of multiple LLM backbones (GPT-3.5, GPT-4, GPT-4o, Llama3, Claude-3.7).\\\n4. Robustness analysis against synthetic noise and extraction errors.\\\n5. Detailed prompt designs and ablation on extraction performance (Table 5).\\\n6. Computational cost analysis showing practical efficiency.\n\nThe results are clearly presented in tables and figures, and the methodology is thoroughly described.", "audience_yn": "Yes", "audience_explanation": "The paper introduces Path-of-Thoughts, which decomposes relational reasoning into graph extraction, path selection, and inference, and achieves strong results across benchmarks with robustness to extraction noise. These topics align with TMLR’s focus on LLM reasoning and neuro-symbolic methods.", "requested_changes": "Critical for Acceptance:\\\nError Analysis: Provide a qualitative analysis of failure cases (e.g., incorrect path selection, reasoning errors) to better understand limitations.\\\nSymbolic Rule Scalability: Discuss the limitations of symbolic reasoning in domains with very large relation sets (e.g., Chinese kinship) and propose or evaluate potential solutions (e.g., rule learning, hybrid approaches).\n\nWould Strengthen the Work:\\\nHuman Evaluation: Include human evaluation for extraction quality or reasoning correctness, especially for ambiguous cases.\nBroader Domains: Test PoT on at least one additional domain (e.g., temporal reasoning, social networks) to demonstrate broader applicability.\\\nBetter Task Allocation of LLMs: \\\nThe article mentions the application of LLM in reasoning, which can refer to the article \" HCTA:Hierarchical Cooperative Task Allocation in Multi-Agent Reinforcement Learning\". This article introduces a novel framework called Hierarchical Cooperative Task Allocation (HCTA) that is based on long-term behavior analysis.", "broader_impact_concerns": "The work presents a general framework for improving relational reasoning in LLMs and does not raise significant ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)