OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Models
Authors: Junda Wu
Xintong Li
Ruoyu Wang
Yu Xia
Yuxin Xiong
Jianing Wang
Tong Yu
Xiang Chen
Branislav Kveton
Lina Yao
Jingbo Shang
Julian McAuley
Venue: ICLR 2025 Poster (2025)
Keywords: chain-of-thought
large language models
offline policy evaluation
agentic
Primary area: foundation or frontier models, including LLMs

Abstract: Offline evaluation of LLMs is crucial in understanding their capacities, though current methods remain underexplored in existing research. In this work, we focus on the offline evaluation of the chain-of-thought capabilities and show how to optimize LLMs based on the proposed evaluation method. To enable offline feedback with rich knowledge and reasoning paths, we use knowledge graphs (KGs) (e.g., Wikidata5M) to provide feedback on the generated chain of thoughts. Due to the heterogeneity between LLM reasoning and KG structures, direct interaction and feedback from knowledge graphs on LLM behavior are challenging, as they require accurate entity linking and grounding of LLM-generated chains of thought in the KG. To address the above challenge, we propose an offline chain-of-thought evaluation framework, OCEAN, which models chain-of-thought reasoning in LLMs as a Markov Decision Process (MDP), and evaluate the policy’s alignment with KG preference modeling. To overcome the reasoning heterogeneity and grounding problems, we leverage on-policy KG exploration and reinforcement learning to model a KG policy that generates token-level likelihood distributions for LLM-generated chain-of-thought reasoning paths, simulating KG reasoning preference. Then we incorporate the knowledge-graph feedback on the validity and alignment of the generated reasoning paths into inverse propensity scores and propose KG-IPS estimator. Theoretically, we prove the unbiasedness of the proposed KG-IPS estimator and provide a lower bound on its variance. With the off-policy evaluated value function, we can directly enable off-policy optimization to further enhance chain-of-thought alignment. Our empirical study shows that OCEAN can be efficiently optimized for generating chain-of-thought reasoning paths with higher estimated values without affecting LLMs’ general abilities in downstream tasks or their internal knowledge.

Published as a conference paper at ICLR 2025
OCEAN: OFFLINE CHAIN-OF-THOUGH EVALUATION AND ALIGNMENT IN LARGE LANGUAGE MODELS
Junda Wu'; Xintong Li Ruoyu Wang", Yu Xia', Yuxin Xiong', JJaning Wang",
Tong Yu', Xiang Chen', Branislay Kveton', Lina Yao?., Jingbo Shang', Julian McAuley!
l UCC San Diego "Th University of New South Wales
East China Normal University *Adobe Research *CSIRO's Data61
[juw069, xi1240, www.eddu.aa. au lygwine gmail com
yxxioo.,.7xigg..... jshang, jmcauley @ucsd.
wangs) @ uunss. (ty xiangche, kveton) @ @adobe com lina. paoodaaaaaaaa. cww.me................m
ABSTRACT
Offline evaluation of LLMs is crucial in understanding their capacities, though current methods remain underexplored cus on the offline evaluation of the chain-of-thought capabilities and show how to optimize LLMs based on the proposed evaluation method. To enable offline feedback with rich knowledge and reasoning paths, we use knowledge graphs (KGs) (e.g., Wikidata5M) to provide feedback on the generated chain of thoughts. Due the heterogeneity between LLM reasoning and KG structures, direct inter- action and feedback from knowledge graphs on LLM behavior are challenging, as they require accurate entity linking and grounding of LLM -L--eenerated chains thought in the KG. To address the above challenge, we propose an offline chain-of- thought evaluation framework, OCEAN, which models chain-of-though reasoning in LLMs as a Markov Decision Process (MDP), and evaluate the policy's align- ment with KG preference modeling. To overcome the reasoning heterogeneity and leverage on-policy KG exploration and reinforcement
existing research. In this work, we fo-
grounding problems, learning to model a KG policy that generates token-level likelihood distributions for LLM-generated chain-of-thought reasoning paths, simulating KG reasoning preference. Then we incorporate the knowledge-graph feedback on the validity and aligmment of the generated reasoning paths into inverse propensity scores and propose KG-IPS estimator. Theoretically. we prove the unbiasedness of the pro- posed KG-IPS estimator and provide a lower bound on its variance. With the off-policy evaluated value function, we can directly enable off-policy optimiza- tion to further enhance shain-of-though alignment. Our empirical study shows that OCEAN can be efficiently optimized for generating chain-of-thought reason- ing paths with higher estimated values without affecting LLMs' general abilities in downstream tasks their internal knowledge.
INTRODUCTION
Offline policy evaluation aims to estimate target policy model's performance with only collected data, without requiring direct interactions between the target policy and realistic environments. Pre- vious offline evaluation methods focus on decision-making policies in recommender systems (Li ct al. 201T), healthcare (Bang & Robins 2005), and other scenarios where online experimentation
is costly (Thomas et al. 2015; Bhargava et al. 2024), risky, and impractical (Yu et al. 2021). Re- cent studies in LLMs leverage human feedback align models' behaviors with human preferences in single-turn generation (Ouyang et al. 2022 Rafailov et al., (024)) and multi-step reasoning tasks (Joshi et al. 2024). In addition, complicated LLM agentic frameworks, involving multi-agent col- laboration, orchestration, and cooperation, rely heavily on efficient (Roucher et al. 20225 Wu et al.)
2023ab, robust (Masterman al. 2024 Nguyen et al. 2024a), and proactive (Yao al. 2023 Xia
et al.
2025 Ma et al.) 2023) chain-ol of-tooogthhuhhh reasoning abilities, which need to be finetuned offline
"These authors contributed equally this work.

---

Published as a conference paper at ICLR 2025
(Putta et al. 2024 before deploying them online. Due to the high cost of deploying LLMs online and interacting with human feedback. Bhhrgava et al. (2024) further enables offline evaluation of LLMs from logged human feedback to align LLMs' response generation.
However, considering annotators may not have comprchensive knowledge in various types of knowl- edge backgrounds, human feedback chain-of-thougle reasoning (Joshi al. 2024) can be more challenging to collect. In addition, since making process, the volume of collected human feedback may increase exponentially. Due such challenges. conventional reinforcement learning from human feedback (RLHF) methods (Ouyang al. 2022 Bai et al. 2022a) can suffer from training inefficiencies and scalability issues.
reasoning involves a sequential decision-
Motivated by recent works in using knowledge graphs (KGs) as side information for prompt engi- neering (Wang et al. 2024cc Xia et al. (024b), self-correction (Zhao al. [2023] Wang al. 2023
Li al. 2024b Wu et al., 2024b), evaluating chain-of- -f-thughttt (Nguyen et al., 2024b), and model fine-tuning (Wang et |l.||2024bb [ Tang et al. (224)) we propose leveraging KGs as weak yet control- lable knowledge reasoners to effectively measure the alignment between LLMs' multi-step chain- of-thought reasoning and multi-hop KG trajectories by inverse propensity scores (IPS) (Joachims) et al. 2017). Unlike the chain-of-thought evaluation method (Nguyen et al. 2024b), which depends on accurate chain-of toain-of-thought grounding in specific KGs, we propose to verbalize KG trajectories and develop KG policy as a verbal reasoning mechanism over the graphs. This approach bridges the gap between KG and L.MM reasoning and generalizes the KG policy to various L.LMs.
To enable controllable chain-of-thought alignment in LLMs, we principally track LLMs' decision- making process in generating thain-of-thhoghh reasoning steps, by formulating the process as a Markov Decision Process (MDP) whose goal is to reach the correct final answer with minimal knowledge exploration and exploitation Lissandrini et al. (2020b a): Wu et al. (2024a). Then, we propose offline thain-of-though offthought evaluation and alignment, OCEAN, which evaluates the generated chain of thoughts from off-policy LLMs through collected offline data samples with feedback from a KG. The improved Knowledge Graph Inverse Propensity Scores (KG-IPS) approach considers the effects of feedback from the KG policy that aligns the model's chain-of-though generation and the behavior policy, which prevents model degeneration. We prove that the KG-IPS estimator provides an unbiased estimate of the target policy, with a lower bound for the variance, and establish confidence intervals using sub-Gaussian concentration inequalities. To enable direct optimization of LLM policies, we leverage the proposed KG-IPS policy evaluation approach for LLM fine-tuning by directly maximizing estimated policy values through gradient descent. Then we empirically evaluate the optimized LLM policy on three types of cfain-of-thought reasoning tasks, and demonstrate the effectiveness of the proposed policy optimization method, without affecting L.Ms' generalizability or generation quality. We summarize our contributions as follows:
We propose an offline evaluation framework, OCEAN, which bridges the heterogeneity be- tween LLM and KG reasoning, for effective evaluations of chain-of-thoughh
With the evaluation framework, we further develop a direct policy optimization method which enables efficient alignment with automatic feedback from the KG.
To facilitate the evaluation and optimization, we model the KG preference and derive feed- back by developing policy which verbalizes KG trajectories.
We provide a theoretical analysis of the unbiasedness and establish a lower bound for the variance of our KG-IPS estimator.
Through comprehensive experiments, we demonstrate OCEAN's effectiveness in aligning LLMs' chain-of-thought reasoning through direct optimization of the estimated policy value. OCEAN also achieves better performance on various downstream tasks without af- fecting LLMs' generalizability.
RELATED WORK
Offline Policy Evaluation Offline policy evaluation (OPE) is essential when online policy learning is risky and impractical (Levine et al. 2020). OPE has been applied to various practical applica- tions, including evaluating the recommender system's behavior with offline collected user feedback (Gilotte et al. 2018 Jeunen 2019). Recent work (Gao et al. 2024) also develops an OPE estima- for for LLM evaluation based on human feedback. Different from previous works, we study and
2

---

Published as a conference paper at ICLR 2025
formulate chain-of-thouu generation in LLM MDP and use knowledge graph reasoning as automatic feedback to develop a KG-IPS policy value estimator.
LLM Aligmment Reinforcement Learning from Human Feedback (RLHF) has been the dominant approach, optimizing LLMs using human-annotaie -anotated data to align model behavior with user prefer- ences (Quyang et al. 2022 Bai al. 2022a). DPO (Rafailov al. (024) and RRHF (Yuan et al., 2023) are proposed to reduce the training instability of RLHF. Wu et al. (2023b) utilizes varying densities of human feedback offer fine-grained rewards for RL finetuning, and [unn al. (2024a)
focuses on aligning LLMs with reward models driven by human-defined principles. To address RLHF's limitations such as heavy reliance on human input. alternative approaches like Reinforce- ment Learning from AI Feedback (RLAIF) (Bai et al., [022b; Lee et al., [223)] [Liu et al. 2023) and self-alignment methods (Sun et al.) 2024b) have been proposed, using Al-generated feedback to scale and automate alignment. Despite advancements, key challenge remains in aligning LLMs' internal knowledge with their reasoning, resulting flawed reasoning even after factual errors are corrected. Our approach focuses improving chain-of-thought alignment by modeling reasoning paths as an MDP and using KGs to ensure both factual accuracy and human-like reasoning.
Chain-of-the ofthouhtt Reasoning Chain-of-thought prompting has been widely applied to elicit the strong reasoning abilities of LLMs (Wei et al. 2022] Chu et al. 2023
Xia et al. (024a). By decom-
posing complex problem into a sequence of intermediate sub- tasks. LLMs can focus important
details and solve the problem step by step (Huang & Chang 2023 Yu et al.
2023). Despite the
remarkable performance improvements, recent studies have found that LLMS often generate
faithful chain-of- of-thought reasoning paths that contain factually incorrect rationales (Turpin et al., 2023 Lanham al., 2023). To address this, number of works leverage LLMs' self-evaluation abilities to verity and refine each reasoning step (Ling et al.] [2023] Madaan et al.] 2023). As the factual errors the generated chain-of th-thht may also be caused by the limited or outdated para- metric knowledge of LLMs, recent methods incorporate external knowledge sources to further edit unfaithful content in the reasoning path (Zhao al., 2023 Wang al. 2023 Li et al. 2024bb Wang et al., a)) While these methods focus knowledge augmentation and editing through prompts, our method, in comparison, directly aligns LLM internal knowledge with faithful and factual ffaiin-of-thoughh which avoids potential knowledge conflicts between parametric and non- parametric knowledge when generating reasoning paths.
3 PRELIMINARY
We first provide the formulation of shain-of-thouglee reasoning in LLMs as an MDP. Then we discuss conventional knowledge graph reasoning, as an alternative to free-form generation by verbalizing structured knowledge graph reasoning paths into natural language, which is more statistically trollable and generates faithful reasoning paths to the knowledge graph.
3.1 PROBLEM FORMULATION: CHAIN-OF-THOUGHE AS AN MDP
Given the prompt instruction q. chain-of-thought reasoning process in a causal language model ma includes the generation of trajectory of reasoning steps e (c1, c2,..., CT), before the final answer prediction
a ~ ao(. q: Cet) Cet = ((1,...,.+-1;; y~ m no c) II no mo(cela,
where each reasoning step & comprises a sequence of tokens and the number of reasoning step T is determined by the model's generation. Controllable chain-of-thougee generation can be challenging due its nature in autoregressive sequential sampling (Lin et al.) (020). which produces a high- dimensional action space in sampling reasoning step x (Celg) containing multiple tokens.
Chain-of-thougee reasoning can viewed as a Markov Decision Process (MDP) (Sutton 2018): starting with the instruction prompt q, the LLM sequentially decides and generates the next-step reasoning path e that navigates until it arrives at a target final answer y. Given the LLM policy ma at time step f, each state s, S comprises of the instruction prompt g and previously generated reasoning paths (-))==] The action space (1...., MVI in LLMss is sequence of N, tokens as
3

---

Published as a conference paper at ICLR 2025
knowledge graph entity or relation identified single thought, sampled from identical and finite vocabulary set V. The LLM policy Th samples next-step thought based on current state as de cu To(- |s)), which is sub-sequence in the reasoning path a, Ct identified the knowledge graph. The surrounding context & (a other than the knowledge graph entity or relation is deterministically generated by L.MMs. The transition in chain-of-thought is concatenating each reasoning path the current state as S(41 [se, ce). Then the reward function is to evaluate each thought given the state as Fr r(st, ct). Although such formulation of chain-oftthought enables direct LLM on-policy optimization via reinforcement learning, direct interaction with knowledge graphs to collect per- step reward in LLMs can be practically challenging and require a large effort of engineering due to the discrepancy between the unstructured generation of LLMs and structured knowledge graphs (Pan) al. 2024). Therefore, we propose to offline evaluate and optimize the target policy aligning with knowledge graph preference.
3.2
VERBALIZED KNOWLEDGE GRAPH REASONING
In contrast to chain-of-thought reasoning, conventional knowledge graph reasoning methods (Lin) et al. 2018 Saxena et al. 2020) sample entity-relation pair (re, ex) at step / from subset of the graph E, D) consisting of the outgoing edges of current entity 6t-1,
(r)) {(r', e')(ee)r),,,,
E g).
(1)
where the transition feasibility of the entity 6t-1 to all the outgoing edges is entirely determined by G. Knowledge graph reasoning starts with triplet (eu, (e,,,1,,1 61) and produces a chain of triplets h = (cu, 01..,,, Pr. er) by sampling from a policy 14,
(r), ~ i ((re, ex)leo: 71, C1,..., f,,,...,---,,,t- 1(-1,
where the goal of such knowledge graph exploration is to arrive at the correct answer entity at the end of the search step T. By knowledge graph exploration, we can collect a set of trajectories HI = -thu Jm= which are used to estimate a parametric probabilistic policy ps as a proxy to model the preference of the knowledge graph.
To align the action space between the knowledge graph preference policy p and the target policy TT, we leverage small language model as the backbone of ps and fine-tune the model on verbal- ized trajectories as natural language contexts. Inspired by existing efforts in verbalizing structured knowledge graphs into natural language query (Seyler et al., (017) and context (Agarwal et al. 2020 Wang et al.) (022a). we leverage the GPT-4 (Achiam et al.) (2033) model f to verbalize each chain triplets into a chain-of-thoughts e = [(h). The verbalized knowledge-grae trajectories are used to model knowledge graph preference in Section 4.2
4 OCEAN: OFFLINE CHAIN-OF-THOUGGT EVALUATION AND ALIGNMENT
We propose an offline evaluation of the chain-of ofaiin-ff-though generation process aligned with knowledge graph preference. The off off-policy estimator can be used for policy optimization that aligns LLMs with more faithful reasoning paths from knowledge graphs (Lin et al. 2023). We develop a small language model behavior policy that models the knowledge graph preference. In Figure 1 illustrate the workflow of our proposed framework OCEAN.
4.1 OFFLINE EVALUATION AND OPTIMIZATION
One of the most broadly used offline evaluation approaches is inverse propensity scores (Ionides) 2008 Dudik et al. (20TT), which has been used for LLM--ased offline policy evaluation for various purposes (Bhargava et al. 2024 Dhawan et al. 2024 Wu et al. 2022). Given the offline logged
chain-of-thought trajectories D D= [Ti) where
(s) c. rt (i) D we propose a KG-
IPS estimator considering two-folded weights of entity tokens in the knowledge graph preference policy He and of non-entity tokens in the base L.M policy mo
T
To((ss (0)
VKG-IPS(0)-
= N
T
log To (v)s(())
(3)
i=1
4

---

Published as a conference paper at ICLR 2025
Stop [Section 3.2]: Verbalized Knowledge Graph
Stop (Section 4.1 OCEAN): Offline Chain-of-thouge
Sampled Trajectories Reasoning
Evaluation and Optimization
(A Collection 1984-1900 1989, performed by. Jane Siberry
(Jnn Siberny, was born Toronto)
Target Policy (UMM)
Policy Gradient
(Torento, castle) (castic,
(Equation
Gratiest prosegation
Trajectory Verbalization
stop
performed Siberry lane Siberry born Toronto. The castle Toronto named Casa Loma.
KG IPS Estimator (Equation
(Pretrained
Step (Section 4.2): Knowledge Graph Preference Modeling
Policy Gradient
Preference Medel (Small
Preference Policy (Mmll
Optimization
An illustration of our framework OCEAN. We illustrate the framework in three steps. Step samples trajectories from the Wikidata5M knowledge graph and verbalizes the reasoning trajectories. With the collected trajectories, Step trains a knowledge graph preference modeling, which is fixed and used during -fthuuhf- evaluation and optimization Step 3 We highlight the learnable models in red and the frozen models in gray.
Figure 1:
where T, is the CoT steps for trajectory i; |ol²1 is the number of tokens within the reasoning step q. The propensity ratio in equation is To (u))s) (/\css") where X(uls{") = 1f E a; ( Mo (vls + 1f E cf vaf) ()) (() In this formulation, the two probabilities are combined to account for cases where sole (0) might be undefined those tokens are surrounding texts that cannot be identified on the knowledge graph. Specifically, we replace the undefined probability with the fallback probability uu s()). We follow (Zhang al. 2224)) to use the log likellhoodo score of each token in the base policy To as the reward function. Establishing the unbiasedness of the KG-IPS estimator is essential for reliable policy evaluation (Tiang & Li 2016 Bhargava al., 2024). We formalize this in the following lemma:
Lemma 1. The KG-IPS estimator provides unbiased estimate of the target policy To-
Intuitively, by re-weighting the token-level likelihoods from the target policy Te with the behavior policy Pé (for entity tokens) and the base policy Wo (for non-entity tokens), ensure that estimator compensates for the off-policy distribution, leading to an unbiased estimate of the true value function. The detailed proof is provided in Appendix [
The standard IPS estimator is known to have a high variance (Metelli et al., 2018) considering large
behavior discrepancies (Fe(v)s)") ( (vls())) between the behavior policy μs and the target policy Ty. In addition, by separately weighting the entity and non-entity tokens with Ps and To respectively, avoid the increasing variance accumulated from the long chain-of thought reasoning process and maintain the LLM's behaviors on non non-entity tokens without model degeneration. To further formalize our approach and illustrate the variance inherent in the KG-IPS estimator, we present the following Lemma, which provides a lower bound on the variance,
Lemma 2. The variance of the KG-IPS estimator is lower bounded by S2( ((M))) where M denotes the maximum value of the weighted terms, and n is the number of samples. For a target policy no.
let the true value function be defined V(0) : = na(e)ss)
[ Tt where rt. E [0, is the reward associated with selecting entity e in state 8t and Po is the behavior policy under which the data is collected. Applying the concentration inequality for sub- Gabusaassian variables, the KG-IPS estimator satisfies the following confidence interval with probability at least 1 1-
v(0) VKG-IP(0) -()) 50 (Molog(1/8)/m)
detailed analysis of the variance and confidence interval can be found in Appendix B
To further support our findings. demonstrate that the optimal policy for the final reward is con- sistent with the optimal policy for the entity-based knowledge graph reward, which means the non- entity-based LLMM reward can be considered as a regularization term that does not affect the optimal
5

---

Published as a conference paper at ICLR 2025
policy. See Appendix C for complete analysis. In the end, could directly optimize the target policy by maximizing the estimated value function through policy gradient,
0 - 0 V VoVKG-IPS(O).
(4)
4.2 KNOWLEDGE GRAPH PREFERENCE MODELING
To facilitate the evaluation and optimization, we model knowledge graph preference and derive feedback by developing the behavior policy Me which verbalizes knowledge-graph trajectories. Ran- domly sampled trajectories H from 9 in Section 3.2 contain samples that may not be transformed into a chain of thoughts leading consistent measurement (Wang et al., 2022b Manakul et al. (023), given sampled trajectory h and its verbalized chain of thoughts c, we prompt the GPT-4 model to propose a question q related to the first entity eg E h whose answer should be exactly the last entity eT h. and query the GPT-4 model with the proposed question,
reasonable question answering. Following conventional self-
f(gloo. er, 1 f(yl,c.. R(hlc) E fer -]],
where the reward of the trajectory is determined by the answer accuracy. We estimate the reward function R(h]e) c) as the normalized question-answering accuracy (detailed in Appendix (D). Then we fine-tune the preference policy pe directly via policy gradient optimization,
-1
VOJ(o) V
R(hx/ck) log uo(uk, Vk.ce),
where J(,) denotes the overall objective function representing the expected cumulative reward of the policy. Based on the distribution of relations (Figure[4h) 4b)) and entities (Figure[4c) in the sampled knowledge graph trajectories, we observe that the relation distribution relatively more skewed toward the most frequent relations. This suggests that the verbalized knowledge graph reasoning policy is likely to focus on more frequent reasoning transitions, potentially enhancing its ability to learn meaningful patterns. In contrast, the entity distribution shows relatively short tail, which may help mitigate the risk of overfitting to specific entities or knowledge biases.
5 EXPERIMENTS
In this section, we evaluate our proposed method, OCEAN, by conducting chain-of-thought align- ment on four LLM backbone models and evaluating several downstream tasks. We show our method's effectiveness in chain-of tfithuugtthh alignment and its generalizability in various tasks to understand (i) whether the proposed optimization approach sufficiently aligns LLMs' chain-of- thought behaviors with higher estimated values on multi-hop -answernn tasks and (iii) whether
tasks, (ii) how
the proposed method performs on chowledge-intensiee the post-alignment LLM generalizes on commonsense reasoning tasks.
5.1 IMPLEMENTATION DETAILS
Datasets. Following Zhang et al. we evaluate our approach on three aspects of question answering.
For knowledgge-ntenssin reasoning, we use datasets that require deep domain understanding. ARC
(Clark et al. 2018) tests advanced reasoning with grade-school science questions, PubMedQA (Jin
2019) assesses biomedical reasoning from abstracts, and SciQA (Auer et al.) (023) challenges models using the Open Research Knowledge Graph. For multi-hop reasoning, where models com- bine multiple sources, we use HotpotQA (Yang al. 2018) (reasoning across Wikipedia articles),
MuSiQue (Trivedi et al. (2022) (requiring 2-4 inference hops), and Strategy (Geva et al.) 2021) (testing implicit reasoning). For commonsense reasoning, we evaluate using three commonsense
benchmarks (CSQA (Talmor al., 2021), CSQA2 (Saha al. 2018), and CSQA-COT100N LLi
et al. 2024a ), along with OpenBookQA (Mihaylov et al.] 2018) and WinoGrande (Sakaguchi
al.
2021). These tasks test models' general commonsense
-antiionnnnwwringg... abilities.
Baselines. We experiment with four backbone LLMs: Gemma-2 (Team) 2024) with 2B model pa- rameters, Llama-3 (et@ 2024) with 8B model parameters, Phi-3.5-mini (Abdin et al. 2024)

---

Published as a conference paper at ICLR 2025
with 3.8B model parameters, and Mistral-0.2 (Jiang et al. 2023) with 7B model parameters. We use the instruction fine-tuned version of backbone LLMs for better instruction following abilities in question-amsweringg
use the CWQ question- answering dataset (Talmor& & Berant 2018) as the source data, in which the question-answering pairs developed from knowledge graphs. OCEAN only uses CWQ questions for the L.M generate chain-of-though reasoning paths, which are further aligned using the knowledge graph preference model, without directly supervised learning on the ground-truth answers. To compare with direct supervised learning, we also enable instruction-tundt totiing as a baseline (SFT), which is fine-tuned with the question as instruction and the answer as the response.
For thoouht alignment in OCEAN,
5.2 MULTI-HOP QUESTION ANSWERING
We evaluate the chain-of-thought reasoning performance of OCEAN compared with base LLMs and supervised fine-tuning (SFT), in three multi-hop question-answering tasks in Table [ Comparing SFT and Base LLMs, we observe similar knowledge inconsistency in knowledge-mnennt intensive tasks. Although SFT improves on MuSiQue with Gemma-2 and Mistral-0.2 backbones whose base mod- els' performance is relatively inferior this task, such knowledge-incoosisten problems result in worse performance on other downstream tasks.
HoipoiQA
MuSiQue
StralegyQA
Model
Method
w/ c/ctx (%)
wlo ctx
V(9)
ctx (%)
V(9)
w/ (%)
wlo (%)
V(())
Base
32,78
31.54
-10.35
11.59
-9.90
77.73
59.53
-9.25
L
(-24.56)
16.49 (-17.05)
22.28
1.80 (-9.9)
-17.09
(-11.21)
(-7.71)
OCEAN
33.38 (+166)
33.75 (+0.21)
-8.10
11.67 (+0.08)
-9.77
75.40 (-2.33)
59.83 (+0.3)
-5.53
Base
26.33
18.58
-31888
5.84
-26.41
76.71
60.99
-14.06
Gomma-2
SFT
29.75 (+3,42)
15.91 (-2,67)
-46.22
12.53 (+6. 69)
-40.5
64.77 (-11.94)
51.97 (-9.02)
-23.27
OCEAN
(-0.13)
19.70(++.11)
.26.43
(+1.03)
-22.15
74.24 (-2.47)
66.23 (+5.24)
.13.52
Base
32.13
26.14
.19449
11.85
-15.30
73.51
58.37
-13.87
Phi-3.5
SFT
21.99 (-10.14)
7,87 (-18.27)
-44.57
6.01 (-5.84)
.42.10
63.03 (-10.48)
50.95 (47.42)
-21.66
OCEAN
(+3.0)
26.23 (40.09)
-14.84
10.82 (-1,03)
-13.47
72.20 (-1.1)
57.64 (-0.73)
-12.25
Base
26.82
28.15
-19.08
5.67
-6.40
79.33
58.22
-11.36
Mistral-0.2
SFT
20.88 (-5.94)
14.49 (-13.64)
-18:53
7.73 (+2.06)
-12.24
52.40 (-26.93)
51.53 (-6.69)
-15.39
OCEAN
27.24 (+0.42)
27.54 (-0.59)
-3.12
5.15 (-0.52)
-5.94
17.29 (-2,04)
56.62 (-1.6)
-11.21
Table 1: Comparison results of OCEAN, base LLMs (Base), and supervised fine-tuning (SFT), on three Multi-hop Question-answering tasks. We report with context (w/ etx) and without con- text (w/o ctx) answer results with the Exact Match (EM) metric on HotpotQA and the Accuracy metric on Strategy Performance on MuSiQue dataset is EM with context. We also use each test/validation split for each dataset and report policy evaluation (()) results. We highlight the best- performed metric in bold font and the second-best underline for each task.
Since OCEAN is aligned to incorporate more shooledde-faithful ofriin-o--houghh reasoning patterns learned from knowledge graph reasoning policy without directly editing its internal knowledge, OCEAN maintains its generalizability in adapting to downstream tasks. We observe that OCEAN consistently improves on the policy estimated value ((0) through direct policy optimization pro- posed in equation 4 which demonstrates the effectiveness of the developed optimization method. Regarding the inesstin--nswerte accuracy, OCEAN improves base L.Mss, which achieves the best performance on HotpotQA and StrategyQA without context.
5.3 KNOWLEDGGF- QUESTION ANSWERING
To understand the effectiveness of OCEAN in knowledge-intensive question-answerigg tasks, we show performance comparison with base LLMMs (Base) and supervised fine-tuning (SFT) in Ta- ble Comparing SFT and Base LLMs, we observe that directly aligning knowledge graphs with LLMS may suffer from domain and knowledge inconsistency when downstream tasks require spe- cific domain knowledge, conflicting with the knowledge graph in the fine-tuning stage. We also observe that SET achieves 4.85% and 0.55% average improvements on the PubMedQA dataset, with and without context respectively, whereas it suffers from 29.60%, 8.35%, 13.6% average per- formance decreases on the remaining tasks. Such significant discrepancies in SFT's effects across different downstream tasks further show the risk in direct knowledge editing in L.MMs.
7

---

Published as a conference paper at ICLR 2025
ARC
PubMadQA
SciQA
Model
Method
w/o cix (98
((0)
w/ ctx (8)
ctx (9)
((0)
w/ cix (%6
w/o cox (96
(())
Base
79.93
-10.38
63,60
58.60
-25.40
83.10
57.10
-22.91
L
SFT
61.87 (-18.06)
-18.42
75.80 (+12.2)
58.00 (-0.6)
26.03
67.10 (-160)
35.80 (-21.3)
23.84
OCEAN
80.60 (+0.67)
-12.45
66.00 (+2.4)
59.80 (+1.2)
-9.37
83,20 (+0.3)
57.70 (+0.6)
-16.63
Base
65.89
15.36
24.61
Gemma-2
SFT
18.06 (-47.83)
-25.22
35 60 (+1.2)
21.00 (-19.6)
26.55
79,80 (+3.3)
$1.50 (++.4)
-3661
OCEAN
(-2.68)
(+10.2)
(+1.0)
-18.72
72.20 (443)
(+0.4)
Base
87.29
-7.86
70.40
41.80
28.48
83.50
58,90
-4444
Phi-3.5
SFT
65.22 (-22.07)
-9.02
62.40 (-8.0)
50.20 (+8.4)
-28.40
76.90 (-6.6)
43.80 (-15.1)
-14662
OCEAN
87.63 (+0.34)
-7.94
68.40 (:2.0)
47.60 (+5.8)
-11.45
84.70 (+1.2)
63.50 (+4.6)
-13.40
Base
73.91
-9.99
51.00
36.20
-13.01
78.50
58.00
-11.77
Mistral-0.
43.48 (:30.43)
65.60 (+14.0)
50.20 (+14.0)
-21.77
(-14.1)
35.50 (:22.5)
OCEAN
68.90 (-5.01)
-10.89
52.60 (+1.0)
33.20
-12.42
79.10 (40.6)
58.40 (+0.4)
-12.00
Table 2: Comparison results of OCEAN, base LLMs (Base), and supervised fine-tuning (SFT), on three Knowledge-intensivve Question-answerinn tasks. We report answers with context (w/ ctx) and without context (w/o ctx) on Exact Match (EM) metric on PubMedQQ and SciQA. The EM performance on ARC dataset without context. We also use the test/validation split for each dataset to report estimated policy values (0). We highlight the best metric in bold font for each task.
With the enhancement of OCEAN, question-answering accuracy of knowledge-intensive tasks gener- ally improved, while OCEAN fine-tuned LLMs achieving the best performance all three datasets, except for PubMedQA without context where SFT achieves better performance due to knowledge transfer from knowledge graph dataset. We also observe consistent policy value improvement on PubMedQA and SciQA, where the original policy values of base L.LMs are relatively lower. For tasks like ARC, which does not require additional reference knowledge from context and reason- ing in an easier chain of thought, OCEAN still maintains comparable policy value to the base LLM, which demonstrates the robustness and generalizability of the proposed method.
5.4
COMMONSENSE REASONING
Model
Method
CSQA
CSQA-2
CSQA-COTI000
OpenBookQA
Winogrande
Average
Base
65.03
71.39
69.50
58.80
43.09
61.56
Lluma-3
SET
51.19 (-13.84) 84)
57,06 (-14.33)
49.00 (-20.5)
63.20 (44.4)
34.73 (-8.36)
51.04
OCEAN
65.03 (0.0)
68.60 (-2.79)
72.00 (+2.5)
60.40 (+1.6)
41.36 (-1.73)
61.48
Base
57.99
62.57
63.50
$1.80
49.64
57.10
Gemma-2
SFT
14.66 33)
65,80 (43.23)
15.00 (-48.5)
7.40 (-44.4)
50.04 (+0.4)
30.58
OCEAN
67.73 (+9.74)
63.56 (+0.99)
72.50 (+9.0)
$7.20 (+5.4)
50.12 (+0.48)
62.22
Base
68.55
64.70
72.50
72.40
50,51
65.73
Phi-3.5
SFT
69,94 (+1.39)
61.47 (-3.23)
72.00 (-0.5)
69.40 (-3.0)
50.51 (0.0)
64.66
OCEAN
69.62 (+1))7)
62.77 (-1.93)
73,50 (+1.0)
71.20 (-1.2)
50.12
65,44
Base
61.18
68.48
65.00
64.00
46.25
60.98
Mistral-0.2
SIT
35.87 (-25.31)
22.47 (-46.01)
33.00 (-32.0)
34,80 (-29.2)
29.12 1-172
31.05
OCEAN
63,80 (+2.62)
69.19 (+0.71)
67.00 (+2.0)
62.60 (-1.4)
46.49 (+0.24)
61.82
Table 3: Comparison results OCEAN, base LLMs (Base), and supervised fine-tuning (SFT), on five Commonsense Reasoning tasks. We report the Exact Match (EM) metric on these tasks and the average performance. We highlight the best method in bold font for each task and LLM.
Finally, demonstrate OCEAN's generalizability in preserving commonsense knowledge and pre- venting knowledge catastrophic forgetting (Luo et al. 2023 Wu et al. 2025). we evaluate OCEAN with base L.MSs (Base) and supervised fine-tuning (SFT) on five commonsense reasoning tasks in Table Since such tasks do not require external domain knowledge, we only evaluate the accuracy of the model's generated answers. We observe that directly applying supervised fine-tuning (SFT) using knowledge graphs significantly impacts large language models (LLMs), potentially leading to catastrophic forgetting of commonsense knowledge. especially for the backbone LLMs of Gemma- 2 and Mistral-0.2. In contrast, we show that OCEAN achieves robust performance commonsense reasoning by leveraging off- off-policy evaluation and optimization from knowledge graph's 's feedback. OCEAN manages maintain comparable performance of base LLMs (e.g., Llama-3 and Phi-3.5),
8

---

Published as a conference paper at ICLR 2025
which have strong zero-shot commmonsense reasoning abilities. In addition, observe that for base LLM with relatively lower performance (e.g., Gemma-2 and Mistral-O.2), OCEAN enables con- sistent improvements. Therefore, OCEAN serves robust off off-policy alignment paradigm to incor- porating knowledge graph reasoning without affecting the generalizability of pretrained L.Mss.
ANALYSIS
6.1 IN-CONTEXT LEARNING INSTRUCTION TUNING
We conduct further analysis to compare the performance of both the base model and proposed model in the scenarios In-Context Learning and instruction fine-tuning. Specifically, we conduct this analysis using the Gemma-2 and Phi-3.5 models across three benchmark datasets: SST2 (Socher et al. 2013) for sentiment classification, AgNews (Zhang et al. 2013) for topic classification, and the In-context Learning setup, we provide the model with a single example for each task in the prompt. For the Instruction tuning experiments, we apply LoRA (Hu et al. 2021) to the pre-trained model and fine-tune on each dataset for 10 epochs. Throughout these experiments, the rank parameter in LoRA is fixed at 16, and we set LoRA to 32 across all tasks. The results of the In-context Learning and Instruction Tuning are presented in Table 4 Overall, observe that the performance of the base model and proposed model is largely comparable across most scenarios, except in the AG News task with Gemma-2, where OCEAN demonstrates greater performance after instruction tuning.
Bool (Clark et 2019) for reading comprehension.
In-Context Learning
Instruction CaaructioonTunnnng
Model
Method
SST2
BoolQ
AG News
Avg.
SST2
BoolQ
AG News
Avg.
Base OCEAN
87.16
56.
16.47
53,25
96.21
69.03
47.03
70.76
Gemma-2
89.33
55.72
13.14
52.73
96.56
68.66
60.08
75.10
Base
41.28
60.06
31.89
44.41
96.44
68. 13
86.43
83.67
Phi-3.5
OCEAN
40.48
59.11
32.37
43.98
96.44
68.81
86.24
83.83
Performance Comparison of In-Context Learning and Instruction Tuning. All datasets consist of classification tasks or true/false questions, so accuracy is used to evaluate the performance. The performance of the base model and our proposed model is largely comparable.
Table 4:
6.2 EVALUATION OF GENERATION QUALITY POST ALIGNMENT
100
Bose
Base
Base
OCEAN
OCEAN
-1EV-4
80
Dyvinet-2
40
AltynScore
30
30
40
30
20
10
Ganna-2
Miana-0.2
Llamu-3
Mistu-022
Llamu-3
Mistul-22
PI6-3.5
(a) comparison Self-BLEU
(b) comparison Distinct-2
(c) comparison AlignScore
Figure 2: Comparison results of base LLMs and OCEAN on three evaluation metrics, Self-BLEU, Distinct-2, and AlignScore. Lower Belf-BLEU scores and higher Distinct-2 scores indicate better diversity of the generated text, while higher AlignScore indicates better faithfulness.
To further evaluate the generation quality of post-alignment LLMs, we use the Self-BLEU (Zhu et al., 2018) and Distinct-2 (Li et al. 2015) scores to evaluate the diversity of the generation, concerning the similarity between generated texts and the uniqueness generated 2-gram phrases respectively. In addition, AlignScore (Zha et al. (023) is used to evaluate the faithfulness of the generated answer given the question context. The results are presented in Figure 2 which show that post-an post-alignment LLMss achieve comparable or better performances in terms of generation diversity and faithfulness.

---

Published as a conference paper at ICLR 2025
This demonstrates that while OCEAN aligns chain-of-thoupee reasoning with KGs, maintain the text generation qualities of LLMs.
6.3 CASE STUDY
In the previous Section 5.2 and 5.3 we observe efficient chain-of-thought alignment with improve- ment on the estimated policy value (0). To further understand the effects of the alignment, we choose backbone LLMs, Gemma-2 and Llama-3, with significant improvements on (0)), and per- form sample analysis by comparing the outputs the base model and OCEAN on the same set of questions. Our findings demonstrate that the application of our method enhances the precision and conciseness the chain of thought in the generated responses. Some illustrative examples are provided in Figure 3 Specifically, in the first example, the base Llama-3 model incorrectly claims that singing is not primary action associated with playing the guitar, which leads to an erroneous solution to the question. In contrast, method enables the model recognize that singing is common activity when playing the guitar, while also understanding that making music serves as a broader term. In the second example, although both the base model and OCEAN on Gemma-2 pro- vide reasonable answers to the question, our model demonstrates more concise chain of thought, streamlining the reasoning process and arriving the solution with greater simplicity.
Q: What do people typically de while playing guitar! Choose the correct answer from the following cheices.
(A) cry (D) hear sounds (C) singing (D) anhritis (I) making
Inaccurate Information
Base Llama-3:
I. The question asks wha people typically do while playing guitar. .... 4. Playing the guitar typically involves plucking or strumming the strings produce sound, which form making music. The other options don't make the of playing the guitar: .... (C) Singing is an activity that can be done while playing guitar, but it's not the primary action people do while playing the instrument. Therefore, the correct answer is (E) making music.
Wrong
OCEAN Llama-3:
Accurate Information
1. The question asking about what people typically do while playing guitar, ..... 4, Singing common activity that people do while playing guitar, as common way accompany oneself while playing instrument. Making music broad term encompasses wide runge of activities, including playing the guitar.
Based on the above reasoning, correct answer is: C) singing
Correct Answer
Q: What is the height above or below sea level called?
Sample Answer: elevation
Base Gemma-2:
Redundant chain-of-thought
height something above below level? vertical distunce of an object from sea level. 3. What is the term for this vertical distance? This is the height above or below seu level. Final Answer: The beight above or below sea level called **allitude**.
1. What level? Sea is average level of
concise thinking process
OCEAN Gemmu-2:
Answer: Elevation, Reasoning: I. Elevation the height of place above or below sea level.
Figure 3: Sample comparison between the base model and OCEAN on Llama-3 and Gemma-2. Our method enables a more precise and concise chain of thought.
7 CONCLUSION
In this work, we propose OCEAN to address the challenge of offline chain-of-thought evaluation and optimization of LLMs. By modeling the knowledge-graph preference and deriving feedback by developing policy that verbalizes knowledge graph trajectories, we propose KG-IPS estimator to estimate policy values in the alignment of reasoning paths with knowledge graphs. Theoretically, we proved the unbiasedness of the KG-IPS estimator and provided lower bound on its variance. Em- pirically, our framework effectively optimizes thain-of-thoughe reasoning while maintaining LLMs' general downstream task performance, offering a promising solution for enhancing reasoning ca- pabilities in large language models. Our framework not only enhances chain-of-thought reasoning but can also offer a potential offline evaluation mechanism for agentic frameworks, enabling the safe assessment of autonomous decision makinn processes. Future work could explore integrating this approach into reinforcement learning and multi-agent systems to further validate its utility in complex, dynamic environments.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 230x:23330088744,, 2023.
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. Knowledge graph based syn- thetic corpus generation for knowledge-enhanced language model pre-training. arXiv preprint 11Xi:22010.12688, 2020.
Meta. Llama model card. 2024. URL httr::://gtthubb.comm//tettt
hima3/blob// n/MODEL, CARD. md
Sören Auer, Dante AC Barone, Cassiano Bartz, Eduardo G Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitri Pliukhin, Daniil Radyush, et al. The sciqa scientific question answering benchmark for scholarly knowledge. Scientific Reports, 13 (1):7240, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint wxXiv:224405880) 2022a. URL
wtwpiii.orr//ooo /teg//df/22 2204. 058 pd.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. URL https:
//arxiw.orr//off/22222 08073.
Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 6144)962-- 2005.
Aniruddha Bhargava, Lalit Jain, Branislav Kveton, Ge Liu, and Subhojyoti Mukherjee. Off-policy evaluation from logged human feedback. arXiv preprint arXiv:2406. /0030, 2024.
Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. A survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309. 204:22209.15402, 2023.
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1005 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini arXiv: 1803:05555055 2018.
Nikita Dhawan, Leonardo Cotta, Karen Ullrich, Rahul G Krishnan, and Chris Maddison. End-to-end causal effect estimation from unstructured natural language data. arXiv preprint arXiv:2407.07018, 2024.
Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint arXiv: 100.4401, 2011.
Qitong Gao, Ge Gao, Juneheng Dong, Vahid Tarokh, Min Chi, and Miroslav Pajic. Off-policy evaluation for human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoming strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
11

---

Published as a conference paper at ICLR 2025
Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. Offline a/b testing for recommender systems. In Proceedings of the Eleventh ACM International Conference Web Search and Data Mining, pp. 198-206, 2018.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0900 2021.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A sur- vey. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the As- sociation for Computational Linguisties: ACL 2023, pp. 1049-1065, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/2023 htt.5///w122223.mmmnsaaa...0.000 URL org/2033 f indings-aall..
Edward L Ionides. Truncated importance sampling. Journal of Computational and Graphical Statis- fics, 1722)125-311, 2008.
Olivier Jeunen. Revisiting offline evaluation for implicit-feedback recommender systems. In Pro- ceedings of the 13th ACM conference on recommender systems, pp. 596-600, 2019.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint /0000223 2023.
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International conference on machine learning, pp. 652-661. PMLR, 2016.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering, arXiv preprint arXiv:1909.06144 2019.
Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. Unbiased learning-to-rank with bi- ased feedback. In Proceedings of the tenth ACM international conference on web search and data mining, pp. 781-789, 2017.
Nitish Joshi, Koushik Kalyanaraman, Zhiting Hu, Kumar Chellapilla, He He, and Erran Li. Improv- ing multi-hop reasoning in lims by learning from rich human feedhack. 2024.
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Her- nandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of of-thought reasoning. arXiv preprint arXiv:2307.13700 2023.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprini arXiv:2309.00200 002677 2023. URL http://xxxvvorgggge/e/ 2309. 00w.0006...0 F
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint 1.Xii:20005.06433 2020.
Jiachun Li, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Daojian Zeng, Kang Liu, and Jun Zhao. Focus on your question! interpreting and mitigating toxic cot problems in commonsense reasoning. arXiv preprint arXiv:2402. 18344, 2024a.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity objective function for neural conversation models. arXiv preprint 10000:55 2015.
Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual- bandit-based news article recommendation algorithms. In Proceedings of the fourth ACM inter- national conference on Web search and data mining, pp. 297-306, 2011.
Xingxuan Li. Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Li- dong Bing. adapting over heterogeneous sources. In The Tweifth International Conference on Learning Rep- resentations, 2024b. URL http::////pppeeeeeeeeeeeeeeeeeee
Grounding large language models via dynamic knowledge
net/
12

---

Published as a conference paper at ICLR 2025
Chu Chu-Cheng Lin, Aaron Jaech, Xin Li, Matthew R Gormley, and Jason Eisner. Limitations of autore- gressive models and their alternatives. arXiv preprint arXiv:2010.11939, 2020.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. Multi-hop knowledge graph reasoning with reward shaping. arXiv preprint arXiv: I0568, 2018.
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. Ra-dit: Retrieval-angmenntd dual instruction tuning. Xrviv preprint arXiv:2310.01352, 2023.
Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. Deductive verification of chain-of-thought reasoning. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 36407-36433. Curran Associates, Inc., 2023. URL https:://prcest ngs neurips cc httppaparrrrrllssppapr///// fi 71le 7bba 1-33-ppper---e -
Matteo Lissandrini, Davide Mottin, Themis Palpanas, and Yannis Velegrakis. Graph-query sug- gestions for knowledge graph exploration. In Proceedings of The Web Conference 2020, pp- 2549-2555, 2020a.
Matteo Lissandrini, Torben Bach Pedersen, Katja Hose, and Davide Mottin. Knowledge graph exploration: Where are we and where are we going? ACM SIGWEB Newsletter, 2um0mr:))))): 1-8, 2020b.
Ruibo Liu, Ruixin Yang. Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and Soroush Vosoughi. Training socially aligned language models on simulated social interactions. arXiv preprint arXiv:2305.1660 2023.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, arXiv preprint arXiv:2308.08747, 2023.
Kaixin Ma. Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu. Laser: Llm agent with state-space exploration for web navigation. arXiv preprint (rXiv22309.0812. 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023. http:/ppp.e/ppppeeereeeeeeeeeeep www.ee/forrrmideee id= 3h0errrLL
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black- hallu- cination detection for generative large language models. arXiv preprint (.0.i::2333088999 2023.
Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey. ArXiv, abs/2404. 1584, 2024. wttp:oorrpsI::2211181....
URL httthhttp//ppp.. www.ntssmmmnnscccciaccec
Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy optimization via importance sampling. Advances in Neural Information Processing Systems, 31, 2018.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint (rXiv:8809.0288, 2018.
Dang Nguyen. Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al. Gui agents: A survey. arXiv preprint arXiv:2412. 13501,,
Minh- Vuong Nguyen, Linhao Luo, Fatemeh Shiri, Dinh Phung, Yuan-Fang Li, Thuy-Trang Vu, and Gholamreza Haffari. Direct evaluation of chain-of-thoue of-thuutgttth in multi-hop reasoning with knowl- edge graphs. arXiv preprint I 1199, 2024b.
13

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744, Curran Associates, Inc., cc cttps/pccpppppppppppppppepeeeeee wwwps//wpweeeeeeeeeaa htt.pp/.....5bb22444775555555110 coner pdf
2022. URL http:p/se/pproceednssseeeeeeeeee
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering, 2024.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.071900 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kau- nismäki. 'smolagents': smol library to build great agentic systems. https://githhb: 2025.
com wwwgiingaaceemmmll.a.a...
Amrita Saha, Vardaan Pahuja, Mitesh Khapra, Karthik Sankaranarayanan, and Sarath Chandar. Complex sequential question answering: Towards learning to converse over linked question an- swer pairs with knowledge graph. In Proceedings of the AAAI conference artificial intelli- gence, volume 32, 2018.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Apoorv Saxena, Aditay Tripathi, and Partha Talukdar. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 4498-4507, 2020.
Dominic Seyler, Mohamed Yahya, and Klaus Berberich. Knowledge questions from knowledge graphs. In Proceedings of the ACM SIGIR international conference on theory of information retrieval, pp. 11-18, 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Daniel Cox, Yiming Yang. and Chuang Gan. SALMON: Self-alignment with prineiple-following reward models. In The Twelfth International Conference on Learning Representations, 2024a. URL https: /ttp:://ppeeeeeieeeee net www..et/ooruun//ddxxoneeeeee
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from seratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024b.
Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
Alon Talmor and Jonathan Berant. The web as a knowledge-bert for answering complex questions.
arXiv preprint arXiv: 06643, 2018.
14

---

Published as a conference paper at ICLR 2025
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. CommonsenseQA 2.0: Exposing the limits of Al through gamification. In Thirty-fitth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
httt::t:: httt.httss////peeeeeet http///pneeeew..eeeeeeeeeoeeeeer net f=tp:/ffmmmmmmmmm/d10010
net.
Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. In Proceedings of the 47th In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 491-500, 2024.
Gemma Team. Gemma. 2024. doi: 10.347400KAAGLL////3300 URLhttps:/ /wwp:://wwwwaat wwwp://www.........../
Com/m m/ 2tm///33311 3301
Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-contidence off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-5555 554, 2022.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don't al- ways say what they think: Unfaithful explanations
Chain-of-thouge prompting, In A. Oh, T. Naumann, A. Globerson, K. Saenko. M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 74952-74965. Curran Associates, Inc., URL pttpps::/ppoceeeddggg.... neur neurips cc wwtpppapaaaleeeeeeeeeeeeeeeeeeee filess/paerr/023/
2023.
File
pdf
Jianing Wang, Wenkang Huang. Minghui Qiu, Qiuhui Shi, Hongbin Wang, Xiang Li, and Ming Gao. Knowledge prompting in pre-trained language model for natural language understanding. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 3164 3177. As- sociation for Computational Linguistics, 2022a. (RLpsttss://doll http://tt.orr./.ooo//oo/o wtt.://///.....18////.1 18653/v1/ 2022. wmnlp-mailamm.... 207
Jannng Wang, Qiushi Sun, Xiang Li, and Ming Gao. Boosting language models reasoning with -hannlof-knoweddge prompting. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), ACL 2024, Bangkok, Thailand, August 11-16. 2024, pp. 4958 4981. Association for Computational Linguistics. 2024a. URL https: 7/doi. ottp 0. wwt...1663vvv/e. www.:o22...lll1o........ 271
Jianing Wang. Junda Wu, Yupeng Hou, Yao Liu, Ming Gao, and Julian McAuley. Instructgraph: Boosting large language models via graph-centric instruction tuning and preference alignment. arXiv preprint wrXiv:2402.08782 2024b.
Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong, and Zhang Xiong. Knowledge-driven cot: Exploring faithful reasoning in Ilm for knowledge- intensive question answering. arXiv preprint arXiv:2308. 13259, 2023.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9: 17641944 2021.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang. Aakanksha Chowdh- ery, and Denny Zhou. Self- Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.1 171, 2022b.
Yu Wang, Nedim Lipka, Ryan A Rossi, Alexa Siu, Ruiyi Zhang, and Tyler Derr. Knowledge graph prompting for multi-document question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19206-19214, 2024c.
Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li. Xiaojian Ma, and Yitao Liang. Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation. arXiv preprint arXiv:2403.053. 0033400333333 13, 2024d.
15

---

Published as a conference paper at ICLR 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URL httpthtts///pppeeeeeeee forum? d= ViQ 1101MMSBB__ MeSB_J
net.
Junda Wu, Rui Wang. Tong Yu, Ruiyi Zhang, Handong Zhao, Shuai Li, Ricardo Henao, and Ani Nenkova. Context-aware information-thene theoretic causal de- be-biasiigg for interactive sequence label- ing. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2022.
Junda Wu, Cheng- Chun Chang, Tong Yu, Zhankui He, Jianing Wang. Yupeng Hou, and Julian McAuley. Coral: Collaborative retrieval retrieval-augmente -ugmented large language models improve long-tail recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3391-34011 2024a.
Junda Wu, Tong Yu, Xiang Chen, Haoliang Wang, Ryan Rossi, Sungchul Kim, Anup Rao, and Julian McAuley. Decot: Debiasing ohain-of-thought for knowledge-intenn intensive tasks in large language models via causal intervention. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 14073-14077 2024b.
Junda Wu, Yuxin Xiong, Xintong Li, Yu Xia, Ruoyu Wang, Yu Wang, Tong Yu, Sungchul Kim, Ryan A Rossi, Lina Yao, al. Mitigating visual knowledge forgetting in mllm instruction-tuning via modality-decooulley gradient descent. arXiv preprint drXiv:2502.117400 2025.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen 1lm applications via multi- agent conversation. arXiv preprint arXiv:2308.0811 2023a.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives bet- ter rewards for language model training. Advances in Neural Information Processing Sys- tems, 36, 2023b. URL Btsp:tttss://rceeddiin wwwp:inngnnnuuuiipp neurips wwppapprppppppppppppp.apaeee..ee Files/paper/ pdf
Yu Xia, Rui Wang, Xu Liu, Mingyan Li, Tong Yu, Xiang Chen, Julian McAuley, and Shuai Li. Beyond 22231118888888177333377770233322
A survey of chain-of-x paradigms for Ilms. arXiv preprint
arXiv:2404. 15676, 2024a.
Yu Xia, Junda Wu, Sungchul Kim, Tong Yu, Ryan Rossi, Haoliang Wang, and Julian McAuley. Knowledge-aware query expansion with large language models for textual and relational retrieval. arXiv preprint arXiv:2410. 13765, 2024b.
Yu Xia, Subhojyoti Mukherjee, Zhouhang Xie, Junda Wu, Xintong Li, Ryan Aponte, Hanjia Lyu, Joe Barrow, Hongjie Chen, Franck Demoncourt, et al. From selection to generation: A survey of llm-based active learning. urXiv preprint arXiv:2502.1 1767, 2025.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio. William W Cohen, Ruslan Salakhutdinov, and Christopher D) Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: 1809.8800000 2018
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. Towards better thain-offthougee prompting strategies: A survey. arXiv preprint 10.09993 2023.
16

---

Published as a conference paper at ICLR 2025
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank responses to align language models with human feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 1d== GGGCGC4441
htt.:/www.ee..trrteoeeeeeee forum?
Yuheng Zha, Yichi Yang. Ruichen Li, and Zhiting Hu. Alignscore: Evaluating factual consistency with a unified alignment function. arXiv preprint arXiv:2305.16739, 2023.
Chen Zhang, Dading Chong, Feng Jiang, Chengguang Tang, Anningzhe Gao, Guohua Tang, and Haizhou Li. Aligning language models using follow-up likelihood as reward signal. arXiv preprint arXiv:2409.13946 2024.
Huiming Zhang and Song Xi Chen. Concentration inequalities for statistical inference. arXiv preprint arXiv:2011.02255, 2020.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
Zongmeng Zhang, Yufeng Shi, Jinhua Zhu, Wengang Zhou, Xiang Qi, Hougiang Li, et al. Trust- worthy alignment of retrieval-ause augmenttd large language models reinforcement learning. In Forty-first International Conference on Machine Learning.
Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. Verify-and ceify--and-diee A knowledge-enhanceed chain-of-thought framework. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5823-5840, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 18.55////v 1/2023. aong...o.o URL
320
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research development in information retrieval, pp. 1097-1100, 2018.
A MEAN ANALYSIS
To prove that the KG-IPS estimator unbiased, we need to demonstrate that the expected value of the IPS estimator equals the true expected reward under my.
Proof. The value function policy ma can be defined as:
V( = E (e (r(se, at)
[r(o( (6) .e]
E
T.
where r(s,, at) represent the reward obtained by taking action a, under state 5t.
Given that our value function consists of two cases: the first case considers the reward derived from the entity tokens under the knowledge graph preference policy (4a and the second case focuses on the reward derived from the non-entity tokens under the base LLM policy To- We separately prove the unbiasedness by showing that the expected value of either the entity-based or non-entity-baser estimators is equal the true expected reward under their respective policies.
17

---

Published as a conference paper at ICLR 2025
The expected value of the entity tokens in the knowledge graph is:
(o(c)) (6) (
E
[KKG(O)
1
log To(ls,
N
5)
Tompl (110
(emP(())
(1)
N
T2
To(e|
P(c = 015! ,c)
N
T:
Telo
(9)
(
(
N
Σ
E
[((e 5/ss (1) .)]
N 1=1
T.
(5)
en-ia (.16
1=1
[r(s(T,e) ((TT)
N
eeota (-169)
For non-entity tokens, the proof is similar:
N
T
(o(cls
E
Voase (0)
E )en
log To To(el (1)
N
To(els (1))
im
T
To(e|s (1)
1
E
P(e = 0115 c))
N
(8)
(11)())
To(els?
i=
2=1
T.
1
IP(e أ15 (9) e)] V
Tice
(8)
E
(1))
This completes the proof.
B VARIANCE ANALYSIS
In this section, we derive a confidence bound from the confidence interval and calculate the lower bound of the variance for the KG-IPS estimator.
pa(e[ss) ranging over all entity tokens This quantity M represents the largest discrepancy between the target policy my and the behavior policy Pag
Proof. Let M be the maximum value of mo(eles)
Since each reward r. lies in [0, 1). it 4-sub-Gaussian, Consequently, multiplying r, by a constant factor of at most M produces a random variable of the form
Te(e)s))
X₁
pp(E(s:)
that is (2ft) -as---uu-uussaat Let VKG-IPs (0) be the average of n i.i.d. samples (X,)== From the property Gabbsaassinn variables, if each X, is ( M1² J-sub-Gaussiii then the average
VRO.PS = Σ X
(
[M2 D--suuuGuuussa In particular, this implies that the variance cannot be smaller than 2 M2 4n
indicating an irreducible noise level of order M
For any Gaussian random variable X with variance o², the concentration inequality ZZhng & Chen 2020) holds:
x- E
s 2 log (
Plugging the variables above into the inequality, get the following bound for the KG-IPS estima- tor:
log 1/5)
ViG-IPS (0) v(0) < MM
O(M (og((1/)m))
2n
18

---

Published as a conference paper at ICLR 2025
with probability at least - 6.
The variance of Sub-Gaussiin random variable is close to its variance proxy, which means the lower bound on the variance of the KG-IPS estimator is S2( A²²) In addition, by standard concentration inequalities, we can get (M (log(1/))/n) confidence intervals on our estimator for policy ne. C
C THEORETICAL ANALYSIS
The value function of policy wo is defined as:
WT (st, a1) Ea (1s)) az)).
Based on our settings, we optimize the target policy for entity tokens aligning with knowledge graph preference. The reward function is formulated as:
(e(())))
KG (8t, (2) = Σ
log (e(())))
(5)
(a(e)s))
CER;
where Ao is the knowledge graph preference policy.
To reduce variance, the logged rewards for non-entity tokens under the base LLM policy mo incorporated as regularization term in the reward function, formulated as:
(u|s)) no (u(())
maa (st, ae)
Σ
log (v(ss))
(6)
WE Voz
where To is the base LLM policy. This helps mitigate disturbances, ensuring the LLM's behavior non-entity tokens remains stable and preventing model degeneration.
The final reward is:
r(st, a1) = (st,
+ + ( az),
Since both ( (t, ar) and -0 0e) are reweightings of the log-based reward log mo (v|s:). they belong to the same equivalence class. By leveraging Lemma 2 from DPO (Rafailov et al. 2024), we show that the optimal policy for the task-specific reward r aligns with the optimal policy for the final reward This implies that both rewards induce the same optimal policy.
1200
1000
Probubility
3
Freq
800
300
2-
600
200
400
100
200
0
0.0
0.2
0.4
0.8
Accuracy
Sampled Relations
Sampled Entities
(a) Distribution of QA Accuracy
(b) Distribution of Relations
(c) Distribution of Entities
Figure 4: Sampling distributions of (a) trajectories the knowledge graph that are verbalized as multi-step QA tasks and successfully answered by the LLM itself, (b) relations, and (c) entities in the knowledge graphs and their frequencies the appearance in the trajectories sampled from the Wikidata5M (Wang et al. 202T) knowledge graph.
D DETAILS OF KNOWLEDGE GRAPH PREFERENCE MODELING
The verbalized trajectories have in average 141.64 tokens with standard deviation of 34.39. The average trajectory length 4.79 steps with a standard deviation of 0.56. For each trajectory, there in average 5.79 entities with standard deviation of 0.56, and 4.11 relations with standard
19

---

Published as a conference paper at ICLR 2025
present the probability distribution of sampled trajectories, with respect to the number of correct answers generated per trajectory from ten differently sampled questions associated with each trajectory. Based such confsscnns measurement, we estimate the reward function R( as the normalized question-answeringg accuracy.
deviation of 0.90. In Figure 4a
Knowledge Graph Preference Model. The knowledge graph preference model is developed based on the pre-trained GPT2-Medium model (Radford et al. (201)). We collected 6K question-answerinn pairs from the Wikidata5M (Wang et al., 2021) knowledge graph based on the sampling strategy in Section 4.2 The sampled knowledge graph trajectories are composed into natural language prefixed by the corresponding questions by the GPT-4 model, which verbalizes the knowledge graph reason- ing trajectories and aligns with generative language models' behaviors. The model is then fine-tuned with base learning rate of le 4 for 10 epochs with a linear learning scheduler.
20


Reviews:
[{"review_id": "9F5qiEIRSu", "reviewer": "Reviewer_QYsv", "summary": "The paper explores a novel approach for enhancing the chain-of-thought (CoT) reasoning in large language models (LLMs) through an offline evaluation and optimization framework. The proposed OCEAN framework utilizes knowledge graphs (KGs) to provide feedback and improve alignment in CoT tasks by modeling reasoning paths as a Markov Decision Process (MDP). By employing a KG-based inverse propensity score (KG-IPS) estimator, the framework enables unbiased offline evaluation, bridging the gap between LLMs and structured KG reasoning to ensure consistent, accurate CoT reasoning without undermining LLMs' performance in general tasks.", "questions": "1. Could you elaborate on the specific challenges encountered in balancing entity and non-entity token weights in the KG-IPS estimator? How sensitive is the estimator to variations in these weights across different types of reasoning tasks?\n\n2. Since your framework relies on structured knowledge graphs, how does OCEAN perform on CoT reasoning tasks that are less fact-based and more abstract or conceptual? Have you considered extending the model to handle these types of reasoning, and what might be the challenges?\n\n3. Given the framework's reliance on multiple components (e.g., MDP modeling, KG-IPS estimator), what is the additional computational overhead compared to baseline CoT methods? Could you provide some insights into how to make OCEAN more computationally efficient?\n\n4. How well does the OCEAN framework adapt to different types of knowledge graphs (e.g., smaller or domain-specific KGs)? What, if any, modifications are necessary when using a KG that differs substantially in structure or domain from Wikidata5M?\n\n5. In some tasks, you noted the possibility of knowledge conflicts when aligning LLM outputs with knowledge graphs. How does OCEAN address or mitigate conflicts between an LLM's parametric knowledge and KG-based feedback, especially in cases where both knowledge sources might offer valid, but differing, information?\n\n6. The paper mentions using sub-Gaussian concentration inequalities to establish confidence intervals for the KG-IPS estimator. Could you explain why this choice was made and how these intervals compare to other possible statistical measures?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The concept of aligning CoT reasoning with knowledge graphs via a KG-IPS estimator is well-motivated, particularly in addressing the challenges associated with human feedback limitations and online deployment costs. The offline nature of the OCEAN framework allows for feasible and efficient model alignment using KGs as a structured feedback source.\n\n2. The paper provides a thorough theoretical foundation for the KG-IPS estimator, proving its unbiasedness and offering a variance lower bound. This adds robustness to the evaluation framework and offers a sound basis for its application in LLM alignment.\n\n3. The authors conduct a wide range of experiments across multi-hop question answering, knowledge-intensive tasks, and commonsense reasoning. These diverse evaluations demonstrate OCEAN's ability to align reasoning paths effectively while maintaining or enhancing the generalization capabilities of the LLMs across domains.", "weaknesses": "1. While the OCEAN framework is well-conceived, the reliance on multiple components (MDP formulation, KG-IPS estimator, verbalized KG policy) might limit its accessibility for practitioners. Simplifying or modularizing the system, potentially with more explicit guidance for implementation, could improve its applicability.\n\n2. Limited Scope in KG-Based Reasoning Tasks: Although the alignment with KGs enhances factual consistency, the paper does not deeply explore its limitations in more abstract CoT reasoning, where LLMs rely on conceptual, rather than fact-based, reasoning. This could be an area for future work, as abstract reasoning is a critical facet of human-like reasoning in LLMs.\n\n3. The paper focuses on large-scale LLMs, which are often resource-intensive to train and evaluate. While the experiments are comprehensive, further validation on smaller LLMs could strengthen the claim of OCEAN's adaptability across various scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xofAdgm7da", "reviewer": "Reviewer_wXum", "summary": "The paper proposes a novel technique for performing offline evaluation of the chain-of-thought capabilities of LLMs using trajectories generated on knowledge graphs. Specifically they propose a well thought out framework, coined OCEAN, which models CoT reasoning in language models as an MDP, where they leverage on-policy reinforcement learning methods to learn a knowledge graph policy which generates likelihood distributions for LLM reasoning paths. They then define and introduce the KG-IPS estimator to evaluate the policy, which they prove is unbiased while also providing a lower bound on its variance. They then discuss how this estimator can be used for policy optimization which would align LLMs to favor the reasoning paths generated in the knowledge graph. Finally, they conduct experiments to evaluate their approach in knowledge-intensive, multi-hop and common sense reasoning tasks compared to various baseline and supervised fine-tuned language models.", "questions": "Some questions were raised above. \n\n1. Why did you choose to 'verbalize' the trajectories generated on the knowledge graphs instead of inversely trying to reformat the LLM reasoning chain as a trajectory using entity and relationship linking? This approach seems initially more intuitive and less likely to cause errors?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper considers a practical approach for the offline evaluation of LLMs, which is an important area of research given that the dominant approach of RLHF is highly labor intensive and expensive.\n2. The notion of using knowledge graphs for LLM alignment is very interesting, and their approach consisting of verbalizing the KG trajectories is novel to the best of my knowledge.\n3. The paper contains very detailed and thorough experiments to demonstrate their results empirically.\n4. Theoretical proofs look sound.\n5. The paper is very well-written and enjoyable to read.", "weaknesses": "1. One aspect of the methodology that was not clear is the case where there are multiple trajectories that lead to the correct answer, which I believe can often be the case in reasoning tasks solved by LLMs. How will this be represented in the knowledge graph preference policy? Additionally, does the current methodology hold the potential to penalize reasoning paths that are correct but just not identified in a knowledge graph?\n\n2. In a similar vein, it is not clear that the knowledge graph used in this model (which seems to be Wikidata5M) is extensive enough to capture all possible reasoning paths given I believe it only contains entities and relations captured on wikipedia. I think it is imperative that the authors provide more information on the specific knowledge graph they are using and why it is sufficient to model reasoning queries given to LLMs.\n\n3. It is not clear from the tables provided in Section 5 that the OCEAN model significantly outperforms the baseline. A more detailed discussion explaining this seems necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1SaA2WhNrw", "reviewer": "Reviewer_hg5c", "summary": "This paper focuses on evaluating Chain-of-Thought (CoT) reasoning with guidance from a Knowledge Graph (KG). The authors propose training a KG model as a reference and using KG-IPS to assess the target language model's performance based on the reference model’s output. They conduct extensive experiments to support this approach.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, I find the paper’s idea and methodology compelling. The provided theoretical analysis is nice and the experimental results can support their statements.", "weaknesses": "However, improvements in writing would enhance readability and comprehension.\n\nPlease use the full name when introducing KG-IPS for the first time.\n\nWhat role does the base LLM policy $\\pi_0$ play? Including a figure to illustrate the entire pipeline—showing the knowledge graph reference policy $\\mu_\\phi$, target policy $\\pi_\\theta$, and base LLM policy $\\pi_0$—would improve clarity and help readers follow the workflow more easily. \n\nCould you expand on Equation 3, especially explaining the term $T_i | c_t^{(I)}$ and the intuition behind defining this term after the third summation symbol?\n\nWhat does the x-axis in Figures 1(b) and 1(c) represent? This clarification is essential to verify the conclusions in the final paragraph of Section 4.\n\nReaders might also wonder about the performance of the reference models and how the architecture of these models impacts the evaluation of the target model.\n\nFinally, how would the reference model perform if it included process-based reward signals rather than relying solely on binary outcome signals defined by answer accuracy? Is a binary signal sufficient, or would process-based feedback add value?\n\nIf the author can address my concern, I would like to raise my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f9yAUKDdVz", "reviewer": "Reviewer_uYKA", "summary": "Evaluating the chain-of-thought (CoT) reasoning of large language models (LLMs) is challenging due to the difficulty in gathering accurate labels. Since CoT reasoning involves a sequential decision-making process, using traditional feedback methods like reinforcement learning from human feedback (RLHF) becomes impractical. The author leverages knowledge graphs to address the challenge of evaluating LLMs CoT offline. The authors propose aligning LLMs' multi-step reasoning with multi-hop knowledge graph trajectories using inverse propensity scores (IPS). Specifically, the author proposes OCEAN, an estimator that evaluates a model's reasoning ability using CoTs from knowledge graphs. The authors demonstrated that their proposed offline evaluation method performed well across several reasoning tasks, including multi-hop question answering, knowledge-intensive question answering, and commonsense reasoning.", "questions": "- How does the proposed IPS approach compare to simply doing the direct method (DM) for evaluation?\n- What is the offline policy evaluation performance without performing updated according to equation 4? How does the offline policy evlauation change as you update the policy using equation 4?\n- How does SFT + normal COT perform?\n- How are you computing the value function estimate for SFT and the base model?\n- Did you start training OCEAN with the SFT model?\n- Did you also train SFT on the reasoning steps, or just on the questions and answers? If it was only on questions and answers, how does it perform when trained on reasoning steps similar to OCEAN?\n- How does the index k depend on the question q and answer y for the policy gradient?\n- How does SFT + normal CoT perform?\n- What does context mean in w/ctx and w/o ctx?\n\n[1] Rewarding Progress: Scaling Automated Process\nVerifiers for LLM Reasoning by Setlur et al.\n[2] Quiet-STaR: Language Models Can Teach Themselves to\nThink Before Speaking by Zelikman et al.\n[3] V-STaR: Training Verifiers for Self-Taught Reasoners by Hosseini et al.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors are focused on addressing a very important problem.\n- The authors performed experiments across three types of datasets and four different models.\n- The idea of evaluating multi-reasoning with a multi-hop knowlege graph is interesting.", "weaknesses": "- Reasoning only requires the model to generate sequences of thoughts with itself, which means we can optimize these thoughts online rather than offline [1,2,3].\n- The ability to evaluate a model of multi-step reasoning based only on a multi-hop knowledge graph is very limiting.\n- The paper lacks ablation experiments to clearly explain the performance gains of OCEAN, which come from the objective in equation 3 and training on GPT-4's verbalized structured chain of thoughts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1OclnjDt5t", "reviewer": "Reviewer_3w4e", "summary": "This paper introduces a novel approach for aligning LLMs with chain-of-thought reasoning by leveraging offline knowledge graph data. Specifically, a knowledge graph preference policy is first trained using reinforcement learning on question-answering data derived from verbalized exploration trajectories within the offline knowledge graph. This policy, which serves as a proxy for the knowledge graph’s preferences, is then used to fine-tune the target LLM policy through a novel KG-IPS estimator that assesses the difference between the proxy knowledge graph preference policy and the target policy. Extensive experiments are conducted to evaluate both the chain-of-thought reasoning alignment performance and the impact of this alignment on the base LLM's general capabilities.", "questions": "# Task formulation\n- Line 156, what is the \"surrounding context $c_t  \\backslash a_t$\"? Could you please elaborate on why $a_t$ \"is a sub-sequence in the reasoning path $a_t \\in c_t$,\"? Providing some intuitive examples may help improve understanding.\n- In line 169, \"the current edge $e_{t-1}$\" implies that $e$ represents edges in graphs. However, in line 172, \"the entity $e_{t-1}$\" implies that it represents entities. Could you please clarify the meaning of the notation $e$? The notation $e$ also appears in Equation 3. Please see related questions below.\n\n# Methodology\n- As most of readers would be not familiar with the Wikidata5M dataset, could you please present some cases of sampled trajectories $h$ from the knowledge graph $\\mathcal{G}$? In addition, some cases of the verbalizing transformation from $h$ to $c$ using GPT-4 would also be very interesting to see.\n- What are the transition dynamics in the exploration of the knowledge graph?\n- Could you please provide more detailed information about the resulting offline dataset used for knowledge graph preference modeling?\n- Could you please clarify whether the training of the knowledge graph preference policy is conducted in an online or offline manner?\n- About Equation 3:\n  - $e \\in c_t$ implies that $e$ is a sub-set of $c_t$. Could you please explain a little bit on the relation between these three notations: $e$, $a_t$, and $c_t$? Could we simply understand that $e$ represents a sequence of tokens?\n  - $c_t^{(i)}$ is outside of the summation $\\sum\\limits_{t=1}^{T_i}$. The subscript $t$ for this variable is unassigned. \n  - Should $e \\in c_t$ be $e \\in c_t^{(i)}$?\n- Line 221, is it correct that the reward function is defined as $r_t^{(i)} = \\log \\pi_0(e|s_t^{(i)})$ for the training of the target policy? If so, why we consider the reward in the token level, instead of the reward in the reasoning step level? Utilizing the reward function at the token level conflicts with the formulation in line 159, where $r_t = r(s_t, c_t)$. \n\n# Minor suggestions\n- Some background knowledge about the conventional inverse propensity scores (IPS) method could be interesting for readers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The research goal of aligning LLMs with chain-of-thought reasoning is both interesting and important.\n- The approach of using graphs as supplementary information to align LLMs with chain-of-thought reasoning is a promising avenue for exploration.\n- The paper is clearly presented and easy to follow, making it a pleasure to read overall.\n- The experiments are thorough and extensive.", "weaknesses": "Several key concepts and notations in the task formulation and methodology are not clearly defined, and certain details about the dataset are missing. Please refer to the questions for further details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)