Agent Planning with World Knowledge Model
Authors: Shuofei Qiao
Runnan Fang
Ningyu Zhang
Yuqi Zhu
Xiang Chen
Shumin Deng
Yong Jiang
Pengjun Xie
Fei Huang
Huajun Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: world knowledge model
agent planning
large language models
Primary area: natural_language_processing
TL;DR: We introduce parametric World Knowledge Model (WKM) which provides global prior task knowledge and local dynamic state knowledge to facilitate agent planning.

Abstract: Recent endeavors towards directly using large language models (LLMs) as agent models to execute interactive planning tasks have shown commendable results. Despite their achievements, however, they still struggle with brainless trial-and-error in global planning and generating hallucinatory actions in local planning due to their poor understanding of the "real" physical world. Imitating humans' mental world knowledge model which provides global prior knowledge before the task and maintains local dynamic knowledge during the task, in this paper, we introduce parametric World Knowledge Model (WKM) to facilitate agent planning. Concretely, we steer the agent model to self-synthesize knowledge from both expert and sampled trajectories. Then we develop WKM, providing prior task knowledge to guide the global planning and dynamic state knowledge to assist the local planning. Experimental results on three real-world simulated datasets with Mistral-7B, Gemma-7B, and Llama-3-8B demonstrate that our method can achieve superior performance compared to various strong baselines. Besides, we analyze to illustrate that our WKM can effectively alleviate the blind trial-and-error and hallucinatory action issues, providing strong support for the agent's understanding of the world. Other interesting findings include: 1) our instance-level task knowledge can generalize better to unseen tasks, 2) weak WKM can guide strong agent model planning, and 3) unified WKM training has promising potential for further development.

Agent Planning with World Knowledge Model
Shuofei Qiao Runnan Fang"; ; Ningyu ? Yuqi Zhu Xiang Chen Shumin Deng*, Yong Jiang Pengjun Xie Fei Huang Huajun Chen of
Zhejiang University * National University Singapore, NUS-NCS Joint Lab O Alibaba Group Zhejiang Key Laboratory of Big Data Intelligent Computing shuofei, zhangningyu @ ju.jedu. edu.....comm cn.
Abstract
Recent endeavors towards directly using large language models (LLMs) agent models to execute interactive planning tasks have shown commendable results. Despite their achievements, however, they still struggle with brainless trial-and- error in global planning and generating hallucinatory actions local planning due to their poor understanding of the "real" physical world. Imitating humans' mental world knowledge model which provides global prior knowledge before the task and maintains local dynamic knowledge during the task, in this paper, we introduce parametric World Knowledge Model (WKM) to facilitate agent planning. Concretely, steer the agent model to self-synthesize knowledge from both expert and sampled trajectories. Then we develop WKM, providing prior task knowledge guide the global planning and dynamic state knowledge to assist the local planning. Experimental results on three complex real-world simulated datasets with three state-of-the-ard open-source LLMs, Mistral-7B, Gemma-7B, and Llama-3-8B, demonstrate that our method can achieve superior performance compared to various strong baselines. Other interesting findings include: 1) our instance-level task knowledge can generalize better to unseen tasks, 2) weak WKM can guide strong agent model planning, and unified WKM training has promising potential for further development
Pgyntt
jridl-and-rror
path
first step
taak knowlolge
step
EDFFEct path
know bedgee
knew Seete
wprnt prode
8
world knowledge model
kallacinatuvyy
trajecturion
Figure 1: Traditional agent planning Agent planning with world knowledge model.
Introduction
The remarkable advances in Large Language Models (LLMss) have witnessed rapid development of various natural language processing tasks 225
16
28
47
60 33 Recently, multiple attempts that
Equal Contribution.
Corresponding Author.
'The code is available at /ttpcpps///ghhhuuccc//////
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

directly exploit LLMs as agent models to address physical world planning tasks have demonstrated
promising achievements [54 57 56
However, as most ofate-of-the-art LLMs are autoregressive models trained with next-token prediction, they lack the ability to essentially understand the real world, leading to generating hallucinatory actions and performing brainless trial-and-error in the environment as shown in Figure
38
44.
In contrast LLMs, humans possess mental knowledge model about the physical world II [8 17 30] When facing a specific task, they will first briefly rehearse the entire process in mind using their rich prior knowledge before performing mindless actions. We call this kind of knowledge global task knowledge (a.k.a. environment/tast commonsense). In addition, during the task procedure, the mental world knowledge model will constantly maintain a kind of local state knowledge, representing humans' cogmition of the current world state. For example, imagine you are in a room and your task to put a clean egg in mimmiccoonn crowave. The task knowledge may refer to The egg is most l likely in the fridge... ... The workfl flwss are: 1) 1 lccate and take the egg; 2) clean the egg using sinkbasin ... The state knowledge possibly refers to My task is to I have found and taked the egg ... Next I should... ... The absence of world knowledge can lead to blind trial-and-error in the early planning stages when environmental information is limited. Conversely, in later stages when information is redundant, it can easily result in confused cognition of the current world state and generate hallucinatory actions.
The process by which humans handle planning tasks reminds us to develop a parametric World Knowledge Model (WKM) facilitate agent planning. As humans typically acquire knowledge from expertise and practical experience, we build WKM based on knowledge learned from both expert and explored trajectories. Specifically, we first steer the agent model to synthesize task knowledge from the comparison between expert and sampled trajectories. Then we prompt it to summarize state knowledge for each planning step from expert trajectories and combine the previous and next actions to build state knowledge base. Lastly, we integrate the generated knowledge into expert trajectories and train a WKM. The agent model needs to be retrained to adapt to the task knowledge. Note our agent and knowledge model are both trained with LoRA 12 sharing the same backbone.
During the planning phase, we use the WKM provide global prior task knowledge and maintain local dynamic state knowledge for the agent model as shown in Figure [b). The task knowledge will be concatenated in natural language form following the specific task to guide the agent model's trial-and-error. At each planning step, to prevent the occurrence of hallucinatory actions, we utilize the generated state knowledge as the query to conduct kNN retrieval from the pre-built state knowledge base. We then use the constraints from the previous action, the probabilities of the retrieved next actions, and the probabilities from the agent model to make a weighted prediction for the next action.
We evaluate our method on three real-world simulated planning tasks: ALFWorld [1], WebShop [53], and ScienceWort [50) with three state-of-the-
spen-source LLMs: Mistral-7B [6]. Gemma- 7B [24), and Llama-3-8B (25). Empirical results demonstrate that our method achieves superior performance compared to various strong baselines on both seen and unseen tasks. Moreover, further analytical results show that 1) our WKM can effectively reduce blind trial-and-error and hallucinatory actions, 2) our goner-generated instance-level knowledge can generalize better to unseen tasks, 3) weak beak-guide-strong is feasible, 4) multi-task unified WKM possesses strong potential, and 5) explicit state knowledge will hurt the performance of agent planning.
2 Preliminaries
We mainly focus on interactive tasks with partial observations from environments. Following the task formulation in (44). the problem can be viewed Partially Observable Markov Decision Process (POMDP): (4, S, O, T). The instruction space U defines the task and its corresponding regulations. Si is the state space, O is the observation space, and A is the action space. T : S Sx.A A - S defines the transition function, which we assume to be given by the environments. is noticed that U, A. and O are subspaces of the natural language space in the language agent scenarios.
Based on the above, the historical trajectory hy that consists of list of actions and observations at time can be represented as:
Ct, or), (u, ag, 0o, di, O1,..., 0r
where u E l is the task instruction and a E A, o O are the action and the observation. Given a task, the language agent with parameter 0 serves the policy model na responsible for generating

---

Training Phase
Task: put two newspapers in drawer
Planning Phase
World Knowledge Model
Training
plasning wil
input output
around you armoharr cablnet
quackly
agent model
drower
Taak: put two newspapors drawer.
a. sota f...
know ledge medel
TT Knowledge:
Whon trying place muliple objects in drawece you should locale otjecte, then go drawer one at timc, place each object
stata lnsouredd boos
[Ht, St, 86-1)
Agent: go iroge
You midile soo.....
agant
State Knowledge taak
to You are checking
inicroonnne
Trusk Knowwedge::
8O
before closing
The action riflows are: Go the drawer
from fnnowledgg model
Agentt open
eop
In
The wookkfloow
Place one object ivian the drawer
From anviranment
Close the drivies
Repoat steps 2-4 for each object.
State Knoosedge
Knoutedge
@urrmonization
Synthesis
Agantt Mgls
Obs: sots you See orcoioardd
Pspert Trojectory
Task: put clean egg microwave
Sempled Trajectory
Creen in prestpape on a.
Mnvopapera drawerr You checking solu
siptato Knowledge: Your Taok
taleo
g+
Agent go troge
Agant: go to countartop 1 creditcard 2 dishsponge
Agent [ike Iopoppr from sols
Bhat
Neat
Agent open Indgee
lettuce
Obe The Vidge open, in
Obe On countertop 2 oos ceediiraed a pen a pan
mescapappraa
Gbe: You piik up the sevapaper from the anta
found
Knowledge: task
should find
State knowlddge will not app model during training and inferenee.
Agent put app Piln microsmoon
2. nesopsper I...
another
Rewentt
O
A.Agent: put revopppprr 2 iniion the drawerr
Roward 1.0
Figure 2: Overview of our WKM. We train a world knowledge model on the knowledge synthesized by the agent model itself from both expert and explored trajectories, providing prior task knowledge to guide global planning and dynamic state knowledge assist local planning.
the action a++1 based on ht at each time step t + 1:
a++1 (T( 1hc).
(2)
Specifically, au ~ TT(- (u) is generated according to the task instruction u. The whole trajectory concludes when the task is completed or exceeds the maximum time steps. Then the production of the entire trajectory with time length n can be modeled as:
To (T( = II no (ar+1 (r))l() 2=0
Ultimately, the final reward r(u, 7) E [0, 1] representing the task completion rate is calculated. Note that we follow a REACT-style [54] trajectory that includes rationales before each action. We use a to represent the action with rationales for convenience.
World Knowledge Model. World knowledge model serves as humans' mental cognition of the physical environment, more intricate than the word knowledge model which LLM-powered agent models are trained to be [0 52 [3]. Our "world" here refers to the simulated environment the task. Based on the static environment of the task and the dynamic changes during interaction with the agent, we define world knowledge as a combination of prior global knowledge and dynamic local knowledge, corresponding to the blind fial-and-eree problem in global planning and the hallucinatory action issue in local planning in traditional agent models, respectively. To attain precise and efficient agent planning, we develop parametric WKM to simulate the mental W WKM of humans.
3 Method
As shown in Figure 2 we steer the agent model to self-synthesize the task knowledge from the comparison of expert and sampled trajectories ($3.1). Then prompt the agent model to self- summarize the state knowledge based on historical behavior and construct a state knowledge base ($3.2). The generated knowledge will be integrated into the expert trajectories for training the WKM. After the training process ($3.3), we augment the agent model with the world knowledge model to achieve effective and accurate planning ($34))
3.1
Task Knowledge Synthesis
The task knowledge serves as the prior knowledge to guide the agent model's global planning and prevent it from dropping into blind trial-and-error.
3

---

Experienced Agent Exploration. We primarily acquire task knowledge through the comparison of preference trajectories (chosen vs. rejected). In order improve the quality of rejected trajectories and obtain more targeted task knowledge, we employ an experienced agent for exploration. Firstly, we train a vanilla language model with expert trajectories from the training set to obtain an experienced agent. Subsequently, the experienced agent explores the training set tasks again to generate rejected trajectories. Our purpose is to extract superior task knowledge that cannot be acquired solely through supervised fine-tuning on chosen trajectories, thus further effectively boosting the agent's 's capabilities.
Self Knowledge Synthesis. With the expert trajectories as the chosen ones and the trajectories sampled from the experienced agent as the rejected ones, we prompt the agent model itself to synthesize the task knowledge. Supposing IC is the task knowledge space:
To [od=||PaskKnow U, Twr Ti):
where & E K is the task knowledge, PTaskKnow stands for the prompt to instruct the task knowledge extraction, and task u, Tu and 77 always satisfy r(u, Tw) = 2 r(u, TT). Even when r(u, Tw) = r(u, n)), we still consider trajectories sampled from the experienced agent as rejected ones. This is because expert trajectories often have shorter step lengths, enabling the agent to learn more knowledge of efficient
are the chosen and rejected trajectories respectively. Note that given the same
planning, For detailed prompts of task knowledge synthesis, please refer to Appendix [.]
3.2 State Knowledge Summarization
The state knowledge serves the dynamic knowledge constrain the agent model's local planning and prevent it from generating hallucinatory actions. We prompt the agent model to self-summarize state knowledge at each planning step based on the expert trajectories detailed prompts of state knowledge summarization, please refer to Appendix [.2 Supposing the
guarantee quality. For
prompt used summarize state knowledge is DStateKnow and the state knowledge s E & is part of the state space S, the generation of state knowledge at time t can be represented as:
To ( PStateknow /14)-
(5)
State Knowledge Base Construction. To avoid confusion caused by excessive additional infor- mation, instead of explicitly concatenating the state knowledge to the context, construct state knowledge base for retrieval (we analyze in $4.3 how explicit state knowledge may affect the perfor- mance of agent model). We combine the state knowledge with the previous action a, and next action 0+ from the expert trajectory to form action-state-acttin triplet (at, 8t, 0++1). After iterat- ing through all expert trajectories, obtain State Knowledge Base B {(s, apre Qnext) (6)) J==1 where Opre- a, Onext 01+1, and [8] is the size of the state knowledge base.
3.3 Model Training
We integrate the generated world knowledge into expert trajectories and train a world knowledge model. The agent model needs be re-trained to adapt to the incorporation of task knowledge. Note that our agent model and knowledge model are both trained with LoRA sharing the same backbone. We list the examples of training data for both the agent model and WKM in Appendix F
Agent Model Training. Given the expert trajectories dataset = {(u, K, Tw (0) | with task knowledge K generated in $3.1 train the agent model to follow the task knowledge generate actions. Under an auto-regressive manner, the loss of the agent model can be formulated as:
Lagone (TO) = - -E..-D (aa.-//TTTTTuu [Tw [u, s)]
(6)
Suppose X' = (II, 21x)) ) is the token sequence of the trajectory Two we have:
To (u, 1))
Σ (1(zg E A) log To(x, k, 3=1
Here (ax E A) is the indicator function to mask tokens unrelated to actions. Please note that Tu here does not include the state knowledge mentioned in $3.2
For details on how to collect expert trajectories, please refer to Appendix A

---

World Knowledge Model Training. The main difference in the training data between the agent and knowledge model is the added state knowledge. Given the expert trajectories dataset with both
task and state knowledge D' {(u, N. ) J where T (do, 00, 80, ................................ Ong On, 8n), the loss
of the knowledge model T can formulated as:
Lienow (TT) = Bu, (T)())
k)]
(8)
Suppose X" = (21, 22 ., (xx) is the token sequence of the expert trajectory with state knowledge Th and Y' (y1, 122 ................................ M1y)) represents the token sequence of the task knowledge have:
[T(() =
log No (u u,, yci)
(9)
1241
Te (T (1, k) =
(1 E x log N (o(r/lu, x,
(10)
j=1
where 1 (ry S) is the indicator function to mask tokens unrelated to state knowledge.
3.4 Agent Planning with World Knowledge Model
At inference time, the agent model plans the evaluation tasks with the aid of the world knowledge .a.....a,,
model. We redefine the historical trajectory h, = (u, K, ag. On, 01 01,..., at, 0t). Given a specific task instruction u. the knowledge model first generates the task knowledge
To(-lu), then the agent model starts planning. Assuming the available action set A., C A for the task u is (aiu) (1) (2)
..... O'a (lll)) ), any time i. 2 0, instead of directly generating next action 0++1 E A.. based on h4, we first employ the world knowledge model to generate the current state knowledge
(0)g St (-1a) and leverage 8 to query the state knowledge base B {(s, apre (next) Ji=1 With the state knowledge as the key, we retrieve N nearest triplets from where apre = a, based on semantic similarity and collect the corresponding next actions Onext- We count the probability of
each action Pkenow (al()) = X where N₁ is the occurrence number of action au in all the collected anext: Therefore, we get the probability acquired from the state knowledge base:
LAul
Pkuoo (An) = (Pisnow (P (a.
Pknow Pknnow (),,))))-- Pknow (a (A..)))))
Pknow (a =1 (11)
Afterward, we sample the probability distribution of the first token for each action aa 1 [A,)] from the last layer the agent model and apply softmax function to normalize the probability distribution. We define the probability acquired from the agent model as:
(8)
[Ag
Pa Pagent (Au) (Pagent (a()),, Pagent (a ). ..a.....PPPaae.... Pagent (a AA 1)),
Σ
Pagent (a (
(12)
Finally, we determine the next action by combining the above two probabilities:
a++1 arg max
(7 Pagent (a(") + (1 - 7) Pleuow (a)
(13)
where y is the hyperparameter that controls the proportion of P Pagent (Au). Based on the above, we enhance the agent planning by global guidance from task knowledge and local constraints from state knowledge generated by our WKM. Due to the WKM and retrieval, the inference stage incurs additional time overhead compared to the pure agent model. The approximate ratio around 1.
Experiments
4.1 Experimental Settings
Datasets and Metrics. We evaluate our method on three real-world simulated planning datasets: ALFWorld 41]. WebShop [53], and Science [50]. AlFWorld and ScienceWorld include
5

---

Table I: Main Results. The best results marked in bold and the second-best results marked with underline. All the prompt-based baselines are evaluated under one-shot prompting and all the fine-tumm line-tuiing- based baselines ( ) ) are trained through LoRA. Red represents the changes of WKM relative to the optimal results in the baselines. WKM and agent model are different LoRAs sharing the same backbone.
ALFWorld
Science
Backhone
Method
WebShop
Seen
Unseen
Seen
Unseen
GPT-3.5-Turee GPT-4
8.57
5.97
44.37 62.76
15.41
13.99
REACT
44.29
38.05
67.32
65.09
REACT
7.86
5.22
14,63
20.72
17.65
Reflexion
11.56
6.00
16.64
21.07
18.11
NAT
64.43
68.96
61.01
57.12
50.79
Mistral-7B
FTO
66.84
71.43
64.09
58.17
$1.85
KNOWAGENT
70.44
70.72
61.28
59.32
47.24
WKM
73.57 +3.13
76.87
+5.44
65.48 +1.39
62.12
+2.80
53.62 +1.77
O
REACT
6.43
2.24
5.93
3.58
3.51
Reflexion
7.14
2,99
7.71
4.94
3,93
Gemma-7B
NAT
67.86
65.88
55.82
47.63
44.98
ETO
66.43
68 66
50.44
47.84
KNOWAGENT
69.29
67.60
58.80
48.55
45.28
WKM
70.71 +1.42
70.40 +1.74
63.75 +1.08
53.68
+3.24
49.24 +1.40
REACT
19.32
24.76
22.66
Reflexion
4,29
4,48
22.73
27.23
25.41
NAT
60.71
59.70
61.60
55.24
48.76
Llama-3-88
ETO
64,29
64,18
64.57
57.90
52.33
KNOWAGENT
66.71
62.69
64.40
58.67
49.18
WKM
68.57
+1.86
65.93
+1.75
66.64
+2.07
60.12
+1.55
54.75
+2.42
unseen tasks to evaluate the agent's generalization ability. The reward of ALFWorld is binary 0 or 1, indicating whether the agent has completed the task or not. WebShop and Science World provide dense rewards from 0 to 1 measure the completion level of the task. For all the datasets, we apply average reward as the final metrics. Please refer to Appendix B for detailed dataset information.
Models and Baselines. We evaluate three state-of-the-ard open-source models: 1) Mistral-7B [6]. the
version. 2) Gemma-7B [24), the Gemma-1. 1.1--Biit version. 3) Llama-3-8B 25 the Betta-Llama-3-88-.1
version. We compare method with prompt- based baselines: REACT [54) and Reflexion [40]. Besides, we adopt two strong baselines that introduce rejected trajectories into the training process learn from experience: NAT [49], learn from rejected trajectories through SFT, and ETO [44, learn from rejected trajectories through DPO planning method KNOWAGENT. We [27] and GPT-4 (gpt-4-32K-0613 (28 for comparison. All the prompt-base based baselines are tested under one-shot and all the fine-tuning based baselines are trained with LoRA [12. Please refer to Appendix C for baselines and re- re-producing details.
33. Moreover, we compare with a knowledge-augnie
also include ChatGPT (gpt-3.5-uu 1..--turbo-0125
Training and Inference Setups. We fine-tune the proposed approach with LoRA [12 using the LlamaFactory (2) framework. During training, the model is tuned after finishing the entire trajectory rather than each step of action. The learning rate is le-4 and the sequence length 2048 for all the models. The training epoch is 3 and the batch size is 32. We adopt the Adam optimizer [22] with a cosine learning scheduler. During inference, we apply the embedding layer WKM as the encoder and use the cosine similarity between sentences for retrieval. The number of retrieved action-state-actioo triplets N set to 3000 and the Pagent (A,.) weight 7 is set to (0.4, 0.5, 0.7). the training and inference experiments conducted 8 NVIDIA V100 32G GPUs within 12 hours. Please refer to Appendix D for detailed hyperparameters used in our paper.
4.2 Results
Main Results. As shown in Table for prompt-basen baselines on open-source models, both RE- ACT and Reflexion exhibit poor performance, far behind our method and fine-tuning-base baselines on various datasets. GPT-3.5-Tuubo performs ordinarily two datasets other than WehShop, and it even falls behind Mistral-7B and -lama-3-8B's REACT performance on ScienceWorld However, GPT-4 exhibits strong performance across various datasets. Nevertheless, approach, through

---

stote
troldcstatcc
w/o rejected
prompt
Heward
7A.
70-
AL.Woorld
WebShhp
Selence
Figure 3: Ablation Study on Mistral-7B. w/o all means the vanilla experienced agent model training with pure expert trajectories. w/ state is testing agent model with only state knowledge base constraints. w/ task stands for guiding agent model with only task knowledge. w/ task &state is our WKM with both task knowledge guidance and state knowledge constraints. w/o rejected means synthesizing task knowledge solely through expert trajectories, merge stands for training WKM and the agent model together with one single model. prompt means using few-shot prompts replace the WKM for providing knowledge.
Table 2: Average Steps. The maximum number of steps Table 3: Hallucinatory Action Rates on ALFWorld. We calculate the proportion of trajectories containing invalid actions regard- less of their correctness.
ALFWorld and WehShop is 40 and 10. In Science World, the number of steps ranges from 10 to 120 depending on the task type, with an average of around 40.
ALFWorld
ScienceWorld
ALFWorld
Method
WebShop
Method
Seen
Unseen
Seen
Unseen
Seen
Unseen
NAT
23.27
23.42
4.08
20.18
21.21
NAT 45.71%
50.00%
ETO
19.82
22.29
3,99
24.13
26.35
ETO
34.29%
36.57%
KNOWAGENT
18.51
24.56
4.01
21.06
24.74
KNOWAGENT
33.57%
44.78%
WKM
17.66
17.92
3.97
18.74
19.59
WKM 32.86%
29.85%
LoRA training alone, surpasses GPT-4 on ALFWorld (44.29-733.57 on seen, 38.05-+76.87 on unseen) and WebShop (62.76-66.66 For fine-tuning-based baselines, both NAT' and ETO fall behind our method, implying that just integrating world knowledge for agent models is worth more than further fussy SFT or DPO negative examples. Our method also performs better than KNOWA- GENT which brings human-designed fixed action knowledge and long action paths into trajectories. This suggests the effectiveness of our WKM which is responsible for generating instance-level task knowledge and maintaining implicit action constraints. Furthermore, KNOWAGENT'S performance on unseen tasks is not as impressive as on seen tasks, while WKM can keep its advantage. This phenomenon also demonstrates the generalization ability of WKM.
example, we decompose the key components of WKM to examine the roles of the task and state knowledge separately. In a macro view, removing each module results in a clear drop in the agent's performance, which validates the power of our world knowledge. Furthermore, the improvement through task knowledge (w/ task) is more pronounced than that through state knowledge (w/ state), suggesting the necessity of global prior knowledge for agent planning. A more micro observation reveals that the impact of state knowledge more significant on seen tasks compared to unseen tasks, while the influence of task knowledge is sustainable across seen and unseen tasks. This may be attributed that although our real-time state knowledge is generated by WKM, the state knowledge base built on the training set, which may weaken generalization to some extent. Additionally, to validate our motivation of allowing the agent to learn task knowledge from both expert and generated trajectories, we exclude the rejected trajectories during the synthesis of task knowledge, instructing the agent model to synthesize knowledge solely based on the chosen trajectories. The results (w/o rejected) demonstrate that learning from the contrast between chosen and rejected trajectories is more effective than learning from chosen examples alone. This procedure is a little similar to DPO, but we achieve it through knowledge augmentation rather than directly converting it into a loss calculation between chosen and rejected trajectories. Additional results can further evident that training a WKM separately performs better than training one single model together with the agent model as well using few-shot prompts to replace WKM for providing knowledge.
Approach Ablations. As shown in Figure 3 taking Mistral-7B
4.3 Analysis
World knowledge can mitigate blind trial-and-error and reduce hallucinatory actions. We compare the number of planning steps for each dataset between three strong baselines and WKM and calculate the average steps of each method. As depicted in Figure 9 (in Appendix F. WKM
7

---

demonstrates the ability to complete a significant proportion of tasks using the shortest trajectory. indicating that guidance from world knowledge can effectively reduce the agent's s blind tria-aan-erro in the environment. Taking further perspective from an average standpoint in Table 2 it can be observed that WKM exhibits lower average planning steps compared to other baselines. As ALFWorld can respond to invalid actions, in Table we count the percentage of hallucinatory actions that occurred in trajectories from ALFWorld for each method. The results confirm the effectiveness of our world knowledge model to decrease hallucinatory actions. Furthermore, it is worth noting that most baselines show a prominent increase in the average number steps and percentage invalid actions when transitioning from seen tasks to unseen tasks, but WKM can still maintain a relatively low level. This reflects laterally that our world knowledge can still effectively guide the agent model on unseen tasks, highlighting the knowledge generalization brought by the world knowledge model. To see how our world knowledge works, please refer to our case study in Appendix
Our instance-level knowledge can generalize better to unseen tasks. To further explore the benefit of using a knowledge model to generate instance-level task knowledge, we carefully survey the task
Husmnn
Awwagg Roward
WKat molo state
knowledge generated by our WKM and abstract it into dataset-level knowledge for each dataset. Then we retrain the agent model to
69-
adapt to new dataset-level knowledge As illustrated in Figure 4 we compare the performance of dataset-level knowledge with our instance-level task knowledge (WKM w/o state) on ALFWorld and Science World. It can be observed that our model-generatee instance- level knowledge not only surpasses human- designed knowledge on seen tasks but also exhibits even more remarkable performance on unseen tasks, with the improvement in performance on unseen tasks significantly greater than that on seen tasks. This phenomenon straightly reflects the strong generalization ability of our knowledge model compared to rigidly designed knowledge by humans.
AL ALrmordd sareer
NatWorld servera
ALEWard
World
Figure 4: Performance of human- designed dataset-level knowledge compared to WKM generated instance-level knowledge.
Table 4: Weak-guide-strong. The knowl-
Weak knowledge model guides strong agent model planning. In our main experiments, the knowledge model and agent model are based on the same backbone. Here, explore ALFWorld what will happen if we
edge model here is based on Mistral-7B.
ALFWorld
Backbone
Method
Seon
Unseen
a weak knowledge model to guide a strong agent model.
We choose Mistral-7B as the backbone of the knowledge model and ChatGPT and GPT-4 as the agent model. Since cannot get the token distribution from OpenAl API, only apply task knowledge to the agent model. As exhibited in Table the results of both ChatGPT and GPT-4 show distinct advances after being guided by the Mistral-7B world knowledge model, indicating the weak world knowledge model also contains knowledge that the strong model may lack. In the era of L.Mss, this inspires with to its lightweight nature, the weak knowledge model can flexibly adjust its parameters based on the needs of the agent model, which can address the difficulty of large agent models in adapting to new environments through fine-tuning.
(PT:3.5./urra
WKM w/o state
REACT
12.86
8.57
8.96
5.97
GPT-4
REACT
44.29
38.05
WKM w/o state
50.71
47,01
agent learning paradigm: weak-guide saak-gudde-troong. Due
ALFWoll
2001
Unified World Knowledge Model Training. We mix the world knowledge collected from all three datasets and jointly train WonnsWWoid
Spa
ALFWerkl BEaOcE
single world knowledge model to investigate the effect of multi-task world knowledge learning. Figure 5 illustrates the relative perfor- mance comparison between multi-task WKM and various baselines, from which we can observe that multi-task WKM not only does not lead to performance degradation but also exhibits visible im- provements compared single-task WKM. especially on WebShop and SidenceWorldd Similar to [57, 58, 3 which endeavor to train unified agent model and achieve strong generalization ability to held-out tasks, this observation inspires us with the potential of train- ing unified world knowledge model that can be applied to help
Schox:Worid sear
WelSlop
Figure 5: Relative performance of multi-task WKM compared to various baselines.
Detailedd manually designed dataset-level knowledge prompt be found in Appendix L.3
8

---

various held-in agent models and also generalize to guide held-out agent models. A more daring idea is whether unified agent model combined with a unified world knowledge model is the key to Artificial General Intelligence (AGI).
Explicit state knowledge will hurt the planning performance.
Exptitl Stace
To demonstrate the rationality of our choice construct state knowledge base, we explore the effect of directly incorporating state knowledge into the context of the agent model (we retrain the agent model to follow both the task and state knowledge), as shown in Figure [6] The performance of explicit state knowledge is far inferior to our approach of retrieving from a state knowledge base and utilizing probabilistic constraints. even performs worse than when we remove state knowledge and only include task knowledge.
Averay Rorard
5541
ALFWorldd socs
ALFWeeid unseen
Figure 6: Performance of ex- plicit state knowledge.
This clearly indicates that blindly extending prompts with large amount of explicit natural language feedback lose-more-than thanr--hann-gain for agent planning, and implicit knowledge constraints may be sometimes more prudent.
Case Study. In Figure 10 (Appendix HD. we list the trajectories of ETO and our WKM within the same task in ALFWorld to illustrate how world knowledge functions. The rationales before each action have been omitted to guarantee a clear illustration. The task is to clean s some soapbar and put it in cabinet. Initially, ETO blindly searches for the soapbar in the counter op and cabinet, introducing a lot of irrelevant information and unnecessary context. In the later stages of planning, ETO experiences the hallucination and executes the put action after close the cabinet, causing the environment to become unrecognizable and resulting in a collapse. On the contrary, guided by task knowledge, WKM directly identified the possible locations of the soapbar and successfully found it in the first attempt. Subsequently, WKM efficiently completed the task with precision, adhering to the constraints of state knowledge.
Related Work
LLM Agents.
LLMs have emerged a promising avenue towards unlocking the potential of Artificial General Intelligence, offering robust support for the development of agent systems [48] 51
8 63.
Existing works in this field mainly focuses agent planning [14] 21 [44 42), external tools
harnessing 33
23
43
29
32 35 44), code generation [45] 31 etc. Recently, there has been
an increasing focus on endowing open-source LLMs with agent functionalities through fine-tuning However, these approaches rely on blindly fitting the probabilities of tokens to learn planning. without having intimate cognition of the environment. The lack of knowledge can lead to the agent blindly attempting trial-an-erroo and generating hallucinatory actions.
17
56 38
49
Knowledge Augmented Agent Planning. Planning 15 is crucial capability for intelligent agents to accomplish real-world tasks, often requiring agents to possess rich knowledge and environmental commonsense. Few works have explored the field of knowledge-augmenttd agent planning. 14 61 5) utilize the rich parametric knowledge stored in pre-trained language models to assist agent planners. 7 20 (5. (4] design structured or natural language knowledge to regulate the actions. However, the above studies require the manual design of fixed prompt templates task procedures, making it challenging to transfer across different task environments. [63 55| 6 propose the automation of knowledge generation using language models. However, their knowledge either consists of only global workflow or only local action principles. In contrast, we train our world knowledge model both on global task knowledge and local state knowledge assist agent planning, and these knowledge sources are derived from the model's self-summary rather than hand-curated.
LLM- World Model. World model and agent model often co-occur in the domain of rein- forcement learning and robotics I3 9 19 37 26 4]. With LLMs commonly deemed as the most powerful intelligent machines constructed by humans thus far, the LLM-bbcked world models have been proposed [6] [0 [3]. In our paper, we attempt to self-synthesize world knowledge and train to obtain a world knowledge model. However, we consider our model be a world knowledge model rather than world model based on the reason that our model is temporarily unable to utilize search algorithms (e.g. MCTS) in conjunction with the agent model make predictions about the world and we leave this for our future work.

---

6 Conclusion and Future Work
In this paper, we strive to develop a parametric world knowledge model (WKM) to augment language agent model planning. Our WKM can generate prior task knowledge to guide global planning as well as dynamic state knowledge to regulate local planning. Our extensive results show that our world knowledge superior performance compared to various strong baselines. Analytical experiments validate that our WKM can 1) reduce brainless trial-and-error and invalid actions, 2) generalize better to unseen tasks, and 4) be effectively extended to unified world knowledge training. Potential future directions include: 1) building unified world knowledge model, 2) learning to predict the world like a world model, 3) applying to multi-modal agent planning, etc.
work both GPT-4 and tfte-of-the-art open-source models and achieve
3) achieve weak-guide-strooee
Limitations
Despite our best efforts, this paper still has some limitations: 1) Our primary intention behind designing the WKM is compensate for the lack of world knowledge in the agent model. However, determining what a language model knows and doesn't know has been an ongoing challenge that remains unresolved. 2) It is widely acknowledged that world knowledge extends beyond textual representations. While our world knowledge is currently limited to textual information, exploring multi-modal world knowledge models is indeed one of our important future tasks. 3) Our world knowledge model cannot dynamically update with the changes of the world and feedback from the agent. 4) Generating world knowledge can introduce additional inference overhead.
Acknowledgments and Disclosure of Funding
We would like to express our great gratitude to the anonymous reviewers for their kind comments. This work was supported by the National Natural Science Foundation of China (No. 62206246, No. NSFCC2332055, No. NSFCU19B2027), the Fundamental Research Funds for the Central Universities 226-2023-001388, Zhejiang Provincial Natural Science Foundation of China (No. LGG22F03001 1), Yongjiang Talent Introduction Programme 2221AA-15-GG) CIPSC-SMI HUPSC-SMPZZhip Zhipu Large Model Cross Pross-Discpplinary Fund, Ningbo Science and Technology Special Projects under Grant No. 2023Z212, Information Technology Center and State Key Lab of CAD&CG, Zhejiang University, and the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2023) (Grant 1-8001996-00-00) We gratefully acknowledge the support of Zhejiang University Education Foundation Qizhen Scholar Foundation.
NUS-NCS Joint Laboratory (A-0008542-00-00
References
[1] Robert Eamon Briscoe. Mental imagery and the varieties of amodal perception. Pacific Philosophical Quarterly, 92(2):153-173, 2011.
[2] Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. CoRR, abs/2310.05915 2023.
[3] Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flam Designing data and methods of effective agent tuning for large language models. CoRR, abs/2403. 12881, 2024.
[4] Anna Dawid and Yann LeCun. Introduction to latent variable energy-based models: A path towards autonomous machine intelligence. CoRR, abs/2306.02572, 2023.
[5] Yan Ding, Xiaohan Zhang, Saeid Amiri, Nieqing Cao, Hao Yang, Andy Kaminski, Chad Esselink, and Shiqi Zhang. Integrating action knowledge and llms for task planning and situation handling open worlds. Auton. Robots, 4788)9981-997, 2023.
[6] Yao Fu, Dong-Ki Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selection of state-aware guide- lines for large language model agents. CoRR, abs/2403.08978, 2024.
10

---

[7] Jian Guan, Wei Wu, Zujie Wen, Peng Xu, Hongning Wang, and Minlie Huang. AMOR: A recipe for building adaptable modular knowledge agents through process feedback. CoRR. abs/2402.01469, 2024.
[8] Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey progress and challenges. CoRR, abs/2402.01680, 2024.
[9] David Ha and Jürgen Schmidhuber. World models. CoRR, abs/ 11011803,111221 2018.
[10] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong. Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. EMNLP 2023, Singapore, December 6-10, 2023, pages 8154-8173. Association for Computational Linguistics, 2023.
[11] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. Metagpt: Meta programming for multi-agent collaborative framework. CoRR, abs/2308.00352, 2023.
[12] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[13] Zhiting Hu and Tianmin Shu. Language models, agent models, and world models: The LAW for machine reasoning and planning. CoRR, abs/2312.05230, 2023.
[14] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Muchine Learning Research, pages 9118-9147. PMLR, 2022.
[15] Xu Huang. Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of L.MM agents: A survey. CoRR, abs/2402.027166 2024.
[16] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abss2331000682. 2023.
[17] Philip N Johnson-Laird. Mental models and human reasoning. Proceedings of the National Academy of Sciences, 107(43):12243-188) 2010.
[18] Philip Nicholas Johnson-Laird. Mental models: Towards a cognitive science of language, inference, and conscionsness. Harvard University Press, 1983.
[19] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[20] Zelong Li, Wenyue Hua, Hao Wang, He Zhu, and Yongfeng Zhang. Formal-llm: Integrating formal language and natural language for controllable bame agents. CoRR, abx/2402.00098, 2024.
11

---

[21] Lajanugen Logeswaran, Yao Fu, Moontae Lee, and Honglak Lee. Few-shot subgoal plan- ning with language models. In Marine Carpuat Maaie-Catherinn de Marneffe, and Iván Vladimir Meza Ruíz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI 2022, Seattle, WA, United States, July 10-15, 2022, pages 3493-5506. Association for Computational Linguistics, 2022.
[22] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA,, USA, May 6-9, 2019. OpenReview.net, 2019.
[23] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-We Chang, Ying Nian Wu, Song- Chng-Chun Zhu, and Jianfeng Gao. Chameleon: ana-play compositional reasoning with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz. Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[24] Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, and et al. Gemma: Open models based on gemini research and technology. CoRR, abs/2403.08225. 2024.
[25] Meta. Introducing meta llama 3: The most capable openly available Ilm to date, 2024. https
7/ai .ew... www.clog//occooocccoo.ooooo.oooo hetemmmtttaaaaaaaa
[26] Thomas M. Moerland, Joost Broekens, Aske Plaat, and Catholijn M. Jonker. Model-based
reinforcement learning: A survey. Found. Trends Mach. Learn., 16(1):1-118 2023.
[27] OpenAL. Chatgpt: Optimizing language models for dialogue, 2022. attps://opent com/ hem/blog//ee chtt
[28] OpenAl. GPT-4 technical report. CoRR, abs/2303.08771 2023.
[29] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. CoRR, abs/2305.15334, 2023.
[30] RT Pramod. Michael Cohen, Kirsten Lydic, Josh Tenenbaum, and Nancy Kanwisher. Evidence that the brain's 's physics engine runs forward simulations of what will happen next. Journal of Vision, 20(1 12:151115221 2020.
[31] Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. CoRR, abs/2307.0799 0,0924,, 2023.
[32] Shuofei Qiao, Honghao Gui, Huajun Chen, and Ningyu Zhang. Making language models better tool learners with execution feedback. CoRR, abs/2305. 13068, 2023.
[33] Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. Reasoning with language model prompting: A survey. In Anna Rogers, Jordan L.. Boyd-Graben and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 5368-5393. Association for Computational Linguistics, 2023.
[34] Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. AUTOACT: automatic agent learning from scratch via self-planning. CoRR, abs/2401.05268,, 2024.
[35] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ realdworld apis. CoRR, abs/2307.16789, 2023.
12

---

[36] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[37] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nar., 588(7839) 2020.
[38] Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. Small Ilms are weak tool learners: A multi-llm agent. CoRR, abs/2401.07324, 2024.
[39] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yucting Zhuang. Hugginggpt: Solving AI tasks with chatgpt and its friends in hugging face. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[40] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[41] Mohit Shridhar, Xingdi Yuan, Mare-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[42] Chan Hee Song, Brian M. Sadler, Jiaman Wu, Wei-Lun Chao, Clayton Washington, and Yu Su. LIm-planner: Few-shot grounded planning for embodied agents with large language models. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 2986-2997. IEEE, 2023.
[43] Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. Rest- gpt: Connecting large language models with real-world applications via restful apis. CoRR, abs/2306.06624, 2023.
[44] Yifan Song, Da Yin. Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-basee trajectory optimization for LLM agents. CoRR, abs/2403. 025502, 2024.
[45] Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA,, USA, December 10 16, 2023, 2023.
[46] Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolal- paca: Generalized tool learning for language models with 3000 simulated cases. CoRR, abs/2306.05301, 2023.
[47] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, and et al. Llama Open foundation and fine-tuned chat models. CoRR, 025/2307700288. 2023.
2:
[48] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents. Frontiers Comput. Sci., 18(6):18634) 2024.
13

---

[49] Renxi Wang, Haonan Li. Xudong Han, Yixuan Zhang, and Timothy Baldwin. Learning from failure: Integrating negative examples when finettuning large language models as agents. CoRR, abs/2402. 1651, 2024.
[50] Ruoyao Wang. Peter A. Jansen, Marc- Alexandre Côté, and Prithviraj Ammanabrolu. Science- world: Is your agent smarter than a 5th grader? In Yoav Goldberg, Zornitsa Kozareva, and Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11279- 11298. Association for Computational Linguisties, 2022.
Yue Zhang, editors, Proceedings of the 2022 Conference
[51] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang. Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huan, and Tao Gui. The rise and potential of large language model based 2023.
agents: A survey. CoRR, abs/2309.0788
[52] Jiannan Xiang. Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang. Zichao Yang, and Zhiting Hu. Language models meet world models: Embodied experiences enhance language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[53] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[54] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRevie 2023.
[55] Yining Ye, Xin Cong, Shizuo Tian, Jiannan Cao, Hao Wang. Yujia Qin, Yaxi Lu, Heyang Yu, Huadong Wang, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Proagent: From robotic process automation to agentic process automation. CoRR, abs/2311.1074 2023.
[56] Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Lumos: Learning agents with unified data, modular design, and open-source llms. CoRR, abs/2311 1a.p2231110557,,, 2023.
[57] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for 11ms. CoRR, a0. 12823, 2023.
[58] Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan. Thai Hoang, liangwei Yang, Yihao Feng, Zuxin Liu, Tulika Awalgaonkar, Juan Carlos Niebles, Silvio Savarese, Shelby Heinecke, Huan Wang, and Caiming Xiong. Agentohana: Design unified data and training pipeline for effective agent learning. CoRR, abs/2402.15506, 2024.
[59] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: LLM agents are experiential learners. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty- -iihhth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty- Sixth Conference on Innovative Applications of Artificial Intelligence. IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 19632-19642 AAAI Press, 2024.
[60] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303. 18223, 2023.
14

---

[61] Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models commonsense knowledge for large-scale task planning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[62] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. CoRR, 1353/2403. 2024.
[63] Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. Agents: An open-source framework for autonomous language agents. CoRR, abs/2309.07870, 2023.
[64] Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Ningyu Zhang, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Knowagent: Knowledge-augmented planning for Ilm-based agents. CoRR, abs/2403.03100 2024.
15

---

A Expert Trajectories Collection
We mainly use the expert trajectories with a REACT-style [54 collected from 44
1. ALFWorld 41
The dataset provides human-annotated trajectories.
2. WebShop 53
Except for numan-annotater trajectories, GPT-4 is also applied generate trajectories with reward larger than 0.7 being reserved.
3. ScienceWorld [50] The dataset offers heuristic algorithms to search golden trajectories for each sub-task.
Since the original golden trajectories do not contain rationales, GPT-4 is further leveraged to generate the corresponding information.
B Dataset Information
We evaluate our method on three real-worke simulated agent planning datasets: ALFWorld 41 WebShop [53], and Science World [50].
ALFWorld a household dataset requiring the agent to navigate through the room and manipulate objects. Except for seen tasks, AlFWorld also includes unseen tasks to evaluate the agent's generalization ability. The reward of ALFWorld is binary 0 or 1, indicating whether the agent has completed the task or not.
2. WebShop is online shopping dataset in website environment. provides dense final rewards from 0 to to measure the completion level of the task.
3. Science a scientific reasoning dataset which is at the level of standard elementary school science curriculum. It also possesses both seen and unseen parts and a dense reward function from to 1.
For all the datasets, we apply average reward the final metrics. Table illustraates the statisties of each dataset.
Table 5: Dataset statistics.
Dataset
Train
Text-Seen
Text- Uextennseen
ALFWorld
3.119
140
134
WebShop
1,824
200
Woild
1,483
194
211
C Compared Baselines
Here detailedly introduce the baselines we compare with and re-produce details.
1. REACT [54). The first approach incorporates Chain-of Thouught (CoT) prompting in agent planning tasks with format of Thought-Action- Observation one-shot prompting for REACT6
-atigntt-tttioo--oeattttooo loop. In our paper, we apply
2. Reflexion 40 A strong prompt-based baseline reinforces agent planning with verbal feedback. Manually designed prompts are used to enable the agent to reflect on the historical trajectory and re-plan based on the feedback. In our paper, we utilize one-shot prompting for reflection and select the first reflect iteration as our result due to limited contex
3. NAT 49 NAT includes negative trajectories by employing different prompts during agent fine-tuning. When evaluating, only positive prompts are used to encourage the language
16

---

agent to generate correct trajectories. As it also follows the REAcT-style format, we directly use the default positive and negative prompts and train with LoRA in our paper
4. ETO Another baseline includes negative trajectories during agent training. The method contains two training phases, of which the first phase is behavior cloning which fine-tunes the agent on expert trajectories, and the second phase is learning from failures which further fine-tunes the agent through Direct Preference Optimization (DPO) [36). In our paper, remove the one-shot prompt for fairness and retain all the default hyperparameters proposed in ETO except for LoRA training l9
5. KNOWAGENT (64] KNOWAGENT is
agent planning baseline that applies action knowledge in the prompt and maintains an action path in the context during planning constrain the agent's 's action. We directly use the default prompt mentioned in KNOWAGENT for ALFWorld and carefully extend it to WebShop and Science by following similar forma 10
All the prompt-based baselines are tested under one-shot and all the fine-tuning- based baselines are
trained with LoRA 12
D Hyperparameters
The detailed hyperparameters we use during training and inference are shown in Table 6 We employ identical hyperparameters for different models. The temperature of the agent model is set to 0.0 when conducting exploration and 0.5 when introduced into WKM. The temperature WKM is set to 0.0 for all the time. The P Pagent (Aw) weight is set to 0.4 for ALFWorld, 0.5 for WebShop, and 0.7 for Sience World.
Table Detailed hyperparameters used our paper.
Name
Value
lora
8
lora alpha
16
lora dropout
0.05
lora target modules
q q_proj. v v_proj
cutoff len
2048
epochs
batch size
32
batch size per device gradient accumulation steps
4
2
learning rate warmup ratio temperature
lc-4
0.03
0.0, 0.5
retrieved state knowledge N
3000
Pagent (Au) weight
0.4, 0.5, 0.7
E
Training Data Example
We show the training data example for agent model and world knowledge model in Figure and Figure During training, transparent tokens do not contribute to the calculation of the loss. Only the parts the model responsible for output (agent model: purple, knowledge world model: green), participate in the loss calculation.
17

---

Training Data Example for Agent Model
Task Instruction: You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 1, a drawer 5, a drawer 4, a drawer 3, a drawer 2, drawer 1, a dresser 1, garbagecan 1, a laundryhamper 1, and shelf Your task is to: examine the book with the desklamp.
When you cannot find the object required for the task under the desklamp, first check all the drawers and shelves in the room, then revisit the previous locations and double-check before giving up and looking in the garbage can or laundry hamper. The action workflows are as follows: 1. For each drawer and shelf, go to the location and open it to check the object is inside. 2. If the object is not found, close the drawer or shelf and move on to the next one. 3. After checking all the drawers and shelves, revisit the previous locations and double-check if the object might have been overlooked. 4. If the object is still not found, look in the garbage can or laundry hamper as last resort.
Task Knowledge:
Thought: The task requires me to examine a book with the desklamp. First, need to locate the book.
Action: go to bed
Observation: On the bed 1, you see book 1, laptop 1, and pillow 1.
Thought: I see a book on the bed. should pick it up to examine it with the desklamp.
Action: take book 1 from bed
Observation: You pick up the book from the bed 1. Thought: Now that have the book, I need to find the desklamp examine the book under
its light.
Action: go to desk
Observation: On the desk you see an alarm clock 1, a bowl 1, a CD 2, a mug 3, a mug 2, a mug 1, a pen and pencil 2.
Thought: The desklamp isn't 't on the desk. need to look around the room to find where the desklamp located.
Action: go to dresser
Observation: On the dresser 1, you see a cellphone 1, a desklamp 1, and a keychain 2.
Thought: I've found the desklamp on the dresser. should use it to examine the book.
Action: use desklamp
Figure 7: Training Data Example for Agent Model.
Training Data Example for World Knowledge Model
Task Instruction: You are in the middle of a room. Looking quickly around you, you see bed 1, desk 1, a drawer 5, a drawer 4, drawer 3, a drawer 2, drawer 1, a dresser 1, garbagecan 1, laundryhamper 1, and shelf 1. Your task is to: examine the book with the desklamp.
Task Knowledge: When you cannot find the object required for the task under the desklamp, first check all the drawers and shelves in the room, then revisit the previous locations and double-check before giving up and looking in the garbage can or laundry hamper. The action workflows are as follows: 1. For each drawer and shelf, go to the location and open to check if the object is inside. 2. If the object is not found, close the drawer or shelf and move on to the next one. 3. After checking all the drawers and shelves, revisit the previous locations and double-check if the object might have been overlooked. 4. If the object is still not found, look in the garbage can or laundry hamper as a last resort.
Thought: The task requires me to examine book with the desklamp. First, I need to locate the book.
Action: go to bed 1
Observation: On the bed 1, you see a book 1, a laptop 1, and a pillow 1.
Your task is examine book with the desklamp. You have not found the desklamp yet, and have only checked the bed 1.
State Knowledge:
Thought: I see book on the bed. should pick it up to examine it with the desklamp.
Action: take book from bed 1
18

---

Observation: You pick up the book from the bed 1. State Knowledge: Your task is to examine a book using the desklamp, Your have located the book bed 1 and picked it up. Thought: Now that have the book, need to find the desklamp to examine the book under
its light.
Action: go to desk
Observation: On the desk 1, you an alarm clock 1, bowl 1, CD mug mug a mug 1, a pen 1, and pencil 2.
State Knowledge: Your task is to examine book with the desklamp. You have located the book on bed and picked it up. Next, You need to find the desklamp to examine the book under its light.
Thought: The desklamp isn't on the desk. need to look around the room to find where the desklamp located.
Action: go to dresser
Observation: On the dresser 1, you see cellphone a desklamp 1, and a keychain 2. State Knowledge: Your task is to examine book with the desklamp. You have located the book on the bed and picked up, now you find desklamp on a dresser.
Figure 8: Training Data Example for World Knowledge Model.
Win Rate of Planning Steps
See Figure g
WKM
FTO
Know
TBT
NAT
ALFWorld
37.1
12.8
31.4
ALJ World unsren
38,8
142
20.9
261
WetSSop
11.0
15.0
65
WoWrrl sem
47.4
171
17.5
180
SdWorld unsecn
50.5
144
370
181
100
Rate (%%)
Figure Win Rate Planning Steps. We choose method with the shortest steps each task and calculate the proportion.
G
Impact of
In fact, the ratio y can viewed as a signal to reflect whether knowledge or planning is more important for task. To understand which part of the output action has the most significant impact, we further analyze 7 0 (fully trust state knowledge base) and 7 = (fully trust agent model, equivalent to remove state knowledge in Figure 3). The empirical results can be seen in Table 7 It can be observed that state knowledge primarily serves as a constraint to alleviate hallucinated actions for the agent model. However, when we fully trust it (y 0), its lack of generalization significantly harms the performance of the agent model.
H Case Study
we list the trajectories of ETO and our WKM within the same task ALFWorld to illustrate how world knowledge functions. The rationales before each action have been omitted to guarantee clear illustration. The task is to c clean some soapbar and put in cabi net. Initially, ETO blindly searches for the scapbar in the countertop and cabinet, introducing
In Figure 10,
19

---

Table Impact y. fact, the ratio 7 can be viewed as signal to reflect whether knowledge or planning more important for task. To understand which part of the oulput action has the most significant impact, we further analyze 7 = 0 (fully trust state knowledge base) and r 1 (fully trust agent model, equivalent to remove state knowledge in Figure It can be observed that state knowledge primarily serves constraint to alleviate hallucinated actions the agent model. However, when we fully trust ( 0), lack of generalization significantly harms the performance of the agent model.
ALF
Science
Method
WebShop
Seen
Unseen
Scen
Unseen
1.58
0.00
25.83
18.69
15.37
7
69.29
75.37
63.68
60.81
53.42
WKM
73.57
76.87
65.48
62.12
53.62
lot of irrelevant information and unnecessary context. In the later stages of planning, ETO experiences the hallucination and executes the put action after close the cabinet, causing the environment become unrecognizable and resulting in collapse. On the contrary, guided by task knowledge, WKM directly identified the possible locations of the soapbar and successfully found it in the first attempt. Subsequently, WKM efficiently completed the task with precision, adhering to the constraints of state knowledge.
I
Prompts
In this section, we illustrate all the prompts used in our paper. Part of our prompts are taken from AutoGuide (
I.1
Task Knowledge Synthesis Prompt
See Figure
1.2
State Knowledge Summarization Prompt
See Figure
12
1.3
Dataset-Level Knowledge Prompt
See Figure
13
J Ethics Statement
This research was conducted following the ethical standards and best practices. All our experiments publicly available datasets (as detailed in Appendix B avoiding ethical concerns related to privacy, confidentiality, or misuse of personal biological information. However, despite our best efforts, it is not avoidable if someone maliciously modifies the world knowledge model contradict the world's knowledge and leads the agent to engage in unethical behavior.
20

---

Case Study of WKM
ETO
WKM
Task Instruction:
You are the middle of a room. Looking quickly around you, you cabinet 4, a cabinet a cabinet 2, a cabinet 1, handtowel-
Task Instruction:
Task Knowledge: and place an object, you should first locate the object. The possible locations of the soapbar are: toilet, sinbasin. Once the object found, you should clean using the sinkbasin, then put it in the cabinet. The action workflows as follows: 1. Go the place where the object located. 2. Take the object from the place. Clean the object using the sinkbasin. 4. Go to the cabinet. 5. Put the object in/on the cabinet.
When the task is to clean
countertop garbagecan holder 2, a handtowelholder 1, sinkbasin 2, a sinkbasin 1, toilet 1, toiletpaperhanger 1, and towelholder 1. Your task is to: clean some soapbar and put it in cabinet.
# Blind trial-and-era -and-erorr begins.
Agent: go countertop
Observation: On the countertop I, you see a candle 3, candle 1, soapbottle spray-
bottle 2, and toiletpaper 2.
Agent: go to toilet
Agent: go to cabinet
Observation: On the toilet 1, you see soap- bar 1, and a spraybottle 3.
Observation: The cabinet 1 is closed.
Agent: open cabinet
State Knowledge: Your task is to clean some soapbar and put it in cabinet. Now you are checking toilet and there is soapbar 1. Agent: take soapbar 1 from toilet
Observation: You open the cabinet 1. The cabinet is open. it, you see soapbottle 2.
Agent: close cabinet 1
Observation: You pick up the soapbar from the toilet 1.
Observation: You close the cabinet 1.
# Blind trial-and-error ends.
State Knowledge: Your task is to clean some soapbar and put it in cabinet. You have checked the toilet and found the soapbar 1.
Agent: go toilet 1
Observation: On the toilet 1, you see a soap- bar 1, and spraybottle 3.
Agent: take soapbar from toilet Observation: You pick up the soapbar 1 from
Agent: go to sinkbasin
Observation: On the sinkbasin you see nothing.
the toilet 1.
Agent: go sinkbasin
State Knowledge: Your task is to clean some soapbar and put it in cabinet. You have checked the toilet and found the soapbar 1. Now you are at the sinkbasin.
Observation: On the sinkbasin 1, you see nothing.
Agent: clean soapbar with sinkbasin
Observation: You clean the soapbar 1 using the sinkbasin 1.
Agent: clean soapbar with sinkbasin 1 Observation: You clean the soapbar using the sinkbasin 1.
Agent: go cabinet
Observation: The cabinet 1 is closed.
State Knowledge:
Your task is to clean some soapbar and put in cabinet. You have cleaned the soapbar 1 using the sinkbasin 1. Agent: go to cabinet
Agent: open cabinet
Observation: You open the cabinet 1. The cabinet is open. it, you see soaphottle
2.
Observation: The cabinet is closed. State Knowledge: Your task is to clean some soapbar and put it in cabinet. You have cleaned the soapbar using the sinkbasin Now you checking cabinet 1.
Agent: close cabinet
Observation: You close the cabinet 1. # Hallucinatory action.
Agent: put soapbar 1 in/on cabinet I
Observation: Nothing happens.
Agent: open cabinet 1
Agent: check cabinet
Observation: You open the cabinet 1. In it, you see a soaphottle 2.
Observation: Nothing happens.
Agent: go exit
State Knowledge: Your task is to clean some soapbar and put in cabinet. You have cleaned the soapbar and opened the cab- inet 1.
Observation: Nothing happens.
Agent: exit
Observation: Nothing happens.
Agent: put soapbar in/on cabinet Reward: 1.0
Reward: 0.0
Figure 10: Case study for our WKM.
21

---

Prompt for Task Knowledge Synthesis
Task Knowledge
Prompt for Synthesis: I will provide you with an analysis of both successful trajectory and an explored trajectory for the same task. By comparing the two, we can identify the key factors that contribute to success, Based on this analysis, you need to generate task-related task knowledge to help increase the success rate of future endeavors. Success Trajectory: Success_"
Explored Trajectory: Explored T
The task knowledge should specify what to do in what task. Here task knowledge example:
Task Knowledge Example
You should make your answer concise. Put your answer in this format: Task Knowledge: When you should (or should not) The action workflows are:
Figure 11: Prompt for Task Knowledge Synthesis.
Prompt State Knowledge Synthesis
State Knowledge
Prompt for Synthesis: You'll get segment of a trajectory of a text-based task. Your task is generate brief and general state knowledge of the task state now, following "State Knowledge: Keep wise and general for the same task. Here an example: State Knowledge Example
Now it's your turn. Here is the trajectory:
Trajectory
Make sure your output is within 128 tokens.
Put your answer in this format: State Knowledge: ...
Figure 12: Prompt for State Knowledge Summarization.
22

---

Task Knowledge example
Alfworld Task Knowledge example
When picking an object, heat it, and place it, you should first go to the possible locations of the object, then take the object, heat it with microwave, and put in place.
The action workflows are as follows:
go to receptacle
2) take object from receptacle
heat object with receptacle
4) go to the place to put the object
put object in/on receptacle
Webshop Task Knowledge example
When looking for an object you want to buy, you should first search with relevant keywords tailored to the product you looking for, and then click the relevant tag to view the product details, if the description matches the characteristics of the target item, click[buy now].
The action workflows are as follows:
1) search with keywords or examples, if you are searching for a laptop. you might search[laptop, 14-inch, Intel Core i7]
2) click the most relevant tag to view the detailed product page.
3) check the product details one by one, like color, size, type, and price, and make sure the price is within budget.
if find the right items, click[buy now] to buy it.
Seiworld Task Knowledge example
When tasked with boiling apple juice, focus on locating the kitchen first. Then, locate the apple juice in the fridge. Activate the stove. pour the apple juice into metal pot, and move the metal pot to the stove. Monitor the stove until the apple juice reaches a boiling point. Once boiled, remove the pot from the stove.
The action workflows are:
teleport the kitchen.
2) look around to find the apple juice in the fridge.
activate the stove.
4) pour apple juice into metal pot.
move the metal pot the stove.
6) look at stove.
examine apple juice to confirm boiling.
8) repeat step 6,7 until apple juice is boiled.
Figure 13: Dataset-Level Task Knowledge Examples.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction do accurately reflect
our paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section Limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
Our paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 4 and Appendix Hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification: We include our source code and data in the supplemental material submission, and outline the data generation procedure, the evaluation protocol, the training regime, and everything else necessary for reproduction either in the main body of the paper or in the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpp::/niip)) cc/
public/guides/ guwdes/ httl://guiis//ooeuusiiii Sodde ssisnSoossoooo conoocccy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NewrIPS code and data submission guidelines (https?
//nips http://ccppbbbccggiieeeeeeeee http://giieeeeeoeeueeeeeeee Code as cesssnnoooiiyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 4.1 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We include the average planning steps of each method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
26

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have carefully checked the NeurIPS Code of Ethics and ensure our code aligns with the request.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix I.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not have high risk PLMs or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited all the corresponding papers.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper thcode com dataaetteett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "OeEy38dkHD", "reviewer": "Reviewer_vJYv", "summary": "This paper introduces a parametric World Knowledge Model (WKM) to enhance agent planning by integrating both global task knowledge and dynamic state knowledge. The authors claim that their approach can mitigate issues like blind trial-and-error and hallucinated actions in large language model (LLM) agents. They demonstrate the effectiveness of WKM through experiments on three real-world simulated datasets (ALFWorld, WebShop, and ScienceWorld) using state-of-the-art LLMs, showing superior performance compared to strong baselines.", "questions": "Methodology:\n\n1. The motivation for rejecting trajectories is not sufficiently justified. (1) Why not and What if directly derive task knowledge from expert trajectories?  (2) Why assume that agent-generated trajectories are always inferior to the dataset trajectories? Could they not be better, and what would be the impact of this assumption? (3) Even if the above assumption holds, if agent-generated trajectories are always rejected, how can the authors claim, \"Our purpose is to extract superior task knowledge that cannot be acquired solely through supervised fine-tuning on chosen trajectories, thus further effectively boosting the agent’s capabilities\" (line 107-108)? (4) Ignoring the previous issues, in Line 104, the authors train an experienced agent to generate reject trajectories. However, trajectory generation requires an environment and an experienced agent. How can trajectories be generated without training the environment model? Is that done by interacting with the environment?\n\n2. The definition of state knowledge is ambiguous. Lines 119-120 suggest that state knowledge is a local summarization of the policy function to instruct actions, but lines 123-124 define it as part of the MDP's state space. Besides, Figure 12's prompt also does not define state knowledge yet asks the LLM to generate it (How can an LLM generate knowledge without a definition?)\n\n3. The knowledge model is trained to output task knowledge and state knowledge using data directly labeled by the LLM through prompts. Why not directly use these prompts to output task knowledge and state knowledge instead of retraining a model?\n\nExperiments:\n\n1. Can the paper \"AutoGuide: Automated Generation and Selection of State-Aware Guidelines for Large Language Model Agents\" be a baseline of this paper?\n2. Can the authors compare results with gamma=1 and gamma=0? It is essential to understand which part of the output action has the most significant impact.", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The motivation for the methodology is clear and logically presented.", "weaknesses": "My primary concerns with this paper are related to the methodology's implementation and the sufficiency of the experiments. Specific issues are detailed in the following questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for your valuable time and insightful feedback. Below are our detailed responses to your concerns.\n\n**Q1: The motivation for rejecting trajectories is not sufficiently justified.**\n\nIn fact, regarding your concern about the rejected trajectories, **we have explained in lines 113-117 of the paper**. We will consolidate this with lines 102-108 to make it more convenient for readers. We sincerely apologize for any inconvenience this has caused you. Below is a further explanation addressing your concern:\n\n(1) To intuitively demonstrate the effect of introducing rejected trajectories, we conduct experiments synthesizing task knowledge based solely on expert trajectories. The advantage of introducing rejected trajectories is very significant:\n\n|Mistral-7B|ALFWorld seen|ALFWorld unseen|WebShop|SciWorld seen|SciWorld unseen|\n|-----------------|-------------|---------------|-------|-------------|---------------|\n|w/o rejected traj|67.19|71.57|63.97|55.49|48.38|\n|WKM|73.57|76.87|65.48|62.12|53.62|\n\n(2) The assumption is based on the premise that expert trajectories are manually labeled, and they always achieve the best in quality and final reward. As we mentioned in lines 113-117 since expert trajectories are gold, their final reward $r(u,\\tau_w)$ always satisfies $r(u,\\tau_w)=1$. Therefore, the final reward $r(u,\\tau_l)$ of agent-generated trajectories always satisfies $r(u,\\tau_l) \\leq r(u,\\tau_w)$. If $r(u,\\tau_l) < r(u,\\tau_w)$, there is no doubt that expert trajectories are better; if $r(u,\\tau_l) = r(u,\\tau_w)$, **since expert trajectories are gold with no excess planning steps**, they are shorter and more efficient. We want the agent model to learn how to perform more efficient planning and avoid blind trial-and-error from these trajectories, thus we also consider agent-generated trajectories to be rejected.\n\n(3) Our experienced agent has undergone SFT on the chosen (expert) trajectories. As we all know, training on a dataset doesn't mean that the model has fully learned all the knowledge in that dataset. Therefore, we let the experienced agent run through the training data again to generate explored trajectories, so that the agent can summarize knowledge that cannot be learned solely through SFT. This step is a little similar to DPO, but we achieve it through knowledge augmentation rather than directly converting it into a loss calculation between chosen and rejected trajectories.\n\n(4) Yes, the generation of rejected trajectories involves direct interaction between the experienced agent model and the environment of the training set.\n\n**Q2: The definition of state knowledge is ambiguous.**\n\nOur state knowledge here is a natural language description of the current environment and the agent's state, so we define it as a part of the MDP's state space. As shown in Fig 12, we teach the agent to summarize state knowledge through few-shot examples, rather than zero-shot instruction. In the main paper Fig 2 and Appx F Fig 9, we provide some cases where you can see the specific appearance of state knowledge.\n\n**Q3: Why not directly use prompts to output task and state knowledge instead of retraining a model?**\n\nFirstly, task and state knowledge are obtained with the need for expert trajectories. Since we cannot obtain expert trajectories on the test set, it's hard to provide high-quality knowledge through prompts. Secondly, even if we use the knowledge obtained from the training set as few-shot prompts, it is evident that training a model is more generalizable than simply using prompts. In fact, **as one of our baselines, the knowledge for KnowAgent is provided through prompts and our dataset-level knowledge analysis in Fig 4 also uses prompts to provide knowledge. WKM's performance is significantly better.**\n\nTo completely address your concern, we also conduct experiments using high-quality task and state knowledge summarized from the training set as few-shot prompts. The experimental results can once again prove that training a model performs better:\n\n|Mistral-7B|ALFWorld seen|ALFWorld unseen|WebShop|SciWorld seen|SciWorld unseen|\n|----------------|-------------|---------------|-------|-------------|---------------|\n|prompt knowledge|65.14|67.40|61.03|56.36|45.27|\n|WKM|73.57|76.87|65.48|62.12|53.62|\n\n**Q4: AutoGuide as a baseline**\n\nIn fact, we greatly wanted to include AutoGuide as a baseline when we began our experiments. **However, at that time and even now, it doesn't have open-source code available.** We attempted to replicate it solely based on its paper, but some implementation details including some specific prompts cannot be obtained solely from the paper. In fact, as a prompt-based baseline, it relies on the strong GPT-4. And using 7/8B models would significantly degrade its performance, making it far less effective than fine-tuning-based methods, not to say our WKM.\n\n**Q5: Compare results with $\\gamma=1$ and $\\gamma=0$**\n\n$\\gamma=1$ is equivalent to removing state knowledge, and **our ablation experiment (Figure 3 w/ task) already includes this scenario**.\n\nWe further conduct experiments specifically for $\\gamma=0$, comparing it with $\\gamma=1$ and WKM:\n\n|Mistral-7B|ALFWorld seen|ALFWorld unseen|WebShop|SciWorld seen|SciWorld unseen|\n|----------|-------------|---------------|-------|-------------|---------------|\n|$\\gamma=0$|1.58|0.00|25.83|18.69|15.37|\n|$\\gamma=1$|69.29|75.37|63.68|60.81|53.42|\n|WKM|73.57|76.87|65.48|62.12|53.62|\n\nIt can be observed that state knowledge primarily serves as a constraint to alleviate hallucinated actions for the agent model. **However, when we fully trust it($\\gamma=0$), its lack of generalization significantly harms the performance of the agent model.**\n\nThank you again for your constructive suggestions!\n\n**Please let us know if you have any further questions. If you find that our response addresses some of your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bf3qBQjQ2D", "reviewer": "Reviewer_pvgu", "summary": "The paper presents a parametric World Knowledge Model (WKM) to enhance agent planning by providing both global prior task knowledge and local dynamic state knowledge. \nTraditional LLMs often perform trial-and-error actions and generate hallucinatory actions due to their limited understanding of the physical world. \nBy imitating human cognitive processes, the WKM synthesizes knowledge from expert and sampled trajectories to guide agents more effectively. \nExperimental results with state-of-the-art LLMs (Mistral-7B, Gemma-7B, and Llama-3-8B) show that this method improves performance on complex real-world tasks and reduces blind trial-and-error and hallucinatory actions.", "questions": "See weeknesses", "limitations": "no potential negative societal impact", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "WKM mimics human mental models, incorporating both global prior task knowledge and local dynamic state knowledge, and offers a novel and effective solution to the limitations of traditional LLMs in understanding the physical world. The human-guided action search would eliminate a lot of inappropriate hallucinations. \n\nThe method is rigorously tested on various complex real-world simulated datasets using Mistral-7B, Gemma-7B, and Llama-3-8B. The superior performance compared to strong baselines demonstrates the practical effectiveness and robustness of the proposed WKM, providing solid evidence for its advantages. Even these \"small language models\" prove the effectiveness of WKM.\n\nThis simulated knowledge base helps in better guiding, planning and assisting local planning, significantly improving the agent's overall performance and understanding of tasks.", "weaknesses": "The approach heavily relies on expert trajectories to synthesize both task and state knowledge. It takes a lot of effort to obtain such a database and pre-train/fine-tuning and off-line WKM. \n\nWKM depends on the world dynamic as well as human demonstration.  In my opinion, the three \"real-world simulated planning datasets\":\nALFWorld, WebShop, and ScienceWorld are not all real. The ALFWorld and ScienceWorld are simulated environments with a discreet Embodied mechanism. The methodology of WKM may not be easily adapted to problem-solving in real scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for your valuable time and insightful feedback. Below are our detailed responses to your concerns.\n\n**Q1: The approach heavily relies on expert trajectories to synthesize both task and state knowledge.**\n\nIn fact, most mainstream agent planning methods currently either rely on proprietary models like GPT-4 (e.g., AutoGuide) or utilize open-source models for incremental training on expert trajectories and explored trajectories through techniques such as SFT or DPO (e.g., ETO, NAT). These approaches do not consume fewer resources (be it monetary or computational resources) than our method. **Our method, without reliance on GPT-4, enables small-scale open-source models to autonomously synthesize knowledge and train into parameterized knowledge models.** This small-scale knowledge model can not only guide fine-tuning-based agents of the same scale (Table 1) but also enhance powerful proprietary models like GPT-3.5/4 (Table 4). Moreover, the multi-task unified WKM has demonstrated superior performance (Figure 5), providing insights into the path toward exploring AGI in the future.\n\n**Q2: The methodology of WKM may not be easily adapted to problem-solving in real scenarios.**\n\nThe current mainstream LLM-based agent planning benchmarks are all based on simulated environments of the real world. The three datasets we utilize—ALFWorld, WebShop, and SciWorld—are also commonly used in LLM-based agent planning. We understand the significance of applying the World Knowledge Model (WKM) to real-world scenarios, and we are actively working towards this, such as exploring a unified knowledge model that can be generalized to real-world situations. We hope to pair this with a generalizable unified agent model to achieve true AGI (Artificial General Intelligence). Although this path is still a long way off, our work, like the work of others, is striving towards this goal.\n\nThank you again for your constructive suggestions!\n\n**Please let us know if you have any further questions, as we are happy to continue the discussion. If you find that our response addresses some of your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s1tKKnahgi", "reviewer": "Reviewer_TdHL", "summary": "This work is concerned with LLMs planning abilities in agent datasets. Instead of only fine-tuning the agent model on expert trajectories, they add “task knowledge” information. This information is created by comparing reject trajectories and expert trajectories, following previous work (NAT; Wang et al., 2024). In addition, the agent model is prompted to summarize the state, which helps avoid generating invalid actions.", "questions": "**Suggestions**\n* Case study in Appendix F is very important to understand the paper, providing intuition and a qualitative analysis which explains why preference trajectories data is useful. I would put it in the main paper to guide the reader.\n\n**Questions**\n* Line 135 mentions that the WKM and the agent are the same backbone model. What about the LM that generates the reject trajectories, is it the same?", "limitations": "The authors discuss limitations of their work.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea of explicating the preference trajectories data (NAT; Wang et al., 2024) into task knowledge is interesting (subsection 3.1 and case study in Appendix F).\n* Strong results when combining both task knowledge and state knowledge.\n* Weak-guide-strong analysis (table 4) interestingly shows the benefits of explicating the trajectories preference knowledge.", "weaknesses": "- When removing the state knowledge (figure 3), it seems that this approach does not outperform NAT, which uses SFT on the same trajectories preference data.\n    \n- Missing qualitative analysis of the generated task knowledge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for your valuable time and insightful feedback. Below are our detailed responses to your concerns.\n\n**Q1: When removing the state knowledge (figure 3), it seems that this approach does not outperform NAT, which uses SFT on the same trajectory preference data.**\n\nWe greatly appreciate your meticulous observation. **However, in fact, even without the state knowledge, our method still outperforms NAT.** We feel so sorry that maybe the distance between Table 1 and Figure 3 in the paper has misled your judgment and we will improve this in our revision. Here we list the specific values in the table below for your convenience in observation:\n\n| Mistral-7B    | ALFWorld seen | ALFWorld unseen | WebShop | SciWorld seen | SciWorld unseen |\n| ------------- | ------------- | --------------- | ------- | ------------- | --------------- |\n| NAT           | 64.43         | 68.96           | 61.01   | 57.12         | 50.79           |\n| WKM w/o state | 69.29         | 75.37           | 63.68   | 60.81         | 53.42           |\n| WKM w/o task  | 67.86         | 70.67           | 62.44   | 55.04         | 51.52           |\n| WKM           | 73.57         | 76.87           | 65.48   | 62.12         | 53.62           |\n\n**Q2: Qualitive analysis and Appendix F**\n\nWe feel so sorry that due to page limitations and the lengthy trajectories of the case in Figure 9, we did not include Appendix F in the main paper initially. **We will incorporate the textual analysis in Appendix F into the main paper in the upcoming revision.** In fact, on the right side of Figure 2, we also display part of a case's steps, from which you can clearly see the role and effectiveness of our task and state knowledge.\n\n**Q3: Line 135 mentions that the WKM and the agent are the same backbone model. What about the LM that generates the reject trajectories, is it the same?**\n\nYes, you are right. They all share the same backbone model but with different LoRAs. All the training involved in our paper is based on LoRA, which allows the knowledge model and the agent model to be plug-and-play, significantly saving computational power while making the model switch and extend more flexibly and efficiently.\n\nThank you again for your constructive suggestions!\n\n**Please let us know if you have any further questions, as we are happy to continue the discussion. If you find that our response addresses your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QrYc0tDRtO", "reviewer": "Reviewer_8XFL", "summary": "This paper presents a parametric world knowledge model designed to enhance agent planning. The model synthesizes knowledge from expert and sampled trajectories for training purposes. It incorporates prior task knowledge for global planning and dynamic state knowledge for local planning. The implementation shows improved performance over several robust baselines using open-source LLMs such as Mistral-7B and Gemma-7B. Incorporating a world knowledge model into LLM-based agents for planning purposes is a novel approach that also helps mitigate common issues like hallucinations and invalid actions in language agents.", "questions": "Please refer to weaknes.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is overall well-written and the method is novel. It also show promising results using only 7B models that could barely perform planning with basic methods. It's also quite reasonable to introduce a world knowledge model to enhance task specific knowledge during planning.", "weaknesses": "Here are some additional points that I believe could be improved: \n\n1. Clarity on hyperparameters and settings: Is the WKM tuned after each step of action or finishing an entire trial? Also what is the split for seen/unseen task? Some tasks are very similar in Alfworld and Scienceworld. What is the structure of the retriever? What is the number of generations for each action? \n\n2. Is WKM training more important for commmon-sense intensive tasks like Scienceworld? Have you previously tested on pure planning tasks? e.g. Blocksworlds, Game 24.  Also, does the ratio $\\gamma$ reflects whether knowledge or planning is more important for a task? \n\n3. What is the additional computation overhead for inferencing using a WKM? Also beam search is necessary to obtain more than one action, what is the computation time compared to ReAct? \n\n4. What is the difference between state knowledge and thoughts/ reflection? It seems to be also interleaved between actions. Does the content of state knowledge greatly affect the performance? e.g. using the WKM to enhance ReAct thoughts. The state-knowledge examples in Figure 8 seems more similar to thoughts rather than world knowledge, i.e. give a general commonsense about the environment.\n\n5. Can WKM and agent training be merged together? i.e. using a single model to learn both knowledge and trajectory and update the loss together. \n\n6. Can this method be applied to an online setting? Where knowledge base is directly updated after each step of action so that later actions could use results from exploration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for your valuable time and insightful feedback. Below are our detailed responses to your concerns.\n\n**Q1: Clarity on hyperparameters and settings**\n\nWe sincerely apologize for any confusion caused by the details not clarified in the paper.\n\n(1) **As shown in Eqn 8-10**, our WKM is fine-tuned over the entire trajectory, where the loss calculation only involves knowledge, and other irrelevant tokens will be masked.\n\n(2) The seen and unseen tasks are predefined within the dataset. For the unseen tasks, the model has not previously encountered the task types in the training set, requiring the application of more extensive generalization capabilities to address them.\n\n(3) Our retrieval is primarily based on cosine similarity. The encoder is WKM's own embedding layer (for example, the embedding layer of Mistral-7B), and similarity is calculated based on the cosine similarity between sentences.\n\n(4) At each step, we only generate an action once. Since we use open-source models, we can directly obtain the probability distribution of each action's first token from the last layer of the agent model at each step. **As we mentioned in lines 162-163**, we normalize the probability distribution of the action tokens with a softmax function to get the final action distribution $P_{\\rm agent}(\\mathcal{A}_u)$ from the agent model.\n\n**Q2: Is WKM training more important for commonsense intensive tasks like Sciworld?**\n\nIn this paper, we focus more on embodied planning tasks that interact with the environment, which require world knowledge to be resolved. **Tasks such as Blocksworld and Game24 are, in our understanding, more like reasoning tasks**, where the model only needs to reason based on the problem without the need for the environment.\n\nRegarding the ratio $\\gamma$ reflecting whether knowledge or planning is more important for a task, your understanding is very insightful. We have also found that for tasks in SciWorld that require a large amount of domain knowledge, the value of $\\gamma$ is smaller (knowledge occupies a larger proportion).\n\n**Q3: The additional computation overhead for inferencing using a WKM**\n\n**In fact in lines 167-168, we have discussed the issue of computation overhead.** When accounting for the inference time of the knowledge model and the retrieval time, our total inference time is 2.5 times that of a pure agent model fine-tuned with expert trajectories. In Q1, we explained that we do not employ beam search, and because ReAct is a prompt-based method, which tends to have poorer performance and thus results in significantly longer trajectory lengths, the ratio of our method's inference time compared to ReAct will be even smaller.\n\n**Q4: The difference between state knowledge and thoughts/reflection**\n\nThe original intention behind the design of state knowledge is due to our observation that a significant issue in agent planning arises when the trajectory is lengthy, causing the model to experience hallucinations due to difficulties in processing long contexts. Thus, state knowledge acts more like an \"outside\" prompter, providing the agent with dynamic knowledge of the current environment rather than prior commensense knowledge, to make it focus more on the current action. In contrast, the agent's thoughts resemble an \"inside\" judger of the current state, which can easily be influenced by long contexts and deviate from the correct path.\n\n**Q5: Can WKM and agent training be merged together?**\n\nWe train WKM and the agent model separately for two reasons: 1) Numerous studies have indicated that the division of labor and collaboration among models can lead to the emergence of group intelligence, which achieves better results than individual intelligence; 2) Decoupling the WKM from the agent model allows for more flexible expansion, such as guiding a stronger agent model with a smaller WKM (Table 4), and realizing a unified WKM (Figure 5). To fully show the advantages of separate training, we further conduct experiments where a single model learns planning and knowledge at the same time:\n\n| Mistral-7B   | ALFWorld seen | ALFWorld unseen | WebShop | SciWorld seen | SciWorld unseen |\n| ------------ | ------------- | --------------- | ------- | ------------- | --------------- |\n| single-model | 65.46         | 70.71           | 63.70   | 58.51         | 51.78           |\n| WKM          | 73.57         | 76.87           | 65.48   | 62.12         | 53.62           |\n\nThe experiment has shown that the merging of WKM and agent model results in a worse effect, which also confirms our viewpoint.\n\n**Q6: Can this method be applied to an online setting?**\n\nIn fact, our initial plan was to create an online version of WKM, but we encountered the following challenges:\n\n1) **Dynamic adjustment of $\\gamma$**. In the early stages, when the state knowledge base contains too few samples to cover common scenarios, we obviously cannot trust its probabilities, so we need to set $\\gamma$ to a high value. Conversely, when the state knowledge base is sufficiently populated, $\\gamma$ should be set to a relatively lower value. Thus, $\\gamma$ should gradually decrease and eventually stabilize over the process, but controlling the speed of decrease and the time to reach stability is challenging.\n\n2) **Parameter updating of the WKM**. How to adjust the parameters of WKM when new knowledge is acquired involves the frequency of parameter updates. An excessively high frequency could lead to extremely low efficiency, while an insufficient frequency may not meet the needs of the agent model.\n\nWe are also conducting further research on the online version of WKM and hope to make significant progress in the future.\n\nThank you again for your constructive suggestions!\n\n**Please let us know if you have any further questions. If you find that our response addresses some of your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear all reviewers,

Thank you for your thoughtful reviews! We appreciate all of your **positive comments** highlighting the strengths of our work for a summary:

## **Our Strengths Summarized by Reviewers**

- **Reasonable motivation**: 
  - "The motivation for the methodology is clear and logically presented."(reviewer vJYv)
  - " It's also quite reasonable to introduce a world knowledge model to enhance task-specific knowledge during planning."(reviewer 8XFL)
- **Interesting, novel, and easy to follow**:
  - "The paper is well-written and easy to follow."(reviewer vJYv)
  - "offers a novel and effective solution"(reviewer pvgu)
  - "The idea of ... is interesting."(reviewer TdHL)
  - "the method is novel"(reviewer 8XFL)
- **Superior performance, promising results, and interesting analysis**:
  - "The superior performance compared to strong baselines demonstrates the practical effectiveness and robustness of the proposed WKM, providing solid evidence for its advantages."(reviewer pvgu)
  - "Strong results when combining both task knowledge and state knowledge."(reviewer TdHL)
  - "Weak-guide-strong analysis (table 4) interestingly shows the benefits of explicating the trajectories preference knowledge."(reviewer TdHL)
  - "It also shows promising results using only 7B models that could barely perform planning with basic methods."(reviewer 8XFL)
- **Well-written** (reviewer vJYv, reviewer 8XFL)

## **Our Supplement Experiments**

We also sincerely thank reviewers for your constructive feedback and questions to improve our manuscript. In response to the reviewers' questions, **we mainly supplement the following experiments**:

1. For reviewer vJYv: **To intuitively demonstrate the effect of introducing rejected trajectories**, we conducted experiments synthesizing task knowledge based solely on expert trajectories.
2. For reviewer vJYv: **To address the concerns about the necessity to retrain a knowledge model**, we conduct experiments using high-quality task and state knowledge summarized from the training set as few-shot prompts for knowledge models to provide knowledge.
3. For reviewer vJYv: **For the convenience of understanding which part of the output action has the most significant impact**, we compare the results with $\gamma=0$, $\gamma=1$, and our WKM.
4. For reviewer 8XFL: **To fully show the advantages of separate training**, we further conduct experiments where a single model learns planning and knowledge at the same time.

**We have added supplementary experiments 1, 2, and 4 as parts of our Ablation Study and revised our Figure 3. We have also added supplementary experiment 3 to our Appendix with a table. The revised Figure 3 and the added $\gamma$ analysis table can be seen in our submitted PDF.** We will add the corresponding textual analysis in our main paper.

We will continue to further enhance the quality of this paper according to the discussion with reviewers.

Last but not least, we wish to **reiterate the motivation and main contributions** of our paper.

## **Our Motivation**

As most state-of-the-art LLMs are autoregressive models trained with next-token prediction, they lack the ability to essentially understand the real world, leading to generating hallucinatory actions in local planning and performing brainless trial-and-error in global planning. In contrast to LLMs, humans possess a mental knowledge model about the physical world. When facing a specific task, they will first briefly rehearse the entire process in mind using their rich prior knowledge and constantly maintain a dynamic cognition of the current world state. **The process by which humans handle planning tasks reminds us to develop a parametric World Knowledge Model (WKM) to facilitate agent planning.**

## **Our Contributions**

- Imitating humans' mental knowledge model, we introduce parametric World Knowledge Model (WKM), providing prior task knowledge to guide the global planning and dynamic state knowledge to assist the local planning.
- Experimental results on three complex real-world simulated datasets with three state-of-the-art open-source LLMs, Mistral-7B, Gemma-7B, and Llama-3-8B, demonstrate that our method can achieve superior performance compared to various strong baselines.
- We analyze to illustrate that our WKM can effectively alleviate the blind trial-and-error and hallucinatory action issues, providing strong support for the agent’s understanding of the world.
- Other interesting findings of our paper include: 1) our instance-level task knowledge can generalize better to unseen tasks, 2) weak WKM can guide strong agent model planning, and 3) unified WKM training has promising potential for further development.

## **Thank you!**

We sincerely thank the reviewers for their constructive suggestions and questions to enhance our paper. Please reply if you have any further questions, and we will be more than happy to continue the discussion.

Meta-review: The authors provide task knowledge and state knowledge to improve global and local planning in smaller LLMs and shows promising generalization. The paper is well written with many experiments and additional analyses that strengthens their novel approach. The authors and reviewers have engaged in thorough discussions, and in spite of some disagreement between the authors and one of the reviewers, the paper is recommended for acceptance as a poster, conditional on integrating the new results and responses to reviewers in the paper.
Decision: Accept (poster)