Agent Planning with World Knowledge Model

Authors: shuofei  qiao
runnan  fang
ningyu  zhang
yuqi  zhu
shumin  deng
yong  jiang
pengjun  xie
fei  huang
huajun  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 09:40:52

Abstract: Recent endeavors towards directly using large language models (LLMs) as agent models to execute interactive planning tasks have shown commendable results. Despite their achievements, however, they still struggle with brainless trial-and-error in global planning and generating hallucinatory actions in local planning due to their poor understanding of the "real" physical world. Imitating humans' mental world knowledge model which provides global prior knowledge before the task and maintains local dynamic knowledge during the task, in this paper, we introduce parametric World Knowledge Model (WKM) to facilitate agent planning. Concretely, we steer the agent model to self-synthesize knowledge from both expert and sampled trajectories. Then we develop WKM, providing prior task knowledge to guide the global planning and dynamic state knowledge to assist the local planning. Experimental results on three real-world simulated datasets with Mistral-7B, Gemma-7B, and Llama-3-8B demonstrate that our method can achieve superior performance compared to various strong baselines. Besides, we analyze to illustrate that our WKM can effectively alleviate the blind trial-and-error and hallucinatory action issues, providing strong support for the agent's understanding of the world. Other interesting findings include: 1) our instance-level task knowledge can generalize better to unseen tasks, 2) weak WKM can guide strong agent model planning, and 3) unified WKM training has promising potential for further development.

Main paper: 
 introduction: 
 The remarkable advances in Large Language Models (LLMs) have witnessed a rapid development of various natural language processing tasks [25, 16, 28, 47, 60, 33] . Recently, multiple attempts that directly exploit LLMs as agent models to address physical world planning tasks have demonstrated promising achievements [54, 57, 56, 34, 38, 64, 44] . However, as most state-of-the-art LLMs are autoregressive models trained with next-token prediction, they lack the ability to essentially understand the real world, leading to generating hallucinatory actions and performing brainless trial-and-error in the environment as shown in Figure 1(a) .In contrast to LLMs, humans possess a mental knowledge model about the physical world [1, 18, 17, 30] . When facing a specific task, they will first briefly rehearse the entire process in mind using their rich prior knowledge before performing mindless actions. We call this kind of knowledge global task knowledge (a.k.a. environment/task commonsense). In addition, during the task procedure, the mental world knowledge model will constantly maintain a kind of local state knowledge, representing humans' cognition of the current world state. For example, imagine you are in a room and your task is to put a clean egg in microwave. The task knowledge may refer to The egg is most likely in the fridge ... The workflows are: 1) locate and take the egg; 2) clean the egg using sinkbasin ... The state knowledge possibly refers to My task is to ... I have found and taked the egg ... Next I should ... The absence of world knowledge can lead to blind trial-and-error in the early planning stages when environmental information is limited. Conversely, in later stages when information is redundant, it can easily result in a confused cognition of the current world state and generate hallucinatory actions.The process by which humans handle planning tasks reminds us to develop a parametric World Knowledge Model (WKM) to facilitate agent planning. As humans typically acquire knowledge from expertise and practical experience, we build WKM based on knowledge learned from both expert and explored trajectories. Specifically, we first steer the agent model to synthesize task knowledge from the comparison between expert and sampled trajectories. Then we prompt it to summarize state knowledge for each planning step from expert trajectories and combine the previous and next actions to build a state knowledge base. Lastly, we integrate the generated knowledge into expert trajectories and train a WKM. The agent model needs to be retrained to adapt to the task knowledge. Note our agent and knowledge model are both trained with LoRA [12] sharing the same backbone.During the planning phase, we use the WKM to provide global prior task knowledge and maintain local dynamic state knowledge for the agent model as shown in Figure 1 (b). The task knowledge will be concatenated in natural language form following the specific task to guide the agent model's trial-and-error. At each planning step, to prevent the occurrence of hallucinatory actions, we utilize the generated state knowledge as the query to conduct kNN retrieval from the pre-built state knowledge base. We then use the constraints from the previous action, the probabilities of the retrieved next actions, and the probabilities from the agent model to make a weighted prediction for the next action.We evaluate our method on three real-world simulated planning tasks: ALFWorld [41] , WebShop [53] , and ScienceWorld [50] with three state-of-the-art open-source LLMs: Mistral-7B [16] , Gemma-7B [24] , and Llama-3-8B [25] . Empirical results demonstrate that our method achieves superior performance compared to various strong baselines on both seen and unseen tasks. Moreover, further analytical results show that 1) our WKM can effectively reduce blind trial-and-error and hallucinatory actions, 2) our model-generated instance-level knowledge can generalize better to unseen tasks, 3) weak-guide-strong is feasible, 4) multi-task unified WKM possesses strong potential, and 5) explicit state knowledge will hurt the performance of agent planning. 
preliminaries: 
 We mainly focus on interactive tasks with partial observations from environments. Following the task formulation in [44] , the problem can be viewed as a Partially Observable Markov Decision Process (POMDP): (U, S, A, O, T ). The instruction space U defines the task and its corresponding regulations. S is the state space, O is the observation space, and A is the action space. T : S ×A → S defines the transition function, which we assume to be given by the environments. It is noticed that U, A, and O are subspaces of the natural language space in the language agent scenarios.Based on the above, the historical trajectory h t that consists of a list of actions and observations at time t can be represented as:EQUATIONwhere u ∈ U is the task instruction and a ∈ A, o ∈ O are the action and the observation. Given a task, the language agent with parameter θ serves as the policy model π θ responsible for generating the action a t+1 based on h t at each time step t + 1:EQUATIONSpecifically, a 0 ∼ π θ (•|u) is generated according to the task instruction u. The whole trajectory τ concludes when the task is completed or exceeds the maximum time steps. Then the production of the entire trajectory with time length n can be modeled as:EQUATIONUltimately, the final reward r(u, τ ) ∈ [0, 1] representing the task completion rate is calculated. Note that we follow a REACT-style [54] trajectory that includes rationales before each action. We use a to represent the action with rationales for convenience.World Knowledge Model. World knowledge model serves as humans' mental cognition of the physical environment, more intricate than the word knowledge model which LLM-powered agent models are trained to be [61, 10, 52, 13] . Our "world" here refers to the simulated environment of the task. Based on the static environment of the task and the dynamic changes during interaction with the agent, we define world knowledge as a combination of prior global knowledge and dynamic local knowledge, corresponding to the blind trial-and-error problem in global planning and the hallucinatory action issue in local planning in traditional agent models, respectively. To attain precise and efficient agent planning, we develop a parametric WKM to simulate the mental WKM of humans. 
method: 
 As shown in Figure 2 , we steer the agent model to self-synthesize the task knowledge from the comparison of expert and sampled trajectories ( §3.1). Then we prompt the agent model to selfsummarize the state knowledge based on historical behavior and construct a state knowledge base ( §3.2). The generated knowledge will be integrated into the expert trajectories for training the WKM.After the training process ( §3.3), we augment the agent model with the world knowledge model to achieve effective and accurate planning ( §3.4).The task knowledge serves as the prior knowledge to guide the agent model's global planning and prevent it from dropping into blind trial-and-error.Experienced Agent Exploration. We primarily acquire task knowledge through the comparison of preference trajectories (chosen vs. rejected). In order to improve the quality of rejected trajectories and obtain more targeted task knowledge, we employ an experienced agent for exploration. Firstly, we train a vanilla language model with expert trajectories4 from the training set to obtain an experienced agent. Subsequently, the experienced agent explores the training set tasks again to generate rejected trajectories. Our purpose is to extract superior task knowledge that cannot be acquired solely through supervised fine-tuning on chosen trajectories, thus further effectively boosting the agent's capabilities.Self Knowledge Synthesis. With the expert trajectories as the chosen ones and the trajectories sampled from the experienced agent as the rejected ones, we prompt the agent model itself to synthesize the task knowledge. Supposing K is the task knowledge space:EQUATIONwhere κ ∈ K is the task knowledge, ρ TaskKnow stands for the prompt to instruct the task knowledge extraction, and τ w , τ l are the chosen and rejected trajectories respectively. Note that given the same task u, τ w and τ l always satisfy r(u, τ w ) = 1 ≥ r(u, τ l ). Even when r(u, τ w ) = r(u, τ l ), we still consider trajectories sampled from the experienced agent as rejected ones. This is because expert trajectories often have shorter step lengths, enabling the agent to learn more knowledge of efficient planning. For detailed prompts of task knowledge synthesis, please refer to Appendix I.1.The state knowledge serves as the dynamic knowledge to constrain the agent model's local planning and prevent it from generating hallucinatory actions. We prompt the agent model to self-summarize state knowledge at each planning step based on the expert trajectories to guarantee quality. For detailed prompts of state knowledge summarization, please refer to Appendix I.2. Supposing the prompt used to summarize state knowledge is ρ StateKnow and the state knowledge s ∈ S is a part of the state space S, the generation of state knowledge at time t can be represented as:EQUATIONState Knowledge Base Construction. To avoid confusion caused by excessive additional information, instead of explicitly concatenating the state knowledge to the context, we construct a state knowledge base for retrieval (we analyze in §4.3 how explicit state knowledge may affect the performance of agent model). We combine the state knowledge s t with the previous action a t and next action a t+1 from the expert trajectory to form a action-state-action triplet (a t , s t , a t+1 ). After iterating through all expert trajectories, we obtain a State Knowledge Base B = {(s, a pre , a next) (i) } |B| i=1, where a pre = a t , a next = a t+1 , and |B| is the size of the state knowledge base.We integrate the generated world knowledge into expert trajectories and train a world knowledge model. The agent model needs to be re-trained to adapt to the incorporation of task knowledge. Note that our agent model and knowledge model are both trained with LoRA sharing the same backbone. We list the examples of training data for both the agent model and WKM in Appendix E.D = {(u, κ, τ w ) (i) } |D| i=1with task knowledge κ generated in §3.1, we train the agent model to follow the task knowledge to generate actions. Under an auto-regressive manner, the loss of the agent model can be formulated as:EQUATIONSuppose X = (x 1 , x 2 , . . . , x |X | ) is the token sequence of the trajectory τ w , we have:EQUATIONHere 1(x j ∈ A) is the indicator function to mask tokens unrelated to actions. Please note that τ w here does not include the state knowledge mentioned in §3.2.World Knowledge Model Training. The main difference in the training data between the agent and knowledge model is the added state knowledge. Given the expert trajectories dataset with both task and state knowledgeD ′ = {(u, κ, τ ′ w ) (i) } |D ′ |i=1 where τ ′ w = (a 0 , o 0 , s 0 , . . . , a n , o n , s n ), the loss of the knowledge model π ϕ can be formulated as:EQUATIONSupposeX ′ = (x ′ 1 , x ′ 2 , . . . , x ′ |X ′ | )is the token sequence of the expert trajectory with state knowledge τ ′ w and Y = (y 1 , y 2 , . . . , y |Y| ) represents the token sequence of the task knowledge κ, we have:EQUATIONEQUATIONwhere 1(x j ∈ S) is the indicator function to mask tokens unrelated to state knowledge.At inference time, the agent model plans on the evaluation tasks with the aid of the world knowledge model. We redefine the historical trajectory h t = (u, κ, a 0 , o 0 , a u , . . . , α), at any time t ≥ 0, instead of directly generating a next action a t+1 ∈ A u based on h t , we first employ the world knowledge model to generate the current state knowledge s t ∼ π ϕ (•|h t ) and leverage s t to query the state knowledge base B = {(s, a pre , a next ) (i) } |B| i=1 . With the state knowledge as the key, we retrieve N nearest triplets from where a pre = a t based on semantic similarity and collect the corresponding next actions a next . We count the probability of each action p know (α(i) u ) = Ni N, where N i is the occurrence number of action α(i)u in all the collected a next . Therefore, we get the probability acquired from the state knowledge base:EQUATIONAfterward, we sample the probability distribution of the first token for each action α(i)u , 1 ≤ i ≤ |A u | from the last layer of the agent model and apply a softmax function to normalize the probability distribution. We define the probability acquired from the agent model as:EQUATIONFinally, we determine the next action by combining the above two probabilities:EQUATION)where γ is the hyperparameter that controls the proportion of P agent (A u ). Based on the above, we enhance the agent planning by global guidance from task knowledge and local constraints from state knowledge generated by our WKM. Due to the WKM and retrieval, the inference stage incurs additional time overhead compared to the pure agent model. The approximate ratio is around 2.5:1. 
experimental settings: 
 Datasets and Metrics. We evaluate our method on three real-world simulated planning datasets: ALFWorld [41] , WebShop [53] , and ScienceWorld [50] . AlFWorld and ScienceWorld include unseen tasks to evaluate the agent's generalization ability. The reward of ALFWorld is binary 0 or 1, indicating whether the agent has completed the task or not. WebShop and ScienceWorld provide dense rewards from 0 to 1 to measure the completion level of the task. For all the datasets, we apply average reward as the final metrics. Please refer to Appendix B for detailed dataset information.Models and Baselines. We evaluate on three state-of-the-art open-source models: 1) Mistral-7B [16] , the Mistral-7B-Instruct-v0.2 version. 2) Gemma-7B [24] , the Gemma-1.1-7B-it version. 3) Llama-3-8B [25] , the Meta-Llama-3-8B-Instruct version. We compare our method with two promptbased baselines: REACT [54] and Reflexion [40] . Besides, we adopt two strong baselines that introduce rejected trajectories into the training process to learn from experience: NAT [49] , learn from rejected trajectories through SFT, and ETO [44] , learn from rejected trajectories through DPO [36] . Moreover, we compare with a knowledge-augmented planning method KNOWAGENT. We also include ChatGPT (gpt-3.5-turbo-0125) [27] and GPT-4 (gpt-4-32K-0613) [28] for comparison. All the prompt-based baselines are tested under one-shot and all the fine-tuning-based baselines are trained with LoRA [12] . Please refer to Appendix C for baselines and re-producing details.Training and Inference Setups. We fine-tune the proposed approach with LoRA [12] using the LlamaFactory [62] framework. During training, the model is tuned after finishing the entire trajectory rather than each step of action. The learning rate is 1e-4 and the sequence length is 2048 for all the models. The training epoch is 3 and the batch size is 32. We adopt the AdamW optimizer [22] with a cosine learning scheduler. During inference, we apply the embedding layer of WKM as the encoder and use the cosine similarity between sentences for retrieval. The number of retrieved action-state-action triplets N is set to 3000 and the P agent (A u ) weight γ is set to {0.4, 0.5, 0.7}. All the training and inference experiments are conducted on 8 NVIDIA V100 32G GPUs within 12 hours. Please refer to Appendix D for detailed hyperparameters used in our paper.Main Results. As shown in Table 1 , for prompt-based baselines on open-source models, both RE-ACT and Reflexion exhibit poor performance, far behind our method and fine-tuning-based baselines on various datasets. GPT-3.5-Turbo performs ordinarily on two datasets other than WebShop, and it even falls behind Mistral-7B and Llama-3-8B's REACT performance on ScienceWorld. However, GPT-4 exhibits strong performance across various datasets. Nevertheless, our approach, through LoRA training alone, surpasses GPT-4 on ALFWorld (44.29→73.57 on seen, 38.05→76.87 on unseen) and WebShop (62.76→66.64). For fine-tuning-based baselines, both NAT and ETO fall behind our method, implying that just integrating world knowledge for agent models is worth more than further fussy SFT or DPO on negative examples. Our method also performs better than KNOWA-GENT which brings human-designed fixed action knowledge and long action paths into trajectories. This suggests the effectiveness of our WKM which is responsible for generating instance-level task knowledge and maintaining implicit action constraints. Furthermore, KNOWAGENT's performance on unseen tasks is not as impressive as on seen tasks, while WKM can keep its advantage. This phenomenon also demonstrates the generalization ability of WKM.Approach Ablations. As shown in Figure 3 , taking Mistral-7B as an example, we decompose the key components of WKM to examine the roles of the task and state knowledge separately. In a macro view, removing each module results in a clear drop in the agent's performance, which validates the power of our world knowledge. Furthermore, the improvement through task knowledge (w/ task) is more pronounced than that through state knowledge (w/ state), suggesting the necessity of global prior knowledge for agent planning. A more micro observation reveals that the impact of state knowledge is more significant on seen tasks compared to unseen tasks, while the influence of task knowledge is sustainable across seen and unseen tasks. This may be attributed that although our real-time state knowledge is generated by WKM, the state knowledge base is built on the training set, which may weaken generalization to some extent. Additionally, to validate our motivation of allowing the agent to learn task knowledge from both expert and generated trajectories, we exclude the rejected trajectories during the synthesis of task knowledge, instructing the agent model to synthesize knowledge solely based on the chosen trajectories. The results (w/o rejected) demonstrate that learning from the contrast between chosen and rejected trajectories is more effective than learning from chosen examples alone. This procedure is a little similar to DPO, but we achieve it through knowledge augmentation rather than directly converting it into a loss calculation between chosen and rejected trajectories. Additional results can further evident that training a WKM separately performs better than training one single model together with the agent model as well as using few-shot prompts to replace WKM for providing knowledge.World knowledge can mitigate blind trial-and-error and reduce hallucinatory actions. We compare the number of planning steps for each dataset between three strong baselines and WKM and calculate the average steps of each method. As depicted in Figure 9 (in Appendix F), WKM demonstrates the ability to complete a significant proportion of tasks using the shortest trajectory, indicating that guidance from world knowledge can effectively reduce the agent's blind trial-and-error in the environment. Taking a further perspective from an average standpoint in Table 2 , it can be observed that WKM exhibits lower average planning steps compared to other baselines. As ALFWorld can respond to invalid actions, in Table 3 , we count the percentage of hallucinatory actions that occurred in trajectories from ALFWorld for each method. The results confirm the effectiveness of our world knowledge model to decrease hallucinatory actions. Furthermore, it is worth noting that most baselines show a prominent increase in the average number of steps and percentage of invalid actions when transitioning from seen tasks to unseen tasks, but WKM can still maintain a relatively low level. This reflects laterally that our world knowledge can still effectively guide the agent model on unseen tasks, highlighting the knowledge generalization brought by the world knowledge model.To see how our world knowledge works, please refer to our case study in Appendix H. Our instance-level knowledge can generalize better to unseen tasks. To further explore the benefit of using a knowledge model to generate instance-level task knowledge, we carefully survey the task knowledge generated by our WKM and abstract it into dataset-level knowledge for each dataset. Then we retrain the agent model to adapt to new dataset-level knowledge 5 . As illustrated in Figure 4 , we compare the performance of dataset-level knowledge with our instance-level task knowledge (WKM w/o state) on ALFWorld and ScienceWorld. It can be observed that our model-generated instancelevel knowledge not only surpasses human-designed knowledge on seen tasks but also exhibits even more remarkable performance on unseen tasks, with the improvement in performance on unseen tasks significantly greater than that on seen tasks. This phenomenon straightly reflects the strong generalization ability of our knowledge model compared to rigidly designed knowledge by humans. We choose Mistral-7B as the backbone of the knowledge model and ChatGPT and GPT-4 as the agent model. Since we cannot get the token distribution from OpenAI API, we only apply task knowledge to the agent model. As exhibited in Table 4 , the results of both ChatGPT and GPT-4 show distinct advances after being guided by the Mistral-7B world knowledge model, indicating the weak world knowledge model also contains knowledge that the strong model may lack. In the era of LLMs, this inspires us with a new agent learning paradigm: weak-guide-strong. Due to its lightweight nature, the weak knowledge model can flexibly adjust its parameters based on the needs of the agent model, which can address the difficulty of large agent models in adapting to new environments through fine-tuning. Unified World Knowledge Model Training. We mix the world knowledge collected from all three datasets and jointly train one single world knowledge model to investigate the effect of multi-task world knowledge learning. Figure 5 illustrates the relative performance comparison between multi-task WKM and various baselines, from which we can observe that multi-task WKM not only does not lead to performance degradation but also exhibits visible improvements compared to single-task WKM, especially on WebShop and ScienceWorld. Similar to [57, 58, 3] which endeavor to train a unified agent model and achieve strong generalization ability to held-out tasks, this observation inspires us with the potential of training a unified world knowledge model that can be applied to help various held-in agent models and also generalize to guide held-out agent models. A more daring idea is whether a unified agent model combined with a unified world knowledge model is the key to Artificial General Intelligence (AGI). Explicit state knowledge will hurt the planning performance.To demonstrate the rationality of our choice to construct a state knowledge base, we explore the effect of directly incorporating state knowledge into the context of the agent model (we retrain the agent model to follow both the task and state knowledge), as shown in Figure 6 . The performance of explicit state knowledge is far inferior to our approach of retrieving from a state knowledge base and utilizing probabilistic constraints. It even performs worse than when we remove state knowledge and only include task knowledge. This clearly indicates that blindly extending prompts with a large amount of explicit natural language feedback is lose-more-than-gain for agent planning, and implicit knowledge constraints may be sometimes more prudent.Case Study. In Figure 10 (Appendix H), we list the trajectories of ETO and our WKM within the same task in ALFWorld to illustrate how world knowledge functions. The rationales before each action have been omitted to guarantee a clear illustration. The task is to clean some soapbar and put it in cabinet. Initially, ETO blindly searches for the soapbar in the countertop and cabinet, introducing a lot of irrelevant information and unnecessary context. In the later stages of planning, ETO experiences the hallucination and executes the put action after close the cabinet, causing the environment to become unrecognizable and resulting in a collapse. On the contrary, guided by task knowledge, WKM directly identified the possible locations of the soapbar and successfully found it in the first attempt. Subsequently, WKM efficiently completed the task with precision, adhering to the constraints of state knowledge. 
related work: 
 LLM Agents. LLMs have emerged as a promising avenue towards unlocking the potential of Artificial General Intelligence, offering robust support for the development of agent systems [48, 51, 8, 63] . Existing works in this field mainly focuses on agent planning [14, 21, 54, 42] , external tools harnessing [39, 23, 43, 29, 32, 35, 46] , code generation [45, 21, 31, 11] , etc. Recently, there has been an increasing focus on endowing open-source LLMs with agent functionalities through fine-tuning [2, 57, 56, 38, 44, 49] . However, these approaches rely on blindly fitting the probabilities of tokens to learn planning, without having an intimate cognition of the environment. The lack of knowledge can lead to the agent blindly attempting trial-and-error and generating hallucinatory actions.Knowledge Augmented Agent Planning. Planning [15] is a crucial capability for intelligent agents to accomplish real-world tasks, often requiring agents to possess rich knowledge and environmental commonsense. Few works have explored the field of knowledge-augmented agent planning. [14, 61, 5] utilize the rich parametric knowledge stored in pre-trained language models to assist agent planners. [7, 20, 59, 64] design structured or natural language knowledge to regulate the actions. However, the above studies require the manual design of fixed prompt templates or task procedures, making it challenging to transfer across different task environments. [63, 55, 6] propose the automation of knowledge generation using language models. However, their knowledge either consists of only global workflow or only local action principles. In contrast, we train our world knowledge model both on global task knowledge and local state knowledge to assist agent planning, and these knowledge sources are derived from the model's self-summary rather than hand-curated.LLM-based World Model. World model and agent model often co-occur in the domain of reinforcement learning and robotics [13, 9, 19, 37, 26, 4] . With LLMs commonly deemed as the most powerful intelligent machines constructed by humans thus far, the LLM-backed world models have been proposed [61, 10, 13] . In our paper, we attempt to self-synthesize world knowledge and train to obtain a world knowledge model. However, we consider our model to be a world knowledge model rather than a world model based on the reason that our model is temporarily unable to utilize search algorithms (e.g. MCTS) in conjunction with the agent model to make predictions about the world and we leave this for our future work. 
conclusion and future work: 
 In this paper, we strive to develop a parametric world knowledge model (WKM) to augment language agent model planning. Our WKM can generate prior task knowledge to guide global planning as well as dynamic state knowledge to regulate local planning. Our extensive results show that our world knowledge can work on both GPT-4 and state-of-the-art open-source models and achieve superior performance compared to various strong baselines. Analytical experiments validate that our WKM can 1) reduce brainless trial-and-error and invalid actions, 2) generalize better to unseen tasks, 3) achieve weak-guide-strong, and 4) be effectively extended to unified world knowledge training. Potential future directions include: 1) building a unified world knowledge model, 2) learning to predict the world like a world model, 3) applying to multi-modal agent planning, etc. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper does not have high risk PLMs or datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited all the corresponding papers.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Despite our best efforts, this paper still has some limitations: 1) Our primary intention behind designing the WKM is to compensate for the lack of world knowledge in the agent model. However, determining what a language model knows and doesn't know has been an ongoing challenge that remains unresolved. 2) It is widely acknowledged that world knowledge extends beyond textual representations. While our world knowledge is currently limited to textual information, exploring multi-modal world knowledge models is indeed one of our important future tasks. 3) Our world knowledge model cannot dynamically update with the changes of the world and feedback from the agent. 4) Generating world knowledge can introduce additional inference overhead.We mainly use the expert trajectories with a REACT-style [54] collected from [44] :1. ALFWorld [41] . The dataset provides human-annotated trajectories.2. WebSahop [53] . Except for human-annotated trajectories, GPT-4 is also applied to generate trajectories with a reward larger than 0.7 being reserved.3. ScienceWorld [50] . The dataset offers heuristic algorithms to search golden trajectories for each sub-task.Since the original golden trajectories do not contain rationales, GPT-4 is further leveraged to generate the corresponding information.We evaluate our method on three real-world simulated agent planning datasets: ALFWorld [41] , WebShop [53] , and ScienceWorld [50] .1. ALFWorld is a household dataset requiring the agent to navigate through the room and manipulate objects. Except for seen tasks, AlFWorld also includes unseen tasks to evaluate the agent's generalization ability. The reward of ALFWorld is binary 0 or 1, indicating whether the agent has completed the task or not.WebShop is an online shopping dataset in a website environment. It provides dense final rewards from 0 to 1 to measure the completion level of the task.3. ScienceWorld is a scientific reasoning dataset which is at the level of a standard elementary school science curriculum. It also possesses both seen and unseen parts and a dense reward function from 0 to 1.For all the datasets, we apply average reward as the final metrics. Table 5 illustrates the statistics of each dataset. Here we detailedly introduce the baselines we compare with and our re-produce details.1. REACT [54] . The first approach incorporates Chain-of-Thought (COT) prompting in agent planning tasks with a format of Thought-Action-Observation loop. In our paper, we apply one-shot prompting for REACT 6 .2. Reflexion [40] . A strong prompt-based baseline reinforces agent planning with verbal feedback. Manually designed prompts are used to enable the agent to reflect on the historical trajectory and re-plan based on the feedback. In our paper, we utilize one-shot prompting for reflection and select the first reflect iteration as our result due to limited context7 .3. NAT [49] . NAT includes negative trajectories by employing different prompts during agent fine-tuning. When evaluating, only positive prompts are used to encourage the language agent to generate correct trajectories. As it also follows the REACT-style format, we directly use the default positive and negative prompts and train with LoRA in our paper8 .4. ETO [44] . Another baseline includes negative trajectories during agent training. The method contains two training phases, of which the first phase is behavior cloning which fine-tunes the agent on expert trajectories, and the second phase is learning from failures which further fine-tunes the agent through Direct Preference Optimization (DPO) [36] . In our paper, we remove the one-shot prompt for fairness and retain all the default hyperparameters proposed in ETO except for LoRA training 9 .5. KNOWAGENT [64] . KNOWAGENT is a knowledge-augmented agent planning baseline that applies action knowledge in the prompt and maintains an action path in the context during planning to constrain the agent's action. We directly use the default prompt mentioned in KNOWAGENT for ALFWorld and carefully extend it to WebShop and ScienceWorld by following a similar format 10 .All the prompt-based baselines are tested under one-shot and all the fine-tuning-based baselines are trained with LoRA [12] .The detailed hyperparameters we use during training and inference are shown in Table 6 . We employ identical hyperparameters for different models. The temperature of the agent model is set to 0.0 when conducting exploration and 0.5 when introduced into WKM. The temperature of WKM is set to 0.0 for all the time. The P agent (A u ) weight γ is set to 0.4 for ALFWorld, 0.5 for WebShop, and 0.7 for SienceWorld. We show the training data example for agent model and world knowledge model in Figure 7 and Task Instruction: You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 1, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a dresser 1, a garbagecan 1, a laundryhamper 1, and a shelf 1. Your task is to: examine the book with the desklamp.Task Knowledge: When you cannot find the object required for the task under the desklamp, first check all the drawers and shelves in the room, then revisit the previous locations and double-check before giving up and looking in the garbage can or laundry hamper. The action workflows are as follows: 1. For each drawer and shelf, go to the location and open it to check if the object is inside. 2. If the object is not found, close the drawer or shelf and move on to the next one. 3. After checking all the drawers and shelves, revisit the previous locations and double-check if the object might have been overlooked. 4. If the object is still not found, look in the garbage can or laundry hamper as a last resort.Thought: The task requires me to examine a book with the desklamp. First, I need to locate the book. Action: go to bed 1 Observation: On the bed 1, you see a book 1, a laptop 1, and a pillow 1.Thought: I see a book on the bed. I should pick it up to examine it with the desklamp. Action: take book 1 from bed 1 Observation: You pick up the book 1 from the bed 1.Thought: Now that I have the book, I need to find the desklamp to examine the book under its light. Action: go to desk 1 Observation: On the desk 1, you see an alarm clock 1, a bowl 1, a CD 2, a mug 3, a mug 2, a mug 1, a pen 1, and a pencil 2.Thought: The desklamp isn't on the desk. I need to look around the room to find where the desklamp is located. Action: go to dresser 1 Observation: On the dresser 1, you see a cellphone 1, a desklamp 1, and a keychain 2. Thought: I've found the desklamp on the dresser. I should use it to examine the book. Action: use desklamp 1 Task Instruction: You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 1, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a dresser 1, a garbagecan 1, a laundryhamper 1, and a shelf 1. Your task is to: examine the book with the desklamp.Task Knowledge: When you cannot find the object required for the task under the desklamp, first check all the drawers and shelves in the room, then revisit the previous locations and double-check before giving up and looking in the garbage can or laundry hamper. The action workflows are as follows: 1. For each drawer and shelf, go to the location and open it to check if the object is inside. 2. If the object is not found, close the drawer or shelf and move on to the next one. 3. After checking all the drawers and shelves, revisit the previous locations and double-check if the object might have been overlooked. 4. If the object is still not found, look in the garbage can or laundry hamper as a last resort. Thought: The task requires me to examine a book with the desklamp. First, I need to locate the book. Action: go to bed 1 Observation: On the bed 1, you see a book 1, a laptop 1, and a pillow 1.State Knowledge: Your task is to examine a book with the desklamp. You have not found the desklamp yet, and have only checked the bed 1.Thought: I see a book on the bed. I should pick it up to examine it with the desklamp. Action: take book 1 from bed 1Observation: You pick up the book 1 from the bed 1.State Knowledge: Your task is to examine a book using the desklamp. Your have located the book on bed 1 and picked it up. Thought: Now that I have the book, I need to find the desklamp to examine the book under its light. Action: go to desk 1 Observation: On the desk 1, you see an alarm clock 1, a bowl 1, a CD 2, a mug 3, a mug 2, a mug 1, a pen 1, and a pencil 2.State Knowledge: Your task is to examine a book with the desklamp. You have located the book on bed 1 and picked it up. Next, You need to find the desklamp to examine the book under its light. Thought: The desklamp isn't on the desk. I need to look around the room to find where the desklamp is located. Action: go to dresser 1 Observation: On the dresser 1, you see a cellphone 1, a desklamp 1, and a keychain 2.State Knowledge: Your task is to examine a book with the desklamp. You have located the book on the bed and picked it up, now you find a desklamp on a dresser. See Figure 9 . 0In fact, the ratio γ can be viewed as a signal to reflect whether knowledge or planning is more important for a task. To understand which part of the output action has the most significant impact, we further analyze γ = 0 (fully trust state knowledge base) and γ = 1 (fully trust agent model, equivalent to remove state knowledge in Figure 3 ). The empirical results can be seen in Table 7 . It can be observed that state knowledge primarily serves as a constraint to alleviate hallucinated actions for the agent model. However, when we fully trust it (γ = 0), its lack of generalization significantly harms the performance of the agent model.In Figure 10 , we list the trajectories of ETO and our WKM within the same task in ALFWorld to illustrate how world knowledge functions. The rationales before each action have been omitted to guarantee a clear illustration. The task is to clean some soapbar and put it in cabinet. Initially, ETO blindly searches for the soapbar in the countertop and cabinet, introducing a Table 7 : Impact of γ. In fact, the ratio γ can be viewed as a signal to reflect whether knowledge or planning is more important for a task. To understand which part of the output action has the most significant impact, we further analyze γ = 0 (fully trust state knowledge base) and γ = 1 (fully trust agent model, equivalent to remove state knowledge in Figure 3 ). It can be observed that state knowledge primarily serves as a constraint to alleviate hallucinated actions for the agent model. However, when we fully trust it (γ = 0), its lack of generalization significantly harms the performance of the agent model. lot of irrelevant information and unnecessary context. In the later stages of planning, ETO experiences the hallucination and executes the put action after close the cabinet, causing the environment to become unrecognizable and resulting in a collapse. On the contrary, guided by task knowledge, WKM directly identified the possible locations of the soapbar and successfully found it in the first attempt. Subsequently, WKM efficiently completed the task with precision, adhering to the constraints of state knowledge.In this section, we illustrate all the prompts used in our paper. Part of our prompts are taken from AutoGuide [6] .See Figure 11 .See Figure 12 .See Figure 13 .This research was conducted following the ethical standards and best practices. All our experiments use publicly available datasets (as detailed in Appendix B), avoiding ethical concerns related to privacy, confidentiality, or misuse of personal biological information. However, despite our best efforts, it is not avoidable if someone maliciously modifies the world knowledge model to contradict the world's knowledge and leads the agent to engage in unethical behavior.Task Knowledge exampleWhen picking an object, heat it, and place it, you should first go to the possible locations of the object, then take the object, heat it with microwave, and put it in place. The action workflows are as follows:1) go to receptacle 2) take object from receptacle 3) heat object with receptacle 4) go to the place to put the object 5) put object in/on receptacle 2) click the most relevant tag to view the detailed product page.3) check the product details one by one, like color, size, type, and price, and make sure the price is within budget. 4) if find the right items, click[buy now] to buy it.When tasked with boiling apple juice, focus on locating the kitchen first. Then, locate the apple juice in the fridge. Activate the stove, pour the apple juice into a metal pot, and move the metal pot to the stove. Monitor the stove until the apple juice reaches a boiling point. Once boiled, remove the pot from the stove. The action workflows are: 1) teleport to the kitchen.2) look around to find the apple juice in the fridge.3) activate the stove. 4) pour apple juice into a metal pot. 5) move the metal pot to the stove. 6) look at stove. 7) examine apple juice to confirm boiling. 8) repeat step 6,7 until apple juice is boiled. Answer: [Yes] Justification: We include our source code and data in the supplemental material submission, and we outline the data generation procedure, the evaluation protocol, the training regime, and everything else necessary for reproduction either in the main body of the paper or in the appendix.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).For details on how to collect expert trajectories, please refer to Appendix A.Detailed manually designed dataset-level knowledge prompt can be found in Appendix I.3https://github.com/ysymyth/ReActhttps://github.com/noahshinn/reflexionhttps://github.com/Reason-Wang/NAThttps://github.com/Yifan-Song793/ETOhttps://github.com/zjunlp/KnowAgent 


Decision: The authors provide task knowledge and state knowledge to improve global and local planning in smaller LLMs and shows promising generalization. The paper is well written with many experiments and additional analyses that strengthens their novel approach. The authors and reviewers have engaged in thorough discussions, and in spite of some disagreement between the authors and one of the reviewers, the paper is recommended for acceptance as a poster, conditional on integrating the new results and responses to reviewers in the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces a parametric World Knowledge Model (WKM) to enhance agent planning by integrating both global task knowledge and dynamic state knowledge. The authors claim that their approach can mitigate issues like blind trial-and-error and hallucinated actions in large language model (LLM) agents. They demonstrate the effectiveness of WKM through experiments on three real-world simulated datasets (ALFWorld, WebShop, and ScienceWorld) using state-of-the-art LLMs, showing superior performance compared to strong baselines.
strength_weakness: strength_weakness: My primary concerns with this paper are related to the methodology's implementation and the sufficiency of the experiments. Specific issues are detailed in the following questions.
questions: questions: Methodology:

1. The motivation for rejecting trajectories is not sufficiently justified. (1) Why not and What if directly derive task knowledge from expert trajectories?  (2) Why assume that agent-generated trajectories are always inferior to the dataset trajectories? Could they not be better, and what would be the impact of this assumption? (3) Even if the above assumption holds, if agent-generated trajectories are always rejected, how can the authors claim, "Our purpose is to extract superior task knowledge that cannot be acquired solely through supervised fine-tuning on chosen trajectories, thus further effectively boosting the agent’s capabilities" (line 107-108)? (4) Ignoring the previous issues, in Line 104, the authors train an experienced agent to generate reject trajectories. However, trajectory generation requires an environment and an experienced agent. How can trajectories be generated without training the environment model? Is that done by interacting with the environment?

2. The definition of state knowledge is ambiguous. Lines 119-120 suggest that state knowledge is a local summarization of the policy function to instruct actions, but lines 123-124 define it as part of the MDP's state space. Besides, Figure 12's prompt also does not define state knowledge yet asks the LLM to generate it (How can an LLM generate knowledge without a definition?)

3. The knowledge model is trained to output task knowledge and state knowledge using data directly labeled by the LLM through prompts. Why not directly use these prompts to output task knowledge and state knowledge instead of retraining a model?

Experiments:

1. Can the paper "AutoGuide: Automated Generation and Selection of State-Aware Guidelines for Large Language Model Agents" be a baseline of this paper?
2. Can the authors compare results with gamma=1 and gamma=0? It is essential to understand which part of the output action has the most significant impact.
limitations: limitations: NA
paper_summary: paper_summary: The paper presents a parametric World Knowledge Model (WKM) to enhance agent planning by providing both global prior task knowledge and local dynamic state knowledge. 
Traditional LLMs often perform trial-and-error actions and generate hallucinatory actions due to their limited understanding of the physical world. 
By imitating human cognitive processes, the WKM synthesizes knowledge from expert and sampled trajectories to guide agents more effectively. 
Experimental results with state-of-the-art LLMs (Mistral-7B, Gemma-7B, and Llama-3-8B) show that this method improves performance on complex real-world tasks and reduces blind trial-and-error and hallucinatory actions.
strength_weakness: strength_weakness: The approach heavily relies on expert trajectories to synthesize both task and state knowledge. It takes a lot of effort to obtain such a database and pre-train/fine-tuning and off-line WKM. 

WKM depends on the world dynamic as well as human demonstration.  In my opinion, the three "real-world simulated planning datasets":
ALFWorld, WebShop, and ScienceWorld are not all real. The ALFWorld and ScienceWorld are simulated environments with a discreet Embodied mechanism. The methodology of WKM may not be easily adapted to problem-solving in real scenarios.
questions: questions: See weeknesses
limitations: limitations: no potential negative societal impact
paper_summary: paper_summary: This work is concerned with LLMs planning abilities in agent datasets. Instead of only fine-tuning the agent model on expert trajectories, they add “task knowledge” information. This information is created by comparing reject trajectories and expert trajectories, following previous work (NAT; Wang et al., 2024). In addition, the agent model is prompted to summarize the state, which helps avoid generating invalid actions.
strength_weakness: strength_weakness: - When removing the state knowledge (figure 3), it seems that this approach does not outperform NAT, which uses SFT on the same trajectories preference data.
    
- Missing qualitative analysis of the generated task knowledge.
questions: questions: **Suggestions**
* Case study in Appendix F is very important to understand the paper, providing intuition and a qualitative analysis which explains why preference trajectories data is useful. I would put it in the main paper to guide the reader.

**Questions**
* Line 135 mentions that the WKM and the agent are the same backbone model. What about the LM that generates the reject trajectories, is it the same?
limitations: limitations: The authors discuss limitations of their work.
paper_summary: paper_summary: This paper presents a parametric world knowledge model designed to enhance agent planning. The model synthesizes knowledge from expert and sampled trajectories for training purposes. It incorporates prior task knowledge for global planning and dynamic state knowledge for local planning. The implementation shows improved performance over several robust baselines using open-source LLMs such as Mistral-7B and Gemma-7B. Incorporating a world knowledge model into LLM-based agents for planning purposes is a novel approach that also helps mitigate common issues like hallucinations and invalid actions in language agents.
strength_weakness: strength_weakness: Here are some additional points that I believe could be improved: 

1. Clarity on hyperparameters and settings: Is the WKM tuned after each step of action or finishing an entire trial? Also what is the split for seen/unseen task? Some tasks are very similar in Alfworld and Scienceworld. What is the structure of the retriever? What is the number of generations for each action? 

2. Is WKM training more important for commmon-sense intensive tasks like Scienceworld? Have you previously tested on pure planning tasks? e.g. Blocksworlds, Game 24.  Also, does the ratio $\gamma$ reflects whether knowledge or planning is more important for a task? 

3. What is the additional computation overhead for inferencing using a WKM? Also beam search is necessary to obtain more than one action, what is the computation time compared to ReAct? 

4. What is the difference between state knowledge and thoughts/ reflection? It seems to be also interleaved between actions. Does the content of state knowledge greatly affect the performance? e.g. using the WKM to enhance ReAct thoughts. The state-knowledge examples in Figure 8 seems more similar to thoughts rather than world knowledge, i.e. give a general commonsense about the environment.

5. Can WKM and agent training be merged together? i.e. using a single model to learn both knowledge and trajectory and update the loss together. 

6. Can this method be applied to an online setting? Where knowledge base is directly updated after each step of action so that later actions could use results from exploration.
questions: questions: Please refer to weaknes.
limitations: limitations: Yes.

Comments: title: Thanks for your valuable feedback!
Thanks for the rebuttal
Thank you for your in time feedback!
Thank you for your patient response!
Thank you for your continuous feedback!
Thanks for your response!
Response to rebuttal
comment: Thank you for your reply and the recognition of our work. Your feedback is very important for us to further improve our paper. Thank you once again.
Your rebuttal has clearly addressed my concerns and helped me gain a better understanding of world knowledge training. I have adjusted the score accordingly.
We are very delighted that our response can address your concerns, and we sincerely appreciate your recognition of our work!

We have indeed realized that in the ablation study, the "w/o" may convey a clearer message than "w/", and we will make a corresponding revision to further enhance the readability of our paper.

Thank you again for your prompt feedback!
Dear reviewer,

We greatly respect your position, but we have to say that we still insist on our viewpoint. In the era of deep learning, we cannot say that a method cannot be used because it may lack some interpretability despite being effective. Moreover, **our goal is to train a World Knowledge Model that provides both task knowledge and state knowledge**. High-quality task knowledge requires comparison between positive and negative examples to be obtained, which cannot be provided solely through prompts (as we have demonstrated in our supplementary experiments 1 and 2, our comparison with KnowAgent, and our analysis in Figure 4). Therefore, **it would be contrary to our approach of training a unified WKM if we were to train a model to provide task knowledge while prompting another model to provide state knowledge**.

With still two days remaining until the end of the discussion period, we also warmly welcome any further questions you may have. At the same time, we are very grateful for your patient review of our responses during the rebuttal phase. Thank you once again!
As the rebuttal period is concluding, I would like to finally point out that self-training should not be interpreted in the same manner as the authors propose. Aligning with the notations in our discussion, for [3], in order to label $y$ in $D=\\{ (X,y) \\}$ for fine-tuning, (1) they filter the generated rationales ($y$ includes rationales followed by an answer in this setting) from model $A$ to retain only those that result in correct answers; (2) for incorrect answers, they provide the correct answer as a hint to the model and ask model $A$ to generate rationales in the same style as in the previous rationale generation step, instead of using the label $y$ generated by model $A$ directly. For [6], they employ RL optimization, where rewards are assigned by model $A$ based on the label $y$ generated by model $A$. This approach is still intuitive, as studies have shown that a model's evaluation capability generally surpasses its generation capability [7]. Similarly, they do not use the label $y$ generated by model $A$ directly for fine-tuning.

I would like to thank the authors' detailed response and the efforts made in the rebuttal. At this point, I maintain my original concerns; however, I am open to reconsidering my evaluation based on the suggestions from the Area Chairs.

[7] Language Model Self-improvement by Reinforcement Learning Contemplation
Here is our further clarification of your questions:

1. We sincerely apologize for any confusion and we need to sincerely say that we still have not understood why you suggested that our statement could reason that DPO should perform better on seen tasks. We need to reiterate that our method enables the model to read both correct and incorrect trajectories for the same task, using $\rho_{\rm TaskKnow}$ (detailed in Figure 11) to summarize the knowledge why the correct trajectories are better than the incorrect ones, and training the WKM to learn to generate this kind of knowledge to augment the agent model. On the other hand, DPO directly trains the agent model by minimizing the loss (increasing the distribution gap between correct and incorrect trajectories) to favor the generation of correct trajectories. Therefore, we believe there is no direct relationship or conflict between the two approaches. Our previous statement of "similar to DPO" might have caused misunderstanding; it only proves that the improvement of our agent model augmented by synthetic task knowledge is better than training the agent model with DPO.

2. **(1)** As mentioned in lines 122-123, the prompt $\rho_{\rm StateKnow}$ that we use to summarize state knowledge is detailed in Appendix I.2, which is displayed in Figure 12. In response to Q2, we have clarified to you that the key part of our prompt is the State Knowledge Example (colored in purple), which is actually the few-shot examples. Therefore, we stated that generating state knowledge directly with Equation (5) is equivalent to the experiment we supplemented in Q3, where knowledge was generated directly using few-shot examples. As shown in our previous table, while it is certainly possible to generate knowledge by few-shot examples, it is clear that its effectiveness is not as good as that of the WKM.


   **(2)** Regarding the question you raised about why training Model A with data annotated by Model A itself can lead to improvements for A, **this issue has been extensively validated in the field of LLM synthetic data (also known as self-training) [1][2][3][4][5][6]**. The key to self-training lies in ensuring the quality of self-generated data. For instance, [3] achieves this by implying the correct labels of the data to the model, while [6] relies on the model's self-judgment capability to assess the quality of the data. Our approach ensures the quality of synthesized knowledge by meticulously designing few-shot examples and using entirely correct expert trajectories. The advantage of this method is that it does not depend on a large amount of manually annotated data or powerful closed-source models (e.g., GPT-4), fully leveraging the model's own potential. However, this method usually has an upper-bound because we cannot guarantee that the synthesized data is 100% correct. As the model generates more data, the diversity of the data is also likely to decrease. This is a key research direction in the field of data synthesis today, that is, how to improve the diversity and quality of synthesized data. The principle behind self-training is still under investigation; we speculate that it may be because the data annotated by the model itself is more in line with the "data distribution" understood by the model. From the model's perspective, this distribution may be smoother. Although this method may not be as effective as training on fully human-annotated data, the most important aspect is that it solves the resource consumption brought about by a large amount of manual data annotation. And if one day human-annotated data is exhausted or LLM grows into a super model that humans cannot supervise, we can only rely on the data synthesized by the model itself to improve the ability of LLM, so this is a very promising research direction in the LLM community.

   [1] On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey.

   [2] KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents.

   [3] STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning.

   [4] Best Practices and Lessons Learned on Synthetic Data for Language Models. (Google DeepMind)

   [5] Self-training Language Models for Arithmetic Reasoning.

   [6] Self-Rewarding Language Models.

We look forward to your further reply and would like to continue the discussion.

Thanks again!
Thank you for the prompt response.

Follow-up question for Q1: I would like to clarify my question again. This question follows up on your statement: "Therefore, we let the experienced agent run through the training data again to generate explored trajectories, so that the agent can summarize knowledge that cannot be learned solely through SFT. This step is somewhat similar to DPO, but we achieve it through knowledge augmentation rather than directly converting it into a loss calculation between chosen and rejected trajectories." It is somewhat intuitive, but following this line of reasoning, using DPO directly should yield better results compared to the WKM variant w/o state knowledge, i.e., WKM w/ task in your ablation studies, at least in the seen tasks. However, it seems contradictory that we observe the opposite result.

Follow-up question for Q3: Could the authors clarify why Equation (5) (with your redefined ${h_t}$) cannot be used to output the state knowledge? Additionally, I do not grasp the idea of few-shot prompts, as using Equation (5) to output state knowledge does not necessarily depend on them. Generally speaking, why can we assert that using model ${A(y|X)}$ to label a dataset ${D=\{(X,y)}\}$, and then fine-tuning ${A}$ with ${D}$ to obtain ${A'}$, results in a model ${A'}$ that is superior to ${A}$?
We are very fortunate that our rebuttal could address some of your questions and are extremely grateful for the opportunity to engage in further discussion with you. In response to your follow-up questions, we would like to provide the following explanations:

**Follow-up question for Q1**

You may consider our approach to be somewhat similar to the idea behind DPO, but while DPO optimizes the model by comparing the losses between chosen and rejected trajectories, requiring the memorization of both types of data, our method contrasts chosen and rejected trajectories to summarize knowledge. During training, the model only needs to learn from this knowledge without the additional requirement to memorize the rejected trajectories. This also validates that our knowledge enhancement allows the agent model to acquire knowledge that cannot be learned through SFT loss calculations.

**Follow-up question for Q3**

We would like to address this question from two perspectives:

1. The notation $h_t$ in Equation (5) is defined according to Equation (1), which does not include the task knowledge $\kappa$. However, **as mentioned in line 152, we have redefined $h_t$ to include the task knowledge $\kappa$**. Therefore, the inputs defined in Equation (5) and line 157 are different. We apologize if the redefinition of $h_t$ was not clear and may have caused some confusion. We will emphasize this in the revision by bolding it to improve readability.

2. **Even if we disregard the changes of $h_t$, as we clarified in our rebuttal Q2, the prompt to summarize state knowledge ($\rho_{\rm StateKnow}$) in Equation (5) are essentially few-shot examples.** This aligns with the experimental setup we supplemented in the rebuttal for Q3 if we directly use $\rho_{\rm StateKnow}$ at the inference time, and the results of our experiments have demonstrated that the performance of providing knowledge through prompts is poor.

**For Additional Suggestions**

We sincerely apologize for any confusion caused in your reading.

1. By separating the synthesis of task knowledge into two processes, our initial intention was to clarify the logic, but we did not anticipate the difficulties it might cause for you. We are truly sorry for this oversight. As we have clarified, we will integrate lines 102-117 in the revision to achieve better readability and make it easier for readers to understand.

2. Initially, we defined the state knowledge within the state space to facilitate comprehension for readers. This definition might indeed lack rigor, and we will redefine it with another symbol similar to the task knowledge in the revision to ensure clearer expression. We are genuinely sorry for any inconvenience this has caused you!

Once again, we greatly appreciate your feedback. Your comments are invaluable to the improvement of our work.

**We look forward to your further reply and would like to continue the discussion. Would you kindly consider raising your rating score for our paper if you find that our response addresses some of your concerns? We greatly appreciate your consideration.**
Thank you for your response, which addressed some of my concerns.

**Follow-up question for Q1:**
Thank you for the clarification. I have a follow-up question: Why does the method with task knowledge *only* perform better than the DPO-style baselines even in seen tasks? Since your expert trajectories are optimal, at least in the seen tasks, the fine-tuning method should be the better one as it only needs to memorize the given data.

**Follow-up question for Q3:**
I would like to clarify my question. In Equation (5), state knowledge is constructed using LLM by prompting with $\rho_{stateKnow}$ and $h_t$. However, in Line 157, state knowledge is generated from $\pi_\phi(\cdot|h_t)$, i.e., the World Knowledge Model. Why not use the same method as in Equation (5) directly in Line 157 as they share the same inputs?

**Additional Suggestions:**
After the author's clarification, I understand more about the methodology. However, I find the writing to be informal. Despite the trend towards more relaxed expressions, I believe that for a NeurIPS paper, the presentation should be more rigorous. For example, 
1. "Subsequently, the experienced agent explores the training set tasks again to generate rejected trajectories. Our purpose is to extract superior task knowledge that cannot be acquired solely through supervised fine-tuning on chosen trajectories, thus further effectively boosting the agent’s capabilities." As clarify by the authors, there is a significant logical gap between these two sentences. Generating rejected trajectories alone does not lead to extracting superior task knowledge that cannot be acquired through supervised fine-tuning on chosen trajectories. 
2. "The state knowledge serves as the dynamic knowledge to constrain the agent model’s local planning and prevent it from generating hallucinatory actions." If you are talking about MDP, it is incorrect to say the state constrains the agent’s ... abilities. The state is just defined to provide full information of the environment for the policy to make decisions [1], rather than enhancing or constraining the agent’s abilities.

[1] Reinforcement Learning: An Introduction.
I appreciate the authors' attention to my concerns and their efforts to clarify any misunderstandings.

Concerning my initial issue, I believe the confusion arose because Figure 3 was labeled "w/ state" instead of "w/o task," which led me to misinterpret the results. The labels in the table provided by the authors offers a clearer perspective in my opinion, and I now recognize that their approach without state indeed outperforms NAT.

I have adjusted my score to reflect my revised understanding.