LLM-Based World Models Can Make Decisions Solely, But Rigorous Evaluations are Needed
Authors: Chang Yang
Xinrun Wang
Junzhe Jiang
Qinggang Zhang
Xiao Huang
Venue: Accepted by TMLR ()

Abstract: World model emerges as a key module in decision making, where MuZero and Dreamer achieve remarkable successes in complex tasks. Recent work leverages Large Language Models (LLMs) as general world simulators to simulate the dynamics of the world due to their generalizability. LLMs also serve as the world model for deliberative reasoning in Reasoning via Planning (RAP) and Tree of Thought (ToT). However, the world model is either evaluated as a general world simulator, or as a functional module of the agent, i.e., predicting the transitions to assist the planning. This paper argues that LLM-based world models can make decisions solely, but rigorous evaluations are needed. We first present the two key observations to showcase how LLM-based world models can make decisions solely, and then present the three key observations to demonstrate why current evaluation framework of LLM-based world models is not sufficient. Then, we present our suggested evaluation framework: policy verification, action proposal, and policy planning, where the world model is used for decision making solely, and finally we leverage the 31 diverse environments from (Wang et al., 2023; 2024) and curate the rule-based policy of each environment for diverse evaluations. The key findings include: i) GPT-4o significantly outperforms GPT-4o-mini on the three main tasks, especially for the tasks which require the domain knowledge, e.g., scientific tasks, ii) the performance of the LLM-based world models depends predominantly on their performance in key steps, while the total number of steps required for task completion is a weak indicator of task difficulty with critical bottleneck steps playing a more decisive role and iii) the combination of world models’ functionalities
for decision making tends to increase performance instability in our experiments, which can partially obscure the performance gap between stronger and weaker model.

Published in Transactions on Machine Learning Research (03/2026)
LLM-Based World Models Can Make Decisions Solely, But Rigorous Evaluations are Needed
Chang Yang', Xinrun Wang"", Junzhe Jiang', Qinggang Zhang', Xiao Huang
The Hong Kong Polytechnic University, Singapore Management University
chang- wta.yyg@oooneee.polu.... grweng@sma.cdd edu.s.
Reviewed on OpenReview: https: oppenrevee.. net/ per/fo id- XmYCERErcD
Abstract
World model emerges as a key module in decision making, where MuZero and Dreamer achiove remarkable successes in complex tasks. Recent work leverages Large Language Models (LLMs) as general world simulators to simulate the dynamics of the world due to their generalizability. LLMs also serve the world model for deliberative reasoning in Reasoning via Planning (RAP) and Tree of Thought (ToT). However, the world model is either evaluated as general world simulator, or as a functional module of the agent, Le., predicting the transitions to assist the planning. This paper argues that LLM-based world models can make decisions solely, but rigorous evaluations are needed. We first present the two key observations lo showcase how LLM-based world models can make decisions solely, and then present the three key observations to demonstrate why current evaluation framework of LLM--based world models is not sufficient. Then, present our suggested evaluation framework: policy verification, action proposal, and policy planning, where the world model is used for decision making solely, and finally we leverage the 31 diverse environments from (Wang et al., 2023; 2024) and curate the rule-based policy of each environment for diverse evaluations. The key findings include: i) GPT-4o significantly outperforms GPT-4o-mini on the three main lasks, especially for the tasks which require the domain knowledge, e.g., scientific tasks, ii) the performance of the LLM-based world models depends predominantly on their performance in key steps, while the total number of steps required for task completion is weak indicator of task difficulty with critical bottleneck steps playing more decisive role and iii) the combination of world models' functionalities for decision making tends to increase performance instability in our experiments, which can partially obscure the performance gap between stronger and weaker model.
1 Introduction
The remarkable achievements MuZero (Schrittwicser et al., 2020) and Dreamer (Hafner et al., 2019; 2021; 2025a;b) have established world models (Ha & Schmidhuber, 2018) as fundamental components decision-making systems. World models serve as learned simulators that encode rich representations of enviromment dynamics, enabling agents predict future states conditioned on their actions. Rather than learning direct policy mappings from observations to actions, world models learn the underlying transition dynamics as) + s' and reward functions R(s, a), providing compressed yet informative representation of how the environment evolves. These models have demonstrated effectiveness across several key areas with concrete achievements: i) Generalization to novel tasks. World models enable transfer learning by capturing reusable environment dynamics (Byravan et al., 2020). For instance, agents trained with world models can adapt to new task objectives without relearning dynamics (Robey al., 2021), and demonstrate
*Corresponding author

---

Published in Transactions on Machine Learning Research (03/2026)
robust performance across distribution shifts in robotics applications (Young al., 2023). This capability is particularly valuable when the cost of re-training from scratch is prohibitive. iij Efficient planming. By simulating possible futures in the learned model, agents can evaluate action sequences through "mental simulation' rather than costly real-world interaction (Sckar et al., 2020; Hamrick et al., 2021). MuZero leverages this capability to achieve superhuman performance in Go, chess, and Atari games by planning over imagined trajectories (Schrittwicser et al., 2020). The ability to explore hypothetical scenarios without environmental interaction dramatically improves sample efficiency and enables safer exploration in high- risk domains. iii) Offtine learning from fixed datasets. World models have proven especially valuable in offline reinforcement learning settings (Schrittwieser et al., 2021; Yu et al., 2020; 2021), where agents must learn purely from pre-collected datasets without additional environment interaction. By learning accurate dynamics models from historical data, agents can perform policy optimization in the model space, generating synthetic rollouts to augment limited real data. This capability opens possibilities for training agents in domains where online interaction is impractical or expensive, such as healthcare and autonomous driving.
Recent developments have extended world models beyond traditional reinforcement learning to interactive simulation systems, demonstrating their potential as general-purpose world understanding engines. Notable examples include the Genie series (Bruce et al., 2024; Parker-Holder et al., 2024; Ball et al., 2025), which learns controllable world models from unlabeled video data and enables users to interact with generated game-like environments through action inputs. Vista (Gao et al., 2024) extends this paradigm to autonomous driving, learning world models that can simulate diverse driving scenarios and enable policy training in purely synthetic environments. Additional systems such as MatrixGame (Zhang et al., 2025; He et al., 2025), which generates playable game environments from natural language descriptions, and Gameuat (Li et al., 2025), which creates interactive 3D game worlds, further illustrate the expanding capabilities of world models in interactive domains. These systems share common theme: they learn predictive models of com- plex, dighedimensional dynamics (visual observations, physics, agent interactions) and expose these models through interactive interfaces. These advances point toward world models serving as foundational com- ponents for artificial general intelligence systems with grounded understanding of world dynamics, moving beyond narrow task-specific applications to general-purpose simulation and interaction.
Large Language Models (LLMs) have emerged as a promising approach to world modeling, leveraging their broad knowledge acquired from large-scale text corpora. Building on their success in natural language tasks (Brown et al., 2020; OpenAT, 2023), recent works explore LLMs as general world models that provide enviromment knowledge for complex reasoning and peeiiionnmmking tasks. The key hypothesis is that LLMs, having been exposed to vast amounts of text describing how the world works, have internalized implicit models of common-sense physics, social dynamics, and domain-specific knowledge (e.g., mathematics, sci- ence, games). Through fine-tuning on environment interaction data, i..., sequences of states, actions, and outcomes, LLMs can predict action sequences across diverse tasks while maintaining their broad capabili- ties (Xiang et al., 2023; Wang et al., 2024; Xie et al., 2024). For example, task demonstrations can predict valid action sequences for cooking or cleaning tasks while retaining its language understanding and reasoning abilitics. LLMs serve as world models in both explicit and implicit ways, depending on how the state prediction capability is utilized: i) Explicil world modeling. In approaches like Reasoning via Planning (RAP) (Hao et al., 2023) and Reason for Future, Act for Now (RAFA) (Liu et al., 2023), LLMs directly predict next states given current states and actions, functioning as explicit transition models TiLM(s,, a.) - 8141. For instance, in BlocksWorle (Valmeekam et al., 2023), given state description like "Block A is on Block B, Block C is on the table" and an action "move Block A to Block C," the LLM predicts the resulting state: "Block A is on Block C, Block B is on the table, Block C is on the table." These predicted states are then used by planning algorithms (c.g., Monte Carlo Tree Search, breadth-first. search) to evaluate action sequences and select optimal plans. The explicit separation allows for modular analysis of world model accuracy independent of planning algorithm performance. ii) Implicit world modeling. In widely-used approaches like Tree of Thoughts (ToT) (Yao et al., 2023) and Graph of Thoughts (GoT) (Besta et al., 2024), LLMs perform world modeling implicitly as part of their reasoning pro- cess. Rather than explicitly outputting state descriptions, these methods have LLMs generate and evaluate intermediate reasoning steps ("thoughts") that implicitly represent states in a problem-solving trajectory. For example, when solving math problem, each thought represents a partial solution state, and the LLM must implicitly model how different reasoning steps transition between these states. The LLM evaluates the
LLM fine-tuned on household
2

---

Published in Transactions on Machine Learning Research (03/2026)
promise of each thought (analogous to value estimation in RL) to guide search through the solution space. We provide comprehensive review of related work in Section 6.
Despite these celebrating progress, the role of LLM-based world models in decision models is still unclear for researchers. In this paper, we present a comprehensive view: LLM-based world models have the potential to make decisions solely through the combination of policy verification and action proposal capabilities, but rigorous evaluation frameworks are needed to assess this capability. We first present the two key observations to showcase how LLM-based world models can make decisions, and then present the three key observations to demonstrate why current evaluation framework of LLM-based world models is not sufficient. Then, we present our suggested evaluation framework: i) policy verification: verifying whether the policy can complete the (ask, ii) action proposal: proposing the top- actions that can potentially complete the task, and iii) policy planning: finding the policy solely with the combination of the different functionalities, i.e., policy verification and action proposal. Finally, we leverage 31 diverse environments from (Wang et al., 2023; 2024) with different tasks varying from daily tasks, c.g., washing clothes, to scientific (asks, e-g., forging keys, and curale the rule-based policy for each environment for the evaluation and conduct the comprehensive evaluation of the advanced LLMs, ie., GPT-4o and GPT--4-- mini, on the environments for three tasks under various scttings, which serves as a proof of concept for the suggested evaluation framework. The key findings include: i) GPT-44 significantly outperforms GPT-40- mini on the three main tasks, especially for the tasks which requires the domain knowledge, e.g., scientific tasks, ii) the performance of the LLM-based world models depends predominantly on their performance in key steps, while the total number of steps required for task completion is a weaker indicator of task difficulty and critical bottleneck steps play a more decisive role and iii) the combination of world models' functionalities for decision making tends to increase performance instability in our experiments, which can partially obscure the performance gap between stronger and weaker model. We hope this work can encourage researchers to rethink the evaluation of world models and further advance the research of the world model field. Code can be access at: https: thub. com/ hawpaaaayynn//ooodoooaoooooo
2 Preliminaries
Markov Decision Process (MDP). A decision making problem is usually represented as a Markov decision process (MDP) (Sutton & Barto, 2018), defined by the tuple (S, A, T, R),), where S is the state space, A is the action space, T : S x A S is the transition dynamics, which specifies the next state s' given the current state and action a, R 5 x A + IR is the reward function, which specifies the agent's 's reward given the current state s and action a, and is the discount factor. The agent's 's policy is defined by :o x A- + [0, 1], paramcterized by 6, which takes the state s as the input and outputs the action a to be executed. The objective of the agent is to learn an optimal policy x² == arg max, E, EE 7're|0o] is the expected return and 8p is the initial state.
Large Language Models (LLMs). Large Language models (LLMs) learn from text data using unsuper- vised learning. LLMs optimize the joint probabilities of variable-length symbol sequences the product of conditional probabilities by P(x) I P(s.lsi,..., 81-1) where (81, (2, is the pariable-lengti sequence of symbols. With the billions of parameters and extensive training data, the vast amounts of common knowledge encoded in LLMs lead to the remarkable generalization across various NLP tasks with simple prompting and in-context learning, without task-specific fine-tuning (Touvron et al., 2023; OpenAT, 2023). Given the generalizability LLMs present a promising foundation for general world models.
LLM-based World Models. The world model Ω2 is introduced to predict the dynamics of the environment, thus supporting the decision making process. Specifically, the world model is trained prompted to predict the next state s', the reward r, and the terminal function d, given the current state s and action a. The world model can be one or multiple neural networks specially trained on the environments for the three prediction tasks (Hafner et al., 2019; Schrittwieser et al., 2020), which cannot generalize across different environments. Recent work leverages chain-of-thought (CoT) prompting (Xic et al., 2024), in-context learning (Wang et al., 2024), angmentee generation (Yang et al., 2025), and fine-tuning methods (Xiang et al., 2023; Lin et al., 2024) to transform L.LMs to world models.
3

---

Published in Transactions on Machine Learning Research (03/2026)
Scope of This Work. In this work, we intend to investigate the core roles of LLM-based world models in decision making and design the comprehensive evaluation framework. Rather than evaluating world models as general simulators or as components coupled with other modules, we propose three decoupled evaluation tasks: policy verification, action proposal, and policy planning. We empirically evaluate GPT-4o and GPT- 4o-mini across 31 diverse text-based environments as a proof of concept to assess the reasonability of the suggested evaluation framework.
3 LLM-Based World Models Can Make Decisions Solely
We present the two observations to illustrate how LLM-based world models can make decisions solely.
Observation 3.1: Selecting potential actions should be an important feature for world models. Most of the previous works in world model focus on next state and reward prediction, and the action selection is usually completed by the actors, i... the model trained for generate a single action for executing. We argue that with more knowledge about the world, the world model may make a better selection of the potential actions. Besides, selecting set of potential actions, e.g., 10 potential actions, may significantly reduce the difficulties of the tasks and improve the performance when combing with planning. World models can also be viewed as game engines (Valevski et al., 2024), which have to provide potential actions to guide fresh players to complete tasks, c.g., Red Dead Redemption 2 (Tan et al., 2024). Therefore, action proposal should be considered for evaluation, which can be easily implemented for the LLM-based world models.
Observation 3.2: Planning with world models can find the policies solcly. With the prediction of the next states and the action proposal, we can leverage planming methods or search methods to find the policies. It is ob- served that most state-of-the-art methods for complex decision-making lasks, e.g., Chess or Go, is based on the planning with an accu- rate simulator (Silver et al., 2018; Monroe & Chalmers, 2024) or the world model (Schrit- twieser et al., 2020). Most works introduce the critic (i.e., the value function) to evaluate the actions immediately for efficient planning (Schrittwieser et al., 2020; Hao et al., 2023). We note that the critic is not necessary for finding policies (as showed in Figure 1) and may also influence the performance. Therefore, we argue that only the planning with the next state prediction and the action proposal is necessary when incorporating the world model in decision making. This focused approach allows for better isolation and evaluation of the world models and further highlight the importance of world models.
so
a1
a2
Q: am al so. if take the actions (a2. ag), can complete the task?
S2
WM: You are at Sor if you take you will tramsit to At sn, if you take da you will reach Sy, which is the target state. So you will complete the task.
a3
a4
S3
S4
Figure 1: An example to show that the critic is not neces- sary. We can query all action sequences and let the world model to determine whether the action sequence can com- plete the task, which may be insufficient but still possible.
True Value
Case 1 Case 2
4 But Rigorous Evaluations are Needed
200
80
100
70
Observation 4.1: Prediction is important, but not that important. An illustrative ex- ample is displayed in Figure 2, which indicates that more accurate predictions do not lead to correct decisions.' This phenomenon is also ob- served in other decision making scenarios, e.g., financial trading (Sun et al., 2023). The success of MuZero Unplugged (Schrittwieser et al., 2021) also demonstrate that we can learn good policies from inaccurate world models which are trained only with limited data. This motivates that
A
B
B
B
Figure 2: Picking the action with higher value. The true value of A and B are 80 and 100, respectively, therefore, B is the correct action. Case I's prediction 200 is worse than in Case 2's prediction 70 (compared the true value 80). However, in Case the most valuable action B, which is also the case in the true value setting. This shows that more accurate predictions (Case 2) do not always lead to correct decisions (Case 1).
The issue in Figure 2 can be elicited by various methods, e.g. rank prediction. This just. illustrate the discrepancy between prediction and decision, motivating us reconsider the evaluation of the world model for decision making.
4

---

Published in Transactions on Machine Learning Research (03/2026)
the evaluation of the world models for decision making should focus the predictions which relevant to the desired policy, rather than as general world simulators. Besides, the decision making usually involves multiple steps and the errors of the one-step predictions are accurulattd when the number of steps increases. Therefore, the accuracy of the one-step predictions is not adoquate for the evaluation of the world model for decision making.
Observation 4.2: Not all state are created equal for decision making. An illustrative example is dis- played in Figure 3, where the agent needs to go through gridworld from start to goal while avoiding falling off a cliff. In this example, the states in the red rectangle are more important than the states in the blue rectangle, as these red states are closer to the optimal path and the blue states are rarely explored by the agent when learning to complete the tasks. This observation is supported by
Opiimal
Figure Cliff Walking (Sutton & Barto, 2018). The optimal path is marked as the red arrow.
the fact that only small portion of the state space will be visited (Svisi < S) when computing the optimal policy, c.g., AlphaZero finds the super-human policy (Sil- ver et al., 2018) by only exploring a small proportion (less than 1%) of the state space. This observation suggests that world model evaluation should prioritize states that are critical for task completion.
Observation 4.3: Performance is usually coupled with other modules. An illustrative example is dis- played in Figure 4. The states in the red rectangle is explored by MCTS, while the optimal node (marked the red star) is in the unexplored states in the blue rect- angle. Therefore, even the world model can provide the accurate prodictions, the agent's performance will be re- stricted by the search algorithm. In model-based planning approaches like MuZero (Schrittwieser et al., 2020), the world model's performance cannot be isolated from the search algorithm, c.g.: MCTS, value network, and pol- icy network (Hafner et al., 2025a). When these compo- nents are tightly integrated, it becomes difficult to deter- mine whether poor performance stems from an inaccurate world model, suboptimal planning, or ineffective value es- timation. This coupling obscures our understanding of the world model's true capabilities and limitations. To address this issue, the evaluation should decouple the world models with other modules and design the tasks which specifically test the world model's functionalities.
plyl
Figure 4: Figure adpted from MuZero (Schrit- twieser et al., 2020) to illustrate search process, i.c., MCTS, in model-based planing.
5 Our Suggested Evaluation Framework and Experiment Results
In this section, we introduce our evaluation framework and the experiment results. Specif- ically, we will introduce the next state pre- diction, the reward and terminal prediction. Then, we will introduce how the world model will be used to complete the considered three main tasks, i.e, policy verification, action pro- posal, and policy planning. We provide the relationship between the three tasks and the two kinds predictions in Figure for bet- ter understanding of the rationale behind the three tasks. We then introduce the environ- ments and ruse- policies. Finally, we will
Next State Prediction
Task
Policy Verification
Reward/Terminal Prediction
Policy Planning
Action
Proposal
Task
Figure 5: With the basic predictions, i.e., next state and re- ward/terminal the policy verification can be completed. With policy verification and action proposal of the world model, policy planning can be completed.
5

---

Published in Transactions on Machine Learning Research (03/2026)
introduce each task, the evaluation protocols, and the corresponding experimental results. Our experiments utilize GPT-4o and GPT-4o-mini as the backbone LLM architectures for the world model.? To ensure repro- ducibility and minimize stochastic variance in model outputs, we configure the temperature parameter to O. All experimental results are averaged across 30 independent trials to handle environmental stochasticity and provide statistically robust metrics. Our evaluation results should be interpreted as diagnostic assessments of specific capabilities under controlled conditions to demonstrate the reasonability of the suggested evaluation framework, rather than measures of general world modeling ability or readiness for deployment.
The world model considered in this work mainly follows the design in (Wang et al., 2024), where the reprosentation of the states includes the objects in the environments and their properties. The prompts to the LLM, e.g., GPT-40, also include the object rules, the action rules, and the score rules, which provides the necessary knowledge of the environments for the LLM to make accurate predictions. For the next state prediction, we ask the LLM predict the state changes, i.c., the change of the objects' properties, which is demonstrated to be efficient for the prediction (Wang et al., 2024). With the predicted state changes, we can recover the full state for further predictions. For the reward reward/terming terminal prediction, the LLM needs to predict three features: i) gameScore: the reward received from the environment, ii) gameOver: whether the task is terminated, and iii) gameWon: whether the task is successfully completed or not. For the rules used for the prediction, we refer to (Wang et al., 2024) for more details and the code to generate the prompts is also provided in Appendix C.1 for completeness.
Environments. We leverage the 31 diverse environments from (Wang et al., 2023)* with different tasks varying from daily tasks, e.g., washing clothes, to scientific tasks, egg,, forging keys. This task suite is more related to the real physical world, including the physical objects, e-g., bulb and bathtub, and the iterations with these physical objects, i.e,, turn the hot tap to improve the temperature of the water in the bathtub. Compared with other widely used environments, such as the grid world, e.g., BabyAI (Chevalier-Boiswe et al., 2019) and the web environments, e.g., MiniWob++ (Shi et al., 2017), this task suite is more relevant to the common knowledge encoded in the LLMs. A full descriptions of the environments is in Appendix B.1.
60
Steps
40
20
paint
blood
thermo
energy
bulb
scale
bandage
hang
multi
volume
sunburn
space
sweep
container
bathtub
conduct
campfire
fridge
stone
bird
weigh-
metal
cooking
ice
heavy-
photo
plant
boil
key
plane
washing
Game
Figure Number of steps to complete the tasks
Rule-based Policies. There are various randomness in the environments, including the specific tasks, e.g., the target color can be "orange", "purple", "green", "black" in the paint task. However, for each task, only single playthrough is provided in ( (Wang et al., 2024), which is not enough for a comprehensive evaluation of the world model for decision making. Therefore, we curate the rule-based policy for each environment and verify the correctness for 200 runs. The scripts for the rule-based policics are provided in Appendix B.2, which can help readers to undersland the process to complete the tasks. We provide the statics of the number of steps to complete the tasks for 200 runs in Figure 6.5
5.1 Task I: Policy Verification
Task Description. Motivated by Observations 4.1 and 4.2, we propose policy verification, one of the most straightforward tasks evaluate the world models in terms of the multi-step predictions. The basic
2Due to the limited budget, we do not take a full list of LLMs, egg, Claude and Gemini, into consideration.
Due to the space constraints and the extreme length of the prompts, we cannot provide complete example in the paper. results.
We will open-source all the codes for readers replicate
We note that there 32 cuvironments in (Wang al., 2023) and the dish-washing cuvironment is used the example the world model, which is excluded for fair evaluation.
She names on figure may differ from (Wang al., 2023) for plotting. Please refer Table correspondence.

---

Published in Transactions on Machine Learning Research (03/2026)
0.75) (b) (p 0.5) (a) 0.25)
Accuracy
0.8
gpt-4o
0.6
gpt-4g-mini
0.4
0.2
0.0
1.0
Accuraee e
0.8
gpt-4o
0.6
go--iniiini
0.4
0.2
(d) (p 1100 (p- (p75.5)
1.0
Accuracy
0.8
gpt-4o
0.6
gpt-4o-trii
0.4
0.0
1.0
8 0.8
9pt-4o
0.6
gpt-4o-mini
0.4
0.2
0.0
paint
blood
thermo
energy
bulb
scale
bandage
hang
multi
volume
sunbum
space
sweep
container
bathtub
conduct
campfire
fridge-
stane
bird
weigh
metal
cooking
ice
heavy
photo
plant
boil
key
plane
washing
Games
Figure 7: The accuracy of the world model to verify the correct policies
idea of policy verification is given an action sequence, the world model need to prediction whether the sequence can complete the task or not. The process for the policy verification is displayed Algorithm 1. Specifically, given the environment env, the action sequence a with length N verify and p denotes the fraction of the action sequence for which the world model is responsible for snfffcatiio/plamninn We will run the game for the first (1 p) N steps (Line 5 of Algorithm 1), and leverage the world model to continue the last pN steps (Line 7 of Algorithm 1). The returned results rv, dy will be compared with the true results from the environment to evaluate the world model.
Evaluation Protocol. Given the action sequence a generated by the rule- based policy, we leverage the world model to verify the last P proportion of the policy, where p E (0.25, 0.5, 0.75, 1.03. We note that when p 1.0, the world model will verify the full action sequence with only the initial observation of the environment. We say the verification of the policy is correct if all three features, i.c., gameScore, gameOver, and gom, are correct. Our policy verification evaluation only tests correct (positive) policies. A complete verification capability would also need to reject incorrect (negative) policies, which we do not test. This means we measure "false negative rate" (incorrectly rejecting valid policies) but not "false positive rate" (incorrectly accepting invalid policies). True utility for planning would require low rates on both. Generating meaningful negative policies challenging: random action sequences are trivially wrong, while plausible-but-ffawee policies require domain expertise construct. We leave comprchensive negative policy evaluation to future work, acknowledging this significantly limits claims about verification reliability for search applications. Furthermore, there would also be other action sequences to finish the tasks, where we cannot enumcrate all policies to complete the tasks.
Evaluation Results. The policy verification results are displayed in Figure 7.6 We observe that GPT-4o outperform GPT-4o-miní in most tasks and especially on the tasks which requires the domain knowledge, e.g., bandage, hang, and campfire. We also observe that with more steps of the verified policies, the performance gap between GPT-4o and GPT-4o-min is increase. With larger proportion of the action sequences to verify, i.c., P increase, the accuracy of the verification is decreased, which indicates that the accumulation of the errors in the world model, either on the next state prediction or the reward prediction, will influence the performance of the world model. This observation is consistent to the fact that the LLM may not perform well in long-term decision making tasks. We also observe that more steps to complete the tasks do not necessarily lead to the worse performance, which indicates that the domains the tasks play a more important role for the policy verification, i.e., for the tasks where the LLM has enough domain knowledge, e.g., conduct, stone, weigh and photo (Wang et al., 2024), the task would be easy even when the number of steps is large, We also provide the accuracy of the three prediction tasks separately in Appendix F, and we found that
Note that the result of the policy verification either 1, so the error bar not plotted.

---

Published in Transactions on Machine Learning Research (03/2026)
both GPT-4o and GPT-4o-mini performs worse for predicting gameScore, while performs much better for predicting gameOver and gameWon. This indicates that the value prediction is more difficult for LLMs, which is consistent the observations from other works. During the experiments, both models frequently returned empty dictionarics, suggesting they may fail to properly follow the instructions.
Takeaways
Policy verification evaluates multi-step prediction accuracy by verifying whether action sequences
complete tasks, addressing the limitations of single-step prediction evaluation
Performance degrades with more verified steps, i.e., error accumulation in long-horizon predictions
GPT-4o significantly outperforms GPT-4o-mini, especially on tasks requiring domain knowledge
Task difficulty depends more on domain knowledge required than the total number of completion steps
Algorithm Policy Verification
A Algorithm 2 Policy Planning
1:
Given the env, the action sequence a to verify with . len(a), p the proportion of to verify, the world model
1: Given the env, the action sequence a with N len(a), the proportion of a for planning, the world model Ω the planning sequence a' =
so env()
2: for tE {1, ..., (1 p). N do
3: for / E (I, N - 1} do
3: St41. = env (a.) a' aappend(((a)
4: if 1 < (1 p) N then
4: end for
5: $141, It, de = env(a,)
5: for E [(1 p) N, ..... ... p)N) do
6: clse
6: (())
7: 8141, de = (((),
7: St+1.Te, dg a(s az), a' aapeeddddd(s))
8: end if
8: if de then break
end for
9: end for
10: return dN dd
10: return a
5.2 Task II: Action Proposal
Task Description. The action proposal is novel task for world model, based on Observations 3.1 and 4.2. Basically, we will ask the world model to recommend top- K actions that can potentially complete the task. Specifically, we follow the representation of the state in the next state prediction, with the additional information: i) the examples of actions, and ii) the previons actions. The previous actions can help the LLMs to understand the game progress. The code to LLM for the action proposal is displayed in Appendix C.2. One key issue for the action proposal is that the action generated by the world model may not be valid for the game at the current state. Therefore, given the predicted action a' and the of possible actions to be executed at the current state A', we leverage the text-embedding model, i.e., text-embedding- --mmmillnn--smll (OpenAT, 2022) to query the most similar actions with the cosine similarity, i.c., a* = arg max max((mta (emb(a', a), Va E A').
Evaluation Protocol. The action proposal requires the world model to generate the top-K potential actions to complete the tasks, where K E {1, 5, 10}. Specifically, given the action sequence generated by the rulo-based policies?, we will let the world model to generate the potential actions with the states along with the path of a to complete the task. We say the action proposal correct if the actions in a in the generated actions by the world model. The results of the accuracy are averaged over the steps over the action sequence and 30 runs of each environment. The action sequence a generated by the rule-based policy is not the only sequence to complete the task and we cannot enumerate all possible actions which can lead to the completion of the task. We note that the number of available actions in the environments is usually larger than 500, which brings difficulties to the RL methods for training and indicates the necessity to gencrate potential actions to facilitate the lcarning.
Evaluation Results. The action proposal results are displayed in Figure 8. Overall, GPT-4o consistently outperforms GPT-4o-mini across different tasks and different values of K. With the increase of the number of steps to complete the tasks, where more analysis of the previous actions is needed to understand the
We note that there may more than one palicy complete the task. However, we cannot enumerate all possible policies and we view this evaluation as the lower bound performance the model in action proposal.
8

---

Published in Transactions on Machine Learning Research (03/2026)
1.0
(a) ( 11
0.8
gpt-4o
0.6
3pk-4o-mini
0.2
1.0
(b) (K=2)
Accuracy
0.8
9Pt-40
apt-4o-mini
0.4
0.2
1.0
(c)
Accuracy
0.8
gpt4g
0.6
apt-4o-mini
0.4
0.2
0.0
(d) (K 5)
0.8
gpt-4o
0.6
apt-4o.mini
0.4
0.2
0.0
(e) [K = 10)
0.8
apt-4o
0.6
gpt-4o-mini
0.2
0.0
paint
blood
thermo
energy
bulb
scale
bandage
hang
mutti
volume
sunburn
space
container
bathtub
conduct
campfire
fridge
stone
bird
weigh
metal
cooking
ice
heavy
photo
plant
bo
key
plane
washing
Games
Figure 8: The accuracy of the world model to generate the potential actions
game progress, GPT-40 maintains the better accuracy, while GPT-4o-mini shows a substantial drop of the accuracy. The performance gap between the two models is generally increased when the number of steps complete the tasks increase. When K 10, the accuracy of the action proposal for GPT-4o is very high in most tasks. With approximately 800 possible actions available at each time step, the results demonstrate that GPT-4o effectively identifies and selects relevant actions while filtering out irrelevant ones. This capability shows promising potential for successful task completion. Furthermore, we still observe that both models obtain lower values in the lasks requiring the domain knowledge, i.e., blood and conduct, which is consistent to the observation in (Wang et al., 2024) that LLMs, e.g., GPT-4, is more likely to make errors when scientific knowledge needed. We also provide the step accuracy of the action proposal Appendix G to illustrato the prediction of the relevant actions along with the steps. We observe that there has extremely low accuracies, which indicates that the critical steps significantly influences the difficulties of the tasks, rather than the number of steps to complete the tasks, which differs from the traditional RL. We also observe that both GPT-4o and GPT-4o-mini can generate wrong actions even when the action rules are given, especially for the environments where the domain scientific knowledge is needed, e.g., 'space-walk'.
key steps that
Takeaways
Action proposal evaluates the world model's ability to generate potential actions that can lead task completion, going beyond traditional state and reward prediction
Performance improves substantially when proposing more candidate actions
Critical steps with extremely low accuracy indicate that specific decision points drive task difficulty more than overall step count, which differs from the traditional decision making
5.3 Task III: Policy Planning
Task Description. The policy planning task is motivated by Observations 3.2 and 4.3, which combines the policy verification and the aclion proposal (as displayed in Figure 5). The process of policy planning is displayed in Algorithm 2. Specifically, we execute the actions in the given a on the environment for (1 N1 steps (Line 3 in Algorithm 2) and then plan for 2 steps, i.e,, {(1 p)., (1 N)N} with the world
8The budget 2nN allows safety factor accommodate potential backtracking dead-ends in the generated policy. In practice, the loop terminates early when the world model predicts task completion.

---

Published in Transactions on Machine Learning Research (03/2026)
0.25)
1.0
0.8
gpt-4o
5
0.6
gpk-4o-mini
(a)
0.2
0.0
(d) 1,0) (c) (o 0.75) (b) 0.5)
Success Rate Suc Rap 8
1.0
0.8
gpt-4o
0.6
gpt-4o-mini
0.4-
0.2
1.0
gpt-4o
0.6
apt-4o-mini
0.4
0.0
1.0
access R0g
0.8
gpt-4o
0.6
gpt-4o-mini
0.4
0.2
0.0
paint
blood
thermo
energy
bulb
scale
bandage
hang
multi
volume.
sunbum
space
sweep
contairer
bathtub
conduct:
campfire:
fridge
stone
bird
weigh
metal
cooking
ice
heavy
photo-
plant-
boil
key
plane
washing
Games
Figure 9: The success rate of the world model to complete the tasks
model (Line 7 in Algorithm 2), where both the action to execute and the state transitions are generated by the world model. The returned action sequence a' will be evaluated in the environment to verify the correctness. Only top-1 action is generated Algorithm 2 for illustration. When more actions are generated, we need to enumerate all possible outcomes or leverage search methods, which will be tackled in future work.
Evaluation Protocol. The policy planning is based on the policy verification and the action proposal, showed in Algorithm 2. Similar to the policy verification, we let p E {0.25, 0.5, 0.75, 1.0] to vary the number of steps for the planning. We only consider the case with K = 1, i.e., the world model only generates the top-1 action with the given states. Finally, we evaluate the planned policy a' in the environment to verify the correctness. We note that when K 1, no advanced search method is needed, while when K > 1, we cannot enumerate all possible outcomes for larger steps, e.g., 10. Besides, critic is also needed choose among the outcomes for verifying in the enviromments. Therefore, we only consider the case with K 1 and leave the case K > 1 into future work.
The policy planning results are displayed in Figure 9, where GPT-4o and GPT-4o- mini achieve comparable performance for the tasks with less steps and smaller values of p, e.g., 0.25 and GPT- 4o generally achieve better results in tasks with more steps. When the value of p increases, the performance is generally docreasing. With the coupling of the policy verification and action proposal, we observe more unstabilities of the performances of models over tasks and settings. This indicates the necessity of decoupling the funetionalities of the world model for the evaluation. Similar to the model-based RT (Schrittwieser et al., 2020), where introducing the world model may bring the training unstabilities, apply the LLM-based world models to the decision making due to the inherent complexity. During the experiments, we also observe the format errors of the outputs from both models, which may interrupt the running of the experiments. The frequency of these failures depends on the environments, where the 'hang- painting', 'space-walk', and campere are the three environments we experiences most of the failures. Therefore, with the interaction of the different functionalities of the world model, the system is more unstable.
Evaluation Results.
need to be carefully to
Takeaways
Policy planning demonstrates that world models can make decisions solely by combining policy verifi- cation and action proposal without requiring separate critic modules
Porformance becomes more unstable when coupling multiple world model functionalitics, partially ob- scuring performance gaps between strong and weak models
5.4
Summary
The three evaluation tasks work together validate our framework: Policy verification (Task I) confirms that world models assess multi-step consequences, addressing Observations 4.1 and 4.2. Action proposal
10

---

Published in Transactions on Machine Learning Research (03/2026)
(Task II) demonstrates capability to identify promising actions, addressing Observation 3.1. Policy planning (Task ITT) shows these capabilities can be combined for autonomous decision making without separate actors, addressing Observation 3.2, though with increased instability as suggested by Observation 4.3. Together, these results support our position that LLM-based world models can make decisions solely, while highlighting the importance of decoupled evaluation to understand component capabilities and failure modes. We believe these results validate the reasonability of the suggested evaluation framework and illuminate future rescarch directions for understanding LLM-based world models in decision-making holistically.
6 Related Work
World Models in Decision Making. World models are actively explored by researchers to further improve the agent's 's performance and the sample efficiency (Ha & Schmidhuber, 2018; Janner al., 2019; Hafner et al., 2019; Schrittwieser et al., 2020). Dreamer (Hafner et al., 2019; 2025b;a) is a practical model-based reinforcement Icarning algorithm that introduces the belief over states as a part of the input to the model-free DRL algorithm used. Trajectory Transformer (Janner et al., 2021) trains the transformer to prediction the next state and action as a sequence modeling problem for continuous robot control. MuZero (Schrittwieser et al., 2020) is a remarkable success of model- based RL, which learns the world model and conduct the planning in the latent space. MuZero achieves the superior performances over other model- based and model- free RL methods. The world models trained in these methods are problem-specific and cannot be generalized to other problems, which motivates researchers to seek to more generalizable world models, e.g., LLMs world models. The world model with LLM in (Xiang et al., 2023) is trained to gain the environment knowledge, while maintaining other capabilities of the LLMs. Dynalang (Lin et al., 2024) proposes the multimodal world model to unify videos and texts for future predictions in decision making.
LLMs as World Simulators. World simulators are developed to model the dynamics of the world (Bruce et al., 2024). LLMs serve as the world simulator due their generalizability across tasks. Specifically, The LLMs (i.c., GPT-3.5 and GPT-4) is evaluated to predict the state transitions, the game progress and scores with the given object, action, and score rules, where these rules are demonstrated to be crucial to the world model predictions (Wang el al., 2024). The world models with LLMs in (Xie et al., 2024) need to additionally identify the valid actions. Recently, the Genie series (Bruce et al., 2024; Parker-Holder et al., 2024; Ball et al., 2025) Icarns controllable world models from unlabeled video data and enables users to interact with generated game-like environments through action inputs, followed by MatrixGame (Zhang el al., 2025; He et al., 2025) and GameCraft (Li et al., 2025). These work demonstrate the (multimodal) LLMs' potential to serve as general world simulators. We move a step further to ask the world model to propose the potential actions to complete the tasks. Both methods focus next state prediction which may be not suitable for the evaluation of world models for decision making.
LLM-based World Models for Decision Making. The concept of world model also be explored in the deliberation reasoning of LLMs. Specifically, Reasoning via Planning (RAP) (Hao et al., 2023) leverage the planning methods (e-g., Monte Carlo Tree Search (MCTS)) with the world model with L.LMs for plan generation and math reasoning, where LLMs need to predict the next state and the reward to guide the search. Tree of Thought (ToT) (Yao al., 2023) implicitly leverage the LLMs as the world model to predict the next state and the reward for the search over different thoughts. Reason for future, act for now (RAFA) (Liu et al., 2023) combine the planning and reflection with the world model for complex reasoning tasks. Recent work considers LLM-based world models for web agents (Chae et al., 2025; Gu et al., 2024), game agents (Hafner et al., 2025b), and even autonomous driving (Liao et al., 2025). However, these methods do not focus on the evaluation of world models, and several interdependent modules are coupled for completing the task.
7 Discussions
Are (LLM-based) World Models Necessary? Despite the success of MuZero (Schrittwieser et al., 2020) and Dreamer (Hafner et al., 2025a;b), world models remain relatively unpopular the rescarch community. The main barriers are the added complexity of training separate world models and their lack of general- irability. Although I.I.M-based world models offer improved generalization capabilities, most current LLM
11

---

Published in Transactions on Machine Learning Research (03/2026)
agents operate without them. This raises an important question: are (LLM-based) world models nec- essary? We argne that world models will become critical components of LTM agents for several compelling reasons. First, as agents are deployed in real-world scenarios where actions carry significant consequences, e.g., high-stake scenarios, world models enable essential counterfactual reasoning and outcome prediction prior to execution. This capability allows agents to simulate potential consequences without incurring the costs and risks of trial-and-crror in physical environments. Second, world models enhance interpretability by explicitly modeling future states and outcomes, thereby making agent behavior more auditable and trans- parent to human stakeholders. Third, world models facilitate sample-efficien learning by enabling agents to plan and refine strategies through internal simulation rather than requiring extensive real-world interaction. Most critically, world models will play an essential role in safety assurance and governance, serving as a foundational safeguard in the development of safe artificial general intelligence (AGI). As we advance toward increasingly autonomous and complicated Al systems, the capacity to accurately model and reason about environmental dynamics and action consequences becomes not merely with better performance, but also necessary for responsible Al deployment in the real world.
Limitations. There are several limitations of the current framework. i) Our framework relies on rule-based policies as ground truth, which represents only one valid solution path per task. Real-world decision making often admits multiple valid strategies. This single-path assumption may penalize world models that propose alternative but potentially successful approaches, leading to underestimation of their true capabilities. ii) For policy planning, we only evaluate K due to computational constraints. This limitation prevents us from assessing the world model to maintain consistency across multiple predicted trajectories, which is important for tree search methods. Extending to K>1 1 with tree search is important. future work to better demonstrate full planning capabilitics. To facilitate the understanding, we provide a comprchensive discussion about the role of K in Appendix A.5. iii) While we argue that decoupling evaluation of world model functionalities is critical, this approach may miss important interactions between components. The performance degradation in policy planning suggests that integrated evaluation still provides valuable insights. iv) Due to the limited budget, we only conduct the experiments on two widely used models. Our claims should be interpreted as applying to the lested models, and validation across diverse model families (including open-source models like Qwen-72BB Llama-70B, and closed models like Claude and Gemini) remains important future work.
Conclusion. In this work, argue that LLM-based world models can make decisions solely through the combination of policy verification and action proposal capabilities, but require rigorous, decoupled evalu- ation to understand their true strengths and limitations. Our proposed bottom-up evaluation framework, i.c., policy verification, action proposal, and policy planning tasks, enables systematic asscssment across 31 diverse environments spanning daily and scientific domains. Key findings reveal that: i) GPT-40 signifi- cantly outperforms GPT-4o-mini, particularly on domain knowledge-intensive tasks, ii) performance depends predominantly on critical steps rather than total step count, and iii) integrating multiple world model func- tionalities introduces instability that obscures capability differences between models. While our framework has
including reliance on single-path ground truth policies and restricted evaluation of multi- action planning-it it establishes foundation for systematic world model asscssment that shifts focus from general simulation accuracy to decision-oriented performance. As LLM-based agents advance toward deploy- ment in high-stakes real-world scenarios, our evaluation framework becomes essential for ensuring their safe and effective operation in complex docision-making tasks.
Acknowledgments
This work was partially supported by the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant (Proposal ID: 23-SIS-SMUU-037) The work described in this paper was also partially supported by a grant from the Innovation and Technology Commission of the Hong Kong Special Administrative Region, China (Project No. GH))391/22).
References
Phil Ball, Jakob Bauer, et al. Genie 3: A new frontier for world models, 2025. URL https:// google/blog/ haalee/bloggggeiieeeeeee wawerrraaaaaatttiir-oooooooaaaaa
12

---

Published in Transactions on Machine Learning Research (03/2026)
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyezyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In AAAI, pp. 17682 17690, 2024.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NewrIPS, pp. 1877-1901, 2020.
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In TCML, 2024.
Arunkumar Byravan, Jost Tobias Springenberg, Abbas Abdolmaleki, Roland Hafner, Michael Neunert, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Imagined value gradients: Model- based policy optimization with tranferable latent dynamics models. In CoRL,, pp. 566 589. PMLR, 2020.
Hyungjoo Chac, Namyoung Kim, Kai Tzu iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navigation. In ICLR, 2025.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Adilya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Tgor Mordatch. Decision transformer: reinforcement learning via sequence modeling. In NewrIPS, pp. 15084 15097, 2021.
Maxime -hevelier-Boisvert,, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. BabyAl: First steps towards grounded language learning with a human in the loop. In ICLR, 2019.
Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllabil- ity. Xixiv preprint erxiv:2405. 17398, 2024.
Nicolas Gontier, Pau Rodriguez, Issam Laradji, David Vazquez, and Christopher Pal. Language decision transformers with exponential tilt for interactive text environments. arXiv preprint orXh::2902.05507, 2023.
Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al. Is your LLM secretly a world model of the internet? model-based planning for web agents. arXiv preprint arXiv:2411006559 2024.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint orXin: 10122, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learning behaviors by latent. imagination. In TCLR, 2019.
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering Atari with discrete world models. In ICLR, 2021.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, pp. 1-7, 2025a.
Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models. an:ivv preprint Xti::2509.24521 2025b.
Jessica B Hamrick, Abram L. Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Witherspoon, Thomas Anthony, Lars Holger Buesing, Petar Veličkovié, and Theophanc Weber. On the role of planning in model-based deep reinforcement learning. In ICLR, 2021.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXio preprint ar Xrv:22305.111 14992, 2023.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, et al. Matrix-game 2.0: An open-source, real-time, and streaming interactive world model. arXiv preprint arXiv:2508. 2025.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. arXiv preprint arXiv:1906.1 190.::1906.0823, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In NeurIPS, 2021.
Jiaqi Li, Junshu Tang, Zhiyong Xu, Longhuang Wu, Yuan Zhou, Shuai Shao, Tianbao Yu, Zhiguo Cao, and Qinglin Lu. Gameuana High-dynamic interactive game video generation with hybrid history condition. arXiv preprint arXiv:2566.11 0125506672011 17201, 2025.
Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, et al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. In CVPR, pp. 12037-12047, 2025.
Jessy Lin, Yuging Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning model the world with language. In ICML, 2024.
Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang. Reason for future, act for now: A principled framework for autonomous Ilm agents with provable sample efficiency. arXiv preprint ar Xrx::22309. 1 1782,, 2023.
Daniel Monroc and Philip A Chalmors. Mastering chess with a transformer model. arXiv preprint arXiv:2409. 00.222249.12272, 2024.
OpenAI. New and improved embedding model, 2022.
URL https://openai.com/in
new-and- hex-and-improvee
OpenAl. GPT-4 technical report. arxiv preprint arXiv:2303.0877 2023.
Jack Parker-Holder, Stephen Spencer, al. Genie A large-scale foundation world model, 2024. URL genie-2-a-large https://aeepnind.goollee. lawpee/llooggooonaa-aaaaee
Alexander Robey, George Pappas, and Hamed Hassani. Model-based domain generalization. In NeurIPS, pp. 00210,00229, 2021.
Julian Schrittwieser, Ioannis Antonoglon, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588(7839): 009, 2020.
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Toannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. In NewrIPS, pp. 2758,-27591, 2021.
Ramanan Sckar, Olch Rybkin, Kostas Daniilidis, Picter Abbeel, Danijar Hafner, and Deepak Pathak. Plan- ning to explore via self-supervised world models. In ICML, pp. 8583 8592, 2020.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open- domain platform for web-based agents. In ICML, pp. 3135 3144, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforeement learning algorithm that masters chess, shogi, and Go through self-play. Science, 462(4411) 2018.
Shuo Sun, Rundong Wang, and Bo An. Reinforcement learning for quantitative trading. ACM Transactions on Intelligent Systems and Technology, 14(3):1-29, 2023.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Jumpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu. Cradle: Empowering foundation agents towards general computer control. orXiv preprinl ar xrXiv:2403.031 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Ami Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et. al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408. 144877 2024.
Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 38975 38987, 2023.
Ruoyao Wang, Graham Todd, Xingdi Yuan, Ziang Xiao, Marc-Alexandre Côté, and Peter Jansen. Byte- Sized32: A corpus and challenge task for generaling lask-specific world models expressed as text games. In EMNLP, 2023.
Ruoyao Wang, Graham Todd, Ziang Xiao, Xingdi Yuan, Mare-Alexandre Côté, Peter Clark, and Peter Jansen. Can language models serve as text-based world simulators? arXie preprint arXiv:2406. 06485, 2024.
Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Language mod- els meet world models: Embodied experiences enhance language models. arXiv preprint or 20.:2335..0626, 2023.
Kaige Xie, lan Yang, John Gunerli, and Mark Riedl. Making large language models into world models with precondition and effect knowledge. arXiv preprint arXiv:2409.1222 2024.
Chang Yang, Xinrun Wang, Qinggang Zhang, Qi Jiang, and Xiao Huang. Efficient integration of external knowledge to Ilm-based world models via aerrivval-nngmenter generation and reinforcement learning. In Findings of FMNLP, pp. 9484-9501 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Grilliths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: deliberate problem solving with large language models. In NewrIPS, pp. 11809-11822, 2023.
Kenny Young, Aditya Ramesh, Louis Kirsch, and Jürgen Schmidhuber. The benefits of model- gener- alization reinforcement learning. In ICML, pp. 40254-40276, 2023.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offinne policy optimization. In NewrIPS, pp. 14129-14142, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: conservative offine mosel-based policy optimization. In NewrIPS, pp. 28954-28967, 2021.
Yifan Zhang, Chunli Peng, Boyang Wang, Puyi Wang, Qingeheng Zhu, Fei Kang, Biao Jiang, Zedong Gao, Eric Li, Yang Liu, et al. Matrix-game: Interactive world foundation model. arXiv preprint arXiv:2506.1877 2025.
15

---

Published in Transactions on Machine Learning Research (03/2026)
A Frequently Asked Questions (FAQs)
A.1 Why World Models and Why World Models with LLMs?
i) Generalization to novel tasks: World models have demonstrated impressive transfer learning abilities (Byra- van et al., 2020), allowing agents to adapt to previously unseen sccnarios by leveraging their learned un- derstanding of world dynamics. This generalization capacily is particularly valuable in robotics and control applications where agents must handle diverse situations (Robey et al., 2021; Young et al., 2023). ii) Efficient planning: The predictive capabilities of world models enable the sophisticated planning algorithms without any online interactions (Sekar al., 2020; Hamrick et al., 2021; Schrittwieser et al., 2020). By simulat- ing possible futures, agents can evaluate different action sequences and select optimal strategies without requiring actual interaction with the environment. This "imagination" or "mental simulation" capability dramatically improves sample efficiency and safely in decision-making, iii) Offline learning: World models have proven especially valuable in offline reinforcement learning settings (Schrittwieser et al., 2021; Yu et al., 2020; 2021), where agents must learn from pre-collected datasets without direct environment interaction. The ability to learn accurate dynamics models from historical data has opened new possibilities for training agents in scenarios where online interaction is impractical or costly.
A.2 More Justifications of the Proposed Tasks
In this section, we will provide a mode detailed justification of the three tasks proposed in this work.
Policy Verification. Policy verification can be viewed as a generalization of the next state prediction. Instead of focusing on the accuracy of the one-step prediction about the next states and the reward/Lermon terminal prediction, which is considered in most previous works, policy verification may accumulate the multi-step prodictions and judging whether the given policy can complete the task or not. This task is more relevant to the world model for decision making, as if the world model can verify any given policy correctly, with the enough number of sampling of the policy, i.e., action sequences, we can complete the task in the end.
Action Proposal. As observing in the ToT (Yao et al., 2023), generating useful thoughts is critical which can significantly improve the performance. However, with multiple thoughts generated, we have to select one to executed. We can test these thoughts the environments, however, this is not always doable. Therefore, building a world model is the straightforward way to do this. On the other hand, action proposal is necessary for the world model as the game engine to guide the fresh players to complete the game. With increasing the number of recommend actions, the difficulty of the action proposal is decreased. However, this task is not considered in the previous work for the world model. We believe that this task should be an important task to evaluate the world model for decision making.
Policy Planning. Policy planning is combination of policy verification and action proposal. Conceptu- ally, if the world model performs well on both tasks, we can obtain the policy with world model only and no actor is needed. This can help us to understand the world model through decoupling the world model with any other modules. Besides, this planning task is consistent with the System thinking, iee. with more time for the planning, the world model may find better policies.
A.3 The Objectives of This Paper
The primary objective of this paper is proposing the new evaluation tasks for the evaluation of the world models with LLMs for decision making.
Evaluating world model for decision making is difficult, given that the decision making tasks usually involves multiple steps of the predictions. Therefore, the one-step prediction tasks considered in most previous works is not suitable.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Instead of treating the world models as world simulators supporting modules for the actors, we identify that the world model can solve the tasks solely with the combination of the policy verification and action proposal. Therefore, the world model should be researched with the same importance.
World models have traditionally been evaluated through a top-down approach, where complex eys- tems are constructed to complete tasks, constraining analysis to high-level observations. By exam- ining fundamental capabilities like policy verification and action proposal, we propose a bottom-up evaluation that enables more systematic and granular assessment of world models.
A.4 Selection of Backbone LLMs
We need to evaluate the three novel tasks over 31 environments, and each with 30 runs, we roughly use 3000 dollars for all the experiments for GPT-4o and GPT-4o-mini. Due the limited budget, we cannot afford to test on Claude and Gemini.
For the open-sourced models, we test on some open-sourced models, e.g., Qwen 7B, and find that current open-sourced models still camnot generate the responses with correct formats, i.e, JSON. This brings difficuliess for the evaluation.
A.5 The Role of K
The choice of K in policy planning, i.c., the number of candidate actions proposed at each step, deserves care- ful discussion, as it governs the trade-off between world model isolation and practical planning performance, as well as the practical efficiency of applying world models for planning.
== 1 as a Principled Baseline. Our main evaluation is with fixed = which represents the most straightforward test of the world model: at each step, the model must commit to a single action with no opportunity for correction. This is a deliberate methodological choice rather than a computational limitation. When K > 1, the planner must select among candidate sequences using a search strategy, e.g., beam search, majority voting, or MCTS, and success becomes joint function of world model quality and search algorithm design. This reintroduces preciscly the coupling problem identified in Observation 4.3, which our decoupled evaluation framework is designed to avoid. K = therefore provides the cleanest signal of the world model's intrinsic planning capability, and establishes a meaningful lower bound.
What K > 1 Would Look Like. The action proposal results in Section 5.2 provide an indirect but informative window into K > planning behavior, since action proposal is essentially policy planning evaluated one step at a time. Two observations are relevant. First, at K = 10, GPT--4o achieves near-perfect per-step coverage on the majority of environments, meaning the correct action is almost always present in the top-10 candidates. The K 1 planning bottleneck is therefore greedy commit at critical steps, not a lack of world model knowledge. A simple K > 1 selection strategy such as majority voting would be expected to substantially improve planning success rates, particularly on environments where a single bottleneck step has low K = 1 accuracy but high K = 10 coverage (e.g., make- campfee, walc)- Second, for environments with long sequential dependencies (e.g., wash-clothes), increasing K alone is insufficient: an error early in the sequence propagates regardless of how many candidates are generated later. In such cases, > 1 planning would need to be paired with policy verification as a subroutine, i.c., generating K candidate continuations and verifying each before executing, which is natural combination of our framework.
The instability concern. Our experiment results in Subsection 5.3 show that combining world model functionalities already increases performance instability at K = 1. K > planning would amplify this effect: each additional branch multiplies the surface area for compounding prediction errors and format failures. This suggests that the evaluation of K > 1 planning should proceed carefully, with explicit ablations over search strategy and K separately, rather than treating K > as a straightforward extension. We leave this systematic study to future work, but emphasize that our framework directly supports it, i.c., policy verification and action proposal are already evaluated as independent components, and their combination under K3 >> search is well-defined next step.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Summary. ==1 is the starting point for understanding world model capability in isolation and K >> 1 is the next step for understanding world model capability in practice. The gap between them is bridged by search strategy design, which we treat as an orthogonal research question. Our results suggest that this gap is largely attributed to bottlencck steps rather than general knowledge deficiency, and that policy verification is the natural mechanism for exploiting K candidates effectively.
A.6 Comparison with Decision Transformer and Trajectory Transformer
Decision transformer (DT) (Chen et al., 2021) trains the transformer to predict the action conditional on the experiences and the target reward or the goal. Language DT (LDT) (Gontier et al., 2023) extends DT to consider the rext-based games and include the state prediction in the training as an auxiliary tasks. However, during the inference, i.e., decision making, the model still generates the action directly, which is not based the world model because the state prediction is only used for training and not for acting.
Trajectory Transformer (TT) (Janner et al., 2021) also consider the decision making problem modeling problem, where the transformer is trained to predict the state, the action and the reward. Compared with DT, TT is more related to the world model, where the state and reward prediction used to generating actions and the search methods, e-g., beam search, used. However, only the continuous robot control is considered in TT (Janner al., 2021) and the trained TT model is domain and problem specific, which cannot generalize other problems.
sequence
Recently, LLMs provide promising way to build the general world model and the world model with LLMs emerge a novel research field. However, most of these work focus on the single-step prediction and comprehensive evaluate is needed. This work is inspired by TT and extends the insights from TT to the world model with LLMs for text-based games. Specifically, we consider the policy verification, the action proposal and the policy planning tasks, where the TT combines these lasks generate the actions and only investigate the performance for the decision. Instead of only considering the performance of the dccision makings, our three tasks provide bottom-up analysis of the world model for decision making.
A.7 Broader Impact Statement
This work contributes an evaluation methodology for LLM-basod world models towards AI safety and re- sponsible development. Rigorous evaluation frameworks help prevent insecure deployment of vulnerable systems in high-stakes scenarios. However, we acknowledge potential dual-use concerns: improved world models could enhance autonomous agent capabilities, which could be misused for harmful automation. We emphasize that our work is alignment remains necessary before deploying world-model-basee agents in real-world applications.
and that substantial research on safety, robustness, and value
A.8
Code and Dataset Availability
The code and datasets can be access at:
htclib..ccooooo// htap://oomanaan////olllddoeooo., TMML.
18

---

Published in Transactions on Machine Learning Research (03/2026)
B Environments
B.1 Introduction of Tasks
There are 32 environment in (Wang et al., 2023) and dish-washing is selected as the example in the prompt, which is excluded for fair evaluation. Specifically, the environments can be categorized into two domains:
Daily-life tasks, including use-bandage, hang-painting sunburn, sweep-floor, bath-tub-water- temperature, make-campfire, refrigerate-foode cooking, take-photo, plant-tree, boil-water, and wash- clothes. For these tasks, the world model need to have the common knowledge about the procodure of completing these tasks, e.g.. first collecting the dirty clothes, then put them into the washing machine, then use the detergent and turn the washing machine for the wash-clothes task.
Scientific tasks, including mix-paint, blood-type, thermometer, clean-energy, lit-lightbulb, scale- weigh, multimeter, volume, space-walk, volume-container, conductivity, volume-stone, bird-life- cycle, balance-scale-weeghh metal-dctector, make-ico-cubes, forge-key and inclined-plan. These tasks requires the scientific knowledge to complete the tasks, e.g,, the world model need to know that the friction may decrease the speed of a block sliding down of the plane for the inclined-plars Then, the world model need to generate build a micro-simulltion to compare the frictions of the two planes.
B.2 Code for Demo Actions Generation
Only one playthrough of the game is provided in (Wang et al., 2024), which is not enough due to the randomness in the environments. Therefore, we curate the rule-based policy for each environment. Note that due the randomness the enviromments, e.g.; the largel color the mix-paint lask, the generated action sequences are different for different instantances of the same environment.
Code Sample 1: mix-paint
dededmmaationsseeceeeeeeeessseee (self):
target_ color self beffu_infoo [o] paint_ self. useful_infs [1]
color
dict color_dict>
"red": (1, 0, 0)
"orange": ["red", "yellox"], "yellow"; (0, 1, 0),
"green": ["yellou", "blue"), "blue": (0, 0, 1) "purple": ["red", "blue"),
"black": ["red", "yellou", "blue"],
paint_ paint_names_to_idr
for paint_idx: idx paint_nane enumerate (paint_namea)
paint_names_ idx [paint_psseeeeeeeeeeeeee name] paint_idx color_mixt pilor_mix_plan color diet [target_color]
demo demo_actions actions []
to -1
for color_ide idx color color_mix mix in enumerate (ctporrattcooooo_xx_____l
if color_id idx 0:
to_idx sainnt_ammes_t.oo. to_ idx continue
(ddo_dooccooroo [color_mix]
lemo_actions (
"pour O paint (ID: ()) in cup 0 (ID: formaat
color_miz,
(paint_nanes) to idx moolor_mixi + 1) 1, to to_idx (to_idx 1),
append ("nix cup (ID: ()) format (to_idx, (to_ids 1)))
19

---

Published in Transactions on Machine Learning Research (03/2026)
demo_ actinactions
Code Sample 2: blood-type
def demo aatnoo_ctionnseeee (self); _seful_info self. useful_inie info [1]
return
"give Type O O blood (ID: 3) to patient (ID: 2) ")".fo ffommtt aseful_infol]
Cerfulinffo00
),
"take Type 0 O blood (ID: 3)" formmat (aaeoulllllfff Lellllinfoool [0], useful_info (1]),
"give Type O O blood (ID: 3) to patient (ID: 2) (ormatt
useful_info [0], useful_info
Code Sample thermometer
def get_demo camooaacctinnntiieee (self): demo_actions =
"take thermoneter (ID: 4)",
"use thermometer (ID: 4) Rater (ID: 3)",
"answer () Celeius degree". format (self. water teaperature),
demo actions
Code Sample 4: clean-energy
def trr_demo_actione
demo_ actions []
change_station {
"sun": "solar farm", "water"; "hydroelectric power station",
"uind": "vind farm",
for region self regions:
demo actiona append( "change O O)" formatt
region. name, change_ (region. properties ['resource"]]
return demo_actions
Code Sample 5: lit-lightbulb
def get_demo_actions actions (self):
"connect light bulb (ID: 2) terminal1 red vire (ID: 3) terminall", "connect red wire (ID: terminal2 battery (ID: anode",
"connect battery (ID: 6) cathode black (ID: terminsli",
"connect black (ID: 4) terminal2 light bulb (ID: terninal2",
Code Sample scale-weigh
def get_demo aettinnnatiionneee (self): (self): demo_actions = "take () formfot (self. useful, info [0] name),
"put O ()" "f'.formateee (self iseful_info [o] name, self useful info (1] name),
"look",
"anawer 0."".frree (aelf, target_weightt
return demo_acte demo_actions
20

---

Published in Transactions on Machine Learning Research (03/2026)
Code Sample 7: use-bandage
def get_demo dedeeeoo_ationneCCiccccoo actions (self);
demo _ctioos -
"open bandage box (ID: 8)",
"look",
"take bandage (ID: 9)",
demo_ aemonactions
"put bandage (ID:
(ID: ")" (self useful_infuu (0])
Code Sample 8: hang-painting
def Leem__actions(ee (self); demo_actien actionss -
"take nail (ID: 7)",
"take hanmer (ID: 6)",
"hanmer nail (ID: 7) O with hamner (ID: 6) format (
self target_wall name
),
"take ()" formmtt (self sarget_picture name),
"hang ( on nail (ID: 7) fformat (self target.picture. name),
return
Code Sample 9: multimeter
def get_ aet__demo_actiine actions (self):
demo Lemo_actions -
"set multineter (ID: 2) resistance mode",
"connect multineter (ID: 2) terminall resistor O (ID: 3) terminall" ifotaaalli....atttt
self. tar.etrttrrrsistor..t
"connect multineter (ID: 2) terminal2 resistor ( (ID: 3) terminal2- .fommat
self targot_rosistor.tt
"look",
"ansver ohm" fornat (self... target
return demo actions
Code Sample 10: volume
def get_demo_actions(Ce actions (self): demo_actions =
"take ()" ffrmat (self useful info [1]. name), "measure the length of the { with the O)" format self. useful_info [0] name, self usfuu_effu.info [1]
"measure width
formoraat(
self useful_info [0] name, self. useful_infffooo name
"measure the height of the O with the formatt paeful_infofll info [0] name, self, uwwful_infouoooooeeeeeeelaeeiiue [1] name
self.um useful
"anawer ( cubic am" format (self. target_box_volurg
return
Code Sample 11: sunburn
def get_demo ret__dmo_acttona actions (self):
21

---

Published in Transactions on Machine Learning Research (03/2026)
"use sunscreen (ID: 4)",
"move to beach (ID: 3)", "look",
"take ball (ID: 8)",
"nove to house (ID: 2)",
"put ball (ID: in box (ID: 5)"
Code Sample 12: space-walk
def wtt__demo_cctione (self);
return
"put on вpace suit (ID: 7)*, "open inner door (ID: 5)", "nove to airlock (ID: 3)", "look",
"close inner door (ID: 5)",
"open door (ID: 6)", "move apace (ID: 4)",
Code Sample 13: sweep-flood floor
def cet_demo_actiona (self):
sveep_actions for garbage self useful_info:
sxeep_actions. ieriions.appende append
"sveep O to dustpan (ID: 3) with broon (ID: 2)", "fornat (garbage name)
demo actions =
[" take broon (ID: 2)", "take dustpan (ID: 3)
actep_actions
[
"open garbage can (ID: 4)", "dump dustpan (ID: 3) to garbage can (ID: 4)",
]
demo_actions
Code Sample 14: volume-container
def get_demo_ actions _et__emo_actionn (self):
demo_ remo_actions
"take format (self, usw.ul. [0] name),
"put O in sink (ID: ffornat (self. useful_inf [0] name),
"turn on sink (ID: 2)".
"turn off sink (ID: 2)",
"take ()", formatt (self www.ull [0] name),
"pour water in O into O"" formatt
self, useful_info (0) name, self useful [1] name
),
"look",
"ansver ( mL", format (self.
return demp actions
Code Sample 15: bullh-tubh-walerkempppaauuuu
def det_ antions ons (self);
sater_temp useful. info [o]
22

---

Published in Transactions on Machine Learning Research (03/2026)
cooling
"tura tap (ID: 5)",
"turn off cold tap (ID: 5)*
thermometer (ID: 6)
(ID: 3)",
hotting
"turn on hot tap (ID: 4)",
"turn off hot tap (ID: 4)",
"use thermoneter (ID: 6) on water (ID: 3)",
if water pater_temp > 40:
cooling_time times (vater_temp 35) // 5
vater_actio cooling * cooling_ times
elif water temp 35:
notting_times times (40 water_temp) //
vater_action hotting hotting.times
else:
water actions
demo actions
"take thermometer (ID: 6)",
"use thermometer (ID: 6) on water (ID: 3)",
]
water_ actioos ["bath"]
return demo_actions
Code Sample 16: conductivity
def get_demo
actiona _et__emo_actiona( (self):
actions
"connect light bulb (ID: 2) terminal1 red (ID: terminali",
"connect red wire (ID: 3) terminal2 to battery (ID: 6) anode",
"connect battery (ID: 6) cathode to black wire (ID: 4) terminall",
"connect black wire (ID: 4) terminal2
fork (ID: 7) terminali",
"connect fork (ID: 7) terminal2 to blue wire (ID: 5) terminali",
"connect blus wire (ID: 5) terminal2 to light bulb (ID: 2) terminal2", "look",
"take fork (ID: 7)",
if self.uses www.uuueeuuleeeeeeeee [0]:
return demo_actions ["put fork (ID: 7) in red box (ID: 8)"]
else:
return demo_actions ["put fork (ID: 7) black box (ID: 9]
Code Sample 17: make-campfire
def ret__deo_actionn actions (self):
return
"take axe (ID: 4)",
"use axe (ID: 4) on tree (ID: 5)",
"look",
"use axe (ID: 4) on chopped down tree (ID: 5)",
"look",
"take firevood (ID: 5)",
"put firevood (ID: 5) in fire pit (ID: 2)",
"take natch (ID: 3)",
"use match (ID: 3) firevood (ID: ")",
23

---

Published in Transactions on Machine Learning Research (03/2026)
Code Sample 18: refrigerate-food
def get_demo_actions actions take_objects []
put_objects [
for food in self. useful_info:
take_objee take_objects. ("ake O ffomatt (food. nane))
put_objecte, apppeeeee..pppeddee......
append ("put O in fridge (ID: 2) " fornat (food. name))
Lemo_action
take_ _ojectt
open fridge (ID: ")
put_objects
"close fridge (ID: ")",
"look",
"look",
"look",
1
return
Code Sample 19: volumo-stons
def get_ der_ddem_acctooneCeec actions (self):
demo_actions
"take measuring cup (ID: 4)",
"put measuring cup (ID:
sink (ID: 2)",
"turn on (ID: 2)",
"turn sink (ID: 2)",
"take measuring cup (ID: 4)",
"examine measuring cup (ID: 4)",
"take stone (ID: 3)",
"put stone (ID: 3) in zeasuring cup (ID: 4)", "examine measuring cup (ID: 4)", "ansver ffr.format (self.anaves anaver_volume
return
Code Sample 20: bird-life-cycle
def get_demo_actte (self);
"sit egg",
"ait egg", "sit egg",
"sit on egg", "sit on egg", "feed young bird",
"feed young bird",
"feed young bird".
"feed young bird", "feed young bird",
Code Sample 21: balance-scale--eigg!
def get dettoooooaccoon
actions (self):
weight_list [1. 1. 2. 5, 10]
weight_list_ins index [False]
def cond_coobination()
remaining welf.cube_vee veight
24

---

Published in Transactions on Machine Learning Research (03/2026)
using
remaining 10:
weight, list feat_indexx_=_e [-1]
remaining
using if remaining Baght_llitt_iddes [-2] = True renaining -
using 2
if remaining >- 2:
weight_ lintt_liit_inex_____ [-3] = True
remaining -
if remaining
if remaining
seight_list 1ist_istt_______ _ist__nnexx____ee index [0]
remaining
weight_ List___ist____ index [0]
weight_list_ index [i]
_.idd_cominnation
nighttaattion []
for idx, weight_list_ _dx in envoraat
if seight_list_1d
veight_actions +=
"take O*" format (self, useful info [idx].name) name),
"put O in right side of the balance scale (ID: 4)" formatt (
self celfuuseful_inte [idx] [idx].name name
"lock",
demo actions
"take cube (ID: 1
"put cube (ID: 10) in left side of the balance scale (ID: 3)",
]
["answer ".g".format (self, cube_ueight))
return demo_actions actions
Code Sample 22: metal-dctector
def get_demo actiona (self):
[es_ddmo_acction(C agent_ing init_position useful info [o]
targe_ targe_position ueffuuseful_infoll
direction
targe_ porge_position [0] agent_ position [0],
targe_position agent_init, position
poniitiottppositon [1],
h_dir fist_list- =
["south"] direction[0] if direction [0] >
else ["north"] * (-direction(0))
v_dir v_dir_liet_ list ['east"] * direction[1] [1] if direction[1) > else ["vest"] (-direction (1))
dir_list b_dir_ w_dir_list self. shuffle (dir_list)
25

---

Published in Transactions on Machine Learning Research (03/2026)
detect_actions "detect with netal detector (ID: 1
dir_ in dir_list:
astect_actions append("move Q"" fornat (dir_step))
detect_actions. append append("detec, ("detect with metal detector (ID: 15)
demo actions
("take metal detector (ID: 15)", "take shovel (ID: 16)
petect_actions
["look", "dig with shovel (ID: " "look", "take metal case (ID: 11)
return demo_action
Code Sample 23: cooking
def get_demo___ cooking_actiom.
(self):
for cooking,iter iten self receipt:
operations self receipt [cooking_iten]
cooking_actions actions += ["take format (cooking_iten )amet)
for operation in operations:
if operation in ("alice", "dice", "chop"]:
booing_actions += "() O with formattt
operation, cooking item name, self useful_ints ("knife") nane
if operation in ["fry"]:
cooking. actions cooking_actions (frmoormat
"cook cooking_itens item. self www.uluefuuueeeeee stove
operation in ["roast"]:
cooking_actien actions +- "cook O in (ormtor
cooking iten name, self useuul_ffu ["vven"] name
demo actions
"take format (self useful_info lnl.iinfooccee "cook_book"" nane), "read () format (self (self.useful. wwwuuluelllaee info infiiinofooooooooo sccok_bookk" nane),
"take 0 n. format (self (self.useful_ useuul_ifiitiieeeee ["knife"] nane),
]
acttinn
"prepare meal",
return demo actionctions
Code Sample 24: make-ice-cubes
def get_demo_acte actions (self):
return
"open freezer (ID: 2)
"examine freezer (ID: 2)*
"take ice cube tray (ID: 3)",
"put tray (ID: sink (ID: 4)",
(ID: 4)",
26

---

Published in Transactions on Machine Learning Research (03/2026)
"turn off sink (ID: 4)",
"take cube tray (ID: 3)", "put ice tray (ID: "close freezer (ID: 2)", "look", "look",
(ID: ")",
"look",
Code Sample 25; balance-scale- heaviest
def dettoociiooo (self);
semo_actions
"take Q)" format (self wwwuuuueelliieeee info [o] [o] nane), side self. useful_infieeeeeee
"put
balance
(ID: 3) format
[o] name
), "take forforraa formattttttt.. (self. useful, info [1] name),
"put O in right side of the balance scale (ID: 4.". formatt self. useful aaeful_info info [1] [o] name
, "look".
if len (self. useful_ info) == 2:
if folf.useful_ing [0] > self (1 # left is heavier
demo ceno_actions append t"take f'".formatsoi (self useful_infe [0] [o], name))
demo_actions append ( "put O O ()".fo" (ornatt useful. info [0] [0] name, self ansver bbx name
return demo_actions
elif self useuul [0] [i]
self uswful_ [1]:
# right is heavier demo_actions (append("saa) ff".forat (self (self wseful_info [1] [o] name))
demo_actions appeedd(
"put ( in for.format
uslfuuefful,att [1] [0] name, self awsver box. name
return detonnctions
else:
deno actions append ("take ()" O)" format (self useful_infe [0] [o], name))
demo_actions append (
"put () ",",fo (
uslfuus info [0] [0] name, self ansver
deno, demo_actions actions append tatake format (self. peefullinffe [1] [0] name))
Leno_action append "put Q in formatt
aelf unffuuseful.itt [1] [0] name, self ansver box. name
return demonactions
on_seal [0, 1]
for in range (2, len (self. useful_info))
if self wwa.uuusfulliteee inw.iiifoooonea [on_sale [o]] [1] self useful_info [oc__oooo_aaaaaaa [1]] [t]: demo_actions
"take ()" fornat useful_infe [cn_e (1)] [0] nane),
ffrmatt (self useful, [i][o]] name),
"put right
(ID: ")fforrmtt
27

---

Published in Transactions on Machine Learning Research (03/2026)
www.uu.efuluuaae [1] [0]
look",
on_acale [1]
elae:
demo_actions +=
"take O." format (self useful_infee scans [0]] [0] name),
"take formfornat wefffuuseuuliiffee [i] name),
"put O in left aide of the balance scale (ID: 3) ffrmmat (
unetuueeful_nd [1] (0) nane
"look",
on_scale [0] max_ weight -
if self useuul_luiieee [on_ [oo__naallee__ooeoeeao_ee__eleel [o]] [1] self helf.useful. iatesu___inoooo_sclalllll [1]][1]: [t]][[]]]]]:
demo_actions append("take formorr (self. useful info lon_scaleeeelllllleelllle [0] [0] name))
demo_actions append(
"put O in ()".format:
[0]] nane, self answer_ boo.eerrrooooeeeeeeee
max_ weight self useful info in scale [0]] [1]
elae:
demo acmonactions append (" "'".formae (self useful [on_scale [1]] [0] name))
demo _att_cctionsscoo append
"put O in formatt
unffuucffuliinfoooe inn [1]] [0] name, self answer box boo.eerrrbooooeeeeeeeeeeeee
name
max seight self useful info [on_e [1]] [1]
for cube, cube_mase mass self useful_info
if cube_mer mass max_veight:
actinnetione +=
"take O)" format (cube (eube.nane).
"put
format (cube. name, self anever_ box..ame)
return demo_actiona
Code Sample 26: take-photo
def get_demo_ www.ooaetionseeeeeeeeeeeeeeeeeso (self):
speed self properties [" cearrret__hutter_eee iso self. properties ["current_isee
aperture self camera properties "uurreett_pperuure""
target_aperture target_speed self useful info target_iso
useful, [o]
aseful_infoll
adjust_action =
"focus O". fornat (self.
if target_aperture > aperture:
idjust_action +* ['rotate aperture clockwise"] target_ aperture
elif target_aperture aperture:
adjust_actions += ['rotate aperture anticlockvise"]
target aperture
else:
pass
target_speed speed:
adjust_actions ["retate speed clockvise"]
28

---

Published in Transactions on Machine Learning Research (03/2026)
target_apeed speed
elif Larget_spece speed:
r'retate speed anticlockvise"]
speed speed
else:
pass
if target_iso is0:
adjust_actions += ('rotate iso clockwise"] (target_isa iso)
elif target_1ao 1s0:
adjust_action ** ["rotate iso anticlockvise"] (-target_iso + isc)
else:
demo actions
["take camera (ID: 2) "] adjust_actions ["press shutter"]
demo actions
Code Sample 27: plant-tree
def det_demo_actionsseee (self);
return
"take shovel (ID: 2)",
"dig with shovel (ID: 2)",
"take tree (ID: 8)", "look",
"put tree (ID: in hole (ID: 9)*,
"inventory",
"put soil (ID:
hole (ID: 9)",
"take jug (ID: 7)",
"put jug (ID: 7) in sink (ID: 5)",
"turn on sink (ID: 5)",
"turn off sink (ID: 5)",
"take jug (ID: 7)",
"pour water in jug (ID: 7) into soil (ID: 10)",
Code Sample 28: boil-water
def
(self):
return
"take pot (ID: 4)",
"put pot (ID: 4) in sink (ID: 3)",
"examine sink (ID: 3)",
"turn on sink (ID: 3)", "examine sink (ID: 3)",
"turn off sink (ID: 3)", "take pot (ID: 4)", "look",
"put pot (ID: 4) on stove (ID: 2)*,
"examine stove (ID: 2)".
"turn on stove (ID: 2)".
"examine (ID: 2)",
"examine (ID: 2)",
"examine (ID: 2)",
Code Sample 29: forgo-key
def www.ddemmmttonneeeeeeeeeeeeeeeee (self): demo_actions =
"take copper ingot (ID: 4)",
"put copper ingot (ID: 4) in foundry (ID: ")", "turn foundry (ID: 3)",
"look",
"look"
29

---

Published in Transactions on Machine Learning Research (03/2026)
"look"
"look"
"look",
"look",
"pour copper (liquid) (ID: 4) into key mold (ID: 6)",
"look",
"look",
"take copper key (ID: 4)",
"open door (ID: 5) with copper key (ID: 4)",
return actions
Code Sample 30: inclined- inclined-plane
def get, get_demo, actions (self):
look_ table
["look"] 5,
["look"] 6,
1.5: ["look"] 5.
2: ["look"] 5,
al aelf. www.uueeeuuuseeeeeeee [0] [0]
a2 molf.useful_inte [0] [1]
a1 a1_look book_table [a1]
look = took_table [a2]
demo actions
"take stopwatch (ID: 5)", "take block (ID: 4)",
"put block (ID: 4) inclined plane (ID: 2)",
"activate stopsatch (ID: 5)",
]
a1_look
[
"deactivate stopwatch (ID: 5)*,
"examine stopvatch (ID: 5)",
"reset stopwatch (ID: 5)",
"take block (ID: 4)",
"put block (ID: 4) on inclined plane 2 (ID: 3)",
"activate stopwatch (ID: 5)",
]
a2_look
"deactivate stopwatch (ID: 5)*,
"examine stopvatch (ID: 5)",
1
actions ["focus inclined plane (ID: "]
else:
return actions ['focus on inclined plane (ID: 2)
Code Sample 31: wash-clothes
def get_dema catto_actionneeeee (self))
washing, drying, busketing = 0. Q,, []
for cloth in self clothes
washing += [
"take formatt (cloth namlth..mmm))
"put O in washing machine (ID: "o..forma nommet(cllthnnmee)
drying [
30

---

Published in Transactions on Machine Learning Research (03/2026)
"take 0 format (cloth name), "put O in dryer (ID: .format (cloth name),
busketing "take .format (eloth. name),
"put O in basket (ID: 12) format (cloth name),
for cloth in celfn clothes:
busketing += [
"take 0 format (cloth. name))
"put O in basket (ID: format name),
demo actions
"open washing machine (ID: 2)*,
washing
"use bottle detergent (ID: 4) on washing machine (ID: 2)",
"close washing machine (ID: 2)",
"turn on washing machine (ID: 2)",
"zait",
"look",
"look",
"open washing machine (ID: 2)*,
"open dryer (ID: 3)",
]
drying
[
"close dryer (ID: 3)", "turn dryer (ID: 3)",
"sait",
"look",
"look",
"open dryer (ID: 3)",
"take skirt (ID: ")",
]
busketing
return
B.3 Analysis of Demo Actions
Figure 10 displays the numbers of steps of the generated rule-based policies for environments to complete the tasks. We note that the number of steps may vary due the randomness in the environments. For example, in mix paint, if the target color is black, 3 steps are needed, and other colors may only require 2 steps.
60
Steps
40
20
weigh H
0
paint
blood
thermo
energy
bulb
scale
bandage
hang
muiti
volume
sunbum
space
sweep
container
bathtub
conduct
campfire
fridge
stone
bird
metal
cooking
ice
heavy
photo
plant
boil
key
plane
washing
Game
Figure Steps to complete the tasks
31

---

Published in Transactions on Machine Learning Research (03/2026)
Table Environments (Wang et al., 2024)
Environments
Task Description
mix-paint (paint)
Your task is to use chemistry to create black paint.
blood-type (blood)
Your task is to give a correct type of blood to the patient.
Your task is to figure out the temperature of the water the pot.
thermometer (thermo)
Your task change all fossil-fucl power stations to use renewable energy while keeping the same capacily.
clean-energy (energy)
lit-lightbulb (bulb)
Your task is to lit the light bulb.
scale-weigh (scale)
Your task is to figure out the weight of the apple. Your task is to put bandages on any cuts. Your task is to hang the picture of a girl (ID: 11) on the back wall (ID: 5).
use-bandage (bandage)
hang-painting (hang)
multimeter (multi)
Your task to figure out the resistance of the resistor 0. Your task to figure out the volume of the green box. It is summer noon. The sky is clear. Your task is to take ball from the beach and put it in the box in the house. Protect yourself from sunburn! Your task is to conduct space walk.
volume (volume)
sunburn (sunburn)
space-walk (space)
Your task is to clean the garbage on the ground to the garbage can.
sweep-lloor (sweep)
volume-container tainer)
(con-
Your task to figure out the volume of the glass.
Your task is to make the temperature of the water in the bath tub to 35 40 Celsius degree by adding water from the taps. When you are done, take the action 'bath". Your task to figure out if the fork is conductive or not. If the fork is conductive, put it in the red box. Otherwise, put in the black box.
bath-tub-water- temperature (bathtub)
conductivity (conduct)
make-campfire (campfire)
Your task to make fire in the fire pit. Your task is to prevent the foods from spoiling. Your task is to figure out the volume of the stone.
refrigerate-food (fridge)
volume-stone (stone)
hard-life-yyol (bird)
Your task is to hatch the egg and raise the baby bird.
balance-scale-weigh (weigh)
Your task is to figure out the weight of the cube. Use the answer action to give your answer.
Your task is to find the buried metal case on the beach. You win the game by putting the metal case in your inventory. Your task is to prepare meal following the instructions of the cook book.
metal-detector (metal)
cooking (cooking)
make-ice-cubes (ice)
Your task is to make ice cubes.
balance-scale-heavisst
Your task is to put all heaviest cubes into the box.
(heavy)
Your lask to take a nice picture of orange (ID: 4), using camera with shutter speed of 1/2, aperture of 16, and iso of 1600.
take-photo (photo)
plant-tree (plant) boil-water (boil)
Your task is to plant the tree and water it.
Your task is to boil water.
forge-key (key)
Your task is to forge a key to open the door.
Here are two inclined planes with the same angle. Your task figure out which of the two inclined planes has the most friction. Focus on the inclined plane with the most friction after your experiment.
inclined-plane (plane)
wash-clothes (washing)
Your task is to wash the dirty clothes and dry them.
32

---

Published in Transactions on Machine Learning Research (03/2026)
C Prompts for World Model
C.1 Prompt for Next State and Reward/ Neward/Termingg Terminal Predictions
Code Sample 32: Code for Prompts of Generating Potential Actions.
prompt
"You are simulator of text gane. Read the task description of text game...
"Given the current gane state in JSON, "you need to decide the
state after taking action including the
score. An
prompt
"Your response should be the J8ON format. "It should
keys: 'nodified', 'removed', 'score'. of
"The 'nodified' key stores
object states are added
changed after taking the action.
"Keep it an empty list if no object added or modified.
"The 'renoved' key stores a 1ist of uuids of the objecte that are removed.
"Keep it an empty list if no object is removed.
"The 'score' key stores a JSON with three keys:
"'score', 'gameOver', and 'gameilon'. "'score' stores the current game acore. ">gameOver' stores bool value on shether the gane is over, "and 'ganeWon" stores bool value on whether the game is won. \n"
last_action if len (self. last_actions) == else self UUID itw.liibiipor_mmmee nodule gane_name). UUID
[-1]
current_atate None:
current_state get_state game, game, last_action, max_OUID, self game_name name)
make gero_aamm__eeeeeeeeeeeeeeeeeeeee state (current_state) max_uuid current_state ["max_UUID"]
else:
print ("use the predicted state")
current, serrent__tatte.e state_fateeeeeeee forpprompt surrent_state max aax_ len (eurrent_stated saanetttttt
start adding examples prompt self. build examples
prompt to example_prompt
# end of adding examples
# Task
prompt += "Here is the gane that need simulate An"
prompt "Task Description:\a" prompt ([faelf.task_desc/W
load rules
obj_dese preprocess_ctl cei.eesss.ci__ddsceeeeee.e.e oue [self. game_name]]
action self action, aectionrrlessl [self. game_name]
score_desc
score_rules [self game_name]
prompt "Here are the descriptions of all gane objects properties:\a" prompt derippp
prompt "\n"
prompt += "Here are the descriptions of all gane actions:\n"
prompt += detion,desc.strip)
prompt += "\n"
prompt += "Here is description of the game score function:An"
prompt += grore_desc.stripp
prompt += "\n"
data_state, data_ UUID base, data_action None, None, None prompt += "Here is the gane state:\n"
prompt
prompt "\n"
33

---

Published in Transactions on Machine Learning Research (03/2026)
prompt "TThe current game UUID base (max_ unid)"" prompt I"The action take 18:\n(action)(a"
Code Sample 33: The Predicted Next State and NeewrrdTTerminng in lit-lightbulb (bulb) environment.
{ game_state'; H "a'ame': 'room (ID: 1),, 'uuid'; 1, 'type'; 'World', 'properties':
isContainer's True, 'iaMoveable': True, 'ia0penable': False, 'in0pen': True, containerPrefix': 'in',, 'contains': ['agent (ID: 0),, 'light bulb (ID: 2)', 'red wire (
ID: 3),, 'black wire (ID: 4)', bbue wire (ID: 5)', 'battery (ID: 6) "), ('name': 'agent
(ID: 0)', 'uuid': 0, 'type': 'Agent', 'properties': ('isContainer's True, 'LaMoveable's True, 'ielpenable': False, 'is0pen': True, 'containetPrefix': 'in'), 'contains: (}), ( 'name'; 'light bulb (ID: 2)², 'uuid': 2, 'type': 'LightBulb', 'properties': isContainer': False, 'isMoveable': True, ii__eletrrial_oojee"""
True, 'conuuctive::
True, 'connects'; ('terminell': [3, 'terminall'), 'terminal2'; [None. None]), 'on':
False), 'contains'; (]), ('name': 'red wire (ID: 3)", 'uuid': 3, 'type': 'Wire', properties': ('isContainer'; False, 'laMoveable': True, 'be_electrical_objeer objectt: True, conductive'; True, 'coneccta;; ('terminall'; [2. 'terminall',, 'teriinal'' [None, None 1}, 'is_wire': True), 'contains'; (}) ('name's 'black
(ID: 4)', 'uuld'; 'type': 'Wire', 'properties'; ('lsContainer's False. 'isMoveable': True. "Ie_electrical True, 'conductive'; True, 'connecta'; ('terminall'; (None, Nonne), 'terminal2': (None. None)), 'is_wire'; True), 'contains': [}), ('name': 'blue wire (ID: 5)', 'uuid': 5, type': 'Wire', 'properties': ('isContainer's False, 'isMoveable'; True, is 1s.electricl__ojeee"" True, 'conductive': True, 'connecta'; ('terminall'; (None, None) 'terminal2'; (Nane, None)], 'is_vire': True), 'contains'; (}), ('name': "battery (ID: 6)', 'uuid'; 6, 'type': 'Battery', 'properties'; ('inContainer's False, 'isMoveable': True, dss_electrica__bjee None), 'anode'; (None, Nons) 'contains'; [}}}, ('score'; 0, 'gameOver': False, gameWon': False}
True, 'conductive'; True, 'connects' ': ('cathode'; (None,
C.2 Prompts of Generating Potential Actions.
Code Sample 34: Code for Prompts of Generating Potential Actions.
prompt
"You are a simulator a text gane. "Read the task description and the descriptions of all game actions of a text
game
"Given the current gane state in JSON, and the previous actions that lead to the current game state, "you need to decide the most O actions "that can help to complete the task step by step the current state, format
prompt +=
"Each of your action should in phrase with verb and objects
operates on.
includes:\n" "nove south" " ",An"
"Examples
"detect with netal detector (ID: 15) *n "dig with shovel (ID: 16) + ",An"
"open freezer (ID: 2)" + ",'n"
"put ice cube tray (ID: 3)
sink (ID: 4)* ",
"dice patato (ID: 2) with knife (ID: 8)" ",'
"give Type 0 negative blood (ID: 3) to patient (ID: 2) ", An" "read cook book (ID: 7)" ",an"
prompt +=
"Your response should be in the J8ON format. "It should have key: actain_ which includes the list the recomnended actions. \n"
action *n (self actions) self lastllastttiiiie actions [-1]
34

---

Published in Transactions on Machine Learning Research (03/2026)
max_UUID importlib... import_ module (self game_ UUID current_state None:
current_state get_ (self. game, last_action, max UUID, game_ seantt_state_fo_rroeeee
pake_game__ state (current__taaaae max_ current_ ("max_UUID")
elae:
print ("use the predicted state")
caantt_sttt__foo__pP__ max_uuic len @ancurrent_state ("ameetttteaaaatttttattaee
current_state
start adding examples
# example_pronpt self buill_
# prompt += example_prompt
end of adding examples
Task
prompt +* "Here is the game that you need simulate:\n"
prompt "Task Description:\al
prompt ""(eelf.takkdees)
obj_desc preprocess.obj_ddest (self obj. (et.eddeeeee...oo__rueeeeeee (self game__amee__m
cotton_rulesseeeeeeeeeeeeseeeeee action desc self action rules [self game game_name]
score_deac self score [self game_name]
prompt += "Here are the descriptions of all gane objects properties:\a"
prompt += des_ddec..tripp
prompt += "\n"
prompt * "Here are the descriptions of all actions:\n"
sesenddesssstrrp...... prompt += "An"
prompt +=
prompt +* "Here is description of the game score function:An"
prompt +- geore_desc.strige
prompt +- "(n"
data_ data_UUID. base. data_action None, None, None prompt "Here is gane state:\n" prompt f"[current_ sore prompt "\n"
prompt [TThe current game UUID base (max_uuid)\a)
if len (self lastaast llastactions) == 0: pronpt += "There is no previous actions."
else:
prompt *- "The previous actions format "is" if len (self. last_actions) == 1 else "are"
for action in self last_actions prompt to action + "\n"
Code Sample 35: The Generated Actions make-ice-cubes (ice) environment.
['open freezer (ID: 2) 'take ice cube tray (ID: 3) from freezer (ID: 2) sink (ID: ')', tray (ID: 3) from sink (ID: ")', 'put ice cube tray (ID:
'put cube tray (ID:
on sink (ID: 4)', 'take cube
freezer (ID: 2) 'close freezer (ID: "),, 'wait for ice form', 'open freezer (ID: 2) *check cube tray ice'
35

---

Published in Transactions on Machine Learning Research (03/2026)
D Action Matching
We provide detailed explanation of the action matching process:
for the matching of actions. Candidate set A' formation: At each state 8t, the game engine exposes the full set of valid actions through its getValidketious( interface. This set typically contains 500-800 actions depending on the environment and the current state. We enumerate all valid actions returned by the environment at the current state form A'. We have added this clarification to the revised manuscript. Threshold and tie-breaking: We do not apply any similarity threshold. the highest cosine similarity. In cases of exact ties (which are extremely rare in practice due to the continuous nature of embedding vectors), we select the first action in the enumeration order. We have addod this specification to the revisod paper.
Embedding model: We use OpenAl's text-embeeddng----aaae
always select the action with
E Format Error Handling
As noted in the main context, we observe that format errors occasionally occur during evaluation. To mitigate this, we rerun the experiments to ensure robustness. Since we primarily evaluate closed-source models, our ability to intervene the decoding post-processing level is inherently limited. Nevertheless, these occasional format errors do not affect our overall conclusions. Moreover, large language models continue to evolve, their Instruction-followite capabilities expected to improve steadily, and such format errors should become increasingly rare in future iterations.
36

---

Published in Transactions on Machine Learning Research (03/2026)
F Accuracy of Policy Verification
We provide the accuracy of the policy verification regarding the three criteria, i.c., score, gameWon and gameOver. We note that the performance on gameWWn and gameOver predictions are far better than the prediction of score.
(d) (0 1.0) (c) (p 0.75) (b) (p 0.5) (a) (p 0.25)
1.0
0.8
0.6
apt-Re-mini
0.4
0.2
0.0
1.0
got-4o
0.6
got
0.4
0.2
0.0
1.0
0.8
apt-4c
0.6
gpt-4c.mini
0.4
0.2
0.0
1.0
0.8
apt-4c
0.6
apt-do-mini
0.4
0.2
0.0
paint
blood
themoo
energy
bulb
scale
bandage
hang
multi
volume
sunbumn burn
space
sweep
container
bathtub
conduct
campfire
fridge
stone
bird
weigh
metal
cooking
heavy
photo
plant
boil.
key
plane
washing
Games
Figure 11: The accuracy of the world model verify the correct policies on score
to 1.0) (c) Up 0.75) (b) (p= 0.5) (a) (o 0.25)
1.0
0.8
got-4o
0.6
0.4
0.2
0.0
1.0
0.8
gpt-4o
0.6
apt-4o-mini
0.4
0.0
1.0
0.8
got-4o
0.6
gpt-4o-mini
0.4
0.2
0.0
1.0
0.8
got-4o
0.6
gpt-4o-mini
0.4
0.2
0.0
paint
blood
thermo
energy
bulb
scale
bandage
hang
multi
volume
sunburn
space
sweep
container
bathtub
conduct
campfire
fridge
stone
bird
weigh
metal
cooking
Ice.
heavy
photo
plant
boll.
key
plane
washing
Games
Figure 12: The accuracy of the world model to verify the correct policies on gameOver
37

---

Published in Transactions on Machine Learning Research (03/2026)
(c) (p - 0.75) (b) (p 0.5) (a) (p 0.25)
1.0
& 0.8
got-4o
0.6
apt-4o-mini
0.4
0.2
0.0
1.0
0.8
get-4o
0.6
9pt-4o-mini
0.4
0.2
0.0
1.0
0.8
got-4o
0.6
gpt-4o-mini
0.4
0.0
1.0
(d) (o 1.0
0.8
got-4o
0.6
apt-4e-mini
0.4
0.2
0.0
paint
blood
thermo
energy
bulb
scale
bandage
hang
multi
volume
sunbum
space
sweep
container
bathtub
conduct
campfire
fridge
stone
bird
weigh
metal
cooking
ice-
heavy
photo
plant
boil-
key
plane
washing
Games
Figure 13: The accuracy of the world model to verify the correct policies game Won
38

---

Published in Transactions on Machine Learning Research (03/2026)
G Step Accuracy of Action Proposal
We also provide the accuracy of cach steps for the action proposal tasks. We observe for most of the task, there is some key steps that the world model has low accuracy for the action proposal, which brings difficulties for the world model to complete the tasks.
1.3
1.3
10
10
0.5
0.5
0.8
0.8
0.8
0.2
0.2
0.2
02
0.0
0.0
0.0
0.0
0.0
paint
Mlod
thermo
erergy
bub
1.0
1.0
1.0
1.0
0.5
0.5
0.8
0.8
0.8
0.2
0.2
0.2
0.2
0.2
11
0.0
0.0
0.0
0.0
0.0
bandage
hang
muti
volume
1.0
1.0
1.0
1.0
3.0
0.8
0.8
0.8
0.8-
0.4
0.2
0.2
0.2
0.2
0.2-
0.0
0.0
0.0
0.0
2.5
5.0
0.0
30
Sarbum
space
sarepp
contaiiner
1.0
1.0
1.0
1.0-
1.0-
0.8
0.8-
0.8-
0.8
0.8
0.4
0.4
0.2
0.2
022
02-
0.2
0.0
0.0
0.0
2.5 campfire
5.0
0.0
0.0
midgc
stone
1.9-
1.04
1.0.
1.0.
1.0
0.8
0.4
0.2
6.2
0.2
0.2
02
10
0.0
10
weigh
cooking
Icc
heavy
1.0
1.0
1.0
1.0
1.0
0.4
0.4
0.4
0.4
0.2
6.2
0.2
0.2
0.2
0.0
1.0
key
1.0
0.4
0.2
Wasthing
Figure 14: Step correctness of the action proposal of GPT-4o-mini
39

---

Published in Transactions on Machine Learning Research (03/2026)
1.0
1.0
10
100
0.5
0.5
0.
0.8
0.8
0.2
0.2
02
02
0.2
0.0
0.0
0.0
0.0
0.0
peint
themmo
enwey
bub
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
0.4
0.4
0.2
0.2
02
02
0.2
10
0.0
0.0
0.0
0.0
0.0
bandage
hang
mull
volume
1.0
1.0
1.
1.0.
1.0.
0.8
0.5
0.8
0.8-
0.8-
0.4
0.4
0.2
0.2
0.2
0.2
0.2
10
-0
50
0.0
2.5
5.0
30
sumbum
space
Swoop
container
1.0
1.0
1.0
1.0
1.0
0.4
0.2
0.2
0.2
02
52
11-10
0.0
0.0
2.5
5.0
Cmmpfree
frigge
St0n6
1.0
1.0
1.0
1.0.
1.0
0.4
0.4
0.4
0.2
0.2
02
0.2
0.2
1.0
1.0
weigh
cooking
heavy
1.0
1.0
1.0
1.0
1.0
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
10
100
key
1.0
0.4
0.2
Figure 15: Step correctness of the action proposal of GPT-40
40


Reviews:
[{"review_id": "wOQS00sDBf", "reviewer": "Reviewer_q59p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:** This paper argues that the LLM-world models can make decisions solely, but they need more rigorous evaluation. To support the first part of this claim, the authors provide two observations to demonstrate how  LLM-based world models can make decisions solely. For the latter, they present three observations to show that the current evaluation frameworks are not sufficient. Then, they proposed an evaluation framework, which includes policy verification, action proposal, and policy planning. They conduct experiments on 31 environments and then provide their findings.\n\n**Strength:**\n- Overall, the paper is well motivated. The paper studies and interesting topic, and the findings are quite interesting. I think they could pave the way for researchers and practitioners in the LLM-based world model.\n- The bottom-up evaluation proposed in the paper is well motivated and systematically make the evaluation of LLM-based worlds more comprehensive. \n- Experiments were conducted in a quite extensive number of environments (which are 31), that cover tasks from daily life to some specific domain.\n- The authors provide the rule-based policies, which make the evaluation process possible.\n\n**Weakness:**\n- My main concern is that the “the LLM-world models can make decisions solely” argument is a bit over-claimed. They only test the method on one family of the LLMs. While I agree that the cost of the closed-source LLMs could be high, and the author said in the paper that they have tested on open-source models, I encourage the authors to provide the results on the open-source LLMs like Qwen or Llama.  Without testing results on a diverse family of LLMs, the claim is not well supported. The authors might consider adjusting the claim to the things that were tested in the paper.\n- The use of closed-source LLMs makes reproducibility harder.", "claims_and_evidence_yn": "No", "claims_explanation": "The claim “the LLM-world models can make decisions solely” is not supported well, as the authors only evaluate on a family of LLMs. Besides that, the policy planning is only evaluated with a single step (K=1) with no more search, which can make the “planning” term weak.", "audience_yn": "Yes", "audience_explanation": "I believe the proposed bottom-up evaluation and findings could be interesting to researchers working on LLM planning, world models, and LLM agents.", "requested_changes": "As my main concern is about the claim “the LLM-world models can make decisions solely”, I suggest the authors could provide more experimental results on the other LLMs family, and evaluate with k > 1.", "broader_impact_concerns": "No concerns", "additional_comments": "- Can you provide the pseudo-code for the evaluation protocol in section 5.2.\n- In the evaluation protocol of the action proposal, is there a risk of incorrect assessment, given that the rule-based policy's generated action sequence is not the only path to task completion?", "extra_scores": {}, "extra_text": {}}, {"review_id": "BNDpBBsN85", "reviewer": "Reviewer_JViA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper argues that LLM-based world models can, in principle, make decisions solely, without a separate actor or critic, but that existing evaluation practices are insufficient to assess this capability. Rather than proposing a new agent architecture, the paper’s main contribution is an evaluation-oriented reframing of world models for decision making.\n\nThe authors introduce a bottom-up evaluation framework consisting of three decoupled tasks—policy verification, action proposal, and policy planning—designed to isolate decision-relevant capabilities. The framework is evaluated across 31 diverse text-based environments using curated rule-based policies, comparing GPT-4o and GPT-4o-mini.\n\nThe experiments reveal consistent performance gaps between models, strong dependence on correctness in critical steps, degradation over longer horizons, and instability when combining world-model functionalities. Overall, the work contributes a clear conceptual argument and a principled evaluation methodology for decision-centric assessment of LLM-based world models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are supported by a combination of clear conceptual arguments and extensive empirical evidence. The paper does not rely on a single metric or task, but instead evaluates three well-motivated capabilities across many environments and settings. Figures and algorithms are consistent with the stated evaluation procedures, and the reported trends (e.g., error accumulation, importance of critical steps, instability from coupling) align with both intuition and prior literature.\n\nImportantly, the authors are careful not to overclaim: they explicitly document failure cases, format errors, instability, and the limitations imposed by computational constraints and ground-truth policy design. This transparency strengthens confidence in the validity of the conclusions.", "audience_yn": "Yes", "audience_explanation": "The paper directly addresses a central methodological question in modern reinforcement learning and LLM-based agent research: how to evaluate world models when used for decision making rather than pure simulation. Researchers working on model-based RL, planning with LLMs, evaluation methodology, and safe deployment of autonomous agents are likely to find the proposed framework and empirical findings informative.\n\nEven for readers who may disagree with some modeling choices, the work provides a useful lens for diagnosing model behavior, making it relevant to a broad segment of the TMLR audience.", "requested_changes": "**Critical Changes**\n1. **Clarify the scope of the main claim (“can make decisions solely”) earlier and more explicitly**.\n\n   While the paper’s argument is defensible, the phrasing can be interpreted as a claim of practical superiority rather than an existential or evaluative claim. The introduction and abstract would benefit from explicitly stating that the contribution is conceptual and evaluative, not a proposal of a new optimal decision-making system.\n2. **More clearly frame the evaluation results as diagnostic rather than definitive.** \n\n   The experiments convincingly show differences between models and reveal failure modes, but the paper should more explicitly caution against interpreting absolute success rates as measures of general intelligence or full task competence, especially given the reliance on single-path rule-based policies.\n\n**Non-Critical Changes**\n1. **Discuss alternative valid policies more prominently when interpreting results**. \n\n   Although mentioned in the limitations, the fact that many environments admit multiple valid strategies deserves stronger emphasis in the main experimental analysis, particularly for the action proposal and policy planning tasks.\n\n2. **Minor language and phrasing revisions for polish**. \n\n   There are small grammatical issues and occasional awkward wording (e.g., “unstability,” subject–verb agreement). These do not affect understanding but should be cleaned up for final publication quality.\n\n3. **Optional: add a short synthesis paragraph at the end of Section 5.**\n\n   While each task has clear takeaways, a brief summary connecting the three tasks back to the motivating observations would further improve readability.", "broader_impact_concerns": "The paper raises no major unresolved ethical concerns. It appropriately highlights the relevance of rigorous evaluation for safe and reliable deployment of autonomous systems. A minor risk—overestimating decision competence from simulation-only evaluations—is already acknowledged and largely mitigated by the paper’s central message. No additional broader impact statement is required.", "additional_comments": "This is a thoughtful, well-executed paper that prioritizes clarity and rigor over novelty for its own sake. While it does not introduce a new agent or learning algorithm, it makes a meaningful contribution by shifting the discussion toward what it actually means to evaluate world models for decision making.\n\nThe decoupled evaluation framework is particularly valuable as a diagnostic tool and may influence how future work reports results in this area. With minor clarifications and polish, the paper would be a solid fit for TMLR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jhnwcD3gai", "reviewer": "Reviewer_581V", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "-- Summary\n\nLLM-based world models can be used for decision making without a separate actor, but current evaluation setups don’t really test that cleanly. The authors propose a three-part evaluation: (1) policy verification (can the model judge if an action sequence will succeed), (2) action proposal (can it suggest top-K useful next actions), and (3) policy planning (can it combine those abilities to complete tasks). They test GPT-4o and GPT-4o-mini on 31 ByteSized32 text-game environments and add rule-based policies to make evaluation repeatable under randomness.\n\n-- Strengths\n\n1) The framing is useful: it tries to separate “world model capability” from planner/agent glue, which is often confounded.\n2) The experimental sweep is large (31 envs, multiple ρ and K, many trials).\n3) Curating rule-based policies is a practical artifact that makes systematic evaluation possible.\n\n-- Weakness\n\n1) Only two models from one provider → hard to generalize to “LLM world models” broadly. \n2) Action proposal is scored by matching the scripted next step, so alternative valid actions get penalized.\n3) Verification tests only correct policies, so it doesn’t measure false positives (real verification needs negatives).\n4) Planning is limited (K=1 and partial roll-in), so “decisions solely” is not strongly demonstrated.\n5) Nearest-action snapping via embeddings + format failures need ablations/quantification.", "claims_and_evidence_yn": "No", "claims_explanation": "The results support narrower claims (GPT-4o > GPT-4o-mini; long-horizon error accumulation; top-K helps; coupling increases instability). But the headline claim that world models can “make decisions solely” is not convincingly shown because planning is constrained (K=1, roll-in), action proposal is evaluated against a single reference action sequence, and verification has no negative cases so reliability is unknown.", "audience_yn": "Yes", "audience_explanation": "The evaluation decomposition (verification/proposal/planning) and the “critical steps matter more than step count” insight are relevant to LLM agents and model-based RL, and the 31 environment policy curation is reusable.", "requested_changes": "-- Either strengthen the “decision making solely” evidence (planning from start with p=0, and at least some k > 1 search) or narrow the claim/title to what’s actually tested. \n-- Add negative policies for verification and report false positives/False negatives. \n-- Reduce single-trajectory bias: evaluate proposals by environment progress/success, not only “did you include the scripted next action.” \n-- Add at least one baseline “standard LLM agent” (direct next-action, or ToT-style), and include aggregated summaries (mean/median across envs; daily-life vs scientific splits).", "broader_impact_concerns": "Low risk in this benchmark setting, but the paper should briefly acknowledge brittlness/instability and reproducibility constraint (proprietary models + cost), since the direction targets more agentic systems.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UqR9FoxjPO", "reviewer": "Reviewer_QyyD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission argues that **LLM-based world models can be used not only as transition predictors (support modules) but also as decision-making components on their own**, and that **current evaluation practices are insufficient** for assessing them in that role. \n\nConcretely, the paper:\n\n1. **Motivates a decision-centric view of world-model evaluation** via several “observations,” e.g., that prediction accuracy does not always translate to correct decisions (illustrative value-ranking example), that only a subset of states matter for decision quality (cliff-walking example), and that end-to-end agent performance is often confounded by other modules like search/value models. \n2. **Proposes a “bottom-up” evaluation framework** with three tasks intended to decouple world-model capabilities:\n\n   * **Policy verification:** given an action sequence, predict whether executing it completes the task (multi-step rollouts using the world model).\n   * **Action proposal:** propose top‑K actions that could lead to task completion from the current state.\n   * **Policy planning:** combine action proposal + transition prediction to generate a policy without a separate actor/critic. \n3. **Implements and evaluates these tasks** on **31 text-game environments** drawn from Wang et al. (2023; 2024) and **curates rule-based policies** for each environment to produce “ground-truth” successful action sequences across random task instantiations. \n4. **Runs experiments comparing GPT‑4o vs GPT‑4o‑mini** on these tasks across multiple settings (varying fraction of rollout planned/verified; varying K for action proposal) and reports qualitative findings about relative performance, error accumulation, “critical steps,” and instability when composing capabilities. \n\n### Key strengths\n\n* **Clear high-level motivation**: the paper is right that evaluating LLM “world models” purely as one-step predictors or only through an end-to-end agent stack can miss important aspects of decision usefulness. \n* **Decoupled task design is potentially useful**: policy verification and action proposal are plausible unit tests for world-model-driven planning, and “policy planning” as a composed test can reveal integration instability. \n* **Diverse task suite and procedural grounding**: the 31 environments span both daily-life and “scientific” tasks, and the paper invests effort in curating executable rule-based policies and reporting step counts. \n* **Practical implementation details** (prompt structure, JSON state deltas, etc.) are at least partially described and the paper claims an anonymous repo. \n\n### Key weaknesses\n\n* **Core notation/protocol ambiguity (likely an error) around the rollout fraction ρ and the pseudocode**: the written description of what ρ means appears inconsistent with Algorithm 1/2 as typeset, and some reported trends imply the opposite convention. This is a clarity/reproducibility blocker. \n* **Evaluation signals are somewhat misaligned with “decision quality” in places**:\n\n  * Action proposal is scored by whether the *single rule-based policy action* appears in the top‑K list, which penalizes alternative valid actions and conflates “matching one reference trajectory” with “good decision-making.” \n  * Policy verification evaluates only **successful** policies; there is no false-positive analysis (can the model detect that *bad* policies fail?), which matters if verification is used for search. \n* **Some headline claims are only weakly quantified** (e.g., “steps aren’t a good difficulty indicator,” “instability obscures gaps”)—the evidence is suggestive but not yet rigorous. \n* **Heavy dependence on closed models and tooling choices** (GPT‑4o/mini + an OpenAI embedding model used to map generated actions to valid actions) raises reproducibility and confounding concerns unless fully specified and reported.", "claims_and_evidence_yn": "No", "claims_explanation": "### Claims that are fairly well supported\n\n* **GPT‑4o outperforms GPT‑4o‑mini** across the proposed tasks, especially on tasks characterized as requiring more domain/scientific knowledge. This appears consistent across the policy verification and action proposal plots. \n* **Error accumulation with longer-horizon simulation** is supported qualitatively: performance decreases as the model is responsible for more steps of verification/planning (as described in the text and consistent with the plots). \n* **Formatting/structured-output failures occur** (“empty dictionaries,” format errors) and can impact evaluation stability. This is explicitly reported and is important evidence that current evaluation/usage pipelines can be brittle. \n\n### Claims that need stronger evidence and/or clearer framing\n\n* **“Total number of steps is not a reliable indicator of task difficulty.”**\n  The paper provides step-count distributions and qualitative discussion, plus step-wise accuracy plots that suggest “critical steps” dominate. However, the evidence is mostly anecdotal/visual. I would expect at minimum:\n\n  * a correlation analysis (e.g., Pearson/Spearman between step count and accuracy, overall and by domain),\n  * and a clearer operational definition of “critical steps” (e.g., lowest‑p percentile steps, or steps where action space entropy spikes, etc.). \n* **“Combining functionalities introduces instability and obscures the performance gap.”**\n  The policy planning results do look noisier/less separable than action proposal or verification, but the claim “obscures the gap” should be backed by explicit measurements (e.g., distribution of GPT‑4o minus GPT‑4o‑mini across tasks for each evaluation mode; variance/CoV across seeds; bootstrap CIs). Right now it reads as plausible but not rigorously demonstrated. \n* **“World models can make decisions solely.”**\n  The proposed “policy planning” task is intended to demonstrate this, but the evaluation as written has a few issues:\n\n  * The planning setting is restricted to **K=1** with no search; that is a legitimate first step, but it weakens the generality of the “planning” claim. \n  * More importantly, there appears to be **confusion/possible mistakes in the definitions of ρ and the algorithms** (see below). If the reader cannot precisely tell what fraction of the trajectory is executed in the real environment vs simulated, it is hard to interpret “solely.” \n\n### Major clarity/accuracy issue: inconsistency around ρ and algorithms\n\nA central control parameter ρ is described in a way that appears inconsistent with the pseudocode:\n\n* In the **policy verification** section, the paper states that the world model verifies the **last ρ proportion** and that **when ρ=1.0, the world model verifies the full action sequence from the initial observation**. \n* But **Algorithm 1** as printed executes the environment when (t < \\rho N) and uses the world model otherwise—which would imply the opposite: when ρ=1.0, the environment executes essentially everything and the world model verifies almost nothing. \n\nSimilar confusion appears in **Algorithm 2** (policy planning) versus the textual description and the qualitative trend statements. \n\nThis is not just a cosmetic issue: it affects how readers interpret the reported curves and conclusions about “long-horizon” behavior. I consider this a **critical fix** for the paper’s claims to be “clear evidence” in the TMLR sense.", "audience_yn": "Yes", "audience_explanation": "I believe this work is likely interesting to multiple sub-communities:\n\n* **LLM agents / planning / tool use**: The paper reframes LLM “world models” as decision components and proposes evaluation tasks aligned with planning.\n* **Model-based RL and world-model research**: It highlights confounds in end-to-end evaluations and argues for decoupling capabilities, which echoes longstanding issues in model-based evaluation. \n* **Benchmarking and evaluation methodology**: A curated suite of procedural environments + policies, along with unit-test-style tasks (verification/proposal), could become a useful diagnostic benchmark if the protocol is clarified and made robust. \n\nEven if the empirical results on GPT‑4o vs GPT‑4o‑mini are not themselves “surprising,” the *evaluation framing* and the curated evaluation harness could be of interest.", "requested_changes": "Below is a list of proposed adjustments. For each, I mark whether it is **Critical** (I would need it to recommend acceptance) or **Strengthening** (would improve the work but not strictly required if claims are adjusted).\n\n1. **Fix and clarify the definition of ρ across the paper, and correct Algorithms 1 and 2 accordingly.** (**Critical**)\n\n   * Ensure the pseudocode matches the text and the experiments.\n   * Explicitly define: does ρ denote the fraction of steps simulated by the world model, or executed in the environment?\n   * Verify the reported trend statements (e.g., “as ρ increases performance decreases”) are consistent with the corrected definitions. \n\n2. **Clean up Algorithm 2 formatting/indentation and specify the rollout budget precisely (why (2(1-\\rho)N)?).** (**Critical**)\n\n   * The current presentation makes it hard to replicate the planning procedure.\n   * Explain the rationale for the extra factor 2 and whether it applies uniformly across tasks. \n\n3. **Fully specify the action canonicalization/matching procedure used in action proposal.** (**Critical**)\n   The paper states that generated actions may be invalid and are mapped to valid actions using an OpenAI text-embedding similarity search. \n   Please add:\n\n   * Which embedding model/version is used, with parameters.\n   * What the candidate set (A') is exactly (all legal actions at the state? all template actions? how enumerated?).\n   * Whether the mapping is applied to each of the top‑K proposals individually.\n   * How ties/low-similarity cases are handled.\n     This step is a potential confound and needs to be transparent.\n\n4. **Report structured-output failure rates and how they are handled (empty dicts, invalid JSON, etc.).** (**Critical**)\n\n   * Provide per-task (or at least aggregated) rates of format violations.\n   * State whether failures are counted as incorrect, retried, or repaired.\n     Given the paper notes frequent failures, this is essential for interpreting “instability” and model comparisons. \n\n5. **Quantify the “steps vs difficulty” claim rather than relying on qualitative interpretation.** (**Critical unless the claim is softened**)\n   Minimal acceptable addition:\n\n   * a scatter plot of step count vs performance (per task) and correlation coefficients, with confidence intervals;\n   * optionally stratified by “daily-life” vs “scientific” tasks (the paper already defines these categories). \n\n6. **Quantify the “instability / obscured gap” claim.** (**Critical unless the claim is softened**)\n   Suggestions:\n\n   * compute the distribution over tasks of (GPT‑4o − GPT‑4o‑mini) for each evaluation mode (verification/proposal/planning);\n   * compute variance across runs for planning vs other tasks (even with temperature=0, environment randomness remains);\n   * report bootstrap confidence intervals for success rates/accuracies. \n\n7. **Make the evaluation less “single-reference-policy dependent,” or explicitly bound the interpretation.** (**Strengthening; could be Critical if left unacknowledged**)\n\n   * For action proposal, consider metrics that do not assume a unique correct action (e.g., whether sampled actions from the top‑K can still complete the task; or rank-based metrics with multiple acceptable actions).\n   * Alternatively, clearly frame results as “agreement with a particular reference policy,” not “decision correctness.” \n\n8. **Add negative examples for policy verification (false positive analysis).** (**Strengthening**)\n   If policy verification is meant to support planning/search, we need to know whether the world model can *reject* incorrect policies. Even a limited negative set (perturbed policies, swapped actions, random prefixes) would help.\n\n9. **Add a baseline that separates “world model” benefits from “LLM as a policy.”** (**Strengthening**)\n   For example:\n\n   * an “actor-only” baseline: directly ask the LLM for next action without state-transition prediction;\n   * or a heuristic baseline that uses access to the action rules.\n     This would strengthen the argument that world-model structure matters.\n\n10. **Improve writing quality and terminology consistency.** (**Strengthening**)\n\n* Fix typos (“unstability” → “instability”, etc.).\n* Ensure consistent use of “policy planning” vs “planning,” and define “decision solely” precisely.", "broader_impact_concerns": "This paper is primarily an **evaluation framework/benchmarking** contribution, which is generally **low risk** and arguably *beneficial* for safety, since better evaluations can prevent overconfident deployment. The submission also explicitly discusses “high-stakes” deployment motivation. \n\nPotential concerns (minor, but worth acknowledging):\n\n* Improved evaluation and planning-with-LLMs can contribute to **more capable autonomous agents**, which could be repurposed for misuse (e.g., automation of harmful tasks). This is a general dual-use concern common to agentic AI research.\n* The work uses proprietary models; this is not an ethical harm per se, but it can limit transparency and independent verification.\n\nI would not call these concerns severe, but I recommend adding a **brief Broader Impact Statement** (if none exists) that:\n\n* frames the work as evaluation/safety-oriented,\n* acknowledges dual-use at a high level, and\n* emphasizes responsible deployment and the role of rigorous evaluation in mitigating harms.", "additional_comments": "* I like the **central thrust**: shifting evaluation from “predict the next state well” toward “support decisions/planning” is a helpful perspective, and the decoupling into verification/proposal/planning is a reasonable structure. \n* The paper would benefit from being more explicit about what it means for a world model to “make decisions solely.” Right now the message is clear conceptually, but the experimental protocol ambiguities (ρ) undermine the clarity.\n* Consider adding a small section that explicitly states:\n  **(a)** what aspects of “world modeling” are being tested (rule-following? commonsense? long-horizon consistency?), and\n  **(b)** what is *not* tested (generalization to unseen action schemas, learning dynamics from data, multimodal grounding, etc.).\n* The appendix contains helpful policy snippets and prompt code fragments; however, the paper’s main text still feels like it relies on “trust us, the prompt contains rules.” Even one fully worked example environment (with actual object/action/score rules excerpted) would make the paper more self-contained.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)