Graph World Model
Authors: Tao Feng
Yexin Wu
Guanyu Lin
Jiaxuan You
Venue: ICML 2025 poster (2025)
Keywords: Graph world model
Unstructured data
Multi-modal information
Zero-shot/few-shot capabilities
Primary area: deep_learning->graph_neural_networks
TL;DR: The Graph World Model (GWM) efficiently integrates unstructured and graph-structured multi-modal data, enhancing diverse tasks through a unified framework.

Abstract: World models (WMs) demonstrate strong capabilities in prediction, generation, and planning tasks.
Existing WMs primarily focus on unstructured data while cannot leverage the ubiquitous structured data, often represented as graphs, in the digital world. While multiple graph foundation models have been proposed, they focus on graph learning tasks and cannot extend to diverse multi-modal data and interdisciplinary tasks. To address these challenges, we propose the Graph World Model (GWM), a world model that supports both unstructured and graph-structured states with multi-modal information and represents diverse tasks as actions. The core of a GWM is a generic message-passing algorithm to aggregate structured information, either over a unified multi-modal token space by converting multi-modal data into text (GWM-T) or a unified multi-modal embedding space by modality-specific encoders (GWM-E). Notably, GWM introduces action nodes to support diverse tasks, where action nodes are linked to other nodes via direct reference or similarity computation. Extensive experiments on 6 tasks from diverse domains, including multi-modal generation and matching, recommendation, graph prediction, multi-agent, retrieval-augmented generation, and planning and optimization, show that the same GWM outperforms or matches domain-specific baselines' performance, benefits from multi-hop structures, and demonstrate strong zero-shot/few-shot capabilities on unseen new tasks. Our codes for GWM is released at https://github.com/ulab-uiuc/GWM.

Graph World Model
Tao Feng'
Yexin Wu²
Guanyu Lin
Jiaxuan You'
Abstract
1. Introduction
A world model (WM) (Ha Schmidhuber, 2018) constructs the world observations as states and predicts future states based on given actions. Modern world models with massive data (Liu et al., 2024b; Cui & Gao), demon- strating successful prediction, generation, and planning ca- pabilities. However, existing world models do not directly generalize to structured data, primarily graphs, that are ubiq- uitous in science (Jin et al., 2018; You et al., 2018) and industry (Ying et al., 2018; You et al., 2022) and can be fur- ther enriched with multi-modal information (Ektefaic et al., 2023). Therefore, our paper aims to raise attention this pressing research question: Can we extend a WM to handle graph-structured data across a broad range of tasks?
World models (WMs) demonstrate strong capa- bilities in prediction, generation, and planning tasks. Existing WMs primarily focus on unstruc- tured data while cannot leverage the ubiquitous structured data, often represented as graphs, in the digital world. While multiple graph foun- dation models have been proposed, they foeus on graph learning tasks and cannot extend diverse
trained
data and interdisciplinary tasks. To address these challenges, we propose the Graph World Model (GWM), world model that supports both unstructured and graph-structured states with modti information and repre- sents diverse tasks as actions. The core of a GWM is generic message-passing algorithm aggregate structured information, either over unified multi-modal token space by converting multi- modal data into text (GWM-T) or a uni- fied multi-modal embedding space by modality- specific encoders (GWM-E). Notably, GWM in- troduces action nodes to support diverse tasks, where action nodes are linked to other nodes via direct reference or similarity computation. Exten- sive experiments on tasks from diverse domains, including multi-mon modll generation and matching, recommendation. graph prediction, multi-agent, retreeval-aummented generation, and planning and optimization, show that the same GWM out- performs or matches domain-specific baselines' performance, benefits from multi-hop structures, and demonstrate strong sero-shot/few-shot ca- pabilities on unseen new tasks. Our codes for GWM is released at https:/ / //tgt/gggttt/t//////// com/ ul ulab-uiuc/CWM
Existing WMs mainly focus on unstructured data. For ex- ample, iVideoGPT (Wu et al., 2024a) and Genie (Bruce et al., 2024) are successful world models over video data. However, the relations and structures in the data are rarely explored in these works. Although some works (Zhang et al., 2021; Zhu et al., 2022) attempt to model structured data in WM using graphs, they have focused solely on planning problems in a specific domain. In recent years, researchers have also explored the concept of the Graph Foundation Model (GFM) (Liu al., 2023a; Chen al., 2024a). How- ever, these methods are confined to predefined graph learn- ing tasks, which cannot easily extend to: (I) multi-modal input data including images and text, (2) diverse tasks be- yond standard graph prediction tasks, and (3) data without explicit structure, i.e. standard unstructured data.
To address these challenges, we propose the Graph World Model (GWM) that embeds the capabilities of the graph into the WM, which models the current state as graph and the action existing methods). Various tasks can be expressed as action nodes; for example, in a graph prediction task, predicting the label of a given node/edge/subgraph leads to intended action nodes that link relevant nodes in the state graph, i.e., target nodes, to the action node; in sirievall-augenneee generation (RAG) task, we can also represent user query an unintended action node that links to target nodes in the state graph via embedding similarities.
node (see Table 1 for comparison with
'Department of Computer Science, University of Illinois Ur- bana Champaign Urbana, IL, USA. Correspondence to: Tao Feng <taofeng2 @ illinois. Jiaxuan You <jiaxuan@ idu
Proceedings of the nrnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To build GWMs, we first introduce a simplified token-based GWM (GWM-T), which integrates multi-modaa data like

---

Graph World Model
is task-related. It can be a real-world operation, code function in the digital world, and even some queries and instructions. (3) Transition. The transition P(s++1 (s, at) depicts the transit probability from the current state st to its next state 8141 after the execution of action Oz.
Table Comparison with existing representative works from three perspectives: task type, data structure, and model type. Compared with existing WM and GFM, GWM can tackle multi- domain tasks and be applied to both structured and unstructured data.
Method
Task Type
Data Structure Model Type
Genie (Bruce et al., 2024)
Video generation
Unstructured
WM
2.2. Multi-modal World Represented by Graphs
L (Zhang et al., 2021)
Planning
Structured
WM
BioBridge (Wang al.)
Biomedical domain
Strcttred
GFM
Graph for state modeling. We define the world state as a graph g = (V,E) to represent multi-modal data with complex relationships, shown in Figure 1. Specifically, V= = =v} represents the set of nodes where each node v = [ur, ob, .v] consists of mulli-moda information, including image v", table v², and text ee modality; when modality is absent, the corresponding tensor will be empty. & = E U Em } is the edge set consists of explicit edges E, and implicit edges En Explicit edges Ep are often those established through expert knowledge or ground truth observations. For example, in the ogbn-arxiv dataset (Hu et al., 2020), edges are determined based on references between papers and historical collaborations between authors. Implicit edges Em are those constructed through connections represented by embedding similarities in the dataset. A typical example is in many protein datasets (Heumos et al., 2023; Stuart et al., 2019; Stuart & Satija, 2019), where edges are obtained based on the similarity of certain node feature embeddings.
LLAGA (Chen et ol., 2024a)
Graph domain
Strettred
GFM
GWM-T GWM-E
Multiple domains
Both
GWM
Multiple domains
Both
GWM
image, table, and text into text modality and represents them as nodes a graph state. We further develop a token-level pesssage-aassing algorithm that aggregates the neighbor in- formation to update the text representation of the state node. Finally, the target nodes on the state graph and prompted action nodes will be fed into multi-modal decoders such as LLMs and Stable Diffusion (Rombach et al., 2022). Despite its simplicity, GWM-T sometimes suffers from high token costs and limited context length. Inspired by latent diffusion models, which introduced modeling in latent space rather than directly on pixels like diffusion to enhance model per- formance and efficiency, further develop an embedding- based GWM (GWM-E). GWM-E first employs modality- specific encoders to process different modalities into node embeddings. Then it utilizes embedding-level message pass- ing to update the node embedding. Finally, the multi-modal information in target state nodes consolidated through a multi-hop projector before passing them to the decoders.
Different levels of world action and state transition. We model the action an action node that queries the current state nodes to obtain the target nodes v, using function R, whose process can be formulated as v, = R(2, a). We fur- ther categorize the world's actions into two types, as shown in Figure 1: one is directly related to the specific structures on the graph, called intended action ag. It includes three levels: node-level, edge-level, and graph-level. The other is indirectly related the specific structures on the graph through semantic relations such Retrioval-augmenned Generation (RAG), called unintended action a. As shown in Figure 1, to implement this action, we can first calculate the similarity between the action node and state nodes, and then retrieve the top-k state nodes for querying. According to the introduction in Section 2.1, we can conclude that an action causes transition of state 81+11 = [er( ae), which includes three types: update nodes, update edges, and up- date graphs. Here, fer means a transition function, which can be a neural network.
We conduct extensive experiments on 6 tasks from di- verse domains, including world prediction (multi-modal generation and matching, recommendation, graph predic- tion), world generation (multi-agent collaboration, retrieval- augmented generation), and world optimization (planning and optimization), with both proposed GWM variants and domain-speeift baselines. Results show that (I) GWMs generalize across domains, as the same GWM outperforms or matches domain-spectin baselines' performance, (2) graph information matters in GWM, as GWMs benefit from multi-hop graph information, and (3) GWMs demonstrate strong vero-shot/fewwwee capabilities on unseen new tasks.
2. Graph World Model
2.3. Instantiations of GWM
2.1. World Model Preliminaries
As shown in Figure 2, We have listed some representative instantiations that can be unified into a graph world model from three aspects: (a) world prediction, (b) world genera- tion, and (c) world optimization.
A world model aims to predict future states based the current state and action, which contains the following main components: (1) State. The state s of the world model estimates the observation of the world. It usually consists of multi-modal information and the state of the t step/time slot can be depicted as st. (2) Action. The action a at step/time
World prediction. (1) Multi-modal generation and matching. As shown in Figure 2(a), the task includes
2

---

Graph World Model
State
Action
State tel
Legend
Mutti-modal
Action
Node-koval
Edge-love
Graph-level
Update graphs
State node
Level
Image
Node
Action node
Table
Target Nodes
Action Node
Targel node
Weys to obtain target nodes
Select action-dreectly
Calculate nodo Rietneve Top-k
State stato edga
related modes
similiaitt
nodes
Action -largel link
Update edges
Edge
Update nodes
Node-node similarty
Intended
Unintended
Updated nodiseddge! graph
Action
AAction
Figure 1. Multi-modal world state transition can modeled via graphs. We model the current state as graph and each node contains more modalities from image, table, and text. Further, the world action is modeled as action node that queries the current state nodes. We categorize actions into two types: intended actions, which include three levels node, edge, and graph- and unintended actions, whose implementation involves similarity computation similar to RAG. Finally, the transition function updates states at three different levels based on state and action: update nodes, update edges, and update graphs.
two subtasks. The multi- generation task (Rombach et al., 2022; Zhang et al., 2023a) involves predicting miss- ing modalities given the available modal information and their interconnections. Specifically, considers clusters of corresponding modalities (such as an image, table, and text describing the same entity) as state nodes, and the re- lationships between clusters, such as similarity, are treated as edges. Thus, the action node here is at the node level. The multi-modal matching task (Rombach et al., 2022), similar to CLIP's pre- treiniai task (Radford et al., 2021), predicts the correspondence between modalities. It treats each modality as a state node and the correspondences be- tween modalities (including cross-modality similarity re- lationships) (Jin et al., 2024) as edges. Here, the action node is at the edge level. (2) Recommendation. Recom- mendations (Ni et al., 2023; Isinkaye et al., 2015; Ko et al., 2022) are based on the historical interactions and features of users and items to predict future interactions, as shown in Figure 2(b). Specifically, it models the user nodes and item nodes as state nodes. Moreover, the action node is edge- level. (3) Traditional graph prediction. Traditional graph prediction (Kipf & Welling, 2016; Veličković et al., 2017b; Hamilton et al., 2017b) primarily focuses on three types of tasks: node-level, edge-level, and graph-level. We follow previous work's 's settings of nodes and edges and define the action nodes of three levels.
2023b). (5) Retrieval-augmented generation. The purpose of Retrieval-Augmenttee Generation (RAG) is to enhance the generation capabilities of Large Language Models (LLMs) by retrieving information from external knowledge (Lewis et al., 2020; Gao et al., 2023; Zhao et al., 2024). Recent studies such as GraphRAG (Edge et al., 2024; Peng et al., 2024) have shown that modeling the relationships between data chunks in external knowledge tive capabilities of RAG. As illustrated in Figure 2(e), we model data chunks as nodes and the similarity of embed- dings between chunks as edges. As introduced in Section 2.2, we design an unintended action node for RAG tasks.
enhance the genera-
World optimization. (6) Planning and optimization. World optimization involves generating the next best de- cision based on sequence of historical decisions (Chen et al., 2021; Zheng et al., 2022; Siebenborn et al., 2022). Many studies have shown that modeling the relationships between historical decisions using graphs can enhance the decision- making effecttiveness of world optimization (Jiang et al., 2018; Munikoti et al., 2023). Following them, we model decision states nodes. The edges between these state nodes are often modeled based on their relationships, such as distance relationships (Prates et al., 2019) and the similarity of embeddings (Munikoti et al., 2023; Jiang et al., 2). We model the action node as the graph level.
World generation. (4) Multi-agent collaboration. As shown in Figure 2(d), the purpose (Zhuge al., 2024; Liu et al., 2023b; Wu al., 2024b) of this task is to gener- ate task-triented outputs based on the interaction between agents and external knowledge, as well as communica- tion among agents. Specifically, its state nodes consist of agent nodes with different profiles, along with multi- modal nodes in external knowledge. Its edges include agent-agent and agent-knowledge relationships. The ac- tion node is graph level and the target nodes primarily in- clude various agent nodes (Zhuge et al., 2024; Liu et al.,
3. Token-based GFM
3.1. Multi-modality token
One of the easiest ways unify multi-modalities is to trans- fer them into text. Specifically, as shown in Figure 3, for im- age nodes va, we utilize pretrained image-to-text LLaVA model (Liu et al., 2024a) L to transform them into text nodes ta = I(v For table nodes U², we employ a table-prompt model T to transform them into text nodes yhh = T(b*) given the column names and feature values. Specifically, each value paired with the corresponding column in the
3

---

Graph World Model
(a) Mutti-modal generation Multi-modaal generation and matching
(b) Recommendation
(e) Traditional graph prediction
Image
User Node
Bem Node
Action Node
Edge
Node
Action Node
Node
Image
Table
Node fevel
8
Graph--avel
Sub-graph
Action Node
Action Node
Action Node
Node-lovel
+
Edge
Edge-level
Edga-leval
Text
Table,
8
Edge
Edgellevel
Multi-modall matching
Interaction prediction
(c) Multi-agent collaboration
(e) Retrievall-ugmeetee generation
(i) Planning and optimization Decision Node
knowledge
Image Node
Chunk Node
Test Node
Read
Communication
Action Node,
Edge
Similarity
Retrieve
Top
Edge Relationship Decision sequence, O.O.O
Next deciaion
Action Node Graph-loval
Unintended Action Node
Generation
Table Node
Graph-lovel
Data Chunks
Agent Node
Figure Instantiations of GWM. (a) Multi-modal generation and matching contains two sub-tasks. For multi-modal generation, it matching, it models nodes for each modality. It includes edges that represent inter-modal correspondences and cross cross-modal similarities. For these two types of subtasks, there are node-level and edge-level action nodes, respectively, (b) The state nodes of recommendation include user nodes and item nodes. Morcover, its edges are primarily derived from user-item interactions. We model edge-level action nodes to perform interaction prediction. (c) In traditional graph prediction, we follow existing work construct task nodes and edges, and model three levels of action nodes according different types of tasks. (d) The state nodes of multi-agent collaboration include agent nodes and multi-modal nodes from external knowledge. Its edges primarily consist of communications between agents and interactions hetween agents and externa knowledge. We set up a graph-level action node to generate content based on the interactions of agents. (c) Retrieval-auge augmented generation treats each data chunk as a state node and builds edges through the embedding similarity between chunks, For this task, we have established unintended action nodes. (f) For planning and optimization, we model each decision a state node and construct edges based on their relationships. We have established graph-level action nodes to generate the next decision.
models modal clusters nodes, whereas for
format of "[column }} is (value 1). {column 2} is {value 2). ..." Finally, we used prompt template Pa (specified Table 23 in Appendix C) to unify the three modal nodes into a single text node Ue- = Pu (u veb 22²).
3.3. Instruction tuning
Based on token-level message passing, we can obtain node text representations h, for each node. Combining the dis- cussion in Section 2.2, we identify the target nodes u, and their node text representations hor, along with the action node a and the state node. Further, we describe the action node using text and utilize a task-oriented prompt template Psu (specified in Appendix C) to combine the information from the target nodes and the action node, as shown in the right part of Figure 3. In response to the different modalities in next states, we designed two types of decoders. We first designed stable diffusion (SD) to generate images.
3.2. Token-level message passing
In contrast to traditional graph message passing (Kipf & Welling, 2016; Hamilton et al., 2017b; Veličković et al., 2017b), we employ token-level message passing here, which aggregates the text information of neighboring nodes. Specifically, as shown in the middle part of Figure 3, for the each unified text node Ve, the node embeddings update of the (-th layer is represented as:
Instruction tuning of SD. SD operates by performing dif- fusion in a compressed latent space rather than directly on pixels. Initially, the system maps input image to lower-dimensional latent code z Enc(z; xx) through an en- coder network. The generated latent representation z' is sub- sequently transformed back into image space via a decoder network, producing the final output x' = Dec(z'). This latent representation z' is generated by the diffusion model using textual guidance from a prompt er P.a (her, a). The fundamental optimization objective for training SD can
h = = (CONCATT(H)-))
(())))))) (1)
where (4) is the node text presentation after I iterations,
( hy (0) has been initialized as h, (0) = Ue. In addition, (v)
denotes the direct neighbors of node u and f.( denotes
prompting strategy functions (specified in Table 24 of Ap- pendix C) to unify nodes information of different hops.
4

---

Graph World Model
Token-based GWM
Legend
Multi-modallt as token (Current Stato)
Token-leval message passing
Image
[e], t 04,
Token
State node
image-to-text Model
Node
Node
Prompt (Target nodes)
Instruction tuning Next state
Stable Diffusion
Action node
Prompling
Multi-hop
aggregation
Edge
Edge
Target node
Teble
Tablo-Prompt Model
Target nodes
Action node
Next state
Next state LLM
Embedding- based GWM
Multi-modaiily as embedding (Currett State)
Embedding-levee message passing
Image
Image Encoder
Node
(g 62i 6t]
Node
Diyt
Target nodes
embedding
Multii-hpp Projector
Projector tuning
Stable Diffusion
Next state
Apcregated node embedding Target node embedding
Text
Multi-hop aggregation
Concat
Edge
Edgg
State state edge Message pessing
Table
Prompt
Text Encoder
Target
Action node
Token
LLM
nodes
Figure 3. Framework of GWM. For both token-based and embedding-based GFM, we initially unify the multi-modal current into graph nodes, conduct message passing. and then combine actions to predict the next state across different modalities through respective decoders. The key distinctions are: 1) Token- based GWM integrates multi- modalities into text, whereas embedding -bsed GWM uses modality-speciffc encoders to process them into embeddings: 2) Token-based GWM utilizes text-based methods for message passing, while embedding- GWM operates at the embedding level; 3) Token-based GWM converts information into prompt form the decoder, while embedding based GWM employs multi-hop projector to manage embedding-level information.
be expressed mathematically as:
4.1. Multi-modality as embedding
C = Bee.Iiadaaeenncc--ooo... [[le co(2e, h(cr))
(2)
The embedding-based GFM unifics the multi-modal nodes in the embedding space. Firstly, as shown in Figure 3, for each modality of the node, we would assign a specific encoder. For the text modality of, utilize BERT model as the encoder Eg to obtain its embedding ét. As for the table node v², we first transform it into text description as discussed in Section 3.1 and then utilize a BERT model as the encoder Es to obtain its embedding Gg- Finally, we deploy a CLIP E. model to encode the image node v into image embedding eu. We finally obtain the node embedding e, CONCAT(Ca, (a, cb) by concatenating the embeddings of all modalities involved with this node. Specifically, if a node misses some modalities, we use zero vectors for them.
During each iterative step 4, a specialized denoising net- work c0(+) estimates noise patterns by jointly processing three inputs: the current latent state Zt, temporal posi- tion indicator f, and encoded text features h(CT). The text features h(cr)= E R dxxeer are extracted using CLIP's text encoder (Radford et al., 2021) h(CT) = CLIP(cT), where ler represents the prompt length and d denotes the feature dimensionality.
Instruction tuning of LLM. We then design LLM to gen- erate texts for both text and table modalities. We followed the standard instruction tuning practice (Zhang et al., 2023b; Peng et al., 2023), which encourages L.LMs to adhere to user requests when returning the outputs. For the input in- struction Pea (bur: a), we supply a response representing the next predicted states, consisting of tokens and denoted as y = (V1....yeh We train the LLM to yield fser via
4.2. Embedding message passing
In this section, we first model the relationships between nodes on the graph through multi-hop aggregation, then ag- gregate the information from different modalities within the nodes through cross-modal fusion into unified embeddings to pass to the subsequent decoders.
Lser =
Σ
Elog LoggPser (VilP (bur, a), 91,.. ... Vt (3)
Multi-hop aggregation. We designed a simplified GCN (Wu et al., 2019; He et al., 2020a) to implement multi-hop aggregation, which directly accomplishes parameter-free free feature aggregation at the node feature level. Specifically, for the adjacency matrix A between nodes, we first nor- malize to obtain the matrix A = D- AD where D represents the degree matrix of A. Then, for the node vec- tor X. composed of all node embeddings eg, we use the obtained normalized adjacency matrix Á to perform I-hop
4. Embedding-bbaed GFM
Although token-based GFM can relatively simply construct a multi-modas world, it is still limited by the high token cost and restricted multi-hop field of view. Inspired by stable diffusion, which introduces modeling in latent space rather than directly on pixels to enhance model performance and efficiency, we have introduced embedding-based GFM as shown in the bottom half of Figure 3.
5

---

Graph World Model
graph aggregation: x = A' * Xx. where x is the I-hop graph embedding, We retain the embeddings of the first L hops [Xe, x(") X!!) to the subsequent modules.
investigate the node-level multi-modal generation task, where the goal is to predict missing images based tex- tual captions. We use data from Goodreads (Jin et al., 2024) and the Multi-Modaa-Pappe dataset (detailed in Ap- pendix A.I). The generated images are evaluated using CLIP Score (Radford et al., 2021) and DINOv2 (Oquab et al., 2023). We compare our approach against several basclines, including Stable Diffusson 1.5 (SD-1.5) (Rom- bach et al., 2022), its fine-tuned variant (SD-115 FT), the image-to-image model ControlNet (Zhang et al., 2023a), and the SOTA INSTRUCTG21 model (Jin et al., 2024), Meanwhile, our edge-level multi-modal matching task evaluates the correspondence between different modal- ities, using Contrastive MLP (Liu et al., 2022), CLIP (Radford et al., 2021), and fine-tuned CLIP on metries such as Accuracy, Recall, and F1. Please note that since the multi matching task of Multi-Modal Paper also includes matching between text and tables, CLIP cannot be applied to this subtask.
Cross-modal fusion. We further apply a parameterized projector fe to transform the multi-modalithe in the node to unified embeddings: x!" = fc(X!"), where x!" is the unified node embedding. Specifically, we utilize a simple MLP as projector Se and output the L. hops embeddings x(!)
XG = [Xx; X! ,..., x to the decoders. Note that GWM-E can be extended to heterogeneous graphs by per- forming separate multi-hop aggregations for each edge type, followed by flattening the resulting node embeddings into sequence format suitable for input into the LLM decoder.
4.3. Projector tuning
As discussed in Section 3.3, in this section we discuss the tuning of projectors for two different modalities separately.
Projector tuning of SD. We first incorporate graph con- ditioning tokens ha (cG))) (cG) = XG into the SD models, func- tioning concurrently with the pre-existing text conditions (or):: M(cr, cG) = [hr(cr), (caa) E Rdxx (lopptttee Hee), where lea is the length of the graph condition. The training objective then becomes:
(2) Recommendation (Rec): As Table 11 of Appendix A.2 illustrates, utilize three benchmark datasets of varying scales-Baby, Sports, and Clothing- frroin Ama- zon's real-world product collections (McAuley et al., 2015). These datasets are commonly used in existing multi-modal graph recommendation systems (Wei et al., 90; 2020a). For these edge-level tasks, we benchmark our GWM model against recent state-of totte--o--he-rrr recom- mendation approaches, including FREEDOM (Zhou & Shen, 2023), as well representative graph baseh mod- els such as LightGCN (He et al., 2020a), MMGCN (Wei et al., 2019b), and GRCN (Wei et al., 2020b). We use Recall and F1 Score as the primary evaluation metrics. (3) Traditional graph prediction (Graph): We utilize Cora (Chen et al., 2024b), PubMed (Chen et al., 2024b), and HIV (Wu et al., 2018) datasets. For the Cora and PubMed datasets, we perform node-level and edge-level tasks, while for the HIV dataset, we undertake graph- level tasks. We compare GWM against two traditional graph baselines, GCN (Kipf & Welling, 2016) and GAT (Veličković et al., 2017b), as well as two GFM baselines, LLAGA (Chen et al., 2024a) and OFA (Liu et al., 2023a). We adopt accuracy as the metric. Details can be seen in Appendix A.3.
C = E -ee-Buttt.eerrcc..oo10.... [lle- co(2e, L, Mes.,MeTCCo c
(4)
Projector tuning of LLM. We describe the action node a using text as in Section 3.3. We further introduce graph tokens X into LLM. The training objective of LLM is to maximize the probability of generating the correct next states. Combining the discussion in Section 3.3, we train the LLM to yield fSFT via
Csrt= -
Xx, an, Ui,... 1t-1))
(5)
We use training approach similar to prefix tuning (Li & Liang, 2021). where we fix the LLM's parameters and only fine-tune the projector fo 's parameters.
5. Experiments
We employ one unified GWM model across multiple tasks, comparing its performance against domain-specific methods. Initially, we introduce the tasks within the GWM framework.
World generation: It contains two sub-tasks. (1) Multi- agent collaboration (Multi-agent): We utilize multi- modal agent benchmark called AgentClinic (Schmidgall et al., 2024) (in Appendix A.4) to evaluate LLMs within simulated clinical environments. This environment is structured as graph, with nodes representing different profile- agents such as patients, measurements, and moderators, and containing various modalities of external knowledge including medical images and patient records. The edges represent interactions between agents and their
Task deseription. The details of the tasks are summarized across three aspects in Table 9 of the Appendix, with further information on tasks and datasets available in Appendix A, and specific action node prompts in Appendix C.
World prediction: It contains three subtasks. (1) Multi- modal generation and matching (Multi-modal): We
6

---

Graph World Model
Table 2. Multi-modal generation results on Goodreads and Multi-Modal-Paper This task is predict the missing modality hased on the given modality. Compared to specific baselines in image generation, GWM achieved the best results.
Table 4. Recommendation on Baby, Sports, and Clothing. Com- pared with three classical graph baselines, GWM achieved state- th-the-art results on most metrics.
Goodreads
Moatii-Mdal-PPee Paape
Baby
Sports
Clothing
Model
Recall
Fl Score
Recall
Fl Score
Recall
Score
Model
CLIPP
DINOv2
CLIP
DINOv2
FREEDOM
60.35
66.16
63.47
70,53
70.20
78.40
SD-1.5
42.16
14.84
52.62
23.64
SD-1.5 FT
45.81
18.97
58.49
24.13
LightGCN
51.11
38.22
85.36
91.32
69.08
77.21
ControlNct
42.20
19.77
52.89
24.77
MMGCN
57.34
61.31
61.69
68.08
64.09
71.26
INSTRUCIG21
50.37
25.54
56.37
18.80
GRCN
24.35
82.47
57.31
61.23
57.60
61.74
GWM-T GWM-E 45,23
47.46
20.91
59.92
23.10
GWM-T
70.84
75.08
84.29
88.60
71.73
74.26
20.87
59.84
26.03
GWM-E
76.72
84.74
88.78
90.32
75.27
84.06
Table Multi-modal matching results on Goodreads and Multi-ModallPappee It aims to predict the correspondence be- tween different modal-ities. For Goodreads, this task is to predict text-image correspondences. As for Multi-Modal- Papeer, it aims to predict text-image, text-table, and table-image correspondences. Note that CLIP and CLIP FT cannot be applied Multi-Modal- Paper since it includes matching tasks beyond text-image.
Table S. Traditional graph prediction results on Cora, Pub.Med and HIV. It covers representative tasks at the node-level, edge- level, and graph-level. Compared classic graph baselines and GFM methods, our GWM can match their performance with one unified model.
Model
Cora
PubMed
HIV
Task Type
Node
Link
Node
Link
Graph
Goodzeads
Multi Muiti-Modall -Paper
GCN
78.86
90.40
74.49
91.10
86.72
Model
Accuracy
Recall Fl Score
Accuracy
Recall
Fl Score
GAT
82.76
93.70
75.24
91.20
87.84
Contrustive MLP
54,70
54.67
54.79
51.77
51.55
50.31
LLAGA
89.22
89.18
95.03
89.18
85.42
CLIP CLIP FT
83.80
83.80
83.84
OFA
73.21
93.12
77.80
96.39
92.04
92.60
92.58
92.61
GWM-T
81.92
88.24
92.91
91.88
92.20
GWM.T GWM-E
84.22
85.66
85.29
88.26
90.35
90.11
GWM-E
83.03
94.31
84.22
94.01
93.86
88.82
89.73
89.06
96.23
97.21
97.13
et al., 2024) attempt optimization tasks by imitating the trajectory of expert strategies. Here, we utilize the expert strategy dataset from the text-based embodied task ALF- World (Shridhar et al., 2020; Yang et al., 2024). We model each decision state as graph nodes and derive the edges between nodes based on the similarity of the state images associated with the decisions. We compare GWM with three LLM baselines-COS (Wei et al., 2022). TOT (Yao et al., 2024), and T5 (Raffel et al., 2020) fine-tuned on our dataset (T5 FT) -using BERT-Score (Zhang et al., 2019) (Precision, Recall, and F1 Score) metrics. Details can be seen in Appendix A.6.
engagement with knowledge resources. Given the objec- tive of integrating information from all agents to answer medical questions, we define this as a graph-level task. We compare our approach with three LLM-based base- lines: CoT (Wei et al., 2022), ToT (Yao et al., 2024), and Few-Shot (Madotto et al., 2021), as well as two ad- ditional baselines fine-tuned on the AgentClinic dataset. FT refers to a LLaMA-3-SB model fine-tuned directly on the task. Longformer (Beltagy et al., 2020) is a strong baseline for dong-document understanding. We use Accu- racy, Recall, and F1 Score as evaluation metrics to assess the correctness of the generated responses. (2) Retrieval- augmented generation (RAG): We utilize LongBench v2 (Bai et al., 2024), a benchmark designed for challeng- ing long-context question-ansve answering (in Appendix A.5). Following previous work like GraphRAG (Edge et al., 2024), we divide long context into chunks as nodes of the graph, and the edges between nodes are the simi- larity of their BERT embeddings. We conduct compar- isons with two et al., 2009) and Dragon (Lin et al., 2023)-and three long-context LLMs (128k), including Mistral Large 2, Command R+, and GPT-4o mini. Accuracy serves as our evaluation metric.
Implementation details. We train and test single GWM all tasks, comparing it with domain-specific baselines for each task. Specifically, for the LLM module, we uniformly use Llama-3-8B, and for stable diffusion, we use SD-v1- 5. For the image-to-text model used in GWM-T, we use LLa
The image encoder and text decoder used in GWM-E are CLIP and BERT models, respectively. In addition, our multi-hop projector uses an n-hop MLP to aggregate features from different hops, where n-hop refers to the number of neighborhood hops of the graph nodes used. To ensure the training efficiency of the models, we set the maximum token length for all models at 2k. We use Adam optimizer (Diederik, 2014) for model training and gradually decay the learning rate with I Lambdal.R scheduler.
baselines BM25 (Robertson
World optimization (Optimization): Many existing works (Ho& & Ermon, 2016; Hussein et al., 2017; Yang
7

---

Graph World Model
All the experiments are conducted on NVIDIA A6000 GPUs. Please refer to Appendix B for other implementation details.
Table 8, Plamning and optimization results on ALFWorld. is to evaluate how well the methods can imitate the trajectory of expert strategies to effectively assist in solving optimization problems. Compared to classic LLM baselines and text generation baselines, GWM-E has achieved the best results.
Table 6. Multi-agent collaboration results on AgentClinic. This task is answer medical questions by leveraging interactions between agents and external knowledge sources. Compared to classic LLM baselines, GWM-T achieves achieecs state-of-the-art results.
Model
Precision
Recall Fl Score
Normal COT
89.62
88.86
89.21
Model
Accuracy
Recall
F1 Score
86.87
87.74
87.27
COT
45.00
33,42
32.25
T5 FT
92.06
91.52
91.82
TOT
35.00
33.71
29.71
GWM-T GWM-E
88.10
87.05
87.42
Fow-shots
40.00
40.63
29.37
93.27
92.36
92.13
Longformer 25.00
20.20
14.00
FT 45.00
45,40
44.00
GWM-T 50.00
46.42
48.20
to domain-specinee baselines in other tasks. This demon- strates GWM's ability to generalize and its applicability across a broad range of tasks. (2) GWM demonstrates promising capabilities in some highly challenging tasks, such as long-context RAG (shown in Table 7). GWM with context length of 2k can outperform LLM models with a context length of 128k in RAG tasks, showcasing GWM's potential in understanding and reasoning with long texts. (3) The design of the latent embedding enables GWM-E to outperform GWM-T in five out of seven tasks with approxi- mately 5- 5-10 times fewer token costs. This demonstrates the efficiency and effectiveness of umbedding-based message passing.
GWM-E
45.00
39.57
35,56
5.1. A single GWM matches the performance of domain-specific methods across multiple tasks
We train a unified GWM on all tasks and test it across all tasks without further fine-tuning, compared with domain- specific baselines under each task. Specifically, for world prediction, we first report the multi-modal generation and matching results in Table 2 and Table 3. Subsequently, we report the recommendation results in Table 4, and the traditional graph prediction results in Table 5. As for world generation, we report the multi-agen collaboration results in Table 6 and the strrival-auumente generation results in Table 7. For world optimization, report the results in Table 8.
5.2. GWM benefits from multi-hop graphs
To explore whether multi-hop graphs can enhance the per- formance of GWM, we compared the effectiveness of four different hop settings with a no-graph baseline using GWM- E on six tasks, as illustrated in Figure 4. Specifically, we measured the average performance across five settings for all tasks. For Multi-modal tasks, we use DINOv2 and F1 Score to calculate average performance, while for Rec, Agent, and Optimization tasks, we exclusively use the Fl Score. Accuracy metrics were employed for the remaining tasks. Graphs have consistently enhanced GWM-E performance across all tasks, showing a minimum relative gain of 20% on graph-related tasks. However, an increased hop number does not always lead better performance since it can cause over-smoothing and introduce redundant information.
We can observe that: (1) A single GWM achieves SOTA re- sults in multi- generation Molti-MModal-Papert multi- agent collaboration, retrieval-augie -aggeened generation, as well as planning and optimization, and also performs comparably
Table Retrieval-augmented generation on LongBench v2. It is challenging long-context guestion-answering task that is cat- egorized into easy and hard levels. Compared to classic RAG basclines and LLM models with extended contexts, GWM with limited context length achieved the best results. The result also demonstrates the superiority of GWM-E over GWM-T in tasks involving long contexts.
5.3. GWM boosts zero-shot/few-shot performance
Model
Overall
Easy
Hard
BM25 (2k)
27.45
41.18
20.59
To validate the zero-shot/few-shh capabilities of GWM, we conduct experiments with GWM-E and GWM-T on the Agent and RAG tasks, as shown in Figure 5 ("-T" and "EE" respectively represent the experimental results of GWM-T and GWM-E). Here, Single Data refers to training GWM solely on the Agent or RAG task. Zero-shot refers to train- ing GWM on tasks other than Agent or RAG and testing it on Agent or RAG. Fine-tuned GWM refers to training
Dragon (2k)
23.53
35.29
17.65
Mistral Large 2 (128k)
26.31
29.42
24.45
Command R+ (128k)
27.43
30.19
26.32
GPT-4o mini (128k)
29.01
30.23
28.03
GWM-T (2k)
29.40
35.71
21.74
GWM-F (2k)
33.32
39.16
29.52
8

---

Graph World Model
0.5
Single Data
Performance Across Hops
No Graph
Zero-shot
RAG
1-Hop
Fine-tuned GWM
2-Hop
Performance
3-Hop
4-Hop
0.3
Optimization
16.
Multi-modal
0.1
0.0
Agent-T
RAG-T
Agent-E
RAG-E
Figure 5. GWM boosts zero-shot/few-shot performance on multi-agent collaboration (Agent) and retrieeal-augmented generation (RAG) tasks. Note that "-T" and "-E" respectively represent the experimental results of GWM-T and GWMMEE It can be observed that GWMM can quickly adapt to new tasks with a small amount of domain-specific training data. Moreover, GWM's strong generalization ability can boost the performance of Agent and RAG.
Recommendation
Multi-agent
Graph
Figure 4. Multi-hop graphs enhance GWM's performance representative tasks in six domains. We can observe that the introduction of graphs has benefited GWM-E across all tasks compared to no graph. Morcover, excessive hops can lead to over-smoothing, thereby decreasing performance.
states based on given actions. Existing WMs (Wu et al., 2024a; Bruce et al., 2024) primarily focus on how to utilize unstructured data to predict state transitions, thereby enhanc- ing the effectiveness of sequence generation tasks. Genie (Bruce al., 2024) trained foundation world model using massive amount of unlabelled, serialized internet videos, which has provided benefits for the planning outcomes of downstream tasks. Additionally, some WMs (Zhang et al., 2021; Zhu et al., 2022) have attempted to integrate struc- tured data with GWM. & (Zhang et al., 2021) uses graphs to model each step of the agent's pociision-maknn process and their connections, thus enhancing scalable planning in reinforcement learning. However, they are still largely con- fined planning and optimization scenarios, which limits their potential for task generalization as WMs. Thus we develop GWM that integrates the capabilities of graphs with WM to generalize across diverse tasks.
GWM on tasks other than Agent RAG, followed by few- shot fine-tuning with 10% of the data from Agent or RAG. We can observe that GWM adapts effectively to new tasks using only small amount of domain-specific training data. Moreover, we observe that the zero-shot results of GWM on the RAG task are even better than those from Single Data, indicating that GWM's strong generalization ability greatly benefits tasks with limited training data like Agent RAG.
6. Additoona Related Work
Graph for Modelling Relations. Graphs are highly effec- tive modeling complex relationships (Fey et al., 2023; Cao et al., 2023; Gao & Xu, 2020; Chen et al., 2022; Wu al., 2022; Yang al., 2021), extracting nodes and edges to model relational data with embeddings. Graph Neural Hamilton al., 20;7a; Veličković et al., 70; Schlichtkrull et al., 2017) have emerged as dominant approach, particularly in recom- mendation systems (Min et al., 2022) and social networks (Wu et al., 2020). further address the vast array tasks and data, scholars have proposed the GFM (Chen et al., 2024a; Liu al., 2023a) to explore GNNs' zero-shot few-shot capabilities (Fey et al., 2023; Cao et al., 2023; Gao & Xu, 2020; Chen et al., 2022) to tackle challenges such as the cold start problem in recommendations.
7. Conclusion
We propose GWM, a unified framework that uses graph world state to tackle diverse prediction, generation, and planning tasks. Across six benchmarks, GWM matches domain-speciffe baselines while benefiting from multi-hop hop graph structures, showing strong generality and flexibility. It also improves zero-shot and few-shot performance, in- dicating strong cross-task generalization. GWM currently supports text, table, and image modalities. with plans to extend more. While the current implementation focuses homophilous graphs, aim to expand it to support both homophilous and non-homophilous structures for broader applicability. Its modular design also makes it a flexible base for future multi-modal graph reasoning tasks.
Networks (GNNs) (Kipf& & Welling, 2017;
World Model. The WM (Ha & Schmidhuber, 2018) to construct the world observations as states and predict future

---

Graph World Model
Impact Statement
Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steiger- wald, R., Apps, C.,, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
The WM serves as a unified framework for prediction, gen- eration, and decision-making across various applications. While traditional WMs are constrained and unstructured data, our proposed GWM enhances them by embedding-level message passing and aggregation to integrate structured and multi-modal data, bridging the gap between unstructured and structured processing, GWM demonstrates significant potential as a foundational graph- based model for real-world multi-modal tasks; however, its current scope limited by the number of supported modal- ities and the simplicity of its graph architecture, necessi- tating further advancements for broader applicability and enhanced relational modeling. Future applications should also prioritize ethical considerations, recognizing that ef- forts are needed to ensure that GWM's responses are reli- able, unbiased, and safe in real-world deployments, thereby preventing potential harm to users. In addition, the data utilized in this work are collected in compliance with appli- cable laws and licensing agreements. Their usage is also transparent and harmless.
Cao, K., You, J., and Leskovee, J. Relational multi-task learning: Modeling relations between data and tasks. arXiv preprini arXiv:2303.07660 2023.
Chen, L., Lu, K., Rajeswaran, A.. Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
Chen, R., Zhao, T., Jaiswal, A., Shah, N., and Wang, Z. Llaga: Large language and graph assistant. arXiv preprint 242.:2240,.08170, 2024a.
Chen, S., Hong. Z., Xie, G., Peng, Q., You, X., Ding, W. and Shao, L. Gndaa: Graph navigated dual attention network for zero-shot learning. IEEE transactions neural networks and learning systems, 35(4):4516-4522, 2022.
The security of Large Language Models (LLMS) has always been a concern. Unfortunately, current LLMs sometimes produce harmful and biased information unexpectedly. Our proposed method uses LLMs to generate simulated queries and summary responses, which are only used to construct graph of records and connect text chunks from long doc- uments. However, more work is needed in real-world ap- plications to ensure that LLMs' responses are reliable and harmless, so that they do not harm users.
Chen, Z., Mao, H., Li, H., Jin, W., Wen, H., Wei, X., Wang, S. Yin. D., Fan, W. Liu, H., et al. Exploring the po- tential of large language models (Ums) in learning on graphs. ACM SIGKDD Explorations Newsletter, 25(2): 42-61, 2024b.
Cui, H. and Gao, Y. universal world model learned from large scale and diverse videos. In NeurIPS 2023 Founda- tion Models for Decision Making Workshop.
References
Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., Cao, S., Xu, J., Hou, L., Dong, Y., et al. Long- bench v2: Towards deeper understanding and reason- ing on realistic long-context multitasks. arXiv preprint 2rXiv:2412.15204 2024.
Diederik, K. Adam: A method for stochastic optimization. (No Title), 2014.
Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., and Larson, From local to global: graph rag approach to query-focusse summarization. arXiv preprint arXfv:2404.161.0, 2024.
Beltagy, I. Peters, M. E., and Cohan, A. Long- former: The long-document transformer. arXiv preprint arXiv:2004.055 150, 2020.
Ektefaie, Y., Dasoulas, G., Noori, A., Farhat, M., and Zitnik, M. Multimodal learning with graphs. Nature Machine Intelligence, 5(4):340-350. 2023.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., derberttVosss. A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
Fey, M., Hu, W.. Huang, K., Lenssen, J. E.. Ranjan, R., Robinson, J., Ying, R., You, J., and Leskovec, J. Rela- tional deep learning: Graph representation learning on relational databases. arXiv preprint 1rX 2023.
J., Winter, C., Hesse, C., Chen, M., Sigler, E.. Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020. URL https:// arxiv. v.iiv... org/ / abs 20005. 65.
Gao, J. and Xu, C. Ci-gnn: Building a category-instance graph for zero-shot video classification. IEEE Transac- tions on Multimedia, 22(12):3088-311 2020.
Bruce, J., Dennis, M. D., Edwards, A., Parker- Holder, J.,
10

---

Graph World Model
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. Rerrieval-augmented generation for large language models: A survey. arXiv preprint 2023.
Jiang, J., Dun, C., Huang, T., and Lu, Z. Graph convolutional reinforcement learning. arXiv preprint 2018.
Jin, B., Pang. Z., Guo, B., Wang, Y.-X., You, J., and Han, J. Instructg2i: Synthesizing images from multimodal at- tributed graphs. arXiv preprint arXiv:2410.07155 2024.
Ha, D. and Schmidhuber, J. Recurrent world models facil- itate policy evolution. Advances in neural information processing systems, 31, 2018.
Jin, W. Barzilay, R., and Jaakkola, T. Junction tree varia- tional autoencoder for molecular graph generation. In- ternational Conference on Machine Learning (ICML), 2018.
Hamilton, W., Ying, Z. and Leskovec, J. Inductive repre- sentation learning on large graphs. Advances in neural information processing systems, 30, 2017a.
Hamilton, W., Ying, Z., and Leskovec, J. Inductive repre- sentation learning on large graphs. Advances neural information processing systems, 30, 2017b.
Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. arXiv preprint arXiv: /609666999007. 2016.
He, X., Deng, K., Wang. X., Li, Y., Zhang, Y., and Wang, M. Lightgnn: Simplifying and powering graph convolu- tion network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pp. 639-648, 2020a.
Kipf, N. and Welling, M. Semi-supervised classification with graph convolutional networks. In ICLR (Poster). OpenReview.net, 2017.
Ko, H., Lee, S., Park, Y., and Choi, A. A survey of recom- mendation systems: recommendation models, techniques, and application fields. Electronics, 11(1):141, 2022.
He, X., Deng, K., Wang. X., Li, Y., Zhang, Y., and Wang, M. Lightgnn: Simplifying and powering graph convolu- tion network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 639-648, 2020b.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t,, Rocktäschel, T., et al. Retrieval-augmentee generation for knowledge- intensive nlp tasks. Advances Neural Information Pro- cessing Systems, 33:9459-9474, 2020.
Heumos, L., Schaar, A. C., Lance, C., Litinetskaya, A., Drost, F., Zappia, L., Lücken, M. D., Strobl, D. C., Henao, J., Curion, F., et al. Best practices for single-cell analysis across modalities. Nature Reviews Genetics, 24(8):550- 572, 2023.
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprini 1rXiv:2101.00199 2021.
Lin, S.-C., Asai, A., Li, M., Oguz, B., Lin, J., Mchdad, Y., Yih, W.-t, and Chen, X. How to train your dragon: Di- verse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07455 2023.
Ho, J. and Ermon, S. Generative adversarial imitation learn- ing. Advances in neural information processing systems, 29, 2016.
Liu, H., Feng, J., Kong, L., Liang, N., Tao, D., Chen, Y., and Zhang, M. One for all: Towards training one graph model for all classification tasks. arXiv preprint arXiv:23 231.0.1149, 2023a.
Hu, E. J., Shen, Y., Wallis, P., Allen -hu, Z., Li, Y., Wang, S., Wang, I.., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 10600685, 2021.
Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovec, J. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 15:22118-22133. 2020.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing. Advances neural information processing systems, 36, 2024a.
Liu, H., Yan, W., Zaharia, M., and Abbeel, World model on million-length video and language with blockwise ringattention. CoRR, 2024b.
Hussein, A., Gaber, M. M., Elyan, E., and Jayne, C. Im- itation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
Liu, S., Ounis, I., and Macdonald, C. An mip-based algo- rithm for efficient contrastive graph recommendations. Proceedings of the 45th international ACM SIGIR conference on research and development in information 2022.
Isinkaye, F. O., Folajimi, Y. O., and Ojokoh, B. A. Recom- mendation systems: Principles, methods and evaluation. Egyptian informatics journal, 16(3):261-273. 2015.
retrieval, pp.

---

Graph World Model
Liu, Z., Zhang, Y., Li, P., Liu, Y., and Yang, D. Dy- namic Imm-agent network: An Ilm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.021100 2023b.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Madotto, A., Lin, Z., Winata, G. L., and Fung, P. Few-shot bot: Prompt-based based learning for dialogue systems. arXiv preprint ww..vv:210008//11 2021.
Robertson, S., Zaragoza, H., et al. The probabilistic rele- vance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-389, 2009.
McAuley, J., Targett, C., Shi, Q. and Van Den Hengel, A. Image-basee based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43-52, 2015.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Min, E., Rong, Y., Xu, T., Bian, Y., Zhao, P., Huang, J., Luo, D., Lin, K., and Ananiadou, S. Masked transformer for -aighhourhood-awarre click-through rate prediction. CoRR, abs/2201. 13311, 2022.
Schlichtkrull, M., Kipf, T., Bloem, P., Van Den Berg, R., Titov, I., and Welling, M. Modeling relational data with graph convolutional networks. arxiv. arXiv preprint 2017.
arXiv:
Munikoti, S., Agarwal, D., Das, L., Halappanavar, M., and Natarajan, B. Challenges and opportunities deep rein- forcement learning with graph neural networks: A com- prehensive review of algorithms and applications. IEEE Iransactions neural networks and learning systems, 2023.
Schmidgall, S., Ziaei, R., Harris, C., Reis, E., Jopling, J., and Moor, M. Agentclinic: a multimodal agent bench- mark to evaluate ai in simulated clinical environments. arXiv preprint arXiv:2405.000 405607960, 2024.
Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020.
Ni, Y., Cheng, Y., Liu, X., Fu, J., Li, Y., He, X., Zhang, Y., and Yuan, F. A content-driven micro-viiee recommen- dation dataset at scale. arXiv preprint arXiv:2309.1533 15379, 2023.
Siebenborn, M., Belousov, B., Huang, J., and Peters, How crucial is transformer in decision transformer? arXiv preprint arXiv:221 1.6555, 2022.
Oquab, M.. Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Stuart, T. and Satija, R. Integrative single-cell analysis.
Nature reviews genetics, 20(5):257-272, 2019.
Stuart, T., Butler, A., Hoffman, P., Hafemeister, C., Papalexi, E., Mauck, W. M., Hao, Y., Stoeckius, M., Smibert, P., and Satija, R. Comprehensive integration of single-cell data. cell, (7()):1888-192. 2019.
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruc- tion tuning with gpt-4, arXiv preprint wwXiv:230.003277, 2023.
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. Graph attention networks. arXiv preprint arXiv: 1710. 10903, 2017a
Peng, B., Zhu, Y., Liu, Y., Bo, X., Shi, H. Hong. C., Zhang, Y., and Tang, S. Graph etrieval-augmenter generation: A survey. arXiv preprint arXiv:2408 089211 2024.
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. Graph attention networks. arXiv preprint 1rXiv:1710.1000 2017b.
Prates, M., Avelar, P. H., Lemos, H., Lamb, L. C., and Vardi,
M. Y. Learning to solve np-complete problems: A graph neural network for decision tsp. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4731-4738, 2019. 2019.
Wan, M., Misra, R., Nakashole, N., and McAuley, J. Fine- grained spoiler detection from large-scale review corpora. In Korhonen, A., Traum, D., and Mårquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Asxo- ciation for Computational Linguistics, pp. 2605-2610, Florence, Italy, July 2019. Association for Computa- tional Linguistics. doi: 10. 10.18653/w1/P19--224 1248. URL hwwphhhoo.oo..oooo/11111140 1t9
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
12

---

Graph World Model
Wang, Z., Wang, Z., Srinivasan, B., Ioannidis, V. N., Rang- wala, H., and ANUBHAI, R. Biobridge: Bridging biomedical foundation models via knowledge graphs. In The Twelfth International Conference on Learning Repre- sentations.
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Ge- niesse, C., Pappu, A. S., Leswing, K., and Pande, V. Moleculenet: a benchmark for molecular machine learn- ing. Chemical science, 9(2):113-530 2018.
Yang, L., Liu, Z., Dou, Y., Ma, J., and Yu, P. Con- sisrec: Enhancing gnn for social recommendation via consistent neighbor aggregation. In Proceedings of the 44th international ACM SIGIR conference on Research and development in information retrieval, pp. 2111-2144444 2021.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of soe---ff-hhouuu thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Wei, Y., Wang, X., Nie, L., He, X., Hong, R., and Chua, T.-S. Mmgen: Multi- moltal graph convolution network for personalized recommendation of micro-video. In Proceedings of the 27th ACM international conference on multimedia, pp. 1437-1445, 2019a.
Yang. Y., Zhou, T., Li, K., Tao, D., Li, L., Shen, L., He, X., Jiang. J., and Shi, Y. Embodied multi-modal agent trained by an Ilm from a parallel textworld. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26275-26285, 2024.
Wei, Y., Wang. X., Nie, L.., He, X., Hong, R., and Chua, T.-S. Mmgen: Multi-modal graph convolution network for personalized recommendation of micro-video. In Proceedings of the 27th ACM international conference on multimedia, pp. 1437-1445, 2019b.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Wei, Y., Wang. X., Nie. L., He, X., and Chua, T.-S. Graph- refined convolutional network for multimedia recommen- dation with implicit feedback. In Proceedings of the 28th ACM international conference on multimedia, pp- 3541-3549, 2020a.
Ying, R., He, R., Chen, K., Eksombatchai, P., Hamilton, W. L.., and Leskovec, J. Graph convolutional neural networks for web-scale recommender systems. ACM SIGKDD International Conference on Knowledge Dis- covery and Data Mining (KDD), 2018.
Wei, Y., Wang, X., Nie, L.,, He, X., and Chua, T.-S. Graph- refined convolutional network for multimedia recommen- dation with implicit feedback. In Proceedings of the 28th ACM international conference on multimedia, pp. 3541-3549, 2020b.
You, J., Liu, B., Ying, R.,, Pande, V., and Leskovec, Graph convolutional policy network for goal-directed molecu- lar graph generation. Advances in Neural Information Processing Systems (NeurIPS), 2018.
You, J., Du, T., and Leskovec, J. Roland: graph learning framework for dynamic graphs In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp. 2358 2358-2366, 2022.
Wu, F., Souza, A., Zhang. T., Fifty, C., Yu, T., and Wein- berger, K. Simplifying graph convolutional networks. In International conference on machine learning, pp. 6861- 6871. PMLR, 2019.
Zhang, L., Yang, G., and Stadie, B. C. World model as a graph: Learning latent landmarks for planning. In International conference on machine learning, pp. 12611- 12620. PMLR, 2021.
Wu, J., Yin, S., Feng, N., He, X., Li, D., Hao, J., and Long, M. ivideogpt: Interactive videogpts are scalable world models. arXiv preprint arXiv:2405.15225 2024a.
Wu, S., Sun, F., Zhang, W., Xie, X., and Cui, B. Graph neural networks in recommender systems: a survey. ACM Computing Surveys, 55(5):1-37, 2022.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1833--347,, 2023a.
Wu, Y., Lian, D., Xu, Y., Wu, L., and Chen, E. Graph convolutional networks with markov random field reason- ing for social spammer detection. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 1061, 2020.
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F,, al. Instruction tuning for large language models: A survey. arXiv preprint arXiv: 2308. 10792, 2023b.
Wu, Y., Fan, Y.. Min, S. Y., Prabhumoye, S., McAleer, S., Bisk, Y., Salakhutdinov, R., Li, Y., and Mitchell, T. Agen- tkit: Flow engincering with graphs, not coding. arXiv preprint arX4v:2404. /483, 2024b.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint 1rXiv:1904.00000 1904.0965, 2019.
13

---

Graph World Model
Zhao, P., Zhang, H., Yu, Q. Wang, Z., Geng, Y., Fu, F,, Yang, L.,, Zhang, W., and Cui, B. Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402. 240.::2402.19473, 2024.
A. More GWM Task
This section discusses the detailed processing procedure for each dataset collected in GWM. We summarize their general information in Table
Zheng, Q.. Zhang, A., and Grover, A. Online decision trans- former. In international conference on machine learning, pp. 27042-27059, PMLR, 2022.
A.I. Multi-modal generation and matching
Dataset descriptions. this study, we utilize two datasets for multi- colli-moddl generation and matching: the Goodreads dataset (Wan al., 2019) and our curated Multi-Modal- Paper dataset.
Zhou, X. Mmrec: Simplifying multimodal recommendation. In Proceedings of the 5th ACM International Conference on Mullimedia in Asia Workshops, pp. 1-2, 2023.
Zhou, X. and Shen, Z. A tale of two graphs: Freezing and denoising graph structures for multimodal recommen- dation. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 935-943, 2023.
The Goodreads dataset is a large-scale col- lection of book-related metadata, textual descriptions, and cover images, widely used in prior multi-modal research (Jin et al., 2024). The Goodreads dataset is structured as graph, where each book is represented node, and edges signify similar-book semantics.
(1) Goodreads:
Zhu, D., Li, L. E.,, and Elhoseiny, M. Value memory graph: A graph-structured world model for offline reinforcement learning. arXiv preprint arXiv:2206. 04384, 2022.
(2) Pape:: Multi-Modal-Paun dataset is curated dataset of academic papers, incorporating textual content, figures, tables, and metadata to facilitate research on scholarly document analysis. The raw LaTeX files of the papers were collected from ArXiv using the ArXiv API, in accordance with the papers' licenses, which primarily CCO CC-BY 4.0, permitting redistribution and sharing. Using carefully selected survey papers from various domains of Artificial Intelligence (AI), including Natural Language Processing (NLP). Computer Vision (CV), Bio- informatics, and Robotics, as seed papers, we employ a breadth- frestth-first search (BFS) algorithm to gather cited papers. Then, we traverse through the abstract syntax tree built from the LaTeX file to extract graph structured multi-modal data for each gathered paper.
Zhuge, M.. Wang. W., Kirsch, L., Faccio, F., Khizbullin, D., and Schmidhuber, J. Language agents as optimizable graphs. arXiv preprint arXiv:2402.1662. 2024.
For the multi-modal generation task, we sample figure- caption pairs for training and evaluation. GWM and base- line models are tasked with reconstructing the figures from the provided captions. For the multi-modal matching task, we sample cited-citing pairs using reference relationship, namely the macro command "(ref". The cited-citing pairs are usually figure-text or table-text pairs.
A detailed statistical overview of the Goodreads and Multi- Modal-Paper datasets presented in Table 10.
Baselines details. For multi-modal generation tasks, we em- ploy two text-to-image baseline models, SD-1.5 and SD-1.5 FT, along with an image-to toage-to-imann to-image baseline model, Control- Net.
SD-1.5: A pre-trained Stable Diffusion v1.5 model (Rom- bach et al., 2022) used for text-to-image generation with- out task-specific fine-tuning.
hupps//arxiv.orr/ We thank ArXiv for providing open access interoperability.
14

---

Graph World Model
Table Detailed summarization of all collected datasets GWM. We summarize dataset names, tasks, action level, multi modaiiy
nodes, and edges in the table.
Dataset
Task
Action Level
Multi-modality
Nodes
Edges
Goodreads
Modai-modal generationn/atchhng
Node level
Text/image
Text/image nodes
Similar- semantic
Multi-ModallPayg Paper- Multi-modal generation/matthing
Node level
Text/image/table
Text/imac/ttbble nodes User/item nodes
References
Baby
Recommendation
Link level
Text/table/image
User-item interactions
Sports
Recommendation
Link level
Text/talle/image
User/item nodes
User-item interactions
Clothing
Recommendation
Link level
Texthable/inage
User/item nodes
User-item interactions
Cora
Traditional graph prediction
Node/edge level
Text
Research paper nodes
Citation
PubMed
Traditional graph prediction
Node/edge level
Text
Research paper nodes
Citation
HIV
Traditional graph prediction
Graph level
Text
Atoms nodes
Atoms bonds
AgentClinic
Multi-agen collaboration
Graph level
Text/image
Agen//imageftext nodes
Agent- agent/nnag//eex
LongBench v2 ALFWorld
Retrieval-auge -uggemetd generation
Unintended action
Text
Chunk nodes
Chunk-chunk similarity
Planning and optimization
Graph level
Text/image
State nodes
State images similarity
Table 10. Data statistics for multi-modal generation and match- ing. In Calti-ModallPappeee there 58565 text-nodes, 7380 figure-nodes, and 6792 table-nodes.
Table II. Data statistics for recommendation. includes three datasets of different scales, with the sizes ranging from small large as follows: Bahy, Sports, and Clothing.
Dataset
#Node
#Edges
Dataset
#User
#Item
#Edges
Sparsity
Goodreads
93,475
637,210
Baby 19,445
7,050
160,792
99.883%
Multe-Modaa-Paper
72,737
51,840
Sports
35,598
18,357
296,337
99.955%
Clothing
39,387
23,033
278,677
99.969%
SD-1,5 Stable Diffusion v1.5 models, each fine-tuned separately on the training splits the Goodreads and Multi-Modal-Pappe Papeer datasets.
A.2. Recommendation
Dataset descriptions. In the recommendation task, we conduct extensive evaluations using three Amazon datasets extensively recognized in prior research (McAuley al., 2015), specifically: Baby, Sports, and Outdoors, well as Clothing Shoes, and Jewelry. For simplicity, these datasets are hereafter referred to as Baby, Sports, and Clothing, re- spectively. Utilizing the 5-core setting, we filter inactive users and items with less than five interactions. Each dataset encompasses both visual and textual modalities and we use the extracted visual and textual features from existing work (Zhou, 2023). Here the visual modality is the product image and the textual modality is the product description. The characteristics of these datasets are shown in Table 1.
ControlNet: An extension of Stable Diffusion that incor- porates structural guidance, such as edge maps or depth maps, enhance control over generated images (Zhang et al., 2023a).
We use three baselines for multi-modal matching: Con- trastive MIP, CLIP, and CLIP FT. Since the vertices of the edge in the Multi-Modal-PPe Papprr dataset can include tables, namely the text-table pair that are not suitable for the CLIP model, we exclusively use Contrastive MLP for this dataset.
Contrastive MLP (Liu al., 2022): A multi-layer per- ceptron trained to predict multi-modal matching by pro- cessing embeddings from different modalities. The text and table embeddings are encoded by BERT while the image embedding is encoded by CLIP. Embeddings from different modalities are padding to the same dimension.
Baselines details. We compare GWM with three represen- tative GNN baselines.
LightGCN: Employs a simplified graph convolutional network to learn user and item interaction graph (He et al., 2020b).
A pre-trained vision-language language model designed for image-text alignment, using contrastive learning to map corresponding image and text embeddings into a shared space (Radford al., 2021).
CLIP:
MMGCN: Learns user preferences across multiple modalities via meessae--passing on condality-specifii user- item graphs, improving recommendations in multimedia contexts (Wei al., 2019a).
CLIP FT: A fine-tuned version of CLIP, adapted to the specific dataset to enhance multi-modas matching perfor- mance.
GRCN: Refines interaction graphs using multimedia con- tent to identify and remove noisy edges, thereby sharpen- ing the recommendation process (Wei et al., 2020a).
15

---

Graph World Model
A.3. Traditional graph prediction
and test sets using 4:1:1 split ratio. To simulate real-world clinical procedures, we employ the simulated clinical en- vironments from AgentClinic to gather dialogues between the patient and doctor, along with physical examination results. This environment is modeled as a graph, where nodes represent various profile-based agents and edges cap- ture the interactions between agents and their engagement with knowledge resources. We utilize Meta-Llama-3-708t Instruct as backbone model for simulation. In the final diagnosis procedure, we apply both GWM and LLM base- lines for comparison, where the graph information is con- verted into textual format before being processed by LLM baselines.
Dataset descriptions. In the traditional graph prediction task, we evaluate GWM Cora, PubMed, and HIV datasets. (I) Cora (Chen et al., 2024b): Cora is citation network in the computer science domain, where nodes represent re- search papers and edges denote citation relationships. Each node includes the paper's 's title and abstract as text features, with labels indicating paper categories. Tasks on Cora in- clude category prediction (node level) and citation link iden- tification (link level). (2) PubMed (Chen et al., 2024b): PbbMed is a biomedical citation network, similar to Cora, with nodes representing papers and edges indicating cita- tion relationships. (3) HIV (Liu et al., 2023a): HIV is a molecular dataset constructed from MOLHIV dataset (Wu et al., 2018) () that contains over 40,000 compounds annotated for their ability to inhibit HIV replication. Molecular struc- tures and graph representations are generated from SMILES strings, with atoms (nodes) and bonds (edges) described using natural language.
Baselines details. We compare GWM with three classic LLM baselines adopting different reasoning strategies. We use
S--ntrtruct' the backbone model to align with GWM.
. CoT: Adopts Chain-of Chain-of-Thought (Wei et al., 2022) prompt- ing, which enhances reasoning by decomposing complex problems into intermediate steps, improving performance on multi-step reasoning tasks.
Baselines details. The settings for the baselines primar- ily follow LLAGA (Chen et al., 2024a) and OFA (Liu et al., 2023a). We convert all nodes and labels the Cora, PubMed, and HIV datasets into text. For all methods, we use BERT to obtain text embedding. We divide all datasets into training, validation, and test sets in 8:1:1 ratio.
ToT: Adopts Tree-of Thoughtt (Yao et al., 2024) prompt- ing, which explores multiple reasoning paths in tree-like structure, enabling iterative evaluation and refinement for more robust decision-making.
GCN (Kipf& & Welling, 2017): Applies spectral- based convolution operations to capture local graph structures and propagate information across nodes, serving as a fundamental baseline for graph-based learning.
Few-shots: Adopts Few-shot (Brown et al., 2020) prompt- ing, where the model is provided with a limited number of in-context examples to guide task-specific reasoning without requiring fine-tuning.
GAT (Veličković et al., 2017a): Enhances node represen- tation learning by incorporating attention mechanisms, allowing adaptive weighting of neighboring nodes to im- prove feature aggregation.
A.5. Retrieval-augmenned generation
The purpose of Retrieval-Augmented Generation (RAG) to enhance the generation capabilities of Large Language Models (LLMs) by retrieving information from external knowledge (Lewis et al., 2020; Gao et al., 2023; Zhao et al., 2024). We introduce its dataset and baselines as follows:
LLAGA (Chen et al., 2024a): Integrates LLM with graph structures to enhance reasoning and information retrieval in multi-modal and structured data scenarios.
OFA (Liu et al., 2023a): Unifies vision, language, and multi-modal learning tasks within a single frame- work, leveraging pre-trained knowledge to facilitate cross- modal understanding and adaptation.
Dataset descriptions. We employ LongBench v2 (Bai et al., 2024), a benchmark specifically designed to test long-context understanding and reasoning. This benchmark comprises 503 challenging multiple-choice questions, with contextual lengths ranging from 8,000 to 2 million words, spanning six major task categories: Single-Doc QA, Multi- Doc QA, Long In-context Learning, Long-dialogu History Understanding, Code Repository Understanding, and Long Structured Data Understanding. The questions are stratified into easy and hard levels based on the difficulty encountered by human experts and models during their resolution. In
A.4. Multi-agent collaboration
Dataset deseriptions. In the multi-agent collaboration task, we evaluate GWM on AgentClinic (Schmidgall et al., 2024) benchmark, specifically AgentClinic-NEIN, collected from the New England Journal of Medicine (NEJM) case chal- lenges. Each case in AgentClini-NEJM is multimodal, comprising a case description, patient profile, clinical pho- tograph, measurement results, and five candidate diagnoses. We partition AgentClinic-Nea into training, validation,
Instruct htnn://lnugiinaaacccooeeesmmmmee
Instruct
16

---

Graph World Model
alignment with methodologies from previous studies such as GraphRAG (Edge al., 2024), segment long contexts into chunks that serve graph nodes, with edges defined by the similarity of their BERT embeddings. Building on this, we select the Top-k (k=5 in our setting) chunks with the highest similarity to the question's ' embedding to feed into the GWM. For this task, we divided the dataset into training. validation, and test sets in an 8:1:1 ratio.
Dataset descriptions. We employ the expert strategy dataset from the text-hased embodied task framework, ALF- World (Shridhar et al., 2020; Yang et al., 2024). This dataset provides detailed descriptions of the expert's 's strategic state at each decision point, incorporating both images and text, along with the corresponding decisions made in text format. In our approach, we represent each decision state as nodes within a graph and establish edges between these nodes based on the similarity of the state images linked to each decision. Our total sample size is 10,000, and it is divided into training, validation, and test sets in an 8:1:1 ratio.
Baselines details. We conduct comparisons with two RAG-based Seetinnes-MM2) (Robertson et al., 2009) and Dragon (Lin et al., 2023)-amm and three long-context LLMs (128k), including Mistral Large 24, Command R+3, and GPT-4o mini". Their details are as follows:
Baselines details. For all baselines, we first use LLa 7B to convert the image of each state into a text description.
BM25: A widely-used ranking function in information sparse retrieval. It inputs the retrieved context along with the question into the Llama-3-8B model to generate response.
Normal: It directly inputs the text description of the current state into Llama-3-8B to get the response.
COT: It adopts Chain-of-Thoure (Wei et al., 2022) prompting into baseline Normal to enhance reasoning ability when predicting.
Dragon: It employs contrastive learning and other train- ing tricks to finetune its ability to retrieve memory chunks. Using the Llama-3-88 model, it processes the retrieved context and the question to produce a response.
T5 FT (Raffel et al., 2020): It is a versatile language model designed by Google Research, which treats every language problem as a text-to-text task, enhancing its adaptability across broad range of NLP applications. Here we finetune it on the dataset of this task.
Mistral Large 2: Mistral Large 2 from Mistral AI boasts 123 billion parameters, with a context limit of 128 k to- kens. This model is one of the largest currently available, offering exceptional depth in language understanding and generation capabilities, suited for tackling the most de- manding NI.P tasks across various domains.
B. Hyper-parameters
For the GWM-F, we employ n-hop MLP, where for the LLM decoder each MLP has dimensions of 2048 and for the SD decoder, each MLP has dimensions of 2048*768. We fix the parameters of LLM and only fine-tune the pa- rameters of ML.. For GWM-T, we select a maximum of 2k token-limited hops for each task to ensure balance between efficiency and performance. We apply Lora (Lora rank 8) (Hu et al., 1) for efficient training. We have summarized the hyperparameters for training different models in Table 12.
Command R+: Command R+ by Cohere is a massive lan- guage model with 104 billion parameters, also supporting context size of up to 128 k tokens. It is optimized for un- derstanding and executing complex commands, making it particularly effective in interactive applications where precise and nuanced language comprehension critical.
GPT-4o mini: GPT-4o mini, developed by OpenAl, is a variant of the GPT-4 series. Unlike its larger counterparts, specific details about the model's size in terms of param- eters are not provided, but it is designed to handle max- imum context size of 128k tokens. This model is geared towards applications requiring high-quality text genera- tion with potentially limited computational resources.
Table 12. Syper-parameter configuration for model training.
Parameter
GWM-TLLM
GWM-T SD
GWM-ELLN GWM-E SD
Optimizer
AdamW
AdamW
AdamW
AdamW
Adam e
1c-8
lc-8
Le-8
Le-8 (0.9, 0.999)
Adam (g)
(0.9. 0.999)
(0.9. 0.999)
(0.99 0.999
Weight decay
le.2
le-2
le.2
Batch size per GPU
4
10
16
Gnadient Accumulation
A.6. Planning and optimization
Epochs
1
Resolution
512
256
This task is designed to measure how effectively different methods can imitate the trajectory of expert strategies, which is very helpful for planning and optimization tasks.
Leuningg rale
3e-4
lc-5
3e-4
le-5
Backbone SD
Llama-3-88
SD-v1-5
Lhama-3-8
SD-x1-5
"haps://huggingfacecco httms///tnngiiigaaaccoommmmiiiia htt.:///ww.....iiiaaaaaaaaaaaaaa
C. Prompt Usage of GWM
https//hhggingfaccc.o/oooooooeee cww.oobreFaarlllcciiccoommmaeee. http:///ww....mm/////////////o/a effecientimtelligennee
We summarize all the prompts we used in GWM in this section. We first introduce the action prompts in GWM.
17

---

Graph World Model
Specifically, have summarized the action prompts for multi-modal generation and matching in Tables 13 and 14. The action prompts for recommendations are summarized in Table 15. Additionally, the action prompts for traditional graph prediction are outlined in Tables 16, 17, 18, and 19. Moreover, we have also summarized the action prompts for multi-agent collaboration, eerieval-augmented generation, and planning and optimization in Tables 20, 21, and 22. Then, introduce prompts used in GWM-T. We summa- rize the prompt P.. of multi-modality as tokens in GWM-T in Table 23. Moreover, we introduce the prompt fo(+) of aggregating central node and neighbor nodes in GWM-T in Table 24.
For GWM-E, takes approximately 7 hours (~ of GWM- T) of training time four NVIDIA A6000 GPUs descrited in implementation details of section 5, and the inference time per case averages 0.213s (similar to GWM-T),. More- over GWM-E significantly reduces memory usage with a shorter token length of 140.23 (~ 1
14 GWM-T).
D. Qualitative Comparisons for All Tasks of GWM
These tables present comprehensive qualitative comparisons across task categories evaluated in our GWM frame- work study. Each table demonstrates the superior perfor- mance of our Graph World Model (GWM) variants com- pared to state-of-th the-art baselines through concrete exam- ples. Table 25 showcases multi-modal generation capabili- ties where GWM-T successfully predicts missing modalities from given inputs. Table 26 illustrates modti match- ing tasks where GWM-E accurately determines correspon- dence between different modalities. Table 27 demonstrates recommendation performance where GWM-E correctly pre- dicts user-item connections. Table 28 highlights traditional graph prediction tasks where GWM-E excels in node clas- sification. Table 29 presents multi- agget collaboration sce- narios where GWM-T integrates multiple agent contexts for medical diagnosis. Table 30 shows retrievallaugmenned generation capabilities where GWM-E effectively combines retrieved documents with user queries. Finally, Table 31 demonstrates planning and optimization tasks where GWM- E predicts optimal decision-making behaviors in embodied environments. These qualitative results consistently vali- date the effectiveness of our approach across diverse task domains.
E. Training and Inference Efficiency
Accurately comparing the training and inference efficiency of GWM with other FMs is highly challenging because many FMS are not designed to address multimodal prob- lems and utilize various architectures. We can only compare the efficiency berween GWM and LLM-bbsee FMs from principles. For GWM-T, its efficiency shows no fundamen- tal difference from other LLM-based FMs, as both based on the standard instruction tuning. For GWM-E, its training process only requires fine-tuning the projector as stated in section 4.3, and its embedding-basee -based method also saves a significant amount of token cost, making it more efficient.
18

---

Graph World Model
Table 13. Action prompt of multi-modal generation.
This is generation task. Please predict the missing modality based on the given modality: {modality).
Table 14. Action prompt of multi-modal matching.
This task involves matching multi-modal information. Given two modalities: (modality 1} and (modality 2). please determine whether they correspond with each other.
Table 15, Action prompt of recommendation.
This is a recommendation task. Given the user node and item node: {user node} and {item node), please tell me whether these two nodes should connect to each other.
Table 16. Action prompt of node classification of Cora.
Given a node-centered graph: {node}, each node represents paper, we need to classify the center node into 7 classes: Case Based, Genetic Algorithms, Neural Networks, Probabilistic Methods, Reinforcement Learning, Rule Learning, Theory, please tell me which class the center node belongs to?
Table 17. Action prompt of node classification of PubMed.
Given a node-centered graph: [node], each node represents a paper about Diabetes, we need to classify the center node into classes: Diabetes Mellitus Experimental, Diabetes Mellitus Type and Diabetes Mellitus Type 2, please tell me which class the center node belongs to?
Table 18. Action prompt of link prediction of Cora and PubMed.
Given two nodes information: (node }} and (node 2). please tell me whether two center nodes in the subgraphs should connect to each other.
Table 19. Action prompt of graph classification of HIV.
Human immunodeficiency viruses (HIV) are type of retrovirus, which induces acquired immune deficiency syndrome (AIDs). Please determine whether this molecule (molecule) effective for this assay.
19

---

Graph World Model
Table 20. Action prompt of multi-agent collaboration.
This is Multi-Agent Collaborative Generation task for creating dynamic conversational interactions. Given a user query: (user query) and context of three distinet agents: (Patient Agent Context), [Measurement Agent Context], and (Moderato Agent Context), Please generate well-rounded response the user's question.
Table 21. Action prompt of retrieval-augmented generation.
This is a Retrieval-Augmented Generation task for improving response quality in dialogue systems. Given a user query: {user query) and set of retrieved documents: fretrieved documents), the goal is to generate a coherent and contextually relevant response. Please generate a response that integrates information from the retrieved documents to accurately address the user's query.
Table 22. Action prompt of planning and optimization.
This is embodied household task, please predict the next decision-making behavior based on multimodal historical information: (historical information).
Table 23. Prompt P., of multi--modality as token GWM-T. The image's 's text description is: {image's 's text description), original text is: (original text), table description is: (table description).
Table 24, Prompt fo(-) aggregating central node and neighbor nodes in GWM-T.
The text description of the central node is: (center node), and the text descriptions of the neighboring nodes are: (neighbor nodes).
20

---

Graph World Model
Table 25. Task description and output comparison of multi-modal generation. This task is to predict the missing modality based on the given modality. Here utilize one case of Goodreads dataset as examples. performing GWM, and ControlNet, the strongest baseline.
We show the output results of GWM-T, the best
Task Description
Task Name
Multi-modal generation
Given modality
Action prompt
Ground truth
THE SHARK- INFESTED CUSTARD
This is multi-modal generation task. Please predict the missing modality based on the given modality: (modality).
Afteett
Title: The Shark-Infested Custard
CHARLES WILLEFORD
Method
Output Results
ControlNet
Auila syante Like dhned SAUNNAL
EET
maratt
GWM-T
21

---

Graph World Model
Table 26. Task description and output comparison of multi-modal matching. This task is predict whether modalities correspond with each other. Here we utilize one case of Multi-Modall-apge dataset as examples. We show the oulput results of GWM-E, the best performing GWM, and Contrastive MLP, the strongest baseline. Note that modality can be image, table, text.
Task Description
Task Name
Multi-modal matching
Modality 1: figure
Modality 2: text
Action prompt
Ground truth
Feature Map Cross Layer Similarity
Illustration of different formats of STI expressions. (a) Different expression formats of the same STL. (b) The binary tree representation of STL.
This task involves matching multi-modal information. Given two modalities: (modality 1} and {modality 2), please determine whether they correspond with each other.
VIT-300-304
Festure Simllrity with
Tefiren-iT-20000
yes
Depth
Method
Output Results
Contrastive MLP
no
GWM-E
yes
Table 27. Task description and output comparison recommendation. This task is to predict whether the user node and the item node connected. Here utilize of Baby dataset examples. We show the output results of GWM-E, the best performing GWM, and LightGCN, the strongest baseline.
Task Description
Task Name
Recommendation
User node
Item node
Action prompt
Ground truth
User online product reviews series: struggled to find full slips, especially larger ones. The first one was too small; the second fit well and was affordable. The beads looked stunning, perfect for beadwork. The earrings broke immediately due to poor quality. love the 3 flower sister Hawaii glass beads my Pandora bracelet. They're re pretty and large, and my eight-year-old finds them comfy enough to sleep in...
This is recommendation task. Given the user node and item node: (user node) } and (item node), ) please tell me whether these two nodes should connect to each other.
no
Method
Output Results
LightGCN
yes
GWM-E
22

---

Graph World Model
Table 28. Task description and output comparison of traditional graph prediction. This task aims to perform predictions at different levels: node, link, and graph. Here utilize one case of Cora's node prediction. best performing GWM, LLAGA, strongest baseline.
We show the output results of GWM-F, the
Task Description
Task Name
Traditional graph prediction
Node
Action prompt
Ground truth
Learning under persistent drift: In this paper study learning algorithms for environments which are changing over time. Unlike most previous work, we are interested in the case where the changes might rapid but their "direction" is relatively constant. We model this type of change by assuming that the target distribution is changing continuously extreme distribution to another. We show in this case how to use simple weighting scheme to estimate the error of an hypothesis, and using this estimate, to minimize the error of the prediction.
Given node-centered graph: (node), each node represents paper, we need to classify the center node into 7 classes: Case Based,
Genetic Algorithms, Neural Networks, Probabilistic Methods, Reinforcement Learning, Rule Learning, Theory, please tell me which class the center node belongs
Theory
constant rate from one
to?
Method
Output Results
LLAGA
Neural Networks
GWM-E
Theory
23

---

Graph World Model
Table 29, Multi-agent collaboration task and output comparison. rexcase-related query answering through agent interaction and external knowledge. Results show GWM-T vs COT baseline. Note: Medical images may cause discomfort but are from real datasets.
Task Description
Task
Multi-agent collaboration
User Query
Patient Agent
Moderator Agent
533-yar-olld man, 3-year history:
itchy rash, Raynaud's, dysphagia, burning hands.
Exam: firm papules on forehead with glabellar grooves, waxy papules hands with thickening and contractures. Similar changes on nose, lips,
Role: 53-year-old patient with 3-year symptoms including itchy rash, firm forehead papules causing brow grooves, waxy hand papules with thickening and
Moderator organizing case information:
cars, trunk, feet. No lelangiectasia/calciosss Sensory neuropathy in hands/arns/face. Normal thyroid. IgG-monoclonal gammopathy, normal bone marrow.
Test results: Normal thyroid function, IgG-monoclonal gammopathy detected, normal bone marrow biopsy.
finger contractures. Experience Raynaud's 's phenomenon, dysphagia, burning hands, and numbness in hands/arms/face. Aware of normal thyroid tests and abnormal blood protein but unaware of diagnosis implications.
Key findings: Extracellular yellow-brows brown deposits in dermis on skin biopsy.
Choices: (A) amyloidosis (B) Multiple myeloma (C) Scleredema (D)
Scleromyxedema (E) Systemic sclerosis
Measurement Agent
Action Prompt
Ground Truth
Multi-Agent Collaborative Generation
task dynamic conversational interactions. Generate response using
D
Patient, Measurement, and Moderator agent contexts for the given user query.
Method
Output Results
Analysis of patient symptoms and test results suggests monoclonal gammopathy with characteristic skin findings. DIAGNOSIS: Multiple myeloma (B)
COT
GWM-T
D
24

---

Graph World Model
Table 30. Task description and output comparison of reumeeell-augmented generation. This task is generate a response that integrates information from the retrieved documents to accurately address the user's query. Here we utilize one case of LongBench v2 dataset examples. We show the output results of GWM-E. best performing GWM, and GPT-4o mini, the strongest baseline.
Task Description
Task Name
betrieval-augmented generation
User query
Document
Action prompt
Ground truth
What is the correct answer to this question: You are given a grammar book of Kalamang language, now translate the following Kalamang sentence into English: Faisal emun me mindi don bolonet me ma kademor. Choices: (A) Faisal's s mother is still angry him for little thing like that. (B) Faisal's
There very few households with two fluent
This is Retrieval-Augment Generation task for improving response quality in dialogue systems. Given a user query: {user query) } and set of retrieved documents: fretrieved documents), the goal is generate a coherent and contextually relevant response. Please generate a response that integrates information from the retrieved documents to accurately address the user's query.
aalamang-speaking parents and children born after 1990, but even in those households the children are not raised Kalamang. As indicated above, non- non-fluent speakers have good passive command of Kalamang Fluent Kalamang speakers do not necessarily shift to Papuan Malay when they join the conversation, but they are not expected to actively contribute, although they can express themselves in simple way in Kalamang ...
mother turns furious at him for big thing like that. (C) Faisal's mother gets frustrated at him for a big thing like this. (D) Faisal's mother gets angry at him for
little thing like that. Format your response follows: "The correct answer is (insert answer here)".
Method
Output Results
GPT-4o mini
A
GWM-E
D
Table 31. Planning and optimization task and output comparison. Agent decision decision-making prediction using ALFWorld dataset. Results
show GWM-E T5 FT baseline.
Task Description
Task
Planning and optimization
Image
Text
Action Prompt
Ground Truth
Embodied household task: predict next decision-makin bchavior based on multimodal information.
Task: put potato countertop
go garbagecan
Method
Output Results
FT FT
go to microwave
GWM-E
go to garbagecan
25


Reviews:
[{"review_id": "ZyphYlKHCt", "reviewer": "Reviewer_5yBi", "summary": "This paper proposes a Graph World Model that supports both unstructured and graph-structured states with multi-modal data. The proposed model can tackle diverse sets of tasks and act as a graph-based foundation model. The results on numerous datasets and tasks show SOTA or comparable results on most tasks compared to the baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The method, although different in nature, solves a similar task to LANISTR [a], which would be nice to cite and contrast against.**  \n\n**Response:**  Thanks for the valuable suggestions. Indeed, we have already compared two baselines that, like LANISTR, were pre-trained by modality alignment and then fine-tuned on downstream tasks, Contrastive MLP (Liu et al., 2022), CLIP FT (Radford et al., 2021) in ***[lines 348-352] and [Table 3]***. However, we followed the reviewer's suggestions and compared the performance of GWM with LANISTR. Due to the limited time for rebuttal, we primarily focused on comparing the performance of GWM and LANISTR when directly finetuned on the Goodreads dataset in the task of multi-modal matching. It can be observed that GWM achieved a significant performance gain compared to LANISTR. In the next version, we will compare LANISTR in more tasks and evaluate its performance after pretraining on our dataset.\n\n| Model | Accuracy | Recall | F1 Score |\n|-------|-----------|---------|----------|\n| LANISTR | 54.3 | 51.8 | 43.6 |\n| GWM-T | 84.22 | 85.66 | 85.29 |\n| GWM-E | **88.82** | **89.73** | **89.06** |\n\n**Q2. In Tab. 4, the provided baselines are old. Maybe the authors can include FREEDOM [e] or another recent SOTA model for recommendation.** \n\n**Response:**  Thank you for your insightful comments. In the recommendation task, we primarily selected some classic and representative baselines for comparison. Our goal in comparison with task-specific baselines is not to demonstrate that GWM can beat sota in all tasks, but to show that GWM, with a unified model, has good generalization capabilities across many tasks, including recommendation. However, we followed the reviewer's suggestions and compared the performance of GWM with FREEDOM on the recommendation task shown in the following table. We can observe that, compared to FREEDOM, the two methods of GWM achieved better performance, which demonstrates the effectiveness of GWM. We will update the experimental results and discussion in the next version.\n\n| Model | Baby |  | Sports |  | Clothing |  |\n|-------|------|---------|--------|---------|----------|---------|\n| | Recall | F1 Score | Recall | F1 Score | Recall | F1 Score |\n| FREEDOM | 60.35 | 66.16 | 63.47 | 70.53 | 70.20 | 78.40 |\n| GWM-T | 70.84 | 75.08 | 84.29 | 88.60 | 71.73 | 74.26 |\n| GWM-E | **76.72** | **84.74** | **88.78** | **90.32** | **75.27** | **84.06** |\n\n**Q3. The graphics can be improved to make the method clearer and more understandable. Eg. in Figs. 1,3 it is not clear what each color identifies in Fig. 1: what is the difference between intended and unintended action, and what is the difference between the node/edge/graph level actions?** \n\n**Response:** Thanks for your valuable advice. Following your suggestions, we have separated the descriptions of different action levels and the methods for obtaining target nodes in Figure 1. Additionally, we have standardized the colors in corresponding parts of Figures 1 and 3. Lastly, we have added captions to each figure to differentiate the meanings of various colors and shapes. We showed the figures in Figure 1 and 2 of [rebuttal.pdf](https://anonymous.4open.science/r/ICML_2025_14809-7B6C/rebuttal.pdf). We will update these modifications in the next version.", "claims_and_evidence": "- The paper claims to propose a graph world model that can tackle multidomain tasks and be applied to both structured and unstructured\ndata.\n  - The claim is backed by numerous experiments and empirical results on multidomain and multimodal datasets with different combinations of structured (graphs) and unstructured (text, image).\n- The claims are clear and the results are convincing.", "methods_and_evaluation_criteria": "Yes, the proposed method claims to act as a foundation model, and the results on 11 datasets compared to the baselines on those datasets show better or comparable results.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The designed experiments are sound and valid and show the analysis of the proposed model in different tasks, datasets, and scenarios.", "relation_to_broader_scientific_literature": "The paper seems to be novel to the best of my knowledge in terms of using a graph-based world model that considers states and actions as graphs and enables the use and combination of multimodal structured and unstructured data, which is valuable. The method, although different in nature, solves a similar task to LANISTR [a], which would be nice to cite and contrast against.", "essential_references_not_discussed": "Although the proposed method in [a] is different from the one here, they could potentially solve similar tasks. It would be nice to contrast or compare against it.\n\n[a] Ebrahimi, Sayna, et al. \"LANISTR: Multimodal Learning from Structured and Unstructured Data.\" CoRR (2023).", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is very well-written, and it was a delight to read.\n2. The contributions are clear, and the claims are backed by the experiments.\n3. The model is extensively evaluated on multiple tasks and datasets and shows SOTA / comparable results in most scenarios.\n\nWeaknesses:\n1. It would be nice to include the SOTA results for the different tasks. Although for most tasks, the baselines are recent, in Tab. 4, the provided baselines are old. Maybe the authors can include FREEDOM [e] or another recent SOTA model for recommendation.\n\n\n[e] Zhou, Xin, and Zhiqi Shen. \"A tale of two graphs: Freezing and denoising graph structures for multimodal recommendation.\" Proceedings of the 31st ACM International Conference on Multimedia. 2023., https://dl.acm.org/doi/pdf/10.1145/3581783.3611943", "other_comments_or_suggestions": "1. The graphics can be improved to make the method clearer and more understandable. Eg. in Figs. 1,3 it is not clear what each color identifies in Fig. 1: what is the difference between intended and unintended action, and what is the difference between the node/edge/graph level actions?", "questions_for_authors": "Could the authors contrast their work against LANISTR [a] for general multimodal learning and, if possible, compare to FREEDOM [e]?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KSL25VwTQk", "reviewer": "Reviewer_4jVH", "summary": "In this paper, the authors propose Graph World Model (GWM), a framework designed to integrate both unstructured and graph-structured data with multi-modal information. They introduce two GWM variants: a token-based method that transforms multi-modal data into textual representations prior to message passing, and an embedding-based method that operates within a unified embedding space. The framework further incorporates action nodes to address diverse tasks and applies message-passing algorithms to combine structured information. The authors demonstrate competitive or superior performance against domain-specific baselines across six tasks and multiple domains. However, limited experimental comparisons, subpar presentation, and insufficient analysis negatively impact the paper’s overall quality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Can the author supplement the baseline that tuned on the task-specific dataset (e.g., Table 6).** \n\n**Response:** Thanks for the comments. Indeed, we have detailed the specific settings of the baselines for Table 6 in Appendix A.4 (***[lines 779-796]***). We primarily selected three classic agent baselines for comparison: CoT, ToT, and Few-shots. Although these baselines were not additionally fine-tuned on this dataset, the results of GWM shown in Figure 6, as described in ***[lines 382-384]***, were also not additionally fine-tuned. Therefore, from this perspective, the comparison between GWM and the baselines is fair. However, following the reviewer's suggestion, we compared two baselines fine-tuned on AgentClinic: FT is a baseline finetuned on the task using the Llama-3-8B model. Longformer [1] is a strong baseline for long document understanding. The table below reports the comparison results between GWM and these baselines, demonstrating the effectiveness of GWM in AgentClinic.\n\n| Method     | Acc       | Recall    | F1        |\n|------------|-----------|-----------|-----------|\n| FT         | 45.00      | 45.40     | 44.00     |\n| Longformer | 25.00      | 20.20     | 14.00     |\n| GWM-T      | **50.00** | **46.42** | **48.20** |\n| GWM-E      | 45.00     | 39.57     | 35.56     |\n\n***[1]*** Longformer: The long-document transformer.\n\n\n**Q2. The authors should have included baseline graph world models (Zhang et al. 2021; Zhu et al. ICLR 2023) in their experiments.**\n\n**Response:**  Thanks for your feedback. Indeed, the works on graph world models (Zhang et al. 2021; Zhu et al. ICLR 2023) do not consider the processing of multimodal information in their modeling and primarily focus on offline scenarios in the RL field. These limitations restrict their capabilities as world models and also make them inapplicable for direct comparison in our scenario. In contrast, GWM is a world model capable of handling multimodal information and generalizing across various tasks. We will supplement these discussions in the related work section of the next version.\n\n**Q3. The paper fails to address computational complexity and scalability considerations adequately.** \n\n**Response:** Thanks for your valuable comments. In fact, we address computational complexity and scalability from two perspectives. First, we introduce GWM-E, which, as discussed in ***[lines 275-277]***, fixes the LLM's parameters and only fine-tunes the projector parameters, thereby reducing computational complexity. Additionally, as described in section 4.2 ***[lines 234-246]***, we designed a simplified, parameter-free GCN. This not only reduces computational complexity compared to traditional GNNs (Wu et al., 2019; He et al., 2020a) but also benefits from PyTorch's sparse matrix multiplication acceleration, allowing it to scale more effectively to large graphs. As shown in ***Tables 10 and 11*** in the appendix, we have conducted experiments with large graphs at the scale of hundreds of thousands, demonstrating GWM's ability to address computational complexity and scalability.\n\n**Q4. The authors present only quantitative results without providing qualitative comparisons.**\n\n**Response:**  Thanks for the insightful advice. We have already added qualitative comparisons for all tasks in  Tables 1-7 of [rebuttal.pdf](https://anonymous.4open.science/r/ICML_2025_14809-7B6C/rebuttal.pdf). We will complete this information in the appendix of the next version.\n\n**Q5. Figure 3, which illustrates the GWM framework, suffers from quality issues that impede understanding. (1) Target nodes are not clearly specified—variables should be directly labeled. Additionally, the target nodes and prompt in the top-right of Figure 3 are two separate inputs but are not represented with distinct arrows. (2) The use of Stable Diffusion in the top-right creates confusion since it was designed for text-to-image generation, while the diagram shows token inputs and \"next state\" outputs. The authors should clarify that the model operates in Stable Diffusion's latent space.** \n\n**Response:** Thanks for your valuable suggestions. Following your suggestions, we first specified the target nodes in Figure 3. Next, regarding the issue with the top right arrow, we intended to prompt the target nodes, so we changed the content on the arrow to 'Prompt (Target nodes)'. Lastly, concerning the input and output issues of Stable Diffusion in GWM-T, as described in ***[lines 229-234]***, our input included action nodes and target nodes at the token level, and the output image served as the next state. To further clarify this process, we added legends to distinguish between different colors and shapes. We have updated the modified figure in Figure 2 of [rebuttal.pdf](https://anonymous.4open.science/r/ICML_2025_14809-7B6C/rebuttal.pdf) and it will be updated in the next version.", "claims_and_evidence": "The authors propose a framework that is clear.", "methods_and_evaluation_criteria": "- The paper bridges world models with graph-based approaches, addressing a clear gap in the literature. This integration allows handling both structured and unstructured data in a unified framework.\n\n- The benchmark datasets makes sense. The experimental evaluation across six diverse tasks (multi-modal generation/matching, recommendation, graph prediction, multi-agent collaboration, RAG, and planning/optimization) demonstrates the versatility of the approach.", "theoretical_claims": "This paper mainly provides the empirical analysis.", "experimental_designs_or_analyses": "1. **Limited experimental comparison**. The experiments lack comprehensiveness in several ways. (1) Table 6 fails to clarify whether baselines were fine-tuned on task-specific datasets, making it difficult to assess the true comparative performance. (2) The authors should have included baseline graph world models (Zhang et al. 2021; Zhu et al. ICLR 2023) in their experiments, particularly for world planning and optimization tasks, to properly demonstrate the advantages of their proposed methods.\n2. **Scalability and efficiency analysis**. The paper fails to address computational complexity and scalability considerations adequately. For real-world applications involving large graphs, multi-hop message passing could become computationally prohibitive, yet this limitation receives insufficient attention.\n    \n3. **Lack of qualitative analysis**. The authors present only quantitative results without providing qualitative comparisons, particularly for world generation tasks. Including actual examples of model outputs would help readers better understand the practical differences between GWM and baselines.", "relation_to_broader_scientific_literature": "The paper inadequately differentiates its methodology from previous graph world models such as Zhang et al. The authors focus primarily on application differences in the related work section rather than providing a thorough methodological comparison, which would better highlight their novel contributions.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "**Other Weaknesses**\n\n**Poor presentation affecting readability**. Figure 3, which illustrates the GWM framework, suffers from quality issues that impede understanding. (1) Target nodes are not clearly specified—variables should be directly labeled. Additionally, the target nodes and prompt in the top-right of Figure 3 are two separate inputs but are not represented with distinct arrows. (2) The use of Stable Diffusion in the top-right creates confusion since it was designed for text-to-image generation, while the diagram shows token inputs and \"next state\" outputs. The authors should clarify that the model operates in Stable Diffusion's latent space.", "other_comments_or_suggestions": "To enhance the presentation quality, the authors are encouraged to refine the figures and incorporate visualized examples to better illustrate the pipelines.", "questions_for_authors": "Most of the baselines are relatively weak (e.g., Table 6). Can the author supplement the baseline that tuned on the task-specific dataset.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UPhSmiidId", "reviewer": "Reviewer_a6Z5", "summary": "This paper proposes Graph World Model (GWM), a novel framework that integrates graph-structured data and multi-modal information into a unified world model for diverse tasks including prediction, generation, and planning. The authors present two variants (GWM-T and GWM-E) with distinct message-passing strategies and demonstrate competitive performance across six domains compared to domain-specific baselines. While the idea of unifying graph-based reasoning with multi-modal world modeling is promising, the technical depth and methodological clarity need further refinement to fully establish its novelty and scalability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Weaknesses 1:**  Thanks for your questions. We answer your questions one by one:\n\n***[Rationale of cross-modal fusion of GWM.]*** There are many ways to perform multimodal fusion. We selected two representative methods, not to avoid advanced fusion techniques. One is a simple and direct method that unifies multimodal information into text via GWM-T. The other method uses GWM-E, employing a cross-modal projector to unify different modal embeddings efficiently. Moreover, GWM-E does not have a modality dominance issue. We designed baselines named No-image and No-text, which mean training and testing solely on the text modality or image modality compared with GWM-E. We made comparisons on ALFWorld as follows. We can observe that the impact of images and text on GWM-E is almost equivalent, which demonstrates that GWM-E does not have modality dominance and all modalities are important. We will conduct more ablation studies in the next version to further substantiate this finding.\n\n| Method | Precision | Recall | F1 |\n|--------|-----------|--------|-----|\n| No-image | 92.95 | 92.80 | 92.86 |\n| No-text | 92.32 | 91.92 | 92.10 |\n| GWM-E | **97.31** | **96.52** | **96.17** |\n\n***[Rationale of simplified GCN and its adaptiveness to heterogeneous graphs.]*** Indeed, our main experiment is based on homogeneous graphs. Because heterogeneous graphs often rely on predefined node and edge types, but we have multiple tasks and also need the model to generalize to unseen tasks, it is hard to define node and edge types in advance. If heterogeneous type information is represented as tokens or embeddings similar to GWM, ordinary GNNs may solve this, but it would lead to complexity in implementation and be costly. A potential way to extend directly from GWM-E to heterogeneous graphs is to perform separate multi-hop aggregations based on different types of edges and then flatten the resulting node embeddings into sequences to feed into the LLM decoder. However, we believe that our results on six real-world tasks validate GWM’s effectiveness and efficiency, highlighting our contribution. We will leave the exploration of heterogeneous graphs to future work.\n\n**Response to Weaknesses 2:** Thanks for your comments. We have already detailed how GWM models state transitions in section 2.2. As described in ***[lines 56-73]***, we use a graph for general multimodal state modeling. Moreover, to address the diversity of action settings in traditional WM, we model actions as action nodes (***[lines 75-88]***) and unify them into text descriptions as in ***[section 3.3 and section 4.3]***. This allows GWM to use a unified model to solve multi-tasks. Lastly, in ***[lines 89-93 of section 2.2]***, we explain that our GWM essentially learns a transition function, which is aligned with principled world models.\n\n**Response to Weaknesses 3:**  Thanks for your comments. We answer your questions step by step:\n\n***[Avoid over-smoothing.]*** As introduced in ***[lines 247-253]*** of section 4.2, we input the multi-hop embeddings into the decoder. This is equivalent to a skip connection, a standard practice that alleviates oversmoothing [1, 2].\n\n***[GWM handles non-homophilous graphs.]*** Essentially, both versions of GWM transform graphs into a standard form interpretable by LLMs. Existing work (Liu et al., 2023a; Chen et al., 2024a) has demonstrated that LLMs can also effectively understand graphs, thus we anticipate that LLMs may outperform GNNs in comprehending homophilous or non-homophilous graphs. But if we must understand homophilous graphs from the perspective of GNNs, a potential way is to integrate [3] based on GWM-E, but this would introduce complex implementation and significant costs. Moreover, the performance across six representative real-world tasks has already validated the practicability and generalizability of GWM. We will include the above discussions in the next version.\n\n***[Graph transitions align with task objectives.]***  As discussed in section 2.3, our action is essentially a description of the task objective. For example, in the RAG task, the action of GWM is to answer specific queries with retrieved contexts. Thus, the graph transition modelled by GWM is aligned with the task objective without a theoretical gap.\n\n**[1]** Optimization of graph neural networks: Implicit acceleration by skip connections and more depth.\n\n**[2]** Representation learning on graphs with jumping knowledge networks.\n\n**[3]** Revisiting heterophily for graph neural networks.\n\n**Response to missing theory and hyperparameters analysis:**  Thanks for the comment. We prioritize empirical analysis of GWM across multiple tasks over theoretical proofs, which are challenging in LLM research. Given the foundation model's numerous hyperparameters and computational costs, we focused on introducing key hyperparameters (lines 357-372 and Appendix B) and analyzed critical ones like hop num (section 5.2), balancing reproducibility with practical costs.", "claims_and_evidence": "The claims and evidence make sense for the generalization problem.", "methods_and_evaluation_criteria": "The proposed method is reasonable. The evaluation criteria are appropriate for the problem.", "theoretical_claims": "The detailed theoretical analyses are missing", "experimental_designs_or_analyses": "The experiments provide strong empirical validations for the proposed method. However, the analyses on the important hyperparameters are weak.", "relation_to_broader_scientific_literature": "The paper is well-situated within the literature on graph foundation model, multi-modal diffusion and graph-based retrieval-augmented generation", "essential_references_not_discussed": "The important references are fully discussed from my point of view.", "other_strengths_and_weaknesses": "The strengths of this paper are as follows:\n(1) The integration of graph message-passing into multi-modal world models is a novel contribution, enabling unified handling of structured and unstructured data across prediction, generation, and optimization tasks. \n(2) The empirical validation highlights that multi-hop aggregation consistently improves performance (e.g., in recommendation and RAG), underscoring the importance of graph structure for complex reasoning.\n(3) The demonstrated zero-shot/few-shot capabilities on unseen tasks suggest practical value for low-resource scenarios.\n\nThe weaknesses of this paper are as follows:\n(1)\tAmbiguity in Multi-modal Fusion and Aggregation:\na)\tWhile GWM-T converts multi-modal data into text tokens and GWM-E concatenates CLIP/BERT embeddings (Section 3.1, 4.1), the rationale for avoiding advanced fusion techniques (e.g., cross-modal attention) is unclear. For instance, tokenization in GWM-T may lose fine-grained visual semantics, and simple concatenation in GWM-E risks modality dominance (e.g., text overshadowing images).\nb)\tThe parameter-free multi-hop aggregation (Section 4.2) adopts a \"simplified GCN\" design without adaptive mechanisms (e.g., attention in GATv2 or dynamic pruning in GRIT). This limits its ability to handle heterogeneous graphs or dynamically adjust aggregation weights during inference, potentially underutilizing graph structure (e.g., implicit vs. explicit edges in Section 2.2).\n(2)\tThe paper positions GWM as a \"world model\" but lacks a formal connection to classical Markovian state transitions (Section 2.1). For example, how do graph-based actions (Section 2.2) align with traditional WM components like transition probability? The framework currently resembles a graph-enhanced multi-modal system rather than a principled extension of world models.\n(3)\tThe simplified message-passing mechanism lacks theoretical grounding. For instance:\na)\tHow does the parameter-free design (Equation 4.2) avoid over-smoothing in deep aggregation?\nb)\tDoes performance gain primarily stem from graph homophily (e.g., node similarity), and how does GWM handle non-homophilous graphs (common in real-world scenarios)?\nc)\tWhat guarantees exist for the alignment between graph transitions and task objectives (e.g., RAG retrieval vs. graph-level optimization)?", "other_comments_or_suggestions": "I would suggest that the authors formalize the connection to world models and analyze trade-offs between graph complexity and task performance.", "questions_for_authors": "(1)\tWhy prioritize parameter-free aggregation over learnable mechanisms (e.g., attention)? How does this design impact performance on heterogeneous graphs?\n(2)\tHow are modality-specific embeddings aligned in GWM-E? Is there empirical evidence of modality dominance (e.g., text overshadowing images)?\n(3)\tCan the framework theoretically justify the relationship between graph transitions and task objectives (e.g., RAG retrieval vs. planning)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RVdGjzrmLX", "reviewer": "Reviewer_6SxJ", "summary": "This paper introduces the first world model in the graph domain. The proposed method is capable of handling multimodal graph data within a unified framework, accomplishing tasks across multiple domains, including prediction, generation, and optimization. The graph world model demonstrates exceptional performance and generalization capabilities on these tasks. Additional experiments highlight the role of graph structures in these tasks and the ability of the graph world model to leverage them effectively.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Issue 1 (Missing details):** Thanks for the comments. The settings for the baselines primarily follow LLAGA (Chen et al., 2024a) and OFA (Liu et al., 2023a). As stated in Appendix A.3, we convert all nodes and labels in the Cora, PubMed, and HIV datasets into text. For all methods, we use BERT to obtain text embedding. We divide all datasets into training, validation, and test sets in an 8:1:1 ratio. We will add these details to the next version.\n\n**Response to Issue 2 (Experiment costs):** Thanks for your questions. We answer them one by one:\n\n***[Training and inference efficiency.]*** Accurately comparing the training and inference efficiency of GWM with other FMs is highly challenging because many FMs are not designed to address multimodal problems and utilize various architectures. We can only compare the efficiency between GWM and LLM-based FMs from principles. For GWM-T, its efficiency shows no fundamental difference from other LLM-based FMs, as both are based on the standard instruction tuning. For GWM-E, its training process only requires fine-tuning the projector as stated in ***[lines 275-277]***, and its embedding-based method also saves a significant amount of token cost, making it more efficient. For GWM-E, it takes \\~7 hours (\\~$\\frac{1}{4}$ of GWM-T) of training time on four NVIDIA A6000 GPUs described in ***[lines 352-376]***, and the inference time per case averages 0.213s (similar to GWM-T). Moreover, GWM-E significantly reduces memory usage with a shorter token length of 140.23 (~$\\frac{1}{14}$ of GWM-T).\n\n***[Large-scale graphs feasibility.]*** GWM-E can be deployed to large-scale graphs. In section 4.2, we designed a simplified GCN. It not only reduces computational complexity (Wu et al., 2019; He et al., 2020a) but also benefits from PyTorch's sparse matrix multiplication acceleration, allowing it to scale efficiently to large graphs. As shown in ***Tables 10 and 11***, we have conducted experiments with large graphs at the scale of hundreds of thousands, demonstrating GWM's ability to address computational complexity and scalability.\n\n**Response to Issue 3 (Examples) and 4 (A mInor issue):** Thanks for the feedback. These tasks can all be framed as generation tasks when using LLM to construct a GWM. But as stated in section 2.3, the essence of world optimization is to guide agents in decision-making, while world generation primarily enables agents to perform QA responses and generation, thus, their objectives are different. We distinguish these tasks to better demonstrate the capabilities of GWM in various aspects.  Additionally, we have provided detailed descriptions of each task in ***section 2.3, section 5, Table 9, and Tables 13-22*** of the paper. However, following the reviewers' suggestions, we have supplemented the details in Tables 1-7 of [rebuttal.pdf](https://anonymous.4open.science/r/ICML_2025_14809-7B6C/rebuttal.pdf). We will add these details to the next version.\n\n**Response to utilization of graph structures:**  Thanks for the comment.  Both GWM versions fully leverage graph structures for message passing from the perspectives of tokens and embeddings. Furthermore, we verify through the experiment in section 5.2 (No Graph vs N-hop) that graph structures enhance performance on downstream tasks.\n\n**Response to Question 1:** Thanks for your question. We didn't include INSTRUCTG2I (with complex QFormer design) as a baseline since our goal was to design a lightweight framework like GWM-E. Following your advice, we compared performance on multi-modal generation tasks as follows. Results show INSTRUCTG2I outperforms GWM on Goodreads but underperforms on Multi-Modal-Paper. This may be because the large-parameter INSTRUCTG2I also requires a larger dataset to achieve good results. In contrast, our lightweight GWM offers better efficiency while adapting well to multiple scenarios. We'll include this discussion in our next version.\n\n| Model | Goodreads |  | Multi-Modal-Paper |  |\n|-------|-----------|-----------|-------------------|-----------|\n|  | CLIP | DINOv2 | CLIP | DINOv2 |\n| INSTRUCTG2I | **50.37** | **25.54** | 56.37 | 18.80 |\n| GWM-T | 47.46 | 20.91 | **59.92** | 23.10 |\n| GWM-E | 45.23 | 20.87 | 59.84 | **26.03** |\n\n**Response to Question 2:** To validate whether similar conclusions to those in [1] can be achieved on GWM, we conducted an ablation study. Specifically, we designed baselines named Text-only, which means training and testing solely on the text modality compared with GWM-E. We made comparisons in planning and optimization (ALFWorld) as follows. It can be observed that removing other multimodal information leads to a performance decline in GWM, which aligns with the conclusions of [1]. We will conduct more ablation studies in the next version to further substantiate this finding.\n\n| Method | Precision | Recall | F1 |\n|--------|-----------|--------|-----|\n| Text-only | 92.95 | 92.80 | 92.86 |\n| GWM-E | **97.31** | **96.52** | **96.17** |", "claims_and_evidence": "The claims in this paper are well validated, with no significant over-claiming.", "methods_and_evaluation_criteria": "The proposed approach makes sense, but its core idea still relies on mixing structural information into LLM and LVM, followed by fine-tuning techniques. The innovation lies in unifying different tasks, which is commendable, but the utilization of graph structures is somewhat limited.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments are diverse and well-chosen, but there are some issues with task descriptions:\n\n1) **Missing setails** Taking graph prediction as an example, the task splits and the comparison methods are not clearly described. For instance, in the case of supervised GCN, the settings should differ significantly from those of the Graph World Model. How GCN utilizes textual information and other details are not provided. In fact, in a supervised setting, the performance of GNNs should be better.\n\n2) **Experiment costs** The cost of using the Graph World Model has also not been reported. As a reader more focused on graph-related tasks, I am concerned with the differences in training and inference efficiency between the Graph World Model and other foundation models, as well as the feasibility of deploying the Graph World Model to large-scale graphs.\n\n3) **Examples** The content on world generation and world optimization needs large revisions. \nFirst, for the generation task, examples of generated outputs should be provided for comparison. \nSecond, the input and output of the tasks should be clearly explained. I understand that the authors have simplified the descriptions in the main text due to space limitations, but the supplementary details in the appendix are still limited and not enough to help readers who are not deeply familiar with these tasks understand the task settings and configurations.\n\n4) **A mInor issue** The distinction between world generation and world optimization is unclear to me. They seem to be essentially the same type of generation task. Adding examples, outputs, and further clarifications for each task would be very helpful, especially given the broad range of domains covered in this paper.", "relation_to_broader_scientific_literature": "I believe this paper has a good broader impact. The concept of the World Model is an intriguing research direction, and exploring its potential to unify graph tasks would be a valuable attempt. It also provides inspiration for future work in this area.", "essential_references_not_discussed": "I believe the paper does not have any significant references not discussed.", "other_strengths_and_weaknesses": "The strength of this paper lies in its exploration of an interesting task setting, and integrating the World Model is a great attempt at advancing graph foundation models. Additionally, the experiments are well-designed and cover a broad range of domains, making this paper one of the most comprehensive I have read on graph foundation models.\n\nThe main issues with the paper are clarity and technical novelty. In terms of clarity, the descriptions of the task details are too rough, and the comparisons between sub-tasks are limited, making it difficult to evaluate the performance of each task individually. This issue ties into the limited technical novelty that was mentioned earlier.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The main questions have been listed before, and other issues require the authors to provide additional details. Once I have a clearer understanding of the experimental setup, I could raise further points.\n\nOther questions:\n1) Why is INSTRUCTG2I not included as baselines?\n2) In [1], multi-modal information, including images and text, has been shown to benefit predictors in traditional graph tasks. Can the graph world model achieve that as well?\n\n[1] Zhu, Jing, et al. \"Multimodal graph benchmark.\" arXiv preprint arXiv:2406.16321 (2024).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the challenge of addressing both unstructured and graph-structured states with multi-modal information, representing diverse tasks as actions. To tackle this, the authors propose the Graph World Model (GWM), which leverages a generic message-passing algorithm to aggregate structured information. This can be done either over a unified multi-modal token space by converting multi-modal data into text (GWM-T) or a unified multi-modal embedding space using modality-specific encoders (GWM-E). Experiments on several datasets demonstrate the effectiveness of the proposed model.


Strengths:
1. The paper is well-written, and easy to follow.
2. The paper successfully bridges world models with graph-based approaches, addressing a clear gap in the literature. This integration enables the unified handling of both graph-structured and unstructured data.


Weaknesses:
1. Some experiments lack sufficient baselines to convincingly demonstrate the effectiveness of the proposed model. Additional aspects of the experiments, such as scalability and efficiency analysis, as well as a more thorough qualitative analysis, should be enhanced to make the results more persuasive.
2. The theoretical analysis could be expanded to provide more depth and clarity.




Overall, I recommend that the authors further refine the paper in terms of experiments and theoretical analysis to ensure a more comprehensive and convincing presentation in the camera-ready version.
Decision: Accept (poster)