Graph4MM: Weaving Multimodal Learning with Structural Information
Authors: Xuying Ning
Dongqi Fu
Tianxin Wei
Wujiang Xu
Jingrui He
Venue: ICML 2025 poster (2025)
Keywords: Multi-modal Learning
Large Language Models
Graph Neural Networks
Primary area: deep_learning->graph_neural_networks

Abstract: Real-world multimodal data usually exhibit complex structural relationships beyond traditional one-to-one mappings like image-caption pairs. Entities across modalities interact in intricate ways, with images and text forming diverse interconnections through contextual dependencies and co-references. Graphs provide powerful structural information for modeling intra-modal and inter-modal relationships. However, previous works fail to distinguish multi-hop neighbors and treat the graph as a standalone modality, which fragments the overall understanding. This limitation presents two key challenges in multimodal learning: (1) integrating structural information from multi-hop neighbors into foundational models, and (2) fusing modality-specific information in a principled manner. To address these challenges, we revisit the role of graphs in multimodal learning within the era of foundation models and propose Graph4MM, a graph-based multimodal learning framework. To be specific, we introduce Hop-Diffused Attention, which integrates multi-hop structural information into self-attention through causal masking and hop diffusion. Furthermore, we design MM-QFormer, a multi-mapping querying transformer for cross-modal fusion. Through theoretical and empirical analysis, we show that leveraging structures to integrate both intra- and inter-modal interactions improves multimodal understanding beyond treating them as a standalone modality. Experiments on both generative and discriminative tasks show that Graph4MM outperforms larger VLMs, LLMs, and multimodal graph baselines, achieving a 6.93% average improvement.

Graph4MM:
Weaving Multimodal Learning with Structural Information
Xuying Ning Dongqi Fu 2 Tianxin Wei!
Wujiang Xu
Jingrui He
Abstract
Section (pure text)
previous
Real-world multimodal data usually exhibit com- plex structural relationships beyond traditional one-to-one mappings like image-caption pairs. Entities across modalities interact in intricate ways, with images and text forming diverse inter- connections through contextual dependencies and co-references. Graphs provide powerful structural information for modeling intra-modal and inter- modal relationships. However, previous works fail to distinguish nulti-hop neighbors and treat the graph as a standalone modality, which frag- ments the overall understanding. This limita- tion presents two key challenges in multimodal learning: (I) integrating structural information from multi-hop -hop neighbors into foundational mod- els, and (2) fusing modality-specific information in principled manner. To address these chal- lenges, we revisit the role of graphs in multi- modal learning within the era of foundation mod- els and propose Graph4MM, a graph- based multi- modal learning framework. To be specific, we introduce Hop-Diffused Attention, which inte- grates multi-hop structural information into self- attention through causal masking and hop diffu- sion. Furthermore, we design MM-QFormer, a multi-mapping querying transformer for cross- modal fusion. Through theoretical and empiri- cal analysis, we show that leveraging structures to integrate both intra- and inter-modal interac- tions improves multimodal understanding beyond treating them standalone modality. Experi- ments on both generative and discriminative tasks show that Graph4MM outperforms larger VI.Ms, LLMs, and multimodal graph baselines, achieving a 6.93% average improvement.
include
iexT previouu
Text-Tett
Caption
(pure text)
Page
Section II (pure text)
Description
pair
Texc Imago
(pure
include
Image
next
previous
Image-
Section
include
Image
Given the contents and relations, generate the Section III?
Figure I. Multimodal relationships a document, where sections, images, captions, and page descriptions form structured graph. The task aims to generate Section using the neighboring context.
1. Introduction
Data modalities, in general, refer to different types of the data, like text, image, and audio (Ngiam et al., 2011). Jointly considering cross-modality data has great research and ap- plication potential, and it has been widely applied in image generation, question answering. and many more (Gao et al., 2020; Summaira al., 2021; Stahlschmidt al., 2022; Jabeen et al., 2023). Recently, pioneering study (Yoon et al., 2023) discovered that the relationship of data modal- ities is complex and far beyond the one-to-one modeling, like the mmage-caption pair, in real-world scenarios. For example, in an academic paper, the image and text data have different semantic and non-linear correlations, i.e., the image and its caption have a direct pairing relation each other, but the relation between the image and its following section contents and the page summarization is intricate, as shown in Figure 1. However, most Vision-Language Models (VLMs) (Alayrac et al., 2022a: Li et al., 2023c; Koh et al., 2023; Zhang et al., 2024a) are still limited to modeling one- to-one relationships between images and text, making them inadequate for capturing complex multimodal interactions.
Graphs, as relational data structure, have the inherent advantage of modeling complex modality relationships to- gether. To the best of our knowledge, MMGL. (Yoon et al., 2023) is the state-of-the-an work that models modalities into graphs and obtains promising performance in the gen- eration task, as compared to single-mode pre- trained lan- guage models and vision-language models. In particular,
Equal contribution University of Illinois Urbana- Urbana-Champaic Champaign 'Meta Al 'Rutgers University. Correspondence to: Jingrui He <jingrui@ illinois.edu>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Graph4MM: Weaving Multimodal Learning with Structural Information
models each modality data item (e.g., a sentence piece, an image) as node and establishes their edges based the concrete application scenario. For example, when address- ing section summarization, edges are established based on section-subsection hierarchy and co-occurrence of images and text within the same section.
modal fusion.
Revisiting the Role of Graphs in Multimodal Learn- ing, We conduct both theoretical and empirical analysis on the role of graphs in multimodal learning within the era of foundation models. Our findings suggest that leverag- ing topological structures to guide intra- and inter-modal interactions more effective than treating graphs as an independent modality.
Establishing the modality graph can help select and fuse the important knowledge from the raw multimodal data and further contribute to solving the context length limitation for language models. Nevertheless, the modality fusion in the pioneering MMGL (Yoon et al., 2023) is simple and does not fully exploit the complex relationship. To be more specific, although cross-modal data is modeled into multi- modal graphs, MMGL. simply concatenates the neighbors's 's multi-modal data together, where the internal adjacency is largely ignored, e.g., different distant nodes are treated equally. Moreover, MMGL regards the established graph as standalone modality with images and text. However, this approach for injecting graph topology information does not yield the expected performance improvements.
State-of-the-Art Performance Across Generation and Discrimination. We evaluate Graph4MM on generative (c.g., academic paper section summarization) and discrimi- native (e.g., zero-shot fashion classification) tasks. Exten- sive experiments show that consistently outperforms pre- trained VLMs, LLMs, and multimodal graph learning meth- ods, achieving an average improvement of 6.93% across textual and visual metrics.
2. Preliminary
In this section, to pave the way for introducing our mul- timodal framework Graph4MM, we first introduce the multimodal graph modeling and the corresponding advan- tages. Then, introduce two formal tasks the proposed Graph4MM aims to solve.
Motivated by the above observation, we further explore the role of graphs in multimodal learning in the era of foun- dation models from empirical and theoretical analysis in Section 4.3. Briefly, unlike conventional modalities such as text and images with densely pretrained representation, treating graph structures as an independent modality and pro- jecting graph embeddings into the same space as language and vision models often result in suboptimal performance. This is primarily due to the volume of training data and the well-aligned feature spaces of pre-trained language and vision foundation models, which can hinder effective fusion and limit the model's 's ability to do downstream tasks.
2.1. Multimodal Graph Modeling
Existing works (Yoon et al., 2023; Jin et al., 2024; Zhu et al., 2024b) offer varying definitions of multimodal graphs. To unify these perspectives, we define a multimodal graph as an unweighted, undirected structure: G = EV,E, PPP).
Therefore, we propose Graph4MM, a structured multi- modal learning framework that simultaneously captures intra-modal multi-hop structural connectivity and fuses inter- modal representations in a principled manner. Our contribu- tions summarized as follows:
text-to text rolation [ontionui): eg. sestion- subsection
Struct Node
index i text (optional) rz image (optional) coptioning selation
Image- to image relation (optionai): eg. figur-subffgure
A Novel Structure-Guided Paradigm for Multimodal Learning. To address the limitations of existing multi- modal learning methods in capturing complex modality in- teractions, we propose Graph4MM, a multimodal learning framework that integrates structural information from multi- hop neighbors into foundation models and fuses modality- specific representations in principled manner.
Figure 2. Multimodal Graph Modeling
Node. In the multimodal graph G = EV,E, P), we model node with unique index and optional textual and visual attributes. Mathematically, each node v; E V (e.g., a sec- tion in a webpage) is represented by a unique node index i and may optionally include textual attribute Log E T (e.g.. its textual content) and visual attribute Pv. E P (c.g., an associated image).
Designs of Hop-Diffused MM-QFormer. We introduce Hop-Diffused Attention, which incorporates multi-hop con- nectivity into self-attention using causal masking and hop diffusion. Theoretical analysis shows that avoids over- smoothing and does not rely on stacking multiple GNN layers for multi-hop aggregation. Additionally, we design MM-QForm querying transformer, to facilitate cross-
Edge. In the multimodal graph 9 = DVE,T,P) P). we con- sider three kinds of edges, i.c., text-to-text, image-to-image, and -on-to- Based on our node modeling, the text- to to-image relationship, like captioning, is basic and already encoded in each node, Further, the text-to-text edge exists
2

---

Graph4MM: Weaving Multimodal Learning with Structural Information
between nodes and j if and only if both nodes contain non- null text features and share meaningful predefined real- world relationship (e.g,, section-subsection in a paper, de- scriptions of frequently co-purchased items). The same rule also applies to image-to-image edges between two nodes.
3. Proposed Graph4MM Framework
In this section, we systematically present our Graph4MM framework, which synergizes the heterogeneous inputs from vision and language while simultaneously capturing their complex structural interactions. The overview of our frame- work is presented in Figure 3. To be specific, in Subec- tion 3.1, we first introduce an elementary backbone for fusing multimodal information, which serves as a vanilla architecture to inspire the more effective Graph4MM. Then, in Subsection 3.2, we dive into each modality and utilize multi-hop neighbor connectivity in the graph to guide the intra-modaie intra-modality information fusion via the proposed Hop- Diffused Attention. Finally, in Subsection 3.3, we intro- duce multimodal query transformer, MM-QFormer, which captures inter-modal interaction and fuses modal-specific information for downstream foundation models to deal with generation and discrimination tasks.
Through this multimodal graph representation, for a node i, along text-to-text (or image-to-ing to-immge) edges, we can easily induce its textual subgraph G. (V,, E,, Pt) as Vi = {u E V dist(u, (u, 04) - T) where V. includes nodes with textual attributes reachable within 7-hops, and & cap- tures text-text relationships. Similarly, the visual subgraph - (Vp, Ep, Tp. Pp) models image-to-image interactions. These subgraphs, combined with text-image associations at the node level, act as adjacency heuristics that guide causal masking attention, facilitating both mordatyy coherence and inter-modalhe fusion.
2.2. Task Definitions
3.1. The Vanilla Architecture for Fusing Multimodal Information
By modeling multiple modalities into graphs. we then math- ematically introduce the tasks Graph4MM aims to solve.
The emergent reasoning capabilities (Wei et al., 2022a) of large language models are essential for tackling zero-shot open-ended QA tasks (Guo et al., 2023). To effectively pre- serve these abilities of pretrained language models while efficiently integrating relevant visual information, our mul- timodal fusion strategy focuses on projecting visual inputs seamlessly into the LLM's s semantic space, leveraging off- the-shelf frozen pretrained language and vision models.
Generation Task. Given node E V, an instruc- tion prompt I, and its associated multimodal subgraphs G. and Gp. our objective is to generate a response y., = (11, .,,...,, (1) by maximizing:
Fv, arg max Yog
II P(ye I δ1. 02, 0e Ip. 1=1
Textual Context Integration. We define the textual con- text er (G.) derived from the subgraph G. = (Vi, E,, Tı, Pt), where V, consists of the target node e; and its r-hop neigh- bors N (1;)) = {u E V dist(u, v;) S T). The textual input sequence is constructed as:
where G. and Gp represent the textual and visual subgraphs. The generated response leverages multimodal interactions to address the task defined by I. For example, in document understanding, given the instruction "Summarize this sec- tion",, Graph4MM integrates broader page-level multimodal context to generate coherent and context-aware summary.
cr = [hr (hr (to) I E v.Jl E RR'r
(3)
Discriminative Task. Here, we target the more challenging zero-shot classification task, where an LLM is called to generate a language response conditioned on the textual subgraph 9., the visual subgraph Gp. and a classification prompt I. For given node 2/2 E V. LLM generates a response y., for the classification answer and assigns the node to the most relevant class by solving:
where I represents instruction prompts, hr(()) maps text inputs to their corresponding token embeddings. ly is the sequence length of the complete textual input, d is the di- mensionality of the token embeddings, and [-] represents the concatenation operation.
Visual Context Integration. The visual context cp(Gp) is derived from the visual attributes of nodes in 9p (Yp, Tp,Pp) Pp) Using a frozen visual encoder Jing (.), we compute the visual embeddings as:
c(rt)= arg max oly u.y dey), ej
cp(C)) H(mmg |
VgD)] E Rllixdd
(4)
where y., = PLM Gp. I) is the generated response from the pretrained language model, and (6-) denotes similarity function. Here, de represents the embedding of the description of class j. Graph4MM utilizes both node and neighbor attributes for enhanced zero-shot classification, which is generally unavailable in traditional graph learning.
where Pe E Pp denotes the visual attribute of node v, and dp is the dimensionality of the visual embeddings.
Structured Multi-modal Input for PLM. Both textual and visual contexts are processed separately and then con-
3

---

Graph4MM: Weaving Multimodal Learning with Structural Information
Text Subgraph
Hop-Diffused Attention
MM- MMMQFoorer
Anchor
Learnable
* Foord
Hop Ne ghbors
Pre-trained
Causal Mask
Self- Aol-Attentioon Layer
LLM
Diffusion
Cross Aer--Atteeioon
Hop
Food Food Sedl-Forwaar
MM
Language Modal
Ne gegrooor
Adjocency Heunstic
Quary Tokens
Response
Vision
Adjocency Heuristic
Hop Diffused
Multimodal Graph
Anchar
Frrbediin Vector
Self-Attention
Caural Mask
Adapter
Hop Neighbors
Pro pre-trained Encoder
V
Diffusion
Hop
Neighbors
Instruction Node Textual Attributes
Figure 3. Overview of the Graph4MM framework. Hop-Diffused Attention applies adjacency heuristics via causal masking and hop diffusion, while MM-QFormer captures fine-grained text-image interactions hefore passing inputs the L.I.M
catenated to form the input to the pretrained language model:
textual embeddings: Hr E R [ To integrate visual in- formation, visual embeddings are first concatenated over all nodes Vp, and then projected into the token embedding space of the language model via learnable linear transfor- mation Mpnj E RRoxdd Hp Owev, 9img 9mmg(P)) Mpnyj R Rlylxx, The Hop-Diffused Attention module is completely symmetric for both visual embeddings Hp and textual em- beddings Hy; for brevity, we use Hp to illustrate below.
h(CT, cp) = (9.), f(cp((,)) E IR (rrt)))d
where er (9t) E R'rxe xd represents the textual context, and ((p) Riexd maps the visual context (pp) E R into the PLM's semantic space using transfor- mation function R R2lld,, + R which will be comprehensively introduced in the Section 3.3. This map- ping ensures the alignment of visual features with the token embeddings of the language model, and lp is the num- ber of multi-modal (MM) tokens corresponding node. Then, the PLM takes h(er, Cp) as input tokens and generates target outputs based on the task requirements mentioned in the instruction.
Self-Attention. Self-attention captures semantic relation- ships between node embeddings. For Hp E g where each h represents the embedding of node Vi and qu. W Weh ke, Wa the initial attention matrix A R VAlxVV is computed as:
each
In our implementation, the MM tokens corresponding to each node are inserted directly after the textual attribute to- kens of the same node. This arrangement ensures adjacency between textual and MM tokens, effectively highlighting the one-to-one correspondence for downstream pretrained language models to process multimodal inputs.
A = Softmax,
qu, Key
(5)
where W g. Wa Raxd are learnable weight matrices pro- jecting node embeddings into query and key spaces, and A l reflects the importance of node U1 to node v,
Causal Masking, encode the graph structure, we define causal mask M,,, that restricts attention valid neighbors based on E,, the edge set of the subgraph Gp. The mask is defined as:
3.2. Hop-Diffused Attention.
Existing vision- languaee models (Li al., 2023b; Yuan al., 2021; Tsimpoukelli et al., 2021; Radford et al., 2021) primarily align single image-text pairs, overlooking the com- plex relationships among multiple images and text in tasks like ours. For example, LLMs often treat all visual inputs equally, ignoring the greater importance of node's own image. To address this, we propose Hop-Diffusse Attention which integrates graph structure form multi-hop connec- tivity into node embeddings through self-attention, causal masking, and a diffusion mechanism.
Mij 10000 11 if (w,, uy) E Ep,
(6)
otherwise.
This mask ensures that attention is computed only for nodes connected by edges in Ep. The attention scores between nodes are then calculated as: Ai, = Softmax (M,, A.,)). By applying M.,j. nodes are restricted to attend only to their immediate neighbors, preserving the graph's local structure.
This ensures that the attention mechanism aligns with the graph topology and the downstream model can distinguish between semantically similar images with different connec- tivity (e.g., directly connected not connected).
Text and Vision Encoding. We encode the text attributes of nodes in Vú using a frozen pretrained language model. Mean pooling over the sequence length dimension produces

---

Graph4MM: Weaving Multimodal Learning with Structural Information
Diffusion Mechanism. While causal masking ensures local connectivity, multi-hop structural information is captured via a diffusion mechanism that propagates attention across neighbors iteratively. Inspired by Wang et al. (2020), the diffusion matrix A is defined as:
The definition of Dirichlet Energy and the proof of Proposi- tion 3.1 provided in Appendix C.
A Lightweight Alternative: Hop-Aware Attention. To simplify the computation complexity (O(V))- dd)) of Hop- Diffused Attention while incorporating topology, we in- troduce learnable hop embeddings h g E E for each hop h E {0, 7}. where 5 denotes the maximum hop count in the visual subgraphs. For visual embeddings Hp E kl,, hop embeddings Hhup are added based on each node's hop information It;:
A 0,A 1=0
0, = 0.=a(1 a)',, a)', a (0. 1),
(7)
where A' represents the i-power of the masked attention matrix A, and 0g are decay coefficients with E 0: = 1. The parameter a controls the influence of higher-hop neigh- bors, prioritizing closer connections through exponential decay. In practical implementation, we approximate the infinite summation by truncating at a finite number of steps K referred to as the diffusion step.
Hp t Hp + Hhop
Hoop
(ha) Dnop JVEV
(10)
IvE,, where I represents the concatenation of learnable hop embeddings for each neighboring node in Vp. Specifi- cally, neighbors at the same hop distance (e.g.. 1-hop neigh- hors) share the same hop embedding (e.g., h
The node embeddings are updated using the diffused atten- tion matrix with residual connection:
The above process is fully symmetric for textual embed- dings, where hop embeddings are similarly added to the original textual features Hr to create topology- repre- sentations. This approach reduces the computational com- plexity to O(), dd), while preserving critical hop informa- tion to guide the downstream model in adaptively learning the importance of the information from different hops.
H + Hp + AHP,
where H E RR. Vp xd integrates multi-hop structural infor- mation into the node representations after the above update. This formulation enables nodes aggregate information not only from their direct neighbors but also from distant nodes while attenuating the impact of farther nodes. The resulting hop-diffused embeddings Hp and H are then fed into the MM-QFormer, which will be discussed in the Sec- tion 3.3 ensuring effective multimodal fusion and structural information integration.
3.3. Multi-Mapping QFormer
The most relevant work in multimodal graph learning for generative tasks, (Yoon et al., 2023), uses simple lin- ear projection (Tsimpoukelli et al., 2021) to map visual information into the language space but overlooks com- plex text-visual interactions. To address this, we propose the Multi Mapping QFormer (MM-QFormer) framework, which takes the topology-integrated embeddings (e.g.. Hop- Diffused Hp and Hr) as input, inspired by (Li et al., 2023b). MM-QFormer employs learnable query tokens and two-layer transformer, where shared self self-attention facili- tates text-query interactions, and cross-attention selectively extracts visual features relevant to textual inputs.
Theoretical Justification of Hop-Diffused Attention. We establish the validity of Hop-Diffused Attention by proving that the diffused attention matrix preserves key attention properties, ensuring that each row sums to one. This guaran- tees that the mechanism remains a valid attention operation while generalizing Personalized PageRank with an adaptive multi-hop weighting scheme. Detailed proof and theoretical analysis can be found in Appendix A and Appendix B.
We further analyze Hop-Diffused Attention in terms of gra- dient smoothing and demonstrate why it cannot be replaced by traditional graph models like GAT (Veličković et al., 2018). Specifically, we establish the following result:
Shared Self-Attention. We initialize learnable query to- kens as Q! (0) E R for capturing intricate multi-modal information fusion, where 4n = 1Vpl x mg. and n, repre- sents the number of multi-modal tokens per node. At each layer the query tokens from the previous layer Q! (t-1) are concatenated with the textual embeddings Hy to form a joint representation:
Proposition 3.1. Let (1) be the node representation matrix after the Hop-Diffused Attention module, and X (k) be the representation at the k-th layer of GAT. When both methods aggregate information from k- k-hop neighbors, Hop-Diffused Attention retains higher Dirichlet energy. preserving more feature variance and mitigating over-smoothing. Formally, for large k.
Hor (Q.-"), HH| E R [(l++vll)xd
(1I)
To make the learnable query can be condition on text in- formation, the joint representation Hør is then processed through shared self-attention:
HoopDiffused X (1))
EGAT (x))) (k),
(9)
where E() denotes the Dirichlet Energy, which quantifies the smoothness of node representations.
HOr SAT(g = Hor, Hor, v = Hor], (12)
5

---

Graph4MM: Weaving Multimodal Learning with Structural Information
where S-T(-) denotes multi-head self-attention. The up- dated query tokens are: Q = Horl Ip,:] E Rlexd x The query tokens are designed to integrate multimodal in- formation by jointly learning from text and visual features. Through shared self-attention, the query tokens refine their representations while being conditioned on textual context, laying the foundation for better interacting with visual fea- tures in subsequent cross-modal operations.
for LLaMA-1B) for evaluation. Dataset details are pro- vided in the Appendix F. Code is available at https: / wtt.huub..co//eennniiii.i Graphng
Baselines. We evaluate PLMs, VLMs, and MMAG vari- ants from MMGL (Yoon et al., 2023) under different input settings. For PL.Ms we compare OPT-125M (Zhang et al., 2022b) and LLaMA- (Touvron et al., 2023), while for VLMs, we assess BLIP2-OPT-2.77 (Li et al., 2023b) and Qwen2-VL-7BB
Modality Cross-Attention. The cross-attention mecha- nism then aligns the updated query tokens ( with these visual embeddings:
(Wang al., 2024). The compar- ison includes four key settings: (1) using only a node's text (NODE'S S TEXT), (2) adding image features (NODE'S 's TEXT & IMAGE), (3) incorporating subgraph-level text (SUBGRAPH'S ' TEXT), and (4) integrating both subgraph text and images (SUBGRAPH'S TEXT & IMAGE). For PLMs, further explore fine-tuning and graph-enhanced modeling. Besides, in MMGL, a unique setting is SUB- GRAPII' T & + GNN that utilizes GNNs to learn the subgraph's 's topological structure and incorporates it as a new modality into the visual and text input embeddings. Full baseline details are provided in the Appendix G.
CATlq Q k- k Hp, Hy] E Rloxd
(13) where C- denotes multi-head cross-attention. The cross-attention module enables the query tokens to extract relevant visual information by attending to the visual em- beddings. This ensures that the multimodal query tokens are aligned with the most pertinent visual features while maintaining the contextual guidance established by the text.
Feed-Forward Module. Each coss-attention layer is followed by a feed-forward network (FFN) to further pro- cess the updated query tokens: Q." = FFN Qu (,),, where (FN is two-layer fully connected network with a non- linear activation function in between, ensuring dimensional consistency:
4.2. Comparing Results
Our proposed Hop-Diffused and Hop-Aware MM-QFormer consistently outperforms pretrained VLMs, pretrained LLMs, and MMGL across both generative and discrimina- tive tasks with on average 1.77% and 12.09% improvement resectively. From the experiment results, we find that pre- trained VLMs perform the worst due to their pretraining focus on image captioning and simple QA, often generat- ing irrelevant image descriptions instead of directly giving answers to the instructions. Among PLM-based methods, incorporating subgraph context consistently improves per- formance over using only node attributes, further validating the importance of modeling relevant multimodal contexts. MMGL performs well, but adding GCN embeddings de- grades performance, likely due to the semantic gap between graph embeddings and LLM representations, which will be discussed in Section 4.3. Our MM-QFormer, with better text-image interaction modeling, achieves superior results compared to the strongest baselines. The introduction of Hop-Diffused Attention and Hop-Aware Attention further enhances MM-QFormer by modeling multi-hop connectiv- ity, allowing the model to capture structure-aware multi- modal information, thereby improving both generative and discriminative performance.
After L layers of shared celf-attention, cross-attention, and feed-forward modules, the final query tokens Q!" are pro- jected into the multimodal semantic space, f(cp(Gp)) = Q!7T E R Here, f(cr(Gp)) represents the learned multimodal tokens, and 1p = 1/pl x ngg Combined with textual prompts, these tokens form the input sequence for the downstream PLM, enabling it to effectively process open-ended tasks.
4. Experiments
4.1. Experiments Setups
Datasets. We evaluate our approach on two datasets that exhibit many-to-many text-image relationships. For the generative task, we use WIKIWEB2M (Burns et al., 2023), where the goal is to generate a section summary (i.e., first-sentence imputation) based on multimodal web- page content, including the page description, consecutive section text, images, and captions. For the discriminative task, we ELE-FASHION (Zhu et al., 2024b), prod- uct classification dataset where nodes represent products, edges capture co-purchase relationships, and text and im- ages serve as attributes. To evaluate zero-shot node clas- sification, we randomly hold out a set of unseen classes (5 out of 11 classes for the OPT-125M backbone and 9
4.3. Discussions
Ablation Studies. The performance of MM-QFormer without structural modeling is reported in Table 4.1. We ob- serve that incorporating MM-QFormer alone improves upon the baselines in most cases. Besides, we further conduct ablation experiments on both Hop-Diffused MM-QFormer
6

---

Graph4MM: Weaving Multimodal Learning with Structural Information
Table Comparison of approach with VI.Ms, Ps. and MMGL. variants on different backhones. In generative tasks, BLAU-4, ROUGE-L, and CIDEr measure language modeling accuracy (higher is better). In discriminative tasks, ROUGE-L (R-L) evaluates response accuracy, while Accuracy (Acc), Recall (Rec), and Precision (Pre) assess classification correctuess. The best results are bolded. while the best baseline results are underlined.
Generative Task
Discriminative Task
Backbone
Method
BLEU-
ROUGE-L
CIDEr
R-I.
Ace (%)
Rec (%)
(%)
Pre (%)
NODE'S TEXT (T)
0.0000
0.0496
0.0060
0. 1450
21.89
9.62
16.27
VLMS
BLIP
SUBGRAPI''S TEXT
0,0000
0.0530
0,0063
0. 1907
31.37
14.49
31.73
NODE'S TEXT
0.0000
0.1223
0.0069
0.0793
11.00
4.43
10.90
QwenVL
SUBGRAPH'S
0.1192
0.0084
0.1233
12.33
5.78
NODE'S TEXT
0.0078
0,1871
0.0783
0.2270
68.57
40.44
48,76
PLM
SUBGRAPH'S 's TEXT
0,0164
0,2198
0,1551
0.2762
67.68
33.14
40.26
NODE'S TEXT
0,0642
0,3807
0.6241
0.7047
70.33
66.67
66.67
OPT-125M
SUBGRAPH'S
0.0770
0.7606
0.8149
MMGL
NODE'S TEXT IMAGE (1)
0.0643
0,3825
0.6371
0,7180
73.90
68.67
83.33
SUBGRAPH'S TEXT & IMAGE
0.0778
0.4041
0.7712
0.8144
99.85
83.25
83.33
SUBGRAPH'S T & GNN
0,0633
0.3814
0.6326
0.5771
70.89
56,08
62.50
MM-QFORMFE
0.0769
0 0,4044
0,7684
0.8195
100.00
100.00
100.00
GraphIMM (Ours)
HOP-AWARE NO-QFORMER
0.0801
0,7736
0.8097
100.00
100.00
100.00
HoP- DIFFUSED MM- OFOMERR
0.0800
0.4076
0.7831
0.8282
100.00
100.00
100.00
NODE'S TEXT
0.0757
0,3691
0,7185
0.7081
94.76
78.07
83,11
PLM
SUBGRAPH''S TEXT
0.0960
0,3949
0,9179
0.7327
81.42
81.27
NODE'S
0.0947
0,9046
0.7083
Liama- BB
SUBGRAPH'S 's TEXT
0.1087
0.4426
1.0426
0.7375
99.84
81.51
81.58
MMGL
NODE'S TEXT & IMAGE
0.0981
0.4449
0,9367
0.6696
97.26
84.33
88.38
SUBGRAPH'S TEXT & IMAGE
0,1157
0,4685
1,1072
0.7446
98.07
84.55
85,50
SUBGRAPH'S
GNN
0.1003
0.4487
0.9631
0.3823
86.73
80
MM-QFORMER
0.1168
0,4699
1.1108
0.8321
99.88
99.86
99.87
Graph4MM (Ours)
HOP-AWARF MM QFORMER
0.1186
0.4731
1.1262
0.8891
99.87
89.86
89.88
HOP DIP-DIIFUSED MM. QFORMER
0.1
0.4713
1.1221
0.8822
100.00
100.00
100.00
and Hop-Aware MM-QFormer by individually removing structural priors from the image and text modalities. The result is shown in Table 4.3. In all cases, removing struc- tural information leads to a performance drop, reinforcing the importance of multi-hop structural information in guid- ing multimodal learning. Notably, removing the structural information in the image input leads to severer perfor- mance drop, which is because text provide structural information through explicitly added some descriptive hints (e.g., "context from I-hop neighbor") in the prompt. Such prompts, illustrated in Appendix Q, allow the language modality to retain partial access to structural information. In contrast, without structural information, images from dif- ferent hops are treated with the same level of importance, leading to suboptimal performance.
Robustness from Hop-diffused to Hop-aware Attention. Hop-Aware Attention reduces computational complexity while maintaining strong performance. Replacing either the image or text modality with Hop-Diffused Attention across four datasets (results shownin Table 3) shows that even lightweight structural priors ensure stable performance, and hybrid approaches may sometimes outperform full Hop- Diffused Attention, due to the complementary inductive biases introduced by modality-speciff structural modeling.
Table 3. Comparison of replacing text vision structural priors in Hop Coo-DDifused MM- MMM-Former with Hop-Aware Attention on the generative task using OPT-125M and LLaMA-IB Best results are in red, second-best in yellow, and third-best in grey. tor and Por denote text visual subgraphs MM- MM--Formmr, respectively.
Model
Method
BLEU-4
ROUGE-L
CIDEr
Table Ablation study on Hop- Diffused and Hop-Aware Atten- tion in the generative setting with OPT- The first row shows original results, while -tg: and -Pg' indicate the removal of graph adjacency heuristics for text and image subgraphs.
LLaMA-IB OPT-125M
Hop-Diffused (HD)
0.0800
0.4076
0.7831
Hop-Aware (HA)
0.0801
0.4063
0.7736
(g-HA + PgD
0.0822
0.4094
0.7947
(g- + -HA
0.0802
0.4063
0.7826
Hop-Diffused (HD)
0.1177
0.4713
1.1221
Ablation Variant
BLEU-4
ROUGE-L
CIDEr
Hop-Aware (HA)
0.1186
0.4731
1.1262
Hop-Diffused Diffused MM- QFormer
0.0800
0.4076
0.7831
fgr-HA + PDD
0.1176
0.4709
1.1199
-gg Hop-Diffused Attention
0.0786
0.4065
0.7765
fg -HA
0.1179
0.4744
1.1207
Po Doffuuifusedd Attention
0.0769
0.4044
0.7684
Hop-Aware MM-QFormer
0.0801
0.4063
0.7736
Should Graph be Encoded as a Modality?
for Hop-Aware Attention
0.7943
0.4047
0.7736
Pg Hop-Aware Attention
0.0769
0.4044
0.7684
A key question in multi-model graph learning is whether
7

---

Graph4MM: Weaving Multimodal Learning with Structural Information
graph structure should be explicitly encoded as an addi- tional modality. MMGL adopts straightforward approach by using a GCN to encode G. then directly adding its graph representations to the pretrained vision and language em- beddings before feeding them into the LLM. The GCN itself is trained jointly with the downstream task loss. However, as shown in the first two rows of Table 4.3, this method provides little to no improvement and often performs worse than using only raw text and vision attributes. To further investigate this, we experimented with treating graph topol- ogy as an independent modality in the generative setting with OPT-125M. We explored two designs: (1) projecting GCN-learned node embeddings into global graph tokens to represent overall graph structure, and (2) mapping them into node tokens that encode local topology at the node level. Despite these efforts, neither approach yielded meaningful performance gains. A fundamental reason for this failure is the semantic gap between graph-derived topology em- beddings and pretrained vision-language representations. Unlike sarge-scale pretrained models, which align text and image representations through extensive multimodal train- ing, GCNs operate on small, sparsely labeled subgraphs, limiting their expressiveness. To formalize this gap, we propose the following proposition:
5. Related Work
Vision-Language Models Recent vision-language mod- els leverage frozen pretrained components, either fixing the image encoder (Chen al., 2020; Li al., 2020; Zhang et al.; Zhai et al., 2022) or freezing the LLM to utilize its knowledge for et al., 2021; Alayrac et al., 2022b; Chen et al.. 2022a), When using a frozen LLM (Touvron et al., 2023; Zhang et al., 2022b), the key challenge is aligning visual features to the text space. Frozen (Tsimpoukelli et al., 2021) fine- tunes an image encoder as soft prompts for LLM,, while Flamingo (Alayrac et al., 2022b) injects visual features via trainable cross-attention layers. BLIP-2 (Li et al., 2023b) introduces learnable query-based transformers to project visual features into the L.M's space. Recent large-scale VLMs (Wang et al., 2024) employ multi-stage pretraining and multi-task finetuning to further enhance Lassonnlanguuge understanding. However, most VLMs still focus on one-to- one image-text alignment naive concatenation of multi- modal tokens for multi-image scenarios, failing to capture complex many-to-many relationships where multiple im- ages correspond to different textual fragments. We address this by extending multimodal learning to graph structures, enabling structured cross-modal interactions.
generation (T (Tsimpoukelli
Proposition 4.1. Let G be small-scale graph data distribu- tion, G ~ r()), and let X be large-scale data distribution (e.g., text or image), X ~ p(X). Suppose f((] is a GCN- based encoder trained only on G to produce representations (G), while p(x)) is a large-scale pre-trained en- coder (e.g., LLM or ViT) producing Zx- = =f{x) (x). Then, for almost all samples x E X. the mutual information satis- fies: I(2G;X) X) < I(zx; X), implying a fundamental gap in expressive power between zg and zx.
Graph Neural Networks for Multimodal Data. Hetero- geneous Graph Neural Networks (HGNNs) extend standard GNNs (Veličković et al., 2018; Wu et al., 2020) to process multimodal graphs by integrating heterogeneous node at- tributes. This is typically done by extracting embeddings with frozen encoders and fusing modalities at different lev- els: input (Schlichtkrull et al., 2018; Zhang et al., 2019), intermediate (Hu al., 2020), or final representation lay- ers (Yoon et al., 2022). While effective for node classifi- cation and link prediction, HGNNs are not well suited for language generation or open-ended reasoning. More discus- sion on Multi-modal Knowledge Graph (MMKG) can be seen in Appendix H. MMGL (Yoon et al., 2023) introduced multimodal graph learning for text generation by combining textual and visual encoders with GNNs. However, align- ing graph-
2G = - (G),
The proof is in Appendix D. Additionally, pretrained vi- sion and language models benefit from strong cross- cross-modal alignment, allowing text and image representations to be semantically consistent. In contrast, graph embeddings lack such alignment, further widening the gap. Thus, we propose use graph structure as a guide for selecting and fusing mul- timodal data rather than treating it as a separate modality.
graph-based based representations with large language models remains challenging. We address this by graph-heuristic multimodal fusion mechanism to enhance vision-language understanding.
Table 4. Comparison of different methods for incorporating topol- ogy in the generative setting using the OPT-125M backbone.
Method
BLEU-4
ROUGE-I
CIDEr
Learning on the Text-Attributed Graphs. Integrating structured graph data with unstructured text has evolved from early MLP- and transformer-basee fusion meth- ods (Feng et al., 2020; Zhang et al., 2022c; Lin et al., 2019), which struggled to capture complex contextual dependen- cies and were insufficient for language-centric tasks like open-ended QA. Recent efforts explore LLMs for graph learning (He et al., 2023; Zhu et al., 2024c; Lin et al., 2024),
Subgraph's s T & (Emb)
0.0649
0,3800
0,6370
+ Graph Embeddings
0.0633
0,3814
0.6326
Subgraph's s T&
0.0788
0.4051
0.7790
+ Graph Tokens
0.0796
0.4052
0.7736
+ Node Tokens
0.0782
0.4045
0.7651
Ours DiffuuDif))
0.0800
0.4076
0.7831
8

---

Graph4MM: Weaving Multimodal Learning with Structural Information
primarily refining graph representations within the LLM framework. GraphTranslator (Zhang et al., 2024b) aligns graph embeddings with LLMs using textual descriptions, but its dependence on explicit, comprehensive language an- notations of graph structures limits its applicability. Our approach extends multimodal graph learning by integrat- ing vision, enabling structured multimodal fusion for both language generation and graph-based reasoning.
for few-shot learning. In Koyejo, S., Mohamed, S., Agar- wal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Ad- vances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022a.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few- shot learning. Advances neural information processing systems, 35:23716 23736, 2022b.
6. Conclusion
In this paper, we propose representing the complex interactions between different modalities using a mul- timodal attributed graph and introduce a theoretically guided, adjacency-aware multimodal fusion framework, Hop-Diffused and Hop-Aware MM-QFormer. Our approach effectively enhances performance in both generative and dis- criminative tasks within multimodal learning. We also dis- cuss whether graph structure should be treated as separate modality in the era of foundation models.
Ban, Y., Zou, J., Li, Z., Qi, Y. Fu, D., Kang, J., Tong, H., and He, J. Pagerank bandits for link prediction. In NeurIPS, 2024.
Burns, A., Srinivasan, K., Ainslie, J., Brown, G., Plummer, B. A., Saenko, K., Ni, J., and Guo, M. Wikiweb2m: page-level multimodal wikipedia dataset. arXiv preprint arXiv:2305.05432 2023,
Chen, J., Guo, H., Yi, K., Li, B., and Elhoseiny, M. Visual- gpt: Data-efficient adaptation of pretrained language mod- els for image captioning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18030-18040, 2022a.
Acknowledgements
We would like to thank Ke Jiang and Xiao Lin from the University of Illinois Tibana-Chhappln
for their valuable suggestions to improve our work, as well as Jiaxuan You for the insightful discussions during the CS598: Deep Learning with Graphs course. This work is supported by National Science Foundation under Award No. 17902. The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies the government.
Chen, X., Zhang, N., Li, L., Deng, S., Tan, C., Xu, C., Huang, F., Si, L., and Chen, H. Hybrid transformer with multi-level fusion for multimodal knowledge graph completion. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pp. 904-915, 2022b.
Chen, Y-C.,, Li, L., Yu, L., EI Kholy, A., Ahmed, F., Gan Z., Cheng, Y., and Liu, J. Uniter: Universal image-text repre- sentation learning. In European conference on computer vision, pp. 104-100 Springer, 2020.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, Z., Zhang, Y., Fang, Y., Geng, Y., Guo, L.,, Chen, X., Li, Q., Zhang, W., Chen, J., Zhu, Y., et al. Knowledge graphs meet multi-modal learning: A comprehensive sur- vey. arXiv preprint arXiv:2402.0539) 2024.
References
Ai, M., Wei, T., Chen, Y., Guo, Z., and He, J. MIp fusion: Towards efficient fine-tuning of dense and mixture-of- experts language models. arXiv e-prints, pp. arXiv-2307, 2023.
Feng, Y., Chen, X., Lin, B. Y., Wang, P., Yan, J., and Ren, X. Scalable multi-hop relational reasoning for knowledge-awarc question answering. arXiv preprint arXiv:2005. 00066, 2020.
Ferrada, S., Bustos, B., and Hogan, A. Imgpedia: a linked dataset with content-based analysis of wikimedia images. The semantic web-ISWC 2017: 16th international semantic web conference, vienna, Austria, October 21-25, 2017, proceedings, part II 16, pp. 84-93. Springer, 2017.
Alayrac, J., Donahue, J., Lue, P., Miech, A., Barr, I., Has- son, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O.,, Zisserman, A., and Simonyan, K. Flamingo: visual language model
Fu, D. and Hc, J. SDG: A simplified and dynamic graph neural network. In SIGIR, 2021.

---

Graph4MM: Weaving Multimodal Learning with Structural Information
Fu, D., Fang, L., Maciejewski, R., Torvik, V. I., and He, J. Meta-learned metries over multi-evolution temporal graphs. In KDD, 2022.
and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023b.
Li, J., Li, D., Savarese, S., and Hoi, S. C. H. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Krause, A., Brunskill, E.,, Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Ma- chine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. PMLR, 2023c.
Fu, D., Hua, Z., Xie, Y., Fang, J., Zhang. S., Sancak, K., Wu, H., Malevich, A., He, J., and Long, B. yer-graphormer: A mini-batch graph transformer via virtual connections. In ICLR, 2024.
Gao, J., Li, P., Chen, Z.., and Zhang, J. A survey on deep learning for multimodal data fusion. Neural Computation, 32(5):829-866 -64,,,966,226 2020.
Li, T. W., Mci, Q., and Ma, J. A metadata-driven approach to understand graph neural networks. Advances in Neu- ral Information Processing Systems, 36:15320-15340, 2023d.
Guo, J., Li, J., Li, D., Tiong, A. M. Li, B., Tao, D., and Hoi, S. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10867-10877, June 2023.
Li, T.-W., Qiu, R., and Tong, H. Model-free graph data selection under distribution shift, 2025a. URL https: / //arxiv. l/axxiv.ogg/bbb. hbwgg//ba//555 200.//555.111 17993.
He, X., Bresson, X., Laurent, T., Perold, A., LeCun, Y., and Hooi, B. Harnessing explanations: Lim-to-Im interpreter for enhanced aext-attributed graph representation learning. arXiv preprint ar arXiv:2305. /9t/3055./99..,. 19523, 2023.
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al. Oscar: Object- semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16. pp. 121-137. Springer, 2020.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint (Xiv:2/06.00688 2021.
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 2021.
Hu, Z., Dong, Y., Wang, K., and Sun, Y. Heterogeneous graph transformer. In Proceedings of the web conference 2020, pp. 2704-2710, 2020.
Li, Z., Fu, D., and He, J. Everything evolves in personalized pagerank. In wwW, 2023e.
Jabeen, S., Li, X., Shoib, A. M., Omar, B., Li, S., and Jabbar, A. A review on methods and applications in multimodal deep learning. ACM Trans. Multim. Comput. Commun. Appl.,
Li, Z., Zheng, L., Jin, B., Fu, D., Jing, B., Ban, Y., He, J., and Han, J. Can graph neural networks learn language with extremely weak text supervision? arXiv preprint 2024.
19(25):76:1-76::
2023. doi: 10. 145/3545572.
arXiv:2412.0855
Jin, B., Pang, Z. Guo, B., Wang, Y.-X., You, J., and Han, Instructg2i: Synthesizing images from multimodal at- tributed graphs. arXiv preprint arXiv:2410.07157, 2024.
Li, Z., Fu, D., Ai, M., and He, J. Apex2: Adaptive and ex- treme summarization for personalized knowledge graphs. In KDD, 2025b.
Koh, J. Y., Fried, D., and Salakhutdinov, R. Generating images with multimodal language models. In Oh, A., Naumann, T., Globerson, A., Saenko. K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural in- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
Lin, B. Y., Chen, X., Chen, J., and Ren, X. Kagnet: Knowledge-awer aware graph networks for commonsense rea- soning. arXiv preprint arXiv: (00::9990000000 02 0510 2019.
Lin, J., Qian, K., Han, H., Choudhary, N., Wei, T., Wang, Z., Gene, S., Huang, E. W., Wang, S., Subbian, K., et al. Unleashing the power of Ilms as multi-modal en- coders for text and graph-structured data, arXiv preprint arXiv:2410. 1235, 2024.
Li, G., Koutra, D., and Yan, Size generalization of graph neural networks on biological data: Insights and practices from the spectral perspective. arXiv preprint 37511 2023a.
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A. Y. Multimodal deep learning. In Getoor, L. and Scheffer, T. (eds.), Proceedings of the 28th International Conference on Machine Learning, ICML. 201 I, Bellevue,
Li, Li, D., Savarese, S., and Hoi, S. 2lip- Bootstrapping iinguage-image pre-training with frozen image encoders
10

---

Graph4MM: Weaving Multimodal Learning with Structural Information
Washington, USA, June 28 July 2, 2011, pp. 689-696. Omnipress, 2011.
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. Graph attention networks. arXiv preprint arXiv: 1et...100..00. 2018.
Ni, J., Li, and McAuley, J. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 conference on empirical meth- ods natural language processing and the 9th interna- tional joint conference on natural language processing (EMNLP-IJCNLP), pp- 188-197, 2019.
Wang. G., Ying, R., Huang, J., and Leskovec, J. Multi- hop attention graph neural network. arXiv preprint arXiv:2009.4433 2020.
Wang. L.., Hassani, K., Zhang, S., Fu, D., Yuan, B., Cong, W., Hua, Z., Wu, H., Yao, N., and Long, B. Learning graph quantized tokenizers. In ICLR, 2025.
Ning, X., Xu, W., Wei, T., and Liu, X. vae: Interest in- formation augmentation with variational regularizers for cross-domain sequential recommendation. arXiv preprint arXiv:2405.207/4 2024.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing lasoun-anguuage model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., et al. Emergent abilities of large language models. 2022a.
arXiv preprint arXiv:2206.0700
Schlichtkrull, M., Kipf, T. N., Bloem, P., Van Den Berg, R., Titov, I,, and Welling, M. Modeling relational data with graph convolutional networks. In The semantic web: 15th international conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, proceedings 15, pp. 593-607. Springer, 2018.
Wei, T., You, Y., Chen, T., Shen. Y., He, J., and Wang. Z. Augmentations in hypergraph contrastive learning: Fab- ricated and generative. Advances in neural information processing systems, 35:1909-1922, 2022b.
Wei, T., Jin, B., Li, R., Zeng, H., Wang. Z., Sun, J., Yin, Q., Lu, H., Wang. S., He, J., et al. Towards unified multi- modal personalization: Large vision- vision-language models for generative recommendation and beyond. arXiv preprint arXiv:2403. 10667, 2024.
Srinivasan, K., Raman, K., Chen, J., Bendersky, M., and Najork, M. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceed- ings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp. 2443-2449, 2021.
Wu, X., Ajorlou, A., Wu, Z., and Jadbabaic, A. Demys- tifying oversmoothing in attention-basee graph neural networks. Advances in Neural Information Processing Systems, 36:35084 (6:35084-35106 2023.
Stahlschmidt, R., Ulfenborg, B., and Synnergren, J. Multi- modal deep learning for biomedical data fusion: a review. Briefings in Bioinformatics, 23(2) 23b2)bbba 2022.
Wu, Z., Pan, S., Chen, F., Long, G., Zhang. C., and Philip, S. Y. A comprehensive survey on graph neural networks.
IEEE transactions on neural networks and learning sys- tems, 32(1):2 24, 2020.
Summaira, J., Li, X., Shoib, A. M., Li, S., and Abdul, J. Recent advances and trends in multimodal deep learning: A review. arXiv preprint arXiv:2105.1 1087, 2021.
Xu, D., Xu, T., Wu, S., Zhou, J., and Chen, E. Relation- enhanced negative sampling for multimodal knowledge graph completion. In Proceedings of the 30th ACM in- ternational conference on multimedia, pp. 3857-3866.
Tieu, K., Fu, D., Wu, J., and He, J. Invariant link selector problem. In The 28th International Conference on Artificial Intelligence and Statistics, 2025.
for spatial-temporal
2022.
Xu, W., Li, S., Ha, M., Guo, X., Ma. Q. Liu, X., Chen, L., and Zhu, Z. Neural node matching for multi-targe cross domain recommendation. In 2023 IEEE 39th In- ternational Conference on Data Engineering (ICDE), pp. 2154-2166. IEEE, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13977 2023.
Tsimpoukelli, M., Menick, J. L., Cabi, S., Eslami, S., Vinyals, O., and Hill, F. Multimodal few-shot learn- ing with frozen language models. Advances in Neural Information Processing Systems, 34:200-213 212 2021.
Xu, W. Liang, Z., Han, J., Ning, X., Lin, W., Chen, L., Wei, F., and Zhang, Y. Slmrec: empowering small language models for sequential recommendation. arXiv e-prints, pp. arXiv-2405, 2024a.

---

Graph4MM: Weaving Multimodal Learning with Structural Information
Xu, W., Ning, X., Lin, W. Ha, M., Ma, Q. Liang. Q.. Tao, X., Chen, L., Han, B., and Luo, M. Towards open-world cross-domain sequential recommendation: A moodel-agnostic contrastive denoising approach. In Joint European Conference on Machine Learning and Knowl- edge Discovery in Databases, pp. 161-179. Springer, 2024b.
Zhang, M., Sun, M., Wang, P., Fan, S., Mo, Y., Xu, X., Liu, H., Yang, C., and Shi, C. Graphtranslator: Aligning graph model to large language model for open-ended tasks. In Proceedings of the ACM on Web Conference 2024, pp. 1003-1014, 2024b.
Zhang, N., Li, L.., Chen, X., Liang, X., Deng, S., and Chen, H. Multimodal analogical reasoning over knowledge graphs. In The Eleventh International Conference on Learning Representations, 2022a.
Xu, Z., Hassani, K., Zhang, S., Zeng, H., Yasunaga, M., Wang, L., Fu, D., Yao, N., Long, B., and Tong, H. Lan- guage models are graph learners. CoRR, 2024c.
Zhang, P., Li, X., Hu, X., Yang. J., Zhang, L., Wang. L., Choi, Y., and Gao, J. Vinvl: Making visual representa- tions matter in vision-language models. arxiv 2021. arXiv preprint 2:Xi::211.0055::
Yasunaga, M. Aghajanyan, A., Shi, W., James, R., Leskovec, J., Liang, P., Lewis, M., Zettlemoyer, L., and Yih. W.-t. Retrieval-augmented multimodal language modeling. arXiv preprint 2r 12561, 2022.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint www.v:2205.000666 2022b.
Yoon, M., Palowitch, J., Zelle, D., Hu, Z., Salakhutdinov, R., and Perozzi, B. Zero-shot transfer learning within heterogeneous graph via knowledge transfer networks. Advances in Neural Information Processing Systems, 35: 27347-27359, 2022.
Zhang, X., Bosselut, A., Yasunaga, M., Ren, H., Liang, P., Manning, C. D., and Leskovec, J. Greaselm: Graph rea- soning enhanced language models for question answering. 2022c.
Yoon, M., Koh, J. Y., Hooi, B., and Salakhutdinov, R. Multi- modal graph learning for generative tasks. arXiv preprint arXiv:2310.074:: 2023.
arXiv preprint arXiv:2201.08888
Zhang, Y., Chen, Z., Guo, L., Xu, Y., Hu, B., Liu, Z., Zhang, W., and Chen, H. Mixture of modality knowledge ex- perts for robust multi-modal knowledge graph completion. CoRR, abs/2405. 16869, 2024c.
Yuan. L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang. Z.- H., Tay, F. E., Feng, J., and Yan, S. mokenssto-tooker vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 558-567, October 2021.
Zhang, Y., Li. S., Qian, C., Liu, J., Yu. P., Han, C., Fung, Y. R.,, McKeown, K., Zhai, C., Li, M., et al. The law of knowledge overshadowing: Towards understanding, pre- dicting, and preventing Ilm hallucination. arXiv preprint arXiv:2502.1614 2025.
Zeng, Z., Zhang, S., Xia, Y., and Tong, H. Parrot: Position- aware regularized optimal transport for network align- ment. In Proceedings of the ACM Web Conference 2023, pp. 372-382, 2023a.
Zeng. Z., Zhu, R., Xia, Y., Zeng, H., and Tong, H. Gen- erative graph dictionary learning. In International Con- ference on Machine Learning, pp. 40749-40769, PMLR, 2023b.
Zheng, I.., Jing, B., Li, 7., Zeng, Z.,, Wei, T., Ai, M., He, X., Liu, L., Fu, D., You, J., et al. Pyg-ssl: A graph self-supervised learning toolkit. arXiv preprint arXi::4412.21151, 2024.
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., and Beyer, L.. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18123-18133, 2022.
Zheng, S., Wang, W., Qu, J., Yin, H., Chen, W., and Zhao, L. Mmkgr: Multi-hop multi-modal knowledge graph reasoning. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pp. 96-109. IEEE, 2023.
Zhu, J., Li, G., Yang, Y.-A., Zhu, J., Cui, X., and Koutra, D. On the impact of feature heterophily link pre- diction with graph neural networks. arXiv preprint arXiv:2409, 17475, 2024a.
Zhang, C., Song, D., Huang, C., Swami, A., and Chawla, N. V. Heterogeneous graph neural network. In Proceed- ings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pp. 793-803 2019.
Zhu, J., Zhou, Y., Qian, S., He, Z., Zhao, T., Shah, N., and Koutra, D. Multimodal graph benchmark. arXiv preprint arXiv:2406. 2024b.
Zhang, J., Huang, J., Jin, S., and Lu, S. Vision-language models for vision tasks: A survey. IEEE Trans. Pattern Anal. Mach. Intell,, 46(8):56255564) 2024a. doi: 10. 1109/TPAMI. 2024.336969..
Zhu, Q., Zheng, D., Song. X., Zhang, S., Jin, B., Sun, Y., and Karypis, G. Parameter-efficien tuning large language
12

---

Graph4MM: Weaving Multimodal Learning with Structural Information
models for graph representation learning. arXiv preprinl arXiv:2404. 18271, 2024c.
Zhu, X., Li, Z., Wang, X., Jiang, X., Sun, P., Wang, X., Xiao, Y., and Yuan, N. J. Multi-modal knowledge graph con- struction and application: A survey. IEEE Transactions Knowledge and Data Engineering, 36(2):715-735, 2022.
13

---

Graph4MM: Weaving Multimodal Learning with Structural Information
Appendix Content
Appendix A: A Remains the Property of Valid Attention Matrix
Appendix B: Global Connectivity Aware by Hop-diffused Attention
Appendix C:
Hop-Diffused Attention is Less Prone to Over-Smoothing than GAT
C..1. Measuring Over-Smoothing: Dirichlet Energy
C.2. Theoretical Analysis
Appendix D: Mutual Information Gap Between Small-Scale Graph Encoders and Large-Scale Pretrained Models
Appendix E:
Graph Modeling
Appendix F: Dataset Details
F.I. WikiWeb2M
F.2. Ele-Fashion
Appendix G: Detailed Baselines
Appendix H: Related Work about Multi-Modal Knowledge Graphs
Appendix I: Implementation Details
Appendix J: Modeling Graph as a Standalone Modality on Discriminative Setting
Appendix K: Impact of the Graph Density
Appendix Simulation Study on Attention Diffusion vs GAT under Small K
Appendix M: Performance Comparison under Different Random Seeds
Appendix N: Performance Comparison with Link Prediction Setting
Appendix O: Validation Accuracy Curve of OPT-125M During Training Discriminative Task
Appendix Accuracy Evolution Zero-Shot Node Classification with Different Number of Unseen Classes
Appendix Q: Example Prompt and Output
- Q.1. Example Prompt and Output in Generative Setting on WikiWeb2M Dataset
-
Q.2. Example Prompt and Output in Discriminative Setting Ele-Fashion Dataset
14

---

Graph4MM: Weaving Multimodal Learning with Structural Information
A. A Remains the Property of Valid Attention Matrix
Proposition A.1. The diffused attention matrix A= Ei=o 0,A' where A is the i-th power of the base attention matrix A, and 8 a set of positive weights satisfying Ei= 0, remains valid attention matrix with each row summing to i.e. A A. Vi.
Proof. Since A is obtained by row-wise softmax normalization, it satisfies preserves row-wise summation in stochastic matrices, any power A' satisfies Σ, (A')ss = 1 for all 2 0. The weight
Σj Ag 1. Since matrix multiplication
coefficients 0 form probability distribution with Ei=o 0: = 1, ensuring that when computing row sums in A, we have:
Σイ 0,(A') )if
Σ 0: O. 1=0
20.1 1=0
(14)
==
Thus, A is row-stochastic, meaning each row sums to 1. This confirms that the hop-diffused attention mechanism essentially adjusts the attention scores based on global connectivity at different hop distances, maintaining the fundamental property of an attention matrix.
B. Global Connectivity Aware by Hop-diffused Attention
Proposition B.1. Hop-Diffused Atlention is generalized form of Personalized PageRank (PPR) with an adaptive multi-hop weighting scheme. Given an attention transition matrix Agm, its diffusion process follww::
A=
Σ
0; 0:A oar
E0==1,
0, > 0.
(15)
For 0 a11 a)', this recovers the standard PPR fPrrommuattion.
Proof. Personalized PageRank is defined as:
App al (1-a)Aua)
(16)
Expanding the inverse as a Neumann series, we obtain:
Appr
a
Σ (1 a)' A
(17)
Compared with the Hop-Diffused Attention formulation,
A = Σ 0,43 am k=
(18)
where 0, is probability distribution with Σi= 0, 1, we see that PPR is special case where 0, a(1 - a)'. This proposition further illustrates that akin to personalized PageRank, Hop-Diffused Attention preserves the global graph structure while adaptively regulating the influence of multi-hop neighbors, where the impact of more distant neighbors decays exponentially.
C. Hop-Diffused Attention is Less Prone to Over-Smoothing than GAT
C.1. Measuring Over-Smoothing: Dirichlet Energy
Definition C.1 (Dirichlet Energy). Let G (V,E) be a graph with node set y and edge set E. Let X R/ denote the node feature matrix at the n-th layer of GNN. The Dirichlet energy, which quantifies the smoothness of node representations, is defined as:
E(X")
1
Σ
Σ
X X7
(19)
M
JEM;
where Mr denotes the neighborhood of node i. a GNN exhibits sooothin,,,, then node representations converge to similar values across the graph, implying that {(X)) + 0.
15

---

Graph4MM: Weaving Multimodal Learning with Structural Information
C.2. Theoretical Analysis
Proposition C.2. Let x be the node representation matrix after the Hop-Diffused Attention module, and (k) be the representation the kth layer of GAT. When both methods aggregate information from k- k-hop neighbors. Hop-Diffused Attention retains higher Dirichlet energy. preserving feature variance and mitigating over-smoothing. Formally, for large h,
Hiop-Difjusod X x(1) > EGAT( (x))).
(20)
Proof. In GAT, the updated rule for the 1th graph attention layer can be written as:
x = (A (4)))
(21)
where Ant is the learned aggregation matrix, W is trainable transformation matrix, and a is point-wise nonlinear activation function. Expanding recursively for layers, we obtain:
((k) or (A a (A (A O ((0)) ... W(K ())()))
(22)
To handle the nonlinearity a, we assume the activation function satisfies < a(a) s for x + O, and o
d'(0) if (0) is undefined, refering Wu et al. (2023). This holds for common activations such as ReLU and Lea Under this assumption, any activation (y) on vector y can be rewritten as:
(yy) diag (o())
(23)
y
We denote Dl as the diagonal transformation matrix induced by the activation at layer k. satisfying
diag(O) = D(!)) = diag(1).
(24)
The representation x! (1+11 can be recursively expanded as:
((++1)=
Σ J611
w
((0)) A(a) ... D. (0)) x(0) x -j0
Jess
(25)
1=0
())
Here, ((k) is a row-stochastic aggregation matrix. Since all ()k) and (1) have spectral norm less than or equal 1, and each weight matrix ((k) is bounded, it follows that:
| S ch |
(26)
for some constant ( < c < 1. The Dirichlet energy is given by:
(xxT(X = 1 Tr (x(ITIXx(())
(27)
Let Amax be the largest eigenvalue of the Laplacian L for (R).
Tr(z LZ) & Amax Tr(Z' Z) = Amax 1|212..
(28)
Applying this to Eq. (27):
Amax 11)
(x)) <
(29)
n
Because || | s
with r rank (x() S d, we have
Amax r 2h (x000
(x)(()
(30)
16

---

Graph4MM: Weaving Multimodal Learning with Structural Information
Letting c² (0, 1) and absorbing constants, we arrive at:
(x - - )()(
(31)
indicating exponential decay of Dirichlet energy. Hence, + oo, the energy tends zero:
lim EGAT (1)) =
(32)
k-+00
which reflects the over-smoothing effect in deep GATs.
Hop-Diffused Attention Analysis For Hop-Diffused Attention,
weighted of attention powers:
K A = Σ 0,4
(33)
am
100
This modifies the feature propagation as:
X = AA (0) W.
(34)
Applying Dirichlet Energy to the Hop-Diffuse Attention Propagation:
K XW,
Hoop-iiftused (x)) = ΣE NEV PEA,
Σ
0 A
Σ 0m XW
(35)
MV
Using Jensen's 's inequality and the convexity of the squared norm, approximate:
K
&Hop.Diffuueed (X*) S X)
(36)
Using the eigenvalue decomposition of A.u, we approximate:
E(A X {()).
(37)
Thus, the energy formulation simplifies to:
Hop Diffisee (X = Σ 0: {(X).
(38)
Since Σ 0,Ainax remains larger than ** for sufficiently large k; it follows that:
Etop Diffused (x1) (x*)
(39)
Thus, Hop-Diffused Attention retains higher feature heterogeneity, making it less prone to over-smoothing compared to directly applying GAT, We also provide a simulation study comparing Dirichlet energy under small values of in Appendix L.
Mutual Information Gap Between Small-Scale Graph Encoders and Large-Scale Pretrained Models
Proposition D.1. Let be a small-scale graph data distribution, r(g), and let X be a large-scale data distribution
(e.g., text or image), X ~ p(2)). Suppose ((() is a GCN-based encoder trained only on G to produce representations
ZG = f6 (G) (G) (G), while ,(X) is a large-scale pre-trained encoder (e.g., LLM or ViT) producing zx = f(xx (X). Then, for almost all samples x E X, the mutual information satisfies
T(zc: X) & T(zx; X);
(40)
which implies a fundamental gap in expressive power between ag and zx.
17

---

Graph4MM: Weaving Multimodal Learning with Structural Information
Proof. Let G denote graph-structured inputs drawn from small-scale distribution r(9), and let X denote data (text, images, etc.) from a large-scale distribution p(2). We use &g = ./ (G) to denote the representation learned by a GCN on G, and zx ((x) (X) to denote the representation learned by a large-scale pre-trained model X.
Let zg have dimension d. By Shannon's definition of mutual information,
I(za; G) H((a) (zz G).
(41)
For deterministic encoder (()), the conditional entropy H G) is small, so [(zo; G) s (zc).
(42)
Since the dataset De is small, we have
H(z)) S log| Da
(43)
Thus,
I(ze: < log Del.
(44)
implying strict upper bound on the captured information.
Consider zx f/x (xxx)(x)) (X) obtained via pre-training on X. With large data coverage [Dx] and sufficiently expressive
model, the entropy H (zx) can much higher:
H(ax) ae log Dx (where |Dx] >> [Da))
(45)
Hence,
I(ax;X) = H(xx) H(zx (X) = log Dx
(46)
which is significantly larger than log/ Dol in most practical scenarios.
When aligning zg and zx in a shared semantic space, note that
I(zci S G( + X(G;X).
(47)
Since [ G) S Log[Del and X)) typically small (the graph domain is narrow or quite distinct from X), follows that
I(zg: X)
I(zx;
(48)
The above inequalities establish that representations zg learned only on small-scale graph dataset G generally exhibit far lower mutual information with broader data X E X. This accounts for the substantial expressiveness gap and difficulty in aligning zg with z in a common semantic space.
18

---

Graph4MM: Weaving Multimodal Learning with Structural Information
E. Graph Modeling
Defining Multi-modal Data Connectivity. In real-world scenarios, multimodal data often interact in intricate ways. For instance, in document understanding, integrating both text (e.g., section content or page titles) and images (Yasunaga al., 2022) enables language models better comprehend the context and improve QA performance. In e-commerce, user interactions with items, represented as text descriptions and images, often reveal implicit correlations that (Wei et al., 2024), when effectively modeled, enhance recommendation accuracy. Similarly, in biomedicine, understanding interactions between molecular compositions of proteins and their corresponding visual structures can aid in predicting new protein attributes. The challenge lies in organizing and preserving these complex relationships between multimodal data, which are often interdependent and hierarchical. To address this, we propose a graph-base framework that captures these connections explicitly. In the following sections, we outline two key approaches for defining and identifying such multimodal relationships:
(a) When multimodal information exhibits intuitive connectivity in its original form, we leverage this natural structure. For instance, in multimodal documents, closer elements (e.g,, adjacent paragraphs or text and images within the same section) are more relevant. Accordingly, we construct content graphs based on paragraph relationships, text-image links, and image-caption associations.
(b) When no predefined connectivity exists, relevance is task-specific. For example, in e-commerce platforms, graphs are constructed from user co-elick patterns, where frequently co-clicked items indicate higher relevance. Each item is associated with textual descriptions and corresponding images.
F. Dataset Details
E1. WikiWeb2M
WikiWeb2M (Burns et al., 2023) is a multimodal webpage dataset extending WIT (Srinivasan et al., 2021) by incorpo- rating full-page structural metadata, section text, images, captions, and hierarchical relationships. We adopt the section summarization task, where the model generates missing first sentence for section given its text, images, and multimodal context from related sections on the same page, which follows the data processing process in MMGL (Yoon et al., 2023). Each section and the overall page description represented nodes, with text and images as attributes. Edges capture the relationships between consecutive sections, links between sections and their images and captions, caption-to-sectio connections, and links between page descriptions and each section. Due to storage constraints, we randomly sample 10K Wikipedia pages, resulting in 13,539 section summary samples for training and 1,768 for testing.
F.2 Ele-Fashion
Ele- fieFaashoon (Zhu et al.. 2024b) is multimodal node classification dataset constructed from Caazzon-Fashion (Ni et al., 2019), designed for evaluating graph-aware multimodal learning. Each node represents a product, with textual and visual attributes derived from product titles and images. Edges encode co-purchase relationships between products. The classification task aims to predict product categories using both multimodal attributes and graph structure. Ele-Fashion consists of 11 product categories, 97,766 nodes, and 199,602 edges, with an average node degree of 4.08 and an edge homophily score of 0.7675. Each node has textual description, and most nodes are also associated with image. For the experiments with OPT-125M backbone, we held out 5 unseen classes for evaluation: (3.4.7.9,11} and due to the strong reasoning ability of LLaMA-1B, we enhance the task difficulty by holding out 9 unseen classes for evaluation: (0, 9, 6, 10, 1, 11,
G. Detailed Baselines
We compare various transformer-based pretrained language models (PLMs), vision-language models (VLMs), and MMAG learning variants from MMGL (Yoon al., 2023).
For PLMM, we evaluate two backbones: (a) OPT-125M (Zhang et al., 2022b), as used in MMGL; (b) LLaMA-IB (Touvron et al., 2023), a IB-parameter model from the latest LLaMA 3.2 family, chosen as a computationally feasible representative of large language models.
For VLMs, we compare (a) BLIP2-OPT-2.7B (Li et al., 2023b), which employs query- dision-languae alignment
19

---

Graph4MM: Weaving Multimodal Learning with Structural Information
but lacks structured multimodal context modeling, and (b) Qwenn-V1--BB-nstnnnn (Wang al., 2024), state-of-the-art open-source VLM supporting multi-image input.
For each PLM backbone, we evaluate:
NODE'S s TEXT (T): Input single-node text attributes to a frozen PLM.
SUBGRAPH'S T: Input text attributes of node itself and of other nodes in Gz-
For VLM backbones, we compare inference performance on the same two settings PLM. Fine-tuned VLMs are excluded due to the significant scale difference from LLMs.
For MMGL(Yoon et al., 2023), we evaluate its various proposed settings:
NODE'S TEXT: Fine-tune the PLM using single-node text attributes with parameter efficient tuning (PEFT), including prefix tuning (Li & Liang, 2021) 1) or lora (Hu et al., 2021).
SUBGRAPH'S TEXT: Fine-tune the PLM with text attributes of subgraph nodes.
NODE'S s TEXT& & IMAGE (I): Input text and image attributes of a single node. Visual features are mapped to tokens via frozen encoder and linear projection. The PLM is fine-tuned with PEFT.
SUBGRAPH'S TEXT & IMAGE: Extend Node's T & to include text and image attributes of neighboring nodes in G. and Gp.
SUBGRAPH's 's T& I + GNN: Map text and visual features to embeddings, process with a GCN, and add graph embeddings the PLM's token embeddings.
H. Related Work about Multi-Modal Knowledge Graphs
Multimodal Knowledge Graphs (MMKG) integrate structured relational triples (Li et al., 2025b; 2023d; Zeng et al., 2023a)
in graph learning (Fu & He, 2021; Fu et al., 2022; Xu et al., 2023; Zeng et al., 2023b; Zheng et al., 2024; Li et al., 2024;
Wei al., 2022b; Zhu al., 2024a; Li et al., 2023a; Fu et al., 2024; Xu et al., 2024c; Wang et al., 2025; Li et al., 2025a)
with auxiliary modalities such as images, text, numerical attributes (Chen et al., 2024). Early efforts focused on MMKG
dataset construction (Zhu et al., 2022), enriching entities in traditional KGs (e-g., Freebase, DBpedia (Ferrada al., 2017))
with visual features to support tasks like link prediction (Ban al., 2024; Tieu al., 2025) entity matching (Li et al., 2023e). Recent work expands to multimodal knowledge graph completion (Chen et al., 2022b; Xu et al., 2022), cross-modal reasoning (Zheng et al., 2023; Zhang et al., 2022a), and adaptive entity representation via modality-aware fusion (Zhang et al., 2024c). For example, MOMOK (Zhang et al., 2024c) proposes a mixture-of-experts model (Ai et al., 2023) that dynamically attends to different modalities during embedding learning, while MarT (Zhang et al., 2022a) tackles analogical reasoning by aligning multimodal relational patterns. These methods largely operate at the entity or triple level, using contrastive or attention-based fusion (Xu al., 2024b) enhance representation learning, thus lacking the capacity support both generative (Xu et al., 2024a; Zhang al., 2025; Wei et al., 2024; Ning et al., 2024) and discriminative tasks simultaneously.
I. Implementation Details
Subgraph Extraction. Different strategies are used based on graph density. In WikiWeb2M, where edges and images are sparse, we follow MMGL's approach, selecting 10 textual neighbors and 4 image neighbors, along with the target node's attributes. In contrast, Ele-Fashion has denser structure with abundant edges. To balance structure retention and complexity, we keep all first-hop neighbors and randomly sample two second-hop neighbors, capping textual and visual subgraphs at 11 nodes (target + 10 neighbors).
PLM Input Length. For single-node input, we set 512 tokens for generative tasks and 128 tokens for discriminative tasks. When including subgraph context, limits increase to 1024 tokens (generative) and 512 tokens (discriminative) to accommodate richer information.
20

---

Graph4MM: Weaving Multimodal Learning with Structural Information
Parameter-Efficien Tuning. We apply Prefix-Tuning for OPT-125M and LoRA for LLaMA-IB, optimizing adaptability while ensuring efficient updates. The syper-parameters of PEFT, including the number of prefix tokens (20) and the row-rank(64) of lora, keeps the same for all baselines and our method for fair comparison. The vision encoder is CLIP. All experiments were conducted on computing nodes equipped with 2 NVIDIA A100 or 2 NVIDIA Ada A6000 GPUs.
Discriminative Task Setup. The classification task involves 11 classes with different zero-shot settings per backbone. For OPT-125M, we train on 6 classes and infer zero-shot on 5. For LLaMA-IB. IB,, given its superior generalization, we train on only 2 classes and evaluate on the remaining testing its zero-shot adaptability. We match the generated response the closest class label using SequenceMatcher, fuzzy string matching algorithm that measures textual similarity. The vision encoder is CLIP. All experiments were conducted computing nodes equipped with 2 NVIDIA A100 2 NVIDIA Ada A6000 GPUs.
hyper-parameete Setting. The key hyperparameter setting is shown in Table
Table 5. Hyperparameter settings for generative and discriminative tasks.
Hyperparameter
Generative
Discriminative
Learning Rate
x 10
1 x 10
Max Input Length
1024
512
Max Output Length
128
32
Text Neighbor Sampling
11
11
Image Neighbor Sampling
11
Batch Size (per device)
Gradient Accumulation Steps
16
Visual Tokens per Image
LoRA Rank
64
64
Prefix Tuning Virtual Tokens
20
20
Attention Diffusion Steps
Number of MM-QFormer Block
Attention Diffusion a
0.1
0.1
Number of Attention Heads
Training Epochs (OPT-125M)
Training Epochs (LLaMA-IB)
J. Modeling Graph as a Standalone Modality on Discriminative Setting
To further evaluate the role of graph independent modality, conduct additional experiments node classification using the OPT-125M backbone. Following the generative protocol described in Section 4.3, we feed GCN-derived node embeddings into the LLM as soft prompts. in a manner consistent with visual and textual tokens. This allows a fair comparison between graph-only modeling and our proposed fusion strategy.
Table 6. Performance of modeling graph as standalone modality in the discriminative setting.
Method
ROUGE-L
Accuracy
Recall
Precision
Subgraph's ' T&I
0.8144
99.85
83.25
83.33
+Graph Token
0.6648
99.96
89.91
89.98
+Node Token
0.6892
99.53
88.78
89.98
Ours (Hop-Diffused)
0.8282
100.00
100.00
100.00
These results reinforce our finding that directly modeling the graph as separate modality brings limited gains. This may be due to semantic mismatches between GNN-derived embeddings and pretrained lasion-language features. Our Hop-Diffused approach significantly outperforms the others, providing further evidence for the advantage of structure-aware generation as stated in Proposition 4.1.
21

---

Graph4MM: Weaving Multimodal Learning with Structural Information
K. Impact of the Graph Density
We analyze the impact of graph density under the generative setting by varying the number of (text, vision) neighbors from sparse (5,2) dense (15,8). Table shows that our model exhibits stable performance across varying densities and benefits from richer structural context, with consistent improvements in generation metrics.
Table 7. Impact of graph density Graph4MM the generative setting with OPT-125M backbone.
Density
BLEU-4
ROUGE-L
CIDEr
Sparse (5,2)
0.0788
0,4049
0.7795
Medium (11,5)
0.0800
0.4076
0.7831
Dense (15,8)
0.0809
0.4086
0.8018
L. Simulation Study on Attention Diffusion vs GAT under Small K
To examine over-smoothing at small diffusion depths, we conduct a simulation study using the Cora dataset. We measure the Dirichlet energy of node embeddings from both Hop-Diffussee Attention and GAT over = to 4. As shown in Table Hop-Diffused Attention preserves energy better, especially at small K, justifying our default choice of
Table 8. Dirichlet energy comparison across diffusion layers.
Method
K
K
K = 2
K
= 3
K = 4
Hop-Diffused Attention
3.2445
3,0105
2.9504
2.8360
2.7843
GAT
3.2445
0.9775
1.1345
0.6129
0.5448
M. Performance Comparison under Different Random Seeds
We evaluate model stability under three random seeds in the generative setting. Table 9 compares Graph4MM with MMGL and reports mean + standard deviation for each metric.
Table 9. Performance under different random seeds in the generative setting.
Method
BLEU-4
ROUGE-L
CIDEr
Graph4MM
0.07991 + 0.00066
0.40725 + 0.00028
0.78904 # 0.00305
MMGL
0.07762 + 0.00059
0.40504 + 0.00076
0.76907 # 0.00286
N. Performance Comparison with Link Prediction Setting
To further assess generalization, we introduce new link prediction task on the Amazon-Sports dataset from MM-Graph Benchmark (Zhu al., 2024b). We sample 10k positive and negative node pairs and use an 8:1:1 train/val/test split, The OPT-125M backbone and hyperparameters follow our node classification setup. Table 10 shows that Graph4MM significantly outperforms all baselines, with an average gain of 7.7% over the second-best method.
Table 10. Link prediction performance on the Amazon-Sport Sports datasct with OPT-125M backbone.
Method
ROUGE-L
Accuracy
Recall
Precision
(PLM) Node's Text
0.1563
52.35
51.87
72.67
(PLM) Node's Subgraph
0.2871
56.81
56.26
74.56
(MMGL) Node's Text & Image
0.5603
93.92
93.86
95.51
(MMGL) Subgraph's Text & Image
0.7352
95.46
95.44
94.35
(Graph4MM) Hop-Diffused MM- For-rrrmer
0.8904
99.98
99.97
99.86
22

---

Graph4MM: Weaving Multimodal Learning with Structural Information
0. Validation Accuracy Curve of OPT-125M During Training Discriminative Task
1.0
1.0
0.8
0.8
0.6
0.4
0.4
0.2
0.2
Epoch Number
Epoch Number
(a) Hop-Diffised ROUGE Accuracy
(b) Hop-Diffused Recall-Precision
1.0
< Aalary
0.8
0.6
0.4
0.2
2
0
2
3
Epoch Number (c) Hop-Aware ROUGE Accuracy
Epoch Number
(d) Hop-Aware Recall-Precision
Figure 4. Validation performance across epochs in the discriminative task for Hop-Diffused and Hop-Aware methods on OPT-125M.
P. Accuracy Evolution in Zero-Shot Node Classification with Different Number of Unseen Classes
1.0
1.0
0.6
0.6
0.2
0.2
ROUGE-L
Accuracy
Recall
Precision
(a) ROUGE-L and Accuracy for different number of unseen classes
(b) Recall and Precision for different
number of unseen classes
Figure 5. The performance in discriminative task with OPT- OPT-125M 125M backbonc across different number of unseen classes (1, 3,5, 7, 9).
23

---

Graph4MM: Weaving Multimodal Learning with Structural Information
Example Prompt and Output
Q.I. Example Prompt and Output in Generative Setting on WikiWeb2M Dataset
Task:
Summarize the given section content.
Section Content: In 1682, he published a sermon titled "By what means can ministers best win souls?" and in 1692, a letter to a minister in the country-suue supposed to his eldest brother, William (1640), minister of Borthwick, Midlothian entitled "A Vindication of the Protestant Doctrine concerning Justification and of its Preachers and Professors from the unjust Charge of Antinomianism." This "angry letter," as Dr. Calamy calls it, was occasioned by the violent controversy that broke out among the dissenting ministers of London after the republication in 1690 of the works of Dr. Tobias Crisp. Donald Macleod called it "unrivalled." Charges of Antinomianism side and of Arminianism on the other, and Traill was distinguished for his zeal against Arminianism. A somewhat similar controversy, known as the Marrow Controversy, followed in Scotland, and as Boston of Ettrick and others took the same side as Traill, his works became very popular among them and their adherents. He later published "Sermons on the Throne of Grace from Heb. iv. 16" (3rd edit. 1731) and "Sermons on the Prayer of Our Saviour, John xvii. 24." These works were devout, plain, and edifying, and were highly favored by those who were attached evangelical religion.
made
Context Information:
Section Image Caption: Rev Robert Traill's New Testament (1656)
1-Hop Neighbor Section Context: Robert Traill's early education was carefully superintended by his father, and at the University of Edinburgh he distinguished himself in both literary and theological studies. At the age of nineteen, he stood beside James Guthrie on the scaffold. He was later associated with John Welsh, minister of Irongray, known for holding armed conventicles. In 1666, and his family were forced into hiding after controversial book was found in their home. In 1667, he was denounced as 'Pentland rebel' and fled Holland, joining his exiled father and other Scottish refugees...
2-Hop Neighbor Section Context: Robert Traill was a church minister at Cranbrook in Kent. He was born at Elie in Fife in 1642. He was incarcerated on the Bass Rock, island in the Firth of Forth, from July 19, 1677, October 5, 1677. His work was often quoted by J. C. Ryle and is still published in the 21st century. (The remaining contexts are omitted for brevity.)
Expected Output:
His first short publication did not occur until he was forty years old, and the next did not appear until he was fifty.
Graph4MM Output:
The first work work was not appear until was thirty years old, it second publication not appear until he was fifty.
1 0.png
8001 0 0_0.pnggggg....
8001 2, 2_0jpg
(a) Section Image
(b) H-Hop Neighbor Image
(e) 2-Hop Neighbor Image
Figure 6. Images from multi-hop neighhoring sections within a webpage.
24

---

Graph4MM: Weaving Multimodal Learning with Structural Information
Q.2. Example Prompt and Output in Discriminative Setting Ele-Fashion Dataset
Task: Classify the given item.
Node Description: Speedo Men's Sonic Warmup Jacket
Available Classes:
O: Sneakers and Men's Formal Shoes
1: Lingerie, Costumes, and Women's 's Footwear
2: Jewelry and Accessories
3: Stockings and Watches
4: Graphic T-Shirts and Sweatshirts
6: Scarves, Suspenders, and Wallets Undergarments and Socks
8: Basic T-Shirts
9: Men's Casual and Formal Shirts
10: Dresses
11: Shoes and Boots
Neighbor Information:
1-Hop Neighbor: Speedo Women's 's Female Sonic Warm-Up Jacket
2-Hop Neighbor: Speedo Women's Female Sonic Warm-Up Pant
2-Hop Neighbor: Speedo Men's Sonie Warmup Jacket
Expected Output:
4:Graphic T-Shirts and Sweatshirts
Graph4MM Output:
4:Graphic T-Shirts and Sweatshirts.
Neighbor
Neighsor
Neighbor
(a) Node Image
(b) I-Hop Neighbor Image
(c) 2-Hop Neighbor Images
Figure Images from multi-ho hopp neighboring nodes.
25


Reviews:
[{"review_id": "1M22vkZf5o", "reviewer": "Reviewer_MweU", "summary": "The paper presents Graph4MM, for modelling multi-hop relationship within and between texts and images, modelled as an undirected graph. This approach enables the foundation model to be aware of the structural information (through the graph topology). They introduce Hop-Diffused MM-QFormer for incorporating multi-hop connectivity information and achieve SOTA results on generative (summarization) and discriminative (zero-shot fashion classification) task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your constructive questions. We provide our response in the form of Q&A.\n\n> **Theoretical Claim (TC) 1**: Softmax after masking.\n\nIn our implementation, we do apply row-wise softmax after causal masking as $A_{i:}= \\text{Softmax}(M_{i:} \\odot A'_{i:})$. This ensures that $A$ is row-stochastic and compatible with our analysis in Proposition A.1. We will revise equations in line 212 for clarity, provide pseudocode in the Appendix, and release our code upon publication.\n***\n> **TC2**: Elaborate the approximation in Proposition C.2 for GAT. \n\na. To strengthen our proof, we adopt a more general assumption inspired by [1] that avoids the approximation with Taylor expansion. Specifically, we assume the activation function satisfies $0 \\le \\frac{\\sigma(x)}{x} \\le 1$ for $x \\ne 0$ and $\\sigma(0) = 0$, which holds for common activations such as ReLU and LeakyReLU. We define $\\sigma(0)/0 := \\sigma'(0)$ or 1 if $\\sigma'(0)$ is undefined. Under this, any activation $\\sigma(y)$ can be rewritten as $\\sigma(y) = \\mathrm{diag}\\left( \\frac{\\sigma(y)}{y} \\right) y$. \n\nb. We further generalize our proof as follows by removing the previous identity-matrix-based approximation and reach the same conclusion. We denote $D_i^{(k)}$ as the diagonal transformation matrix induced by the activation at layer $k$, satisfying $\\text{diag}(0) \\preceq D_i^{(k)} \\preceq \\text{diag}(1)$. The representation $X_i^{(k+1)}$ can be recursively expanded as: $ X_{.i}^{(k+1)} = \\sum_{j_{k+1}=i, (j_k, ..., j_0) \\in [d]^{k+1}} ( \\prod^k_{l=0} W_{j_lj_{l+1}}^{(l)} ) D_{j_{k+1}}^{(k)} A^{(k)} \\cdots D_{j_1}^{(0)} A^{(0)} X_{.j_0}^{(0)}$,  where $A^{(k)}$ is a row-stochastic aggregation matrix. Since all $D^{(k)}$ and $A^{(k)}$ have spectral norm less than or equal to 1, and each weight matrix $W^{(k)}$ is bounded, it follows that $\\||X^{(k)}\\|| \\leq c^k \\cdot \\||X^{(0)}\\||$ for some constant $0 < c < 1$. The Dirichlet energy is given by: $\\mathcal{E}(X^{(k)}) = \\frac{1}{n} \\text{Tr}\\left( X^{(k)\\top} L X^{(k)} \\right)$, where $L$ is the graph Laplacian. Applying the Rayleigh-Ritz Inequality yields: $\\mathcal{E}(X^{(k)}) = O(\\gamma^k \\cdot \\mathcal{E}(X^{(0)}))$ for some $\\gamma \\in (0, 1)$, indicating exponential decay of energy.\n\n[1] Demystifying Oversmoothing in Attention-Based Graph Neural Networks. NeurIPS 2023.\n***\n> **TC3**: Attention Diffusion vs GAT under small $k$ for over-smoothing.\n\nTo examine the over-smoothing behavior under small $k$, we further conducted a simulation study on the Cora dataset. We report the Dirichlet Energy of representations produced by GAT/Hop-diffused Attention across $k=0$ to $4$ in Table 1. (1) As $k$ increases, Hop-Diffused Attention shows slower decay than GAT. (2) Additionally, the Dirichlet Energy gap of two methods is already evident at $k=1$, suggesting that our choice of $k=2$ in the main experiments is valid.\n\n**Table 1**: Simulation results for Dirichlet Energy comparison.\n|Method|K=0|K=1|K=2|K=3|K=4|\n|---|---|----|----|----|----|\n|Hop-Diffused Attention|3.2445|3.0105|2.9504|2.8360|2.7843|\n|GAT|3.2445|1.2307|1.0293|0.8027|0.3730|\n\n***\n\n>**Weakness (W) 1**: Disengage effects of graph modeling and Hop-diffused attention.\n\nTable 2 shows that (1) our graph modeling—adding multimodal neighbors—improves performance (Row 1→2), and (2) only hop-aware attention (Row 2→4) yields consistent gains over GAT-based structure modeling (Row 2→3), confirming the effectiveness of both components.\n\n**Table 2**: Ablation on graph and structural modeling (OPT-125M).\n\n|Method|BLEU-4|ROUGE-L|CIDEr|\n|------|---|----|----|\n|Node's T & I|0.0643|0.3825|0.6371|\n|Subgraph's T & I|0.0788|0.4051|0.7790|\n|GAT for Graph Tokens|0.0796|0.4052|0.7736|\n|Ours(Hop-Diffused)|**0.0800**|**0.4076**|**0.7831**|\n\n\n***\n>**W2**: Computational complexity of Graph4MM.\n\nDue to space limits, please see [Section W1 in our response to Reviewer xJRk](https://openreview.net/forum?id=FB2e8PV6qg&noteId=qC65trdQn8). We show our method has the same complexity order as the baseline and mild runtime overhead.\n***\n\n> **W3**: Should add new setting for comparison.\n\nDue to space limits, please see [Section W2 in our response to Reviewer xJRk](https://openreview.net/forum?id=FB2e8PV6qg&noteId=qC65trdQn8) for complete comparison results under a new setting.\n\n***\n>**Q1**: Representative examples where GraphMM succeeds and MMGL fails.\n\nBelow is a representative case where MMGL confuses node vs. neighbor features, while our structure-guided fusion yields the correct prediction.  \n[Link to the example.](https://anonymous.4open.science/r/Graph4MM-3F30/representative%20example.png)\n***\n\n>**Q2**: Why is BLUE in Table 2 high?\n\nBLEU-4 measures exact n-gram overlap. The inflation is due to the prediction containing more exact matches of domain-specific phrases from the ground truth.\n***\n>**Q3**: Meaning of \"modality-specific structural encoding\".\n\nIt refers to applying different structural modeling (Hop-Diffused / Hop-Aware) to each modality (vision or text).", "claims_and_evidence": "The authors provide a theoretical proof that GAT is more prone to over-smoothing effects but I doubt the proof (elaborated in Theoretical Claims). It would be nice to have it support it with a simulation study and spectral analysis.", "methods_and_evaluation_criteria": "Yes, the benchmarks make sense and the authors present a thorough comparison across models with ablations.", "theoretical_claims": "1. Rather than applying masking *before* softmax, the authors decide to apply it apply it after softmax. That is, $A_{i, j} = M_{i,j} \\cdot A_{i,j}'$, where $A'$ is a row stochastic matrix (because of softmax function) and $M$ is a binary mask. This means that $A$ is not necessarily a row stochastic matrix. Why did the author go for such design choice rather than the conventional choice of masking before softmax (which was also done in Wang et al for attention diffusion? \\\nFor proposition A.1 it relies on row stochasticity, but the $A$ is no longer row stochastic, or am I misinterpreting something?\n\n2. I am not convinced with Proposition C.2 for GAT. Could you elaborate how: \\\na. the first order Taylor expansion (about which point?) yields the said softmax approximation  \\\nb. is the softmax assumption valid? Because effectively, you approximate all the softmax functions to be identity (L747) which doesn't seem very valid.\n\n3. While the authors perform theoretical analysis for infinite diffusion steps/GAT Layers, they keep the number of diffusion steps to be 2. For lower values of k, I am not sure how diffusion is better than GAT (assuming the correctness of Dirichlet Energy proofs)", "experimental_designs_or_analyses": "The experiments are well designed, results carefully analyzed and experiment details provided in the supplementary", "relation_to_broader_scientific_literature": "The paper is surely interesting because it not only achieves SOTA results on the two benchmarks but it also contains interesting findings ans results about incorporating the topological structure of the data for generation tasks", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n1. The paper not only attempts at achieving SOTA for the tasks but also provides general result of using pretrained vision-language representations and also multi-hop representations\n2. The designed MM-QFormer incorporates multimodal information efficiently, and can be applied to other use cases as well\n2. The paper is easy to read and well-presented\n\n\nWeaknesses:\n\nAside from my reservations about the theoretical claims, I have following doubts:\n1. The authors should include a comparison with GAT to make sure that the prowess of their method comes from (1) their way of graph modelling alone, or (2) diffusion/hop-aware attention alone, or (3) both.\n2. The computational complexity of the method might impede its scalability\n3. Comparison on only one generative and one discriminative task might not be enough \n\nI would be willing to increase my score if the authors are able to answer all my questions satisfactorily.", "other_comments_or_suggestions": "Some minor typos:\n\n1. For proposition C.1, the authors use index $i$ for both -- row indexing and diffusion step indexing\n2. Fig 2.: $t_{v_i}$ instead of $t_{i}$ and  $p_{v_i}$ instead of $v_{i}$ for consistency\n3. L356R \"shown in\" in place of \"shownin\"", "questions_for_authors": "1. Can the authors provide representative examples where GraphMM succeeds and MMGL fails?\n2. Can you verify Table 2, row 5 (BLEU score for hop-aware -$t_{G'}$? Why is it significantly high?\n3. Could you elaborate on R358-362? What do you mean by \"modality-specific structural encoding\"?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HFF74KQeB7", "reviewer": "Reviewer_xJRk", "summary": "The paper proposes Graph4MM, a graph based multimodal learning framework. It integrates multi hop structural information into foundation models and fuses modality specific information. The main algorithmic ideas include Hop Diffused Attention and MM QFormer. Experiments show that Graph4MM outperforms larger VLMs, LLMs, and multimodal graph baselines, achieving a 6.93% average improvement.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your constructive questions, each of them are quite actionable. We take them quite seriously and prepare the following QA formatted response.\n\n>**Weakness (W) 1**: A weakness could be that the framework's complexity might limit its scalability.\n\n**A1**: We analyze the complexity of Graph4MM from both time complexity and runtime, showing that it shares the same order of complexity as LLM-based baselines like MMGL, with only a modest runtime overhead introduced by the Hop-Diffused MM-QFormer.\n\nSpecifically, the complexity of our Hop-Diffused Attention is $O(|\\mathcal{V}|^2 \\cdot d + K \\cdot |\\mathcal{E}| \\cdot d)$, and the Q-Former operates at $O(|\\mathcal{V}|^2 \\cdot n_q^2 \\cdot d)$, where $|\\mathcal{V}|$ is the number of nodes, $|\\mathcal{E}|$ is the number of edges, $n_q$ is the number of multimodal queries, and $d$ is the hidden dimension. However, the LLM remains the dominant cost which scales up to $O(|\\mathcal{V}|^2 \\cdot T^2 \\cdot d)$ due to processing per-node textual inputs—similar to MMGL, where $T$ is the average token length per node (usually $T \\gg n_q$ ). We also provide per-batch training and inference time comparisons in Table 2, where the runtime shows only a mild increase.\n\n**Table 1**: Per-batch training and generation time comparison for generative setting on OPT-125M backbone (Mean ± Standard Deviation).\n| Method   | Secs per Batch (Training) | Secs per Batch (Generation) |\n|----------|----------------------------|------------------------------|\n| Graph4MM | 0.2185 ± 0.1259            | 0.0679 ± 0.0002              |\n| MMGL     | 0.1649 ± 0.1033            | 0.0470 ± 0.0002              |\n\nAll LLM-based graph learning methods—including ours and MMGL—have quadratic complexity and are typically limited to small graphs, but they provide strong zero-shot generalization capabilities beyond the reach of traditional GNNs.\n\n***\n\n>**W2**: Also, the performance improvement might be dataset-specific, and more extensive testing is needed.\n\n**A2**: To further validate the effectiveness and generalization of our method, we introduce a new discriminative task—link prediction—on the Amazon-Sports dataset from the Multimodal Graph Benchmark (MM-Graph Bench) [1]. We randomly sample 10k source-target node pairs with equal numbers of positive and negative samples, and split the dataset into train/validation/test sets in an 8:1:1 ratio. To ensure a fair comparison, all methods use the OPT-125M backbone. The hyperparameter settings follow those used in our discriminative node classification experiments. We compare our method against the most representative and promising baselines used in our main paper, including: (a) pretrained language model using nodes' and subgraphs' text, (b) MMGL’s best-performing methods with both text and image modalities, \\(c) our Graph4MM with Hop-Diffused MM-Qformer. \n\nThe results, shown in Table 2, demonstrate that Graph4MM significantly outperforms all baselines, achieving an average improvement of 7.7% over the second-best method across all evaluation metrics. This further demonstrates the strong generalization ability of our method, showing its applicability across different datasets and scenarios.\n\n**Table 2**: Performance comparison in Link Prediction setting on Amazon-Sports dataset (OPT-125M Backbone).\n\n| Method                                | R-L     | Acc (%) | Rec (%) | Pre (%) |\n|---------------------------------------|---------|---------|---------|---------|\n| (PLM) Node's Text                     | 0.1563  | 52.35   | 51.87   | 72.67   |\n| (PLM) Node's Subgraph                 | 0.2871  | 56.81   | 56.26   | 74.56   |\n| (MMGL) Node's Text & Image            | 0.5603  | 93.92   | 93.86   | 95.51   |\n| (MMGL) Subgraph's Text & Image        | 0.7352  | 95.46   | 95.44   | 94.35   |\n| (Graph4MM) Hop-Diffused MM-QFormer    | **0.8904**  | **99.98**  | **99.97**  | **99.86**  |\n\n[1] Zhu, Jing, et al. \"Multimodal graph benchmark.\" arXiv preprint arXiv:2406.16321 (2024).", "claims_and_evidence": "The claims are generally supported by evidence. The authors conduct experiments on two datasets with various baselines. The performance improvements in both generative and discriminative tasks demonstrate the effectiveness of Graph4MM.", "methods_and_evaluation_criteria": "The proposed methods make sense for the problem. Graph4MM uses graph based structures to model multimodal relationships, which is suitable for handling complex multimodal data. The evaluation criteria, including using relevant datasets like WIKIWEB2M and ELE FASHION for generative and discriminative tasks respectively, are appropriate for assessing the model's performance.", "theoretical_claims": "The theoretical claims, such as the analysis of Hop Diffused Attention's properties and its comparison with GAT in terms of over smoothing, are presented with proofs.", "experimental_designs_or_analyses": "The experimental designs are sound. The authors compare Graph4MM with multiple baselines under different input settings. They also conduct ablation studies to analyze the importance of structural information. However, the generalization of the results could be further verified by testing on more diverse datasets.", "relation_to_broader_scientific_literature": "The key contributions of the paper are related to the broader scientific literature. It builds on existing works in vision language models and multimodal graph neural networks.", "essential_references_not_discussed": "There do not seem to be any essential references not discussed.", "other_strengths_and_weaknesses": "A strength of the paper is its innovative approach to multimodal learning using graph structures, which shows significant performance improvements. \n\nA weakness could be that the framework's complexity might limit its scalability. Also, the performance improvement might be dataset - specific, and more extensive testing is needed.", "other_comments_or_suggestions": "none", "questions_for_authors": "none", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z9hnQb3337", "reviewer": "Reviewer_gTEN", "summary": "The paper Graph4MM introduces a graph-based multimodal learning framework that integrates structural relationships into foundation models to improve multimodal understanding. Unlike previous methods that treat graphs as standalone modalities, Graph4MM incorporates Hop-Diffused Attention to model multi-hop connectivity and MM-QFormer for cross-modal fusion using learnable query tokens. The framework achieves an average improvement of 6.93% across generative and discriminative tasks, outperforming large VLMs, LLMs, and multimodal graph baselines. Theoretical and empirical analysis demonstrates that leveraging graph structure enhances multimodal learning beyond traditional one-to-one alignments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your constructive questions, each of them are quite actionable. We provide our response in the form of Q&A as follows.\n\n> **Weakness (W) 1**: Should cite relevant papers working on the multimodal knowledge graphs and talking about the relationship between them and Graph4MM.\n\n**A1**: We thank the reviewer for the suggestion. The survey [1] covers a wide range of MMKG research, including multimodal knowledge graph construction, multimodal representation learning, and cross-modal reasoning. These topics are relevant to our work in terms of constructing multimodal graphs and inter-modality fusion. We will update our Related Work section to include the mentioned survey [1] as well as representative recent works such as MoMoK [2] and MarT [3].\n\n[1] Chen, Zhuo, et al. \"Knowledge graphs meet multi-modal learning: A comprehensive survey.\" arXiv preprint 2024.\n\n[2] Zhang, Yichi, et al. \"Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning.\" ICLR 2025.\n\n[3] Zhang, Ningyu, et al. \"Multimodal analogical reasoning over knowledge graphs.\" ICLR 2023.\n\n***\n> **W2 & Question (Q) 1**: How does Graph4MM compared to Q-Former and GraphTranslator?\n\n**A2**: \n* **Compared to Q-Former**: \n\nWe would like to clarify that we do not claim the Q-Former architecture itself as our core contribution. Q-Former, originally from BLIP-2, is a standard module for visual-language fusion. Our key contribution lies in how we integrate structural heuristics into the multimodal fusion process.\n\n * **Compared to GraphTranslator**: \n\nOur method does not use the Translator module in GraphTranslator. The only similarity between our work and GraphTranslator lies in the use of the Q-Former architecture for cross-modality fusion, as discussed in the previous point. The Translator module in GraphTranslator is specifically designed for text-attributed graphs, fusing GNN-derived graph embeddings and explicit textual descriptions of graph structure using the Q-Former architecture. \n\nIn contrast, our method: (1) Escalates to full multimodal learning, incorporating vision in addition to text and graph structure. (2) Takes a fundamentally different approach to structural integration: instead of relying on explicit textual descriptions to align graph embeddings with the LLM’s space (as in GraphTranslator), we introduce hop-diffused attention—a sparse mechanism that implicitly encodes graph topology in the hidden state during vision-text fusion. (3) Provides both theoretical analysis and empirical evidence demonstrating that treating graphs as a separate modality, as done in GraphTranslator, is sub-optimal in complex multimodal scenarios where vision and language co-occur.\n\nWe believe our contributions are significant and fundamentally different from existing methods.\n\n***\n\n> **W3 & Q2**: Generalization of our method on different datasets.\n\n**A3**: Due to space limitations, we refer the reviewer to [Section \"W2\" in our response to Reviewer xJRk](https://openreview.net/forum?id=FB2e8PV6qg&noteId=qC65trdQn8), where we validate the effectiveness and strong generalization ability of our method under a new link prediction setting.\n\n***\n>**W4 & Q3**: Compared to other Graph Large Language Model-based methods.\n\n**A4**: We did not include graph-LLM methods as baselines because most prior works operate in a unimodal (text-only) setting and are not designed to handle multimodal inputs involving images. To better demonstrate the effectiveness of our method in multimodal scenarios, we conducted two additional experiments:  \n\n(a) We used CLIP-ViT to extract multimodal node embeddings as input to enhance LLaGA;\n\n(b) We enhance GraphPrompter by incorporating visual tokens in the same way as MMGL.\n\nFor a fair comparison, all models were evaluated under the same generative setting and applied the same fine-tuning strategy using the OPT-125M backbone. The results are shown below.\n\n**Table 1**: Comparison with multimodal extensions of Graph-LLM baselines under the generative setting (OPT-125M).\n\n| Method           | BLEU-4 | ROUGE-L | CIDEr  |\n|------|-----|-------|------|\n| LLaGA            | 0.0642 | 0.3738  | 0.6628 |\n| GraphPrompter    | 0.0782 | 0.4045  | 0.7651 |\n| Ours (Hop-Diffused) | **0.0800** | **0.4076** | **0.7831** |\n\nThese results further validate the effectiveness of our proposed approach, which introduces hop-diffused structural heuristics into the latent space of multimodal fusion, leading to superior performance in multimodal graph reasoning.\n\n*** \n> **Q4**: MMGL is evaluated under both frozen and fine-tuned settings?\n\n**A5**: We would like to clarify that all MMGL variants we compared were fine-tuned, following the official MMGL implementation. We provide additional results under the frozen generative setting with OPT-125.\n\n**Table 2**: Comparison on frozen backbone. \n|Method|BLEU-4|ROUGE-L|CIDEr|\n|--|--|---|--|\n|MMGL|0.0606|0.3382|0.6332|\n|Graph4MM|**0.0733**|**0.4018**|**0.7428**|", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, I checked them all and found no obvious errors.", "experimental_designs_or_analyses": "Yes, I checked them all. However, this paper has some limitations in the selection of dataset and baseline. For the details, please refer to weaknesses.", "relation_to_broader_scientific_literature": "1. Beyond One-to-One Vision-Language Models: Unlike BLIP-2 and Flamingo, which align single image-text pairs, Graph4MM models many-to-many multimodal relationships using graph structures.\n2. Advancing Multimodal Graph Learning: Extends MMGL by integrating graph topology into multimodal fusion rather than treating it as a standalone modality.\n3. Improving Graph-Based Attention: Builds on GATs and diffusion-based methods by introducing Hop-Diffused Attention, which encodes multi-hop connectivity while mitigating over-smoothing.\n4. Enhancing Query-Based Transformers: Extends QFormer (used in BLIP-2) by incorporating structural priors, improving cross-modal alignment with structural guidance.", "essential_references_not_discussed": "Line 50 “To the best of our knowledge, MMGL (Yoon et al., 2023) is the state-of-the-art work that models modalities into graphs and obtains promising performance in the generation task, as compared to single-modal pre-trained language models and vision-language models.”\n\nShould cite relevant papers working on the multimodal knowledge graphs and talking about the relationship between them and Graph4MM.\n\nFor example,  some most relevant papers mentioned in the survey:\n\nKnowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey (submitted to arxiv on 8 Feb 2024)\n\nor any other sota works working on this.", "other_strengths_and_weaknesses": "Strengths\n\n1. Structured Multimodal Integration: Effectively incorporates multi-hop graph connectivity into multimodal learning, improving intra- and inter-modal interactions.\n2.  Advanced Attention Mechanism: Hop-Diffused Attention enhances multimodal reasoning by encoding structural dependencies while avoiding over-smoothing.\n3. Good Empirical Results: Outperforms large VLMs, LLMs, and multimodal graph baselines, achieving a 6.93% average improvement across tasks.\n\nWeaknesses\n\n1. Lack of citations for relevant references.  Such as works related to multimodal knowledge graphs. I understand that this type of work is not a baseline for this paper, but it should be cited and briefly discussed in relation to this study.\n2. The proposed module lacks sufficient innovation. The paper introduces MM-QFormer, but its architecture shows little novelty compared to traditional Q-Former. The main difference lies in incorporating structural prior knowledge into the input embeddings, a concept already explored in previous works such as the Translator module in GraphTranslator [1], significantly reducing the originality of the contribution.\n3. Generalizability of the proposed method. The paper evaluates its performance on only one dataset for graph-related tasks, which raises concerns about the generalizability. Using a broader set of graph-based multimodal benchmarks [2] would strengthen the validity of the results and demonstrate the robustness of the approach across diverse datasets.\n4. Incomplete baseline selection. The paper lacks a comprehensive baseline comparison, particularly with Graph Large Language Models related methods [3,4] that can handle text-attributed graphs. Replacing embeddings with unified representations incorporating both text and image information can also show reasoning capabilities in multimodal graph tasks. However, the paper does not sufficiently explore or evaluate these alternatives, limiting the depth of its baseline analysis.\n\n\n[1] GraphTranslator: Aligning Graph Model to Large Language Model for Open-ended Tasks (https://arxiv.org/pdf/2402.07197)\n\n[2] Multimodal Graph Benchmark. (https://arxiv.org/pdf/2406.16321)\n\n[3] LlaGA: Large language and graph assistant (https://arxiv.org/abs/2402.08170)\n\n[4] Can we Soft Prompt LLMs for Graph Learning Tasks? (https://arxiv.org/abs/2402.10359)", "other_comments_or_suggestions": "No", "questions_for_authors": "1. How does it improve upon similar ideas, such as the Translator module in GraphTranslator? MM-QFormer seems to just move it over and use it.\n2. Given that the method is only evaluated on a single dataset for graph-related tasks, how can the authors ensure its robustness and generalizability across different multimodal graph benchmarks?\n3.  Why were Graph Large Language Models related methods (or GNN-based approaches) that integrate text and image embeddings not included as baselines? How would the proposed method compare to such models in multimodal graph reasoning tasks?\n4. The baseline MMGL is evaluated under both frozen and fine-tuned settings, whereas Graph4MM is only tested in the fine-tuned setting.Would a frozen version of Graph4MM still demonstrate effectiveness compared to MMGL?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mI9lrmEi31", "reviewer": "Reviewer_viW1", "summary": "This paper introduces Graph4MM, a novel framework for multimodal learning that leverages graph structures to model complex relationships between text and images. The framework can be divided into two parts: Hop-Diffused Attention and MM-QFormer. By giving both theoretical and empirical analysis, the authors demonstrate the superiority of Graph4MM.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your valuable questions, each of them are quite actionable. We provide our response in the form of Q&A as follows.\n\n> **Claims And Evidence (CE) 1 & CE4**: Graph as a standalone modality in discriminative setting; empirical validation of Proposition 4.1.\n\n**A1**: We conducted additional experiments on node classification (OPT-125M backbone) to assess the effectiveness of modeling the graph as a standalone modality. Following the generative setting described in Lines 392–395, we use GCN-derived graph/node embeddings as soft prompts that serve as input to the downstream LLM, equivalent to vision and language tokens. The results are shown below.\n\n**Table 1**: Performance of \"modeling graph as a standalone modality\" in the discriminative setting.\n|Method|ROUGE-L|Accuracy|Recall|Precision|\n|------|-------|--------|------|----------|\n|Subgraph's T&I|0.8144|99.85|83.25|83.33|\n|+Graph Token|0.6648|99.96|89.91|89.98|\n|+Node Token|0.6892|99.53|88.78|89.98|\n|Ours (Hop-Diffused)|**0.8282**|**100.00**|**100.00**|**100.00**|\n\nThese results reinforce our earlier finding: treating the graph as a separate modality does not lead to consistent gains, likely due to the semantic gap between GNNs with pretrained vision-language features. In contrast, our method performs significantly better, supporting Proposition 4.1 alongside Table 4 in our paper.\n\n*** \n> **CE2 & Methods And Evaluation Criteria (MEC) 3**: Computational Efficiency of our method.\n\n**A2**: Due to space limitations, we refer the reviewer to [Section \"Weakness (W) 1\" in our response to Reviewer xJRk](https://openreview.net/forum?id=FB2e8PV6qg&noteId=qC65trdQn8), where we theoretically and empirically show that our method shares the same order of time complexity as the baseline, with only mild runtime overhead.\n\nFor the memory cost, in the generative setting with the OPT-125M backbone, our method adds 13M trainable parameters via Hop-Diffused MM-QFormer, which is less than 6% of MMGL's 229M total parameters. This overhead would become even smaller as the LLM backbone scales up, thus, the memory usage of Graph4MM remains manageable.\n***\n> **CE3 & MEC2**: Testing on more diverse domains.\n\n**A3**: Due to space limitations, we refer the reviewer to [Section \"W2\" in our response to Reviewer xJRk](https://openreview.net/forum?id=FB2e8PV6qg&noteId=qC65trdQn8), where we validate the effectiveness and strong generalization ability of our method under a completely different domain.\n***\n> **MEC1**: Different number of unseen classes or OPT-125M and LLaMA-1B backbone.\n\n**A4**: For the LLaMA-1B model, the performance gap among different baselines under the 5 unseen-class setting is relatively small, as all methods achieve high classification accuracy. Thus, we increased the task difficulty by expanding the number of unseen classes to 9 for comparison to highlight the different performance between methods. In Appendix K, we provide the performance variation under different numbers of unseen classes.\n***\n\n> **Experiment Design or Analysis (ED) 1**: Ablation of vision and text interaction.\n\n**A5**: We ablate modality-specific structures and inputs by (1) removing structural heuristics from text or vision, and (2) removing the vision modality entirely. As shown in Table 2, removing structural heuristics for either vision or text modality degrades performance, the same as excluding visual information. This highlights the importance of both modalities and their corresponding structural guidance for effective multimodal fusion.\n\n**Table 2**: Modality interaction ablation results in the generative setting (OPT-125M).\n|Method Variant|BLEU-4|ROUGE-L|CIDEr|\n|-------------|------|--------|------|\n|Hop-Diffused MM-QFormer|**0.0800**|**0.4076**|**0.7831**|\n|-$t_{G'}$Hop-Diffused Attention|0.0786|0.4065|0.7765|\n|-$p_{G'}$Hop-Diffused Attention|0.0769|0.4044|0.7684|\n|-Vision Modality|0.0770|0.3992|0.7606|\n***\n> **ED2**: Influence of the random seeds.\n\n**A6**: Under the generative setting with OPT-125M, we report the mean and standard deviation over 3 random seeds (0–2), and compare with the best MMGL baseline under the same setup.\n\n**Table 3**: Model performance across random seeds.\n|Method|BLEU-4|ROUGE-L|CIDER|\n|------|------|--------|------|\n|Graph4MM|**0.07991±0.00066**|**0.40725±0.00028**|**0.78904±0.00305**|\n|MMGL|0.07762±0.00059|0.40504±0.00076|0.76907±0.00286|\n\n***\n> **ED3**: Discussion of the graph density.\n\n**A7**: We study graph density under the generative setting by varying (text, vision) neighbors from sparse (5,2), to medium (11,5), and dense (15,8). As shown in Table 5, our model remains stable across densities and benefits from more neighbors, consistently improving generation performance.\n\n**Table 4**: Impact of graph density on Graph4MM in generative setting (OPT-125M).\n\n|Density|BLEU-4|ROUGE-L|CIDEr|\n|-------|------|--------|------|\n|Sparse|0.0788|0.4049|0.7795|\n|Medium|0.0800|0.4076|0.7831|\n|Dense|0.0809|0.4086|0.8018|", "claims_and_evidence": "Most claims are well supported. But some points can be further improved.\n1.\tThe paper claims that treating graph structure as a guide rather than a standalone modality is more effective, but the results in Table 4 only contain generative task, maybe discriminative task should also be included.\n2.\tThe paper does not discuss computational efficiency trade-offs. Maybe results like runtime, memory usage comparison can be included.\n3.\tThe paper uses WikiWeb2M and ELE-FASHION, but testing on more diverse domains would be better.\n4.\tThere are no experiments about proposition 4.1, more empirical validation of this specific theory would be beneficial.", "methods_and_evaluation_criteria": "Yes, it makes sense. But there are some points:\n1.\tThe different setting for OPT-125M and LLaMA-1B(5 unseen classes vs 9 unseen classes)\n2.\tMore diverse datasets would be better \n3.\tThe concrete analysis for the computational efficiency should be included", "theoretical_claims": "The proofs are generally correct.", "experimental_designs_or_analyses": "1.\tThe interaction between text and vision modalities should be taken into consideration in an ablation study.\n2.\tHow does the random seed influence the results? Authors should claim this, make sure the results are not lucky initialization.\n3.\tThe paper should systematically explore the influence of different graph densities or structures.", "relation_to_broader_scientific_literature": "NO", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "NO", "other_comments_or_suggestions": "NO", "questions_for_authors": "NO", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received overall positive reviews. Reviewers found Graph4MM to be a novel and effective framework for integrating multi-hop structural information into multimodal learning via Hop-Diffused Attention and MM-QFormer. The approach shows strong empirical results on both generative and discriminative tasks. While some concerns were raised about theoretical proofs, baseline completeness, and evaluation scope, the authors provided a clear rebuttal that addressed these issues. The AC concurs with the reviewers and recommends acceptance. The authors are encouraged to expand evaluation and empirical support for theoretical claims in the camera-ready version.
Decision: Accept (poster)