InstructG2I: Synthesizing Images from Multimodal Attributed Graphs
Authors: Bowen Jin
Ziqi Pang
Bingjun Guo
Yu-Xiong Wang
Jiaxuan You
Jiawei Han
Venue: NeurIPS 2024 poster (2024)
Keywords: learning on graphs
image generation
diffusion model
graph neural network
Primary area: graph_neural_networks

Abstract: In this paper, we approach an overlooked yet critical task Graph2Image: generating images from multimodal attributed graphs (MMAGs). This task poses significant challenges due to the explosion in graph size, dependencies among graph entities, and the need for controllability in graph conditions. To address these challenges, we propose a graph context-conditioned diffusion model called InstructG2I. InstructG2I first exploits the graph structure and multimodal information to conduct informative neighbor sampling by combining personalized page rank and re-ranking based on vision-language features. Then, a graph QFormer encoder adaptively encodes the graph nodes into an auxiliary set of graph prompts to guide the denoising process of diffusion. Finally, we propose graph classifier-free guidance, enabling controllable generation by varying the strength of graph guidance and multiple connected edges to a node. Extensive experiments conducted on three datasets from different domains demonstrate the effectiveness and controllability of our approach. The code is available at https://github.com/PeterGriffinJin/InstructG2I.

INSTRUCTG21: Synthesizing Images from Multimodal Attributed Graphs
Bowen Jin, Ziqi Pang, Bingjun Guo, Yu-Xiong Wang, Jiaxuan You, Jiawei Han
Department of Computer Science
University of Illinois at Urbana-Chie Chaampainn
Nowenj44illinoot edu
https://instructte www.ggi...ggttt.uug.ee github. io/
Abstract
In this paper, we approach overlooked yet critical task Graph2Image: gen- erating images from multimodal attributed graphs (MMAGs). This task poses significant challenges due to the explosion in graph size, dependencies among graph entities, and the need for controllability in graph conditions. To address these challenges, we propose graph context-conditioned diffusion model called INSTRUCTG21, INSTRUCTG21 first exploits the graph structure and multimodal information to conduct informative neighbor sampling by combining personalized page rank and re-ranking based on vision- language features. Then, a Graph- QFormer encoder adaptively encodes the graph nodes into an auxiliary set of graph prompls guide the denoising process of diffusion. Finally, we propose graph classifier-free guidance, enabling controllable generation by varying the strength of graph guidance and multiple connected edges to node. Extensive experiments conducted on three datasets from different domains demonstrate the effectiveness and controllability of our approach. The code is available at https: //gi thub. wtt./etterrrieieieeeeeeeo htt.rrrrftffiii///nttttttt.t.... InstructG2I.
Introduction
This paper investigates an overlooked yet critical source of information for image generation: the pervasive stph-structrred relationships of real-world entities. In contrast to the commonly adopted language conditioning in models represented by Stable Diffusion [32], graph connections have combinatorial complexity and cannot be trivially captured a sequence. Such graph-structured relationships among the entities are expressed through "Multimodal Atributed Graphs" (MMAGs), where nodes enriched with image and text information. As concrete example (Figure (a)), the graph of artworks is constructed by nodes containing images (pictures) and texts (titles), as well edges corresponding shared genre and authorship. Such a graph uniquely depicts piece of artwork by its thousands of peers in the graph, beyond the mere description of language.
To this end, we formulate and propose the Graph2lmage challenge, requiring the generative models to synthesize image conditioning on both text descriptions and graph connections of a node. This task featuring the image generation MMAGs is well cell-grounded in real-world applications. For instance, generating an image for a virtual artwork node in the art MMAG is akin to creating virtual artwork according to the nuanced styles of artists and genres [5] (as in Figure (a)). Similarly, generating image for product node connected to other products through co- co-purchase links an e-commerce MMAG equates recommending future products for users [24]. Without surprise, our exploiting the graph-structured information indeed improves the consistency of generated images compared to models only using texts or images conditioning (Figure ((b)).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Alreaa wit Chanch
The
wyjo Graph Candthiingg
wy Grigh Croditrring
(b) Imsage Synthesis Compurison
Mosdy Kiakedy
Glval VoNe
Gomparbann
56%5 Kandhuly
Monet
Urndition
Graph Condition
Painting
?
(a) Muitimodal Attribute Geuph & Graph2lmage (Virtual Artist Application)
day
Controllable Generation
Figure We propose new task Graph2Image featuring image synthesis by conditioning on graph information and introduce novel graph-conditions diffusion model called INSTRUCTG21 to tackle this problem. (a) Graph21mage is supported by prevalent multimodal attributed graphs and is grounded in real-world applications, e.g., virtual artistry.
(b) INSTRUCTG21 outperforms baseline image generation techniques, demonstrating the usefulness of graph information. (c) To accommodate realistic user queries, INSTRUCTG21 exhibits smooth controllability in utilizing text/graph information and managing the strength multiple graph edges.
Despite the usefulness of graph information, existing methods conditioning either text [32] or images [2, 41] are incapable of direct integration with MMAGs. Therefore, we propose a graph context-aware diffusion model INSTRUCTG21 inherited from Stable Diffusion that mitigates gaps. A most prominent challenge directly originates from the combinatorial complexity of graphs, which we term Graph Size Explosion: inputting the entire local subgraph structure to model, including all the images and texts, is impractical due to the exponential increase in size, especially with additional hops. Therefore, INSTRUCTG21 learns to compress the massive amounts of contexts from the graph into a set of graph conditioning tokens with fixed capacity, which functions alongside the common text conditioning tokens in Stable Diffusion. Such compression process is enhanced with a semantic personalized pagerank-bass graph sampling approach to actively select the most informative neighboring nodes based both structural and semantic perspectives.
Besides the number contexts, the graph structures in MMAGs additionally specify the proximity of entities, which is not captured in conventional text or image conditioning. This challenge of "Graph Entity Dependency" reflects the implicit preference image generation: synthesizing a shirt image linked to "light-colored" clothing is likely have "pastel tone" (image-image dependency), and generating a picture titled "a running horse" should reference interconnected animal images rather than scenic ones (text-image dependency). To enable the nuanced proximity understanding on graphs, we further improve our graph conditioning tokens via Graph-Q architecture learning to encode the graph information guided by texts.
Finally, we propose that our graph conditioning a natural interface for controllable generation, reflecting the strength of edges in MMAGs. Take the virtual art generation (Figure ((c)) for example: INSTRUCTG21 can flexibly offer different strengths of graph guidance and can smoothly transition between the style of Monet and Kandinsky, defined by its strength of connection with either of the two artists. Such an advantage is grounded for real-world application and is a plug slug-and-play test-time algorithm inspired by classifier-free guidance [18]. In sum, contributions include:
Formulation and Benchmark. We are the first to identify the usefulness of multimodal attributed graphs (MMAGs) in image synthesis and formulate the Graph21mage problem. Our formulation supported by three benchmarks grounded in the real-world applications of art and e-commerce.
Algorithm. Methodologically, we propose INSTRUCTG21, context-aware diffusion model that ef- fectively encodes graph conditional information as graph prompts for controllable image generation (as shown in Figure 1(b,c)).

---

Experiments and Evaluation. Empirically, we conduct experiments graphs from three different domains, demonstrating that INSTRUCTG21 consistently outperforms competitive baselines shown in Figure (b)).
2 Problem Formulation
2.1 Multimodal Attributed Graphs
Definition (Multimodal Attributed Graphs (MMAGs)) A multimodal attributed graph can be defined as G = (V,EP,, P,D), where V. E, P and D represent the sets of nodes, edges, images, and documents, respectively. Each node U/ E V is associated with some textual information d., and some image information Pv, P.
For example, in an e-commerce product graph, nodes (v V) represent products, edges (e E) denote co-viewed semantic relationships, images (p E P) are product images, and documents (d E 'D product titles. Similarly, in graph (shown in Figure 1), nodes represent artworks, edges signify shared artists or genres, images are artwork pictures, and documents are artwork titles.
In this work, we focus graphs where edges provide semantic correlations between images (nodes). For instance, in an e-commerce product graph, connected products (those frequently co-viewed by many users) highly related. Similarly, in art graph, linked artworks (those created by the same author or within the same genre) are likely to have similar styles.
2.2 Problem Definition
In this work, we explore the problem of node image generation on MMAGs. Given a node vs in an MMAG G, our objective is to generate Pv, based on de, and G. This problem has multiple real-world applications. For example, in e-commerce, it translates to generating the image (pv.) ) for a product (v:) based user query (dv,) and user purchase history (9), which is generative retrieval task. In art domain, involves generating picture (po;) for an artwork (v;) based on its title (d:) and its associated artist style or genre (G), which is a virtual artwork creation task.
Definition 2 (Node Image Generation MMAGs) In multimodal attributed graph G (V, E, P, D), given node v V within the graph G with a textual description du,- the goal is to synthesize Pr,, the corresponding image es with leameed model P... f(1;, d)
Our evaluation emphasizes instance-level similarity, assessing how closely P.; matches Po.. We conduct evaluations on artwork graphs, e-commerce graphs, and literature graphs. More details can be found in Section 4.1.
Methodology
In this section, we present our INSTRUCTG21 framework, overviewed in Figure 2. We begin by introducing graph conditions into stable diffusion models in Section 3.1. Next, we discuss semantic personalized PageRank-bans based sampling select informative graph conditions Section 3.2. Furthermore, we propose Graph- Bapph-- to extract dependency-awer representations for graph conditions Section 3.3. Finally, we introduce controllable generation to balance the condition scale between text and graph guidance, well manage multiple graph guidances in Section 3.4.
3.1 Graph Context-aware Stable Diffusion
Stable Diffusion (SD). INSTRUCTG21 is built upon Stable Diffusion (SD). SD conducts diffuusion in the latent space, where an input image first encoded from pixel space into a latent representation z Enc(:). A decoder then transfers the latent representation z' back to the pixel space, yielding Dec(z'). The diffusion model generates the latent representation z' conditioned text prompt The training objective of SD is defined as follows:
= Ey -n [|e - c0(2,, t,
(I)
3

---

PrePPent NNoms Stare
Graph Q-former
Tea Diaader (Scide (ellaae))
Visium- Feature Similarity
Toe neer
Dise-trwerin
(CUP)
Freeptt
Goeditining
Singe Teal Frenpt
Prompt eg
Structural Relevance
Semantic Relavannce
GraphZZimae Diflusion (Denskring Thzsaamp))
Personalized Page Konk (PPA)
Similartty based Re ranking
(4) Training Setup
PPR-based Neighbor Sampling
(4) Model
The overall framework of INSTRUCTG21, (a) Given a target node with text prompt (e.g., House in Snow) in Multimodal Attributed Graph (MMAG) for which we want to generate an image, (b) we first perform semantic PPR-based neighbor sampling, which involves structure- aware personalized PageRank and semantic-aware similarity-based reranking sample informative neighboring nodes in the graph. (c) These neighboring nodes are then inputted into a Graph-Q -Former,, encoded by multiple self-attention and cross-attention layers, represented as graph tokens and used to guide the denoising process of the diffusion model, together with text prompt tokens.
Figure 2:
At each timestep i, the denoising network ea(-) predicts the noise by conditioning on the current latent representation z₁, timestep and text prompt vectors h(cr). To compute h(cr) E Raxfer, where lor is the length of er and d is the hidden dimension, the text prompt er is processed by the CLIP text encoder [31]: h(CT) CLIP(cT).
Introducing Graph Conditions into SD. In the context of MMAGs, synthesizing the image for a node v; involves not only the text de., but also the semantic information from the node's proximity on the graph. Therefore, we introduce an auxiliary set of graph conditioning tokens hG(cg) to the SD models (as shown in Figure 2(c)), working in parallel with the existing conditions hyr(er).
M(er cG) (er) ha(c)) R Rex((eey Hlag)
where leg is the length of the graph condition. The training objective then becomes:
L Eg (nc(xn)() er...ee))e.....NNNN.. (001).) [[|c ca (z, t, heor, ca))
(3)
For (CT), we can directly use the CLIP text encoder as the original SD. However, determining cG and h-)-) is more complex. We will discuss the details of cg and ha (-) in the following sections.
3.2 Semantic PPR-based Neighbor Sampling
straightforward approach developing ca(u;) involves using the entire local subgraph of es However, this is impractical due to the exponential growth in size with each additional hop, leading excessively long context sequences. To address this, we leverage both graph structure and node semantics to select informative cg.
Structure Proximity: Personalized PageRank (PPR). Inspired by [10], first adopt PPR [15] to identify related nodes from a graph structure perspective. PPR processes the graph structure to derive ranking score P.ş for each node vy relative node v, where higher P., indicates greater degree of "similarity" between iú and Let P E Rax* be the PPR matrix of the graph, where each row p represents PPR vector toward target node U;. The matrix P is determined by:
P 3AP B)T.
where the reset probability for PPR and Å is the normalized adjacency matrix. Once P is computed, we define the PPR-based graph condition cG of node vi as the top-Kpp PPR neighbors of node i;;
CG (Us)
arginax
Σ
P.j.
(5)
GGpp (o)=Vylog (wx))- iN tye0o (14)
Semantic Proximity: Similarity-based Reranking. However, solely relying on PPR may result in graph condition set containing images (e.g., scenery pictures) that are not semantically related to our

---

target node (e.g., a picture titled "running horse"). To address this, we propose using semantic-basee similarity calculation function Sim(d, d. p) (e.g., CLIP) to rerank 8; cG (vi) based on the relatedness of Pe; de.. The final graph condition (v())) is calculated by:
ca(v:)
argmax ca (b. (c (o))) eyena(m))
Σ
Sim
(6)
3.3 Graph Encoding with Text Conditions
After we derive ca(2) from the previous step, the problem comes how can we design ha(+) ( to extract meaningful representations from (vi). Here we focus more on how to utilize the image features from (G(1) (i.e., (Poylu3 |vy E ca (v:))) since we find they are more informative for ey image generation compared with text features from ca(us) (i.e. {dr, Joy ca(v:))) (shown in Section 4.3).
Simple Baseline: Encoding with Pretrained Image Encoders [31]. A straightforward way to obtain representations for By ca(w: is to directly apply some pretrained image encoders Jung (-) (e.g., CLIP [31]]:
9mg((o)) E R ha(co(v.)) @hhn,,l,yeatte E Rdxio
where ① denotes the concatenation operation. However, this simple design has two significant limitations: 1) The encoding for each Pry (v) E (r((w;)) is isolated from others in ca(w) and failed to capture the image-image graph dependency. For example, the style extraction from one picture (Pez) can benefit from the other pictures created by the same artist (in ca (v.).. ) 2) The encoding for each Pv, is independent to de, which fails to capture the text-image graph dependency. For example, when we are creating a picture titled "running horse" (dv,), ), is desired to offer more weight on horse pictures in (1;);) rather than scenery pictures.
as hG(+) to learn representations for eG while considering the graph dependency information. As shown in Figure 2, Graph-QFormer consists of two Transformer [35] modules motivated by [26]: (I) self-attention module that facilitates deep mutual information exchange between previous layer hidden states, capturing image-image dependencies and (2) a cross-attention module that weights samples in co using text guidance, capturing text-image dependencies.
Graph-QFormer. To address these limitations, we propose
Let H H()) ca(e.) &Rixig denote the hidden states outputted by the 1-th Graph-QFormer layer. We use the token embeddings of de, as the input query embeddings to provide text guidance:
H (0) (v:) [x1 .... Ldd.,
(8)
where ay is the k-th token embedding in dy and leo Id The multi-head self-attention layer (MHAsAr) is calculated by
H() (v.) = MHASAT = H ec(lv)) k H (t-1)) H (1-1) (vx)
(o.)..)
(9)
where q, k; v denotes query, key, and value channels in the Transformer. The output (t) inputted to the multi-heate cross-attention layer (MHAcxr), calculated by
is then
(v.)
H (v.) MHACxT (q
H'(t)
ca(t:)) Zea(e.); v
(10)
where (ma) = + [img (Pe,, I0,eee((..) E R ddd represents the image embeddings extracted from a fixed pretrained image encoder and n is the number of embeddings. Finally we adopt ha(co(v)) =
(L) (o.).. where L is the number of layers in Graph-QFormes
Connection between INSTRUCTG21 and GNNs. As illustrated in Figure 2, INSTRUCTG21 employssseeeploys
a Transformer-base based architecture as the graph encoder. However, it can also be interpreted as a Graph Neural Network (GNN) model. GNN models [38] primarily to obtain node representations (Ni) denotes the neighbor set of i);
pppaagatioonaggrggattn paradigm
(1-1) = ( (n. (1-1) h (-") (v)eW()): h !GG!)) (n. (n e(())) a
Similarly, in INSTRUCTG21, (4))5))) can be regarded as the propagation function PROP(),
while the aggregation step (GG corresponds to the combination of Eq.()) and Eq.(10).
5

---

3,4 Controllable Generation
The concept of classifier-free guidance, introduced by [18], enhances the performance of conditional image synthesis by modifying the noise prediction, eg(-), with the output from an unconditional model. This formulated as: ég [zz c) ca ca(z;, @) + s (co(Z,, c) - co (z 2)), where s(> 1) the guidance scale. The intuition is that learns the gradient of the log image distribution and increasing the contribution of (c)c) - co(#)) will enlarge the convergence to the distribution conditioned on c.
In our task, the score network condition cg. We compose the score estimates from these two conditions and introduce two guidance scales, ST and sg. to control the contribution strength of eT and CG to the generated samples respectively. Our modified score estimation function is:
CG, CT) is conditioned on both text cr = dg and the graph
éø Ea(2,,
= (Z,, Ø, 0) + ST (eo(z,, 0, - c0(2), 23, 21) + (ea Ze: Ca, - ea(z,, 2, (T))).
(11)
For cases requiring fine-grained control over multiple graph conditions (i.e., different edges), extend the formula as follows:
Eo (Ze, CG, CT) = Ep (Ze; Ø, Ø) + 8T (Eo(Ze, 0, - Ea(Ze, 2, 01)
Σ (k) (co(2), C CT) cZ (T)))
(12)
where (k) is the k-th graph condition. For example, to create an artwork that combines the styles of Monet and Van Gogh, the neighboring artworks by Monet and Van Gogh on the graph would be c() and <a respectively. Further details on the derivation of our classifier-free guidance formulations can be found in Appendix A.3.
(2)
4 Experiments
4.1 Experimental Setups
Datasets. We conduct experiments on three MMAGs from distinct domains: ART500K [27]. Amazon [16], and Goodreads [37]. ART500K is artwork graph with nodes representing artworks and edges indicating same-author or same-genrs relationships. Each artwork node includes a title (text) and picture (image). Amazon is product graph where nodes represent products and edges denote co-view relationships. Each product is associated with a title (text) and a picture (image). Goodreads literature graph where nodes represent books and edges convey similar-book semantics. Each book node contains a title and front cover image. Dataset statistics can be found in Appendix A.4.
Baselines. We compare INSTRUCTG21 with two groups of baselines: 1) Text-to-image methods: This includes Stable Diffusion 1.5 (SD-1.5) [32] and SD 1.5 fine-tuned on our datasets (SD-1.5 FT). 2) Image-to-image methods: This includes InstructPix2Pix [2] and ControlNet [41]. both initialized with SD 1.5 and fine-tuned on our datasets. We use the most relevant neighbor, as selected in Section 3.2 the input image for these baselines, allowing them partially utilize graph information.
Metrics. As indicated in Section 2.2, our evaluation mainly concerns the consistency of synthesized images with the ground truth image on the node. Therefore, our evaluation adopts the CLIP [31] and DINOv2 [29] score for instance-level similarity, addition to the conventional FID [17] metric for image generation. For the CLIP and DINOv2 scores, we utilize CLIP and DINOv2 to obtain representations for both the generated and ground truth images and then calculate their cosine similarity. For FID, calculate the distance between the distribution of the ground truth images and the distribution of the generated images.
4.2 Main results
Quantitative Evaluation. The quantitative results are presented in Table 1 and Figure 3. From Table 1, we observe the following: 1) INSTRUCTG21 consistently outperforms all the base- line methods, highlighting the importance of graph information in image synthesis on MMAGs. 2) Although InstructPix2Pix and ControlNet partially consider graph context, they fail to cap- ture the semantic signals from the graph comprchensively. In Figure 3, plot the aver-

---

Table 1: Quantitative evaluation of different methods on ART500K, Amazon, and Goodreads datasets. The CLIP score denotes the image-image score. INSTRUCTG21 significantly outperforms the best baseline with p-value < 0.05 and consistently outperforms all the other common baselines in image synthesis, supporting the benefits of graph conditioning.
ART500K
Amazon
Goodreads
Model
CLIP score
DINOv2 score
CLIP score
DINOv2 scure
CLIP scure
DINOv2 score
SD-1.5
58.83
25.86
60.67
32.61
42.16
14.84
SD-1.5 FT
66.55
34.65
65.30
41.52
45.81
18.97
InstructPix2Pix
65.66
33.44
63.86
41.31
47.30
20.94
ControlNet
64.93
32.88
59.88
34.05
42.20
19.77
INSTRUCTG21
73.73
46.45
68.34
51.70
50.37
25.54
Ground-truth
Sampled Neighbors
[a] Ours
(b) Stable Diflusion (c) InstruciPaaPix
(d) ControiNet
Prompt: bh Crater and The Clouds
Prompt: "Painting of My wife And Doughtar
Prumpt: "irickker
fuller kazr
insdantly
thick
seran"
Figure 4: Qualitative evaluation. Our method exhibits better consistency with the ground truth by better utilizing the graph information from neighboring nodes ("Sampled Neighbors" in the figure).
age DINOv2 (x-axis, 1) and FID score (y-axis, J) across the three datasets. INSTRUCTG21 outperforms most baselines on both metrics and achieves the best trade-off between them. InstructPix2Pix obtains better FID score than INSTRUCTG21 because it takes an in-distribution image as imput, constraining the output image to stay close to the original distribution.
110
FID Score
100
SD 1.5
SD 1.5TT
Qualitative Evaluation. We conduct qualitative evaluation by randomly selecting some generated cases. The results are shown Figure 4, where we provide the sampled neighbor im- ages from the graph, text prompts, and the ground truth images. From these results, we observe that INSTRUCTG21 generates images that best fit the semantics of the text prompt and context from the graph. For instance, when generating a picture for "the crater and the clouds". the baselines either capture only the content ("crater" and "clouds") without the style learned from the graph (Stable Diffusion and InstructPix2Pix) or adopt a similar style but lose the desired content (ControlNet). contrast, from the neighbors on the graph and conveys the content accurately.
ControlNet
Ours
InstructPix2Pix
50
0.2
0.25
035
0.4
0.45
DINOv2 Score
Figure 3: INSTRUCTG21 achieves the best trade-off between DI- NOv2 (1) and FID (l) scores.
INSTRUCTG21 effectively learns

---

Table 2: Ablation study on graph condition variants and Graph- Former
ART300K
Amazon
Goodreads
Model
CLIP
DINOv2
CLIP
DINOv2
CLIP
DINOv2
INSTRUCTG21
73.73
46.45
68.34
51.70
50.37
25.54
Graph-QFomma
72.53
44.16
66.97
48.18
47.91
24.74
GraphSAGE
72.26
43.06
66.07
43.40
46.68
21.91
GAT
72.60
43.32
66.73
46.58
46.57
21.45
IP2P neighhor images SD FT neighbor texts
65.89
33.90
63.19
40.32
47.21
21.55
69.72
38.64
65.55
43.51
47.47
22.68
Sampled Neighboos
Random Sampling
PPR-based Sampling
Somanti es basaa Sampling
Ours (Semantic PPR- based Sampling)
Text P Prompt
"The Horse of the Frieze"
sampled Neighbors
Ground Truth
Generated Imsges
Figure 5: Ablation study on semantic PPR-based neighbor sampling. The results indicate that both structural and semantic relevance proposed by method effectively improve the image generation quality and consistency with the graph context.
4.3 Ablation Study
Study of Graph Condition for SD Variants. In INSTRUCTG21, we introduce graph conditions into SD by encoding the images from into graph prompts, which serve as conditions together with text prompts for SD's denoising step. this section, we demonstrate the significance this design by comparing it with other variants that utilize graph conditions in SD: InstructPix2Pix (IP2P) with neighbor images and SD finetuned with neighbor texts. For the first variant, we perform mean pooling on the latent representations of images in cc. according to the IP2P's setting, and use this as the input image representation for IP2P. This variant has the same input information as INSTRUCTG21. For the second variant, utilize text information from neighbors instead of images, concatenate with the text prompt, and fine-tune the SD. The results are shown in Table 2, where INSTRUCTG21 consistently outperforms both variants. This demonstrates the advantage of leveraging image features from cG and the effectiveness of our model design.
Study of Graph-QFormer. We first demonstrate the effectiveness of Graph-QFormer by replacing it with the simple baseline mentioned Eq.(7), denoted as " Graph-QFormerr" QFormer". We then compare with graph neural network (GNN) baselines including GraphSAGE [13] and GAT [36]. integrated into INSTRUCTG21 the same manner. The results, presented in Table 2, show that INSTRUCTG21 with Graph-QFore FFomher consistently outperforms both the ablated version and GNN baselines. This demonstrates the effectiivnnes of FFommrr design.
Study of the Semantic PPR-based Neighbor Sampling. We propose a semantic sampling method that combines structure and semantics for neighbor sampling on graphs, as detailed in Section 3.2. In this section, we demonstrate the effectiveness of this approach by conducting ablation studies that remove either or both components. The results, shown in Figure 5, indicate that our sampling methods effectively identify neighbor images that contribute most significantly to the ground truth in both semantics and style. This underscores the value of integrating both structural and semantic information in sampling approach.

---

Gragh Goldaare
Gentart Coun
Tod Prompt
House In
Wooded Aree.
Sampled Neighbors
Polle Plzzsse
Test
(a) Text and graph guidance study.
(b) Single or multiple graph guidance.
Figure 6: Controllable generation study. (a) The ability of INSTRUCTG21 to balance text guidance and graph guidance. (b) Study of multiple graph guidance. Generated artworks with the input text prompt "a man playing piano" conditioned on single or multiple graph guidance (styles of "Picasso" and "Courbet"). Please refer to Figure for another example between Monet and Kandinsky.
4.4 Controllable Generation
Text Guidance & Graph Guidance. (1I, we discuss the control of guidance from both text and graph conditions. To illustrate its effectiivness,, we provide example in Figure 6(a). The results show that as text guidance increases, the generated image incorporates more of the desired content. Conversely, graph guidance increases, the generated image adopts more desired style. This demonstrates the ability of our method to balance content and style through controlled guidance.
Multiple Graph Guidance: Virtual Artist. In Eq.(12), we demonstrate how multiple graph guidance can be managed for controllable image generation. We present use case, virtual artwork creation, to showcase its effectiveness (shown in Figure 6(b)). The goal of this task is to create an image that depicts specific content (e.g., man playing piano) in the style of Picasso and Courbet). This is akin to adding a new node to the graph that links to the artwork nodes created by the specified artists and generating an image for this node. The results indicate that when single graph guidance is provided, the generated artwork aligns with that artist's 's style. As additional graph guidance is introduced, the styles of the artists blend together. This demonstrates that our method offers the flexibility to meet various control requirements, effectively balancing different types of graph influences.
more artists (e.g.,
4.5 Model Behavior Analysis
Cross-attention Weight Study in Graph-QFormer. We conduct cross-attention study for Graph- QFormer to understand how different sampled neighbors on the graph are selected based on the text prompt and contribute the final image generation. We randomly select a case with the text prompt and neighbor images and plot the cross-attention weight map shown in Figure 7. From the weight map, we can find that which are related to "raising" and "I "Lazarus" in the text prompt respectively. The results indicate that effectively learns to select the images that are most relevant to the text prompt.
learns to assign higher weight to pictures and
5 Related works
Diffusion Models. Recent advancements in diffusion models have demonstrated significant success generative applications. Diffusion models [4, 7] generate compelling examples through a step-wise denoising process, which involves forward process that introduces noise into data distributions and a reverse process that reconstructs the original data [19]. A notable example the Latent Diffusion Model (LDM) [32], which reduces computational costs by applying the diffusion process

---

Sampled Neighbors
Overall
Text prompt
0220
Bangghber Titens
-he
0200
Raising of
Lasarus
Defail.
Text Tokens
Figure 7: Study of Craph-QFormer' cross-attention map. Graph-QFormer effectively learns to select the images that are most relevant to the text prompt.
low-resolution latent space. In the domain of diffusion models, various forms of conditioning are employed to direct the generation process, including labels [6], classifiers [8], texts [28], images [2], and scene graphs [39]. These conditions can be incorporated into diffusion models through latent
concatenation [33], cross-aftention [1], and gradient control [12]. However, most existing works
neglect the relational information between images and cannot be directly applied to image synthesis
on MMAGs.
Learning on Graphs. Early studies learning on graphs primarily focus on representation learning for nodes edges based graph structures [3, 14]. Methods such as Deepwalk [30] and Node2vec [11] perform random walks on graphs to derive vector representation for each node. Graph neural networks (GNNs) [38, 43] are later introduced as a learnable component that incorporates both initial node features and graph structure. GNNs have been applied to various tasks, including classification [25]. link prediction [42], and recommendation [21]. For instance, GraphS [13] employs
propagation and aggregation paradigm for node representation learning, while GAT [36] introduces an attention mechanism into the aggregation process. Recently, research has increasingly focused on integrating text or image features with graph structures [22, 44]. For example, Patton [23] proposes
pretraining language models text-atrrbbted graphs. However, these existing works mainly target representation learning on single-modal graphs and are not directly applicable to the image synthesis from multimodal attributed graph (MMAG) task addressed in this paper.
6 Conclusions
In this paper, we identify the problem of image synthesis on multimodal attributed graphs (MMAGs). To address this challenge, we propose graph context-conditione diffusion model that: Samples related neighbors on the graph using a semantic personalized PageRank-baas method; 2) Effectively encodes graph information graph prompts by considering their dependency with Graph-QFormer; 3) Generates images under control with graph classifier-free guidance. We conduct systematic experiments MMAGs in the domains of art, e-commerce, and literature, demonstrating the effectiveness of our approach compared to competitive baseline methods. Extensive studies validate the design of each component in INSTRUCTG21 and highlight its controllability. Future directions include joint text and image generation on MMAGs and capturing the heterogencous relations between image and text units on MMAGs.
Acknowledgments and Disclosure of Funding
This work was supported by the Apple PhD Fellowship. The research also was supported in part by US DARPA INCAS Program No. HR0011-21-C0165 and BRIES Program No. HR001 1-24-3-0325 National Science Foundation IIS- 19-56151,, the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environment (I-GUIDE) by NSF under Award No. 21 21399. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of DARPA the U.S. Government. The views and conclusions contained this paper are those of the authors and should not be interpreted representing any funding agencies.
10

---

References
[1] Namhyuk Ahn, Junsoo Lee, Chunggi Lee, Kunhee Kim, Daesik Kim, Seung-Hun Nam, and Kibeom Hong. Dreamstyler: Paint by style inversion with text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 674-681, 2024.
10
[2] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392- 2023. 2, 6, 10
[3] Hongyun Cai, Vincent W Zheng. and Kevin Chen-Chuse Chang. A comprehensive survey of graph embedding: Problems, techniques, and applications. IEEE transactions on knowledge and data engineering, 101))161161137, 2018. 10
[4] Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering, 2024.
[5] Eva Cetinic and James She. Understanding and creating art with ai: Review and outlook. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 18(2):
1 1-22, 2022.1
[6] Jian Chen, Ruiyi Zhang, Tong Yu, Rohan Sharma, Zhiqiang Xu, Tong Sun, and Changyou Chen. Label-retrieval-anmmened diffuusin models for learning from noisy labels. Advances in Neural Information Processing Systems, 36, 2024. 10
[7] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021. 10
[9] Rohit Gandikota, Joanna Materzynska, Jaden from diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
and David Bau. Erasing concepts
2023. 14
[10] Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810005997, 2018.
[11] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855-864, 2016. 10
[12] Yingqing Guo, Hui Yuan, Yukang Yang, Minshuo Chen, and Mengdi Wang. Gradient guidance for diffusion models: An optimization perspective. arXiv preprint arXiv:2404.14743, 2024. 10
[13] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, pages 1024-1034, 2017. 8, 10
[14] William L Hamilton, Rex Ying, and Jure Leskovee. Representation learning on graphs: Methods and applications. arXiv preprint arXiv: 2017. 10
[15] Taher H Haveliwala. Topic-sensitive pagerank. In Proceedings of the I 11th international conference on World Wide Web, pages 517-526, 2002. 4
[16] Ruining He and Julian McAuley. Ups and downs: trends with one-class collaborative filtering. In WWW, pages 507-517, 2016.
Modeling the visual evolution of fashion
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 6

---

[18] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. 12X1::2207.12598. 2022. 2,6 6
arXiv preprint
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. 9
[20] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005. 14
[21] Bowen Jin, Chen Gao, Xiangnan He, Depeng Jin, and Yong Li. with graph convolutional networks. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, pages 659-668, 2020. 10
recommendation
[22] Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, and Jiawei Han. Large language models on graphs: A comprehensive survey. arXiv preprint wrr.vv:232..22777 2023. 10
[23] Bowen Jin, Wentao Zhang, Yu Zhang, Yu Meng, Xinyang Zhang, Qi Zhu, and Jiawei Han. Patton: Language model pretraining on text-rich networks. arXiv preprint arXiv:2305.12268 2023. 10
[24] Wei Jin, Haitao Mao, Zheng Li, Haoming Jiang, Chen Luo, Hongzhi Wen, Haoyu Han, Hanqing Lu, Zhengyang Wang. Ruirui Li, et al. Amazon-m2: A multilingual multi-locale shopping session dataset for recommendation and text generation. Advances in Neural Information Processing Systems, 36, 2024.
[25] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint
2016. 10
[26] Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023. 5, 16
[27] Hui Mao, Ming Cheung, and James She. Deepart: Learning joint representations of visual arts. In Proceedings of the 25th ACM international conference on Multimedia, pages 1183-1191.
ACM, 2017. 6
[28] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, lyya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 12. 10741, 2021. 10
[29] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.071999 2023. 6
[30] Bryan Perozzi, Rami Al- and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 701-710, 2014. 10
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-8763. PMLR, 2021. 5,
[32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022. 1, 6, 9,
14
[33] Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. arXiv preprint arXiv:2311.100S:
2023. 10
[34] Anwaar Ulhaq, Naveed Akhtar, and Ganna Pogrebna. Efficient diffusion models for vision: survey, arXiv preprint arXiv:2210.09290 2022. 14
12

---

[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[36] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In JCLR, 2018. 8, 10
[37] Mengting Wan, Rishabh Misra, Ndapandula Nakashole, and Julian McAuley. Fine-grained spoiler detection from large-scale review corpora. In ACL, pages 2605-2610, 2019. 6
[38] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4-24 2020. 5, 10
[39] Ling Yang, Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming-Hsuan Yang. Diffusion-based scene graph to image generation with masked contrastive pre-training. arXiv preprint arXiv:2211.1 2022. 10
[40] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721, 2023. 16
[41] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-imaae diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023. 2,
[42] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018. 10
[43] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI open, 1:57-81, 2020. 10
[44] Jing Zhu, Yuhang Zhou, Shengyi Qian, Zhongmou He, Tong Zhao, Neil Shah, and Danai Koutra. Multimodal graph benchmark. arXiv preprint arXiv:2406. 16321, 2024. 10
[45] Haomin Zhuang, Yihua Zhang, and Sijia Liu, A pilot study of query-free adversarial attack against stable diffusion. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 2384-2391, 2023. 14
13

---

A Appendix
A.1 Limitations
In this work, we focus on node image generation from multimodal attributed graphs, utilizing Stable Diffusion 1.5 as the base model for INSTRUCTG21. Due computational constraints, we leave the exploration of larger diffusion models, such as SDXI., for future work. Additionally, we model the graph as homogeneous, not accounting for heterogeneous node and edge types. Considering that different types of nodes and edges convey distinct semantics, future research could investigate how to perform Graph2Image on heterogeneous graphs.
A.2 Ethical Considerations
While stable diffusion models [32] have demonstrated advanced image generation capabilities, studies highlight several drawbacks, such the uncontrollable generation of NSFW content [9], vulnerability to adversarial attacks [45], and being computationally intensive and time-consuming [34]. In INSTRUCTG21, we address these challenges by introducing graph conditions into the image generation process. However, since INSTRUCTG21 employs stable diffusion as the backbone model, remains susceptible to these limitations.
A.3 Classifier-free Guidance
In Section 3.4, we discuss controllable generation to balance text and graph guidances (cr and cg) as well as managing multiple graph guidances (cg) ()))) ). We introduce sT and &g to control the strength of text conditions and graph conditions and have the modified score estimation shown as follows (copied from Eq.(11)):
Ca(2,, CG, = (0,, 63, 00,05 + ST (co(2;, 0,CT) 60 23, 23)
+8G (eo(Ze, CT) - ea (Ze, 2,07 (T)))
In this section, we will provide mathematical derivation on how these modified score estimations are developed. Noted that INSTRUCTG21 learns P(z|eg, er), the distribution of image latents conditioned on text information Cr and graph information cg. which can expressed as:
P(zlco, = P(z, ca. CT) P(cgler, 2) P(orlz) P(ca,eT)
(13)
e(ca,er)
INSTRUCTG21 learns and estimates the score [20] of the data distribution, which can also be interpreted as the gradient of the log distribution probability. By taking log on both sides of (13() we can attain the following equation:
log( ((z) ca. er)) = (ogg((ccloo z)) + log( (P((c(o()) + log (oz()(z)) log( P(cc, oT)).
(14)
After calculating the derivation on both sides of (1.() we can obtain:
alog( ((zzea,, (T)) Ōlog( P(caler,2) 2)) Ölog ((crlz)) Ölog P(z))
(15)
dz
ôz
Oz
This corresponds our classifier-free guidance equation shown in Eq.(11), where sT controls how the data distribution shifts toward the zone where P(cr (2) assigns a high likelihood to cr and 8g determines how the data distribution leans toward the region where P(cal /CT, z) assigns high likelihood to cg. Although there are other ways to derive the modified score estimation function (e.g., switching er and sa or making it symmetric), we empirically find that our derivation contributes to both advanced performance (since P(er zz) is well learned in the base model) and high efficiency (since the denoising operation only needs be conducted three times rather than four times compared with symmetric setting).
(k) If given multiple graph conditions, we utilize se to control the strength for each of them and have the derived score estimation function as follows (copied from (12))
è [2,, Cas cT) = <o(2, 0,0) + ST (co(z,, Ø, (,Cr) - cn(z,, Ø, 2)) (k)
Σ
$
(K)
(co(Ze, Cg (T) co(a,, 20, (0,r)))
14

---

Table 3: Dataset Statistics
Dataset
# Node
# Edge
ART500K
311,288 178,890
643.008,344
Amazon
3,131,949
Goodreads
93,475
637,210
Table 4:
configuration for model training.
Parameter
ART500K
Amazon
Goodreads
Optimizer
AdamW
AdamW
AdamW
Adam
lc-8
lc-8
Adam (31,82]
(0.9. 0.999)
(0.9, 0.999)
(0.9, 0.999)
Weight decay
le-2
le-2
le-2
Batch size per GPU
16
16
Grudient Accumulation
Epochs
10
10
Resolution
256
256
256
Learning rute
lc-5
le-5
le-5
Backbone
Stable Diffusion
multiple graph conditions are given, (13( then becomes:
P(z ..... G (M) (M) CT) P(z, (1) (M)
Ca CG CT)
P(ea (1)
Ca
(M)
CT, P(cTl)) (16)
P(eg (M)
cc)
P(c
C cr)
(M)
where M is the total number of graph conditions.
Assume cg are independent from each other, then we can attain:
(M) (1) CG (M),,T) .T) = I& P(c (k)
CT. z) (z)
P(z
(17)
P(ee ca 200 .TT)
(M)
Similar to (15) we can obtain:
(log(P(z)ce) (1) ...., CG (T))
(M)
Ölog P(ce) (k) eT, z))
Ölog P((rlz))
Ōlog( P(z)) (18)
dz
Oz
δz
This corresponds to the classifier-fret free guidance equation shown in Eq,(1 where sq determines how the data distribution leans toward the region where P(ce) (R) (er, z) assigns high likelihood to the
(k)
(k)
graph condition CG
A.4 Datasets
The statistics of the three datasets be found in Table 3. Since Amazon and Goodreads both have multiple domains, we select one from each of them considering the graph size: Beauty domain from Amazon and Mystery domain from Goodreads.
A.5 Experimental Settings
We randomly mask 1,000 nodes as testing nodes from the graph for three datasets and serve the remaining nodes and edges the training graph.
In implementing INSTRUCTG21, we initialize the encoder and U-Net with the pretrained pa- rameters from Stable Diffusion 1.5 We use the pretrained CLIP image encoder as our fixed image encoder to extract features from raw images. For Graph-QFormed QFommer, we empirically find that initializing it with the CLIP text encoder parameters can improve performance compared with random initialization.
15

---

We use AdamW as the optimizer to train INSTRUCTG21. The training of all methods including INSTRUCTG21 and baselines on ARTS00K and Amazon are conducted on two A6000 GPUs, while that on Goodreads is performed on four A40 GPUs. Each image is encoded as four feature vectors with the fixed image encoder following [40] and we insert one cross-encoder layer after every two self-attention layers in Staph-QFormme following [26]. The detailed hyperparameters are in Table 4.
16

---

The contributions and scope is reflected in the abstract and introduction. NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of this work in the conclusion section and Appendix section A.I.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
The theoretical derivation of controllable generation is shown in Appendix
A.3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
provide the code at httpss///ggtubb...mm//tettrrrrr
InstructG2I.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the code at httpp://ggtaab..cco/eeeeetrrrrr
InstructG2I.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
cetoomissssooooooooooooooooooooo cesssonooliooooo ooiccyyyy)ll for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ wwcp/bllc///eeeeccceeeceecceelee http//gudeee/eeeee CCtees//ooSSubmieec cessonnoolcyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Detailed experimental settings discussed in Appendix A.5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide the p-value in section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes)
Justification:
The compute resources information is given in Appendix A.5,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /nwurip...p.. http/pullc//ttcctiiiiiaeeee
Answer: [Yes]
Justification:
We strictly follow the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
This is discussed in Appendix A.2.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The licenses for the assets are discussed in Appendix A.4.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The license is provided together with the code in https://githitn com/ wwwerGrrffinJnnnnnstrrree
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "FlRUyG2fRS", "reviewer": "Reviewer_TQRx", "summary": "This paper introduces a novel approach for controllable image generation using both graph and text conditions. The authors propose that additional context information from multimodal attributed graphs (MMAGs) can enhance the performance of diffusion models. Specifically, they formulate the Graph2Image problem and develop the INSTRUCTG2I model to incorporate contextual information during the generation process. Empirical evaluations demonstrate the strong performance of the model.", "questions": "1. Are there any unique scenarios where only graph input can significantly improve SD performance?\n\nAs my review is overdue, I welcome concise feedback and am open to clarifying any potential misunderstandings.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to follow.\n2. The intuition behind the approach is clear.", "weaknesses": "1. The overall setting is questionable. The authors integrate graph information using a Graph-QFormer and context information such as artists and genres, stored in graph prompt tokens. Given the large graph size, they only use subgraph structures. Consequently, the Stable Diffusion (SD) model absorbs additional information from similar artworks, which could be derived from image or text prompts alone. This raises the question of whether an additional condition structure is necessary. I suggest the authors demonstrate a unique application where standard models with text and image prompting capabilities are insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your thoughtful review!\n\nRegarding your questions:\n\n1. **Question about the problem setting.**\n\nThank you for your comment. We would like to answer this from two aspects.\n\n**Why graph is important?**\n\n1) *Graph structure helps discover multiple informative neighbors*: We agree with the reviewer that Graph-QFormer is to transfer “neighbor images” into graph prompt tokens for center node generation. However, how to select such “neighbor images” is important and non-trivial, which needs help from graph structure. As shown in Figure 5, the “neighbor images” condition affects the generation results a lot, and our proposed semantic PPR-based sampling can discover informative neighbors based on *graph information* for high-quality image generation.\n\n2) *We utilize global graph structure and semantics*: We would like to clarify that in our method, we do not “only use subgraph structures” but utilize global graph structure as well as text semantics for neighbor sampling. In semantic PPR-based sampling, we first adopt PPR on the *global graph* to discover informative neighbors based on *graph structure* and then adopt semantic search to find more fine-grained informative neighbors (e.g., if we would like to generate a picture of a “horse”, the neighbors related to “horse” will be more useful, as shown in Figure 5). In addition to the qualitative result, we also show the quantitative analysis of semantic PPR-based sampling, where we can find that utilizing global graph structure (Ours in the table below) outperforms utilizing only subgraph structures (- PPR in the table below), which also demonstrates the importance of graph structure information.\n\n| Dataset             | ART500K | Amazon |\n|---------------------|--------|--------|\n| Ours  | 46.45 | 51.70 |\n| - PPR | 45.06 | 48.40 |\n| - semantic ranking | 46.19 | 51.49 |\n\n3) *Not just one, but many and extract their similarity from graph*: Graph enables discovering *multiple* related neighbor images and extracting their similarity rather than solely based on one image to conduct image generation (which is widely adopted in the literature). This is important for image generation in many scenarios. For example, we would like to generate a “bird” picture drawn by “Monet”. If we only conditioned on his picture of a “scenery”, we may overfit the corresponding content and fail to draw a “bird”. However, if we conditioned on a multiple of his images covering diverse content including animals, the model would better extract his style and successfully conduct image generation.\n\n**Why are image or text prompts alone not enough?**\n\n1) *Text is not enough to describe everything*: For example, in the artwork scenario, for the artists who are not that famous (e.g., my little brother), it is hard to use text to represent them; and for the artists who are new and not seen during model training, it is hard to expect the model can be generalized through text (artist names). However, using graph structure as the condition, we can well solve the not famous artist and new artist problem by discovering informative neighbors in the graph. We add an example of image generation for my little brother on the artwork graph to illustrate this in the rebuttal PDF (in general response). \n\n2) *Image conditions need to be discovered from the graph*: As discussed above, neighboring image conditions can affect the generation result a lot, and graph structure can benefit the informative neighbor image sampling. In addition, solely based on image condition will result in unclear content (e.g., if we want to generate a “bird” image connected to the Picasso node, solely based on the sampled neighboring images does not provide the content indicator of a “bird”).\n\n3) *Graph enables flexible and diverse controllable generation*: Another advantage of adopting graph as the condition is that we can combine multiple graph conditions (e.g., different art styles) with the controllable generation (with any mixed ratio according to the user’s interest), which is extremely hard to express by text only or image only. We add an example of controllable generation for a node connecting to “Pablo Picasso” and my little brother on the artwork graph to illustrate this in the rebuttal PDF (in general response).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ca65UCeO7O", "reviewer": "Reviewer_5RsF", "summary": "The paper introduces a new task graph2image which is to generate images conditioned on both text descriptions and graph information, which improves consistency of generated images compared to conditioned only on texts or images. To address combinatorial complexity of graphs and dependencies among graph entities, the paper proposes a graph context-conditioned diffusion model InstructG2I for generating images from multimodal attributed graph.", "questions": "- Why is the DINOv2 score on Goodreads dataset significantly low compared to that of ART500K or Amazon datsets?", "limitations": "Yes the paper addresses the limitation in Appendix A.1.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- To the best of my knowledge, graph2image is a novel task, and the motivation to use the rich and high-dimensional information of graphs for image generation seems reasonable and interesting. \n- The proposed approach to incorporate graph information into pre-trained text-to-image is new, in particular introducing graph conditioning token and considering scalability of graph size. \n- The generated samples show that using graph information results in better consistency with the ground truth compared to methods that use only text prompts or images.\n- Examples of controllable generation with both text and graph show the ability to balance content and style in a simple manner.", "weaknesses": "While I do not have a major concern, an ablation study on scalability to graph size seems to be missing. How large graphs is the method able to be applied?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your thoughtful review and support of our work!\n\nRegarding your questions:\n\n1. **How large graphs can the method be applied?**\n\nThank you so much for your question. Our method can be adapted to large-scale graphs with millions or even trillions of nodes. In InstructG2I, we only need to conduct offline sampling with semantic PPR-based neighbor sampling **once** to extract useful structure information from the large-scale graphs. Since this step is performed offline, the size of the graph will not introduce sampling bottlenecks to stable diffusion model training and inference. Empirically, the semantic PPR-based neighbor sampling only takes about 10 minutes on a graph with millions of nodes, which is quite short compared to stable diffusion training and inference time cost.\n\n2. **Why is the DINOv2 score on the Goodreads dataset lower than the others?**\n\nThank you for your great observation. 1) *Training data size*: As shown in Table 3, the graph size of Goodreads is the smallest compared with ART500K and Amazon. This means that the training data on Goodreads is smaller than the other two datasets. Since a larger training data size can contribute to more sufficient training, the results on ART500K and Amazon are better than the results on Goodreads. 2) *Data distribution*: In ART500K, Amazon, and Goodreads datasets, the images are art pictures, product pictures, and book cover pictures respectively (some samples can be found in the rebuttal PDF in general response). We believe that compared with book cover pictures, product and art pictures are closer to the distribution of images used in stable diffusion (which is our base model) pretraining. In conclusion, training InstructG2I on ART500K and Amazon will provide better performance than training InstructG2I on the Goodreads dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h1c134H6gx", "reviewer": "Reviewer_nWct", "summary": "This paper focuses on the problem of image synthesis on multimodal attributed graphs (MMAGs) and proposes a graph context-conditioned diffusion model, INSTRUCTG2I, to address the challenge in this setting. In particular, it proposes a semantic personalized PageRank-based method to sample related neighbors in the graph.  Then, the INSTRUCTG2I can effectively encode graph conditional information as graph prompts with Graph-QFormer. Systematic experiments on MMAGs demonstrate the effectiveness of the methods proposed in this paper compared to competitive baseline methods.", "questions": "1. Please see the weaknesses.\n2. I wonder if the authors will compare it to other state-of-the-art image generation models, such as some Multimodal LLMs that are so prevalent nowadays.", "limitations": "This paper has reasonably discussed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.  This paper studies an interesting and meaningful question. It investigates the graph-structured relationships of real-world entities for image generation on MMAGs, a task well-grounded in practical applications. \n2. This paper is well-structured and easy to understand.\n3. The graph context-conditioned diffusion model proposed in this paper is reasonable in solving image generation problems on MMAGs.", "weaknesses": "1. The description in eq.10 may be incorrect. Please check more carefully.\n2. Subsection 3.4 is more challenging to understand when reading. The authors' descriptions of some symbols in Eq. 10 and Eq. 11 are not exhaustive.\n3. The results of the ablation experiments in Table 2 indicate that using a GNN such as GAT or GraphSAGE to aggregate graph information seems to be worse than the straightforward approach in Eq.7. Authors are requested to give a more detailed discussion with a reasonable explanation.\n4. The images sampled by the semantic PPR-based sampling shown in Figure 5 appear to have the same image as the ground truth. Does this indicate that the proposed method suffers from label leakage?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your thoughtful review!\n\nRegarding your questions:\n\n1. **The description in Eq.10 may be incorrect.**\nThank you so much for your comment. We have found the typos and will correct them in the revision.\n\n2. **Descriptions of symbols in subsection 3.4.**\nThis section mainly discusses controllable generation with classifier-free guidance (CFG) on graphs. The main philosophy of CFG is that $\\epsilon_\\theta$ learns the gradient of the log image distribution, and increasing the contribution of $\\epsilon_\\theta(c,z_t)-\\epsilon_\\theta(z_t)$ will enhance convergence towards the distribution conditioned on $c$. We agree with the review that the description of the symbols can be improved to make this section easier to understand. We plan to add detailed illustrations to symbols including $\\varnothing$, $z_t$, $s^{(k)}_G$, and others. We would be grateful if the reviewer could provide us with inputs on what other symbols need further explanation and we are glad to make this section clearer in the revision.\n\n3. **Why do GNNs underperform the straightforward approach in Eq.7?**\nThank you for the comments. The philosophy of vanilla GNN is to propagate and aggregate information from neighbors and compress the neighboring information into *one* embedding for the center node, while the straightforward approach in Eq.7 will directly provide neighboring information separately without aggregation or compression. It is worth noting that the aggregation/compression step could lead to neighbor information loss, while the straightforward approach in Eq.7 does not. Since more information is provided to the stable diffusion model, the latter can perform better for image generation. However, we believe that future work can consider a more advanced design of GNN methods that could keep all the neighboring information and extract the common knowledge for more accurate image generation on graphs.\n\n4. **The images sampled by the semantic PPR-based method appear to have the same image as the ground truth. Does this indicate any label leakage?**\nWe are sorry for the confusion. This is a typo and we will correct it in our revision. We have written data processing codes to ensure that there is no label or data leakage in both training and testing. To demonstrate this, we have uploaded the data processing code here: https://anonymous.4open.science/r/Graph2Image-submit-607E/art500k_dp.ipynb\n\n5. **Comparison with other SOTA multimodal LLMs.**\nThank you for your comments. We would like to argue that the philosophy of introducing graph information into image generation can be not only adopted to the diffusion model (in our paper) but also to any multimodal LLM backbones as mentioned by the reviewer. However, we agree with the reviewer that adding the comparison can make the experiment more comprehensive. As a result, we compare with a recent advanced multimodal LLM called DreamLLM [1] and show the results below:\n\nThe image-image CLIP score:\n\n| Dataset             | ART500K | Amazon | Goodreads |\n|---------------------|--------|--------|------|\n| DreamLLM  | 57.16 | 60.72 | 40.72 |\n| Ours | 73.73 | 68.34 | 50.37 |\n\nThe image-image DINOv2 score:\n\n| Dataset             | ART500K | Amazon | Goodreads |\n|---------------------|--------|--------|------|\n| DreamLLM  | 27.06 | 34.77 | 16.13 |\n| Ours | 46.45 | 51.70 | 25.54 |\n\nFrom the results, we can find that our method can outperform the advanced DreamLLM consistently, which demonstrates the effectiveness of our model design.\n\n[1] DreamLLM: Synergistic Multimodal Comprehension and Creation. ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "znQXTCns5r", "reviewer": "Reviewer_7dPX", "summary": "The authors propose an approach to enhance image synthesis using multimodal attributed graphs, adopting a strategy to condition image generation via a tokenization scheme on graph structure.", "questions": "- Typos: \n  - Line 18: \"graph-structued\"\n\n- The motivation proposed in lines 28-30 is a little bit confusing, since the scenario the authors discuss here (e.g. virtual artwork creation based on nuanced styles of artists and genres) seems like it could be well-handled by text rather than explicitly using graph structure.   \n\n- There is limited prior work in multimodal graph learning, as the authors mention.  The authors may want to reference and position their work with respect to the recent [1] which offers multiple datasets and focuses on utility of GNN methods for node/link/graph-level tasks rather than generative tasks.\n\n- Nit: the notation is a bit awkward compared to conventional graph literature which typically uses $\\mathcal{V}, \\mathcal{E}, \\mathcal{X}, \\mathcal{F}$ or something similar to indicate node-set, edge-set, node-features, and edge-features.  The authors proposed notation in line 72 seems to define P and D as different sets of images / documents compared to the nodes V, but then mentions that each node has some textual information and image information corresponding to P and D (it should be made clear whether this information is just features, or actual node relationships -- if the latter, it seems that P and D should be contained within the nodeset V). \n\n- The process described in line 115 around introducing tokens to help condition the representation using graph structure is also explored in some related works, e.g. [2].  Perhaps the authors could consider adopting a similar approach if it makes sense in this task, since the tokenization scheme as the authors of [2] point out is key in injecting the right level of information to the model.\n\n- Comment: the notation in pages 3-5 is quite heavy and would benefit from a symbol table.\n\n- Section 3.2 proposes a heuristic solution for neighbor selection.   I'd encourage exploring solutions designed for learnable importance of multiple edge types similar to [3].\n\n- Can the authors discuss what sort of techniques were used to incorporate graph conditions for the baseline models like InstructPix2Pix and SD?\n\n- Is there a quantitative understanding or experiment for the PPR neighbor based sampling approach?  It seems this is one of the more heuristic parts of the paper where the design of the sampling procedure (two phase PPR + semantic re-ranking) is less conventional and deviates from other aggregation mechanisms explored in previous literature like attention-based selection, learnable importance of multiple edge types, etc.  The qualitative experiment is helpful but not terribly convincing in terms of the actual performance impact in aggregate.\n\n[1] Multimodal Graph Benchmark (Zhu et al, 2024)\n\n[2] LLaGA: Large Language and Graph Assistant (Chen et al, ICML 2024)\n\n[3] Pathfinder Discovery Networks for Neural Message Passing (Rozemberczki et al, WWW 2021)", "limitations": "Yes, Appendix A.1", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper studies an intersectional topic: leveraging graph learning techniques for image generation, which is a creative application and an area which deserves more focus.\n- The authors' use of qualitative examples (e.g. Figure 5 and 6) is commendable and helps articulate visual improvements.", "weaknesses": "Please see questions and concerns below.  My general feeling is the paper is fairly incremental in its introduction of a mechanism to encode graph condition into the conditioning for generation.  Many design choices for graph conditioning are not discussed well and the quantitative results for some of these choices are missing which hurts the overall impact of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your thoughtful review!\n\nRegarding your questions:\n\n1. **The scenario the authors discuss in lines 28-30 seems like it could be well-handled by only text.**\nWe would like to answer this question from three aspects. 1) On one hand, the problem introduced in this paper can be grounded not only in the virtual artwork scenario but also in others such as the e-commerce scenario, where generating an image for a product node connected to other products equates to recommending future products. Such scenarios are hard to handle by text only (e.g., user interests are implicitly expressed in their purchase history which is hidden in the graph structure). 2) On the other hand, even in the artwork scenario, for the artists who are not that famous (e.g., my little brother), it is hard to use text to represent them; and for the artists who are new and not seen during model training, it is hard to expect the model can be generalized through text (artist names). However, using graph structure as the condition, we can well solve the not famous artist and new artist problem by discovering neighbors in the graph. We add an example of controllable generation for a node connecting to “Pablo Picasso” and my little brother on the artwork graph to illustrate this in the rebuttal PDF (in general response). 3) Another advantage of adopting graph conditions is that we can combine multiple graph conditions (e.g., different art styles) with controllable generation (with mixed ratio according to the user’s interest), which is extremely hard to express by text only.\n\n2. **Add a reference to the recent [1].**\nThank you for bringing this related work. We would like to mention that the reason why we do not reference this paper [1] in our submission is that *it is put on arxiv later than the neurips submission deadline*. However, as the reviewer mentioned and we agreed, this is a related paper and we would like to reference it in our revision.\n\n3. **Question about the notations, especially P, D, and V.**\nWe agree with the reviewer that since this paper tackles multimodal learning on graphs, the notations will be a little different and complex compared with conventional graph literature. As mentioned in the paper, each node $v_i\\in V$ is associated with a text $d_{v_i}\\in D$ and an image $p_{v_i}\\in D$. In other words, $d_{v_i}$ and $p_{v_i}$ can be understood as features for $v_i$ from the conventional graph learning perspective. We will make it clear in the revision.\n\n4. **Perhaps the authors could consider adopting a similar approach [2] if it makes sense in this task.**\nThank you for bringing up this great paper and we are glad to reference it in the revision. At the same time, we would like to answer this question from three aspects: 1) *The design of Graph-QFormer is non-trivial*: as we mentioned in the paper, in MMAGs, we need to tackle the graph entity dependency (including image-image dependency and text-image dependency) to conduct image generation. To extract such dependency, we design Graph-QFormer and show consistently better performance than other designs in Table 2; 2) *Different task with different backbone models*: we agree with the reviewer that LLaGA is a great work, but we would like to emphasize that the problem tackled in LLaGA (text generation on TAGs with LLM) is different from that in this paper (image generation on MMAGs with stable diffusion). Since the problem and the backbones are different, it is non-trivial to directly compare these two methods; 3) We have compared with simple GNN tokenization methods in the ablation study (Table 2) and demonstrate the effectiveness of Graph-QFormer compared with them. \n\n5. **Comment: the notation on pages 3-5 is quite heavy and would benefit from a symbol table.**\nWe will add a symbol table in either the main content or the appendix in the revision according to your suggestion.\n\n6. **Learnable importance of multiple edge types similar to [3].**\nWe agree with the reviewer that learnable sampling is a promising future direction. As we are the first paper to introduce this problem, we would like to tackle this problem in a simple and effective way. As a result, we would like to leave this more complex learnable sampling for future studies.\n\n7. **How graph conditions are introduced to InstructPix2Pix and SD?**\nWe compare our method with InstructPix2Pix and SD with an advanced design for graph information in Table 2. For InstructPix2Pix, we aggregate the neighboring images on the graph with mean pooling and serve the resulting image as the input image condition. For SD, we concatenate the text information from the center nodes’ neighbors on the graph to its original text as text condition. From Table 2, our method outperforms both baselines significantly which demonstrates the effectiveness of the InstructG2I design.\n\n8. **A quantitative understanding of semantic PPR-based neighbor sampling approach.**\nWe are glad to add some quantitative results (DINOv2 score) on ART500K and Amazon datasets as below:\n\n| Dataset             | ART500K | Amazon |\n|---------------------|--------|--------|\n| Ours  | 46.45 | 51.70 |\n| - PPR | 45.06 | 48.40 |\n| - semantic ranking | 46.19 | 51.49 |\n\nFrom the result, we can see that both PPR-based sampling and semantic-based reranking can benefit the InstructG2I. This is demonstrated in both quantitative and qualitative results (Figure 5). The reason we adopt semantic PPR-based sampling rather than attention-based selection and others is that semantic PPR-based sampling can be performed offline only once and it is computationally efficient compared with SD training and inference. This enables us to scale to large-scale graphs in the real world. However, we agree with the reviewer that attention-based selection and others are interesting directions and we would like to leave them for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We sincerely appreciate your valuable feedback and suggestions. We will revise our work based on your reviews. 

We also want to thank the Reviewers for noting the strengths of our paper, namely:

- The problem addressed in our paper is important and well-motivated. (5RsF, nWct, 7dPX)
- Our proposed method is substantial and modern. (TQRx, 5RsF, nWct)
- The paper is clearly written. (TQRx, nWct)
- The empirical results are consistent, solid, and convincing. (TQRx, 5RsF, 7dPX)
- The method offers balanced controllable generation. (5RsF)

We have addressed the individual questions of reviewers in separate responses. In the revised version, we will incorporate all reviewers' suggestions by making the motivation more solid, adding more experimental results/discussion, adding references, and making the symbols more clear. **We have also attached a rebuttal PDF below.**

Here we would like to briefly outline the contribution of this work for the reference of reviewers to start the discussion.

- (Formulation and Benchmark). We are pioneers in recognizing the potential of multimodal attributed graphs (MMAGs) for image synthesis, and we have introduced the Graph2Image problem. Our formulation is validated by three benchmarks based on practical applications in art and e-commerce. Those benchmarks will be valuable for future research.
- (Algorithm). Methodologically, we introduce InstructG2I, a context-aware diffusion model that adeptly encodes graph conditional information as graph prompts for controllable image generation (as illustrated in Figure 1(b,c) and the experiment section).
- (Experiments and Evaluation). Empirically, we perform experiments using graphs from three distinct domains, showing that InstructG2I consistently surpasses competitive baselines (as illustrated in Figure 1(b) and the experiment section).

In closing, we thank the Reviewers again for their time and valuable feedback. If there are further concerns, please let us know, and we will be happy to address them.

Meta-review: The paper introduces the Graph2Image problem and develops the InstructG2I model to incorporate graph structure and multimodal information for better image generation. Empirical evaluations demonstrate the strong performance of the model. The paper is well-written and easy to follow. Reviewers are generally positive about this work. They agree that the paper studies a novel task and proposes a reasonable approach. The authors are suggested to address reviewers' comments, e.g., clearer motivation discussion and notations, for paper revision. 

Based on the above summary, I recommend acceptance.
Decision: Accept (poster)