Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Models
Authors: Bicheng Xu
Qi Yan
Renjie Liao
Lele Wang
Leonid Sigal
Venue: Accepted by TMLR ()

Abstract: A grounded scene graph represents a visual scene as a graph, where nodes denote objects (including labels and spatial locations) and directed edges encode relations among them. In this paper, we introduce a novel framework for joint grounded scene graph - image generation, a challenging task involving high-dimensional, multi-modal structured data. To effectively model this complex joint distribution, we adopt a factorized approach: first generating a grounded scene graph, followed by image generation conditioned on the generated grounded scene graph. While conditional image generation has been widely explored in the literature, our primary focus is on the generation of grounded scene graphs from noise, which provides efficient and interpretable control over the image generation process. This task requires generating plausible grounded scene graphs with heterogeneous attributes for both nodes (objects) and edges (relations among objects), encompassing continuous attributes (e.g., object bounding boxes) and discrete attributes (e.g., object and relation categories). To address this challenge, we introduce DiffuseSG, a novel diffusion model that jointly models the heterogeneous node and edge attributes. We explore different encoding strategies to effectively handle the categorical data. Leveraging a graph transformer as the denoiser, DiffuseSG progressively refines grounded scene graph representations in a continuous space before discretizing them to generate structured outputs. Additionally, we introduce an IoU-based regularization term to enhance empirical performance. Our model outperforms existing methods in grounded scene graph generation on the Visual Genome and COCO-Stuff datasets, excelling in both standard and newly introduced metrics that more accurately capture the task’s complexity. Furthermore, we demonstrate the broader applicability of DiffuseSG in two important downstream tasks: (1) achieving superior results in a range of grounded scene graph completion tasks, and (2) enhancing grounded scene graph detection models by leveraging additional training samples generated by DiffuseSG. Code is available at https://github.com/ubc-vision/DiffuseSG.

Published in Transactions on Machine Learning Research (08/2025)
Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Models
Bicheng Xu" University of British Columbia Vector Institute for A1
bichengn@cs.uber Suc.gprrcssub..co
Qi Yan'
qi. pi.yan&ece..bee ubc...r
University of British Columbia Vector Institute for Al
Renjie Liao
rjliao@ece. ubc.ca
University of British Columbia Vector Institute for AI Canada CIFAR AI Chair
Lele Wang
lelewong@ece.uke ubc.
Universily of British Columbia
Leonid Sigal
Isigal Mcs. ubc.
University of British Columbia
Vector Institute for Al Canada CIFAR Al Chair
Reviewed on OpenReview: https: // openreview. oppere/veeee net/ forum? id= 2cweZI2LO1
Abstract
grounded scene graph represents a visual scene as a graph, where nodes denote objects (including labels and spatial locations) and directed edges encode relations among them. In this paper, we introduce novel franework for joint grounded scene graph image gen- cration, challenging task involving high-dimensional, multi-modal structured data. To effectively model this complex joint distribution,
adopt factorized approach: first gen- erating a grounded scene graph, followed by image generation conditioned on the generated grounded scene graph. While conditional image generation has been widely explored the literature, our primary focus is on the generation of grounded scene graphs from noise, which provides efficient and interpretable control over the image generation process. This task re- quires generating plausible grounded scene graphs with heterogeneous attributes for both nodes (objects) and edges (relations among objects), encompassing continuous attributes (e.g., object bounding boxes) and discrete attributes (e.g., object and relation categories). To address this challenge, we introduce DiffuseSG, novel diffusion model that jointly mod- els the heterogeneous node and edge attributes. We explore different encoding strategies to effectively handle the categorical data. Leveraging a graph transformer as the denoiser, DiffuseSG progressively refines grounded scene graph representations a continuous space before discretizing them generate structured outputs. Additionally, we introduce an loU- based regularization term lo enhance empirical performance. Our model outperforms exist- ing methods in grounded scene graph generation on the Visual Genome and COCO-Stufi datascts, excelling in both standard and newly introduced metrics that more accurately capture the task's ''s complexity. Furthermore, we demonstrate the broader applicability of DiffuseSG in two important downstream tasks: (1) achieving superior results in a range of
Equal contribution

---

Published in Transactions on Machine Learning Research (08/2025)
Grounded Scene Graph Generation
Image Generation
Grounded Scene Graph Generation
Image Generation
har
bow
6ar-1
66r-2
man
shoe
main
6ar-1
near
has
bowi
wearing
kateboart
joan
paw-1
cat
panw-2
hair
joan
Skato-
Figure 1: Joint grounded scene graph and image modeling. We model the joint distribution of grounded scene graph image pairs via two steps: first training proposed DiffuseSG model to produce grounded scene graphs and then utilizing a conditional image generalion model to generate images. Lay- outDiffusion (Zheng et al. 2023b is used to generate images in these examples. Results shown are sampled from models trained on the Visual Genome dataset (Krishna et al. 2017
grounded scene graph completion tasks, and (2) enhancing grounded scene graph detection models by leveraging additional training samples generated by DiffuseSG. Code is available at https: cow http://hwww....oeeieeeee//////aa
1 Introduction
A grounded scene graph is a graph-based representation that caplures semantics of the visual scene, where nodes correspond to the objects (including their identity/labels and spatial locations) and directed edges
correspond to the spatial and functional relations between pairs of objects. Grounded scone graphs have
been widely adopted in a variety of high-level tasks, including image captioning (Yang al. 2019 Zhong
et al. 2020) and visual question answering (Damodaran al.] 2021 Qian et al. 2022). Various models
Kundu & Aakur 2023 Jung et al. 2023 Zheng et al. 2023a Jin et al. 2023 Biswas & Ji 2023 Li et al.
2024 Hayder & He 2024) have been proposed to detect grounded scene graphs from images. Such models
require supervised training with image grounded scene graph pairs, which is costly to annotate.
Motivated by this and the recent successes of diffusson models, in this paper, we tackle the problem of joint generative modeling grounded scene graphs and corresponding images via diffusion models. The benefits of such generative modeling would be multifaceted. First, it can be used to generate synthetic training data to augment training of discriminative grounded scene graph detection approaches discussed above. Second, it can serve generative scene prior which can be tasked with visualizing likely configurations of objects in the scene conditioned on partial observations via diffusion guidance. For example, where is the likely position of the chair given placement of the table and sofa. Third, it can be used for controlled image generation, by allowing users to sample and edit grounded scene graphs, and then conditioned on them, generale corresponding images.
Diffusion models (Soks-Dickssein et al. 2015 Ho al. 2020 Song al. 2021 | have been shown to excel
modeling complex distributions, generating realistic high-resolution images (Saharia al. 2022 Rombach
et al.. 2022 Nair et al. 2023 Ruiz et al. 2023 Epstein et al. 2023) and graphs (Jo et al. 2022 Vignac et al.
2022
Yan et al. 2023). However, joint generation of an image and the corresponding graph representation is
challenging. To simplify the task, we factorize the joint distribution as a product of grounded scene graph prior and a conditional distribution of images given a grounded scene graph. The conditional distribution
has been widely studied the form of layout-to-image generation (Zhao et al., 2019 Sun Wu 2019 2021 Zheng et al.] (023b), task of generating images based on spatial layouts that can be constructed from grounded scene graph:] Therefore, in this paper we mainly focus on modeling the first term, i.e., building generative grounded scene graph model. This task in itself is challenging it requires generation of graphs with heterogencous attributes, e.g., real-valued node attributes like object bounding boxes and categorical edge altributes like relation types.
Thouug, notably, the process, edge information often not utilized.
2

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Characterization of related, but often conflated, concepts encountered in the literature.
The representations used to describe the visual scene are: layout, scene graph, and grounded scene graph.
Attributes Contained
Concept
Object Label
Object Location
Relation Label
Layout
Scene Graph
Grounded Scene Graph
To generate grounded scene graphs, we propose DiffuseSG which a diffusion-based generative model capable of generating plausible grounded scene graphs. To deal with heterogeneous attributes, we explore various encodings of categorical node/cdge representations. We also design a graph transformer architecture that successively denoises the continuous graph representation which in the end produces clean grounded scene graph samples via simple discretization. Moreover, we introduce intersection-over--uiin (loU)-based training loss to better capture the distribution of bounding box locations and sizes. To generate images conditioned on grounded scene graphs, besides using one existing layout-to-image generation model, we also experiment with a relation-aware image generator built upon ControlNet (Zhang et al. 2023b
In summary, our contributions are as follows. (1) We propose a novel joint grounded scene graph - image generation framework, by factorizing the joint distribution into grounded scene graph prior and condi- tional distribution of images given the grounded scene graph. In this context, propose diffusion-based model, named DiffuseSG, for grounded scene graph generation, which jointly models node attributes like object classes and bounding boxes, and edge attributes like object relations. Particularly, the technical no- valties lie in the tensor representation of the grounded scene graph, separate prediction heads for node and odge attributes, an IoU-basod training loss, and exploration among different encoding mechanisms for the calegorical data. (2) We show that our model significantly out outperforms existing unconditional scene graph generation models, layout generation models, and general-purpost graph generative models on both standard and newly introduced metrics that better measure the similarity between observed and generated grounded scene graphs. (3) We show that our model performs well on various grounded scene graph completion tasks using diffusion guidance. Moreover, paired with conditional image generation model, our model generates grounded scene graph image pairs which serve as extra training data for the downstream grounded scene graph detection lask. The observed performance improvement highlights the practical significance of our joint modeling framework in real-world applications.
2 Representation Taxonomy
There are three similar but distinct visual scene representations that are often referenced in the literature. We formally define them here and in Tab. [ to avoid conflating them. We strictly follow these definitions for the remainder of the paper.
Layout. Layouts comprise objects and their corresponding bounding boxes. There is no encoding of semantic relations between these objects beyond those implied by their placement. This representation is widely explored in the layout generation task, and utilized in the task of image generation from layouts.
Scene Graph. Scene graphs encode object labels and their corrosponding relations the form of dircctod graphs. As such, a scene graph can encode that a certain object (e.g., a car) is neal lo another object, but it not precisely localize those objects in scene. This representation is mainly used in the scene graph-to- image generation task. More closely related to our task, some works generate scene graphs unconditionally.
Grounded Scene Graph. Grounded scene graphs can be viewed as a combination of layouts and scene graphs as defined above. A grounded scene graph is a directed graph where nodes correspond to objects encoded by labels and bounding boxes, and odges correspond to relations. This representation extensively studied in the literature on grounded scene graph detection from images. In this paper, we propose DiffuseSG for the task of unconditional grounded scene graph generation.
3

---

Published in Transactions on Machine Learning Research (08/2025)
3
Related Works
Diffusion Models. Diffusion models achieve great success in a variety of generation tasks nowadays, ranging
from image generation Kumari et al. 2023 Kim & Kim 2024 Miao et al. 2024
Zeng et al.
2024b Qu et al.
2024
Lin et al.
2024 Wei et al. 2024), video generation (Sun et al.
2023
Zeng et al.
2024a
Qing et al.
2024
Wang et al.
2024a Skorokhodov et al.
2024
Liang et al.
2024
Gupta et al. 2024
Melnik et al. 2024
Liu et al.
2024b).
, text generation (Li et al.
2022b
Gong et al. 2022
Yuan et al., 2022 Dicleman ct al. 2022
Ye el. al.
2023
Lin el al.) 2023 Wu el al.
2023).
to simple graph generation (Jo el al.) 2022 Vignac el al.
2022
2024), e.g., on molecule datasets (Irwin 2014), which usually have less than 10 node types and less than 5 edge types. These models contain two key processes: forward process, which typically involves adding Gaussian noise to clean data, and a reverse denoising process that is often implemented using architectures such as U-Net (Ronneberger et al.) 2015)) or transformer (Vaswani et al.) 2017). Since the grounded scene graph is fundamentally a graph structure, our proposed grounded scene graph generation model is conceptually similar to the ones for simple graph generation. However, our model goes beyond merely generating the graph structure of node (object) and edge (relation) labels. It also generates the object locations, in the form of bounding boxes. Also, our grounded scene graph data is more complex than the molecule data in terms of the numbers of node and edge types, e.g., the Visual Genome dataset (Krishna et al. 2017) contains 150 node and 50 edge types. This diversity necessitates re-evaluation of many design choices traditionally made in graph generation models.
Jo et al. 2023; Yan et al.
2023 Cho al. 2024 Xu et al.
ct al.
2012 Ramakrishnan et al..
Layout Generation. Layout generation focuses on creating image layouts, which comprise object labels and their corresponding bounding box locations. In contrast, our proposed grounded scene graph generation goes a step further by also generating the relations among these objects. Existing layout generation models
typically take the form of VAF (Jyothi et al., 2019 Lee et al., 2020: Arroyo et al.] 2021) ) GAN (Li et al.
2020a b), transformer or BERT type language models (Gupta et al. 2021
Kikuchi et al. 2021 Kong et al.
2022 Jiang et al. 2023). or diffusion models (Inoue ct al. 2023 Chai et al. 2023 Hui et al. 2023 Levi et al.
2023
Zhang et al. 2023a Shabani et al. 2024). These layout generation models usually ly work on graphical
layout generation problems, G.g., designing layouts for mobile applications (Deka et al. 2017 Liu et al. 2018).
documents (Zhong et al. 2201)), or magazines (Zheng et al. 2019). The object bounding boxes of these layouts are expected to be well-aligned and not overlapping with each other. Thus, the layout generation models are usually measured on the alignment and intersection area of the generated bounding boxes. However, the object bounding boxes in our grounded scene graphs are naturally not aligned and usually occlusions occur. Therefore, we replace the evaluation metrics used in the layout generation literalure with new ones that better capture characteristics of grounded scene graphs.
Unconditional Scene Graph Generation. Garg et al. 20221 introduce the task of unconditional scene graph generation, where a scene graph is generaled from noise. They propose an auloregressive model for the generation, first sampling an initial object, then generating the scene graph in a sequence of steps. Each step generates one object node, followed by sequence of edges connecting to the existing nodes. A follow-up work (Verma et al., 2022 proposes variational autoencoder for this task, where scene graph is viewed as collection of star graphs. During generation, it first samples the pivot graph, and keeps adding star graphs to an existing set. The scene graph in these works is defined to only have object and relation category labels; object bounding box locations are omitted. Compared with these works, we use a diffusion based model for unconditional grounded scene graph generation, where node and edge attributes are generated at the same time. We also include the object bounding box location in our grounded scene graph representation.
Image Generation from Layouts or Scene Graphs. There exist two conditional image generation tasks in the literature: layout-to-image (Zhao et al. 2019) and scene -oapg- {Johnson et al. 2018 The conditions in the layout-to-image task are object labels and their bounding box locations, while the conditions in the task of scene graph-to-image are the object labels and relation labels. These lasks were
initially widely explored within the GAN framework (Li et al. 2019 Ashual & Wolf 2019 Sun & Wu 2019)
Dhamo et al. 2020 Li et al. (202bb; Sun & Wul 2021 He et al.
2021 Sylvain et al. 2021). With the increasing popularity of diffusion models and their excellence at modeling complex distributions, diffusion
models now become the default. choice to accomplish these tasks
(Yang et. al. 2022
Cheng et al.] 2023
4

---

Published in Transactions on Machine Learning Research (08/2025)
Liu et al. 2024a Wang al. 2024b). However, the grounded scene graphs considered in our task contain object labels, object locations, and relation labels. These attributes can not be encoded altogether into any of the existing image generation models. This motivates us to build and explore diffusion-based relation-aware layout-to-image model.
Zheng et al. 2023b Farshad al. 2023 Liu & Liu 2024
4 Joint Grounded Scene Graph Image Pair Modeling
We propose to model the joint distribution of grounded scene graphs and their corresponding images by modeling it as product of two distributions. Denote the grounded scene graph by S and the image by 1 The joint distribution of grounded scene graph and image pairs can be factorized as p[S, = [T)) from which one can easily draw samples in a two-step manner; first from the prior p(S) and then from the conditional p(I ()) Hence, we first build a grounded scene graph generation model to learn p(S); i.e., the underlying prior of grounded scene graphs. Sccond, we employ conditional image generation model caplure p S), the conditional image distribution.
4.1 Grounded Scene Graph Generation
In this section, we first formally define our proposed grounded scene graph generation task. We then present our DiffuseSG model specifically designed for the task. DiffuseSG is a continuous diffusion model, whose training and sampling utilize the stochastic differential equation (SDE) formulation. We begin by providing some background on the SDF-bbaee diffusion modeling, followed by an in-depth explanation of DiffuseSG.
4.1.1 Grounded Scene Graph Generation Task
A grounded scene graph S, consisting of n nodes, can be described using node and edge tensors, denoted as (V. E). We denote the space of node features by V and the space of edge features by E, and the space of grounded scene graphs by S = V x E. The node tensor V= = [V1; U2i...; Un. E Raxd., captures the node labels and their bounding box locations, where d, represents the dimension of the node feature. Each node feature v, [c,, b,] combines a discrete node label, G 21.2..... Zz}, with normalized bounding box position, b, 1]* The bounding box b, is represented by (centerz, center,, width, height) and normalized w.r.t. the image canvas size. The edge tensor E E Rax*, details the directed edge relationships among the nodes. Each edge entry eij corresponds to a discrete relation label, Ci2j 1....... Ze), clarifying the connections between nodes. The symbols Zo and Ze represent the total numbers of semantic object categories and relation categorics of interest, respectively. Notably, Ci2 = indicates the absence of a relation between nodes and j. The task is generale such grounded scene graphs from noise.
4.1.2 Diffusion Model Basics
Preliminaries. Diffusion models Ho et al. 2020 Song et al.) 2021 learn probabilistic distribution po( through matching the score functions of the Gaussian noise perturbed data distribution Va log at various noise levels a E {o,) Following Song et al. (2021); Karras et al. (2022). we use the SDE-based diffusion model for training and sampling, which comes with a continuous time t E [0,T] specified by the
following dynamics:
dx+ = f(x, t)dt + g(t)dw,
(1)
dx_ [f t)dt g(t)²Va log (x) + g(t)dw,
(2)
where Eq. and Eq. denote forward and reverse SDEs, fix i) and g(l) are the drift and diffusion coefficients, and w is the standard Wiener process. The SDEs govern how the probabilistic distribution (x) evolves w.r.t. time t. Specifically, Po(x) is the data distribution, from which we observe a set of i.i.d.
samples X' fm (x models a tractable prior distribution, i.e., Gaussian, from which we can draw
PIn what follows,
Ia for identity matrix and for image data.
We symbols x, this section introduce preliminaries of diffusion model general, regardless of the type of data being modeled.
5

---

Published in Transactions on Machine Learning Research (08/2025)
Sover Ri
Fig. Fg.
fhwerr vass, making
Sounding
Nade MULP
(PA 163)
(122200.226000
g B
Sower
glos
Liwer
SM Alamian
Stah Arteddoo
Dont
Us
213 Atantcon
21 Alaricn
Graph Transtormer
Illustration of the graph transformer during training. Object and relation cate- gories are illustrated using binary-bit encoding. Object bounding box positions are represented by (center,, centery, width, height).
Figure 2:
samples elficiently. In our formulation, we choose linear noise schedule o(I) I, and let [xx () and g(() (2 The SDEs solution yields that Pt in Eq. (2) becomes Po (x) pt(x)
E M a,, o² Ia). E X. Let (x) = Em 5(x - xx) be the Dirac delta distribution for X. We can rewrite Po Po (z) = (x))) ( dae with a Gaussian perturbation kernel Po ( = N (ā; o²;
Training. We train a neural network to learn the score function of Pa (i.e., V (ā))). Following Vincent (201 Karras al. 2022), we reparameterize the score function by denoising function D(t, o) which maps the noise- corruptd data back to the clean data æ. They are connected by Tweedie's formula. (Efron 2011), Va (a (D( o) x)/o². In practice, score function. Given specified distribution over the noise level, denoted as p(ar), which also corresponds to the distribution of forward time t since o(t) t, the overall training objective can be formulated as follows,
train denoiser Do to implicitly capture the
Ep(o)ex (w)p. (a
(a or) - 2013|
(3)
Sampling. To draw samples using the learned diffusion model, we discretize the reverse-time SDE in Eq. (2) and conduct numerical integration, which gradually transitions samples from prior distribution pr to data distribution Po- We choose a of discrete time steps (t+] at which the score function evaluated using the trained model for numerical reverse-SDE solution. We employ second-order solver based Heun's
method (Sūli & Mayers 2003
Karras et al. 20221-
4.1.3 DiffuseSG for Grounded Scene Graph Generation
We model the grounded scene graph distribution p(S) with a continuous state diffusion model. Our model captures the distribution of grounded scene graph topology along with node and edge attributes simultane- ously.
Denoising Objective. During training, we draw noisy grounded scene graph samples from the perturbed distribution Po = (s(S)pp(S)S)dd and train a denoiser Do(S, a) to output the associated noise-free samples S. We relax node and edge label distributions to contimuous space, enabling SDE-based diffusson modeling with smooth Gaussian noise on all attributes. Various methods for encoding discrete labels will be introduced in Section "Encoding Discrete Data" below. Specifically, $ -V, Æ)) denotes the noisy grounded scene graph and PS) Σn 5 - 5,) is the Dirac delta distribution based on training data {S.) We implement the grounded scene graph Gaussian perturbation kernel p> (5) by independently injecting noise to node and edge attributes, í.e., Po ($) p (V V) E) The decomposed kernels are both simple Gaussians: Po(VVV = VVV V.. o² Ia), Po EE = NEE; o La Further, we design denoising network Da with two prediction heads Dý and Dø dedicated to node and edge attributes respectively (detailed in Section "Network Design" below). Our grounded scene. graph denoising loss now becomes:
VVV.V.
Ls = (90 ($,o) - vI DI (S E13
(4)

---

Draw one grounded scene graph sample Draw diffusion timestamp Published in Transactions on Machine Learning Research (08/2025)
Algorithm 1 DiffuseSG Training Process.
Require: node denoiser D edge denoiser DØ diffusion time distribution p(a), dataset A'.
1: repeat
2: sample S X
sample p(o)
(V, V,E)+ S
D Obtain the node and edge lensors
B+V
Obtain the bounding box positions
6: V Pt- V + +o0 with 6 ~~ I)
D Add noise to the node tensor
7: È +-+ E 402 with 62 ~ I0,,
> Add noise to the edge tensor
$ + EV.E)
8:
V D5(5,0)
< Denoise the node tensor
10:
É (5,
Denoise the edge tensor
11:
L IV -V + | El
Eq. (4)
12:
B + V
P Obtain the denoised bounding box positions
13:
Liou = 1 Σ. GloU( Bi)
> Eq. (5)
14:
update via
Vo (Cs + \Liou)
Optimization step
15:
until converged
Network Design. To effectively capture the complex distribution of grounded scene graphs, we develop a transformer architecture, named graph transformer, as the denoiser Dg. Given a noisy grounded scene graph $ (V, E) as input, we construct triplct representations (í.e., generalizod edge representations) by concatenating the subject node, object node, and relation information i [ō,, Gy, ēi.]. Vi, E [m]. where is the number of nodes in the grounded scene graph. The denoising task is essentially node and edge regression in continuous space, trained with stochasticity. For expressive graph representation learning in this context, we consider message passing among all (nn²) triplets as suggested in Morris et al. 2019 2021). Here, each triplet becomes a unit of message passing. However, a naive triplet-to-triplet message passing implementation is space-consuming (O(n*) messages). Inspired by Liu et al. (2021). we employ an approximate triplet-to-triplet message passing using shifted-window attention layers with a window size M, which reduces the space complexity M2?. When pardtoopprtitiiniig is repeated adequately, e.g., at least O(nMM) times, all triplct-to-triplet interactions can be effectively approximated. We tokenize the noisy triplets using linear layer each entry in Q, resulting in x n. triplet tokens of dimension de, represented as Q₂ E Rxxxx,, Our graph transformer then employs repeated hhifeed-innoor attention and downsampling upsampling layers to update the dense triplet-token representations, for predicting the noiso-free node and edge attributes. To generate node and edge attribute predictions of distinct shapes, we employ two MLPs as readout layers for node and edge respectively. The node denoiser Dý and the edge denoiser DE share identical intermediate feature maps and have the same parameters up the final readout layers. The network architecture is further detailed in Appendix A.2
Encoding Discrete Data. To find an appropriate representation ion for the categorical labels, denoting the node label representation as & E Rd and the edge label representation as e E IRd we explore three distinct encoding methods. (1) Scalar: both node and edge labels are expressed as scalar values, with d, = d, = 1. (2) Binary-Bit Encoding: the discrete type indices of node and edge labels are converted into their binary format, represented as a sequence of Os and Here, de [log (Z,)) and de = flogz (Z, + 1)]- (3) One-Hot Encoding: the scalar labels are transformed into their one-hot representations, resulting in de Zu and d, = During training, after encoding the node and edge labels, along with the bounding box positions, inject noise and form the noisy Q. Fig. illustrates the pipeline of our graph transformer during training. During sampling, we start with a Gaussian noised Q, where the node number can either be specified or drawn from some given distribution. After the denoising process, we discretize the continuous-valued representations of node and edge types, e.g., through thresholding for the binary-bit encoding. Detailed explanations of these categorical encodings and discretizations during sampling are in Appendix A. Note that the bounding box positions b, are naturally continuous and there is no need for discretivation while sampling.
Z,. +1+1. The impact of these different encoding methods is further analyzed and compared in Sec. 5.2.3
7

---

Published in Transactions on Machine Learning Research (08/2025)
Additional Bounding Box IoU Loss. To enhance bounding box generation quality, integrate Intersection-over-un (ToU)-based loss, Lion; into the denoising objective. This ToU loss aims to align the denoised bounding boxcs B Rax4 (a partial output of D closely with the ground-truth B E R"x4, The IoU loss is formulated as:
Σ GIo B.), 72
Liou 1
(5)
where n is the number of objects in the grounded scene graph, and GloU the generalized IoU, proposed in Rezatofighi et al. (2019), of the corresponding boxes. The final training loss then becomes:
L= Lad + ALious
(6)
where A is a hyperparameter that adjusts the balance between these two loss components. Note, La is given in Eq. 14. We use X = in our experiments. Algorithm [ shows the training process of DiffuseSG. Note, the node and edge denoisers (Dİ and D share identical intermediate feature maps and have the same parameters up to the final prediction heads, and the bounding box positions are part of the node tensor. The diffusion modeling details are in Appendix A.3
Summary of Technical Contributions, Though the shifted window attention is adopted from Liu et al. (2021), and the U-Net architecture is similar to other diffusion models like the ones in Song et al. (2021) Karras et al. (2022). Our novelties lie in: (1) the tensor representation of the grounded scene graph for DiffuseSG (a continuous diffusion model), (2) the separate read out layers (MLP prediction heads) for object
and relation generation, (3) an additional bounding box loU loss for training, and (4) exploration among different encoding mechanisms for object and relation categories.
4.2 Conditional Image Generation
We two diffusion-based conditional image generators to model the conditional image distribution p(T (S) given generated grounded scene graphs: one existing layout-to-inage generator named Layout sion [Zhnng et al. 2023b),, and one relation- awarre image generator which is built upon ControlNet (Zhang ct al. 2023b by ourselves, termed Relation-ControlNN.
4.2.1 LayoutDiffusion
LayoutDiffusion [Zheng et al. 2023b is diffusion- baffee sayout-to-imas generation model. It uses U-Net architecture for the denoising process, with the layout condition enforced on the hidden features of the U-Net. It first employs transformer-bai -bssed layout fusion module to capture the information in the given layout, and then utilizes a cross-attention mechanism to fuse the image features and the layout representations inside the denoising U-Net. The whole diffusion process is applied on the image pixel space. Following Ho et al. (2020): Ho & Salimans| (2022). LayoutDDiffusion utilizes a standard mcan-squared error loss to train the diffusion model and the classifier-free guidance technique to support the layout condition.
LiffusDnffusson is trained and evaluated on the Visual Genome (Krishna et al. 2017 and COCO-Stuff (Cae- sar ct al. 2018) datasets, which is aligned with our dataset settings as described in Sec. 5.1 Given the superior performance of DiffutDiffusion on these two datasets, we decide to adopt it as one of our conditional image generator candidates. Specifically, we take the model checkpoints provided by the authors which generate images of resolution 256 x 256.
4.2.2 Relation-Aware Layout-to-Image Generation (Relation- ControlNeC)))))NNe
As discussed in Sec. B the existing conditional image generators are either conditioned only on object labels and their locations (layout-to-inage models), or object labels and relation labels but no object locations (scene traph-to-mmage models). To fully utilize the data information generated by our DiffuseSG, we build a relation-aware layout-to-image generator, which generates images conditioned on object labels, locations and relation labels. We call the model, which is based on ControlNet (Zhang et al. 2023b),
8

---

Published in Transactions on Machine Learning Research (08/2025)
Prompt: Bottle behind plate. Fork near plate.
Image
Control Encoder
Bottle
Fork
Plate
Other Objects
SD Encoder Blocks
Encoder Blocks (Copy)
C. Channels
SD Decader
Zero Convolutions
Blocks
Image Decoder
Behind
Near
Other Relations
C, Channels
Relation-ControlNet
Control Input of Relation-ControlNet
Figure 3: Relttion-ControlNNet Illustration. The control input is to provide spatial information for both objects and relations. Each channel the control input has size (256, 256), where the colored areas having values of and the white areas are filled with values of In the figure, SD stands for Stable Diffusion.
ControlNet. ControlNet. (Zhang al., 2023b neural network architecture which add spatial conditioning controls to large pre-trained text-to-image diffusion models. When it is instantiated the pre-trained Stable Diffusion (Rombach et al. 2022 model, the encoder part of the U-Net is first cloned. The image feature inputted to both the original U-Net encoder and the cloned counterpart, while the cloned network blocks receive an additional control input. The output of the cloned U-Net encoder then injected into the original U-Net decoder via zero convolution layers. The control input has the same spatial dimension as the input inputoutputt /output image. Since the U-Net in Stable Diffusion works on the latent image features, the control input also transformed by convolution layers to match the input dimension of the diffusion U-Net. The original training loss in Stable Diffusion is used. During training, only the weights of the cloned U-Net encoder and the newly introduced convolution layers gel updated. We utilize ControlNet with Stable Diffusion V1.5 to build our relation-aware layout-to- tenl--tt-inn cocuuttt--mmage image generation model. Specifically, we tailor the control input and the text prompt of ControlNet to encode the object label, object location, relation label, and relation location information. We train the ControlNet to produce images of resolution 256 256.
Control Input. The control input the ControlNet to provide the spacial information of both objects and relations given grounded scene graph. Specifically, assuming there are Co object categories and C, relation classes given dataset, construct control input of size (256, 256, C. C,) to encode the object and relation locations, where use one channel to represent one specific object/ preectrrelatinn category. That is, the channel C; is to provide the location information of cobject/relattee
specify the spacial information. We utilize this
class C, and all objects or relations having the same class are encoded in the same channel. The first C, channels are for object classes while the last C, channels are for relation categories. The relation location is determined via the "between" operation given the subject and object bounding boxes defined in Hoe al. (2024). The bounding box positions are mapped from [0, 1] in continuous space to [0, 1,..., 255) discrete grids to form the control input.
Text Prompt. Given the specific version of Stable Diffusion (V1.5) used, the text prompt encoder can only encode 77 tokens. Therefore, we let the text prompt only include the relation information. We define a relation sentence as a concatenation of the subject label, the relation label, and the object label for a relation triplet, ending with period, e.g., "cat under chair.". Given the limited capability of the text encoder, we decide to only encode the unique relation sentences given grounded scene graph. Same in Zhang et al. (023b)), during training, the text prompt is replaced with empty string at. a probability of 50%.

---

Published in Transactions on Machine Learning Research (08/2025)
The
is illustrated in Fig. 3 with emphasis the control imput. The training details
can be found in Appendix A.4
5 Experiments
We conduct all experiments on the Visual Genome (Krishna al. 2017) and COCO-Stuff (Caesar et al, 2018) datasets. Our experiments mainly focus on the grounded scene graph generation part since this the main contribution of this work.
5.1 Datasets
Visual Genome (VG). We use the same cre-processing procedure and train/val /val splits as previous grounded
scene graph detection works (Xu al. 2017 Zellers
2018 Tang et al. 2020 Li al.) 2022a). but only the grounded scene graph annotations are used. This pre-processed dataset contains 57, 723 training and 5, 000 validation grounded scene graphs with 150 object and 50 relation categories. For each grounded scene graph, ensure that there will be only one edge label if there exists a directed edge (relation) between two nodes (objects). Each grounded scene graph has node numbers between 2 and 62, with an average of 5.15 relations per instance.
COCO-Stuff. COCO-Stuff contains 171 object types (including 80 thing and 91 stuff categories). comes with object label and bounding box annotations, but no relation labels. Following JJhnsoo al. (2018) we manually assign relation label between two bounding boxes based their relative positions, with relation set of 6 labels: left of, right of, above, below, inside, and surrounding. Same as in Kong al. 2022 remove small bounding boxes (S 2% image area) and instances lagged as "iscrowd", resulting in 118, 262 training and 4, 999 validation grounded scene graphs. Each grounded scene graph is a fully-connected graph without sclf-loop, with node numbers between and 33.
5.2
Grounded Scene Graph Generation Experiments
5.2.1
Evaluation Metrics
We use maximum mean discrepancy (MMD), triplet label total variation difference (Triplet. TV), and our proposed novel object detection- based F1 scores to measure the model performance on the grounded scene graph generation task.
MMDs to measure the similarities between the generated grounded scene graphs and the ground-truth ones the node degree, node label, and edge label distributions respectively. Empirically, let {z; 1 be the generated samples and {y, ) be the ground-trute ones. The MMD value is calculated as 2 E E k(x,,2,) + m Ei Σ k(yi, Uj) 2 Σ Σj k(æ;, 35), where and m are the numbers of generated samples and ground-truth ones respectively, and we use the Gaussian kernel as the kernel function k(.,-). The lower MMD value means the closer to the ground-truth distribution.
MMD. Inspired by the relevant graph generation literature (You et al. 2018 Liao et al. 2019
Triplet TV. As the labels of the <subject, relation, object> triplets lie in a very high dimension (the cross product potential subject, relation, and object types), it is computationally infeasible to calculate the triplet label MMD. As a compromise, we use the total variation difference (TV) measure the marginal distribution difference between the generated triplct labels and the ground-truth ones. Specifically, assuming that the generated empirical distribution is 9 with the ground-truth being i, where ş and & are vectors sized the number of unique triplets combining all generated and moound-ruuth triplets. The TV is calculated I g). Note, if triplet does not exist in either or q, then its relevant entry in á or is 0.
Detection-based F1 Scores. We propose a set of novel object delection-basel F1 scores lo evaluate the generated bounding box layout quality (including both the location and node label). Specifically, for a generated layout, we calculate a Fl score between this generated layout and every ground-truth layout, and take the maximum one as the final score. Assume there are N node categories. Given a pair of generated and ground-truth layouts, the F1 score is calculated F1 Evex(W.- F1a), where F1, is the F1 score for
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: Different attributes that can be generated from different methods.
Generation
Attributes Generated Object Location Relation Label
Method
Type
Object Label
SceneGraphGen
Scene Graph
VarScene
Scene Graph
D3PM
Scene Graph
BLT
Layout
V
LayoutDM
Layout
DiGress
Grounded Scene Graph
DiffuseSG
Grounded Scone Graph
a node category с and we is its weighting coefficient. Calculating Fl, needs to decide whether two bounding boxes match or not. We use 10 different bounding box IoU thresholds ranging from 0.05 to 0.5 with a step size of 0.05 to decide the bounding box match. That is, Fl, = to (ooue[0.0500000000 (ioull),) where Fi(ioulc) means a Fl score between two layouts given a specific node category с and a loU threshold iou. We calculate 4 different types of F1 scores: (1) F1-Vinilla (F1-V), where we is set to N for every node
calegory; (2) F1-Area (F1-A), where we is set to EEeN Area(c) and Area(c) is the average bounding box area
Area(c)
Fregíe) in the validation set for the node category c; (3) F1-Frequency (F1-F), where wa is set to EeeN Freq(o] and
Freq(c) is the frequency of the node category in the validation set; (4) FI-BBox Only (F1-BO), where the F1 calculation is purely based on the bounding box locations, that is, we treat all bounding boxes as having single node category (I-V) = 1 and we = 1). The motivation of having F1-Area and F1- Fl-Frequency is that we want some metrics to be slightly biased to those salient objects (appearing either in a large size in general or more frequently). The higher the Fl scores, the better.
5.2.2 Baselines
We consider the following six baselines to compare our DiffuseSG model.
(1) SceneGraphGen Garr al. 2021 is a scene graph generalive model based on recurrent networks. It is an autoregressive model where one object label or one relation label is generated at time. This model is not capable of producing object bounding boxes, and we can not specify the number of objects in a scene graph to be generated during inference.
(2) VarScene (Verma et al., 2022) is a variational autoencoder- generative model for scene graphs. Similar SceneGraphGen, it generates scene graphs without bounding boxes and can not accept number of objects as parameter during inference. Using the released code we successfully replicate the results the VG dataset but encounter issues with COCO-Stuff, as its symmetric modeling of edges between nodes prevents VarScene from generating directed graphs. Consequently, we report directed graph results on VG but undirected graph results on COCO-Stuff.
(3) D3PM (Austin et al. 2021 is a discrete denoising diffusion probabilistic framework designed for discrete data generation. We adopt its image generation model for scene graph generation; details are in Appendix B This model only generales node and edge labels.
(4) BLT (Kong et al. 2022 is a transformer- layout generation model where only object labels and bounding boxes are generated. The transformer is non-antoregressive, where all the attributes of the layout are generated at the same time as discrete tokens. Bounding box locations are quantized into integers.
(5) DMyoutDM (Inoue et al. 2023 is a discrete state-space diffusion model for layout generation. Similar to BLT, this model is also not able model relation labels and the bounding box locations are discretized.

---

Published in Transactions on Machine Learning Research (08/2025)
Table 3: Grounded scene graph generation results on the Visual Genome and COCO-Stuf validation sets. In each column, the best value is bolded. N-MMD, D-MMD, and E-MMD are the MMD values caleulated based on node label distribution, node degree distribution, and edge label distribution respectively, T-TV (val) / (train) is the Triplet TV calculated against validation training triplet statistics. The training set has a larger set of triplets than the validation, giving more comprehensive evaluation.
Visual Genome (VG)
Method
N-MMDJ
FI-VI
F1-A
F1-F1
FI-BOr
D-MMDI
E-MMD4
T-TV (val)}
T-TV (train)
La
9.44c-3
0.161
0.291
0.368
0.766
SconeGraphien
8.77e-3
3.79o-2
2.29c-2
0.987
0.979
VarScene
2,58c-2
1.04e-2
3,91c-2
0.988
0.981
DiffuseSG*
9.52c-3
0.188
0.331
0.369
0.749
6.35e-3
3.25c-2
0.735
0.566
BET
2.70e-2
0.181
0.300
0.376
0.708
D3PM
7.60e-3
3.07e-2
2.00e-2
0.816
0.772
DiGress
7.94e-3
0.157
0.263
0.282
0.732
8.89e-3
8.02e-3
0.718
0.706
DiffuseSG
6.64e-3
0.184
0.308
0.292
0.747
5.26e-3
3.46e-2
0.702
0.685
COCO-Stuff
Method
N-MMD4
F1-V1
FI-A1
FI-FT
F1-BO*
D-MMDL
E-MMD
T-TV (val)
T-TV (train)
LayoutDM
3.40e-4
0.274
0.330
0.508
0.824
SceneGraphGen
3.79e-4
2.59e-3
7.24e-4
0.904
0.895
VarScene
2.60e-2
3.07e-1
9.32e-2
0.949
0.949
DiffuseSG*
1.22e-3
0.439
0.500
0.639
0.822
1.44e-4
1.59e-4
0.229
0.287
BLT
1.09c-1
0.322
0.389
0.526
0.807
D3PM
4.92e-4
0
1.29e-1
0.341
0.305
DiGress
1.06e-3
0.342
0.387
0.570
0.782
0
4.44e-
0.515
0.398
DiffuseSG
5.53e-4
0.421
0.485
0.637
0.830
0
7.25e-5
0.270
0.219
(6) DiGress (Vignac et al. 2022 is a discrete denoising diffusion model for generating graphs with cate- gorical node and edge labels. We add an additional input of discrete bounding box representation, same the one used in LayoutDM, to incorporate bounding box generation.
Tab. 2 shows the different attributes that can be generated among all the baselines and DiffuseSG. Among the six baselines, SceneGraphGen, VarScene, and LayoutDM can not deal with specification on the number of objects for the scene graphs or layouts to be generated during sampling, while the other three can. Comparing to those diffusion-based basclines (D3PM, LayoutDM, and DiGress), DiffuseSG performs the diffusion process in the continuous space. For all the baselines, we train them from scratch using the authors' released code, with slight adaptation the datasets.
5.2.3 Grounded Scene Graph Generation Results
Our DiffuseSG is trained with Eq. (6] and uses the binary-bit input representation. To compare with SceneGraphGen, VarScene, and LayoutDM, while sampling with DiffuseSG, we do not fix the object numbers but draw the number of objects from its empirical distribution on the validation set. This line of results is denoted as D Since the other three models (D3PM, BLT, and DiGress) can specify the object numbers during inference, when comparing with these three models, the ground-truth object numbers are used for grounded scene graph generation. For all our models and basclines, we consistently randomly sample fixed set of 1, 000 training samples to do model selection.
Tab. shows the quantitative results. For each model, we generate 3 sets of layout, scene graph, grounded scene graph samples, where the size of one set samples is equal to the number of instances in the respective validation set. Reported results are the averaged results over the 3 sample sets. From the table, we can see that our D and DiffuseSG achieves the best results on most evaluation metrics. Comparing DiffuseSG* with LayoutDM, SceneGraphGen, and VarSeene, DiffuseSG* better than these basclines on F1- V, FI-A, and F1-F scores (describing both object labels and bounding box locations). D-MMD (measuring the graph connectivity), and Triplet-TV (capturing the co-occurrence between objects and their relevant
12

---

Published in Transactions on Machine Learning Research (08/2025)
Grounded Scene Graph Generation
Image Generation
Grounded Scene Graph Generation
Image Generation
roof
logo
person
roof
person
train
person
window
above
surrounding
near
near
has
Togo
person
below
inside
train
window
ooor
Sur- board
Inside
surfboard
srreunding
pilow-
light-1
light-2
light-2
Inside
window
window other
pillow-2 2
light-1
left.
Hight
right left
above
right of
of
right
bed
har
pillow- 2
bad
of left
Welll
wall
above
concress
right of
wood
conbrrree
Visual Genome
COCO-Stuff
Figure 4: Grounded scene graph image pair generation qualitative results. Grounded scene graphs are generated by DiffuseSG and the corresponding 256 x 256 images are produced by LiyoutDiffusion (Zheng ct al. 2023b).
Table 4: Ablations regarding the different categorical input representations and the IoU loss on both the Visual Genome and COCO-Stuff validation sets. The model without the loU loss is denoted DiffuseSG The evaluation metrics the same the ones used in Tab. 3
Visual Genome (VG)
Method
N-MMD4
F1-Vt
FI-Af
F1-Ft
F1-BO1
D-MMD4
E-MMDI
T-TV (val)]
T-TV (train)]
DilfuseSG (bit)
6.64e-3
0.184
0.308
0.292
0.747
5.26e-3
3.46e-2
0.702
0.685
DiffuseSG (scalar)
4.91e-3
0.109
0.210
0.262
0.742
4.91e-3
7.61e-2
0.982
0.824
DifluseSG (one-hot)
2.94e-2
0.108
0.196
0.260
0.802
1,84--2
2.96e-2
0.978
0.825
DiffuseSG (bit)
6.57e-3
0.173
0.285
0.283
0.736
7.85e-3
3.40e-2
0.709
0.692
DiffuseSG (scalar)
8, 6.65--3
0.168
0.267
0.276
0.712
7.69e-3
4.776-2
0.729
0.713
DiffuseSG (one-hot)
3.05e-3
0.142
0.249
0.253
0.689
8.94e-3
5.77622
0.795
0.751
COCO-Stuff
Method
N-MMD4
F1-Vt
FI-At
F1-Ft
F1-BOt
D-MMD4 E-MMD/
T-TV (val))
T-TV (train)
DiffuseSG (bit)
5.53e-4
0.421
0.485
0.637
0.830
7.25e-5
0.270
0.219
DiffuseSG (scalar)
1.24c-2
0.172
0.222
0.368
0.822
1.41e-5
0.895
0.915
DiffuseSG (one-hot)
1.86e-2
0.161
0.206
0.317
0.824
2.89e-3
0.807
0.793
DiffuseSG (bit)
6.63e-4
0.422
0.481
0.634
0.821
7.94e-5
0.272
0,225
DilfuseSG (scalar)
8.72c44
0.380
0.432
0.605
0.788
4.65e-4
0.312
0.282
DiffuseSG (one-hot)
2.35e-3
0.365
0.372
0.553
0.762
1.82e-3
0.439
0.332
edge labels). Notably, DiffuseSG* better than the second best model SceneGraphGen by 83.25% (VG) and 94.44% (COCO-StufT) regarding D-MMD, and by 25.53% (VG) and 74.67% with respect to T-TV (val). DiffuseSG* is only worse than SceneGraphGen on N-MMD (VG) and E-MMD (VG), and LayoutDM on N-MMD (COCO-Stuff) and F1-BO (VG and COCO-StulT;; but the differences are marginal. Comparing DiffuseSG with BLT, D3PM, and DiGress, DiffuseSG is better than these baselines all F1 scores (except F1-F on VG), and Triplet-TV. Remarkably, on COCO-Stuff, DiffuseSG is 23.10%, 25.32%, and 11.75% better than DiGress with regard to F1-V, FI-A, and F1-F respectively. DiffuseSG is only worse than D3PM on N-MMD (COCO-StufT), BLT on F1-F (VG), and DiGress on F-MMD (VG); but all the gaps are small.
Qualitative Results. Some qualitative results of DiffuseSG are shown in Fig. 4 (morc Appendix D.I As can be seen, the generated grounded scene graphs are reasonable. On VG, DiffuseSG learns the sparsity of the semantic edges. While COCO-Stuff, DiffuseSG captures the fully-connected graph pattern.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Table 5: Grounded scene graph completion results on VG validation set.
Single Node Label Completion
Single Edge Label Completion
Method
HR@1
HR@10
HR@50
HR@100
mA
HR@1
HR@10
HR
HR@05
mA
DiGress
12.4
21.2
65.3
91.2
20.7
9.8
30.9
41.4
63.2
15.8
DiffuseSG
13.9
25.7
73.2
94.5
23.6
10.1
35.7
46.2
65.3
19.4
man
Nowe
head
bird-1
wing
hooa
hise
am
bird-2
wing-2
weoding
Y
log
shirt
hand
bird-3
wing-3
Figure 5: Single bounding box completion. The left figure shows the input grounded scene graph, where only the edges and corresponding node labels are shown. The blue node's bounding box has been masked out. The middle figure shows the untouched (input) bounding boxes with labels in red, the one masked out in blue, along with the corresponding ground-truth image. The right figure shows our generated bounding box heatmap in white along with the target ground-truth bounding box (to completed) in blue. The heatmap is obtained via generating the bounding box 100 times; the whiter the area, the more overlap at the location.
Ablations. We conduct ablations with different categorical imput representations (binary-bic scalar, and one-hot), both with and without the IoU loss (Eq.. (5) ). Tab. shows the ablation results, where the model without the IoU loss is denoted as DiffuseSG Comparing among the three different categorical input encoding methods on both DiffuseSG and DiffuseSG the binary-bil representation works the best on both Visual Genome and COCO-Stuff datasets. Comparing DiffuseSG (bit) with DiffuseSG (bit) on the Fl scores, we can see the effectiveness of our proposed IoU loss. The IoU loss seems not working well while using scalar and one-hot encodings. We believe that with these two types of representations the hyperparameters need to be further tuned. Current set of hyperparameters is tuned using the binary-bit representation. Overall, using the binary-bit representation with the IoU loss gives us the best performance on both datasets.
5.2.4 Grounded Scene Graph Completion
Our DiffuseSG versatile; besides doing the pure grounded scene graph generation, it can also achieve variety of grounded scene graph complction tasks. We mainly follow Lugmayr et al. 2022 for all the completions. All completion tasks are conducted on VG.
Single Node Label Completion. In this setting, we randomly masked out one node label per grounded scene graph in the validation set. For each grounded scene graph, we keep all the bounding box locations and edge labels, and let the model complete one node label given the remaining ones. The node whose label is masked out has degree (sum of in-degree and out-degree) at least 1. This random masking is only done once, that is, it is fixed when evaluating the models. For each validation grounded scene graph, we conduct the completion 200 times, and report the Hit Rate @ K (HR@K) and mean accuracy (mA) values. For each masked grounded scene graph, to calculate the HR@K, 200 completions over the 150 object categories, and then keep the predictions from the K most frequently predicted node calegories. We assign a score of 1 if there is one prediction from the kept prediction set matches the ground-truth node label, and a score of otherwise. boundary when selecting the top K predicted catogorics, we randomly select some of them make exact K categories. Accuracy defined as the ratio of the correct predictions (matched the ground-truth) among the 200 predictions. We then respectively take the average value of HR@K and accuracy the
first build node label histogram from the
there are multiple categories on the
14

---

Published in Transactions on Machine Learning Research (08/2025)
Table 6: Grounded scene graph classification evaluation results in the PredCIs and SGCIs settings.
PredCls
SGCIs
Method
Mean Triplet Acc f
Object Acc f
Mean Triplet Acc
FID +
Ground-Truth Images
30.65
70.31
15.77
0.0
Layout Diffusson
27.85
56.79
10.04
15.73
Object-ControlNy ControlNet
26.68
55.05
8.35
15.32
Delation-Controlle
27.08
47.94
9.07
15.99
whole validation set to get the final validation HR@K and mA scores. Tab. 5 shows the and mA results on the single node label completion task. As the results suggested, on all evaluation metrics, our DiffuseSG is consistently better than the DiGress (Vignac al. 2022) baseline.
100
Single Edge Label Completion. Similar the above single node label completion task, we conduct another single edge label completion task, where one edge label is masked out per validation grounded scene graph. The experiment setting and evaluation metrics are the same the single node label completion setting. For each partially masked grounded scene graph, we predict the edge label 200 times. Tab. 5 shows the (RR@1/1022/500 and mA results. Again, our DiffuseSG is consistently better than DiGress on all evaluation metrics.
Single Bounding Box Completion. Some qualitative examples from DiffuscSG on the single bounding box completion task are shown in Fig. (more in Appendix D.2p, where one bounding box location is masked out given validation grounded scene graph, with all other information untouched. The node whose bounding box is masked out has degree at least 1. Note that neither the image nor any image feature is given to the model for the completion task; the image is only for visualization. As seen, DiffuseSG can complete the bounding box in reasonable locations.
5.3 Conditional Image Generator Discussion
As described in Sec. 4.2 have two conditional image generators: layout-to-image model Liyout sion (Zheng et al. 2023b) and relation-aware model Relation-ControlNNe. In this section, we are going to compare the image generation quality between these two generators. The COCO-Stuff dataset only has spatial relation types, which are less semantically meaningful becanse these relations can be simply decided from relative object bounding box locations. Thus we only use the VG dataset for the comparison.
5.3.1 Grounded Scene Graph Classification Evaluation
The grounded scene graph classification evaluation is to mcasure whether the generated images follow the condition control, that is, the object labels and locations, and the relation labels. For each generated image, we use trained grounded scene graph classification models to classify grounded scene graphs under two settings: (1) PredCls, where given the ground-truth object labels and bounding box locations, to classify the relation labels; and (2) SGCIs, where given the object bounding box locations, to classify both object labels and relation labels. The PredCls setting to evaluate whether the generated images contain the input relations at appropriate locations, while the SGCIs setting is check whether the objects and relations are generated properly as a whole the given locations.
We use classification accuracies the evaluation metrics. We calculate two types of accuracies: one for object labels, and one for triplet labels (the subject, relation, object labels in the <subject, relation, object> triplets). Given the long-tailed nature of the relation labels in the VG dataset, while calculating the triplet label accuracy, we first take the average of the accuracies for each relation category, and then average across all relation categories, which weighs cach relation category equally. The triplct accuracy is calculated under both the PredCls and SGCIs settings while the object accuracy is caleulated only under SGCIs, since all the object. information is given in PredCls.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Table 7: Relation control evaluation results under the PredCIs and SGCIs settings.
PredCls
SGCIs
Method
Mean Triplet Acc f
Object Acc f
Mean Triplet Acc 1
Layout
21.10
58.32
9.10
Contr--Conttoleett
17.88
56.28
7.95
Cilation-ConnroNNet 24.23
55.21
10.95
The grounded scene graph classification models used are the MOTIFS-SUM-TDE models Tang et al, 2020h, given their popularity, casy access, and reasonable performance. Besides Layout and Relation-ControoNNe,, we also consider another variant of ControlNet as baseline, named Object-ControlNet. The differences between Object-ControlNe and Relation-ControlNe are that in Object-ControlNe trol imput contains only the object information and the text prompt includes all English words of object labels (allowing repetitions). This can be considered as an ablation of Relation-ControlNer ControlNet, which omitss relation information. Note, both LifoutDiffusion and Object-ControlNer generate images conditioned on layout, while Melation-ControlNee generates images from grounded scene graphs.
the con-
Given an image generator, we use the 5, 000 grounded scene graphs from the VG's validation set, each to generate 5 images, resulting in 25, 000 images. The accuracy results are presented in Tab. [ where the object label accuracy is denoted as Object Acc and the triplet label accuracy is denoted as Mcan Triplet Acc. We also report the accuracies of the grounded scene graph classifiers on VG's ground-truth validation images, to show the upper bounds of the accuracy values. We also present the FID scores of the generated images to show the general image generation quality.
As the results suggested, the images generated from all conditional image generators have comparable general image quality (similar FIDs). However, different image generators show different control abilities. Though both LayoutDiffusion and Object-ControlNel receive the same object control information: object labels and bounding box locations, the Object- Coneett-onttoleett shows worse object renderings, which also results in worse relation renderings. We believe this result difference is due to the different model architectures and training strategies between these two models. The Conttion-ControlNet includes relation labels as an additional control input compared to Object- CottroNen,,,,,,, thus it has better triplet accuracies even with worse object accuracy. The worse object renderings of Relation-ControlNNe is reasonable. It is a common observation that it is more difficult to generate images properly with more condition inputs. Note that even the triplet accuracies of Delation-Controlly (0.4 under PredCls and 0.72 under SGCIs), and they are still worse than those of LayoutDiffusion. This is due to the fact that in VG, many relations can be inferred from bounding box positions. This is an artifact of the dataset and not a general problem at hand.
are better than those of Objeci-ControlNet, the differences are marginal
5.3.2 Relation Control Evaluation
To show the benefits of Relation- ControlNet, we build an evaluation set specifically designed for relation control evaluation. Among the 50 VG relation categories, we choose the subset containing carrying, eating, holding, laying on, looking at, lying on, playing, riding, sitting on, standing on, using, and watching. These are the relations which can not be easily determined by relative bounding box locations. In the validation set, we find the <subject, relation, object> triplets containing those relation categories. For each such triplet, we find its closest triplet inside the validation set. The closest triplct is to have the same subject and object labels, but different relation label as the original one. The relative bounding box iosition of the subject and the object of the closest triplet is also the same as the original one We discard the triplets for which we can not find the closest counterpart. If there are more than one such closest triplets, we choose the one which has the most similar subject and object bounding box aspoct ratios as
TThe model checkpoints are downloaded from https: hotpaa/cc..//aaaaaaaaaaeeeeeeeee Cranhee--rraBBBeBBee Benchmark, pytorch As discussed in Appendix C.1 the result calculation this TTbble is actually micrior to LayoutDillusion, However, it achieves the best accuracies compared both Dbject-ControlNe and Relation- Control-e......... 8Te relative bounding box positions are defined as the same way used in the COCO-Stuff dataset.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Condition
LayoutDiffusion
beacch
women
woman
Condition
Object-ControlNet
boacc
Deach
women
Condition
Relation-ControoNet
wwoman standing beach>
woman
Triplet: Pwoman walking on beach>
Figure Relation control evaluation qualitative results. The same layout condition containing an object woman and an object beach with their corresponding bounding box locations is fed to all the models. Additionally, the relation information is given to the Relation- ControlNet model. We draw 10 image samples from both LiffutDi and the relations <woman standing on beach>> and <woman walking beach>> respectively.
while with Relation-Controlyy
draw samples
the original triplet. We then swap the relation labels between the triplet and its closest counterpart. We keep all the triplets, their closest triplets, and the triplets with swapped relations. This process gives 328
17

---

Published in Transactions on Machine Learning Research (08/2025)
Table 8: Grounded scene graph detection (SGDet)
results of SGTR.
Data Setting
mR@50
mR@100
R@50
R@@0000
Head
Body
Tail
Original Additional 12.7
11.9
16.0
23.7
27.0
26.6
19.8
9.0
16.7
23.6
26.8
26.4
20.7
9.7
triplets. The swapping procedure guarantees that for the same pair of subject and object, there exist two different relation types, which is agnostic the object-only conditioned models, but important to the relation conditioned model. For each triplet, we generate 5 images from Layout Relation-Controlle respectively. The corresponding object and triplet accuracies reported in Tab. 7
Object-Controlnn
and
This setting shows the benefits of Delation-ControlNer ControlNet is worse than LiffutDiffusion and Object-ControlNNe in terms of object accuracy, its triplet accuracies are siguificantly better than the other two. This is because Layout Diffusion and Object-ControllN can not take the relation label as input but Relation- CoationnControoNet can. As the qualitative examples shown in Fig. 0 when the relation information is inputted to the model, the appropriate postures for the woman, either standing walking depending on the condition. However, for both LayoutDiffusion and Object-ControlNett since these models can not read in the relation informa- tion, the models have to render the postures of the woman via their own decisions, which may be standing, walking, or even bending the knees. Though our Relation-ControoNee is not perfect, it implics the usefulness of generating grounded scene graphs. Generating grounded scene graphs, which contain object labels and locations along with relation labels, can provide more controllability for image generation.
As indicated in the results, though Relation-
can constantly render
Ceation-ConntolNeee is designed for harder task, where the relation information also part of the condition. However, given the inferior performance of Relation-ControlNe in the real validation setting (Tab. and limitations of current datasets (many of the relations in VG can be easily determined by relative bounding box positions); we use
as the conditional image generator for experiments that follow.
5.4 Grounded Scene Graph Detection Evaluation
We take the DiffuseSG trained on VG, and pair it with the trained LifoutDiffusion model (Zheng et all 2023b) to form 5, 000 grounded scene graph image pairs, treated as additional training data for the down- stream grounded scene graph detection (SGDet) task: given image, detecting the object labels and locations, as well as the relation labels. Those additional grounded scene graphs only contain body relations as defined in Li et al. 2021a). Detailed process is in Appendix C.2
We use the SGTR model (Li et al. 2022a). as an example, to show the value of our generated grounded scene graph image pairs. We train SGTR for the SGDet lask under two training data settings: (1) Original, where the training data is the original Visual Genome training data in Li et al. (2022a); (2) Additional, where besides the original training data, we add in our generated 5, 000 grounded scene graph image pairs. Note that these two settings only differ in the training data; both validation and testing data is still the original data for the SGDet task. For both data settings, we train SGTR 4 times and the averaged test results are reported in Tab. g where the model for testing is selected via best mR@100 on the validation set. We report results on mean Recalls (mR@50 and mR@100), Recalls (R@50 and R@100), and mR@100 on the head, body, and tail relation partitions (respectively indicated as Head, Body, and Tail in the Table).
Comparing our Additional results with the Original ones, we can see that our generated grounded scene graph immaee pairs do have value, which brings improved results on Body and Tail, and comparable results Head, which results in increased results mean Recalls and comparable results on Recalls. As suggested in Tang et al. (2020), mean Recall is a better evaluation metric than Recall for the grounded scene graph detection task, because it is less biased to the dominant relation classes. Note that although the generated grounded scene graphs in the additional training data only contain body relations, the Tail results also get improved. This reasonable, because grounded scene graph is a structure, increasing the confidence of some part of the structural prediction will increase the confidence of other part as well, especially for the tail relations, where the prediction confidence is usually low.
18

---

Published in Transactions on Machine Learning Research (08/2025)
6 Conclusion
In this work, propose novel framework for joint grounded scene graph image generation. As part of this, we propose DiffuseSG, a diffusion- model for generating grounded scene graphs that adeptly handles mixed discrete and continuous attributes. DiffuseSG demonstrates superior performance on both unconditional grounded scene graph generation and conditional grounded scene graph complction tasks. By pairing DiffuseSG with a conditional image generation model, the joint grounded scene graph image pair distribution can be obtained. We illustrate the benefits of DiffuseSG both on its own and as part of joint grounded scene graph image generation. In the future, interested in modeling the joint distribution with single model.
Acknowledgments
This work was funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chairs, NSERC CRC, and NSERC DGs. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, the Digital Research Alliance of Canada alliance. can ca companies sponsoring the Vector Institute, and Advanced Research Computing at the University of British Columbia. Additional hardware support was provided by John R. Evans Leaders Fund CFI grant and Compute Canada under the Resource Allocation Competition award. Qi Yan is supported by the UBC Four Year Doctoral Fellowship.
References
Diego Martin Arroyo, Janis Postels, and Federico Tombari. Variational transformer networks for layout generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pallern Recognition, pp. 13642-138 13652, 2021.
Oron Ashual and Lior Wolf. Specifying object attributes and relations interactive scene generation. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 4561-4568 2019.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Newral Information Processing Systems, 34:17981-17993, 2021.
Bashirul Azam Biswas and Qiang Ji. Probabilistic debiasing scene graphs. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 10429 10438, 2023.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209-1218, 2018.
Shang Chai, Liansheng Zhuang, and Fengying Yan. Layoutdm: Transformer-based diffusion model for layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 18349 18358, 2023.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXim:2208 04202, 2022.
Jiaxin Cheng, Xiao Liang, Xingjian Shi, Tong He, Tianjun Xiao, and Mu Li. Layoutdiffuse: Adapting foundational diffusion models for layout-to-image generation. arXiv preprint 2XXi:2302.088000 2023.
Hyuna Cho, Minjae Jeong, Sooyeon Jeon, Sungsoo Ahn, and Won Hwa Kim. Multi-resolution spectral coherence for graph generation with score-based diffusion. Advances in Neural Information Processing Systems, 36, 2024.
Vinay Damodaran, Sharanya Chakravarthy, Akshay Kumar, Anjana Umapathy, Teruko Mitamura, Yuta Nakashima, Noa Garcia, and Chenhui Chu. Understanding the role of scene graphs in visual question answering. or Xiv preprint ar 2511:2200..05/79, 2021.
19

---

Published in Transactions on Machine Learning Research (08/2025)
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annuall ACM symposium on user interface software and technology, pp. 845 854, 2017.
Helisa Dhamo, Azade Farshad, Iro Laina, Nassir Navab, Gregory D Hager, Federico Tombari, and Christian Rupprecht. Semantic image manipulation using scene graphs. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp.
2020.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffusion for categorical data. arXic preprint orXiu:2211.1008 2022.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106 (496):1602 1614, 2011. doi: 10.1198/jasa, 2011.am.021111.mm11mmm URL http://do..oog11..1113.00 2011.
tm1 1181 PMID: 22505788.
Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. Advances in Newral Information Processing Systems, (e.16222-162333 2023.
Azade Farshad, Yousef Yeganeh, Yu Chi, Chengzhi Shen, Böjrn Ommer, and Nassir Navab. Scenegenie: Scene graph guided diffusion models for image synthesis. In Proceedings of the IEEE/CVF International Conference on Compuler Vision Workshop, pp. 88 98, 2023.
Sarthak Garg, Helisa Dhamo, Azade Farshad, Sabrina Musatian, Nassir Navab, and Federico Tombari. Unconditional scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. (6362-16371, 2021.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong, Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210 [ta:2210.08990 08933, 2022.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. In European Conference on Computer Springer, 2024.
Vision, pp.
Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry S Davis, Vijay Mahadevan, and Abhinav Shrivas- tava. Layouttrausformer: Layout generation and completion with attentinnnn In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1004-1014, 2021.
Zeeshan Hayder and Xuming He. Dsgg: Dense relation transformer for an end-to-end scene graph generation. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pollern Recognition, pp. 28317 28326, 2024.
Sen He, Wentong Liao, Michael Ying Yang, Yongxin Yang, Yi-Zhe Song, Bodo Rosenhahn, and Tao Xi- ang. Context-aware layout to image generation with euhanced object appearance. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15049-15058, 2021.
Dan Hendrycks and Kevin Gimpel. Gaussian error lincar units (gclus). arXiv preprint arXiv:6606.08,11 2016.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12500 2022.
Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, volume 33, pp. 6840 6851. Curran As- sociates, Inc., 2020. URL https: http:///ppooeeiinngnnrri....... http://aaar.illeapppp/////////// Rcbocte66544110077711107777aaaa
Jiun Tian Hoc, Xudong Jiang, Chee Seng Chan, Yap-Peng Tan, and Weipeng Hu. Interactdiffusion: Interac- tion control in text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 61 80 6180-6189, 6189, 2024.
20

---

Published in Transactions on Machine Learning Research (08/2025)
Mude Hui, Zhizheng Zhang, Xiaoyi Zhang, Wenxuan Xie, Yuwang Wang, and Yan Lu. Unifying layout generation with a decoupled diffusion model. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 1942 1951, 2023.
Naoto Inoue, Kotaro Kikuchi, Edgar SirraS Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete dif- fusion model for controllable layout generation. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 10167-1011 2023.
John Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman. Zine: free tool to discover chemistry for biology. Journal of chemical information and modeling, 52(7):1157-1168, 2012.
Zhaoyun Jiang, Jiaqi Guo, Shizhao Sun, Huayu Deng, Zhongkai Wu, Vuksan Mijovic, Zijiang James Yang, Gan-Guang Lou, and Dongmei Zhang. Layoutformer++ Conditional graphic layout generation via con- straint serialization and decoding space restriction. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 18403 18412, 2023.
Tianlei Jin, Fangtai Guo, Qiwei Mong, Shiqiang Zhu, Xiangming Xi, Wen Wang, Zonghao Mu, and Wei Song. Fast contextual scene graph generation with unbiased context augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 8802-6311 2023.
Jachyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. International Conference on Machine Learning, pp. 10362 10383. PMLR, 2022.
Jachyeong Jo, Dongki Kim, and Sung Ju Hwang. Graph generation with diffusion mixture. orXiv preprint arXiv:8302.00 03596, 2023.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1219-1228, 2018.
Deunsol Jung, Sanghyun Kim, Won Hwa Kim, and Minsu Cho. Devil's on the edges: Selective quad attention for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18664-18674, 2023.
Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochastic scene layout generation from a label set. In Proceedings of the SEEE/CVE International Conference on Computer Vision, pp. 9895 9904, 2019.
Tero Karras, Miika Ailtala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565 26577, 2022.
Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Constrained graphic layout gener- ation via latent optimization. In Proceedings of the 29th ACM International Conference Multimedia, pp. 88-96, 2021.
Jinseok Kim and Tae-Kyun Kim. Arbitrary-scale image generation and upsampling using latent diffusion model and implicit neural decoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2211- 2024.
Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, and Irfan Essa. Blt: bidirec- tional layout transformer for controllable layout generation. In Europeon Conference Computer Vision, pp. 474-490. Springer, 2022.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73 73, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
21

---

Published in Transactions on Machine Learning Research (08/2025)
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept cus- tomization of text-to-image diffusion. In Proceedings of the IEFE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 1931- 1941, 2023.
Sanjoy Kundu and Sathyanarayanan N Aakur. Is-ggt: Iterative scene graph generation with generative transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 6292 6301, 2023.
Hsin- Ying Lee, Lu Jiang, Irfan Essa, Phuong B Le, Haifeng Gong, Ming-Hsuai Yang. and Weilong Yang. Neural design network: Graphic layout generation with constraints. In Compuler Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 491-506. Springer, 2020.
Elad Levi, Eli Brosh, Mykola Mykhailych, and Meir Perez. Dlt: Conditioned layout generation with joint discrete-contimtous diffusion layout transformer. arXiv preprint 3rXiv22303.0375, 2023.
Jianan Li, Jimei Yang, Aaron Hertzmann, Jiamming Zhang, and Tingfa Xu. Layoutgan: Synthesizing graphic vector-wirefrane wireframe adversarial networks. IEEE Transactions on Pottern Analysis and Machine Intelligence, 43(7):2385) 2299-399992 2020a.
layouts with vector
Jianan Li, Jimei Yang, Jianming Zhang, Chang Liu, Christina Wang, and Tingfa Xu. Attribateeconditiooned layout gan for automatic graphic design. IEEE Transactions on Visualization and Computer Graphics, 27 (10):4039-400) 4048, 2020b.
Jiankai Li, Yunhong Wang, Xiefan Guo, Ruijic Yang, and Weixin Li. Leveraging predicate and triplet learning for scene graph generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 28369 28379, 2024.
Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In Proceedings of the IEEE/CVP Conference Computer Vision and Paltern Recognition, pp. 11109 11119, 2021a.
Rongjic Li, Songyang Zhang, and Xuming He Sgtr: End-to-cnd scene graph generation with transformer. In proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 19486-19496, 2022a.
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang and Tatsunori Hashimoto. Diffusion-lm improves controllable text generation. Advances Neural Information Processing Systems, 35:4328-433 4343,, 2022b.
Yikang Li, Tao Ma, Yeqi Bai, Nan Duan, Sining Wei, and Xiaogang Wang. Pastegan: A semi-parametric method to generate image from scene graph. Advances in Neural Information Processing Systems, 32, 2019.
Zejian Li, Jingyu Wu, Immanuel Koh, Yongehuan Tang, and Lingyun Sun. Image synthesis from layout with locality-aware mask adaption. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13819-13828, 2021b.
Jingyun Liang, Yuchen Fan, Kai Zhang, Radu Timofte, Luc Van Gool, and Rakesh Ranjan. Movidoo: Motion-aware video generation with diffussin model. In European Conference on Compuler Vision, pp. 74-74.60 Springer, 2024.
Renjie Liao, Yujia Li, Yang Song, Shenlong Wang, Will Hamilton, David K Duvenaud, Raquel Urtasun, and Richard Zemel. Efficient graph generation with graph recurrent attention networks. Advances in newral information processing systems, 32, 2019.
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhn Chen. Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise. In International Conference on Machine Learning, pp. 21051-21064. PMT.R 2023.
22

---

Published in Transactions on Machine Learning Research (08/2025)
Zhihang Lin, Mingbao Lin, Meng Zhao, and Rongrong Ji. Accdiffusion: An accurate method for higher- resolution image generation. In European Conference on Compuler Vision, pp. 38 09-53. Springer, 2024.
Jinxiu Liu and Qi Liu. R3ed: Scene graph image generation with relation-aware compositional contrastive control diffusion. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 38, pp. 3657- 3665, 2024.
Shanyuan Liu, Ao Ma, Xiaoyu Wu, Dawei Leng, Yuhui Yin, et al. Hico: Hierarchical controllable diffusion model for layout-to-image generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Thomas F Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. Learning design semantics for mobile apps. In Proceedings of the 31st Arnual ACM Symposium on User Interface Software and Technology, pp. 569-579, 2018.
Xiaohong Liu, Xiongkuo Min, Guangtao Zhai, Chunyi Li, Tengchuan Kou, Wei Sun, Haoning Wu, Yixuan Gao, Yuqin Cao, Zicheng Zhang, et al. Ntire 2024 quality assessment of ai-generated content challenge. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6337-6362, 2024b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVT Conference Computer Vision and Pattern Recognition, pp. 114111 2022.
Andrew Melnik, Michal Ljubljanac, Cong Lu, Qi Yan, Weiming Ren, and Helge Ritter, Video diffusion models: A survey. arXi preprint 005.0000 2024.
Zichen Miao, Jiang Wang, Ze Wang, Zhengyuan Yang, Lijuan Wang, Qiang Qiu, and Zicheng Liu. Training diffusion models towards diverse image generation with reinforcement lcarning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 10841-10853, 2024.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher-order graph neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 33(0)):4602-44000 Jul. 2019. doi: 10.1609/ /a1600/aaai..o v33101. v33.10 33014602. URL https://ojs. aaai
ieaai.orge org/index. phpp /SS/AAAA//aaaa www..//aaiieeeeee/////aa.a.aaaae
Christopher Morris, Yaron Lipman, Haggai Maron, Bastian Rieck, Nils M Kriege, Martin Grohe, Matthias Fey, and Karsten Borgwardt. Weisfeiler and leman go machine learning: The story so far. arXiv preprint arXiv:21 12.12..0000 090990 2021.
Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Akino, Vishal M Patel, and Tim K Marks. Steered diffusion: A generalized framework for plug-and-plas conditional image syn- thesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20850-20860, 2023.
Tianwen Qian, Jingjing Chen, Shaoxiang Chen, Bo Wu, and Yu-Gang Jiang. Scene graph refinement network for visual question answering. IEEE Transactions on Multimedia, 2022.
Zhiwu Qing, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yujie Wei, Yingva Zhang, Changxin Gao, and Nong Sang. Hierarchical spatio-temporal decoupling for text-to-video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6635 6645, 2024.
Leigang Qu, Wenjie Wang, Yongqi Li, Hanwang Zhang, Liqiang Nic, and Tat-Seng Chua. Discriminative probing and tuning for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Pattern Recognation, pp. 7434-7444 2024.
23

---

Published in Transactions on Machine Learning Research (08/2025)
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Gen- eralized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 658-666, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medicnl Image Computing and Computer- Computer-Assisted Assisted Citrreetioo-MMICC. 2015: 18th International Conference, Munách, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241. Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22511 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479 36494, 2022.
Mohammad Amin Shabani, Zhaowen Wang, Difan Liu, Nanxuan Zhao, Jimei Yang, and Yasutaka Furukawa. Visual layout composer: Image-vector dual diffusion model for design layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9222 9231, 2024.
Ivan Skorokhodov, Willi Menapace, Aliaksandr Siarohin, and Sergey Tulyakov. Hierarchical patch diffusion models for high-resolution video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7569-7579, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMI.R, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL http:://ppaareeeeaaaaaa///rrraap
Mingzhen Sun, Weining Wang, Zihan Qin, Jiahui Sun, Sihan Chen, and Jing Liu. Glober: coherent non- autoregressive video generation via global guided video decoder. Advonces in Neural Information Process- ing Systems, 36, 2023.
Wei Sun and Tianfu Wu. Image synthesis from reconfigurable layout and style. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10531-105400 2019.
Wei Sun and Tianfu Wu. Learning layout and style reconfigurable gans for controllable image synthesis. IEEE transactions on pattern analysis and machine intelligence, 44(9)55000-5087, 2021.
Tristan Sylvain, Pengchuan Zhang, Yoshua Bengio, Devon Hjelm, and Shikhar Sharma. Object-centric image generation from layouts. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 2647-2655, 2021.
Endre Süili and David F. Mayers. An Introduction Numerical Analysis. Cambridge University Press, 2003. doi: 10.1017/CR000785118111111
24

---

Published in Transactions on Machine Learning Research (08/2025)
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph gener- ation from biased training. In Proceedings of the IEEE/CV conference on computer vision and pattern recognition, pp. 3716 3725, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Tathagat Verma, Abir De, Yateesh Agrawal, Vishwa Vinay, and Soumen Chakrabarti. Varscene: A deep generalive model for realistic scene graph synthesis. In International Conference on Machine Learning, pp. 22122-216222 22183. PMLR, 2022.
Clement Vignac, Igor Krawczuk, Antoine Sirandin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint ar arXiv:2209. 14794, 2022.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Compulalion, 23 (7):1661- 1674, 2011.
Yanhui Wang, Jianmin Bao, Wenming Weng, Ruoyu Feng, Dacheng Yin, Tao Yang, Jingxu Zhang, Qi Dai, Zhiyuan Zhao, Chunyu Wang, et al. Microcinema: A divide-and-conquer approach for text-to-video generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8414 8424, 2024a.
Yumnan Wang, Ziqiang Li, Zequn Zhang, Wenyao Zhang, Baao Xie, Xihui Liu, Wenjun Zeng, and Xin Jin. Scene graph disentanglement and composition for generalizable complex image generation. arXic preprint arXiv:241000044 2024b.
Fanyue Wei, Wei Zeng, Zhenyang Li, Dawei Yin, Lixin Duan, and Wen Li. Powerful and flexible: Personalized text-to-image generation via reinforcement learning. In European Conference on Computer Vision, pp. 394-410. Springer, 2024.
Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, et al. Ar-diffusion: rego-ssgres diffusion model for text generation. Advances in Neural Information Processing Systems, 01:39977-39977 2023.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5410- 5419, 2017.
Zhe Xu, Ruizhong Qiu, Yuzhong Chen, Huiyuan Chen, Xiran Fan, Menghai Pan, Zhichen Zeng, Mahashweta Das, and Hanghang Tong. Discrete-state continuous-time diffusion for graph generation. arXiv preprint arXiv:2405.114 14166, 2024.
Qi Yan, Zhengyang Liang, Yang Song, Renjic Liao, and Lele Wang. Swingnn: Rethinking permutation invariance in diffusion models for graph generation. arXiv preprint orXiv:2377.066// 2023.
Ling Yang, Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming-Hsuan Hsuan Yang. Diflusion-based scene graph to image generation with masked contrastive pre- training. aixiv preprint arXiv:2211.11133 2022.
Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10685-10694 2019.
Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. Dinoiser: Diffused conditional
sequence learning by manipulating noises. arXio preprint 1000022302.10025, 2023.
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovee. Graphrnn: Generating realistic
graphs with deep auto-regressive models. In International conference on machine learning, pp. 5708-5717. PMLR, 2018.
25

---

Published in Transactions on Machine Learning Research (08/2025)
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. Seqdiffuseq: Text diffusion with encoder-decoder transformers. arXiv preprinl or arx:2:2112 10325, 2022.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference computer vision and pattern recognation, pp- 5831-5840, 2018.
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video gencration. Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 8850 8860, 2024a.
Yu Zeng, Vishal M Patel, Haochen Wang, Xun Huang, Ting-Chun Chun Wang, Ming-Ye Yu Liu, and Yogesh Balaji. Jedi: iiaag diffusion models for finetuning free personalized text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6786-6795, 2024b.
Junyi Zhang, Jiaqi Guo, Shizhao Sun, Jian-Guang Lou, and Dongmei 'Zhang. Layoutdiffusion: Improving graphic layout generation by discrete diffusion probabilistic models. arXiv preprint 1rXiu22303.1158,
2023a.
Lviin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-inage diffusson models. In Proceedings of the LEEE/CVF International Conference on Computer Vision, pp. 1836-3847, 2023b.
Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8584 8593, 2019.
Chaofan Zheng, Xinyu Lyu, Lianli Gao, Bo Dai, and Jingkuan Song. Prototype-based embedding network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22783 22792, 2023a.
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Con- trollable diffusion model for layout-to-image generation. Proceedings of the IEFE/CVV Conference on Computer Vision and Pattern Recognétion, pp. 22490-22499, 2023b.
Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau. Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics (TOG), 38(4):1-15, 2019.
Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset for document layout analysis. In 2019 International Conference on Document Analysis and Recognilion (ICDAR), pp. 1015 1022. IEEE, 2019.
Yiwu Zhong, Liwei Wang, Jianshu Chen, Dong Yu, and Yin Li. Comprehensive image captioning via scene graph decomposition. In Computer Vision-ECCO EECV 2020: 16th European Conference, Glasgow, UK August 23 28, 2020, Proceedings, Port XIV 16, pp. 211 229. Springer, 2020.
26

---

Published in Transactions on Machine Learning Research (08/2025)
A Implementation Details
A.1 Discrete Input Encodings
To effectively handle the discrete altributes, we implement the following encoding methods.
Scalar. Similar to prior works (Ho et al.) 2020 Song et al. 2021 Karras et al. 2022] Jo et al. 2022). we use zero-based indexing and map the scalar value to the range [1, Specifically, during training, denoting an integer label of an attribute as n and the number of calegories of the attribute as m, where n E (0, ",....... m - 13, the scalar representation becomes m-1 1. During sampling, we first. split, the interval [-,, into equal-sized bins in accordance with the number of node or edge categories. We then decode the mutinuous-valuee network output into discrete label based on the bin into which the output value falls.
Binary-Bit. Following Chen et al. 2022), we first convert the zero-based indexed integer node/edge at- tribute into binary bits, and then remap the 0/1 bit values to -1/1 for improved training stability. During sampling, we binarize each value in the network output based on its sign. That is, a positive value is inter- preted as 1, and a negative value as O. We then convert the binary representation back into an integer node or edge label.
One-Hot.
We remap the 0/1 1 values in the one-hot encoding of the original integer node/edge label to -1/1. During sampling, we take the argmax value of the network output to obtain the categorical label.
A.2 DiffuseSG Network Architecture
Our proposed graph transformer has two essential components: (1) shifted-window attention mechanism, and (2) downsampling/ haaaiinggg/ammpinggggg.g. layers. In the case of graphs comprising n nodes, their adjacency matrices, entries.
incorporating both node and edge attributes, are conceptualized
high-order tensors with To handle graphs of varying sizes, we standardize the size of these adjacency matrices through padding, Consoquently, for different datascts, we accordingly adjust the design paramcters to ensure the network is proportionate and suitable for the specific requirements of each dataset.
Shifted- Window Attention. We adopt the shifted window attention technique from Liu et al. (002) which partitions the original grid-like feature map into smaller subregions. Within these subregions, local message passing is executed using self-attention mechanisms. Additionally, the windows are interleavedly shifted, facilitating wiosswwnndww message passing, thereby enhancing the overall efficiency and effectiveness of the feature extraction process.
Downsampling/ UUpsampling operators to effectively diminish or augment the size of the feature map, thereby constructing hicrarchical representations. During downsampling, the feature map is divided into four segments based on the parity of the row and column indices, followed by a concatenation process along the channel, which serves to reduce the dimensions of height and width. The upsampling process performs the inverse operations. It initially splits the tensors along the channel and then reshapes them, effectively reversing the process conducted in downsampling. The downsampling and upsampling layers are visualized in Fig. 7 We also implement one MLP layer right after each downsampling, huppa//pusigggggggggee layer. In line with the widely recognized U-Net architecture (Song et al. 2021 Karras et al. 2022), our approach also integrates skip-connections for tensors of identical sizes to enhance the network capacity.
Layer.
We incorporate channel mixing-based downsampling/upsanne
The crucial design parameters of our model are detailed in Tab. g It is important to note that within the Down/Up block layers, the initial blocks do not utilize downsampling/ upsampling operations. For instance, in the context of the Visual Genome dataset, we effectively implement 3 downsampling layers, which leads to the successive alteration of the feature map dimensions as 64 + 32 + 16 + 8. In this setup, we opt for a window size of 8, ensuring that the receptive field is sufficiently large to facilitate effective message passing between each pair of nodes. While on COCO-Staff we employ 2 downsampling layers, resulting the feature map dimensions as 40 + 20 + 10, and thus the window size is set to 10.
MLP Prediction Head. The node/edge attribute MLP prediction head implemented as two linear layers with a GELU (Hendrycks & Gimpel 2016) operation injected in between.
27

---

Published in Transactions on Machine Learning Research (08/2025)
Downsampling Layer
Doncet
+ 4.44)
14) 19.9.44
(..4
9 46g)
Lincar Layer
19 2.4)
Upsampling Layer
Spii
( 242
(8.9.14)
(4-9.4)
(4-9.40
(9.9.4)
(9-9.4)
4g)
Figure Visualization of the downsampling and upsampling layers.
Table 9: Architecture details of our graph transformer.
Hyperparameter
VG
COCO-Stuff
Full tensor size (n x n)
64 64
40 40
Down block attention layers
[1, 1, 1]
[1, 2, 6
Up block attention layers
[1, 1, 3,
[1, 2, 6
Number of attention heads
[3, 6, 12, 24]
[3 6, 12]
Window size
10
Token dimension
96 384
96
Feedforward layer dimension
384
A.3 DiffuseSG Diffusion Modeling Details
To ensure the stable training of our diffusion model, we adopt framework based on the stochastic differ- ential equation (SDE), as proposed in Song et al. (2021). Additionally, we incorporate a variety of training techniques that have been proven effective in image generation contexts: network preconditioning (Karras et al. 2022), self-conditioning (Chen al. 2022) and exponential moving average (EMA). For network training, we employ the hyperparameters specified for the ImageNet-64 dataset in Karras et al. (2022) for preconditioning purposes; detailed explanation of these parameters be found therein. We Adam optimizer and learning rate being 0.0002. The EMA coefficients used for evaluation are 0.9999 and 0.999 the Visual Genome and COCO-Stuff datasets respectively.
The pseudocode of our sampling algorithm is presented in Algorithm 2 which follows the stochastic sampler in Karras et al. 2022) but with the additional self-conditioning (Chen et al. 2022) technique.
the algorithm, Do is the denoising network, g(*) is the generated grounded scene graph at step I, and I is the identity matrix. The associated parameters are detailed Tab. IO We opt for = 256 sampling steps to expedite the sampling process, as opposed to the original 1, 1,000 steps used in the DDPM framework (Ho let al.] 2020). In Algorithm [ with slight abuse of notations for simplicity, we consider the generated grounded scene graph (S, Sec), which comprises tuples of node and edge attributes, as a singular tensor, allowing for soraightforward addition or subtraction operations. Practically, this is implemented through separate operations on the node and edge tensors.
A.4
and Coject-ConntooNet Training Details
Following Zhang al. (2023b). we use Stable Diffusion as an instantiation the ControlNet architecture. We use Stable Diffusion V1.5. To train both Relation-Controll and Object-ControlNe use Adam optimizer with 3 being 0.9, B2 being 0.999, and weight decay being 0.01; a constant learning rate 0.00001 is
28

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm 2 DiffuseSG Sampler.
Require: Dg,T, T, (41))20 (w))-1
sample s (0, S 0.
for = to - do noise I)
3: sample c~ (O, 2
4:
4
(1+7)44
s -
1 fle
S t 1o(S (
(
6:
sc f d (5 -50 )/4
sc
7:
S s(it1)
8:
5
S
+ (4111 l;)
S
(1+1) S 0c ,t+11)
9:
t DD(s
d' + (5 (5(ttt))
(1+1)
-s(itt) )/4++1
10:
11: g(1t1) + 5
+ + (d, + d,)
12: end for
13: return 5 (T)
Table 10: Sampling parameters in the denoising process.
min = 0.002, max = 80, 7
Simin = 0.05, Sumax = 50, S Suiss = 1.003, Schurn 40, T = 256
t, max 7 T (m maxx : ))
% Sunin Stunn min( Sulquen V2 - 1)
used train the models. Both models are trained for 200 epochs with batch size of 120. Following Zhang ct al. 2023b), during training, the text prompts are randomly replaced with empty strings at chance of 50%.
Images are generated in the resolution of 256 256.
B D3PM Baseline
Our D3PM (Austin et al. 2021 b baseline based on the image generation model on the CIFAR-10 dataset (Krizhevsky et al., 2009 Given a scene graph S = (V, (V with n nodes, where V N is the node vector containing the inleger node labels, and E E Nax is the adjacency matrix containing the in- teger edge labels, the imput our D3PM baseline is represented Q Nuxax3 where Q [V, V E.yl. Vi,j E
We use two separate discretized Gaussian transition matrices, one for the node category and one for the edge category, to add noise on Q, resulting in the noised Q. We then use a U-Net with two separate prediction heads, cach implemented as two convolution layers with sigmoid operation before each of the convolution layers, to respectively produce the logits of the denoised V. and Ê, which then form the logits of the denoised D3PM the logits of Q to train
Q. We use the Tx=0.001 (Eq. (5) in Austin et al. (2021) )) calculated
baseline. We use Adam optimizer and learning rate being 0.00005 for training. We use T 1, 000 noising and denoising steps and take the model with EMA coefficient 0.9999 for evaluation.
The 3: (in Fq. (8) in the Appendix of Austin et al. 2021 of the discretized Gaussian transition matrix is increased linearly, for t E 21, T3. On the Visual Genome dataset, we set n to be 64, and 3e is increased linearly from 0.0001 to 0.02 for both node and edge categories. On the COCO-Stufe dataset, n set to be 36. The Bg is increased linearly from 0.0001 to 0.02 for the node category and from 0.04 to 0.1 for the edge category.
WW slightly ahuse the notations here, specifically for the D3PM model, compared the ones in the main text.
29

---

Published in Transactions on Machine Learning Research (08/2025)
C Evaluation Details
C.1 Grounded Scene Graph Classification Evaluation LayoutDiffusion
The LayoutDiffusion model checkpoint that we used is trained on version of the VG dataset annotation which has 178 object categories. This is slightly different from the VG dataset annotation that the grounded scene graph classification models are trained on, which contains 150 object categories. However, between these two versions of VG annotations, there are 131 object categories in common. Thus when generating images from LayoutDiftusion for the grounded scene graph classification evaluation, we only keep the objects whose labels are in the common category set. Specifically, given a. ground-truth VG validation grounded scene graph, we keep the objects in the common category set and discard others, and then generate the corresponding images. But, when calculating the grounded scene graph classification accuracy scores, we still use the ground-truth grounded scene graph without any object filtering, Though this setting is inferior to LayoutDiffusion, it still achieves better accuracics than Object- Coject-ControoNet and Celation-Controlye as shown in Tab. G
When building the evaluation set for the relation control evaluation, we make sure that all the subject and object labels are in the common category set.
C.2 Generating Additional Training Data for the Grounded Scene Graph Detection Task
We take DiffuseSG model trained on the Visual Genome dataset, let it generate set of grounded scene graphs which only contain relations falling into the body partition (as defined in Li et al. (2021a) and then use the pretrained (on VG, with resolution 256 x 256) Liffus model (Zheng et al. 2023b to form the grounded scene graph image pairs. Since there exists some node label set discrepancy between the respective VG annotations used to train the LiffutDifftssin model (178 node categories) and our DiffuseSG model (150 node categorics). When forming the grounded scene graph image pairs, we discard the nodes whose labels are not in the common category set (131 node categories) and their related edges. We randomly choose 5, 000 such generated pairs, where node numbers are restricted to be less than 10, additional training data to train the SGTR model (Li et al.] 2022a) on the grounded scene graph detection task: given an image, detecting a grounded scene graph (node labels and bounding box locations, and edge labels) from it. We guarantee that for those randomly chosen grounded scene graphs, each of them has at least one edge.
The motivations of why we generating the grounded scene graphs only containing body relations are as follows. First, for the grounded scene graph detection task, there are already many training instances for the head classes, so generating additional head relations may not be beneficial at all. Second, for grounded scene graph generation task, since the training data for the tail relations is limited, our grounded scene graph generation model may not be able to model the tail class distribution well.
D More Qualitative Results
D.1 Grounded Scene Graph Image Pair Generation
More qualitative results of grounded scene graph - image pair generation are shown in Figs. g and 9 (Visual Genome) and Figs. 10 and I1 (COCO-Stuff). Grounded scene graphs including the bounding box locations are generated by DiffuseSG and the corresponding images (in resolution 256 x 256) are produced by the pretrained LayoutDiffusion model (Zheng et al. 2023b). Note that the Visual Genome dataset, since there exists some node label set discrepancy between the annotations used to train our DiffuseSG model and the LifoutDiffusion model, we only visualize the grounded scene graphs whose node labels are all in the common node label set (131 node categories).
D.2 Single Bounding Box Completion
More qualitative results of our DiffuseSG on the Visual Genome validation set are shown in Fig. 12 The left. figure shows the input grounded scene graph, where only the edges and corresponding node labels are
30

---

Published in Transactions on Machine Learning Research (08/2025)
shown. The blue node's bounding box has been masked out. The middle figure shows the untouched (input) bounding boxes with labels in red, the one masked out in blue, along with the corresponding ground-truth image. The right figure shows our generated bounding box heatmap in white along with the target ground- truth bounding box (to be completed) in blue. The heatmap is obtained via generating the bounding box 100 times; the whiter the area, the more overlap at the location. Note that neither the image nor any image feature given the model for the completion task; the image is only for visualization.
31

---

Published in Transactions on Machine Learning Research (08/2025)
table
pizza
on
under
bock
tree building
taam
iding
Donind
behiin
train
track
het
tree
snow
hat
boot
woman
boot
woman
building
clock-2 roof
clock-1
Iding
window
835
headi hair
arm
surfboard
Figure 8: Grounded scene graph- image pair generation qualitative results on the Visual Genome dataset.
32

---

Published in Transactions on Machine Learning Research (08/2025)
Winco
building
window-1
window-
window-3
mnn
tile
shirt
temoring
man
glove
sign
Sitinng bn
bus
windshield
DUX
plate
MAnd
pole
window
WITE
bed
bed
lamp
pillow
curtain
Wind dand
logo
sign
train
TINII
windshield
door
engine
man
logo
Shilld
wheel
Figure 9: Grounded scene graph image pair generation qualitative results on the Visual Genome dataset.
33

---

Published in Transactions on Machine Learning Research (08/2025)
frisbee
Gefow
4love
dog
Duppur
surrannding
Arow
tree
trpedde
semounding
zebra
1004
straw
below
200re
sky-other
otherr
Atrvv
basiowe
Below
Insidee
building-other
boxt
Sa Juroonnding
belnve
river
tree
Surtunndng
abowe
snow
snowboard
Legend
bush
free
bush
above
drt
bench
right of
grass
Inside
grass
surrounding
dirt
Figure 10: Grounded scene graph image pair generation qualitative results on the COCO-Stuff datasct.
34

---

Published in Transactions on Machine Learning Research (08/2025)
fog
Gefow
person
Mlooe
persoot
slove
Tallww
playing-fie
nold
ding
sky-other
traffic light
belba
Frofficc
building- other
lndicle
surraunding
sioude
clouds
basioue
giraffe
BeloW
above
tree
THAM
graffe
DolOW
Neft of
right
straw
dini- other
sky-other
buitding-other
besove
Feft of
atoeve
bird
Inside
othir
Lurnountting
tree
Legend
persons
persons
sky-other
above
snow
persan-4
person-3
right of
Inside
person-2
surrounding
person-1
Figure 11: Grounded scene graph image pair generation qualitative results on the COCO-Stuff datasct.
35

---

Published in Transactions on Machine Learning Research (08/2025)
Window
Demind
LHMM
On
INmO
di ttting en
sable-3
0
head
FIOS
Hindow
roof
Hear
hars
Abvv
WAT
COWA--
haLd
1og
ant
has
ritl
Dunk
art
Ing-2
Figure 12: Single bounding box completion qualitative results the Visual Genome validation set.
36


Reviews:
[{"review_id": "TbJt4zCHMt", "reviewer": "Reviewer_o2mt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nTo enhance image fidelity and provide greater control in generative pipelines, this joint modeling approach is compelling, which is similar to DALL·E 2’s two-stage decomposition (which uses image embeddings instead of scene graphs). Modeling a complex marginal distribution through this factorization is a principled direction.\n\nA core contribution of this work is the unconditional scene graph generation, composed of a graph transformer denoiser combined with IoU regularization. This regularization explicitly models object locations, which has not been adequately addressed in prior works.\n\n**Weaknesses**\n\nMy main concern with this work is the practical effectiveness of the unconditional scene graph generation component. Scene graph-guided image generation is a promising direction for improving controllability in generative models, and many recent works have explored this avenue. However, unconditional scene graph generation remains less studied. I believe this is partly because scene graph generation itself is somewhat difficult to apply in many practical settings.\n\nTo strengthen the motivation of this work, could the authors consider conditional scene graph generation instead of the unconditional formulation? Specifically, rather than modeling the joint distribution P(S,I), it may be more effective to model P(S,I | C)=P(I | S,C)P(S | C), where C represents a class label, text description, or any other form of conditioning. If it can be demonstrated that explicitly incorporating a scene graph in this conditional framework improves the fidelity of image generation, or even if the fidelity remains similar but generation controllability improves, this would provide a much stronger and more practical motivation. Furthermore, such a formulation could be beneficial beyond generative modeling, such as in data augmentation for discriminative models, making it more broadly applicable than the purely unconditional approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "DiffuseSG presents a novel approach by integrating diffusion models for the joint generation of scene graphs and images. Unlike traditional methods, DiffuseSG can generate scene graphs unconditionally from noise, capturing both the structure and attributes effectively. Key contribution of this work: \n- Joint modeling: Simultaneously generates scene graphs and corresponding images, ensuring consistency between the two modalities.\n- Graph Transformer Denoiser: Employs a graph transformer as the denoising network within the diffusion process, handling both continuous and discrete attributes\n- IoU Regularization: Introduces IOU regularization to enhance the spatial coherence of generated objects.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Major comments**\n\nBased on my comments in the Weaknesses section, I encourage the authors to strengthen the motivation for why unconditional scene graph generation deserves more attention. While I suggested considering a conditional scene graph generation formulation as a possible alternative, I acknowledge that this may not align with the intended scope or contributions of the paper. Instead, I recommend that the authors more clearly describe the value and potential of the unconditional approach. Clarifying this motivation will help readers better appreciate the relevance and impact of this research direction.\n\n**Minor Comments**\n\n#1. In Section 3.1.2, the authors write:\n “Following Karras et al. (2022), we reparameterize the score function by a denoising function which maps the noise-corrupted data back to the clean data.” I suggest citing an earlier foundational work on this idea: Denoising Score Matching by Pascal Vincent (2010). While Karras et al. (2022) builds on this, Vincent’s work is more appropriate for introducing the original concept.\n\n#2. Please clarify how the total variation difference is computed. Specifically, how is the L1 loss between p^(x)\\widehat{p}(x)p​(x) and q^(x)\\widehat{q}(x)q​(x) calculated?\n\n#3. One key component of this joint model is the scene graph-guided image generation described in Section 3.2. Could you provide a performance comparison between your method (e.g., LayoutDiffusion and Relation-ControlNet) and previous works such as SceneGenie and SGDiff?", "broader_impact_concerns": "I didn't observe any potential ethical concerns or negative societal implications associated with the proposed method.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Awk0UrWMHr", "reviewer": "Reviewer_nn1a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed DiffuseSG method is novel, capable of generating nodes, edges, and bounding boxes, and demonstrates strong empirical performance in scene graph and layout generation tasks.\n- A variety of evaluation setups were explored.\n- Overall, the presentation of this paper is clear. \n\nWeaknesses\n- While this paper claims to introduce a novel joint scene graph-image generation framework, its primary contribution is in the scene graph and layout generation step. Its impact on scene graph-conditioned image generation seems limited, as it primarily adapts existing conditional image generation methods, such as LayoutDiffusion and ControlNet. These two processes remain quite separate rather than being truly \"joint\".\n- From my perspective, the formulation of image generation conditioned on scene graphs is standard and does not constitute a significant contribution.\n- Further clarification is needed in Section 3.1.3 regarding the technical novelty of DiffuseSG. It is unclear which aspects are novel and which are adapted from existing methods. An architecture diagram of DiffuseSG would be beneficial for understanding its technical innovations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel method called DiffuseSG, which effectively generates nodes, edges, and bounding boxes to construct scene graphs and layouts. It shows good empirical performance in scene graph and layout generation tasks compared to a range of existing methods. Additionally, its contributions are assessed across various related tasks, including conditional image generation, scene graph completion, and synthetic data generation to enhance scene graph detection.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The terms \"scene graphs\" and \"layout\" in this paper are somewhat confusing. It appears that the task of DiffuseSG blends elements of scene graph generation and layout generation. Its output includes node labels, bounding boxes, and edge labels (node relations), which differs from the typical definition of scene graphs in the literature, where only nodes and edges are included. This should be further clarified.\n- Baselines such as SceneGraphGen, VarScene, D3PM, DiGress, do not have default bounding box output, while baselines such as BLT, LayoutDM, do not output relation labels. A table that lists these features would be helpful for readers to get a clear idea of the each baseline.\n- Relation-ControlNet is an interesting method. A diagram to show the conditional image generation pipeline should be helpful to understand it.\n- Table 1, experimental results of DiffuseSG with different input representations (scalar, one-hot) with the IoU loss are missing.\n\nQuestions\n- LayoutDiffusion does not utilize relation labels by default. Are relation labels predicted by DiffuseSG used in your experiments of LayoutDiffusion-based conditional image generation?", "broader_impact_concerns": "There is no concern on the ethical implications of this work from my perspective.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "orBu1nYMVN", "reviewer": "Reviewer_3NQy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In general, this paper is well-written. The motivation is clear, and the logic flow is fluent. The notation system is clear self-contained. The scene graph generation task is an important task in vision generation, and the authors conduct a systematic research in the how well does diffusion-based method perform on this particular tasks. \n\nThe experiment sections are also admirable. The performance of DiffuseSG is comprehensively evaluated from both the graph generation quality aspect (Section 4.2) and the downstream scene-to-image generation aspects (Section 4.3). The Appendix sections provided sufficient details in reproducing the experiments.\n\n-------\n**Concern 1. Additional illustration in the training scheme.**\n\nI recommend the authors to add a detailed algorithm panel for clarifying the training process of DiffuseSG, which is outlined in Setion 3.1.3.\n\n**Concern 2. Additional illustration in the network designs.**\n\nI recommend the authors to add a detailed diagram for illustrating the network architecture design of the proposed DiffuseSG, including the best practice of encoding scheme, graph embedding, and graph transformers (i.e., visualizing Appendix A). \n\n**Concern 3. Additional illustration in conditional image generation.**\n\nI recommend the authors to add a detailed diagram for the process of adapting DiffuseSG into pretrained Relation / Object-ControlNet (i.e., elaborating and visualizing Appendix A.4).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed DiffuseSG, a diffusion-based unconditional graph generative model that generates high-quality scene graphs. Specifically, the proposed method is designed to jointly denoise the object nodes and object relation edges via a graph-transformer-based denoising network with two predictive heads. The contributions of this paper is three-folds:\n\n1. To the best of my knowledge, this paper is the first attempt that leverage diffusion-based denoising model in the scene graph generation task. \n2. This paper explores the network design, categorical variable encoding schemes, and  bounding box IoU penalty design in training diffusion-based scene graph generative models.\n3. The proposed DiffuseSG enables scene-graph-to-image generation by providing spatial conditioning controls to existing pre-trained vision generative models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Concern 1-3.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)