Synergistic Dual Spatial-aware Generation of Image-to-text and Text-to-image
Authors: Yu Zhao
Hao Fei
Xiangtai Li
Libo Qin
Jiayi Ji
Hongyuan Zhu
Meishan Zhang
Min Zhang
Jianguo Wei
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion
Spatial Understanding
Dual Learning
Primary area: diffusion_based_models

Abstract: In the visual spatial understanding (VSU) field, spatial image-to-text (SI2T) and spatial text-to-image (ST2I) are two fundamental tasks that appear in dual form. Existing methods for standalone SI2T or ST2I perform imperfectly in spatial understanding, due to the difficulty of 3D-wise spatial feature modeling. In this work, we consider modeling the SI2T and ST2I together under a dual learning framework. During the dual framework, we then propose to represent the 3D spatial scene features with a novel 3D scene graph (3DSG) representation that can be shared and beneficial to both tasks. Further, inspired by the intuition that the easier 3D$\to$image and 3D$\to$text processes also exist symmetrically in the ST2I and SI2T, respectively, we propose the Spatial Dual Discrete Diffusion (SD$^3$) framework, which utilizes the intermediate features of the 3D$\to$X processes to guide the hard X$\to$3D processes, such that the overall ST2I and SI2T will benefit each other. On the visual spatial understanding dataset VSD, our system outperforms the mainstream T2I and I2T methods significantly.
Further in-depth analysis reveals how our dual learning strategy advances.

Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-lmage
Yu Zhao', Hao Fei?; Xiangtai Li Libo Qin Jiayi Ji³ Hongyuan Zhu², Meishan Zhang", Min Zhang Jianguo Wei!
1 Tianjin University 2 National University of Singapore 3 Bytedance Central South University 5 P & CFAR, **START Harbin Institute of Technology (Shenzhen) zhaoyucs@t jue.t..cuuduudo.oo edu. cn, haofei37@nus edu.sg
Abstract
In the visual spatial understanding (VSU) area, spatial image-to-text (SI2T) and spatial text-to-image (ST2I) are two fundamental tasks that appear dual form. Existing methods for standalone SI2T or ST2I perform imperfectly in spatial understanding, due the difficulty of 3D-wise spatial feature modeling. In this work, we consider modeling the SI2T and ST2I together under a dual learning framework. During the dual framework, we then propose represent the 3D spatial scene features with novel 3D scene graph (3DSG) representation that can be shared and beneficial both tasks. Further, inspired by the intuition that the easier 3D 3D>>image and 3D>>text processes also exist symmetrically in the ST2I and SI2T, respectively, we propose the Spatial Dual Discrete Diffusion (SD$) framework, which utilizes the intermediate features of the 3D-+X processes to guide the hard X-3D processes, such that the overall ST2I and SI2T will benefit each other. On the visual spatial understanding dataset VSD, our system outperforms the mainstream T2I and I2T methods significantly. Further in-depth analysis reveals how dual learning strategy advances.
Introduction
Within the research topic of Visual Spatial Understanding (VSU)
42
95
Spatial Image- 74 89
representative task forms across vision and language. SI2T aims to understand the spatial relationships of objects in the given image, while ST2I focuses on synthesizing spatial-faithful image based on the input text prompts. Existing efforts mostly formalize SI2T and ST2I tasks as the normal 12T and T2I problems, applying general-purpose 12T and T2I models for task solutions 995 researchers widely employ the vision-language generative architecture 79
to-Text (SI2T) 195] 97] and Spatial Text-to-Image (ST2I) 58
Technically, for 12T tasks, i... generally, there is visual encoder and a text decoder. For the T2I task, recent diffusion-bart based methods have shown the extraordinary capability of image generation and achieved state-of-the -frt----t-ee-t (SoTA) performance on mount of benchmarks
97
58
12 28
18
Unfortunately, these strong-performin modeling in VSU tasks, largely fall short in precisely extracting spatial features from visual textual inputs. For instance, in the SI2T process, the spatial relationships are often incorrectly recognized due to layout overlap and perspective illusion [97]. This is due the inherent characteristics of 2D images, which, lacking 3D feature modeling, inevitably fail to understand spatial relations. On the other hand, in ST2I, synthetic images frequently fail match strictly the spatial constraints specified
12T and T21 methods, without the deliberate spatial semantics
"Corresponding Author: Hao Fei Institute for Infocomm Research (IRR) & Centre for Frontier Al Research (CFAR), A
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

in the prompts, such as the position, pose, and perspective [58]. This is because language's abstract nature allows only for general description of the content rather than detailed depictions of spatial scenes. Moreover, unlike general image generation, ST2I should place greater emphasis on 3D spatial modeling, while the input sequential language tokens intrinsically do not portray the kind of specific spatial scene.
Let us revisit the human process in solving SI2T and ST2I tasks. We typically process the input image or text within our minds by constructing a 3D spatial scene, which serves as the center for further spatial textual descriptions or image generation. Specifically for SI2T, we intuitively project a 2D image into a reasonable 3D scene based on common sense before describing that scene in words. In contrast, for ST2I tasks, we start by imaginatively converting the text of user instruction into 3D conceptual scene which is then rendered into 2D image. Interestingly, we can actually find that SI2T and ST2I are dual processes, with each task's s imput and output be- ing the reverse of the other, as illustrated in Fig- ure Ta). More importantly. there are two impor- tant observations in these dual tasks. First, [in- termediate processing sharing], they can com- plement and benefit each other. For SI2T, the 'Image+3D'' reasoning process is challenging in acquiring necessary 3D features, whereas the descriptive '3D+Teext' process is relatively eas- ier. Conversely, for ST2I, the 'Text=3D'' pro- cess requires complex reasoning of the 3D scene feature, while rendering 33D IID>Immae' is much more straightforward. Ideally, if complementing the information during each learning process, i.e., letting the easy part aid the hard part, it should enhance the performance of both tasks. Second, [3D scene feature sharing], both dual tasks urgently require modeling of the respective 3D features, where such stereospecific insights in 3D perspective can be essentially shared and also complementary between each other. Intuitively, vision offers very concrete clues describing spatial scenes, e.g., objects and attributes, while the 3D features derived from texts are more likely to define the semantics-orientl relations or constraints, which are often abstract and rough.
SI2T
Easy
Enteatay Strre' From 20 luage
Gonurniing Drseppane
Junm 10 Sonce
Proousiie
Preceding
72c
AH
The mother drivc in
and deaghtr
Banderng Amage
Kenoentay tD
fow
Easy
ST2I
Hard
(a) Complementary dual tasks of ST2I and S12T.
3DSG
plain
-belind
right
nundl
rol
"Car"
Thes"
family of mike one hoving hour ratther drin the soudhter bd
Sharel
"famlyy
Grape
Difloiss
"nsthert
"father"
divink
doughte
Yasal tada
Dosthe
"car"
(b) Diffusion Based 3DSG Generation.
Figure I: Demonstration of SI2T and ST2I tasks.
Inspired by such intuition, in this paper, we introduce novel synergistic dual framework for SI2T and ST2I. To start with, we propose a spatial- 3D scene graph (namely 3DSG). where the spatial objects, their relations and the layouts within the 3D scene are formulated with a semantically structured representation. The 3DSG representation effectively depicts the stereospecific attributes of all objects, and meanwhile models the spatial relations between them, from which both the SI2T and ST2I processes can be beneficial. Technically, 3DSG is obtained from a shared graph diffusion model for both SI2T and ST2I processes, as shown Figure Db.. Trained with our "DDSG-3DSG' pair data, the graph diffusion is learned to propagate and evolve the initial 2D visual SG (parsed from input image at SI2T side) or the textual SG (parsed from input text at ST2I side) into the final 3DSG, i.., by adding all necessary and reasonable spatial details.
With 3DSG, we next implement the whole dual framework of SI2T and ST2I generation. Instead of directly employing the SoTA generative 12T models or diffusion-based T21 methods, we consider a solution fully based on discrete diffusions 3 due to several key rationales. Primarily, for VSU, the most crucial spatial information that determines scene consists of objects and their relationships, which presents the characteristic of discretization and combination in the spatial layout, while other background and spatial unrelated information would be noisy. Thus, the discrete representation is more appropriate to model pure spatial semantics in our scenario. Besides, in our scenario both the textual token and SG representations possess discrete characteristics [54]. which can be perfectly modeled by discrete diffusion. Morcover, the discrete diffusion works on the limited index space
2

---

28 54 35 99], which is much more computationally efficient, especially for visual synthesis tasks. As illustrated in Figure 2 our Spatial Dual Discrete Diffusion (dubbed as SD*) system consists of three components: 1) 3DSG generation, 2) S12T generation and 3) ST2I generation. During dual learning, both S12T and ST2I first induce the 3DSG representations from the input image or text respectively via one shared graph diffusion, where the modality-varian features are simultaneously preserved in 3DSG for unbiased and holistic modeling of 3D spatial scene. Then, the image synthesis (for ST2I) and text generation (SI2T) are carried out via two separate discrete diffusion models, during which the 3DSG feature is integrated for better generation. At the meanwhile, the intermediate features of the '3D+X' (X means text or image) diffusion steps are also passed to the counterpart hard 'X-3D' processes for further facilitation.
We conduct experiments on the VSD dataset [95]. which is a VSU benchmark with paired images and texts of spatial descriptions that allow for both S12T and ST2I. Extensive results demonstrate that our proposed system significantly outperforms all baselines on both ST2I and SI2T, including the vision-language model based and diffusion-based methods. Further analysis reveals that the dual framework helps align the asymmetric spatial semantics across image and text modalities. To our knowledge, this is the first attempt to resolve the SI2T and ST2I tasks with a novel dual perspective, and further successfully investigate synergistic learning in between by sufficiently modeling the spatial 3D scene representations.
2 Related Work
Visual Spatial Understanding, VSU is an important topic within the research of multimodal
learning [51] 17 19 21) VSU aims to extract spatial information from a given scene, developed
within the forms of reasoning [88,, relation extraction [56), role labeling (4), question answering
[4 61 52). amage-to-text generation [95] 97), image synthesis [58]. 3D reconstruction [74] 69]
etc. With the VSU capability, many downstream applications can achieve, such roboties [24 (3).
navigation 33 80, 61, and language grounding [49] Among various VSU tasks, the S12T and ST21 generation attract significant attention due to their fundamental positions in vision-language cross-modal tasks. Current efforts mostly study ST2I or SI2T separately. For the text-to-imaae generation, the diffusion models have emerged as the SoTA approaches [15). For image-to-text generation, i.c., image captioning. it has been a long-standing task and has achieved great progress. Recent advances can be largely attributed to vision-language pre-training (VLP) (79 [O]. Besides conventional VLP methods, [99] are the first to use diffusion models for image captioning (and more specifically, for text generation). In this work, we consider the two tasks together under dual learning framework, via which we aim to achieve mutual benefits of the spatial feature modeling from each other.
Discrete Diffusion Models. The diffusion model is first proposed by 32 and has achieved 65 86
impressive performance for text-to-image generation 63 12
Original diffusion models are parameterized Markov chains trained to translate simple distributions to more sophisticated target distributions in finite set of steps on the continuous data or its latent representations. In this work we adopt the SoTA diffusion- based model as our T2I backbone. Recently, diffusion models
on discrete space are introduced to markedly reduce the computing cost (28 3 34 35). Discrete
diffusion methods are also used in text generation (34) and structure generation (54) due to its natural
adaptability for the data with discrete nature. This work follows the line and takes the SoTA discrete diffusion model as the backbone for image, text and scene graph generation.
Scene Graph Representations. This work is also closely related to scene graph (SG)-based represen- tation learning. Scene graphs have advanced in depicting the intrinsic semantic structures of scenes in images texts [43]]8] [93]. In SGs, key object and attribute nodes relations to describe semantic contexts, which have been shown to be useful as auxiliary features that carry rich contextual and semantic information for a wide range of downstream applications, such as image retrieval [39], image generation 40 86). translation 17.. image captioning [90] 83 and video modeling (96] [8] However, research utilizing 3D scene graph representations to enhance various downstream tasks can be currently very scarce. In this paper, we incorporate both visual and language scene graphs within 3D scope enhance cross-modal alignment learning for better spatial semantics understanding.
connected via pairwise
Dual Learning. The dual learning method is proposed to enhance the coupled tasks that have the same exact input and output but reverse [ 9T). With dual bidirectional learning, the model could capture potential mutual information from the primary and dual tasks, improving their performance.
3

---

Recently, the idea of dual learning has been applied to various tasks, such as intertranslation 88) speech recognition with text-to-speech [77]. question answering with question generation [76 75
and image classification with image generation [[6]. The key point of dual learning is to model the paired and complementary features between the coupled tasks, and then the learning process can reinforce them mutually. In this work, we first connect the dual S12T and ST2I tasks with their shared 3D spatial feature and use dual learning to enhance 3D feature construction. To our knowledge, we are the first to achieve synergy between two spatial- cross-modal dual generations.
3 Preliminaries
3.1 Task Formulation
We process the dual visual spatial understanding tasks, ST2I and SI2T. Given a textual prompt Y, ST2I aims to generate an image Í that semantically matches the spatial constraints with Y. Its dual task is SI2T, which is also known as the visual spatial description (VSD), aiming to generate a piece of textual description Ý based on an input image I. In this paper, we process the two tasks parallelly.
3.2 Discrete Diffusion
Diffusion models [32 are generative models characterized by a forward and reverse Markov process. In the forward process, the given data To with distribution ((xo) is conrupted into a Gaussian distribution variable æT in T steps, formulated as ((#1T)(20) I (1x) In the reverse process, the model learns to recover the original data xo from TT, denoted as (=(.)))) = p( p(xT) II2 po (2e). In order to optimize the generative model ps (20) to fit the data distribution ((tt)), one typically optimizes a variational upper bound on the negative log- likelihood:
T Bay (m)) [Der. | mo) (x)- [mm))]] | (1)
C.tb Eegioo DK.. (((=)|=))]lp(<))
Vanilla diffusion models are defined on continuous space, Recently, the discrete diffusion model has been introduced where transition probability matrix Qi is defined to indicate how Zy transits to Ie for each step of the forward process, where æ, E Z is defined in discrete space. The matrices [Quli g( i|æ;-1
j) defines the probabilities that x;-1 transits to Z,, Then the forward and reverse process could be rewritten as g( [z,-1) = a ) d (2;-1) and q( 1|22, mn) g( | 2o ( -1 /z)) / //((2///))))) where v(x) means the one-hot representation of æ. Appendix A. gives more technical details about the discrete diffusion models.
3.3 Scene Graph Representation
Table 1: Comparison of 3DSG and 2DSG.
The scene graph presents a scene via the objects with their attributes and relation-
TSG
ships in the form of a graph, which can
be constructed from either text (TSG)
Objects: Only key objects
Attributes: Any attributes
[68] or image (VSG) 922 46 45). We
Relationships: Semantic relationships between objects
represent a scene graph as G (V, E}.
where V is the node set, and E is the edge set. There are three types of nodes
2D VSG
Objects: The front objects in one perspective
in the scene graph: object nodes, at-
Attributes: Visual related attributes
tribute nodes, and relationship nodes. Each type of node has ts own unique tag vocabulary.
Relationships: Simple 2D spatial relationships
3DSG
For SI2T and ST2I, we focus on spatial
Objects: All the objects in the 3D scene; High-level
information extraction, especially the ob-
spatial concepts.
Attributes: Any attributes including 3D attributes such pose and texture.
ject relationships in the 3D space. How-
ever, due to the limited information pre-
Relationships: 3D spatial relationships and subordinate relationships between spatial concepts.
sented by text or image, a 2D TSG or VSG hardly fully models these 3D spa- tial semantics. Here, we introduce the apat 3D scene graph (3DSG) 41), which thoroughly depicts 3D scenes. The 3DSG formally equivalent to the 2D scene graph, i.c. the object, attribute, and relationship, but organized

---

ST2I
Reasoning 3D Scene from Text
Rondering Image from 3D Scrme
CLIP
6
Text Input
Graph Diflusion
Image Diffission
Visual Decoder
Parsing
Image
Gruph Encoder
Output
TSG
M 305G
VSG
Shared
ADD
Text Decoder
Parsing]
Output
Image Input
Tost Diffiusion
CLIP
SI2T
Estimating 3D Scone from 2D Image
Description from JD Scone
Representation
Representation
Discrete Image Representation
Discrete Representarion
Global Image/Text Feature
Diserete Representation
Indermediate Processing Sharing
3D Feature Sharing
Figure 2: Overall Framework of the D. The figure presents the dual processes of ST2I and SI2T. The RED block represents the hard X-+D processes, and the GREEN block represents the 3D-X processes. There three diffusion processes in total, i.e., shared graph diffusion model for VSG/TSG-+3DSG generation, and the image diffusion model and text diffusion model.
hierarchically, which contains more abundant elements such nodes for high-level spatial concepts, texture attributes, and spatial relationships in the 3D perception. Table I presents the differences between 2DSG and our 3DSG. Conventionally. the 3DSG should be constructed directly from a 3D scene, i.c., sensor data [64]. point clouds (78). 3D mesh (2], and RGB-D sequences
(87 26). In this
work, we uniform the VSG/TSG/3DSG representations, modeling the object nodes Vobj. attribute nodes Vattr, and relationship nodes Vret by the embedding of their textual tags Cobj, Cattr, Grei E Rdd where d is the dimension of the tag embedding.
Methodology
4.1 Spatial Dual Discrete Diffusion Framework
In Figure 2 we illustrate the overall framework of the proposed Spatial Dual Discrete Diffusion (SD*), consisting of three separate discrete context diffusion models, i.e. the 3DSG diffusion model for the X-3D process, the ST2I diffusion model for the 3D Image process, and the SI2T diffusion model for the 3D-xText process.
X->D: 3DSG Diffusion Model. We set graph diffusion model convert the initial TSG (for ST2I) and VSG (for SI2T) to the 3DSG. We first acquire the initial VSG and TSG from the input image and textual description through the off-the-shelf VSG [68] and TSG [92
parsers. Then the initial VSG and TSG nodes (represented as [Cobj eattr: Crei) }) are encoded to the latent quantized representations by discrete graph auto-encoder (DGAE) (54), which the discrete diffusion process discrete graph representations is conducted to generate the 3DSG. Concretely, we denote the quantized gold Groph (>{]|>{{) 3DSG as z5¹ Then, following conventional discrete diffusion, we can calculate Ps Moreover, on our final dual training, the 3DSG diffusion model leverages the intermediate feature of the following 3D process aid the TSG-3DSC and VSG-+ 3DSG generation process. For TSG-3DSG, the model takes global text features and the intermediate feature of the dual SI2T diffusion, i.e., Pv
Graph (z 1 ),,,).) ).).p Graph (xL xxx), and for VSG-3DSG it will be
Graph (z 15 zf), where the z and z are the step representations of SI2T and ST2I diffusion respectively. Then we align them by:
[=x_]|>ح (2 -.)., Ps
Groph
Py
Lx- = DKL(pg Graph G 12 "lp Groph (2 2Y)),
(2)
-x-1230 = DKI.P,
Graph (z G cf )1l
Graphh (2 12 2()).
Graph
To enable the training for this 2DSG-3DSG process, we should construct the paired 'DSSG3DDSC"
data. We follow previous work 72 78 27 to construct gold 3DSG dataset. After that, we adopt strong captioning model to generate text descriptions from view images. With all the inputs and
5

---

outputs ready, we can train the graph diffusion model on aligned 3DSG-Image-Ten Text data, after which the model will acquire the 3D estimation capability.
3D 3D=Immaee: ST2I Diffusion Model. The ST2I diffusion model is used to generate image with the condition of input textual prompts Y. this task, we adopt a vector quantized variational autoencoder (VQ-VAE) 14 as the quantized model to encode image data to embedding vectors. We denote the quantized image as z and we could also calculate (( x<{-1) and ((zf-1>z(. zt). On the other hand, the ST2I diffusion model takes two conditions, i.e., the global text feature cx extracted via CLIP model [60]. and the shared 3DSG feature c.. Then ex and c are fused and fed to the diffusion model to calculate p ST22 [z) 2 c" ① c) At last, the decoder from VQ- VAE used to generate images from latent codes.
3e=xText: SI2T Diffusion Model. In the SI2T model, the inputs and outputs are reversed. The diffusion is applied on the text representation z) with similar process as ST2L. The latent text representation the word embedding, which functionally equivalent the visual codebook. We use another denoising network p2 S127 (z xy c' a c²), while the input condition becomes the
visual tokens e and also the 3DSG feature e Afterward, the language model decoder generates textual results based on the latent representations.
Image and Text Decoding. Finally, through three diffusion models, we acquire the predicted latent representations of image, text, and 3DSG, denoted as E/. E & For ST2I, we fuse the generated graph vectors Ef to visual vectors E/ by adopting an attention mechanism:
Attni softmar @6 [m] E
+
Σ Attr
26 [m],
(4)
where @ means the concatenation operation. Afterwards, the image decoder Gy is used to reconstruct image from the scene graph enhanced codes 2 2'+:
i i Gy
(5)
For S12T, we append the flatten o after generated textual codes Ey and use language model decoder Gr to generate the description:
Gr (0))
(6)
4.2 Mutual Spatial Synergistic Dual Learning
To exploit the complementarity of the dual processes, we elaborate dual training strategy for our framework. To conduct the strategy effectively, we introduce the essential training objectives.
Dual Learning Objective. Our dual learning framework contains two tasks, i.c. ST2I and SI2T, where their inputs and outputs are just reversed. We denote ST2I as fo : i + U E I, E Y and SI2T as fo y i, I, E Y. Their learning objectives should be:
LsT21 Ei,y log (v())) Lsr2r = E,,y log (yll).
(7)
Based on the dual supervising learning (8). given the duality of the tasks, if the learned ST21 and S12T models are perfect, we should have the probabilistic duality:
(())) p(y)po(ily) = p(i, Vi, y,
8
where p(t) and p(y) are the marginal distributions. Then we add this constraint to the dual learning target as equivalent regularization term:
Ldval = LST21 + Ls12r + log + log po(li)) - log p(y) - Po
Pa (ily)]l.
(9)
where p(1) and log p(v) are the estimated marginal distribution by pre-trained vision-language model, which is illustrated detailly in Appendix $A.4
Loss for Mutual 3D Feature Learning. The 3DSG diffusion model used to estimating 3D scene information from TSG VSG, denoted fo : + 9 and fo y + g, where E I, y Y.g G. Based I and Eq. [ the graph diffusion loss can be acquired, denoted as L12AD and Lr23D
L Ei,yogg log (ogl)()))), LTODD E,, log Po (gly))
(10)

---

where 2 and z are the intermediate features of image diffusion and text diffusion. Along with the X->D alignment loss -x-223 and --31D3 the final mutual 3D Feature learning target is:
Lmuttul = L123D + Lx-123D 123D + LT23D + Lx-T23D-
(11)
Loss for Spatial Feature Alignment. The paired latent visual and textual representations are initialized in different feature spaces. We thus conduct a spatial feature alignment to bridge them by the shared spatial representation of the 3DSG. Concretely, we consider the image decoder, text decoder and graph encoder. Given the paired image I, text Y and the corresponding 3DSG Ghosts we can easily get their quantified representations zf, = =f= and then the fused feature =0 [=0 ] based on the Eq. 5 and Eq. 6 After that, we adopt the reconstruct loss of VQ-VAE Ce and the next token prediction loss Le-dec of text decoder for optimization:
/+ and
ly
-eeden 1| Gr(=
Lt-den =
log Log
(12)
Loss for 3DSG Reconstruction. To initialize the graph encoder and decoder, i.e., the DGAE, we follow [5] to calculate the graph reconstruction loss:
LDGAE = -a (hopp 129)),
(13)
where 20 is the latent representations of the predicted 3DSG and the Ghost is the gold 3DSG,
Training Remarks. To achieve perfect conver- gency, we use four-step strategy for the overall training process, shown in Figure 3 Step-1 DGAE pre-training. First, we pre-train the DGAE model with the 3DSG reconstruction target LDGAE initialize the DGAF and the graph codebook, with the self-supervised train- ing on gold 3DSG data. For this step, we spe- cially choose the 3DSG datasets [72, 78, (2) for training and the graph codebook will be well initialized. Step-2 Spatial Alignment. Then we tune the visual and text decoder with the loss in Eq. 12 During this step, only visual decoder and text decoder are updated with the loss Lv-dec + Li-dec- Step-3 2DSG-3DSG Diffusion Training. In this step, we use the con- structed aligned 3DSG-Imaag- data ($4.1 to train the 2DSG~3DSS graph diffusion, thus the model will acquire the prior 3D estimation capability, The optimization objectives here is Lr23 + LraaD. Step-4 Overall Training, Fi- nally we tune the whole model with dual learn- Figure 3: Illustrations of the training steps of 3D3
Step-1
Step-2 1 Visual Decoder
G Graph Decoder
Text Decoder
Graph Encoder
Graph Encoder 1
Visual Encoder
Text Encoder t
G 30
Step-3
Graph # Encoder
Graph Difluser
Graph Decoder
G3
G. G,
Step-4
* Visual Decoder
Graph Encoder
Image Diftuser
i
G
Graph
Diffuser
Text Decoder
Text Diffuser
Module Updating
Module Frozen
ing objectives, where we update the three diffusion models and freeze other modules. The overall learning target is Ldual + Ematwal-
After above training steps, the SI2T ST2I be launched alone without the intermediate process sharing. At this point, the critical graph diffusion module has learned sufficiently well from the previous training, so without the dual task aiding, the 3DSG generated during inference is also of high quality for the following >D+XX generation.
Experiment
5.1 Experiment Settings
Dataset. To demonstrate the capability of our proposed method for both ST2I and SI2T generation, we conduct experiments on the VSD (95 997] dataset, which constructed for visual spatial under- standing. The VSD dataset contains about 30K images from SpatialSense [89] and Visual Genome
7

---

Table 2: Main results the 256 256-sized VSD dataset with 200 DDIM steps. Bold numbers are the best and the Underline numbers denote the best baselines. Our results are averaged on five running with different seeds.
VSDv1
VSDv2
ST2I
SI2T
ST2I
SI2T
FID
1ST
CLIP
BLEU4
SPICET
FID,
IST
CLIP
BLEU41
SPICET
'21 Baselines
DALLE
62
32.55
17.01
62.16
28.52
21.18
64.58
Cogview
13
32.30
17.07
61.85
28.17
21.74
64.76
LAFITE
30.73
24.39
25.73
25.47
VQ-Diffusion 28
18.34
20.58
63.42
15.66
24.75
66.30
Friodo
15
12.86
25.92
64.65
11.41
26.02
67.01
12T Baselines
3DVSD
97
54.85
68.76
26.40
46.97
MNIC
34,21
66.87
20.01
43.88
FNIC
37.03
66.50
22.62
43.52
DiffCap
29
34.75
66.39
20.27
43,30
DDCap
99
37.93
67.10
23.14
44.07
Singleton
18.05
20.42
63.51
48.77
66.59
14.70
24.62
66.41
23.51
43.70
Singleton 3D
12.56
26.92
65.62
50.05
67.20
10.43
25.62
67.29
25.37
45.13
Vanilla Dual Learning
11.80
27.85
67.18
51.59
67.79
11.67
27.80
68.46
26.10
46.72
SD (Ours)
11.04
29.20
68.31
56.23
68.02
10.09
29.76
71.10
27.63
48.03
43], and each image in this dataset has aligned text for spatial scene description. Now, VSD has two released versions, VSDv1 and VSDv2, where the spatial descriptions in VSDv2 more meticulous.
Evaluation metrics. For spatial text-to-image synthesis, following prior work, we adopt Fréchet Inception Distance (FID) [66]. Inception score (IS) (31) and CLIP score (30] evaluate authentieity
of generated images. For spatial image-to-texs we follow (95] and adopt BLEU4 (55) and SPICE measure text generation. Furthermore, perform human evaluation compare the spatial understanding effectiveness. Appendix B details more experiment settings.
Implementation Details. We use the pre-trained VQ-VAE of VQ-GAN [14), which leverages the GAN loss to get more realistic image. It converts 256 256 images into 32 32 tokens. We follow (88 and remove the useless codes and obtain codebook with size K=2886. For text encoder, we adopt the CLIP model and its tokenizer, which has 77 as the max token lengths. We adopt the pre-trained GPT-2 (59 the text decoder, following its default settings. We freeze the VQ-VAE encoder and CLIP while training. For the user Unet to learn the denoising process for all the three diffusion models. We follow the default settings of DGAE (54) for the graph diffusion model. We optimize the framework using AdamW (50) with 3 0.9 and B3 0.98. The learning rate is set to 5e-5 after 10,000 warmup iterations in the final dual tuning.
5.2 Quantitative Results
Main Comparisons. The main results are shown in Table We first compare some strong baselines of 12T and T21 methods. Overall, the ST2I results on VSDv2 are consistently better, and the SI2T results VSDv1 are better. This is because the VSDv2 contains more complicated and detailed textual descriptions, which is beneficial ST2I but challenging for ST21. We see that the proposed SD3 substantially outperforms the compared baselines both VSDv1 and VSDv2. For the ST2I task, we outperform Frido by 1.82% FID, 3.28% IS. and 3.66% CLIP score on VSDv1 and 1.32% FID, 3.74% IS and 4.09% CLIP score VSDv2 For the SI2T task, surpass 3DVSD by 1.38% BLEU4 and 0.74% SPICE on VSDv1 and 1.23% BLEU4 and 1.06% SPICE on VSDv2. The results directly demonstrate the efficacy of our method.
further show the key module ablation studies on last four lines Table First, removing the 3DSG integration ("Vanilla Dual Learning"), i.e., train the SI2T and ST2I diftusion with loss of Eq. 9 the performance drops much on main metrics, indicating the critical influence of 3DSG guidance. Also, the drops of "Singleton+3D" reveal the effectiveness of our dual learning strategy.

---

GT Image
VQ-Diffusion
Fride
SD
Generated 3DSG
living
laptop
frent
room
10om
couch
On
next
DDCap
man is couch next dog laying down.
A man sitting on chair with dog his lap.
man sitting the sofa.
mon is sitting the couch with a dog aside
floor
lie
Immae
VQ-Difllsion
Frido
Generated
train
railway
stalbon
Behind
sky
bluc
track
Frautt
(weed
GT Text Some weeds the middle in front of the railway tracks.
DDCap train traveling down tracks next to forest.
3DVSD Some weeds lie the right of the railway tracks,
SD Some weeds grow in front of the railway tracks.
forest
pole
next
Figure 4: Qualitative results by different models, where the samples are selected from VSDv2.
When removing both 3DSG modeling and dual learning ("Singleton"", the model decay the simple discrete diffusion model, which is just comparable to the VQ-Diffusiio and DDCap model.
Spatial Evaluation. To compare the model's effectiveness, we conduct human evaluations for both ST2I and S12T. We follow 197] to ask ten volunteers answer the 5-point Likert scale on 100 samples. For SI2T, we collect scores for Spatial Description Accuracy and Spatial Description Diversity. For ST2I, we collect scores for Visual Spatial Accuracy. The results shown in Table 3 Overall, 3D method shows the best spatial understanding capability on S12T and ST2I. More details are put in Appendix
Table 3: Human Evaluation for Spa- tial Accuracy.
ST2I
SI2T
SpaAce.
SpaAcc.
Div.
3D
3,86
4.03
3.62
Frido
3.04
3DVSD
3.03
2.76
5.3 Qualitative Results
To present the ST2I and SI2T performance of our model in a more intuitive way, we show some qualitative results in Figure 4 For ST2I, compared with Frido and VQ-Diffusion, our SD3 generates more spatial- images. For SI2T, compared with 3DVSD and DDCap, our method is able to extract and correctly deseribe the key spatial information.
We visualize the generated images and texts using different methods.
SD
Vanilla Dual
Table 4: Comparing the gen- erated 3DSG with the gold 3GSD with TriRec.
3DSG I VSG la TSG
Frido
3DVSD
16
86
14
84
Hydra (GT)
100
12
82
Initial TSG
57.61
2
10
80
Initial VSG
24.20
STZI SI2T GLIP GLIP
ST2I SIZT TriRec. TriRe..
w/o DiffPre.
56.82
Obj.
Attr.
3D3 w/o Xfeat.
65.71
Figure 6: The change of av-
Figure 5: The evaluation of struc- ture matching on VSDv2.
Full 3D3
84.51
erage number of SG nodes.
5.4 In-depth Analyses and Discussions
Previously, we have verified the effectiveness of our dual learning by thorough numerical evaluations. Now, we explore how our methods advance via the following research questions.
How does 3DSG guidance aid the generation of spatial-faithful image and text? First, after removing the 3DSG integration, comparing the last two lines Table we can see that the 3DSG feature contributes great influence. Further, we follow GLIP 47 to assess how well the objects-

---

attribute matching between imput image/text and generated ones, Also, we assess how well the subject-predicate-bieee -oojectt---ooeec triplets of the gold TSG and VSG could be retrieved in the ones from the generated image/text via the triplets recall (TriRec.) between scene graphs. We compare the SoTA S12T and ST2I baselines and the results are shown in Figure 5 We observe that our SD significantly outperforms the model without 3DSG guidance ("Vanilla Dual") on the two metrics, which is just comparable to the baseline models. This suggests that with the 3DSG guidance, the model could capture correct and sufficient spatial structures from inputs for the generation.
How does the dual intermediate sharing aid the graph diffusion model? On one hand, we report the performance of 3DSG generation on the gold 3DSG dataset [78] to evaluate how well the structures of generated 3DSG could be retrieved in the gold ones. As shown in Table[4 we can see that compared with the initial VSG/TSG the generated 3DSG of our full model highly matches to the ground truth while without the dual intermediate sharing ("w/o Xfeat."), i.e., remove the -X-123D and L -233 in the X-+D diffusion, the performance dramatically drops, showing the effectiveness of intermediate process sharing. Besides, if the graph diffusion pre-training is removed ("w/o DiftPre."), it will lose the ability to generate 3DSG (degrade to initial TSG/VSG). On the other hand, we assess how much the graph diffusion model produce the new structures, compared with the initial TSG/VSG. As shown in Figure 6 after graph diffusion, the average numbers of the three types of elements are enriched substantially.
The influence of the training strategy. To show the effectiveness of our four- step training strategy, here we conduct an objective ablation study. We investigate the model without DGAE pre-training in step 1, the model without spatial align- ment in step 2, and the model without 2DSG-3DSG in step 3, respectively.
Full
w/o DGAE
w/o Spatial Alignment
w/o 2DSG-3DSG
70
SPICE
68
CLIP
68
66
66
64
64
62
62
As shown in Figure 7 we can see both ST2I and SI2T performance decreases when removing any training steps. No- tably, the 2DSG- step contributes the most to the final results. The essential 3D understanding function lies in the capability of the graph diffusion module, which should be largely enhanced via this step.
(a) ST2I
(b) SI2T
Figure 7: Comparison of different training strategy.
Conclusion
In this work, we study the ST2I and SI2T tasks under a dual learning framework. We first propose the spatial-aware 3D scene graph to model the 3D feature which is essentially shared by ST2I and S12T. The 3DSG is constructed by being initialized with a 2D TSG/VSG and then evolved to the 3DSG by a graph diffusion model. We then leverage the dual learning to enhance the 2DSG-+3DSG evolving by sharing clues of the dual 3DSG->Iage/Tee adequately guided and then facilitates the whole ST21 and S12T. On the VSD dataset, our method shows great superiority in both SI2T and ST21. Further analyses demonstrate how our dual learning method could capture 3D spatial structures and then help generate spatial-faithie images and texts.
process, through which the 3DSG diffusion model is
Looking forward, there can be quite rich explorations in future research. First, the graph diffusion model highly relies on the quality of 3DSG training data, which has significant impact the final performance of our method. Correspondingly, we in the future will explore the data augmentation methods to optimize the training. Second, the evaluation for spatial understanding of ST2I and SI2T has not been fully explored in this work, i.e., with human evaluation instead. Also, one promising direction for VSU is constructing multimodal large language models (MLLMs) 22 g5
57 82), especially for 3D spatial understanding (9.
Acknowledgements
This research is supported by National Natural Science Foundation of China (NSFC) under Grant 62336008, A*STAR AME Programmatic Funding A18A2b0046. RobotHTP Seed Fund under Project C211518008, and EDB Space Technology Development Grant under Project S22-19016 STDP.
10

---

References
[1]
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould, SPICE: semantic propositional image caption evaluation. In Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Purt V. volume 9909 of Lecture Notex in Computer Science, pages 382-398. Springer, 2016. doi: 00.10077978--33194444444 24. URL https://Gox org 1007/ 1-1
[2]
Armeni, Zhi- Yang He, Amir Zamir, Jun JunYoung Gwak, Jitendra Malik, Martin Fischer, and Silvio Savarese. 3d scene graph: structure for unified semantics, 3d space, and camera. In 2019 /EEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November URL / //c..//ciiioo org/
2, 2019, pages 5663 5672. IEEE, 2019. doi: 10. ww..1199/CV221190000
10. 10..119//10 wwD//I//C.... 2019. 2019.00576
[3] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Advances in Neurul Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrlPS 2021, December 6-14.
2021, virtual, pages 17981 17993, 2021. URL https /ppccocaiiocccccccco.o.o
www.innggs.nnurrgg nwwrips cappaper/ 2021/hash/ (aaah/958c5500 ///////5653555555 7abcc/ee478888333e 7ww.:/6997777577007979 70 01t.2//777777777-----aaaaa......
[4] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 1737- 1752. PMLR, 2023. URL https:t:::// http:///ooceeedeeeeeeeoee mll press/ satess/v202/. b html
[5] Yoann Boget, Magda Gregorova, and Alexandros Kalousis. Diserete graph auto-encoder. Transactions on Machine Learning Rexearch, 2024, ISSN 2835-8856.
b280bowb3
[6] Allan M. C. Bretas, Alexandre Mendes, Martin Jackson, Riley Clement, Claudio Sanhueza, and Stephan K. Chalup. A decentralised multi-agent system for rail freight traffic management. Ann. Oper. Res.. 320(2):631-666 661, 2023, doi: 10.1007/s10479-021 -411788x. URL 810479-021-04178-8
[7] Angel X. Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halher, Matthias Nicßner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from RGB-D data indoor environments. In 2017 International Conference on 3D Vision, 3DV 2017, Qingdao, China, October 10-12. 2017, pages 667-676. IEEE Computer Society. 2017. doi: 10.1 1099 URL
https::/doi.co org/ 10. 1109/3DV 2ww.:/D.........0 000E1
2017
[8] Boyuan Chen, Zhuo Xu. Sean Kirmani, Brian lchter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas J. Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. CoRR, abs/2401. 121168............. 2024. doi: 10.4555/ARXXI224011122e.
[9] Sijin Chen, Xin Chen, Chi Zhang. Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the JEEF/CVP Conference on Computer Vision and Pattern Recognition, pages 26428 26428-26438, 2024.
[10] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-langusae tasks via text generation. In Proceedings of the 38th International Conference Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings Machine Learning Research, pages 1931-1942. PMLR, 2021. htLl/attp://poceeeee
mi am...mmrrrpp.... www.s.....ppp..................a /139///300 hww///ho21aaaaaaaeeeeeaea1aaaaaa
cho2la. html
[11] Fatemeh Daneshfar, Ako Bartani, and Pardis Lotfi. Image captioning by diffusion models: A survey, Engineering Applications of Artificial Intelligence, 138:109288, 2024.
[12] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image syn- thesis. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 8780-8794, 2021. URL htt.:pp:/procceeiiee .neuuriips.s........ooe.m.. pt/pappr/202////eeeeep 49ad23dlec9fa (Dw.ee9ff4ad87700000022 wtw.aa/fsf.aaassttttttt.m Abstract.
[13] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: Mastering text-to-image generation via transformers. Advances Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 19822 19835, 2021. URL U asdd/eeccbbs1ccceeeeeeee re4 wet.:eecaaaeeeeeeaaaeaaaaa
www.ggg...e...p....... httpppps.cpppprr//eeeeee hatpeeer///////aahaa Abst mtmm

---

[14] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis.
In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021. virtual. June 19-25, 2021,
pages 12873-12883, Computer Vision Foundation IEEE. 2021. doi: D0.11999CPP44444722200000
URL attps:// http///pccccccsccooooo theaacccssssss.eeeeeeeeeee com
www.///////ooo////ooooooooocooct content/ /Wtp///PR22222 http:///wwww.eee....eeeeasa Tami ng.
CVPR 2wt.__222____PPPeee_ html
[15]
Wan-Cyuan Fan, Yen-Chun Chen, Dongdong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido:
Feature pyramid diffusion for complex scene image synthesis. In Thirty-Sevntth AAAI Conference Artificial Intelligence, AAAI 2023, Thirry-Fifth Conference on Innovative Applications of Artificial Intelligence. IAAI 2023. Thirteemth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 579-587, AAAI Press, 2023, doi: 10.1609/ AAALV3711.2513 URL.tttss:///
wttp:///.......a.a.............. 22111
[16] Hao Fei, Shengqiong Wu, Yafeng Ren, and Meishan Zhang. Matching structure for dual learning. In Proceedings of the International Conference on Machine Learning, pages 6373-6391, 2022.
[17] Hao Fei. Qian Liu, Meishan Zhang, Min Zhang, and Tat Seng Chua. Scene graph as pivoting: Inference- time image-fiee free unsupervised multimodal machine translation with visual scene hallucination. In Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 1980-5994, 2023.
[18] Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, and Tat-Seng Chua. Dysen-vdm: Empowering dynamies-aware text-to-video diftusion with 1lms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7641-7653, 2024.
[19] Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thoouht: Step-by-step video reasoning from perception to cognition. In Proceedings of the International Conference on Machine Learning. 2024.
[20] Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision Ilm for understanding, generating, segmenting, editing. 2024.
[21] Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua, and Shuicheng Yan. Enhancing video-language representations with structural spatio-temporal alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[22] Hao Fei, Yuan Yao, Zhuosheng Zhang, Fuxiao Liu, Ao Zhang, and Tat-Seng Chua. From multimodal Ilm to human-level ai: Modality, instruction, reasoning, efficiency and beyond. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation PIRECCCOLING 2024): Tutorial Summaries, pages 1-8, 2024.
[23] Zheng-cong Fei. Fast image caption generation with position alignment. CoRR, 1bs/1912.00365, 2019. URI.http: //atxiv//aaxvvv.o arg//bs/// 260665
[24] Jonathan Francis, Nariaki Kitamura, Felix Labelle, Xiaopeng Lu, Ingrid Navarro, and Jean Oh. Core challenges in embodied vision-language planning. J. Artif. Intell. Res., 74:459-515, 2022. doi: 10.1613/ jair. 1336.. URL BtRphhtts:///oo wtw.o/o.oooooooooooooeooooo 100.1.1......... 1613/ jair 1. 13646
[25] Junlong Gao, Xi Meng, Shiqi Wang, Xia Li, Shanshe Wang. Siwei Ma, and Wen Gao. Masked non- autoregressive image captioning. CoRR, abs/1906.00717, 2019. URL http:://rxiv. org/ abs/
00717
[26] Nishad Gothoskar, Marco F. Cusumano- MusunennoTTowner, Ben Zinberg. Matin Chavanizadeh, Falk Pollok, Austin Garrett, Josh Tenenbaum, Dan Gutfreund, and Vikash K. Mansinghka. 3dp3: 3d scene perception via probabilistic programming.
Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrlPS 2021, December 6-14, 2021, 2021. URL https:::/proceediinoooe virtual, pages SaS.
wwwrips....p.......... wwt.rrips.cccppppeeee 22/2 html
Bash/4Ec66104Eeada666
[27] Elias Greve, Martin Büchner, Niclas Vädisch, Wolfram Burgard, and Abhinav Valada. Collaborative dynamic 3d scene graphs for automated driving. CoRR, 0ttp//239.066635 2023. doi: 10.48550/ARXIV. 2309.06635. URL URL htttts:://////// http:////c...o////0.40///////// www.:0.8850000.8..
06635
[28] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model text-to-image synthesis. In IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10686- 10696. IEEE, 2022. doi: 1011099CCVR52268222222000
URL b /ooi org 10. 1109/
CVPR52638 2022 01043
12

---

[29] Yufeng He, Zefan Cai, Xu Gan, and Baobao Chang. Diffcca: Exploring continuous diffusion on image captioning. CoRR. abs/2305. 121444 2023. doi: 10.48550/ARXXV.23005.. orq/ 10. 4ww548/0008..222222
URL httth::/taa.. //coi
[30] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: reference- free evaluation metrie for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Procesxing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7514-7522 Association for Computational Linguistics, 2021. doi: 10. 18653/V1 2021 WHINL--MAIN
htttps://dci.. org/ 10. 1ww138816/222222 emnip ewn.p-main..n... maii..999
[31] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA,
pages 6626 6637, 2017. URL httpts://pp//.eesses http::////pccceiiiieeeeoeoe ngs wet.rneurrpp.. iparips... Bal 1a11647777e00ffff65557 oot.r//e6e....11111222222ooooo.. html
[32] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion prohabilistic models. In Advances in Neural Information Processing Systems 33; Annual Conference on Neural Information Processing Systems 2020. NeurlPS 2020, December 6-12, 2020, virtual, 2020, URL https: : http::///oceeeeeeeeeeeee neur 01 /9ca4b- wwstract. .tmm
ce/paper 2020/ hww./aaahhhaeaaaa https///ehheeeeceeeeeeeee 384
[33] Yicong Hong, Cristian Rodriguez Opazo, Qi Wu, and Stephen Gould. Sub-instruction aware vision- and-language navigation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 3360-33766 Association for Computational Linguisties, 2020. doi: 100116663/11/222..mmmpmmmm
URL URL httpss:///d/ .arosi.orrg /org/
10. 121138861112000 emn mam..ann.... 271
[34] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Towards non autoregressive language models. CoRR, abs/2102.05379, 2021. URL
kttps://arxiv> www.awwwww...../o.ooaooeeeeaaeaa
2102. 05379
[35] Minghui Hu, Yujic Wang, Tat-Jen Cham, Jianfci Yang, and Ponnuthurai N. Suganthan. Global context with discrcte diffusion in vector quantised modelling for image generation. In /EEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pages 11492-11501. IEEE, 2022. doi: 10. 10 2022.01121. htLp:t /ttttp:://di..oo orgg/
10. 10.1109/CVPR52688 2 2022. 01121
[36] Nathan Hughes, Yun Chang, Siyi Hu, Rajat Talak, Rumaisa Abdulhai, Jared Strader, and Luca Carlone. Foundations of spatial perception for robotics: Hierarchical representations and real-time systems. CoRR, abs/2305.00 07 154, 2023, doi: 10.48550/AXXI..330000777
URL/hctps://dbl http://oo.orrg///////// 488500/
arxiv v. 2305 2305.07154 07154
[37] Jiayi Ji, Yunpeng Luo, Xiaoshuai Sun, Fuhai Chen, Gen Luo, Yongjian Wu, Yue Gao, and Rongrong Ji. Improving image captioning by leveraging intra-and inter-layer global representation in transformer network. In Proceedings of the AAAI conference on artificial intelligence. volume 35, pages 1655-1663, 2021.
[38] Jiayi Ji, Yiwei Ma, Xiaoshuai Sun, Yiyi Zhou, Yongjian Wu, and Rongrong Ji. Knowing what to learn: metric-oriented focal mechanism for image captioning. IEEE Transactions on Image Processing, 31: 4321-4335, 2022.
[39] Justin Johnson et al. Image retrieval using scene graphs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. htLLattps://// wtwp///axvvvv/. http:aa://aartt..oorr/aaa///////
htagg/ab//15000 00325
[40] Justin Johnson et al. Image generation from scene graphs. In European Conference Computer Vision (ECCV). 2018. bttpss/: wrw. abs 1703.60 07370
[41] Ue-Hwan Kim, Jin-Man Park, Tick-Jin Song, and Jong-Hwan Kim. 3-d scene graph: A sparse and semantic representation of physical environments for intelligent agents. IEFE Trans. Cybern., 50(12): 4921-4933, 2020. doi: 10. 1011TCYY.22001 2991922.914.. WtRthhttss://do .eriiorrg//i 10. 1109/TCYH 2019.2931042
[42] Parisa Kordjamshidi, Taher Rahgooy. Marie-Francine Moens, James Pustejovsky, Umar Manzoor, and Kirk Roberts. CLEF 2017: Multimodal spatial role labeling task working notes. Working Notes of CLEF 2017 Conference and Labs of the Evaluation Forum, Dublin, Ireland, September 11-14, 2017, volume 1866 of CEUR Workshop Proceedings. CEUR-WS.org WS.org, 2017. URL http:://////.s....p org/ Vo1-1866/ hat///////iiieeeeee invited paper 19. ...di
13

---

[43] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. Comput. Vix.. 123(1):32-77 73, 2017. doi: 10.1007/S11263- -16-00--7------- URL
111263-016-0981-
[44] Hao Li, Jinfa Huang, Peng Jin, Guoli Song, Qi Wu, and Jie Chen. Toward 3d spatial reasoning for human-like text-hased visual question answering. CoRR, abs/2209. 2022. doi: 10.48550/ARXIV. 2209.10326. httpp:t:t////
2209. 2209.10326
[45] Li Li, Wei Ji, Yiming Wu, Mengze Li, You Qin, Lina Wei, and Roger Zimmermann. Panoptic scene graph generation with semanties prototype p learning. Proceedings of the AAAI Conference on Artificial Intelligence. 38(4):3145 3153, Mar. 2024. doi: 10. 160166//a..33844422800. org/ wn/e/ndexx.../
URL/nttps://o33 aaai
[46] Li Li, You Qin, Wei Ji, Yuxiao Zhou, and Roger Zimmermann. Domain-wise invariant learning for panoptic scene graph generation. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3165-3169, 2024. doi: 10.1 10.1009/4 2024. 10447193.
[47] Liunian Harold Li, Pengchuan Zhang. Haotian Zhang. Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai- Chang, and Jianfeng Gao. Grounded language- image pre-training In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10955-10965. IEFF, 2022. doi: 10. 10.11099CVP526888 2022.01069. URL h //áci org/ 10. 100.11119//222258000
2022. 01069
[48] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11, 2023.
[49] Xiao Liu, Da Yin, Yansong Feng. and Dongyan Zhao. Things not written in text: Exploring spatial commonsense from visual signals. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio. editors. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 2365 2376. Association for Computational Linguistics, 2022. doi: 10. 11.1866/11222222AA111 v1/2022. acl long.
168. URL hctth:h//ooooo org/ 18653/
[50] Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA. May 6-9, 2019. OpenReview.net, 2019.
URL htttps:/ net hwwummormmmmmmm
wowum?dddBuss B 6RiCqY7
[51] Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-elip: End-to-end multi- grained contrastive learning for video-text text retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pages 638-647, 2022.
[52] Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning. and Parisa Kordjamshidi. SPARTQA: A textual question answering benchmark for spatial reasoning. In Proceedings of the 2021 Conference of the North American Chupter of the Association for Computational Linguisties: Human Language Technologies. NAACL-HLT 2021, Online, June 6-11, 2021, pages 4582-4598. Association for Computational Linguistics. 2021. doi: 10. paacc-maile 364
MAIAAAL-MANN... 364, URL httott/ssss..o.ccoooppppooooooct http:pppp///oooooo.11111111.3330
[53] Aditya Mogadala, Marimuthu Kalimuthu, and Dietrich Klakow. Trends in integration of vision and language research: A survey of tasks, datasets, and methods. J. Artif. Intell. Res., 71:1183-1317, 2021. doi: 10.1613/jair.1. 116888. URL. httt:p://doi.orgge
161 1613/jair. 11688
[54] Van Khoa Nguyen, Yoann Boget, Frantzeska Lavda, and Alexandros Kalousis. Discrete latent graph generative modeling with diffusion bridges. CoRR, abs/2403 16883, 2024, doi: 10.48550/ARXIV.2403..
16883. URL wttp::ttp//// //coi or1/10....0o0oo........1 48550/ 20150 www.ariwwiv........ 16883
[55] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing 7hu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6- 12, 2002, Philadelphia, PA, USA, pages 311-318. ACL, 2002. doi: 1013115/1073083. 1073135. URL
pwgppo2-.r...00000 10407
[56] Tao Pu. Video scene graph generation with tempooal knowledge. In Proceedings of the 31st ACM International Conference on Multimedia, MM 2023, Ottawa, ON, Canada, 29 October 2023- 3 November 2023, pages 9340 9344. ACM, 2023. doi: 00.1145/358178833613333 URL atips
//doi. org 10. 111..1115/2000 3581 783. 3613433
36333333333333
14

---

[57] Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting Zhuang. and Siliang Tang. Momentor: Advancing video large language model with fine-grained temporal reasoning. arXiv preprint urXiv:2402.11335 2024.
[58] Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua. Layoutlim-12i: Eliciting layout guidance from Ilm for text-to-image generation. In Proceedings of the 31 ACM International Conference on Multimedia, pages 643-654, 2023.
[59] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, al. Language models are unsupervised multitask learners. OpenAI blog, 1(8).9 2019.
[60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR, 2021. URL BRLLhttp:///ttt http::/procceedngg/ .mr
ntml
[61] Navid Rajabi and Jana Kosecka. Towards grounded visual spatial reasoning in multi-modal vision language models. CoRR, 0bs/2208.099788 2023. doi: 10.48500ARXIV.230000077 URL https
//doi.or//10, 188550/arX111 2308. 09778
[62] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Muchine Learning Research, pages 8821 -821- PMLR, 2021. URL http:::/ppoocccddngggge mww.prees.............m..mmmmm..
v139/ 0a333//mmssh2aahaaee
[63] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA. June 18-24, 2022, pages 10674-10685. IEEE, 2022. doi: 10.1109/CVPRS52288..20 200.PPR852228822220044440
BatLhhtps:/// //ooi 1ttp://a.......111.11..1/1.1.1/.
2022
01042
[64] Antoni Rosinol, Andrew Violette, Marcus Abate, Nathan Hughes, Yun Chang, Jingnan Shi, Arjum Gupta, and Luca Carlone. Kimera: From SLAM to spatial perception with 3d dynamic scene graphs Ins. J. Robotics Res., 40112-114:1110-5544 2021. doi: 10.117770278364921 1056674. URL https:///oo
[65] Chitwan Saharia. William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans, Jonathan Ho. David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep lan- guage understanding. In Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NewrlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. http:tttpppppp.....ett www.spperr..iis..oooppppp wttp:ppp.cppppppp.eeppppp files pp2pr//222// .tml
[66]
Tim Salimans, Ian J. Goodfellow, Wojeiech 7aremba, Vicki Cheung, Alec Radford, and Xi Chen. Im- proved techniques for training gans. In Advances Neural Information Processing Systems 29: An- nual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2226-2234, 2016. URL hhttps::/procceeinnngg neurips................... ips. cc/ httppppp.ccpppppppppappppa 2apprr22015
neur html
hash/ 2ttr/9333633B0 2elt3336eeb799222bb2 wwtpaaaeee7222222222226666660600 0at1:333222022000000000000000000
[67]
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. In 5th International Conference on Learning Representations, ICI. 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:/:: httppps:///ppeeeee /wwpnreviiiieeeeeeeeeeooooe.e.ee ew. net/ ForunniddsJrrCCCC. d=3JrFC6ceg)
[68] Sebastian Schuster, Ranjay Krishna, Angel X. Chang, Li Fei-Fei, and Christopher D. Manning. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the Fourth Workshop on Vision and Language, VL@ EMNLP 2015, Lisbon, Portugal, September 18, 2015, pages 70-80. Association for Computational Linguistics, 2015. doi: 11SS3////11-28122 URI. https://dci.on 10.
[69] Jiansong Sha, Haoyu Zhang, Yuchen Pan. Guang Kou, and Xiaodong Yi. Nerf-is: Explicit neural radiance fields in semantic space. In ACM Multimedia Asia 2023, MMAsia 2023, Tainan, Taiwan, December 6-8, 2023, pages 10:1-10:7, ACM, 2023. doi: 111114///599916362222 URL at
//doi. org 10. 1ee.i1145/5/0 110...1111/39991.... 399699966666999999 3626379
15

---

[70] Jascha Sohl-Dicksted Eric A. Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamiis. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015. Lille, Frunce, 6-11 July 2015, volume of JMLR Workshop and mir
Conference Proceedings. pages 2256 22056.22222222222 JogR.org, 2015. URL .html
press/v37/sce
[71] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL.htttps:///ppereeiieee htt.:eeeeeeeeeeeeeeeeerrrrreeeea wwtp:/eef/oooundstttte1
[72] Jared Strader, Nathan Hughes, William Chen, Alberto Speranzon, and Luca Carlone. Indoor and outdoor 3d scene graph generation via language-enabled spatial ontologies. CoRR, wht/2312.117713.. 2023, doi: 00.48550/ARXIV.2112.2
URL htl.httpa:///ob.... , org/ 10. 48550 2312. 11713
[73] Shang- Yu Su, Chao- Chao-Wei Huang, and Yun-Nung Chen. Dual supervised learning for natural language understanding and generation. In Proceedings of the 57th Conference the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2019, Volume 1: Longg Papers, pages 5472- 5477. Association for Computational Linguistics, 2019. doi: 10. (0.18653/V1/P199 1545, URL [ntpps]
[74] Haifeng Sun, Xiaozheng Zheng, Pengfei Ren, Jingyu Wang, Qi Qi, and Jianxin Liao. SMR: spatial-guided model-based regression for 3d hand pose and mesh reconstruction. IEEE Trans. Circuits Syst. Video Technol. (42:)))999331))1) 314, 2024, doi: 10.1009TCSST.2203338881111 1109/TCSVT 2023. 3285153
UtLhttts:: http:://ddoo.o/////
[75] Yiho Sun, Duyu Tang, Nan Duan, Tao Qin, Shujic Liu, Zhao Yan, Ming Zhou, Yuanhua Lv, Wenpeng Yin. Xiaocheng Feng, Bing Qin, and Ting Liu. Joint learning of question answering and question generation. IEEE Trans. Knowl. Data Eng. 32(5):971-982, 2020. doi: 10.1109/TKDE.2019292999 289.297773. 2013.2877773
[76] Duyu Tang. Nan Duan, Tao Qin, and Ming Zhou. Question answering and question generation as dual tasks. CoRR, abs/1706.00022) 2017. URL CRLLhtt::// http//xxxvv.roo.//o wttgg////////..ooo/////// 02027
[77] Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Listening while speaking: Speech chain by deep learning. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2017, Okinawa, Japan December 16-20, 2017, pages 301 -08. IEEE, 2017. doi: 10.1109/ASRU wTT.///SR.......... 8268950.
URL URL[https:: https: /tdps: http://t...ooo..////// 10. 11009/93330 2017 826895
[78] Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In 2020 IEEE/CVF Conference Computer Vision and Pottern Recogni- tion, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 3960-3969. Computer Vision Foundation IEEE, 2020, doi: 10. RR.11099 200.:2000..22.0000002220 004000
http:/tttttpppaaaaeeaaaaaa wtw.cessss..te....eeeee.ee thecvf. com/
content CVPR 202 //ttm WWald Learning 3D Semantic Scene_eeeeeeeeeeeeeeeee Graphs From 3D
Indoor_ Rrrborr_Rccensttuccte ructionstttoooooo CVw._VCR___. 2020 paperapppa...... html
[79] Peng Wang. An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures. tasks, and modalities through a simple sequence-to- sequence lcarning framework. In International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 23318- 23340. PMLR, 2022. btt pttp///oocccccncccccc/e mlr. wrwssrrsssss.... v wang22a1 html
[80] Shanpeng Wang, Zhenbing Qiu, Panpan Huang, Xiang Yu, Jian Yang, and Lei Guo. A bioinspired navigation system for multirotor UAV by integrating polarization compas/magnntonceriis//sgg IEFE Truns. Ind. Electron., 70(8):8526-855 8536, 2023. doi: 10. 1099000..022..2212222....
https:/::
org/10.1 1119/TIEE..... 2022. 3212421
[81] Yang Wang et al. Scene graph generation by iterative message passing. In European Conference on Computer Vision (ECCV), 2018. URL
owt. wttgg/ba///baeeeeo.oeo/eae/eaaea http//bss//111 1111. 05846
[82] Mingrui Wu, Xinyue Cai, Jiayi Ji, Jiale Li, Oucheng Huang, Gen Luo, Hao Fei, Xiaoshuai Sun, and Rongrong Ji. ControlmlIm: Training-free free visual prompt learning for multimodal large language models. arXiv preprint 24Xx:24077.21344 2024.
[83] Shenggiong Wu. Hao Fei, Wei Ji, and Tat-Seng Chua. Cross2stra: Unpaired cross-lingual image captioning with cross-lingual modal structure-pivoee alignment. arXiv preprint XrXiv:2005 /2260, 2023.
[84] Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards semantic equivalence of tokenization multimodal 1mm. arXiv preprint 0tti::200000012221 2024.
16

---

[85] Shengqiong Wu, Hao Fei, Leigang Qu. Wei Ji, and Tat-Seng Chua. gext-gpt: Any-to-any multimodal Ilm. In Proceedings of the International Conference on Machine Learning, 2024.
[86] Shengqiong Wu, Hao Fei, Hanwang Zhang, and Tat-Seny Seng Chua. Imagine that! abstract-to- io-intricat text-to- image synthesis with scene graph hallucination diffusion. Advances in Neural Information Processing Systems, 36, 2024.
[87] Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scenegraphfusion: Incremental 3d scene graph prediction from RGB-D sequences. In IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2021. virtual, June 19-25, 2021. pages 7515 7525. Computer Vision
Foundation / IEEE, 2021. doi: 10. 11w/////VV/44..22210.00.
URL nt httptttt///tcttc.ttt
httttpp://ppeaceessss. Incremental 3D
thecvi , con./ http://ootett/tccm /Ctpteerrtt////ee222 http:///ww222//m////222 ntm1/Wu /Wu SceneGraphPus
Scene_Graph Praapc_nrddccttoo prom RGB-D Se_teeueneees__eeee Seeeuuences_cPR__2020 [Sccpernee_cck_2222_ppaeeee 2a1p221paperrhtr_ html
[88] Yingee Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, and Yan Liu. Dual supervised learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 20/7, Sydney, NSW; Australia, 6-11 August 2017, volume 70 Proceedings of Machine Learning Research, pages 3789-3798. PMLR, 2017. URL htt.:::/pproeeedingg mlr. wtw..pppeess//O0xxaalll httmmm..mmmm...mmmmmmm.mmm..m
[89] Kaiyu Yang, Olga Russakovsky, and Jia Deng. Spatialsense: An adversarially crowdsourced benchmark for spatial relation recognition. In 2019 IEEE/CVF International Conference on Computer Vision ICCV 2019, Scoul, Korea (South), October 27- November 2, 2019, pages 1-600. IEEE, 2019. doi: 10. 1109///CV.2019900021111 bttttt : orro//oor///
/toor/110000 10. 1109/ CCV 201 00214
[90] Xinlei Yang et al. X-lan: Cross-modal scene graph alignment network for image captioning. In IEEF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. URL httphtttps///axxi.ooo
abs abs/ 11006996..111 11097
[91] Hai Ye. Wenjie Li. and Lu Wang. Jointly learning semantic parser and natural language generator via dual information maximization. Proceedings of the 57th Conference the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 2090- 2101. Association for Computational Linguistics, 2019. doi: 10.18653/V1/P199 1201. URL |RL 7/doi. 1tt.oooorg1o...15....... 10t.0.11/111111122222001
[92] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, pages 5831-5841 Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1109/VPR.2018. 00611. URL http:tt:///opeecccceeaatt
thecvf com/ content_cvpr 201 2018/html /7e1 1ers_
Neural Matrral_Motiia www.ooofffSSeeeeeeeeeeeeee Sww.SSceeeeeee_Pee______ 201 watpeoo.ppppppppa______ html
[93] Mcishan Zhang, Huiyao Chen, Xin Zhang. Jing Chen, and Min Zhang. Faking teacher works! dependency scoring learning and corpus boosting for translation- -aasad based cross-lingual dependency parsing. Dependency Scoring Learning and Corpus Boosting for Translation Based Cross-Lingual Dependency Pursing, 2023.
Utt/https:// ///p::///sa0
sarn. //ssnn.
com/ abstract= 4rraac==666665 626681
[94] Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg- Bridging image- object-level, pixel-level reasoning and understanding. arXiv preprint arXiv:2406. 19389, 2024.
[95] Yu Zhao, Jianguo Wei, Zhichao Lin, Yucheng Sun, Meishan Zhang, and Min Zhang. Visual spatial deseription: Controlled spatial-oriented image-to-text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1437-1449. Association for Computational Linguistics, 2022. doi: 18.18653/ V1/2022. EMNLP WEMML--MMANN....... URL /tttpp::/dii.ogg/// 10. 00./36333///00 .ere22..mnigg p--ippp--i in..inn... 93
[96] Yu Zhao, Hao Fei, Yixin Cao, Bobo Li, Meishan Zhang, Jianguo Wei, Min Zhang, and Tat-Seng Chua. Constructing holistic spatio-temm temporal scene graph for video semantic role labeling. In Proceedings of the si ACM International Conference on Multimedia, pages 5281 1-511, 2023.
[97] Yu Zhao, Hao Fei, Wei Ji, Jianguo Wei, Meishan Zhang, Min Zhang, and Tat-Seng Chua. Generating visual spatial description via holistic scene understanding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 941 2023, pages 7960-797777 7977. Association for Computational Linguistics, 2023. doi: 18.18853/V11 442
LONG..44. URL https::///// /oi http:///to...000.11/510
17

---

[98] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu. Jinhui Xu, and Tong Sun. LAFITE: towards language- free training for texi-to-image generation. CoRR,
abs/2111.13799 2021. URL /tR/ttss////rra..or///aaab/ 2tt1:.//11.1111 13792
[99] Zixin Zhu, Yixuan Wei, Jianfeng Wang. Zhe Gan. Zheng Zhang, Le Wang. Gang Hua, Lijuan Wang. Zicheng Liu, and Hu. Exploring discrete diffusion models for image captioning. CoRR.
abs/2211.11694, 2022. doi: 10.48550/ARXI/2221.... 1694. URL https: :tsppp://dee
arxiv.22. 2211 11694
18

---

A Extension of Technical Details
In this section, we introduce the specific details our method which we cannot present in the main article due to the space limit.
A.1 Discrete Diffusion
We first introduce the discrete representation of continuous data. Given data æ in continuous space, vector quantized model (VQM) is employed to encode æ to embedding vectors. VQM contains an encoder E, a decoder G and pre-trained codebook Z = {cu J E IR k xd, where has a finite number of embedding vectors, K the size of the codebook and d is the code dimension. Given input æ. we obtain sequence of tokens Zg with the encoder e = (xx) and a quantizer that maps its closet codebook entry C₁:
2 = Q(e) = argmin ||e
Ck
(14)
sEZ
Correspondingly, given a quantified tokens 2n, the decoder could faithfully reconstruct the data G(,)).
With this discrete representation, the discrete diffusion model can applied to it. Given the quantized data 2o, the forward diffusion process gradually comrupts 20 through the Markov chain g (<)(),-),,. randomly replacing some tokens in 2t-1- After a fixed number of T steps, the model outputs a sequence of increasingly noisy latent variables 21....: 27, where æT is the pure noise tokens. In the reverse process, the model gradually denoise from žy and reconstruct 2n, by sampling from the distribution ((2 12,, 20) sequentially. Specifically, for token 2 of zp, z takes the index of one entry of codebook, i.e., 21 E {1,..., K'. The probabilities of the transition from 2₁-1 to 2; can be represented by matrix (mm) qzt m|2,-1 n) E RK* Then the forward diffusion process for 2, is:
(- = v ( (z))Qu((Z()).
(15)
where v(2) means the one-hot representation of in K categories. Q.u(z,1) means the categorical distribution over zt. Due to its Markov property, the (Zz could be written as:
[1 v(20)
g(2z|20) = = (z+)
(16)
=v Q(=))(((
By applying Bayes' rule, we can compute the posterior qf [1 - 12: 20) as:
q( 2 g 20 20) ( bt- (20)
g(2,- Fz, z0)
g( 20)
(17)
nu v (2x ( (()((z))
(u (21-1 Q Qu-1)(z)))
Qe is usually defined as the small amount of uniform noises and it can formulated as:
ax + 3g
Br
3
3r
+ 3
3
Q₂
(18)
3
Br
at + 3,
where a, E [0, 1] and 8, = (1 - aK)K,, which means each token has a probability of a; + 3r to remain the previous value one step and has a probability of K B, to be sampled uniformly over all the K categories.
Then noise estimating network p0(21-1/21 c) is trained approximate the conditional transit distribution (2,-1 [22, 2o) with condition c. The network trained to minimize the variational lower bound (VLB) (70]
L = DDI(g((z-
( c))
(19)
19

---

A.2 Discrete Representation for 3DSG
Following [54, given the 3DSG G, we use GNN encoder encode
of node embeddings
Z = _ We subsequently apply a quantization operator. F, on the continuous node embeddings and map them fixed points within the same space. The quantization operates independently on each dimension of the latent space, and the quantization of the j&h dimension of 2; embedding is given by:
Ti F(EJ, L2) :R( tanh 25y). 1 S dz
(20)
where R. is the rounding operator and Ly is the number of quantization levels used for the jth dimension of the node embeddings. The quantisation operator maps any point in the original continuous latent space point from the set Z Z₁. The discrete latent representation of graph g is the set - } = Z.
The quantization operator is permutation equivariant and is the mapping from the initial graph G to its discrete representation in the latent space as long as the graph encoder D is permutation equivariant. Thus for any P permutation matrix the following equivariance relation holds:
P Z = Z) = F(D(P" EP,P" X))
(21)
Using equivariant decoder D(ZC), which will operate on the assumption of a fully-connected graph over the node embeddings n results in reconstructed graphs that are node permutation equivariant with the original input graphs.
A.3 Aligned Image-Text-3DSC Data Construction.
We follow [72] to take the 3D datasets Matterport3D (MP3D) 7 3DSSG 78 and CURB-SG [27]. using the Hydra [36] parser to generate the 3D scene graphs and then leverage large language models to refine it as the ground-truth. Then we use ChatGPT to generate alignede spatial descriptions for the RGB images.
A.4 Marginal Distribution Estimation for the Dual Learning Target
Based on Eq. 9 the marginal distributions p(x) and p(y) can be aqeired directly. Thus we estimate these marginal distribution p(z) and p(y) with a surrogate distribution p(x) and p(y), by
observing the target in the scope of the whole data. For the text, we use a Transformer-based language model that is trained over the specific data to calculate the p(x) [73.. For the image, we follow [16] calculate p(y) I p We serialize the image pixels as x; and use PixcICNN++ 67 model this distribution.
B Detailed Experiment Settings
B.1 Evaluation Metric Implication
We employ Fréchet Inception Distance (FID) 666, Inception score (IS) [31, CLIP score 30 and GLIP [47 used in [15 to quantitatively evaluate the quality the generated images. Additionally, we introduce Triplet Recall (TriRec.) to measure the percentage of the correct relation triplet among all the relations. Technically, given a set ground truth triplets GT, and the TriRec. is computed TriRec.=|PTT n GTI//GT), where PT are the relation triplets extracted from the generated images by a visual or textual SG parser.
denoted
B.2 Human Evaluation Criterion
We conduct a human evaluation to mainly focus on spatial understanding quality.
Then design 5-point Likert scale is designed follows:
S12T Spatial Accuracy: The sentences correctly describe the spatial relationship of the objects in the given image.
SI2T Spatial Diversity: The generated sentences describe diversified spatial semantics.
20

---

ST2I Spatial Accuracy; The image present the spatial description correctly.
Each question will be answered by number from I to 5, denoting "Strongly Disagree". "Disagree",
"Neither disagree nor agree". "Agree" and "Strongly agree". We generated sentences for SI2T and 3 images for ST2I from 100 samples.
C Extended Experiment Results and Analyses
Table 5: Comparison with latest T2I methods.
ST2I
FIDU
CLIP
SD-1.5
20.12
26.35
64.79
SDXL
12.60
27.95
66.41
Vog Ours
13.28
28.63
67.29
11.04
29.20
68.31
C.1 Comparison with More Methods
add more experiments of the latest T2I methods in table Comparing with the recent SD and Cog View, our method exhibit leading performance with the similar scale.
C.2 The Superiority of the Discrete Modeling
To verify the superiority of the discrete modeling, we compare with the continuous diffusion backbone in table We find that the results of continuous backbone drops slightly on ST2I and SI2T while drops more on 3DSG generation, revealing that the discrete model show superiority for structure data modeling and further benefit the final performance.
Table 6: Comparison between discrete diffusion and continuous diffusion.
3DSG
ST2I
SI2T
TriRec.
FID.
1S
CLIP
BLEUU4
SPICE!
w/ Descrete Diff (SD*)
84.51
10.09
29.76
71.10
27.63
48,03
w/ Continuous 83.14
10.25
29.68
70.02
27.51
47.82
C.3
Can Non-diffusion SI2T models be used in our Dual Learning Framework?
We use the diffusion based SI2T model to maintain the symmetry for diffusion based ST2I and 3DSG generation, so that their intermediate feature could be mutually enhanced. Here we explore whether vision-language model (VLM) based SI2T model could be integrated in our dual framwework. We follow 97 to use a OFA model as SI2T backbone. We replace the z in P (2
Graph
@ xxx)
to the OFA encoder output hidden states z Then we train our model with the same training data and training strategy. The results are shown in Table where we find the ST2I performance drops while the SI2T performance keep comparable. This reveal the necessity of choosing diffusion based SI2T.
Table 7: Comparison between diffusion and non non-diffusion SI2T on VSDv2.
ST2I
SI2T
FID
IST
CLIP
BLEU41
SPICE*
SIET (
10.09
29.76
71.10
27.63
48.03
OFA-SI2T
11.20
69.62
27.40
47.91
21

---

Table Efficiency analysis for each step.
Training Parameters
Training Time
Step-1 DGAE
110M
hours
Step-2 Spatial Aligmment
127M
1.5 hours
Step-3 2DSG +3DSG Diflusion Training
350M
hours
Step-4 Overall Training
1.IB
20 hours
C.4 How the quality of 3DSG dataset influences the performance
Instinctive, the quality of 3DSG training data has strong impact to the graph diffusion. To verify this, corrupt the gold 3DSG by random replace the node and edeges with probability and compare the model Figure 8 We can see that with the noise increase, both the 3DSG and the final performance decrease sharply. This means the graph diffusion model is sensitive to the 3DSG data quality.
S12T
ST2I
3DSG
100
80
No Noive
60
Noisy
0
0
20
40
60
80
Noise Rate (%)
Figure 8: The evaluation of structure matching on the gold 3DSSG dataset with noise.
C.5 Visualization of 3DSG Generation Process
we visualize the 3DSG generation process with two examples, where we sample several time steps and plot the generated 3DSG be seen, compared with the initial VSG/TSG our graph diffusion is capable of enrich certain reasonable structures.
In Figure g
C.6 Efficiency Analysis
In table 8 we give the computational complexity analysis as well our training settings.
VSG-to-3DDSG Diffiusion
livingnom
Lastep
VSG
lest
Hippog)
Ippop
Iean
Foof
Liston
(iura
man
loptop.
Bestog
livinanocm
livingnoom
Invet
Sortl
Inork
nest
lake
7SG-to-33DSG Diffusion
Figure 9; Visualization of 3DSG generation process from initial VSG and TSG.
22

---

The man in the yellow striped hoodie holds a colorful umbrella over his head.
Maroon curtain panel in front of window.
We can see a big bridge above the water in the distance.
Figure 10: More cases of ST2I generated by SD
C.7
More Cases
Here we showcase more examples of the ST2I and SIT@ via SD3 in Figure 10 and Figure 11
23

---

There are some fruits in the bowl on the table.
Some grapes are put in white bowl.
A bowl of fruit with lemon and banana in it.
The red chairs are put in front of the wooden table.
Rows of chairs and tables in the room.
A red chair with a counter top in the front.
Two guys stand on the rock with bikes nearby.
Two guys are riding bicycles the mountain.
Two guys handing the bikes standing next to each other.
Figure 11: More cases of SI2T generated by SD
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have highlighted the contributions in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yess Yes]]
Justification:
We discuss limitaitons in the Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

We provide main proof in the Method and full set of assumptions in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide the detailed implementation settings.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
We will open source at Github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips... cc/
publ ic quides Codeeubmme ssion iisionoolics icy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. cc/ wwwblic/guideen / //odiddee/Cdeeeeeeee Cooeeeooeeeeeeeeee SudeSuumiiee wwwmisssiiii wsssnooliccyyyyooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the detailed implementation settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our results are significantly comparable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
27

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We detail in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The experiments are conducted with the Code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
28

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssw hopersswiccoe com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
29

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "JaazbdFGb8", "reviewer": "Reviewer_Urkw", "summary": "The paper introduces a dual learning framework and a 3D scene graph representation for enhancing spatial image-to-text and text-to-image tasks in visual-spatial understanding. The proposed Spatial Dual Discrete Diffusion (SD3) system outperforms existing methods on the VSD dataset, demonstrating the effectiveness of the dual learning strategy in improving spatial feature modeling and task performance.", "questions": "See the weaknesses.", "limitations": "The authors have adequately discussed the limitations and potential negative societal impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors are commended for providing the code, which facilitates further investigation and reproducibility of the research findings.\n2. This manuscript is the first to achieve synergy between two spatial-aware cross-modal dual generations.\n3. The manuscript provides a comprehensive summary of the differences between the 2DSG and the proposed 3DSG.", "weaknesses": "1. Could the authors consider prioritizing the most relevant or impactful references and possibly discuss the contribution of each cited work in more detail? \n2. The introduction would benefit from a concise summary of the manuscript's contributions at its conclusion.\n3. Could the authors briefly discuss how the treatment of spatial features in existing methods compares to the approach taken in this paper, and highlight the distinct contributions of this work?\n4. The complexity of the SD3 framework may impact training and inference efficiency. The authors should provide a computational complexity analysis to assess its practicality.\n5. Please provide complete results for Table 2 and consider additional datasets to validate the findings.\n6. The current reliance on human evaluation for spatial assessment could be complemented with automated evaluation techniques to provide a more comprehensive assessment approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and the careful review for our paper. Your suggestions will definitely help improve our paper. We try to address your questions point by point as follows.\n***\n**Q1:** Could the authors consider prioritizing the most relevant or impactful references and possibly discuss the contribution of each cited work in more detail?\n\n**A1:** Thanks for advising. We will consider reorganize the content of the related work. For the most relevant ones, e.g., discrete SD models and PLMs, we will discuss the contribution of them detailedly. However, due to space limitation, we may leave others via a summary.\n***\n**Q2:** The introduction would benefit from a concise summary of the manuscript's contributions at its conclusion.\n\n**A2:** We will consider give a summary of contributions in the last paragraph in the introduction in the revised version.\n***\n**Q3:** Could the authors briefly discuss how the treatment of spatial features in existing methods compares to the approach taken in this paper, and highlight the distinct contributions of this work?\n\n**A3:** Thanks, the critical distinct contribution of our work is that we use the 3D scene graph to model spatial features, which is a novel way in both I2T and T2I generation. Existing methods takes a general visual embedding or the 2D scene graph to model appearance feature and the spatial features, while our 3D scene graph could capture the 3D spatial relationships via the proposed graph generation pretraining. We will highlight the definition of 3D scene graph and the contributions in the method section.\n***\n**A4:** The complexity of the SD3 framework may impact training and inference efficiency. The authors should provide a computational complexity analysis to assess its practicality.\n\n**Q4:** Thanks. As other reviewers have mentioned the same issue, we will give the computational complexity analysis as well as our training settings in the revised version.\n|          | Training Params/Training time  |\n|  ----  | ----  |\n| Step-1 DGAE  | 110M/1 hours |\n| Step-2 Spatial Alignment  | 127M/1.5 hours |\n| Step-3 2DSG→3DSG Diffusion Training | 350M/ 12 hours |\n| Step-4 Overall Training | 1.1B/ 20 hours |\nAll the training\n***\n**Q5:** Please provide complete results for Table 2 and consider additional datasets to validate the findings.\n\n**A5:** Table 2 has listed the full results on the VSD dataset. We will provide the additional results on other datasets. For larger dataset, the training may cost time, and we are sorry we can not provide the results immediately. We will provide in the final version or the arxiv version.\n***\n**Q6:** The current reliance on human evaluation for spatial assessment could be complemented with automated evaluation techniques to provide a more comprehensive assessment approach.\n\n**A6:** For SI2T, the SPICE could reflect the spatial assessment in to some extent. For ST2I, it is hard to evaluate the spatial assessment directly from the final image with automatic metrics, thus we analyze and evaluate the intermediate spatial scene graph in section 5.4 (Table 4,5 and Figure 5).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kbZ9srRc0s", "reviewer": "Reviewer_WpM7", "summary": "This paper presented a novel dual learning framework for spatial image-to-text and spatial text-to-image generation. The main model is a combination of three discrete diffusion models, where an intermediate 3D presentation is first generated and then the image and text outputs are generated based on the 3D feature. The proposed method creatively divides the SI2T and ST2I to two pairs of dual stages, i.e., the Text-3D with 3D-Text and Image-3D and 3D-Image, and then takes a dual training strategy to enhance the hard X-3D processes with the easy 3D-X processes. The experimental results show that the proposed methods outperform current I2T and T2I models significantly.", "questions": "Please see my weakness comments.", "limitations": "Limitations have been discussed and I do not foresee any other negative impact from this work.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.\tThe novel and effective dual learning solution for the paired ST2I and SI2T tasks.\n2.\tThe proposed methods provide an interesting perspective for the similar dual tasks, which is of value to the community.\n3.\tThe paper is well-organized and easy to read. \nOverall, the idea is novel and the presentation is good. I tend to believe the quality of the paper meets the standard of the conference if the author could address the concerns I raised in the weaknesses.", "weaknesses": "Some confusion about the technical details:\n1.\tHow many node categories does the 3DSG has and how to define the high-level spatial concepts? Does it follow the definition of previous works? Lack of related discussion and references.\n\n2.\tDo the VSG/TSG and 3DSG have the compatible node sets? Do you map the nodes among the three types of SG with a rule?\n\n3.\tAbout the codebooks of the discrete modeling. How do the codebooks (graph, image, and text) be initialized and updated during the whole training process?\n\nThe training of diffusion model is time consuming. I am a little worried about the efficiency problem. Can the author provide the efficiency analysis for each training stage?\n\nThe author takes GPT2 as the text decoder. Could it be replaced by other PLMs and how does it influence the performance? \n\nMinor Issues:\n\n1.\tTwo small font size in Figure 2\n\n2.\tLine 248, the subscript “T23D” and “I23D” are ambiguous.\n\n3.\tBars overlap in Figure 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for going through our paper so deep and carefully. We appreciate that you acknowledge the novelty of our proposed task and the comprehensive experiments. All your possible concerns are addressed as follows. \n***\n**Q1:** How many node categories does the 3DSG has and how to define the high-level spatial concepts? Does it follow the definition of previous works? Lack of related discussion and references.\n\n**A1:** In our dataset it could be more than 200 object tags, which could be the mode categories of the 3DSG. However, limited by the VSG/TSG generator, the input node category number is about 150. Actually, in practical application, the node tags of 3DSG could be open-ended because the graph diffusion model convert the graph only in the latent dense representations. But in order to facilitate evaluation, we use the predefined 150 categories. \n\nAbout the high-level spatial concepts, they are usually the location, places (such as room, living room) of a normal object (such as table, chair). Their characteristic is that they have a hierarchical relationship with general nodes, which are marked via the special edges. In our datasets, there are only two-level spatial concepts, and we have manually divided them.\n\nWe will add these discussions to the third section.\n\n***\n**Q2:** Do the VSG/TSG and 3DSG have the compatible node sets? Do you map the nodes among the three types of SG with a rule?\n\n**A2:** As the response for Q1, the node tags are limited to 150. The tags between VSG and TSG are compatible. For the 3DSG, the graph diffusion may generate unseen tags but we do not need to worry. The following image generator and text generator only take dense graph representations. The only problem occurs in the evaluation if we want to decode the graph. In this case, we handle it through manually established mapping rules.\n***\n\n**Q3:** About the codebooks of the discrete modeling. How do the codebooks (graph, image, and text) be initialized and updated during the whole training process?\n\n**A3:** The graph codebook is randomly initialized in the training of DGAE (Step-1 in Fig. 3). The image and text codebooks are initialized with the off-the-shelf models and will be frozen all the time. Of course, we can also enhance them with a self-supervised image/text generation between the DGAE pre-training and the spatial alignment (Step-1 and Step-2 in Fig. 3).\n***\n\n**Q4:** The training of diffusion model is time consuming. I am a little worried about the efficiency problem. Can the author provide the efficiency analysis for each training stage?\n\n**A4:** The finetuning focus on only part of the parameters. We give the computational complexity analysis as well as our training settings:\n|          | Training Params/Training time  |\n|  ----  | ----  |\n| Step-1 DGAE  | 110M/1 hours |\n| Step-2 Spatial Alignment  | 127M/1.5 hours |\n| Step-3 2DSG→3DSG Diffusion Training | 350M/ 12 hours |\n| Step-4 Overall Training | 1.1B/ 20 hours |\n\n***\n\n**Q5:** The author takes GPT2 as the text decoder. Could it be replaced by other PLMs and how does it influence the performance?\n\n**A5:** Yes, it could be replaced by other PLMs. We take T5-decoder for example.\nWith the same training strategy, the results are not much different:\n|          | BLEU4/SPICE  |\n|  ----  | ----  |\n| GPT2  | 27.63/48.03 |\n| T5-Decoder  | 27.58/48.10 |\n\nWe did not consider larger models, i.e., the LLMs, which may improve the final ST2I performance but not the core textual diffusion performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bltP22J3Tv", "reviewer": "Reviewer_cK3d", "summary": "This paper presented a novel model for the SI2T and ST2I tasks. The proposed model combines the two dual tasks and let them mutually learn via intermediate feature sharing. Through this framework, both SI2T and ST2I are enhanced. The author also provide analysis that how this method works.", "questions": "Please refer the weakness.", "limitations": "No Limitations", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\tThe paper proposed a novel dual learning idea and the implemented model is reasonable.\n\n2.\tGenerally comprehensive analysis on the designed model.", "weaknesses": "There is no major issue in my view. Some questions and advices are listed following: \n\nAbout the motivation: The author claims that the 3D feature is important for spatial understanding, and I agree with that. I wonder if 3D modeling is necessary for these two tasks and what special information could the 3D modeling provide. In this framework, it seems the model acquires the capability of 3D understanding via the pretraining of the 3DSG generator. Is the scene graph a reasonable way to modeling 3D features, and how does it match your task? The author should discuss the above points.\n\nThe DGAE is pretrained by the gold 3DSG dataset. Intuitively, the quality of this 3DSG dataset significantly affects the DGAE results and further the final results. Figure 7 analyzes this problem by an ablation study on the manual noise data. But I still have a concern that the performance of DGAE may be the bottleneck of the whole model. \n\nAbout the discrete modeling: To my knowledge, for the respective I2T or T2I task, the continuous diffusion models could achieve better performance, while the Table 6 presents the better final performance of the proposed dual model. Ignoring the efficiency problems, what superiority does discrete modeling has? \n\nThis work provides a novel way to solve dual tasks. From your perspective, what kind of tasks can be solved with this dual learning method? \n\nTypos:\nLine 767 “alignede” should be “aligned”\n\nLine 810 “eh” should be “the”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for you for your time and provide us with rich and constructive feedback on our paper. And we believe it will surely improve our work. Following we extract your concerns into points and try to address them one by one.\n***\n\n**Q1:** I wonder if 3D modeling is necessary for these two tasks and what special information could the 3D modeling provide. In this framework, it seems the model acquires the capability of 3D understanding via the pretraining of the 3DSG generator. Is the scene graph a reasonable way to modeling 3D features, and how does it match your task? The author should discuss the above points.\n\n**A1:** 3D modeling is very critical for spatial understanding.  The layout overlap and perspective illusion are most common problem. With 3D modeling and pretraining, the model will get the prior knowledges to “imagine” the right spatial relationships between a pair of common objects just like human beings (for example, the books should “on” the shelf, the food should “in” the bowl).\nWe use 3DSG to model the 3D features for two reasons. First, the graph structure could capture the fine-grained object-level relationships, containing more informative features.\nSecond, the 3DSG specifically marks the high-level concepts (room, etc.), which usually be presented as the easily-confused overlapped object in the 2D image. \n***\n\n**Q2:** The DGAE is pretrained by the gold 3DSG dataset. Intuitively, the quality of this 3DSG dataset significantly affects the DGAE results and further the final results. Figure 7 analyzes this problem by an ablation study on the manual noise data. But I still have a concern that the performance of DGAE may be the bottleneck of the whole model.\n\n**A2:** We agree. The bottleneck seems to be the DGAE and it is also the key module. As our responses to reviewer Nx71, the graph module may generate failed 3DSGs and then the final results come incorrect.\nBut we have the viable solution. The straightforward solution is to upgrade this module by scaling up it and training with better data. This effort makes sense and this graph module could be reused in many other tasks.\n\n***\n\n\n\n**Q3:** About the discrete modeling: To my knowledge, for the respective I2T or T2I task, the continuous diffusion models could achieve better performance, while the Table 6 presents the better final performance of the proposed dual model. Ignoring the efficiency problems, what superiority does discrete modeling has?\n\n**A3:** We think the reason coms from two aspects. First, the VSD datasets mainly contains the spatial descriptions while the continuous diffusion models are skilled at handling general descriptions. Second, our model gives stronger guidance to the diffusion process, thus leading more similarity to the reference results. The outputs of continuous diffusion models are more random.\n\n---\n\n**Q4:** This work provides a novel way to solve dual tasks. From your perspective, what kind of tasks can be solved with this dual learning method?\n\n**A4:** The dual learning was first proposed in the machine translation task. Now it has been applied in many areas. The effectiveness of dual learning primarily stems from the asymmetric information transfer between the two dual tasks, where learning in a single direction tends to miss some key features. In such cases, bidirectional dual learning enables both tasks to acquire certain necessary prior information, thereby enhancing performance. Overall, the dual system essentially forms a self-supervised system.\nFrom our perspective, all the dual tasks involving the conversion between two modalities with asymmetric information could find a similar solution.\nWe provide some here: ASR and TTS, I2T and T2I, dual QA, etc.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F3HCQAhQcs", "reviewer": "Reviewer_Nx71", "summary": "Given that previous works for standalone SI2T (Scene-to-Image-to-Text) or ST2I (Scene-to-Text-to-Image) perform imperfectly in spatial understanding due to the difficulty of 3D spatial feature modeling, this paper proposes tp model SI2T and ST2I together under a dual learning framework. Within this dual framework, a novel 3D scene graph (3DSG) representation is introduced, to capture 3D spatial scene features. Moreover,  a Spatial Dual Discrete Diffusion framework is proposed to utilize the intermediate features of the 3D→X processes to guide the hard X→3D processes. Extensive experiments show the proposed method outperforms the mainstream T2I and I2T methods significantly.", "questions": "Please refer tot the \"weakness\" section.", "limitations": "Please refer tot the \"weakness\" section.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The proposed dual learning framework with 3D scene graph (3DSG) representation to enhance the 3D spatial feature modeling is novel.\n\n+  The proposed Spatial Dual Discrete Diffusion framework is simple but effective, which can utilize the intermediate features of the 3D→X processes to guide the hard X→3D processes\n\n+ Extensive experiments on VSD dataset have valid the effectiveness of this method. Compared with previous works (e.g., DALLE, CogView), the proposed works show more competitive performances.", "weaknesses": "- Although the proposed method in this paper has achieved good results, the methods compared in Table 2 are somewhat outdated and not the latest SOTA methods (e.g., DALLE-3, CogView-2). Could the authors compare with some of the latest text-to-image methods, such as the Stable Diffusion series, SD-1.5[1], SDXL[2], etc.?\n\n- The authors should present some failure cases to analyze the shortcomings of the proposed method. For example, can the 3D scene graph always generate perfect outputs? If there are issues in the generated outputs, what impact will it have on the final results? What are the subsequent strategies to address these issues?\n\n\n[1] https://huggingface.co/runwayml/stable-diffusion-v1-5\n[2] Dustin Podell, Zion English and etc. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, in arxiv, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful that you acknowledge the strengths of our work. We've conducted additional experiments and tried every effort to address your concerns. We hope you'll reconsider the evaluation if you find our responses are effective. Following are our outputs.\n***\n\n**Q1:** Although the proposed method in this paper has achieved good results, the methods compared in Table 2 are somewhat outdated and not the latest SOTA methods (e.g., DALLE-3, CogView-2). Could the authors compare with some of the latest text-to-image methods, such as the Stable Diffusion series, SD-1.5, SDXL, etc.?\n\n**A1:** Thanks, we agree and we provide the following results. We compare the continuous SD-1.5 and SDXL on VSD dataset as follows:\n\n|          | FID | IS | CLIP |\n|  ----  | ----  | ----  | ----  |\n| SD-1.5|\t20.12|\t26.35\t|64.79|\n|SDXL\t|12.60|\t27.95|\t66.41|\n|CogView 3\t|13.28|\t28.63|\t67.29|\n|Ours\t|11.04|\t29.20|\t68.31|\n|Ours-XL\t|TBC|\tTBC|\tTBC|\n\nPlease note that both Cogview 3 and SDXL have undergone a significant upgrade in model scale (the Unet backbone), while we did not find the released discrete ones that match this scale of parameters. Also, we are sorry but it is hard for us to train an XL model in the limited time from scratch. Thus, the margins in results in this table do not fully reflect the superiority of our method. \nThe results in our manuscript are sufficient to support our main claim, i.e., the dual learning framework and the 3D scene graph modeling indeed improve the SI2T and ST2I.\nOf course, for the sake of rigor, we will scale up the model and add the new results if we could finish it before the camera-ready deadline. \n***\n\n**Q2:** The authors should present some failure cases to analyze the shortcomings of the proposed method. For example, can the 3D scene graph always generate perfect outputs? If there are issues in the generated outputs, what impact will it have on the final results? What are the subsequent strategies to address these issues?\n\n**A2:** Thanks. We carefully checked the outputs and found that we may have two specific cases that need to be discussed. 1) First, for some simple scenes, the 3DSG may degenerate into the VSG/TSG and lose some surrounding objects and some attribute nodes. In this case, the model will still output the correct final results most of the time, because the main spatial information in the 3DSG is correct. 2) Second, the graph diffuser fails to generate the right 3DSG, leading to inacceptable final results.\nWe believe the main reason for both 1) and 2) is that the graph generation module is not solid enough. One possible solution for this issue is to scale up the graph generator and train it with a better dataset. \nWe will add this fail cases and these discussions in the experiment section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response to All Reviewers

***

Dear Reviewers,

Thanks for all of your insightful and cheerful comments on our manuscript. Your feedback will greatly assist us in enhancing the quality of our paper, and we are committed to incorporating your suggestion in our revision process. 
Meanwhile, we feel encouraged so much that reviewers find our work “novel and effective dual learning” (**Rv#Urkw**, **Rv#WpM7**), “simple but effective” (**Rv#Nx71**), appreciate our 3DSG modeling (**Rv#Nx71**), “comprehensive analysis” (**Rv#cK3d**), and other aspects such as “well-organized and easy to read” (**Rv#WpM7**)


In response to reviewers' comments, we have thoroughly reviewed our paper, performed additional necessary experiments, and prepared a comprehensive response. We will fix all the typos and improve the manuscript according to reviewers' comments.
We hope that our response adequately addresses your concerns and provides further clarification on the contributions presented in our manuscript. We kindly request a re-evaluation of our work based on the updated information, and look forward to your recognition.

Thanks and regards,  
Authors of Submission#1614

Meta-review: The reviewers consistently praise the paper for its novel dual learning framework that effectively enhances spatial feature modeling through the introduction of a 3D scene graph (3DSG) representation. The proposed Spatial Dual Discrete Diffusion (SD3) system is recognized for outperforming existing methods, demonstrating the effectiveness of the approach. Meanwhile, concerns are raised about the complexity and efficiency of the framework, the lack of comparison with the latest state-of-the-art methods, and the need for further clarification on some technical details.

Based on the strengths of the paper, particularly its innovative approach and demonstrated effectiveness, and despite the identified weaknesses, the overall contribution to the field is significant. The paper should be accepted, with the expectation that the authors address the concerns regarding efficiency, comparison with recent methods, and provide clearer technical explanations in the final revision.
Decision: Accept (poster)