Generating Visual Spatial Description via Holistic 3D Scene Understanding
Year: 2023

Abstract:
Visual spatial description (VSD) aims to generate texts that describe the spatial relations of the given objects within images. Existing VSD work merely models the 2D geometrical vision features, thus inevitably falling prey to the problem of skewed spatial understanding of target objects. In this work, we investigate the incorporation of 3D scene features for VSD. With an external 3D scene extractor, we obtain the 3D objects and scene features for input images, based on which we construct a target object-centered 3D spatial scene graph (GO3DSG), such that we model the spatial semantics of target objects within the holistic 3D scenes. Besides, we propose a scene subgraph selecting mechanism, sampling topologically-diverse subgraphs from GO3D-SG, where the diverse local structure features are navigated to yield spatially-diversified text generation. Experimental results on two VSD datasets demonstrate that our framework outperforms the baselines significantly, especially improving on the cases with complex visual spatial relations. Meanwhile, our method can produce more spatially-diversified generation. Code is available at https://github.com/zhaoyucs/VSD.

heading: 1 Introduction
text: Visual spatial description is a newly emerged vision-language task, which aims to generate a textual descriptive sentence of the spatial relationship between two target visual objects in a given image (Zhao et al., 2022). VSD falls into the category of image-to-text generation, while in particular focusing on the visual spatial semantics understanding, which has great values on the real-world humancomputer interaction (HCI) applications (Heuser et al., 2020), e.g., automatic navigation (Pendão and Moreira, 2021; Wang et al., 2023b), personal assistance (Vanhooydonck et al., 2010), and unmanned manipulation (Castaman et al., 2021; Wang et al., 2023a, 2022a).
∗Corresponding author: Hao Fei
Zhao et al. (2022) pioneer the VSD task by manually annotating the spatial descriptions to the images based on the visual spatial classification datasets (Krishna et al., 2017). Also they solve VSD as a general image-to-text (I2T) task via vision-language pre-trained models (VL-PTMs), i.e., inputting images and outputting texts. However, modeling VSD as a regular I2T job with open-ended VL-PTMs can be problematic. Unlike the existing I2T tasks, such as image captioning (Vinyals et al., 2015), verb-specific semantic roles (VSR) guided captioning (Chen et al., 2021) and visual question answering (VQA) (Antol et al., 2015) that focus on the content semantics understanding, VSD emphasizes more on the spatial semantics reasoning, according to its definition. Thus, directly adapting VSD with general-purpose VL-PTMs will lead to inferior task performances. We note that there are at least two observations that should be taken into account for VSD enhancement.
From the image encoding aspect, it is critical to model the holistic 3D scene semantics of the input image. In Zhao et al. (2022), their VL-PTM-based methods model the visual geometrical features at merely 2D flat space (e.g., superficial features). Yet directly perceiving objects from the first-person perspective will inevitably result in skewed angle of view and biased spatial understanding, and thus fail
7960
to handle complex cases (e.g. layout overlap, perspective illusion) or generate incorrect descriptions. For example as in Figure 1(a), with the 2D-level visual understanding, the spatial relation between two cars is wrongly decided. As a reference, we human always first project the visual contents into the 3D space and reckon the scene layout and object attributes (e.g., depth, shapes, camera poses), and then narrate the spatial relations based on such holistic 3D clues.
From the text decoding aspect, it is necessary yet challenging to generate diverse sentences of the object pair relation. In generic I2T task, prior methods strengthen the text diversification by equipping with beam search (Vijayakumar et al., 2018) or integrating external knowledge (Yu et al., 2022). Different from the generic diversified generation, VSD requires the diversification with respect to the spatial descriptions, rather than the diverse linguistics. We can again place the emphasis on the modeling of holistic 3D scene features. For example, with a precise understanding of the spatial relations, it is both viable to generate ‘A is on the left of B’ or ‘B is on the right of A’. Also, as illustrated in Figure 1(b), by comprehensively modeling the surrounding relations of the neighbor objects connecting to the target objects in the holistic scene, more spatially-diverse texts can be yielded via different path traversing.
In this paper, we propose enhancing VSD by modeling the holistic 3D scene semantics. We build an encoder-decoder VSD framework (cf. Figure 2), where the encoder learns the 3D spatial features, and the decoder generates spatially-diversified de-
scriptions based on the spatial semantic features. Specifically, at encoding side, we first employ an off-the-shelf 3D scene extractor (Nie et al., 2020) to produce 3D objects and the corresponding scene features (i.e., layout, location, size and visual features) for the input monocular RGB image, via which we build a target object-centered 3D spatial scene graph (namely, GO3D-S2G). We then present an object-centered graph convolutional network (OCGCN) to encode the GO3D-S2G. At decoding side, we devise a scene subgraph selecting (S3) mechanism to sample topologically-diverse object-neighboring subgraphs from GO3D-S2G, which allows to generate descriptions focusing on the near surroundings of target object. Based on the sampled subgraphs, we then create prompt texts to ground the focused objects and their prototype directions. Finally, a backbone VL-PTM is used to encode the prompt texts, input images as well as 3D scene features, then to produce VSD texts.
We experiment on two versions of VSD datasets (Zhao et al., 2022), where one is with simple annotations and one has more complex and humanfriendly descriptions. The results indicate that our system outperforms the best baseline with significant margins, where our method especially improves on the complex cases, such as layoutoverlapped and irregularly-posed objects. We further reveal how the 3D scene graph modeling as well as the S3 mechanism facilitate the task, and also quantify the influence of the external 3D scene extractor. All in all, this work contributes by verifying that modeling the 3D scene of 2D image helps the understanding of visual spatial semantics.

heading: 2 Methodology
text: Problem Definition Given an image I with two object proposals <O1, O2> in I , VSD generates a sequence of words S = {w1, ..., wn} that describes the spatial relationship between O1 and O2. The input O1 and O2 contain the object tags and their 2D location coordinates. Different from image captioning, the generated sentences of VSD must directly or indirectly express the spatial relation between the target objects.
Overall Framework As shown in Figure 2, our framework (namely 3DVSD) is built upon an encoder-decoder paradigm, where the encoder is responsible for the 3D scene feature modeling, and the decoder generates spatially-diversified descriptions based on the spatial semantic features learned from encoder.

heading: 2.1 Encoder: 3D Scene Feature Modeling
text: We first extract 3D scene features via an external extractor. Then we build a target object-centered 3D spatial scene graph (GO3D-S2G), which is encoded and propagated with an object-centered GCN (OCGCN).
Extracting 3D Scene Features We adopt the 3D scene extractor as in Nie et al. (2020), which is a joint layout estimator and 3D object detector. It first processes the 2D object detection for the input RGB image, based on which the 3D relative coordinates (location) and pose parameters of all the objects will be estimated. Formally, we set up the world system located at the camera center with its vertical axis perpendicular to the floor, and its forward axis toward the camera, such that the camera pose R(β, γ) can be decided by the pitch and roll angles (β, γ). In the world system, an object Oi can be determined by a 3D center loci ∈ R3, spatial size sizei ∈ R3, orientation angle orii ∈ [−π, π)3. Finally, we obtain the loci, sizei, orii and the region-of-interest (RoI) visi (with its representation rvisi ) of each 3D object, which is summarized in Table 1. We extend the 3D scene generating details at Appendix A.1.
Constructing Target Object-centered 3D Spatial Scene Graph Based on the 3D objects and the corresponding 3D scene features, we now construct the GO3D-S2G. The graph is centered on the two target objects, placing the focus on the spatial relationships between the target objects and their surrounding neighbor objects in the scene. Technically, we denote GO3D-S2G as G =(E, V ), where V is the set of 3D nodes vi (i.e., 3D objects). Note that as the input images are likely to contain some noisy objects that are less-informative to the task, we remove those objects by comparing their confidence fi (i.e., the logit from the object detector) with a threshold p. E is the set of edges ei,j , consisting three types:
• Target-pair edge, which connects two given target objects. • Target-surrounding edge, which connects each target object to all their surrounding nontarget objects. • Near-neighbor edge, which connects those non-target objects in near neighbor that may have implicit correlations between each other. We build the edges by calculating their coordinates (loci), with those values larger than a pre-defined threshold d as valid edges.
The edge ei,j=1 when there is an edge between vi and vj . Figure 3 illustrates the edge constructions.

heading: Encoding Graph with Object-centered GCN
text: While graph convolutional network (GCN) (Marcheggiani and Titov, 2017) has been shown effective for aggregating graph data, it may fail to model the centralization of the target objects of GO3D-S2G structure (as GCN treats all nodes equally). Thus, we devise an object-centered GCN, which advances in modeling both the edge features and the target objects. OCGCN first creates initial
FFN
0.6 0.4
0.3
0.03 0.02 0.1 0.01

heading: Step2. Step3. Step1. Scoring edges
text: Eq. (5)
Choosing highest-scored (first-order) neighbors of target objects
➢Target object-1 ( ) perspective
➢Target object-2 ( ) perspective
0.4
0.30.01
0.6
0.02 0.1
0.6 0.4 ✘
Assembling and pruning
➢Assemble two perspectives
➢Pruning lower edge in the cycle
Figure 4: Scene subgraph selecting mechanism.
representations of node svi and edge s e i,j .
sposei = Embed(orii ⊕ sizei) , svi = FFN(s vis i ⊕ sposei ) ,
sei,j = FFN(loci ⊕ locj) , (1)
where Embed() is the looking-up operation, FFN() is the non-linear feedforward network.
Then, OCGCN updates the GO3D-S2G: svi = σ( ∑
j=1
γi,j(Wa · [svi′ ; s e i,j ; s v t ])) ,
γi,j = ei,j · exp(Wb(svj′ ⊕ ei,j ⊕ s v t ))∑
t=1 ei,t · exp(Wb(svt′ ⊕ ei,t ⊕ s v t ))
,
(2)
where sv i′ is the node representation of last layer, as OCGCN has total L layers. svt =s v O1
⊕ svO2 is the summary of the two target objects. [; ] is the concatenation operation. Wa,Wb, b are learnable parameters. The weight γli,j reflects the contribution of each object when propagating the spatial attributes towards target objects.

heading: 2.2 Decoder: Spatially-diversified Text Generation
text: In decoding stage, we use a VL-PLM to generate VSD texts, as shown in Figure 2. We first perform scene subgraph selection over GO3D-S2G, where the diverse local structures lead to spatiallydiversified text generation. Also we create prompt texts to hint model to generate relevant contents.
Scene Subgraph Selecting As cast earlier, we can make use of the neighbor non-target objects of the two target objects in the scene as type of ‘bridges’ to diversify the generation. Thus, we propose a scene subgraph selecting (namely, S3) mechanism to sample sub-structures of GO3D-S2G.
Concretely, S3 contains three steps, as illustrated in Figure 4. First, we calculate the connectingstrength score for each edges in GO3D-S2G via a simple FFN transformation: ai,j=FFN(ŝei,j), where ŝei,j is the edge representation of final-layer OCGCN. In the second step, we take a first-order traversal to search the best neighbor nodes of two target objects, respectively, where the best neigh-
bor nodes have the highest connecting scores to their target objects. Note that we only consider the direct neighbor of target objects (i.e., first-order connection), because including nodes in too distant will rather lead to inaccurate descriptions. In the third step, we assemble the two perspective structures into one, and then prune the conflicting edge with lower connecting score if a cycle exists (i.e. two target objects connects to a common neighbor), resulting in a successful subgraph.
It is noteworthy that during training we sample only one subgraph with highest-scored edge, where reparameterization trick (Blum et al., 2015) is used for gradient propagation. During inference, we sample multiple scene subgraphs with top-k highest score edges, i.e., giving diverse descriptions.
With the subgraph at hand, we create its representations via a mean-pooling operation over it:
λi = δ(vi ∈ Gsub) + at1,i + at2,i∑
l∈G (at1,l + at2,l) ,
rG = MeanPool({λiŝvi , λiŝei,j |G}) , (3)
where ai,j is the strength score, t1 and t2 are the target nodes, δ(exp) funtion outputs 1 when exp is true, otherwise 0 , ŝvi , ŝ e i,j are the node and edge representations of last-layer OCGCN. The local scene graph feature rG will be used during text generation at following stage.
Building Directional Prompts Now we try to guide the VL-PLM to generate contents closely relating to the target objects and the nodes in the sampled subgraph. We thus build two types of prompt texts, 1) target object prompt, e.g.,
<TGT> table <TGT> sofa
and 2) spatial relation prompt, e.g., <OBJ> table <REL> near <OBJ> sofa <OBJ> sofa <REL> left <OBJ> bed Two types of prompts are concatenated as one via a ‘<SEP>’ token. The former enlightens model what the target objects are, and the latter tells model what possible relational triplets are, i.e., “<objecti, relation, objectj>”, where “relation” is the predefined relation term. For each pair of Oi, Oj in GO3D-S2G, we map their edge eij to a specific direction term based on their centroid coordinates.
We maintain a prototype of 3D universal direction-term mapping, as shown in Figure 5, where we define 26 directions in the whole sphere, with each direction binding certain directional terms. Even a strong VL-PLM may fail to map a direction to a term accurately. We thus additionally perform pre-training to strengthen the perception of direction. The detailed mapping rules of universal 3D direction-term are shown in Table 2. With the predefined 26 directions, we compare the cen-
troid coordinates of a pair of objects to decide the direction terms. Note that according to the rules in Table 2, there may be multiple terms for the same direction, e.g., “left up front” and “up left front” and we keep these redundant terms in our implementation. We add an extra term “next to” to describe the situation that two objects are close to each other. For some types of object that not exists during the 3D Scene Extractor pretraining, we just use their 2D locations and treat the depth coordinate to 0.
Moreover, we conduct a pre-training to strengthen the perception of direction for VL-PTM. Concretely, we utilize the 3D scene extractor and relation triplets ground-truth in VSD dataset to generate a set of pseudo data. For example, if we have two target objects O1, O2 with theirs names Tag1, Tag2 ,ground-truth relation term Relg (in VSD annotations), and 2D boxes Box1, Box2, we could get their 3D centroid coordinates loc1, loc2 through off-the-shelf 3D scene extractor. Then we map the 3D centroid coordinates to 3D direction term Relp. We use “<OBJ> Tag1 <REL> Relp <OBJ> Tag2” as inputs and “Tag1, Relg, Tag2” as outputs to train the VL-PTM. Moreover, we randomly replace the Relg with some synonyms for data augmentation.
Generating Text Finally, we feed the prompt text and the raw image as input into our backbone VL-PLM encoder, where the resulting representations rT and rI and the local scene graph feature rG are fused together via cross-attention, i.e., r=CrossAtt(rG, rT , rI ). The VL-PLM decoder then performs text generation based on r.

heading: 3 Experiments

heading: 3.1 Settings
text: Dataset We evaluate our model on two datasets: VSD-v1 and VSD-v2 (Zhao et al., 2022). VSD-v1 is the initial version of VSD datasets, which has a large scale but simple annotations. VSD-v2 has the same image source while more complex and human-friendly descriptions, which is more challenging. We use the original split of train/dev/test set of each dataset.
Implementation Our model takes the pre-trained 3D extractor from Nie et al. (2020), containing the layout estimation network and the 3D object detection network. The hidden size of OCGCN is 768 which is the same with text decoder. The dimension of edge feature sei,j is also 768. We adopt the OFAbase as our backbone VL-PLM.
Evaluation We make comparisons with 1) the existing popular image-to-text VL-PLMs, including OSCAR (Li et al., 2020), VLT5/VLBart (Cho et al., 2021), OFA (Wang et al., 2022b); 2) the models introduced in Zhao et al. (2022), including the pipeline (ppl) and the end-to-end (e2e) paradigms. Zhao et al. (2022) use the visual spatial relations classification (VSRC) results as intermediate features for VSD. Following Zhao et al. (2022), we adopt five automatic metrics to evaluate performances, including BLEU-4, MENTEOR, ROUGE, CIDEr and SPICE. We measure the diversity with three metrics, i.e., mBLEU-4, BLEU-4@K and SPICE@K. All the used VL-PLMs are the base version. Our results are the average scores over five runs. Appendix §B.3 details all the experimental settings.

heading: 3.2 Main Observations
text: Main Results As shown in Table 3, overall, the VSD-v2 can be more challenging than VSD-v1, where model scores on all the metrics are lower. Also we see that four different VL-PTMs show the similar level of performances, due to their general purpose of pre-training for multimodal learning. By taking advantages of the VSRC features, Zhao et al. (2022)’s methods outperform the baseline
vanilla VL-PTMs on the task. However, the improvements from Zhao et al. (2022)’s models can be incremental, due to the reason that Zhao et al. (2022) model the input images with only 2D information. On the contrast, our proposed 3DVSD model achieves significant improvement over the baselines cross two datasets on all the metrics, evidently demonstrating its efficacy.
In addition, our model shows larger improvements on the harder VSD-v2 data than that on VSD-v1. To directly measure the capability of our method, we further collect a subset from VSD-v2, where the target objects in images are irregularlyposed with complex spatial relation, and also there are overlapped layouts. We perform experiments on the set, where the results are shown in Table 4. We can find that our 3DVSD model improves the best-performing baseline with marked boosts, i.e., 1.87 BLEU-4, 1.94 METROR, 1.24 ROUGE and 5.11 CIDEr. This significantly indicates that our method is capable of well understanding the visual spatial semantics and thus generating more diverse and flexible VSD sentences.
Model Ablation Now we quantify the contribution of each design in our systems via model ablation, as shown in Table 5. First, we can see that the 3D scene feature from GO3D-S2G graph modeling gives the biggest influences, i.e., contributing 3.09 BLEU-4 and 3.08 SPICE scores. Besides, the OCGCN encoder, the S3 mechanism as well as the direction-term prompting also plays an essential role to the overall system, respectively.
Evaluation on Spatial-diversity Generation As we equip our system with the S3 mechanism, we can generate spatially-diversified texts. Next, we directly assess the ability on the generation spatialdiversification. We first make comparisons with the beam search method using automatic metrics (Deshpande et al., 2019), including mBLEU-4, BLEU-4@K and SPICE@K. mBLEU-4 compares the 4-gram matching between one of the generated sentence and the remaining generated sentences for an image, and thus lower mBLEU-4 score means more diversity. BLEU-4@K and SPICE@K represent the highest BLEU-4 and SPICE score for the top-k generated sentences for an image, where higher BLEU-4@K and SPICE@K prove that the a model can keep better semantics accuracy while generating diverse results. As shown in Table 6, our S3 mechanism achieves lower mBLEU-4 and higher BLEU-4@K and SPICE@K, demonstrating that our method could generate diversified descriptions with enough semantics accuracy.
We also provide a human evaluation to describe spatial diversity with respect to the Spatial Accuracy, Spatial Diversity and Fluency. We ask 10 English speakers to answer the 5-point Likert scale on 100 samples, where the average results are shown in Table 7. Overall, all the models can achieve competitive score on language fluency, thanks to the superiority of VL pretraining. Also the tendencies of spatial accuracy and diversity is consistent with the automatic evaluation. Remarkably, our S3 mechanism shows the best capability on spatial diversity, demonstrating its effectiveness.

heading: 3.3 Analyses and Discussions
text: To gain an in-depth understanding of our method’s strengths, we try to answer following research questions via further analyses:
• RQ1: How 3D scene features help understanding the spatial semantics of input images?
A: The key of our method is the leverage of 3D scene features. Now, we first consider downgrading the 3D features into the 2D ones, such that we can gain the perception of its necessity. To ensure the fair comparison, we remove the 3D scene extractor and replace the 3D pose feature posei (Eq. 1) with the 2D size. Also we replace the 3D coordinates by 2D coordinates. And the other settings are kept the same. As shown the results in Figure 6, the 2D scene modeling results in the markedly performance decrease.
We can further quantify the contributions of each type of 3D features via feature ablation, including the orientation feature orii and the size feature sizei. As seen in Figure 7, both 3D orientation and
3D size features contribute to the overall system. Also the influence on SPICE is larger, which indicates that the orientation and size of the objects especially help recognize the spatial relation. Finally, in Figure 8 we empirically show the OCGCN’s kernel weight γi,j (Eq. 2) on two pieces of instances, where the attention values reflect the contribution of each object. It is clear that our model has successfully captured the spatial semantics of the target object pairs.
• RQ2: How does S3 mechanism aid the diversified spatial description generation?
A: Next, we consider investigating how exactly the S3 mechanism contribute to the spatial description generation. Through our S3 mechanism, we could generate 4 types of subgraphs: 1) with only target nodes (2-hop); 2) with one non-target neighbor node linked with subject node (3-hop-s); 3) with one non-target neighbor node linked with object node (3-hop-o); 4) with two non-target neighbor nodes. In our implementation, we use a threshold pcut to filter out edges with very low scores. Figure 10 show the distribution of four subgraphs with pcut=0.1 or 0.2. We see that different pcut val-
ues help generates spatial descriptions with varying numbers of objects attended into the subgraphs. That is, S3 mechanism aids the diversified spatial description generation by producing multiple heterogeneous subgraphs structures.
We also empirically show the qualitative results in Figure 8. We notice that the beam search method can generate multiple alternative texts in both cases, where unfortunately the diversification on describing the spatial relation is much inferior and limited. In contrast, with our S3 method, the system generates spatial-diversified descriptions for both two images. Some surrounding objects, e.g., ‘shelf’, ‘table’, ‘door’ in the first case and ‘chair’, ‘bed’, ‘shelf’ and ‘desk’ in the second case, are leveraged to aid describe the target objects.
• RQ3: To what extent the external 3D scene extractor quality influence the VSD performances?
A: As we obtain the initial 3D scene features from the external extractor, the quality of the extractor is key to our final VSD performance. Note that the off-the-shelf 3D extractor is trained on a datasets of indoor scenes, while in VSD dataset the images contain both types of indoor (NYU) and out-
door (Visual Genome (VG) and Flickr). As shown in Figure 9, the indoor images actually are the minority in our VSD data. Here we perform analysis to valid the influence on 3D scene extractor. We split the VSD-v2 test set into indoor and outdoor subsets according to the domain types. Then we run our model on the two sets separately. As shown in the figure, the results on the indoor NYU set exceed those on the outdoor VG&Flickr set clearly, demonstrating the domain shift issue in our system. Given that our system has already secured considerable performance increase than existing models, we presume that when obtaining a 3D extractor capable of detecting higher-quality 3D scene features for any domain and scenario, our method has the greater potential to gain more task improvements.

heading: 4 Related Work
text: Image-to-text (I2T) is a fundamental task category of the vision-language multimodal topic. Existing I2T tasks, e.g., image captioning (Vinyals et al., 2015; Cornia et al., 2019; Mathews et al., 2018) and VQA (Antol et al., 2015; Lubna et al., 2019; Manmadhan and Kovoor, 2020), attempt to generate textual pieces to understand the image content semantics through different perspectives. VSD is also a subtask of I2T, which however places the focus on the spatial semantics understanding. Within recent years, VL-PTMs are extensively employed for the I2T tasks, which have helped achieve state-of-theart performances on many benchmarks (Lu et al., 2019; Chen et al., 2019; Zhou et al., 2020; Li et al., 2020; Wang et al., 2022b). Prior VSD study (Zhao et al., 2022) has benchmarked the VSD task with these general-purpose VL-PTMs. Unfortunately, different from the content-semantic I2T tasks, the goal of VSD is to grasp the spatial semantics.
This work also relates to the theme of visual spatial understanding, which has been intensively investigated in the past years. Yang et al. (2019); Jänner et al. (2018) propose the visual spatial relation classification (VSRC) task, aiming to predict whether a spatial description is reasonable for the input image. Differently, VSD aims to directly generate the descriptions of the spatial relations. The bottleneck of visual spatial understanding tasks is the capturing of spatial semantics. Thus, in this paper, we propose improving the visual spatial understanding for VSD via modeling the holistic 3D scene features. Our work takes the advancements from the topic of 3D scene parsing (Huang
et al., 2018a,b; Nie et al., 2020; Zhang et al., 2021), which reconstructs the 3D scene from a single RGB image. By using the 3D scene features, e.g., 3D layout and 3D object, we achieve the goal of more in-depth understanding of the spatial semantics.

heading: 5 Conclusion
text: In this work we incorporate the 3D scene features for improving the visual spatial description (VSD) task. We first employ an off-the-shelf 3D scene extractor to produce 3D objects and scene features for the input images. We then build a target object-centered 3D spatial scene graph (GO3DS2G) structure, which is encoded with an objectcentered graph convolutional network. Next, we devise a scene subgraph selecting mechanism to sample topologically-diverse subgraphs from GO3DS2G, where the local features are used to guide to generate spatially-diversified descriptions. On the VSD datasets our framework shows superiority on the task, especially for solving the complex cases, such as layout-overlapped and irregularly-posed object. Meanwhile our method can produce more spatially-diversified generation. Finally, we demonstrate the influence of quality of the external 3D scene extractor.

heading: Acknowledgments
text: This research is supported by the National Natural Science Foundation of China under Grant No. 62176180. The work is also supported by the SeaNExT Joint Lab.

heading: Limitations
text: This work has one major risk. As the main idea proposed in this work heavily relies on the external 3D scene extractor, the quality of extractor on our used VSD images largely influences the task performance. However, we reveal in analysis that although suffering from the domain shift issue by the out-of-domain 3D scene extractor, our method still improves the VSD task. We show that when handling the in-domain VSD images as used for training the 3D scene extractor, the VSD performance has been boosted remarkedly. Thus, with better a 3D scene extractor, it can be expected that our system will exhibit much stronger capability and advance the VSD task more significantly.

heading: A Model Specification
text: Here we provide the detailed calculations of our system. For the MeanPool operation in Equation 3, we calculate rG as:
rG = 1
M2
∑
i∈G
∑ j∈G λi ( ŝvi ⊕ ŝei,j ) (4)
where M is the node number of the GO3D-S2G, λi is a weight expansion (shown in Equation 3), which could capture the subgraph features with a very high weight, ŝvi and ŝ e i,j are the node and edge representations of last-layer OCGCN. For VL-PTM encoding, we adopt the approach in OFA model (Wang et al., 2022b). The input image I is first embedded by a inbuilt ResNet and flatted to a tokenized visual feature. Then the visual features are projected to the encoder model dimension, obtaining the global visual feature rI . The process could be formalized as:
r̂I = ResNet(I),
rI = Wvr̂ I + bv,
(5)
where Wc and bc are model parameters. Then the tokenized visual features are concatenated with embedded text sequence as the inputs of VL-PTM encoder:
hI,T = Encoder(rI ⊕ rT ). (6) After that, the encoder outputs and graph feature rG are fused through cross-attention in VL-PTM decoder. The cross-attention operation in a transformer unit could be formalized as:
X = { rG;hI,T } ,
K = WkX,V = WvX,Q = WqDh,
Attention(Q,K,V ) = softmax ( QK√
d
) V ,
(7)
where rG is the graph feature, hI,T is the encoder outputs, Wk,Wv,Wq are model parameters, d is the out dimension of Wv, Dh is the outputs of self-attention unit in Transformer decoder.
A.1 3D Scene Extracting
We employ an external 3D scene extractor from (Nie et al., 2020), which is pretrained on the SUN RGB-D dataset (Song et al., 2015). The model contains three modules: 1. Layout Estimation Network (LEN); 2. 3D Object Detection Network (ODN); 3. Mesh Generation Network (MGN). In our framework, we only use LEN and ODN, which outputs the layout coordinates system and the 3D location
and pose of each object. We also need a 2D Object Detector to generate appearance feature and 2D box as the inputs for ODN. We directly borrow the Faster-RCNN (Ren et al., 2015) to process the images to obtain the necessary RoI features, boxes, and object tags. For the LEN, we need the camera intrinsic parameters to adjust the coordinates system. However, the VSD dataset does not provide this information of the images. We just generate a pseudo camera intrinsic matrix for all the images in our implementation. The method works in VSD because we do not need the accurate location of each object, we just need to capture the relative relations among the object pairs. Thus, the distortion of absolute coordinates will hardly impact the final results of VSD. As shown in Figure 12, though the 3D boxes are distorted, their relative spatial relations remain unchanged.

heading: A.2 Spatial Scene Graph Creating
text: The detailed algorithm of S3 is shown in Algorithm 1. Suppose that the max object number is N (N=36 in our implementation). We first initialize an N × N adjacency matrix A = 0. When adding the Target-pair and Target-surrounding edges, we set the columns and rows of the target object index to 1. When adding the Near-neighbor edge, we traverse the edges and set the one with dist > d in A to 1. At last, we should remove the edges of noisy objects, setting the related items of A to 0.

heading: A.3 Overall Training Procedure
text: We have several pretrained external modules in our system. First, we prepare the pretrained ResNet for image embedding, which will be used in 3D layout estimation, 3D object detection and the OFA image embedding. Before training, we use the pretrained
Algorithm 1: GO3D-S2G Creating Input: max object number N ,
two target objects index o1, o2, confidency of each object f , centroid of each object C, distance threshold d, noise confidency threshold p
Output: adjacency matrix AN×N initialization: A = 0. // target object edges A[o1,:] = 1, A[:, o1] = 1, A[o2,:] = 1, A[:, o2] = 1, // add special edges for i in N do
for j in N do dist = ||Ci − Cj || if dist > d then
Aij = 1 end
end end // remove noise objects for i in N do
if fi < p and oi is not target object then A[i,:] = 0, A[:,i] = 0
end end
Faster-RCNN to preprocess all the images for 2D object detection, obtaining the 2D boxes and RoI features. We also prepare the 3D scene extractor pretrained on SUNRGB-D with the method in (Nie et al., 2020).
Then we collect the three modules and train our whole system, where the 3D extractor and the decoder (VL-PTM) are initialized by pretrained parameters. The OCGCN and other parameters (e.g. edge embedding, connecting-strength scorer) are initialized randomly. Before global training, we
pretrain the VL-PTM with direction terms as Table 2. During global training, the 3D scene extractor are frozen while others will be updated.
We have two main training targets, e.g., the crossentropy between Vl-PTM decoder outputs and the ground-truth description of VSD Lt, and the crossentropy between connecting-strength score distribution and target object index Lc:
Lt = − Nw∑
l=1
log pθ (yl|x,y<l) ,
Lc = − N∑
i=1
δ(i ∈ Og)(log at1,i + logat2,i), (8)
where Nw is the text length, x is the VL-PTM encoder outputs and yl is decoder outputs in the l-th step , N is the node number of GO3D-S2G, ai,j is connecting-strength score during S3, t1, t2 are the two target objects, Og is a set that contains the objects appear in the ground-truth description, and δ (·) is the two value function that outputs 1 when · is true, otherwise 0. We obtain Og by string searching in the descriptions. Then the final loss is L = Lt + Lc.

heading: B Experiment Specification

heading: B.1 Data Analyses
text: The VSD dataset contains images from Visual Genome, Flickr, and NYU-Depth (Zhao et al.,
2022; Yang et al., 2019), where NYU-Depth is the indoor dataset (Silberman et al., 2012).
Figure 13 shows the distribution of object tags in the VSD dataset. Table 8 lists the object types in SUNRGB-D dataset which are used to pretrain 3D scene extractor. There are several common types, such as “person”, “table”, “chair”, “wall” and “window”. Thus the 3D scene extractor could achieve a comparable performance in VSD. For other object types, the built-in 2D object detector (pretrained on VG or COCO) could provide a passable 2D location information.
There are two versions of VSD dataset released, where the VSD-v1 contains large-scale while relative simple descriptions and the VSD-v2 contains small-scale while diversified descriptions. The two versions share the same image set. The statistics of the two datasets are shown in Table 9. Figure 14 shows the difference between the annotations of the two datasets. The annotated sentence in VSD-v1 is relative short and simple while those in VSD-v2 contains more semantics and be more challenging.

heading: B.2 Extended Experimental Implementations
text: Data Preprocess For memory space and time saving, we preprocess the data through FasterRCNN to obtain the RoI featues, object tags and 2D boexs and save them to files. For 3D scene extractor, we also preprocess the images, saving their ResNet features to files.
VL-PTM Configuration For fairly comparison, we employ the OFA-base as our VL-PTM, which has the similar scale of parameters to the baselines. The layers number is 6 for both encoder and decoder, and the head number of the multi-head attention is 6 as well. Besides OFA, we can use any type of VL-PTMs such as VLT5/VLBart, or combination of single VL encoder and text decoder, such as “CLIP-GPT”, “ViT-GPT”. For more comparison, we leave them in future works.

heading: B.3 Hyperparameters
text: Table 11 lists the hyperparameters of our implementation.

heading: B.4 Baselines
text: We use some strong image-to-text models as our baselines.
• Oscar (Li et al., 2020) is a BERT-like visuallanguage pretrained model, only containing the Transformer encoder. Oscar takes tokenized RoI features of the image as visual inputs. This model has shown strong performance on Image Captioning and VQA.
• VLT5/VLBart (Cho et al., 2021) are continue pretrained on image-to-text tasks from generative encoder-decoder language model Bart and T5. Previous work (Zhao et al., 2022) has proven their capability on solving VSD task. • OFA (Wang et al., 2022b) is a new visionlanguage model, which are pretrained by a mount of vision-language multi-modal tasks. It has achieved several SoTA results on existing image-to-text tasks.

heading: C Extended Experiments

heading: C.1 Diversity Evaluation
text: We compare more results of sampling scale for diversity evaluations. Figure 10 reports the results of K=5 and K=10 on VSD-v2. The tendency of K=10 is consistent with that of K=5, where the S3
achieves lower mBLEU-4 and higher BLEU-4@K and SPICE@K.
For human evaluation, our 5-point Likert scale is designed as follows:
• Spatial Accuracy: The sentences correctly describe the spatial relationship of the target objects. • Spatial Diversity: These sentences describe diversified spatial semantics. • Fluency: The sentences are readable and not different from human sentence-making. Each question will be answered by a number from 1 to 5, denoting “Strongly Disagree”, “Disagree”, “Neither disagree nor agraa”, “Agree” and “Strongly agree”. We select 100 samples and generated 5 sentences for each, and sent them to the evaluators for scoring.

heading: C.2 More Case Study
text: Figure 15 provides some more qualitative comparisons among models and human. Compared with baselines, our model could generate description more like human-made.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
section 6
A2. Did you discuss any potential risks of your work? Not applicable. We use public datasets
3 A3. Do the abstract and introduction summarize the paper’s main claims? section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 4 and appendix
3 B1. Did you cite the creators of artifacts you used? section 4 and appendix
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? section 4 and appendix
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? section 4 and appendix
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We use public datasets without people or offensive content.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. appendix
C 3 Did you run computational experiments? section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? section 4 and appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? section 4 and appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 4 and appendix
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 4 and appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: For more comparison, we leave them in future works