Transforming Visual Scene Graphs to Image Captions
Year: 2023

Abstract:
We propose to TransForm Scene Graphs into more descriptive Captions (TFSGC). In TFSGC, we apply multi-head attention (MHA) to design the Graph Neural Network (GNN) for embedding scene graphs. After embedding, different graph embeddings contain diverse specific knowledge for generating the words with different part-of-speech, e.g., object/attribute embedding is good for generating nouns/adjectives. Motivated by this, we design a Mixture-of-Expert (MOE)-based decoder, where each expert is built on MHA, for discriminating the graph embeddings to generate different kinds of words. Since both the encoder and decoder are built based on the MHA, as a result, we construct a simple and homogeneous encoder-decoder unlike the previous heterogeneous ones which usually apply Fully-Connected-based GNN and LSTM-based decoder. The homogeneous architecture enables us to unify the training configuration of the whole model instead of specifying different training strategies for diverse sub-networks as in the heterogeneous pipeline, which releases the training difficulty. Extensive experiments on the MS-COCO captioning benchmark validate the effectiveness of our TFSGC. The code is in: https://anonymous.4open.science/r/ ACL23_TFSGC.

heading: 1 Introduction
text: Image captioning, which aims to generate one sentence for describing multi-aspects of an image, has made huge progress since the proposal of the encoder-decoder framework (Vinyals et al., 2015; Xu et al., 2015). Such a framework contains one visual encoder to extract a series of visual features from the image and one language decoder to generate captions from the extracted visual features. Since the visual encoder is usually well pre-trained by image classification and object detection, the
*Corresponding authors.
extracted features contain abundant knowledge of the object categories, which enables the captioning model to generate object-abundant captions.
However, object category is not the only visual pattern that matters for high-quality captions (Anderson et al., 2018; Jiang et al., 2020), object attributes and relations also play significant roles in generating descriptive captions, i.e., the caption containing multi-aspects of an image. Motivated by this, researchers propose to incorporate additional semantic knowledge, e.g., object categories, attributes, and relations, into the captioning models by using the scene graph as the mediator (Yao et al., 2018; Yang et al., 2020). Scene graphs assign each object node with certain attribute nodes and some pairwise objects with certain relation nodes. These nodes are represented by the corresponding semantic tags, e.g., as shown in Fig. 1, the object “dog” is assigned with the attribute “black” and the pairwise objects “dog” and “fish” have the relation “bite” in between. To exploit the scene graph, Graph Neural Network (GNN) (Battaglia et al., 2018) is deployed to embed the graphs and the output embeddings are input to the decoder for captioning.
The top part of Fig. 1 shows the pipeline of the previous popular GNN-based captioning model (Yao et al., 2018; Yang et al., 2020), which implements GNN as a few Fully-Connected (FC) and non-linear activation layers. To update the node embedding, this GNN maps the concatenated neighbour embeddings into the new one (Xu et al., 2019). Then the updated graph embeddings are input into the language decoder that contains a few LSTM layers and an attention module. The LSTM layers are used to generate the context vector based on the partially generated captions. This context vector works as the query in the attention module for determining which graph embeddings should be used to generate the next word. Compared with the models without GNN, this GNN-LSTM pipeline usually gets better performance.
12427
However, this GNN-LSTM framework implies two flaws which hinder the further improvement of applying scene graphs. First, FC-based GNN and LSTM do not share the same building blocks and thus the constructed model is a heterogeneous structure, which requires well-chosen training strategies, e.g., choosing different learning rates or optimizers for different sub-networks, to achieve the best performance (Yang et al., 2020). Finding such training configurations is a labour-intensive process. Second, the graph embeddings are indiscriminately selected during captioning (the grey embeddings in Fig. 1 top (c) denote such indiscrimination), which causes less descriptive captions. While intuitively, different kinds of node embeddings should be used for generating the words with diverse part-of-speech (POS), e.g., the object/attribute/relation embeddings should be more responsible for the nouns/adjectives/verbs, respectively (Yang et al., 2019b).
To alleviate the above-mentioned flaws, we propose a novel homogeneous captioning model to Transform Scene Graphs (TFSGC) into captions. Our TFSGC is built based on the Transformer (Vaswani et al., 2017) since it is more powerful than LSTM in image captioning (Herdade et al., 2019; Li et al., 2019; Cornia et al., 2020). TFSGC is homogeneous since we use multi-head attention (MHA) to design both the graph encoder to embed the scene graphs and the language decoder to generate the caption.
Our design principle is quite simple where we do not need to revise the self-attention operation but only need to reformulate the input data structure. Specifically, to design GNN by MHA, we
first linearize the scene graph into a token sequence and introduce a binary mask to index which two nodes are connected in the graph. Then we use the masked MHA operation to deal with this linearized token sequence for graph embedding. In this process, each token embedding is added by a learnable type embedding to index the token type (e.g., object/attribute/relation) and we will show that such type embedding can help distinguish the edge type during the attention calculation.
After graph operation, we get a series of object/attribute/relation embeddings, which will be used in the decoder for captioning. To make the decoder discriminate different embeddings for generating different words, we learn from MOE networks (Jacobs et al., 1991; Xue et al., 2022; Du et al., 2022) to revise the original Transformer decoder with two strategies. First, as Fig. 1 bottom (d) shows, we use three encoder-decoder attention layers, which are built on MHA, as three experts to address object/attribute/relation embeddings, respectively. Second, we incorporate an attention-based soft routing network to discriminate which kinds of embeddings should be more responsible for generating the next word. Both the MOE-decoder and the type embedding in the encoder help distinguish node embeddings for better captions. We carry exhaustive ablation studies and comparisons to validate the effectiveness of TFSGC and it achieves 132.3/138.6/139.5 CIDEr scores when using BUTD/Patch/VinVL features.

heading: 2 Related Work
text: Image Captioning. For a long time, the attentionbased CNN-LSTM pipeline (Vinyals et al., 2015;
Xu et al., 2015) is the most popular backbone for captioning and various techniques have been added into it for better performance, including building stronger visual encoders (Lu et al., 2018; Jiang et al., 2020), designing more advanced attention mechanisms (Anderson et al., 2018; Wang et al., 2020), incorporating semantic knowledge (You et al., 2016; Gan et al., 2017; Yao et al., 2018; Yang et al., 2020), and exploiting language structure (Lu et al., 2017; Yang et al., 2019b).
Recently, Transformer (Vaswani et al., 2017) has gradually substituted LSTM as the mainstream language decoder in image captioning (Herdade et al., 2019; Li et al., 2019) since it achieves better performances than the LSTM-based models. Based on this new backbone, researchers develop more advanced strategies for further improving the effectiveness, including designing more sophisticated attention mechanisms (Huang et al., 2019; Pan et al., 2020), introducing additional memory blocks (Cornia et al., 2020; Yang et al., 2021b), distilling knowledge from the large-scale pre-training models (Radford et al., 2021; Li et al., 2021; Xu et al., 2021), and exploiting Transformer-based visual encoders (Wang et al., 2022; Fang et al., 2022), modularized design for large-scale multi-modal pretraining (Li et al., 2022; Xu et al., 2023; Ye et al., 2023). Since the recently proposed SOTA models use Transformer as the backbone, we also built TFSGC based on Transformer for fair comparison.
Graph Neural Network (GNN). Scene Graph abstracts the major visual patterns in a visual scene as a graph. It is usually used as the mediator to narrow the gap between the vision and the language domains. To incorporate scene graphs into deep networks, GNN (Battaglia et al., 2018) is used to embed the discrete node labels into dense embeddings. However, most of the previous GNNs are MLP-based (Yang et al., 2020; Yao et al., 2018; Xu et al., 2019; Milewski et al., 2020; Zhong et al., 2020), which may limit the effectiveness of embedding scene graphs in a Transformer architecture. In our research, we design an MHA-based GNN to remedy this limitation. Moreover, noisy scene graphs may damage the performances (Nguyen et al., 2021), so we use better scene graph parsers to minimize the impact of noise on our model.
Mixture of Experts (MOE). The major idea of MOE is to construct a network with lots of experts where different experts deal with diverse sam-
ples (Jacobs et al., 1991; Shazeer et al., 2017). When a sample is input to the MOE network, a routing network will decide which experts should be more responsible for this input. Thus MOE naturally fits our case where we hope diverse experts can discriminate graph embeddings for generating the words with different POS. Different from the existent MOE-based Transformer (Lepikhin et al., 2020; Xue et al., 2022; Du et al., 2022) which applies various feed-forward networks as different experts, we set three encoder-decoder attention layers as different experts where the query is set to the same context vector while the key and value are set to object/attribute/relation embeddings.

heading: 3 Revisiting of Transformer
text: We first revisit the Transformer-based captioning model and then introduce how to revise it to get our TFSGC in the next section. Transformerbased model contains a visual encoder to calculate the contexts of the extracted visual features and the output embeddings will be input into a language decoder for captioning. For both the encoder and the decoder, the most elemental building block is the multi-head attention (MHA). Given the query, key, and value matrices: Q ∈ RNQ×d, K ∈ RNK×d, V ∈ RNV ×d, MHA calculates the output Y = MHA(Q,K,V ) as:
Input: Q,K,V
Att: Ai = Softmax( QWQi (KW K i ) T
√ d
)
Head : Hi = AiV W Vi ,
Multi-Head: H = [H1,H2, ...,Hh]WH , Output: Y = LN(H +Q),
(1)
where WQi ,W K i ,W V i ∈ Rd×dh , and WHi ∈ Rd×d are all trainable matrices; h is the number of attention heads (set to 8 in the experiments) and dh = d/h; Ai denotes the i-th attention matrix used to calculate the i-th head matrix; [·] means the concatenation operation; and LN is the Layer Normalization operation.
Besides MHA, another important module in Transformer is the Feed-Forward network (FFN):
FFN(Y ) = LN(FC(ReLU(FC(Y ))) + Y ), (2)
where FC denotes the fully-connected layer and ReLU denotes the rectified linear function.
Given MHA and FFN, we can use them to build a Transformer-based captioning model. For the encoder, it stacks 6 identical blocks where each one contains an MHA and an FFN. Given the output of the former block as the input X , the next block
calculates its output as: Input: X,
Self-ATT: Y = MHA(Q = X,K = X,V = X), Output : Z = FFN(Y ), (3) Note that the variables X,Y ,Z used here are “local variables” for conveniently introducing the work flow of Transformer architecture, whose values will be set to the specific values when introducing the concrete captioning model. In “Self-ATT”, Q,K,V are set to the same value and this operation is named as self-attention (Vaswani et al., 2017). After stacking 6 blocks defined in Eq. (3), a visual encoder is built. For the first block, its input is the extracted visual feature set of the given image. The output of the last block will be input into the language decoder.
For the decoder, it also stacks 6 identical blocks where each one contains two MHAs and an FFN. Given the output of the former decoder block XD and the output of the visual encoder XE , the next decoder block calculates its output:
Input: XD,XE Self-ATT: Y1 = MHA(Q = XD,K = V = XD), ED-ATT: Y2 = MHA(Q = Y1,K = V = XE), Output : Z = FFN(Y2), (4) Note that in “ED-ATT”, Q is set to the output of the former decoder block while K,V are set to the output of the visual encoder, and such operation is called encoder-decoder attention (Vaswani et al., 2017). After stacking 6 blocks defined in Eq. (4), a language decoder is built.
For the first block in the decoder, XD in Eq. (4) is set to the word embedding set of the partially generated captions S = {s1, ..., st} at the t-th time step. For all the decoder blocks, the input XE is set to the same value, which is the output of the visual encoder. The output of the last decoder block Z = {z1, ...,zt} is used to calculate the word distribution of the next word:
P (st+1) = Softmax(zt). (5) Given the ground-truth caption S∗, we can train this model by minimizing the cross-entropy loss: LXE = − logP (S∗), (6) or by maximizing a reinforcement learning (RL) based reward (Rennie et al., 2017): RRL = ESs∼P (S)(r(Ss;S∗)), (7) where r is a sentence-level metric for the sampled sentence Ss and the ground-truth S∗, e.g., the CIDEr-D (Vedantam et al., 2015) metric.

heading: 4 Transforming Scene Graphs
text: In this section, we introduce how to revise the Transformer to get our TFSGC. We will first show how to get an MHA-based GNN and then introduce how to design an MOE-based decoder.

heading: 4.1 MHA-GNN
text: A visual scene graph (Krishna et al., 2017) contains three kinds of node embeddings: object/attribute/relationship embeddings o/a/r. These nodes are connected by the following rules: if an object oi has an attribute ak, oi and ak are connected, e.g., o1 connects a1 in Fig. 2 (a). If two objects oi and oj have the relation rk, we connect rk with oi and oj , e.g., r1 connects o1 and o2. Given an image, we extract a series of visual features from the im-
age as the object embeddings: {o1, ...,oNo}. To get the attribute/relation embeddings, we first use the attribute/relation annotations from VG (Krishna et al., 2017) to train the attribute/relation classifiers to predict the labels. Then we use two learnable embedding layers to respectively transform these labels into the dense attribute/relation embeddings {a1, ...,aNa}/ {r1, ..., rNr}.
Given these original node embeddings, GNN will update each one by aggregating the neighbour embeddings. In the previous GNN-LSTM-based models, GNN is usually deployed by the FC layers, which aggregates the contexts by mapping the concatenated neighbour embeddings to the new one (Yao et al., 2018; Yang et al., 2020). However, in our TFSGC, since Transformer is applied as the backbone, we may more hope to use the basic building block of Transformer to deploy the GNN. Such a design principle has two advantages. First, it alleviates the coding implementation difficulty that we do not need to specify some additional GNN operations. Second, which is more important, when the GNN and the Transformer architecture are homogeneous, the whole model will be more easily trained. For example, we do not need to set different training strategies like the learning rate or optimizer to different sub-networks.
Since MHA (Eq. (1)) can learn the context knowledge between the embeddings (Vaswani et al., 2017), it can naturally be used to define the graph operation for aggregating the knowledge. We apply the following two steps to do this. Firstly, as shown in Fig. 2, we linearize the object, attribute, and relation embeddings into one sequence and add the learnable type embeddings as the linearized token set: U = {u1, ...,uN}:
Object: ui = oi + eo, 1 ≤ i ≤ No, Attribute: uNo+i = ai + ea, 1 ≤ i ≤ Na, Relation: uNo+Na+i = oi + er, 1 ≤ i ≤ Nr, (8) where eo/ea/er are respectively learnable type embeddings correspond to object/attribute/relation types and N = No + Na + Nr. For example, in Fig. 2, No/Na/Nr is 3/1/2, the objects o1:3 become u1:3, the attributes a1 becomes u4, and the relations r1:2 become u5:6.
After linearizing, the topological knowledge of the graph is lost, i.e., this token sequence does not show which two nodes are connected or not. To remedy such knowledge, we use a symmetric binary mask matrix M ∈ RN×N to all Transformer blocks to mask the attention weights of the un-
connected nodes to control whether two nodes are connected or not. If two nodes ui and uj are connected in the original scene graph, we set Mi,j = 1 and Mi,j = 0 otherwise. Specifically, the values of M are set as follows: 1) If oi has one attribute aj , we set Mi,j+No = 1, e.g., o2 (u2) and a1 (u4) in Fig. 2 are connected and M2,4 = 1. 2) If rk connects with oi and oj , we set Mi,k+No+Na = Mj,k+No+Na = 1, e.g., r1 (u5) connects o1 (u1) and o2 (u2) and M1,5 = M2,5 = 1. 3) All the object nodes are connected with each other since they are visual features that their contexts play a key role in captioning (Herdade et al., 2019). Thus ∀i, j ≤ No,Mi,j = 1. 4) Since the scene graph is an undirected graph, M should be symmetric: Mi,j = Mj,i.
After getting U and M , we can revise the Transformer encoder to get our MHA-GNN. Specifically, we use U as the input of the encoder defined in Eq.(3) and revise the Att operation in Eq. (1) as the following Masked Att operation:
Ai = Softmax(M ⊙ QW Q i (KW K i ) T
√ d
), (9)
where ⊙ denotes the element-wise product. In this way, the graph operation is defined by MHA. Specifically, for each node embedding, it is updated by weighted summing its neighbour embeddings, where the weights are from the attention heads Ai calculated by the Att operation in Eq. (1). During weighted summing, the binary matrix control whether two nodes are connected or not. Note that the edge type is implicitly embedded in Att operation due to the added node type embedding. For example, after adding node type embeddings eo and er to the object and relation embeddings o and r, respectively, the inner-product becomes*: (o+ eo) T (r + er) = o Tr + eTo r + e T r o+ e T o er, (10) where the right three terms are affected by the node type embedding. Thus, when the edge type changes (e.g., the object-relation edge changes to object-attribute edge), the corresponding node type embeddings also change (e.g., er changes to ea), which means Eq. (10) encodes the knowledge of edge types into the embeddings.
By stacking more such layers, the receptive field is increasing and thus each node can be updated by aggregating more neighbour embeddings, which naturally follows the design principle of GNN (Battaglia et al., 2018). The output graph
*For convenience, we omit the trainable matrices WQ,WK in Eq. (1) in this inner-product operation.
embedding set G are input to the decoder for captioning.

heading: 4.2 MOE-decoder
text: As mentioned before, a caption contains different kinds of words for describing diverse visual patterns, e.g., nouns/adjectives/verbs for objects/attributes/relations (Yang et al., 2019b), which suggests that different experts should be used to address diverse visual knowledge for generating the corresponding words. Motivated by this idea, we design an MOE-based (Jacobs et al., 1991; Du et al., 2022) language decoder to discriminate diverse graph embeddings by setting three encoder-decoder attention layers as different experts. As shown in Fig. 1 (c), the graph embeddings G = {g1, ..., gN} output from the MHAGNN can be naturally divided according to the original token types in the scene graph: object/attribute/relation sets Go = {g1, ..., gNo}/ Ga = {gNo+1, ..., gNo+Na}/Gr = {gNo+Na+1, ..., gN}. Then we only need to input them into the corresponding experts for discriminating them. Fig. 3 sketches the designed MOE-based decoder, which is got by revising the decoder defined in Eq. (4) as:
Input: XD,Go,Ga,Gr SA: X = MHA(Q = K = V = XD),
EXPO: Yo = MHA(Q = X,K = V = Go), EXPA: Ya = MHA(Q = X,K = V = Ga), EXPR: Yr = MHA(Q = X,K = V = Gr),
FFN: Zo,Za,Zr = FFN(Yo,Ya,Yr), SR: Z = SR(Zo,Za,Zr,X)
(11)
where EXPO, EXPA, and EXPR denote three different experts (encoder-decoder attentions) used to address object, attribute, and relation embeddings, respectively. They have the same structure while
with different parameters. Note that the input XD is the word embeddings of the partially generated captions and at t-th step, XD = {x1D, ...,xtD}. Then all the X/Zo/Za/Zr also contain t elements, e.g., Zo = {z1o , ...,zto}. Soft Router (SR) calculates an ensemble embedding z at each time step to construct the embedding set Z = {z1, ...,zt}. Specifically, for each element x/zo/za/zr in X/Zo/Za/Zr, a corresponding output z can be got†:
Input: x,zo,za,zr,
ATT: α = {αo, αa, αr}
= Softmax({xTzo,xTza,xTzr}) Output: z = αozo + αaza + αrzr,
(12)
where ATT operation calculates the soft routing weights, since x accumulates the context knowledge of the partially generated caption, it can help judge which kind of word should be generated at the next step. For example, if the last word of this partially generated caption is an adjective “black”, the next word is more like to be a noun and thus αo should be a large value for using more object embeddings instead of the other embeddings.

heading: 5 Experiments

heading: 5.1 Datasets, Metrics, and Implementation Details
text: Datasets. MSCOCO. We use MSCOCO (Lin et al., 2014) to validate our TFSGC. This dataset has 123,287 images and each one is labeled with 5 captions. We use two splits in the experiments: the offline Karpathy split (113,287/5,000/5,000 train/val/test images) and the Official online split (82,783/40,504/40,775 train/val/test images). Visual Genome (Krishna et al., 2017) provides scene graph annotations for training the scene graph parser. We follow (Yang et al., 2020) to filter the noisy dataset (e.g., lots of labels only appear a few times in the dataset) by removing the attribute/relation labels appearing less than 2000 times and use the remained 103/64 attribute/relation labels to train the attribute/relation classifiers. Implementation Details. In the experiments, we use three kinds of visual features to exhaustively compare to the other SOTA models, which are BUTD (Anderson et al., 2018), Patch (Liu et al., 2021), and VinVL (Zhang et al., 2021). During training and inference, for BUTD/Patch/VinVL, we respectively follow (Yang et al., 2020) and
†For convenience, we remove the superscript representing for different time steps of each embedding.
VinVL’s official parser‡ to parse SGs, where the latter is more powerful. For all the visual features, we set the batch size to 20 and use Adam (Kingma and Ba, 2014) as the optimizer. For BUTD/Patch/VinVL features, We sequentially use cross-entropy loss (Eq. (6)) and RL-based reward (Eq. (7)) to train the models 20/20/30 and 30/30/30 epochs. For BUTD/Patch/VinVL features, the learning rate used in the cross-entropy stage is initialized as 5e−4/5e−4/2e−5 and both decayed by 0.8 every 5 epochs, the learning rate used in the RL-reward stage is reset to 5e−5/2e−5/5e−6 and both decayed by 0.8 every 5 epochs. During inference, we use beam search where the beam size is 5. We evaluate the captions by CIDEr-D (Vedantam et al., 2015), BLEU (Papineni et al., 2002), METEOR(Banerjee and Lavie, 2005), ROUGE (Lin, 2004) and SPICE (Anderson et al., 2016).

heading: 5.2 Ablation Studies
text: To confirm the effectiveness of the proposed MHAGNN and MOE-decoder, we deploy exhaustive ablations as follows. Note that we use BUTD feature in this section. BASE: We directly use the classic Transformer architecture. SG: We incorporate the scene graphs into the Transformer by using the node embeddings without any graph operations. MLP-GNN: We apply MLP-based Graph Neural Network (Xu et al., 2019) for embedding the scene graphs. MHA-GNN w/o e: We apply the proposed MHA-GNN while do not use node type embedding. MHA-GNN: We apply the proposed MHA-GNN and keep the decoder unchanged as BASE. GNNFC:We remove the binary mask matrix M from TFSGC. MOE: We use the proposed MOE-decoder and do not use GNN but input the original node embeddings into the decoder. TFSGC: We apply the integral TFSGC.
Table 1 compares the similarity metrics of the ablation models. Firstly, we can find that the integral TFSGC achieves the highest scores, which confirms its effectiveness. Next, we respectively compare the ablation models to validate the effectiveness of the proposed MHA-GNN and MOEdecoder. By comparing MLP-GNN, SG, and BASE, it is easy to see that using GNN can gain more profits than only using node embeddings. Furthermore, we can find that MHA-GNN has higher CIDEr than MLP-GNN, which suggests that designing GNN by MHA is more powerful than by
‡https://github.com/microsoft/scene_graph_benchmark
MLP in the Transformer architecture. Next, to see whether discriminating the graph embeddings is beneficial or not, we can compare MHA-GNN with MHA-GNN w/o e and find that using node type embedding performs better. From the comparison between GNN-FC and TFSGC, it can be seen that when removing M , the graph becomes a fully-connected graph, which introduces more noises. Also, it can be seen that MOE and TFSGC respectively achieve better performances than SG and MHA-GNN, which validates the effectiveness of the MOE-decoder.
Besides evaluating these ablation models by similarities, in Table 2, we calculate the recalls of the words with different POS to evaluate the descriptiveness. Table 2 shows that the captions generated from TFSGC have the highest recalls, suggesting that TFSGC generates the most descriptive captions. Also, we can find that both the proposed MHA-GNN (MHA-GNN vs. MLP-GNN) and MOE-based decoder (MOE vs. SG) can boost the recalls, suggesting both of them improve the descriptiveness. Moreover, We use Stanford Parser to get the POS to train the route weights by the crossentropy loss, then the CIDEr of TFSGC boosts from 132.3 to 132.9, suggesting the advantage of using POS knowledge. Fig. 4 shows 4 examples of the captions generated from diverse models, where we can see that TFSGC generates more descriptive captions. BASE generates less descriptive captions since it does not use scene graphs and thus loses se-
mantic knowledge compared with SG. MHA-GNN does not use the soft router and thus can not select the most suitable experts for generating corresponding words, which may lose certain details. Also, we show that which expert is more responsible for the words with different POS, e.g., in Fig. 4 (b), the adjective “yellow” is generated by using more knowledge from the attribute expert.

heading: 5.3 Comparisons with SOTA
text: Recently, various SOTA captioning models with diverse settings have been proposed, including using different language decoders (LSTM, GRU, and Transformer), different visual features, and whether distilling knowledge from large-scale pretraining models (CLIP (Radford et al., 2021) or VInVL (Zhang et al., 2021)). In fairness, we compare our TFSGC with the models that also use Transformer-based decoder by three features: BUTD (Anderson et al., 2018), Patch (Liu et al., 2021), and VinVL (Zhang et al., 2021).
We compare with the following SOTA models: ETA (Li et al., 2019), ORT (Herdade et al., 2019), AoANet (Huang et al., 2019), M2 Transformer (Cornia et al., 2020), CATT (Yang et al., 2021b), APN (Yang et al., 2021a), DLCT (Luo et al., 2021), X-Transformer (Pan et al., 2020), Up-Down (Anderson et al., 2018), SGAE (Yang et al., 2019a), NG-SAN (Guo et al., 2020), PureT (Wang et al., 2022), and ViTCAP (Fang et al., 2022). Specifically, ETA and ORT are preliminary Transformer-based models; AoANet and X-Transformer design stronger attention mech-
anisms; CATT and M2 Transformer introduce additional memory networks; and APN exploits the hidden tree structures. We set the training batch size of TFSGC∗ to 50 as X-Transformer for fairly comparing. These methods use BUTD features, while ViTCAP and PureT use Patch features. RSTNet and ViTCAP distill knowledge from pre-trained vision-language BERTs. Note that in VinVL (Zhang et al., 2021), they use OSCAR (Li et al., 2020) as the captioner, while OSCAR is trained on 6.5 million image-text pairs. To fairly compare, we input VinVL feature into the classic Transformer to generate the captions, which is denoted as VinVL(Transformer). Note that we do not compare with extra large scale models trained by millions of image-text pairs or with excessively large parameter sizes, such as RSTNet (Zhou et al., 2020) and LEMON (Hu et al., 2022). All the results are shown in Table 3.
From Table 3 we can find that TFSGC almostly achieves the highest scores in different settings, i.e., achieving 132.3, 138.6, 139.5 CIDEr scores when using BUTD, Patch, and VinVL features, respectively. DLCT’s visual features extractor (Jiang et al., 2020) is stronger than TFSGC(BUTD) (Anderson et al., 2018) and thus DLCT is a little better than TFSGC(BUTD). Among these compared methods, although other SOTAs do not use scene graphs, they usually have some other training burdens. For example, APN and X-Linear apply more complex attention mechanisms and it requires more computation resource for well-training them, while our TFSGC only apply the simplest attention opera-
tion. Moreover, as detailed in Sec 4.1 of ViTCAP, it applies much more training data (9.9M image-text pairs from 4 datasets including VG) to pre-train a concept network to get more powerful discrete tags, while we only use one dataset VG to get scene graphs and achieve better performance, which suggests that connecting discrete tags into a graph is a useful strategy if the discrete tags is not very powerful. Moreover, TFSGC achieves comparable performances with LEMON (Hu et al., 2022) which uses 200 million image-text pairs and uses 12/24-layer, 768/1024-hidden units Transformers(base/huge). To sum up, the advantage of TFSGC is that it can effectively embed and discriminate the semantic knowledge from scene graphs to balance the (usu-
ally more) burden of using more training data or of training more complex networks.
We also submit the single model TFSGCS and 4-ensembled model TFSGCE trained by regionalbased features into the online server for testing, where the results are shown in Table 4. From this table, we can discover that both TFSGCS and TFSGCE have the highest CIDEr-D scores, which further confirm the effectiveness of the proposed TFSGC.

heading: 6 Conclusion
text: We proposed to transform the scene graph into captions (TFSGC) by a simple and homogeneous network. Specifically, we use MHA to design the GNN by linearizing the scene graph and remedying the lost topological knowledge with a binary mask matrix. Furthermore, we add learnable type embedding and design an MOE-based decoder to distinguish node embeddings for more descriptive captions. At last, we compared TFSGC with various SOTA models and demonstrated that our model can achieve comparable performances to some strong benchmarks.
Limitations
There are two major limitations of the proposed TFSGC. The first one is that the effectiveness of TFSGC depends on the quality of the scene graph. Since MSCOCO does not have SG annotations, we evaluate the parsers in Visual Genome: for BUTD/Patch and VinVL, the recall@50 of relation/attribute are respectively 65.2/68.4 and 73.4/76.6. We use VinVL’s SGs in TFSGC(BUTD) and CIDEr improves from 132.3 to 133.1, suggesting better SGs are beneficial. If the scene graph quality is poor, then TFSGC will not achieve good performance. When an incorrect node in the scene graph, it also affects the output of the caption. e.g., in Fig. 4 (e), the correct object label should be "surfboard" instead of "train". In this paper, we use Visual Genome, which contains abundant and useful scene graph annotations for parsing effective scene graphs, but current performance is not the best, and we will improve the scene graph parser based on the latest scene graph parsing methods in the future.
The second limitation of TFSGC is that if the visual features contain abundant attribute or relation knowledge, then the improvement of TFSGC compared with the classic Transformer will be weakened. For example, compared with the BUTD feature case where the relative improvement of CIDErD is 3.6 (TFSGC-BASE in Table 1), the VinVL feature is more powerful since it is trained by much more data samples with more semantic labels, thus the relative improvement is lower, which is 2.2 (TFSGC-VinVL(Transformer) in Table 3).

heading: Acknowledgements
text: This work is supported by National Key R&D Program of China (2018AAA0100104, 2018AAA0100100), National Science Foundation of China (62206048), Natural Science Foundation of Jiangsu Province (BK20220819, BK20211164), and Young Elite Scientists Sponsorship Program of Jiangsu Association for Science and Technology Tj-2022-027.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
The Limitation section is given in page 10.
7 A2. Did you discuss any potential risks of your work? This work deal with image captioning, which is a task that does not have potential moral risks.
3 A3. Do the abstract and introduction summarize the paper’s main claims? The Abstract and Introduction sections are given in the page 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.
3 B1. Did you cite the creators of artifacts you used? The creators of artifacts are cited in the “Datasets, Metrics, and Implementation Details” section in the page 6.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? When we use the artifacts, we cite them in the paper, for example, the metrics are cited in the
“Datasets, Metrics, and Implementation Details” section in the page 6.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We use the evaluation metrics like BLEU or CIDEr-D to measure the similarities between the generated and ground-truth captions, which is consistent with their intended use.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Such information is introduced in the Dataset section in the page 6.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In this paper, we use Visual Genome, which contains abundant and useful scene graph annotations for parsing effective scene graphs, but current performance is not the best, and we will improve the scene graph parser based on the latest scene graph parsing methods in the future. The second limitation of TFSGC is that if the visual features contain abundant attribute or relation knowledge, then the improvement of TFSGC compared with the classic Transformer will be weakened. For example, compared with the BUTD feature case where the relative improvement of CIDErD is 3. 6 (TFSGC-BASE in Table 1), the VinVL feature is more powerful since it is trained by much more data samples with more semantic labels, thus the relative improvement is lower, which is 2. 2 (TFSGC-VinVL(Transformer) in Table 3)