Improving Image Captioning via Predicting Structured Concepts
Authors: Ting Wang
Weidong Chen
Yuanhe Tian
Yan Song
Zhendong Mao
Venue: EMNLP 2023 Main (2023)
Keywords: Image captioning
GCN

Abstract: Having the difficulty of solving the semantic gap between images and texts for the image captioning task, conventional studies in this area paid some attention to treating semantic concepts as a bridge between the two modalities and improved captioning performance accordingly. Although promising results on concept prediction were obtained, the aforementioned studies normally ignore the relationship among concepts, which relies on not only objects in the image, but also word dependencies in the text, so that offers a considerable potential for improving the process of generating good descriptions. In this paper, we propose a structured concept predictor (SCP) to predict concepts and their structures, then we integrate them into captioning, so that enhance the contribution of visual signals in this task via concepts and further use their relations to distinguish cross-modal semantics for better description generation. Particularly, we design weighted graph convolutional networks (W-GCN) to depict concept relations driven by word dependencies, and then learns differentiated contributions from these concepts for following decoding process. Therefore, our approach captures potential relations among concepts and discriminatively learns different concepts, so that effectively facilitates image captioning with inherited information across modalities. Extensive experiments and their results demonstrate the effectiveness of our approach as well as each proposed module in this work.

Having the difficulty of solving the semantic gap between images and texts for the image captioning task, conventional studies this area paid some attention to treating semantic concepts as a bridge between the two modali- ties and improved captioning performance ac- cordingly. Although promising results on con- cept prediction were obtained, the aforemen- tioned studies normally ignore the relationship among concepts, which relies on not only ob- jects the image, but also word dependencies in the text, that offers considerable po- tential for improving the process of generating good descriptions. a structured concept predictor (SCP) to pre- dict concepts and their structures, then we in- tegrate them into captioning, as to enhance the contribution of visual signals in this task via concepts and further use their relations to distinguish cross-modal semanties for better description generation. Particularly, we de- sign weighted graph convolutional networks (W-GCN) to depict concept relations driven by word dependencies, and then learns differ- entiated contributions from these concepts for following decoding process. Therefore, our approach captures potential relations among concepts and discriminatively learns different concepts, so that effectively facilitates image captioning with inherited information across modalities. Extensive experiments and their results demonstrate the effectiveness of our approach as well as each proposed module in this work. Source code is available at: Improving Image Captioning via Predicting Structured Concepts
Ting Wang', Weidong Chen', Yuanhe Tian', Yan Song', Zhendong Mao University of Science and Technology of China
University of Washington wt1023email www.ustcc.... edu.cn, {chenweidong, Jenst.e@uecc.edie yhtian@uw. edu, ciksong@gmail. com
Abstract
Canrep Reddriie
Conanntdd Dec Dreiiroo
monkay
Three monkeys sit fencei eating food.
Sonce
Canarg
person walkinng outsiiee inithe rain while toiding green umbrelia.
rain
umbrod
Figure 1: Illustrations of motivation. Compared with integrating semantic concepts into the image cap- tioning framework, we find that the structured concepts helps reduce over-reliance on linguistic priors in lan- guage generation.
this paper, we propose
an interdisciplinary research of computer vision and natural language processing and has become a popular topic in recent years.
Current methods (Vinyals al., 2015; Anderson et al., 2018; Huang al., 2019; Cornia et al., 2020; Pan al., 2020; Luo al., 2021; Fang et al., 2022; Yang et al., 2022; Wu et al., 2022) usually follow an encoder-decoder framework, using a pre-traiied ob- ject detector/classifier as an encoder to mine visual information in an image, and then feeding it into an RNN- (Zaremba al.. 2014) or Transformer- (Vaswani al., 2017) based decoder for description prediction along with partially generated words. However, in most cases, the extracted visual infor- mation insufficient, even with the use of powerful visual feature extractors. This shortcoming makes the decoder rely too much on partially generated words to predict the remaining words to ensure the fluency of the generated description, that is, the model relics too much linguistic priors during decoding, and sometimes the resulted words does not related the image at all. In short, the ma- jor challenge that the image captioning facing now is that description generation relies too much on linguistic priors and has little to do with images.
1 Introduction
The image captioning task aims at generating a human-like description for a given image, normally requiring recognition and understanding of the con- tent in the image, including objects, attributes, and their relationships, etc. The task is regarded as
Corresponding author: Weidong Chen.

---

11
ractett
Deaa
persen hitting tennis bell with a tann's racket.
Hit
Motrlk Mordk
Festuu Preteo
taare
Image
(3-13
Feed
Adjacency Matrix
-h
Ace horm
Mdt horm
Mult- head
Quaris
Xx--
Xn
Encader
Weighted GCN
Figure 2: The overall framework our approach, including concept prediction, weighted graph convolutional networks (W-GCN) and language decoder. The last weight matrix refers the relations between concepts. Our model can be trained in end-to-end manner.
To this end, some studies(Wu al., 2022) add additional visual information to obtain strong vi- sual features and increase the contribution of vi- sion to generate captions. The other studies (Fang et al., 2022; Gan et al., 2017; Bin et al., 2017: Gao et al., 2020) have noticed the importance of seman- tic concepts based visual content, which can provide rich and accurate semantic understanding, accordingly helping semantic alignment and gen- erating reliable text descriptions. Thus the conven- tional pipeline of their studics to firstly predict the semantic concepts, and then send the predicted semantic concepts along with the visual features into the decoder to predict description. Although promising results obtained by these studies, they still ignore the relations among predicted con- cepts, which would not alleviate the overfitting on linguistic priors. For example, in one possible scenario that a model predicts the semantic words "baby" and "drink" from an image, the model is much more likely to predict the sentence as "baby + drink + milk" if it understands the relations between "baby" and "drink" is "baby - drink". On the contrary, "drink" is more likely be followed by "water" based on the linguistic prior if the model ignores the relations. Another example shown Figure 1, the predicted concept itself makes difficult for language generation to escape linguis-
tic priors. Meanwhile, is seen that concepts have relations, which are not only shown among objects in the image, but also in word dependencies in the text. Structured semantic concepts improve the performance of the model from the perspective of language generation.
In this paper, we propose structured cept predictor (SCP) improve image caption- ing, which not only integrates concept prediction into the end-to-end image captioning, but also pre- diets the structures of obtained concepts based on the word dependencies. Specifically, we propose weighted graph convolutional networks (W-GCN), with its input graph built based mutual informa- tion priors of all descriptions in unsupervised manner. The mutual information priors are the probability of co-occurrence of two words within a certain distance the same description, making the language generation no longer limited to local con- textual information and measuring the relationship between two concept words well. Meanwhile, ap- plying attention over the graph makes the concept features discriminatively learned, thereby reducing linguistic priors in captioning. Thus, structural- ize the semantic concepts and integrate them into the end-to-end image captioning, which keep the description generation associated with image all generation steps and reduce over-reliance on lin-

---

guistic priors. Experiments on the widely used MS COCO benchmark demonstrate the superiority of our approach over strong baselines, with qualitative analyses confirming its ability in better capturing structural relationship among semantic concepts, so that offers good interpretability for our method.
2.2 Concept Prediction
Most existing image captioning works leverage a pre-trained object detector to capture the seman- ties in an image, which are then directly fed into language decoder to generate the descriptive sen- tence. However, the semantic perception capability of the pre-trained detector is severely limited by pre-defined class labels. Based on the grid visual features, predicts concepts from ground-truth captions through a multi-label classification task, These concepts contain rich, comprehensive, accu- rate and refined semantic information, which can be decoded directly by multi-modal decoders, en- couraging the generation of relevant word at each decoding step, greatly benefiting the image caption- ing task.
2 The Approach
In this section, we describe our proposed SCP, which extracts concepts from a given image and utilizes graph convolutional networks to build topo- logical structures between concepts, so that the structured concepts can help generate descriptive text. Figure gives an overview of SCP, which consists of three main components.
2.1 Visual Feature Processing
Specifically, we predict the semantic concepts under the guidance of visual features through a set of concept queries Q. by leveraging N₂ Trans- semantic gap between images and text, and help se- mantic alignment, we extract visual features from images I by using the encoder of CLIP (Mokady et al., 2021) with the ResNet-101 (He et al., 2016) backbone, which has the ability to understand com- plex scenarios, and having excellent domain gen- eralization ability after pre-training with a large dataset. The process can be formulated as:
Extractor. To generate descriptions, the first step is to extract the visual features from images. Fol- lowing (Li et al., 2022b), in order to narrow the
former encoder blocks. The set of learnable queries learns the essential concepts within the images. Through the image interaction, each of our learn- able queries focuses on specific area of the image and learns the information (concepts) contained in the image. These concepts include objects, rela- tive positions between objects, actions, etc. Each Transformer block reinforces concept queries by interacting visual features V with object queries Q through a cross-attention mechanism. The whole process can be formulized as follows:
= fc(
(1)
where fo the visual extractor, and X is the image features.
H (0)
(5)
Encoder. Since the image feature is in the form of 2D, we first flatten X into a sequence {I1, X2, ..... Fs}. H, E Rd, where 2, are patch fea- tures and d is the size of the feature vector. Then we employ N₂ Transformer encoder blocks to fur- sequence. Outputs
H LN(MHA(H V. H ") (6)
where !D indicates the output of the I-th middle hidden layer and the superscript indicates the num- ber of layer. Thus, the output of the Transformer encoder 11 is the output of the last Transformer layer.
ther encode image features are the hidden states encoded from the input fea- tures X extracted from the visual extractor. The whole process can be formulized as follows:
We then feed the output of the last block into a multi-linear perception network to get concept features C::
H = X
C - MM
(7)
H MHA (((-1), H( I1 ! + HI! 1)
(3)
where MLP is the multi-linea perception network with the sigmoid activation.
(4)
where LN is the layer normalization, MHA stands for multi-head attention. H7.") indicates the output of the 1-th middle hidden layer and the superscript indicates the number of layer. In particular, H. is the output of the Transformer encoder. For sim- plicity, let V. denote H.N, v in the rest of paper.
During training, following (Fang et al., 2022), describe it multi-label classification prob- lem. Due to the imbalance of the distribution of concepts, we use asymmetric loss (Ben-Baruch et al., 2020), which can handle the sample imbal- ance problem of multi-label classification tasks

---

well. Asymmetric loss is calculated for concept prediction:
based on the word dependencies extracted from a pre-constructed n-gram lexicon D.
Specifically, we count the frequency of the oc- currences of each word and the frequency of simul- taneous occurrence of any two words within Ni word distance (considering the order) in all sen- tences of the training set. We regard two words within Nr. distance as they have word dependency. Then we calculate the Pointwise Mutual Informa- tion (PMI) score of any two words wi, w2 by the following formula and set a threshold to determine if they are strongly correlated.
L. = as= Ya)
(8)
where Ye denote the visual concept of the ground- truth sentence that corresponds to the concept vo- cabulary (The details of building concept vocabu- lary are introduced in $3.2).
2.3 Weighted Graph Convolutional Network
After obtaining the enriched semantics derived from concept predictor, the most typical way to predict descriptions to directly feed the semantic features, which is obtained by concept predictor, into the RNN/Trasformer-- based language decoder. However, in this way, the language decoder overly relies on the language priors to generate captions, since those concepts are treated independently, and their features are learned independently. A straght- forward example is illustrated in the introduction section.
p WT w2)
PMI(W1; (22) log pl wm (w2))
(9)
where p(w1), ), pl (2)]is the probability of w1, we in the training set, p(wr, w2) is the probability that both wi and w2 are within NE word distance.
We store all strongly correlated word pairs in a word lexicon and refer to to build the graph. If the concept represented by the two nodes in the graph can be found in the lexicon D, then the cor- responding element of its adjacency matrix is ini- tialized correlated, which is set to 1. Otherwise, setting the value to O.
To this end, we propose to construct graph for these concepts, explore the relationship between them by Weighted Graph Convolutional Networks (W-GCN), and obtain structured concepts. Struc- ture concepts learns to estimate the linguistic rela- tive position of semantic word pairs, thereby allo- cating all the semantic words in potential linguistic order as humans. In doing so, the output sequence of structured semantic concepts serve as additional visually-grow grounded language priors, which encour- age the visual contribution in generation.
2.3.2 The Weighted GCN
Based on the adjacency matrix, the W-GCN mod- ule of the L layers concepts. Considering that the contribution of dif- ferent gj to 9i may be different, we further apply the attention mechanism to the adjacency matrix, replacing aij with the weights aiy. For each g, and all its related Jj. we calculate weight auj for the concept pair. In particular, at the 1-th layer, for each Is, all the 9j associated with it can be calculated:
learn from all the input
Concretely, the nodes of the graph represent con- cepts G {g1 92, .... ). and the edges repre- sent the relationship between nodes gi and 9j for Vi,j E {1, 2, .... k), which can be represented by an adjacency matrix A. In A, Oij = 1, if there is an edge between 9i and gj or = j. otherwise
asj exp(h (1-1) (1 (
h (1-1))
(1) 11
pos
0ij =0. 0.
Σj aij exp(h! (1-1) w(ll) h1
(1-1)
(10)
2.3.1 Graph Construction
n h.00 o (LN Σ J=1
(w(()) - 11-11) + b(0))) (11)
As the obtained semantic concepts cannot form a complete sentence, our method cannot leverage existing dependency parsers to estimate their rela- tions. Without such parser, we need an alterna- tive way to find satisfied word pairs to build initial graphs in our W-GCN, which equivalent to build the initial adjacency matrix A. Inspired by the studies (Tian et al., 2020) which leverage chunks (n-grams) as additional features to carry contex- tual information, we propose to construct the graph
where W-(() a trainable parameter, it can model the position relationship between Si and gy (three choices: wx(h) w-(!) ((()) Left right: setf). h 1) is the hid- den vector from layer 1- 1, W()) and ((0) are the trainable matrices and biases of the W-GCN at layer I, LN is layer normalization, and a is the ReLU activation function.

---

Finally, we take the output (L) of the L-th layer the structured concept feature C and feed into the language decoder, which helps to establish syntactic relationships and dependencies of texts, thereby generating more accurate text descriptions.
METEOR (Denkowski and Lavie, 2014). ROUGE (Lin, 2004), CIDEr (Vedantam et al., 2015), and SPICE (Anderson et al., 2016).
3.2 Implementation Details
Our feature extractor is CLIP (Mokady et al., 2021) with the ResNet-101 (He et al.. 2016) backbone and the dimension of the grid visual feature is 2048, Following previous work (Li et al., 2022b), to build concept vocabulary, we filter out low-frequency words and convert all uppercase letters lower- case letters to all caption descriptions. Thus, a con- cept vocabulary that containing 906 words is con- structed. The word distance in the pre-constructed lexicon D is 3. The number of layers in Weighted GCN is set to 2. The Transformer block in the Feature Encoder Module, the Concept Prediction Module and the Language Prediction Module are 3 layers, 6 layers, and 6 layers, respectively. The size of the hidden state features is set to 512. The query size is set to 17. / is set to in this work. Our code is developed based on the COS-Net
2.4 Language Decoder
With the enriched visual tokens V and the position- aware semantic tokens Ö from W-GCN, we integrate them into the Transformer-based de- coder for sentence generation. Formally, let Yee = (wo, W1,..., WT-1) denote the description (T: word number) of the input image The sen- tence decoder takes each word as input and learns to predict the next word ruto-regressively condi- tioned on V and C. The formulations are
H: = MHA(U; V, V) + MMH (w,, Õ,C) (12)
yi+1 IN(H, + wx)
(13)
where Vi+1 is the (i + 1)th word of the pre- dicted sentence, and H; is the hidden state. Y=
[y1 1/2 ..., yT] is the predicted sentence. The loss function of captioning can be defined as:
The model is trained using a typical two-stage training method. In the first stage, we utilize Adam optimizer and cross-entropy loss with learn- ing rate of 0.0005 and take about one hour per epoch. In the second stage, the self-critical se- quence training strategy is used to further optimize the CIDEr scoring model, and the learning rate is set to 0.00005, which takes about 4 hours per epoch. In inference, the beam size is set to 3. The number of parameters our model used is 20M. All experiments are conducted single RTX 3090.
T
Coop = Σ Yz:
(14)
where CE is the cross-entropy loss. Thus, the total loss is the combination of visual concept prediction loss and the language prediction loss:
L - Leup + B Le-
(15)
where 3 is the trol the balance of two losses. To this end, our method can be trained in an end-to-end manner, which is kind to training and faster inference speed.
which aims to con-
Results and Analysis
4.1 Main Results
Our main results are shown in Table 1 and Table 2. All compared methods can be briefly grouped into two kinds. Firstly, the conventional methods, c.g.. Up-Down, M2 Transformer, X- X-Transformer, which utilize the pre-trained Faster R-CNN (backbone: ResNet- 101) to extract visual inputs. The second kind approaches, e.g. CTX+M2, which take the strong CLIP (backbone: ResNet-101) grid features as visual inputs. Compared with these previous studies, the results of our method have improved in each evaluation metrics, especially in CIDEr. Specifically, our method is 11.4 higher than the classic GCN-LSTM method (GCN-based model),
3 Experiment Settings
3.1 Datasets and Metrics
Our experiments are conducted on the MS COCO (Lin et al., 2014), which is the most popular image captioning benchmark dataset. It consists of more than 120,000 images, and each image is equipped with five human-annotated descriptions. We fol- low Karpathy's split, which divides 5,000 images for validation, 5,000 images for testing, and the rest for training (Karpathy and Fei-Fei, 2015). For fair comparison with other techniques, we lever- age pycocoevalcap package to calculate five eval- uation metrics: BLEU-N (Papineni et al., 2002),
'https: //gi thub. com/YehL. //madaaaooaarrreaaaaa ptpenaaleerrree///eeee master/configst wwafffigsmmaaae hts///maeeeeae waaaamaaaaettiieeeeeeeeee caption/cossee

---

Cross Entropy
Method
BLEU-1
BLEU-4
METEOR
ROUGE
CIDEr
SPICE
Up-Down(Anderrsn et al., 2018)
77.2
36.2
27.0
56.4
113.5
20.3
GCN-LSTM(YYao et al., 2018)
77.3
36.8
27.9
57.0
116.3
20.9
AoANei(Huang et al., 2019)
77.4
37.2
28.4
57.5
119.8
21.3
X-Transformer(Pan al., 2020)
77.3
37.0
28.7
57.5
120.0
21.8
ViTCAP (Pang et al., 2022)
35.7
28.8
57.6
121.8
22.1
(Ours)
78.3
38.6
29.3
58.5
125.3
22.4
Table Comparison with the state-of-the-arr methods on COCO Karpathy test split in the first stage, namely Cross Entropy.
CIDEr Optimization METEOR
Method
BLEU-1
BLEU-4
ROUGE
CIDEr
SPICE
Up-Down(Anderson al., 2018)
79.8
36.3
27.7
56.9
120.1
21.4
GCN-LSTM(Yao al., 2018)
80.5
38.2
28.5
58.3
127.6
22.0
AoANet(Huung et al., 2019)
80.2
38.9
29.2
58.8
129.8
22.4
M2 TransformerrCormiaa et al., 2020)
80.8
39.1
29.2
58.6
131.2
22.6
X- Traansforrer(Pan et al., 2020)
80.9
39.7
29.5
59.1
132.8
23.4
RSTNet (Zhang et al., 2021b)
81.1
39.3
29.4
58.8
133.3
23.0
DLCT (Luo et al., 2021)
81.4
39.8
29.5
59.1
133.8
23.0
ReFormer (Yang et al., 2022)
82.3
39.8
29.7
59.8
131.9
23.0
ViTCAP (Fang et al., 2022)
40.1
29.4
59.4
133.1
23.0
CTX+M2 (Kuo and Kira, 2022)
81.5
39.7
30.0
59.5
135.9
23.7
DIFNet (Wu al., 2022)
81.7
40.0
29.7
59.4
136.2
23.2
Ours
82.6
41.5
30.2
60.2
139.0
24.2
Table 2: Comparison with the state-of oftte-ofthhearr the-art methods on COCO Karpathy test split in the second stage, namely Cider Optimization.
and 7.1 higher than the recent ReFormer method. This due our approach considering both visual and textual semanties and finding ways to close the semantic gap between them. In the CP module, the extraction concept distills the textual information. And we use attention map convolutional networks in the W-GCN module to explore the structural re- lationships between concepts, which has been com- pletely ignored in previous studies. It is important that when initializing the graph, fully consider the context information and firmly glue the strongly related word pairs together, so that the generated sentences are more human-like We further test the running speed with other state-of-the-ar meth- ods. (Nguyen et al., 2022) runs at 138 ms/image, (Cornia et al., 2020) runs at 178 ms/image, and the pre-trained based method (Zhang et al., 2021a) runs at 542 ms/image. Our method achieves 214 ms/image running speed, which illustrates the ef- fectiveness of our method. Meanwhile, the results show that the proposed W-GCN module and CP
module have little effect the running speed.
4.2 Qualitative Results
To qualitatively illustrate the effectiveness of proposed approach and make the results more vincing, we present and analysis some qualitative results here. Firstly, some examples are shown Figure 3, where GT is the ground-tran -ruth sentence la- beled by human and Ours the predicted result of our model. It can be seen that our model accurately describes the content of the image, and the words appropriate, logical, and elegant, reaching the human level. It captures the key semantic words, and generated sentence follow the same patterns the GT.
In addition, present structured graph of the concept to illustrate the value of the W-GCN mod- ule in Figure 4. Nodes represent concepts, and the edges indicate the strength of the relationship between the two concepts. It can be seen that, first, our method can predict semantic concepts

---

Image
Predicted Strunttrred Cancapia
Captioeus
couple of people sitting on bench nest to dog.
people
sptiing
OURS:
Two people sitting on bench with white dog.
takes snawe his cell man taking picture mountains with phore.
younn boy feeding
banch
young boy feading giraffe
lnttl
GT. A young child sitting top the khchen counter.
coting
stting
OURS: a little girl sitting kitchen counter eating food.
GT: A cat that is laying down couch.
CT: woman walking holding a pinkk umbrella near train. OURS: woman walking with umbrella
food
OURS: laying fiving
train.
The qualitative results of our approach. Our approach is good at capturing the relations between semantic concepts, improving the preformance.
Figure 3:
Figure 4: Visualization results of the procedure of our method, including the structured concepts prediction. It not difficult to find that the structured concepts we predict are of good quality.
well, including key nouns, verbs and adjectives in the sentence. Meanwhile, in the absence of pervision information, our W-GCN can capture the relations between semantic concepts, includ- ing adjectives pointing to the correct nouns, and nouns pointing to the correct verbs. For example, in the second case, our model can understand the "little->girl", "girl->eatting" and "catting+food" The predicted structured concepts further strength- sentence generation.
"Baseline + CP+ CP+WGGOO which means, matter whether we learn the relations between concepts not, our concept predictions
accu- rate capture the semantic words of the sentence. The accuracy of this keyword prediction will lead to a good result BLEU-1 metric. Thus, there is not much difference BLEU-1 metric hetween the two methods. But the results of other metrics can show the superiority of our W-GCN.
4.3 Ablation Studies
4.3.2 Discussion on Graph Construction
In order to explore the effect of each proposed mod- ule, we perform series of ablation experiments. For the Weighted GCN Module, we also conduct several experiments to verify the effect of different graph construction on the results. All the results are recorded in Table 3.
As aforementioned, graph construction is equiv- alent the initial the adjacency matrix, and we conduct a varicty of experiments to verify the ef- feet of the initial graph constructed by our word dependencies.
"Random" indicates that when initializing the graph, the interrelationships among concepts are ignored, and the corresponding adjacency matrix is randomly generated 0-1 matrix. "1-for-all" indi- cates that when the graph is initialized, all concepts are considered to related, and the elements the adjacency matrix are all set to 1. "MLP" learns the relationship between concepts. Each concept is affected by other concepts to different degrees, which is learned by MLP. "Threshold-N" means that the initial graph is built by the PMI scores of word dependencies, where N is the threshold. For all concepts of an image, we calculate the PMI score between pairs. the PMT score greater than the threshold, the corresponding element the adjacency matrix is set to 1, otherwise to
4.3.1 Effect of the Proposed Modules
"Baseline" is the simple Transformer encoder- decoder structure. "Baseline- CP" represents the "Baseline" combine with the concept prediction module. "Baselin++CP-WGCNOOO best model, which integrate W-GCN on the basis of "Baseline+tCP"
is our
From the results, it can be seen that, first, the two proposed modules "concept prediction" and "Weighted GCN" are both effective and greatly im- prove the captioning performance. When compar- ing BLEU-1 metric, we have an interesting obser- vation that "Baseline CP" is slightly worse than
Since our W-GCN is built upon concept prediction, we cannot conduct -Baseline--WWeeen W-GCN" experiment.

---

Discussion
Method
B-1
B-4
M
R
C
S
Baseline
77.7
37.7
28.8
57.8
121.7
21.8
Proposed Modules
CC
78.2
38.0
29.0
58.0
123.4
22.1
BasciineCCP-WGGNNOOrr)
78.3
38.6
29.3
58.5
125.3
22.4
Random
78.1
38.0
29.2
58.1
123.6
22.2
1-for-all
78.6
38.4
29.2
58.2
124.4
22.4
MLP
77.7
38.1
29.1
58.0
123.7
22.1
Graph
Threshold-0.1
78.5
38.6
29.2
58.3
124.4
22.2
Construction
Threshold-0.3
78.5
38.4
29.1
58.4
124.1
22.1
Threshold-O.5(Ours)))
78.3
38.6
29.3
58.5
125.3
22.4
Threshold-0.7
78.6
39.0
29.2
58.2
124.6
22.3
Table 3: The results of ablation studies (Cross Entropy). Here, B-N, M, R, C, and represent BLEU-N, METEOR. ROUGE, CIDER, SPICE, respectively.
The results are in Table 3, and from the results, we have several observations. First of all, the re- sults of "Random" and "MLP" the worst. can be seen that the relations between concepts is not random and should be established based on some word dependencies. Secondly, the effect of "I-for-all" is not good enough. It considers all concepts to be related. Obviously, this way of han- dling doesn't t make sense, and there is no relations between some semantic words, such "red" and "grass". Last, for word dependencies, it can be seen that the effect reaches best when the threshold is set to 0.5.
tributions to high-quality descriptions. (Chen et al., 2020) encode image into Abstract Scene Graphs, which represent semantics and structure at fine- grained level, thus generating detailed descriptions. (Zhang et al., 2022) not only construct a multi- modal relational graph for images, but also encodes relational graphs for all sentences in the dataset to fully capture language features. Then a cascaded GAN is developed to achieve cross-domain align- ment of image and text pairs, which are fed into the decoder. (Shi et al., 2020) explore the seman- tics available in captions and construct caption- guided visual relationship graph, and leverage to enhance image representation and caption genera- tion. The Concept Token Network introduced to predict concept words based on visual information. These concepts contain rich semantic information, which benefits the image captioning task (Fang et al., 2022; Li et al., 2022b). (Zeng et al., 2022) try capture the hierarchical semantic structure in the text space, helps visual features carry semantics and generates finer-grained and more reasonable phrases and collocations. To effectively extract contextual information, constructing graph is pro- posed to model the relations between words (Tian et al.. 2020).
5 Related Work
Image Captioning is a practical task that has been widely concerned and studied. There have been many studies before (Liu et al., 2019; Guo et al., 2020; Chen et al., 2021; Song et al., 2023b,a; Liu al., 2021; Nie et al., 2021; Tu et al., 2021). In- spired by the Transformer structure in NLP, the Transformer-basee encoder-decoder structure has recently become mainstream, where the interaction between multi-modal information can be enhanced. (Herdade et al., 2019; Li et al., 2022a) integrates the spatial relationships between objects through geo- metric attention based on the Transformer structure. (Nguyen et al., 2022) set of learnable object queries to extract semanties from multi-scale fea- tures, and feed them into Transformer generate captions.
Conclusion
In this paper, explore the significance of con- cepts and their structures for accurately describing images the image captioning task. To predict concepts and their structures, we propose a struc- tured concept predictor (SCP), so that enhance the
The previous researches have shown that explor- ing semantic concepts and their relations make con-

---

contribution of visual signals and further use their relations to distinguish cross-modal semantics for better description generation. Particularly, we de- sign weighted graph convolutional networks (W- GCN) to depict concept relations driven by word dependencies, and then learns differentiated con- tributions from these concepts for following de- coding process. Therefore, our approach captures potential relations among concepts and discrimina- tively learns different concepts, so that effectively facilitates image captioning with inherited infor- mation across modalities. Extensive experiments indicate that our approach achieves competing per- formances on MS COCO benchmark. Qualitative analyses confirm its ability in better capturing struc- tural relationship among semantic concepts, so that offers good interpretability for the proposed model.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086.
Emanuel Ben-Baruch, Tal Ridnik, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. 2020. Asymmetric loss for multi-lahel classification. arXiv preprint arXiv:2009.1411
Yi Bin, Yang Yang, Jie Zhou, Zi Huang, and Heng Tao Shen. 2017. Adaptively attending to visual attributes and linguistic knowledge for captioning. In Proceed- ings of the 25th ACM international conference on multimedia, pages 1345-1353.
Long Chen, Zhihong Jiang, Jun Xiao, and Wei Liu. 2021. Human-like controllable image captioning with verb-specific semantic roles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16846-16856.
Limitations
When predicting structured concepts in this paper, there is premise, that is, predict concepts at first. However, when constructing the semantic concept vocabulary in this paper, we follow the existing work (Li et al., 2022b). The semantic concept vo- cabulary is obtained only by analyzing the corpus and the fireuuency of words. In future work, can consider semantic concepts based on n-grams the corpus. Secondly, we find that in our experiment, for some samples, our model can only predict cou- ple concepts, which make the structured concept prediction meaningless. In future work, we aim to establish better cooss-modal correspondence, so that more concepts can be predicted.
Shizhe Chen, Qin Jin, Peng Wang, and Qi Wu. 2020. Say as you wish: Fine-grained control of image cap- tion generation with abstract scene graphs. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9962-9977 1971..
Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-memory trans- former for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages 10578-10587.
Michael Denkowski and Alon Lavie 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376-380.
Ethics Statement
Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lin Liang. Zhe Gan, Lijuan Wang. Yezhou Yang, and Zicheng Liu. 2022. Injecting semantic concepts into end- ondend image captioning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pat- tern Recognition, pages 18009-18019.
This paper proposed new approach for Image Captioning and use existing datasets for evaluation. We do not foresee any ethic issues of this study.
Acknowledgements
Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu. Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. 2017. Semantic compositional networks for visual captioning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 5630-5639.
This work is supported by the National Natural Sci- ence Foundation of China under Grant 62302474 and the National Science Fund for Excellent Young Scholars under Grant 62222212.
Lianli Gao, Xuanhan Wang, Jingkuan Song, and Yang Liu. 2020. Fused gru with semantic-temporal atten- tion for video captioning. Neurocomputing, 395:222- 228.
References
Peter Anderson. Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Proceedings of the Euro- pean conference on computer vision (ECCV). pages 382-398. Springer.
Longteng Guo, Jing Liu, Xinxin Zhu, Peng Yao, Shichen Lu, and Hanqing Lu. 2020. Normalized and geometry-awan self-attention network for image

---

captioning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 10327-10336.
Fenglin Liu, Xuancheng Ren, Yuanxin Liu, Kai Lei, and Xu Sun. 2019. Exploring and distilling cross- modal information for image captioning. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 5095-5101.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Yunpeng Luo, Jiayi Ji, Xiaoshuai Sun, Liujuan Cao, Yongjian Wu, Feiyue Huang, Chia-Wen Lin, and Rongrong Ji. 2021. Dual-level collaborative trans- former for image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- 35, pages 2286-2293.
Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. 2019. Image captioning: Transforming objects into words. Advances in Neural Information Processing Systems, 32.
Ron Mokady, Amir Hertz, and Amit H Bermano. 2021. Clipeap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734
Lun Huang, Wenmin Wang, Jie Chen, and Xiao- Yong Wei. 2019. Attention on attention for image caption- ing. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4634 4643.
Van-Quang Nguyen, Masanori Suganuma, and Takayuki Okatani. 2022. Grit: Faster and better image caption- ing transformer using dual visual features. In Pro- ceedings of the European conference on computer vision, pages 167-184. Springer.
Andrej Karpathy and Li Fei-Fei. 2015, Deep visual- semantic alignments for generating image descrip- tions. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 1128-3137.
Weizhi Nie, Jiesi Li. Ning Xu, An-An Liu, Xuanya Li, and Yongdong Zhang. 2021. Triangle-rewe reward reinforcement learning: a visual-lingusstc semantic alignment for image captioning. In Proceedings of the 29th ACM international conference on multime- dia, pages 45 1014588.
Chia- Wen Kuo and Zsolt Kira. 2022. Beyond a pre- trained object detector: Cross-modal textual and vi- sual context for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17969-17979.
Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. 2020. X-linear attention networks for image captioning. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 10971- 10980.
Jingyu Li, Zhendong Mao, Shancheng Fang. and Hao Li. 2022a. Er-san: Enhanced-udaptive relation self- attention network for image captioning. In Proceed- ings of the Thirty-Fire First International Joint Confer- ence on Artificial Intelligence, IJCAI-22. Rd. by Lud De Raedt. Main Track. International Joint Confer- ences on Artificial Intelligence Organization, pages 1081-1087.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Anmual Meeting of the Association for Computational Linguistics, pages 311-318.
Yehao Li, Yingwei Pan. Ting Yao, and Tao Mei. 2022b. Comprehending and ordering semantics for image captioning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 17990-17999.
Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu. 2020. Improving image captioning with better use of caption. In Annual Meeting of the Association for Computational Linguistics.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summarics. In Text summarization branches ouf, pages 74-81.
Peipei Song, Dan Guo, Jun Cheng, and Meng Wang. 2023a. Contextual attention network for emotional video captioning, IEEE Transactions on Multimedia, 25: 18518188887
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Proceedings of the European conference computer vision, pages 740- 755. Springer.
Peipei Song, Dan Guo, Jinxing Zhou, Mingliang Xu, and Meng Wang. 2023b. Memorial gan with joint semantic optimization for unpaired image caption- ing. IEEE Transactions Cybernetics, 53(7):4888- 4399.
An-An Liu, Yingchen Zhai, Ning Xu, Weizhi Nie, Wen- hui Li, and Yongdong Zhang. 2021. Region-aware image captioning via interaction learning. IEEE Transactions on Circuits and Systems for Video Tech- nology, 32(6))368533666
Yuanhe Tian, Yan Song, and Fei Xia. 2020. Supertag- ging combinatory categorial grammar with attentive graph convolutional networks. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 6037-6044.

---

Yunbin Tu, Liang Li, Chenggang Yan. Shengxiang Gao. and Zhengtao Yu. 2021. Reenet: Relation-embedee representation reconstruction network for change cap- tioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2319-9329.
Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Ron- grong Ji. 202 Rstnet: Captioning with adaptive attention on visual and non-visual words. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15465-15474.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention all need. Advances in neural information processing systems, 30.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-basse image descrip- tion evaluation. In Proceedings of the IEEE/CVF con- ference computer vision and patlern recognition, pages 4566-4575.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 3156-3164.
Mingrui Wu, Xuying Zhang, Xiaoshuai Sun, Yiyi Zhou, Chao Chen, Jiaxin Gu, Xing Sun, and Rongrong Ji. 2022. Difnet: Boosting visual information flow for image captioning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pages 10020-8009,,
Xuewen Yang. Yingru Liu, and Xin Wang. 2022. Re- former: The relational transformer for image caption- ing. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5398-5406.
Ting Yao, Yingwei Pan, Yehao Li. and Tao Mei. 2018. Exploring visual relationship for image captioning. In Proceedings of the European conference com- puter vision (ECCV), pages 684-699.
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint
Pengpeng Zeng. Jinkuan Zhu, Jingkuan Song, and Lianli Gao. 2022. Progressive tree-structured pro- totype network for end-to-end image captioning. In Proceedings of the 30th ACM International Confer- ence Multimedia, pages 5210-5218.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang. Lijuan Wang, Yejin Choi, and Jian- feng Gao, 2021a. Vinvl: Revisiting visual represen- tations in vision- languaee models. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 5579-5588.
Wenqiao Zhang, Haochen Shi, Jiannan Guo, Shengyu Zhang, Qingpeng Cai, Juncheng Li. Sihui Luo, and Yueting Zhuang. 2022. Magic: Multimodal rela- tional graph adversarial inference for diverse and unpaired text-based image captioning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3335-3343.


Reviews:
[{"review_id": "3Fpe3pURp2", "reviewer": "Reviewer_uJ4d", "summary": "", "questions": "1. Do you have any intuitions as to what kind of content would benefit most from your proposed new technique?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your high evaluation of our paper.\n\nTo the question:\n\nIf describing an image requires multiple-step reasoning between concepts, our method works better since our method can grasp the relations between concepts. For example, if the predicted structured concepts are “rain->man->hold”, the most likely words behind for our method would be “umbrella”. Otherwise, the likely following word of the \"hold\" would be \"on\" for other methods. Meanwhile, if the method needs to fill the blank between the words “man” and “hold”, for our method, it is very likely that no trivial words will be filled. This is because our method leverages prior knowledge of point-wise mutual information (PMI) of any concepts. If the prior knowledge tells the model that there are no words between “man” and “hold”, the predicted sentence would not have words between “man” and “hold”.  In that case, our method also works well.\n\nTo the writing typos:\n\nThanks for pointing out our writing typos. We will carefully check the paper, and make sure there are no writing typos in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Solid literature review.\n2. Insightul and sound new approach to incorporation of concept structures worked out thoroughly.\n3. Clear improvement over the state of the art.\n4. Well written paper.", "reasons_to_reject": "1. No significant weaknesses.", "questions_to_authors": "1. Do you have any intuitions as to what kind of content would benefit most from your proposed new technique?", "paper_topic_and_main_contributions": "This paper proposes incorporation of concept structures that capture relationships between concepts through a weighted graph convolutional network, into image captioning. The authors build a convincing framework using among others a mutual information based approach, and demonstrate significant advancement of the state of the art in image captioning across multiple evaluation criteria or measures.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing references."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typo line 017 replace “so that enhance” with “so as to enhance”\nLine 225 please replace “an stragtforward” with “a straightforward”\nLine 234 please replace “Potiential” with “Potential”\n\nthese are typos I noticed. I have a feeling I might have missed some. Please go through carefully and revise."}}, {"review_id": "ol1Qh1L9v2", "reviewer": "Reviewer_hF5e", "summary": "", "questions": "1. In the process of predicting concepts and their structures, what challenges did you encounter? And how did you mitigate them?\n2. How does your method compare to other existing techniques, not just in terms of performance but also in terms of computational efficiency and complexity?\n3. Can the SCP be modified or adapted for tasks other than image captioning? If yes, what potential challenges do you foresee?\n4. In implementing the W-GCN, how did you decide upon the weighting mechanism? Was there an iterative process or any fine-tuning involved?\n5. It would be insightful to understand more about how you see the future developments or potential extensions of this method. Are there any immediate enhancements you're considering?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your carefully reviewing and detailed suggestions. It would improve the quality of our paper a lot.\n\nTo the Rejections:\n\nTo R1:\n\nThank you for your suggestions. We will cite and discuss these two papers. \n\nBesides, our method is different from the scene graph-based approach. The scene graph is established based on the predicting visual objects and their relationship. The prediction may make errors since images vary and their object relations vary. Meanwhile, the scene graph also has a gap between language descriptions.\n\nFrom the perspective of natural language processing, our concept relationship is learned based on concept word dependencies, which can more accurately capture the concept relationship, and structured concepts have no gap between language descriptions. Thus, our method has a good performance.\n\nTo R2:\n\nThanks for your kind suggestions. We tested the running speed of our method, which is 214 ms/image. It is acceptable. We also ran the speed of some comparison methods, which are listed as follows:\n\nGRIT (ECCV 2022): 138 ms/image\n\nM2 Transformer: 178 ms/image\n\nVin VL_large: 542 ms/image\n\nThe GCN and concept predictor have little influence on the running speed of our method. Our method is still significantly faster than the pre-trained based method. We will add the discussion in the revised paper.\n\nTo R3:\n\nThanks for your suggestions. Our method can indeed be expanded on tasks such as VQA, but this paper focuses on image captioning tasks and their challenges, and the modified method in VQA would be slightly different. In the revised paper, we will add more discussion of the scalability of our method, and in future work, we will add more experiments of different tasks to explore the scalability of the method.\n\nOn the VQA task, we can capture the structured concepts of the question (Q) and the structured concepts of the image (V), and then, we can leverage the multi-modal graph convolutional networks to merge the two graphs to generate answers (A) better. Slightly different from our current method, which only uses structured concepts of the image in captioning.\n\nTo R4:\n\nThanks for your suggestions.\n\nI. We discuss the limitations at the end of the submitted paper. We will substitute it into the error analysis in the revised paper. We find that in our experiment, for some samples, our model can only predict a couple of concepts, which makes the structured concept prediction meaningless.\n\nII. We have done the online test of our method. The result is slightly lower than the offline test, which is also in line with the results on COS-Net (CVPR 2022). The online test results are lower than the offline test in COS-Net. It is worth noting that our code is developed based on COS-Net. \n\nMeanwhile, our online results are still better than the baselines that we compared in the submitted paper, such as COS-Net (CVPR 2022), ReFormer (ACM-MM 2022), and multiple baselines.\n\nThe results of our method are listed as follows:\ncider-c5: 1.368; cider-c40: 1.381\n\nWe will add the results to the revised paper, and thanks for your suggestions.\n\nIII. Thanks for your kind suggestions. We will supplement the results of SOTA methods in the paper and analyze our shortcomings.\n\nIt is worth noting that our main results are conducted on a single RTX3090, and most SOTA methods are conducted on 8*A100.\n\nTo the Questions:\n\nTo Q1:\n\nDuring the experiment, the biggest challenge is how to capture the relationship between concepts. We propose and try a variety of methods, and finally, we propose weighted graph convolutional networks and leverage the prior knowledge of concept word dependencies, so as to accurately capture the concepts and their relations, resulting in better performance.\n\nTo Q2:\n\nAll our experiments are carried out on a single RTX3090, and the performance of our method reaches SOTA. It is worth noting that most of the SOTA methods are conducted on 8*A100. We also test the running speed, which is 214 ms/image. It is acceptable. As we mentioned, we have a comparable running speed, and our method is still significantly faster than the pre-trained based method.\n\nTo Q3:\n\nSCP can be modified to apply in the VQA task. On the VQA task, we can capture the structured concepts of the question (Q) and the structured concepts of the image (V), and then, we can leverage the multi-modal graph Convolutional networks to merge the two graphs to generate answers (A) better. Slightly different from our current method, which only leverages structured concepts of the image in captioning. The potential challenge is how to align the structured concepts of the question (Q) and the structured concepts of the image (V).\n\nTo Q4:\n\nThe initial weight is established based on word dependencies, which is also called point-wise mutual information (PMI) of any concept. We have tried a variety of initial weights, and word dependencies lead to the best results. The most likely reason is that word dependencies are the prior knowledge of the dataset, which helps the model understand natural language better. The subsequent relations are learned through the graph convolutional networks, which are powerful for learning the relations between nodes.\n\nTo Q5:\n\nWe are considering predicting the exact position of the concept in the sentence, and the subsequent text generation can be seen as a process of filling in the blanks. This process of filling in the blanks can refer to bi-directional (forward and backward) concepts and their relations. In this way, during the text generation process, it does not rely too much on word priors, like \"drink\" followed by \"water\" in any case, resulting in image-related sentences.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed approach is sound and well-motived.\n- The manuscript introduces an innovative approach by predicting not just the concepts but also their inter-relationships. This structured concept prediction can offer significant improvements in understanding cross-modal semantics.\n\n2. The proposed W-GCN to capture concept relations and dependencies is interesting.\n\n3. The results prove the effectiveness of the proposed method.", "reasons_to_reject": "1. The paper doesn't sufficiently engage with related works, potentially missing out on drawing distinctions or acknowledging overlaps with other scene graph based methods, e.g., [1][2].\n\n2. A comprehensive evaluation, including a performance metric such as inference speed, might provide a more rounded understanding of the method's practicality.\n\n3. A broader range of applications or tasks, apart from just captioning, might provide a better understanding of the versatility and adaptability of the proposed method.\n\n4. Several experiments are missing.\n- Error analysis.\n- MSCOCO online evaluation.\n- Although the proposed did not achieve state-of-the-art performances, it is still necessary to report the state-of-the-art results in the Tblaes. It can better help the readers understand the strengths and weaknesses of the proposed approach.\n\n[1] R^3Net: Relation-embedded Representation Reconstruction Network for Change Captioning, EMNLP 2021.\n[2] DIFNet: Boosting Visual Information Flow for Image Captioning, CVPR 2022.", "questions_to_authors": "1. In the process of predicting concepts and their structures, what challenges did you encounter? And how did you mitigate them?\n2. How does your method compare to other existing techniques, not just in terms of performance but also in terms of computational efficiency and complexity?\n3. Can the SCP be modified or adapted for tasks other than image captioning? If yes, what potential challenges do you foresee?\n4. In implementing the W-GCN, how did you decide upon the weighting mechanism? Was there an iterative process or any fine-tuning involved?\n5. It would be insightful to understand more about how you see the future developments or potential extensions of this method. Are there any immediate enhancements you're considering?", "paper_topic_and_main_contributions": "The manuscript delves into the challenging task of bridging the semantic gap between images and texts for image captioning. The authors introduce the structured concept predictor (SCP) to predict both concepts and their relationships, then incorporate these into the image captioning. They utilize weighted graph convolutional networks (W-GCN) to depict relations among concepts influenced by word dependencies. By doing so, they aim to improve the understanding of cross-modal semantics and, consequently, the quality of generated image captions. Their results, supported by extensive experimentation, showcase the effectiveness of the proposed approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SlmhenWXMQ", "reviewer": "Reviewer_jqw3", "summary": "", "questions": "1. In L83-85, the authors said that“drink” is more likely to be followed by “water” based on the linguistic prior, which I agree with. However, in the first case of Fig.1, the authors argued that “eating” is followed by “apple” also based on the linguistic prior. This is not convincing to me. Please provide more explanation or change an example.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your sincere review. We will carefully revise our paper following your suggestions.\n\nTo the Rejections:\n\nTo R1:\n\nThanks for your kind suggestions. We will cite this paper and add a comparison in the revised paper.\n\nSimilar to the DIFNet (CVPR 2022) that you pointed out, we also focus on the same challenge. Differently, we solve this challenge from the perspective of natural language processing. First, we will predict the concept words, and then, we leverage the prior knowledge of concept word dependencies to capture the structured concepts in order to guide the generation of trivial words. The DIFNet adds additional visual information (segmentation image) to obtain strong visual features and increase the contribution of vision to generate captions. DIFNet and our method solve this challenge from two perspectives, and our method outperforms DIFNet in metrics.\n\nTo R2:\n\nThanks for pointing out our weakness. We will cite these papers and discuss them. Besides, our method goes a step further than these methods, as our method learns to capture the relations between concepts from the perspective of NLP. Our method proposes weighted graph convolutional networks and leverages the prior knowledge of concept word dependencies to learn the relations between concepts, which would further encourage language generation to be more reasonable.\n\nTo R3:\n\n Thanks for your suggestions.\n\nI. We have done online test of our method. The result is slightly lower than the offline test, which is also in line with the results on COS-Net (CVPR 2022). The online test results are lower than the offline test in COS-Net. It is worth noting that our code is developed based on COS-Net. Meanwhile, our online results are still better than the baselines that we compared in the submitted paper, such as COS-Net (CVPR 2022), ReFormer (ACM-MM 2022) and multiple baselines.The results of our method are listed as follows:\n\ncider-c5: 1.368; cider-c40: 1.381\n\nWe will add the results to the revised paper, and thanks for your suggestions.\n\nII. Thanks for your suggestion. We will supplement the discussion of D in the revised paper. When D<=2, the concepts basically have no initial relations, resulting in the initial matrix being almost equal to the all-0 matrix, making it difficult to learn relations between concepts. The results drop a lot when D<=2. The results in D=4 are similar to D=3. The performance when D=3 is slightly better. When D>=5, almost all concepts have initial relations, and the performance drops a lot. Thus, we choose D=3 in the submitted paper. \n\nTo the Question:\n\nWe will change the example in the revised paper, and thanks for your sincere suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. I appreciate the idea of introducing concepts and modeling their relations, which is key to generate high-quality image captions.\n2. The proposed method achieves state-of-the-art results on a well-known dataset.", "reasons_to_reject": "1.\tThere is a lack of comparison with the previous work that addresses the same problem.  Specifically, in Sec. Introduction L53-64, the authors claimed that extracted visual information is often insufficient, so previous methods rely too much on linguistic priors for caption generation. This claim is similar \nto the previous CVPR work, DIFNet: Boosting Visual Information Flow for Image Captioning (CVPR’22). However, the authors did not make comparison with it in terms of method and experiment. In my opinion, this is not rigorous and not beneficial to highlight the contribution of this work. \n2.\tMissing references. The major contribution of this paper is to predict concepts from ground-truth captions through a multi-label classification task. This idea has been tried by some of the previous works [1-4]. However, the authors did not cite and discuss them. \n\n[1] Semantic Compositional Networks for Visual Captioning, CVPR 2017. \n\n[2] Adaptively Attending to Visual Attributes and Linguistic Knowledge for Captioning, ACM MM 2017.\n\n[3] Fused GRU with semantic-temporal attention for video captioning, Neurocomputing 2020.\n\n[4] R^3Net: Relation-embedded Representation Reconstruction Network for Change Captioning, EMNLP 2021.\n\n3. Insufficient experiment. This paper lacks the experiment on the MS-COCO online testing server, which has been widely done in most previous works. Besides, in L356-357, the word distance in the pre-constructed lexicon D is 3. The authors might consider providing analysis for this hyper-parameter.", "questions_to_authors": "1. In L83-85, the authors said that“drink” is more likely to be followed by “water” based on the linguistic prior, which I agree with. However, in the first case of Fig.1, the authors argued that “eating” is followed by “apple” also based on the linguistic prior. This is not convincing to me. Please provide more explanation or change an example.", "paper_topic_and_main_contributions": "This paper proposes a structured concept predictor to predict concepts and their structures, and integrating these concepts into the decoder for caption generation. Besides, considering that the contribution of different nodes may be different, this paper further designs a weighted GCN to not only construct the connections between different concept nodes, but also assign different weights to these connections. The weighted average concept features are called structured concept features.  Finally, the structured concept features and visual features are fed into a transformer decoder for caption generation. Extensive experiments show that the proposed method achieves SOTA results on the MS COCO dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "None,"}, "extra_text": {"Missing_References": "[1] Semantic Compositional Networks for Visual Captioning, CVPR 2017. \n\n[2] Adaptively Attending to Visual Attributes and Linguistic Knowledge for Captioning, ACM MM 2017.\n\n[3] Fused GRU with semantic-temporal attention for video captioning, Neurocomputing 2020.\n\n[4] R^3Net: Relation-embedded Representation Reconstruction Network for Change Captioning, EMNLP 2021."}}, {"review_id": "diAEuSiCiZ", "reviewer": "Reviewer_y9d1", "summary": "", "questions": "Why do the authors test the method only on captioning tasks? I think the method can also be effective in VQA tasks.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your sincere review. We will carefully revise our paper following your suggestions.\n\nTo the Rejections:\n\nTo R1: \n\nThanks for your kind suggestions. We tested the running speed of our method, which is 214 ms/image. It is acceptable. We also ran the speed of some comparison methods, which are listed as follows:\n\nGRIT (ECCV 2022): 138 ms/image\n\nM2 Transformer: 178 ms/image\n\nVin VL_large: 542 ms/image\n\nThe GCN and concept predictor have little influence on the running speed of our method. Our method is still significantly faster than the pre-trained based method. We will add the discussion in the revised paper.\n\nTo R2:\n\nThanks for pointing out our writing typos. We will carefully check the paper, and make sure there are no writing typos and logical problems in the revised paper. \n\nTo R3:\n\nSorry for the confusion. We will add more details in the revised paper to introduce query embeddings. The learnable query in our paper is similar to BLIP2. However, our query learns the essential concepts within the images. Through the image interaction, each of our learnable queries focuses on a specific area of the image and learns the information (concepts) contained in the image. These concepts include objects, relative positions between objects, actions, etc.\n\nTo the question:\n\nThanks for your suggestions. Our method can indeed be expanded on tasks such as VQA, but this paper focuses on the image captioning task and its challenges, and the extensive method in VQA would be slightly different. In the revised paper, we will add more discussion of the scalability of our method, and in future work, we will add more experiments of different tasks to explore the scalability of the method.\n\nOn the VQA task, we can capture the structured concepts of the question (Q) and the structured concepts of the image (V), and then, we can leverage the multi-modal graph convolutional networks to merge the two graphs to generate answers (A) better. Slightly different from our current method, which only leverages structured concepts of the image in captioning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- In this paper, a structured concept predictor (SCP) is proposed, which is utilized to enhance image captioning. The idea of incorporating structured information between visual concepts is deemed valid and crucial.\n\n- The proposed method demonstrates superior performance compared to strong baselines in captioning tasks.\n\n- To verify the effectiveness of the proposed method, the authors conduct ablation studies.", "reasons_to_reject": "- The method proposed in this study involves several complex modules, such as GCN and the concept predictor. Including an analysis of the inference speed in comparison to other baselines would add strength to the research.\n\n- The writing quality could be improved. For instance, there were difficulties in understanding parts of L172, and it might be helpful if the authors added a \"However\" before that section.\n\n- Regarding the query embedding, more explanation is needed. Are they learnable queries, similar to BLIP2? The authors should provide additional details about the modules to clarify their implementation and functioning.", "questions_to_authors": "Why do the authors test the method only on captioning tasks? I think the method can also be effective in VQA tasks.", "paper_topic_and_main_contributions": "A structured concept predictor (SCP) is proposed to predict concepts and their structures. The researchers integrate these predictions into captioning, aiming to enhance the contribution of visual signals in the task using concepts. Additionally, they leverage the relations among these concepts to distinguish cross-modal semantics for improved description generation. Specifically, the authors employ weighted graph convolutional networks (W-GCN) to depict concept relations based on word dependencies. This enables the model to learn differentiated contributions from the identified concepts during the decoding process. As a result, their approach successfully captures potential relations among concepts and discriminatively learns different concepts, effectively facilitating image captioning by leveraging information across modalities. The effectiveness of the proposed approach, along with each module introduced in this research, is demonstrated through extensive experiments and their corresponding results.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n.a.", "Typos_Grammar_Style_and_Presentation_Improvements": "n.a."}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary**: The reviewers reached a consensus that the proposed method of using concept predictor was well-motivated and well-supported in the paper. Also, reviewers acknowledged that the paper was well-written and state-of-the-art performance is achieved. Therefore, all the reviewers are positive about the results in the paper. In the meantime, several reviewers pointed out a few missing prior works and references which the authors acknowledged during the rebuttal and mentioned they would add them in the final paper. Also, a few reviewers questioned about the inference speed comparison with other methods which was presented by the authors during the rebuttal. Overall, the reviewers seemed satisfied with the authors' responses.

**Reviewers' recommendations**: Four reviewers share that the paper is sound or strong in terms of soundness. (3 'Good's and 1 'Strong'). In terms of the excitement, the reviewers also share the same score distribution which make the paper is sound and somewhat exciting.
Decision: Accept-Main