CWATR: Generating Richer Captions with Object Attributes
Authors: Enes Muvahhid Şahin
Gözde BOZDAĞI AKAR
Venue: Submitted to ICLR 2023 (2023)
Keywords: image captioning
vision and language pretraining
object attributes
machine learning
deep learning
computer vision

Abstract: Image captioning is a popular yet challenging task which is at the intersection of Computer Vision and Natural Language Processing. Recently, transformer-based unified Vision and Language models advanced the state-of-the-art further on image captioning. However, there are still fundamental problems in these models. Even though the generated captions by these models are grammatically correct and describe the input image fairly good, they might overlook important details in the image. In this paper, we demonstrate these problems in a state-of-the-art baseline image captioning method and analyze the reasoning behind these problems. We propose a novel approach, named CWATR (Captioning With ATtRibutes), to integrate object attributes to the generated captions in order to obtain richer and more detailed captions. Our analyses demonstrate that the proposed approach generates richer and more visually grounded captions by integrating attributes of the objects in the scene to the generated captions successfully.

Under review as a conference paper at ICLR 2023
CWATR: GENERATING RICHER CAPTIONS WITH OB- JECT ATTRIBUTES
Anonymous authors Paper under double-blind review
ABSTRACT
Image captioning is a popular yet challenging task which is at the intersection of Computer Vision and Natural Language Processing. Recently, transformer based unified Vision and Language models advanced the state-of-the-as further on im- age captioning. However, there are still fundamental problems in these models. Even though the generated captions by these models are grammatically correct and describe the input image fairly good, they might overlook important details in the image. In this paper, we demonstrate these problems in state-of-the-an baseline image captioning method and analyze the reasoning behind these problems. We propose novel approach, named CWATR (Captioning With A7tRibutes), to inte- grate object attributes to the generated captions in order to obtain richer and more detailed captions. Our analyses demonstrate that the proposed approach generates richer and more visually grounded captions by integrating attributes of the objects in the scene to the generated captions successfully.
INTRODUCTION
With the recent advancements in Computer Vision (CV) and Natural Language Processing (NLP), machines can understand and respond to visual or textual data, and establish relationships between these two modalities. Among several studies working on these two modalities, image captioning aims to generate grammatically and semantically meaningful sentences describing the given input image like the humans do. A good caption should be grammatically correct, natural sounding, rich, and grounded on the image (Stefanini et al. 2022), (Rohrbach et al. 2018). (Zhou et al. (020b).
Design of large transformer- based models (Vaswani et al. 2017) and utilization of large datasets (Chen et al., 2015; Sharma et al.] 2018 Ordonez et al., ID. (Young et al.] (2014), have led to a significant improvement in sfate-of-the-arr in image captioning. OSCAR (Li at. 2020) and VIVO (Hu et al., 2021) achieved very good results in general image captioning (Chen et al. 2015) and
novel object captioning (Agrawal et al. 2019). VinVL (Zhang al. 2021] further improved those two and achieved state-of-the-art by utilizing richer regional features.
Even though theoretical evaluation of state-of-the-art methods results in high scores, there are over- looked problems in these models. These problems arise when actual captioning outputs are ex- amined in detail. There are studies (Yang al. 2019 Ma al. 2020) demonstrating that image captioning models are inclined towards copying phrases from training dataset without paying atten- tion to the input image. Furthermore, these models might hallucinate non-existing objects in the image overlook important details (Yang et al., 2019: Rohrbach et al.) 2018).
Our observations in this study are also in parallel with those findings. The results show that the recent captioning models overlook some aspects of the scene. Most of the time, the generated captions lack details of objects in the scene. An example of such a case demonstrated in Figure I In this example, the caption generated by VIVO (Visual Vocabulary Pretraining) with VL features (Hu et al., 2021 Zhang et al., (02I)) hallucinates a chair. It also overlooks important details in the image such as the fence of the garden and the car in the background. Moreover it does not mention about properties of the objects in the scene, such as "small garden, red car".
In this paper, attack this problem and propose novel approach in order to generate richer captions with additional object attribute information. More precisely, contributions of this paper are as follows:

---

Under review as a conference paper at ICLR 2023
girddn chair and the ground.
Figure 1: An example image and a poor caption generated by VIVO (Hu et 2021 Zhang al. 2021
We analyze shortcomings and problems with tfaee-of-thearr image captioning models and demonstrate how they overlook important details in the image.
We propose novel approach in order improve generated captions with object attribute information. The proposed approach, CWATR, is directly applicable to Vision and Lan- guage Pretraining (VLP) based approaches.
We compare the CWATR with the bascline method, VIVO (with VLn features) (Hu et al. 2021 Zhang et al. 2021), and provide theoretical and visual analyses.
RELATED WORK
Initial image captioning approaches were based on template filling (Kulkarni et al. 2011 Yao al, 2010). Predefined sentence templates were filled with predicted object names, attributes, and prepo- sitions. Rapid progress in deep learning field also influenced the image captioning research and deep learning based approaches were proposed /Vinyals et al. 2015 Karpathy & Fei-Feij 2015). These approaches utilized a Convolutional Neural Network (CNN) as the image encoder and a Recurrent Neural Network (RNN) as the language model.
Later, Xu et al. (015)) proposed Show, Attend, and Tell by integrating an attention mechanism between CNN encoder and RNN decoder in order to generate enhanced and more visually grounded captions. After Show, Attend, and Tell, attention mechanism became a standard and many other works employed and/or improved attention in image captioning methods (Lu al. 2017 Chen
ct al. 2018 2017).
Anderson et al, (2018) introduced another attention mechanism named Bottom-Up attention in addi- tion to the Top-Down attention in Show, Attend, and Tell. In Bottom- attention, an object detection algorithm detects objects in the scene. Later, RNN attends regional features for detected object instead of whole feature map. The idea of exploiting regional features for objects in the scene is
employed in many image captioning methods QQin et al. 2019 et al. 2019 Wang al. 2020).
The invention of transformers Vaswani et al., 2017) caused paradigm shift in NLP and transform- ers have become the go-to method recent approaches (Devlin et al.) 2019 Radford et al.. 2019
Brown et al. 2020). Recently, they become quite popular in CV well. Transtormers are used as a feature extractor similar to CNNs (Dosovitskiy et al.) 2021 Touvron al., 2021). These ap- proaches, dubbed Vision Transformers, divide the image into patches and perform sell-attention these patches. Inspired from these approaches, some image captioning methods employed Vision
Transformers in their Visual Encoder blocks (Liu et al. 2021 Wang et al. 2022).
Another branch of image captioning approaches focus on unified architecture for Visual Encoder and Language Model blocks by utilizing transformers. This kind of approach was first introduced in Zhou et al. (2020a). called Unified- VLP (Unified Vision and Language Pretraining). Unified- VLP, single transformer network is used for both encoding and decoding steps. It is also unified
2

---

Under review as a conference paper at ICLR 2023
in the sense that, a single architecture is pretrained on large image-text paired datasets and then finetuned on task-specific datasets for downstream tasks such as image captioning or visual question answering. OSCAR (Li et al.] 2020 further improved Unified-VL VLP with the integration of object lags which helps alignment between image features and language features. VInVI. (Zhang et al. 202T] has been proposed an improvement over OSCAR. In VinVI., the captioning architecture is the same as OSCAR but the object detection model is improved by training a larger Faster R- R-CNN (Ren al. 2015) model on a collection of datasets (Chen et al. 20155 Kuznetsova al. 2020 Shao et al. 2019| Krishna et al. 2017) instead of just one (Krishna et al. 2017). This helps extracting richer regional features and results in better captioning performance (Stefanimi et al.] 2022).
VIVO (Visual Vocabulary Pretraining) (Hu et al. 2021) is another Unified- -LLP based method which was proposed for Novel Object Captioning where aim is generating captions for images which con- tain unseen objects during training. In VIVO, pretraining is performed on an object detection dataset (Kuznetsova et al., 2020) and finetuning is performed on an image captioning dataset (Chen et al.] 2015) following the restrictions of necaps challenge (Agrawal et al. 2019). VinVL also improves VIVO by just utilizing richer regional features extracted by a larger object detection model while keeping captioning model and strategy the same.
Even though these models generate pretty impressive captions, there are still fundamental problems with them. In Yang et al. (2019), they have shown that image captioning models are prone to biases in the training dataset and language model. They copy phrases from the training set in the generated caption, overlook details in the image and disregard attributes of objects in the image. Various ap- proaches are proposed in order to generate more visually grounded captions (Yang et al. 2019)) (Ma) al. 2020). In Wu et al. ) Yao al. (2016), they proposed predicting attributes, embedding them, and feeding them to RNN- as additional input. In Yang et al.] (2019), they utilized attributes implicitly by training an attribute prediction module to generate richer features. However, effect of object attributes are unexamined in state-of ofate-of-the-rr the-art unified image captioning models.
In this work, we propose a novel approach for integrating object attributes to the state-of-the-ae transformer-based unified image captioning models in order to generate richer captions. We demon- strate the proposed approach on general image captioning and Novel Object Captioning (NOC) tasks. VL is the state-of-the-art model on general image captioning and NOC. For NOC, they make use of VIVO pretraining (Hu et al. 2021). Hence, approaches VIVO and Vin are taken as baseline in this thesis. Analyses and improvement strategies are employed on this baseline.
PROPOSED METHOD
We demonstrate our proposed approach, named CWATR (Captioning With ATtRibutes), for gener- ating richer captions with attributes on nocaps (Agrawal et al. 2019) challenge. Hence, following VIVO (Hu et al. 2021) and VL (Zhang et al. 2021). we pretrain the network on Open Images Kuznetsovaa al., 2020) dataset and finetune on COCO (Chnn al. 2015) dataset.
As the first step in pretraining, finetuning, and testing, we extract regional features, object tags, and object attributes for a given input image, 7. Regional features and object attributes are extracted using a Faster R-CNN (Ren et al., 2015) model with extra attribute prediction head. This model is the same as in VLn (Zhang et al. (021) and it is trained on collection of datasets (Chen et al., 2015 Kuznetsova et (122000)Shaa et al. 2019 Krishna et al. 2017). The model is then linetuned on Visual Genome (Krishna al.] 2017) with extra attribute prediction head in order to generate richer regional features. In CWATR, we exploit these attribute predictions.
Regional features are extracted by projecting bounding box predictions of Faster R-CNN to the backbone output feature map and applying Rol pooling.
Faster R-CNN model predicts a set of attributes M, = {m-a, m,1....: m1..-13 for each detected object in the image where G; is the number of predicted attributes for the i-th object. An example attribute prediction output is illustrated in Figure 2 We obtain the set of overall object attributes
A {ao, ai...., } } where L is the number of attributes for the input image I. We only add maximum of 1 attribute from M, to the overall set A. For an object o, in I, the attribute with the highest confidence in M, which is also not already existing in the overall set A is added to A. Hence, the attributes in A are unique. Such an approach allows extracting a diverse set of most dominant attributes of the objects in the input image.
3

---

Under review as a conference paper at ICLR 2023
Figure An example object-attribute prediction
Aim in nocaps challenge is generating captions for images which contain novel objects that are unseen during captioning training. Models are expected to learn novel objects through Open Images (Kuznetsova et al. 2020) dataset and mention those objects during captioning. Hence, in order to predict novel object tags, we utilize a state-of-the-ar multi-label classification network called ML- Decoder (Ridnik et al. 202T) trained on Open Images dataset. This network predicts object tags in the image out of 600 novel classes.
The process of extracting regional features, object attributes, and object tags is illustrated is Figure 3 Following this procedure, we obtain 3 sets, C, A, and F', corresponding to object tags, object attributes, and regional features, respectively.
Regioral Fartures
Pocling
Faster R-CNN
Regrusior
Dourd Davo
ABTIUU For
Soennt bork)
BLACK
Obaec Tap
oboect Detestor Mutii Label Clssifler
SPOCLARS
Figure 3: Regional feature, object tag, and object attribute extraction. Regional features and object attributes are extracted using Faster R-CNN (Ren et al. (215)) model (with extra attribute predic- tion head) trained on collection of datasets (Chen et al. 20155] Kuznetsova et al. 2020 Shao al, 2019 Krishna et al. (017).. Novel object tags are extracted using ML-Decoder (Ridnik et al. 2021 trained on Open Images (Kuznetsova et al. 2020) dataset.
For novel object captioning, we perform pretraining on Open Images (Kuznetsova et al. 2020 dataset, similar to Hu al. (202T) as the first step. 'he aim in pretraining is learning a visual vocabulary for novel objects. Different from VIVO, we integrate object attributes to the pretraining process in order to allow model to establish the relationship between regional features, object tags, and object attributes.
During pretraining, previously extracted sets, C, A, and F, are fed to the model. Similar to VIVO, the captioning model is transformer-bast based BERT base (Devlin et al. 2019) network. We randomly replace some of the tokens in object tags (set C)) and some of the tokens in object attributes (set A) with special mask token similar to Devlin al. 2019). The aim is predicting outputs corresponding
4

---

Under review as a conference paper at ICLR 2023
to the masked tokens the network output by applying self-attention between object tags, object attributes, and regional features.
Formally, define D {do, di.......... ......ddd-). dM-1) be the set of tokens for masked object tags in C' 2 D and D {dg, di..... .d.............................. ) to be the transformer outputs corresponding to elements in DD where M the number of masked object tags. We define E feu. e1 e1..... eQ-1 } to the set of tokens for masked object attributes in 2 and = (co, c1, e <q- ) to be the transformer outputs corresponding elements in E' where Q is the number of masked attributes.
Input tokens in object tags or object attributes might be fed to the network any order since they are independent. Furthermore, there might be more than one masked token for set (M > 1 or Q 1). In that case, there is an ambiguity at the transformer output since any output for masked token might correspond to any input. As an example, if round and black are two masked input tokens, the network is free to predict these words at any of the outputs since attributes are not ordered by nature unlike words in a sentence. Hungarian Matching between masked words and the network outputs. We employ a similar strategy and perform Hungarian Matching between masked input tokens and corresponding network outputs to obtain an optimal assignment between them.
order to solve this ambiguity, Hu et al.) (202T) proposed applying
Since object tags and object attributes belong to different parts of speech, they are treated separately. Two optimal one-to-one assignments obtained using Hungarian Assignment Algorithm (Kuhn 1953]: a and 3. a assigns masked object tags to transformer outputs for these tags and / assigns masked object attributes to transformer outputs for these attributes. To obtain and B, Hungarian Assignment Algorithm is solved for two square cost matrices Ja E RMMx and J E TRQ respectively for and B. These cost matrices constructed according equation
J = 1 1-p(d)))) mn 11 -((e.)
(1)
where p(d,") and p(etn) are the probabilitics of transformer outputs d, and ēm being the words with vocabulary id ty and In. for masked tag and attribute, respectively. and B3 assigns input indexes to output indexes as in equation 2
a(i) = e E O, - B(m) = n m,n [0, 1
Hungarian Assignment Algorithm finds and such that cost functions in equation are mini- mized.
M-1
Ja = Σ
p()) p(d,
1=0
(3)
Q-1
J3 Σ
p(c
After the assignments are obtained, the network is trained with Masked Token Loss (LMTL)) CMTL for object tags and object attributes are calculated separately according to equation
1
MTL
Σ
Σ
y log log,(d))
M d
j
V
Cottribute
1
Σ
Σ
htth log (
MTL
Q
C... EE
where V the size of the vocabulary (number of output classes), p(d,) is the predicted output prob- ability for output class j for the masked tag token d,, and p(e.). is the predicted output probability for output class for the masked attribute token e. y/ is the binary ground truth label for class for the masked word d, and yn the binary ground truth label for class n for the masked word em- ia and yth are calculated using the assignments a and as in equation 5
w = f1, a j
1, if (m) =
0. otherwise
ynm = 0. otherwise
(5)
5

---

Under review as a conference paper at ICLR 2023
Total CMTL used during pretraining is the sum of individual losses for tags and attributes in equation Sisioon/@hheeer where is used to weight object tag and object attribute losses.
LTT + Lottribute
Chotal
(6)
2
The process of pretraining is illustrated in Figure 4 for example input image from Open Images
dataset. Full attention mask is used during pretraining. Any input token can attend to any other input token. The pretrained model on Open Images dataset is finetuned on COCO dataset for image
Total Loss
Image
Masked Token Loss
Masked Token Loss
Matcher
Hungorian Matcher
en
MULTI-LAYER TRANSFORMER
[CLS]
(MASI]
WATCH
[MADK]]
[SFP]
OLD
[MAGI]
[Mollta
[SEP]
Objest lago - Person, Wattch Binoralers, Itai
Oljjet Aitributes- Okd, Hound, black, Bege
Figure 4: Proposed pretraining approach with object attributes. We feed attributes as an additional set of inputs. Masked object tags and attributes predicted the network output by attending to unmasked input vectors from all sets (object tags, object attributes, regional features).
captioning. Firstly, object tags, object attributes, and regional features are extracted in the same way as in pretraining (Figure B}. The set of words in ground truth caption, S is fed to the model along with extracted set of object tags, C, object attributes, A, and regional features, F.
During finetuning, object tags and attributes are not masked. Only tokens in ground truth caption are randomly masked. The model should predict outputs corresponding to the masked words by attending the other words in the ground truth caption, object tags, object attributes, and regional features. Since the words in ground truth caption are ordered, an assignment is not necessary, unlike pretraining. Formally, let D {do, di,..., dd.-i) 3 be the set of tokens for masked words in S D and D {do, di..... d be the transformer outputs corresponding to elements in D where M is the number of masked words in S. The network is trained with LMTL objective. Cross- Cross-Entropy loss is used in LMTL. LMTL is calculated according to equation]
LMTL M
1
Σ 4.D
Σ
u log p(d))
where V is the size of the vocabulary (number of output classes), p(d)) the predicted output probability for output class j for the masked token d, y is the binary ground truth label for class for the masked word d,
The process of finetuning is illustrated in Figure for for an example input image from COCO dataset. Unidirectional attention mask is used for tokens the ground truth caption in order to model se- quential nature of the language, similar to Hu et al. (2021).
During inference, regressegeee decoding is applied. At each time instant, input token for that time instant is masked and network is asked to predict the current word at the transformer output by attending previously predicted words, object tags, object attributes, and regional features.
4 EXPERIMENTAL RESULTS
As explained in Section 3 we expose the network to object attributes in both pretraining and fine- tuning. In this section, we analyze the CWATR model and compare it with the baseline VIVO

---

Under review as a conference paper at ICLR 2023
Irpitinng
Klrised Iakn
MULTI-LAYER TRANSFORMER
Dien Minexte
Figure 5: Proposed finetuning approach with object attributes. Attributes fed as additional set of inputs. Network is trained with seq2seq objective as in Hu et al. (202I). Masked words in the caption is predicted the network output by attending previous words, object tags, object attributes, and regional features.
model which is not exposed to object attributes. Both methods utilize regional features generated by VinVL. We compare two models both visually and theoretically in the following subsections.
4.1
VISUAL ANALYSIS
In this section, we provide visual outputs of the baseline VIVO model and the proposed CWATR model. Attribute words in the predicted output caption of the CWATR are shown in bold. Additional visual examples are provided in Appendix A
In Table I and Table 2 we share example images from COCO test set, their ground truth captions, and predicted captions by the VIVO and CWATR.
In Table we see that the CWATR model predicts small, wooden, and red attributes. Furthermore, it mentions about the car in the background and does not hallucinate a chair object, unlike VIVO. In
Table A visual example, corresponding ground truth sentences and outputs of VIVO and CWATR. CWATR successfully integrates small, wooden, and red attributes.
Ground Truth
VIvO
CWAIR
simple, fenced as garden containing several kinds of plants. around flower pota it. -A few plants ganden
gardes with chair and plant ground.
A. small garden with weoden
plot with
A small gardon that has vegetables blooming
some dirt a garden gmass bushes and trees.
Table observe that the caption generated by the VIVO is very superficial. On the other hand, CWATR produces very detailed description of the scene by mentioning greater number of objects in the scene and integrating an attribute for each of them such as white, filled, chopped, and wooden. Another visual example, corresponding ground truth sentences and predictions of the models for an
Table 2: A visual example, corresponding ground truth sentences and outputs of VIVO and CWATR. CWATR successfully integrates white, filled, chopped, and wooden attributes.
Image
Ground
VIvO
CWATR
pan filled with veggies and a block of butter. vegetables pun. chopped vegetiblies A pan with butter and other ingrediens for dish.
A hout of food with a spoon it.
A white bowl filled with chopped vegiies wonden spoce.
Spoon in
Pot with variety of chopped vegotables, big chunk spoon.
image in nocaps validation set is given in Table In this example, VIVO does not give details of

---

Under review as a conference paper at ICLR 2023
objects in the scene. Furthermore, it hallucinates a dishwasher object. On the contrary, CWATR associates metal and wooden attributes with relevant objects in the scene and does not hallucinate dishwasher. The success of CWATR in such visual examples urged to examine visual results
Table A visual example, corresponding ground truth sentences and outputs of VIVO and CWATR. CWATR successfully integrates metal and wooden attributes.
Image
Ground Truch
VIVO
CWATR
Wood cabinctry surtonsding a sink with high curved fancet and kinchen
A kitchen with a sink, cabinees, and disbwasher.
kitchen with metal sink and wooden cubinets.
knives on the left side of the counter.
interor kitchen showing the and suroonnding
togs
kitchen
cabinet, sink counter top.
A clean counter top with cnal cohinet furmiure downscand.
kitchen counter and double sink with a towel loying on the counter.
Kitchea sink granise countertop and wooden
sink with knives, plates, and lowel on top of counter.
-A gray sink is placed in the middle of cousteriop.
kinchen sink with many cabinets under the sink. oringo container
Kniies, dish song,
kitchen top of sink.
further by investigating the ground truth sentences and scores of the models for predicted captions.
In Table 4 and Table we provide examples where CWATR correctly integrates object attributes (white, süfting, metal, stainless steel) to the predicted captions but gets lower scores. The reason for these results that the ground truth captions in these examples describe the scene from different perspectives. For example, in Table 4 some ground truth captions mention about what is written on the poles, some mention about their pointing directions. Some mention about two poles, some mention about only one. Furthermore, they do not mention about some or any of the object attributes. Hence, the proposed CWATR model is penalized for integrating attributes even though the predicted sentences describe the scenes perfectly. Another such example is given in Table 6 where the
Table 4: A visual example, corresponding ground truth sentences and outputs of VIVO and CWATR models. CWATR gets lower score despite producing richer caption.
Image
Ground Truth
VIVO
CWAIR
strect sign that reads Greea Garbo Straze"
A close up of street sign with sky background
- couple white street sigus sttting on of metal pols. CIDEr 38.01 SPICE: 14.29
street sign that reads Gremn Garbe Strade. interscction. A strect sign that as pointing in different ways.
Gert:
intersecting while
CIDEr: 61.27
SPICE: 16.6
street siga reading
Giuho
pole.
Table 5: A visual example, corresponding ground truth sentences and outputs of VIVO and CWATR models. CWATR gets lower score despite producing richer caption.
Image
Ground Truth
VIVO
CWATR
We looking a small airline toilet.
bullnoom with tollet sink, and
A white toilet sitting stainless CIDEr 43.61
compact
toiler sink inside.
sepp dispenscr.
A voilet and sink the buihromm an airplane.
CIDE: 63,0
SPICE: 36.71
SPICE: 14.81
A builmoom trilei that has the seat down.
This a lnvatcey of airplane.
predictions of the VIVO and CWATR just differs by two attribute words, white and pink. Despite CWATR identifies and associates these attributes correctly, it gets lower score compared VIVO. None of the ground truth sentences mention about these attributes. All of them define the scene from different perspectives. Hence, CWATR is punished even though it describes the scene very well and detailed.
4.2 THEORETICAL COMPARISON
Comparison of two models on COCO and nocaps datasets are given in Table 7 and Table respec- tively. The results show that the VIVO outperforms the CWATR on both datasets. As discussed in Section I.], visual analysis demonstrates that CWATR successfully generates richer and grounded captions with additional attribute information. However it falls behind VTVO because the ground
8

---

Under review as a conference paper at ICLR 2023
Table 6: A visual example, corresponding ground truth sentences and outputs of VIVO and CWATR models, CWATR gets lower score despite producing richer caption.
Image
Ground Trush
VIVO
CWATR
Creszive centerpiece floral urangement at as oundoar evest.
-A vase filled with flowers sit- ting on top of table.
A vase filled with white and plak floers on of table.
wedding centerpiece made of Bowers and vanius other plant.
flowers sitting top of
207.57
SPICE 333
SPICE:
A flora arragement inside a clear sylinder shaped vise.
Table 7: Evaluation results on COCO Karpathy test split (Karpathy & Fei-Fei 2015) for VIVO
baseline model and proposed CWATR model which utilize object attributes.
COCO
Method
CIDEr
SPICE
VIVO
119.41
21.62
CWATR
110.62
20.7
Table 8: Evaluation results nocaps validation set for VIVO baseline model and proposed CWATR model which utilize object attributes.
in-domain CIDEr
near-domain CIDEr
out-of-domain CIDEr
entire
Method
SPICE
SPICE
SPICE
CIDEr
SPICE
VIVO 85.48
12.84
77.22
12.3
59.53
10.14
74.82
11.96
CWATR
79.29
12.44
72.1
11.74
55.07
9.81
69.68
11.48
truth captions problematic and do always contain relevant attributes. Hence, the model which predicts attributes gets penalized even though the generated captions is correct and more detailed compared the baseline. Additionally, both models' performances are worse nocaps dataset since it is more challenging than COCO dataset due to novel objects.
CONCLUSION
In this paper, we propose method, CWATR, to integrate object attributes to state-of-the-art Vision and Language Pretraining based image captioning algorithms in order to generate richer captions. We explain the method in detail. We compare the proposed method with the statc-of-the-art baseline method, VIVO (with VinVL features) (Hu et al. 2021 Zhang et al. 2021). Our analysis demon- strates that the proposed method successfully integrates object attributes in the scene to the gener- ated caption. Hence, it generates attribute- wise richer and more detailed captions compared to the baseline method. Further analysis reveals that the proposed method obtains lower scores for some captions compared to the baseline even though the generated captions are better and more detailed than that of the baseline. Our analysis shows that this result is due to improper ground truth captions with high variance. Different people describe the scene different level of detail and resulting ground truth captions might lack object attributes. Hence, the proposed method gets penalized in such cases and obtains lower scores. In order to achieve fairer evaluations, we believe that there should be more strict standards while labeling images for captioning (generating ground truth cap- tions). Objects, their attributes, and relationships should available the ground truth captions. Such enhanced and consistent captions in the training dataset well would help training the models better. addition, the proposed algorithm can used to guide the process of enhancing ground truth captions. Attribute-wer wise rich captions predicted by the proposed method can be shown to the annotators as a baseline so that they can originate from these captions and integrate object attributes the ground truth captions.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8947-8956, October 2019. doi: 10.1109/CCCV22119.00000
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6077-6086, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. SCA- CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. doi: 10. 1109/CVPR.2017.655
Xinlei Chen, Hao Fang, Tsung-Yl Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. L. Zitnick. Microsoft COCO Captions: Data Collection and Evaluation Server. ArXiv, 2015.
Xinpeng Chen, Lin Ma, Wenhao Jiang. Jian Yao, and Wei Liu. Regularizing RNNs for Caption Gen- eration by Reconstructing the Past with the Present. In 2018 TEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
June 2018. doi: 10.1 10911099..2008..000 8.00834.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers). pp. June 2019. Association for Computational Linguistics. doi: 10.18653/v 10118653/1/N199111 1423.
Minneapolis, Minnesota,
A. Dosovitskiy, L. Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, M. Dehghani, Matthias Minderer, G. Heigold, S. Gelly, Jakob Uszkoreit, and N. Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. ICLR, 2021.
Xiaowei Hu, Xi Yin, Kevin Lin, Lei Zhang, Jianfeng Gao, Lijuan Wang, and Zicheng Liu. VIVO: Visual Vocabulary Pre- Training for Novel Object Captioning. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 35(2):1575-1583 May 2021. ISSN 2374-3468.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3128- 3137, June 2015. doi: 10. 100100//2R222215.99999..
Lei Ke, Wenjie Pei, Ruiyu Li, Xiaoyong Shen, and Yu-Wing Tai. Reflective Decoding Network for Image Captioning. In 2019 JEEE/CVF International Conference on Computer Vision (ICCV), pp. 8887-8896, October 2019. doi: 10. 10.1109/CCV2221990000
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annota- tions. International Journal of Computer Vision, 123(1):32-73, May 2017. ISSN 0920-5691.
doi:
H. W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quar- terly, 2(1-2):83-97, 1955. ISSN 1931-9193. doi: 10. 1..002/aa.380002000999
10

---

Under review as a conference paper at ICLR 2023
Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. Baby talk: Understanding and generating simple image descriptions. In CVPR 2011, pp. 1601-1608, June 2011. doi: 10. D0.1109CVVR.2011.99992
A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, Ivan Krasin, J. Jusee,, S. Kamali, S. Popov, M. Malloci, Alexander Kolesnikov, Tom Ducrig, and V. Ferrari. The Open Images Dataset V4. International Journal of Computer Vision, 2020. doi: 10.1007/s1 223--22-00316--
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-Semantics Aligned Pre- training for Vision-Language Tasks. In ECCV, 2020. doi:
Wei Liu, Sihan Chen, Longteng Guo, Xinxin Zhu, and Jing Liu. CPTR: Full Transformer Network for Image Captioning. ArXiv, 2021.
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing When Look: Adaptive
Attention via Visual Sentinel for Image Captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 3242-3250, July 2017. doi: 10.1109/CVPR.201.
345.
Chih-Yae Yao Ma, Yannis Kalantidis, G. AlRegib, Péter Vajda, Marcus Rohrbach, and Z. Kira. Learning to Generate Grounded Visual Captions Without Localization Supervision. In ECCV, 2020. doi:
10.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2Text: Describing Images Using 1 Million Captioned Photographs. In Advances Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
Yu Qin, Jiajun Du, Yonghua Zhang, and Hongtao Lu. Look Back and Predict Forward in Image Captioning. In 20/9 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8359-8361 June 2019. doi: 10.1109/CVP..201900000
Alec Radford, Jeff Wu, Rewon Child, D. Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. undefined, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Tal Ridnik, Gilad Sharir, Avi Ben-Cohen, Emanuel Ben-Baruch and Asaf Noy. ML-Decoder: Scal- able and Versatile Classification Head, December 2021.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object Hallucination in Image Captioning. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 4035 4045, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10. 10.18853/v1/D18-11
Shuai Shao, Zeming Li. Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A Large-Scale, High-Quality Dataset for Object Detection. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8429-8438, October 2019. doi: 10. 1109/CCC.2019.0088
Piyush Sharma, N. Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In ACL, 2018. doi: 10. 10.188533/1/P188 1238..
Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. From Show Tell: A Survey Deep Learning-baser Image Captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2022. ISSN 1939-3539. doi: 20110/9/TAMM..0022.3112

---

Benjamin 7. Yao, Xiong Yang, Liang Lin, Mun Wai Lee, and Song-Chun Zhu. 12T: Image Parsing to Text Description. Proceedings of the IEEE, doi: 10.1109/P000.20000005000 Under review as a conference paper at ICLR 2023
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. Pro- ceedings of the 38th International Conference on Machine Learning, pp. 10347-10357. PMLR, July 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All you Need. In Advances in Neural Infor- mation Processing Systems, volume 30. Curran Associates, Inc., 2017.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3156 1556--164, June 2015. doi: 10.1099CVRR.0055529995.
Li Wang, Zechen Bai, Yonghua Zhang, and Hongtao Lu. Show, Recall, and Tell: Image Captioning with Recall Mechanism. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07):
12176-12183, April 2020. ISSN 2374-3468. doi: 10. 10.1609/aaai.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. SimVLM: Simple Visual Language Model Pretraining with Weak Supervision. In International Conference on Learning Representations, May 2022.
Qi Wu, Chunhuu Shen, Linggiao Liu, Anthony Dick, and Anton van den Hengel. What value do explicit high level concepts have in vision language problems?, April 2016.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In Proceedings of the 32nd International Conference on Machine Learning, pp. 2048 2057. PMLR, June 2015.
Xu Yang. Hanwang Zhang, and Jianfei Cai. Learning to Collocate Neural Modules for Image Captioning. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019. doi: 10.1109/CCV.2019.0045..
August 2010. ISSN 1558-2256.
Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei. Boosting Image Captioning with Attributes, November 2016.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014. doi: 10.1162/tuela.000166
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang. Lijuan Wang, Yejin Choi, and Jianfeng Gao. VinVL: Revisiting Visual Representations in Vision-Language Models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5579-5588, 2021.
Luowei Zhou, H. Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. Unified Vision-Language Pre-Training for Image Captioning and VQA. AAAI, 2020a. doi: 16.16 A4A1774417.7005.
Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, and Hanwang Zhang. More Grounded Image Captioning by Distilling Image- Text Matching Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4777-4786, 2020b.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
Additional visual examples are provided below. Examples in Table S and Table 10 are from COCO (Chen al. 2015) dataset. Examples in Table [] and Table [2 are from nocaps (Agrawal al.) 2019) dataset. Examples in Table [T and Table 12 reveal the variance in the ground truth captions since they describe the image from very different perspectives.
Table 9: A visual example, corresponding ground truth sentences and outputs of VIVO and CWATR. CWATR successfully integrates old, rusted, sitting, and yellow attributes.
Image
Ground Truth
VIVO
CWAIR
A old tuck with a husted window in the tall tushes.
An old buck sitting the mid- die of forest.
An old rusted track sitting in the middle yellow flwwes.
old truck sitting overgrown field.
misted
pick-up with flower growing yellow
parked
truck shown docaying among flowers without window.
Table 10:
visual example, corresponding ground truth sentences and outputs of VIVO and CWATR. CWATR successfully integrates double decker and parked attributes.
Image
Ground Truch
VIVO
CWATR
A tall building with foor double docker boses driving along a parking lot.
A group of buses that are sitting in street.
A group double decker buses parked building.
long bus city
double decker
Deuble-decker bises
old building.
Double Secble-Decker Decker buses line up al a bus stup.
Table 11:
visual example, corresponding ground truth sentences and outputs of VIVO and CWATR. CWATR successfully integrates old and white attributes.
Image
Ground Truth
VIVO
CWATR
A shabby garage stands next to brown red residemial building.
A white door open outside a buikling.
An old w building the side road.
An down gumuee to building.
grrngo
brown building
while garage disrepair next to house.
gimgee that is not in good shape nest a building.
The large building uppears
tiny garage made dented
middle of driveway.
gamge contains eeented metallic side wall.
An old white aluminum window less garage, in disrepair, next building.
garage with galvanized steel outside walls.
Table 12: A visual example, corresponding ground truth sentences and outputs of VIVO and CWATR. CWATR successfully integrates shiny, metal, and silver attributes.
Image
Ground
vivo
CWATR
bronze sculpture
that
close of
distontd,
stares
shiny silver
A bronze scalyture with strange expression stands on brick surface.
The sculpture of man made with cooper has the mouth
bust of bronzed
shown with brick background. way.
statue
outside
A bronze statue of man who is making face and has heavy eyebrows.
bronze statue a human face that good looking.
has spot. bronze sculpture
japunese samurni.
13


Reviews:
[{"review_id": "ndrcQqQ7Vz_", "reviewer": "Reviewer_E9nV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an image captioning model, termed CWATR, where object labels, attributes, and visual features are combined together using transformers and masked pre-training methods. Basically, the main contribution of this work is that it shows that introducing rich information like visual features, object labels and attributes into image captioning models can improve the quality of the generated captions, such as detailed descriptions.", "summary_of_the_review": "This paper lacks novelty and convincing experimental results, so I give it a score of 3 in this phase.", "strength_and_weaknesses": "Strength:\n\nS1. The quality of the generated captions is relatively high.\n\nS2: Using Hungarian Assignment Algorithm in the pre-training stage is interesting to me.\n\nWeaknesses:\n\nW1: The model is not novel at all. In many existing works, people have employed transformers to combine visual features, tags and attributes to enhance vision-language representation, such as Oscar, VIVO, VinVL. And the experimental results are not convincing. The author only compares the proposed model with VIVO. Plus, in the section on visual analysis, the author only describes the differences among the generated captions, but deeper explanations are required. Though the captions generated by the proposed model are more detailed with more adjective words, I think the main reason for this is that the proposed approach employs attributes, while VIVO does not.\n\nW2: It seems that the generated captions are more distinctive, but the author only considers CIDEr and SPICE metrics which are not appropriate for distinctiveness. So I suggest using CIDErBtw [1,2].\n\n[1] J. Wang et al. Compare and reweight: Distinctive image captioning using similar image sets. ECCV, 2020.\n\n[2] J. Wang et al. On Distinctive Image Captioning via Comparing and Reweighting. TPAMI, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XPkDAK5_fJ", "reviewer": "Reviewer_hWrE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author proposed an approach which concentrate on attributes generation to general image captioning and NOC tasks. It is an interesting attempt, however, there are some serious defects:\n1. It is mentioned that existing methods “overlook some aspects of the scene”, it is also the motivation of this work. However, the shortcomings of these methods remain unclear. Methods like [1] also trained on Visual Genome with corresponding attribute. 2. As for the experimental results, there are several examples which CWATR outperforms VIVO. However, it can not prove that it works well on the whole dataset, unless a global metric is designed, like CIDEr. In traditional metrics, VIVO is better than this method.\n[1] Xu Yang, Hanwang Zhang, and Jianfei Cai. Learning to Collocate Neural Modules for Image Captioning. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019. doi:10.1109/ICCV.2019.00435\n", "summary_of_the_review": "The direction of this paper is innovative, but needs more explanation and design of experimental methods and more filling of experimental results and conclusions. The author could add more experiments to prove the effectiveness of this method.", "strength_and_weaknesses": "Strength：meaningful attempt in attribute generation.\nWeakness: few illustration of the interpretability of experimental methods；the results and conclusion are inadequate and superficial；experiments are insufficient relatively\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "KjPayidV98n", "reviewer": "Reviewer_4nF4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to address the shortcomings in image captioning models by generating richer captions with additional object attribute information. They pretrain vision language based transformer models using an additional masked attribute loss and match the object information to the corresponding attributes using Hungarian matching algorithm. They further finetune the transformer to generate informative captions conditioned on previous words, object labels , attributes and region based visual features.", "summary_of_the_review": "The contributions and experimental outcomes of the paper are marginal and not well supported by ablations, human studies or architectural changes. I propose major improvements both in the method and evaluation for a good submission. ", "strength_and_weaknesses": "Strengths - The motivation of the paper to generate more descriptive and well grounded captions for an image is well inspired from the limitations of existing methods. They  integrate an additional loss based on object attributes to the pre-training of Vision language transformers. \n\nWeaknesses - #\n\nIn the pretraining stage in Fig 4, a very useful ablation / simpler modeling strategy would be to combine the object tags + attributes and then mask that information randomly for token loss prediction.\n\nDue to the absence of masked region modeling, the model might fail to draw associations between attributes / tags and regions and hence fail to generate novel combinations of attributes + tags (black hat - > black dog) during caption generation.\n\nA study of attention maps from the transformers could help to see what muli-modal attention is being learned by the model. \n\nThe results are far below the baselines and are not justified well by using human evaluation, I recommend the authors to compare using some human evaluation. \n\nAnother metric - SPICE - U [1] could be more useful in measuring such descriptive captions. \n\nThe contributions are very limited compared to VIVO (baseline) and the results worse. \n\n[1]Wang, Zeyu, et al. \"Towards unique and informative captioning of images.\" European Conference on Computer Vision. Springer, Cham, 2020.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject