Learning Visual-Linguistic Adequacy, Fidelity, and Fluency for Novel Object Captioning
Authors: Cheng-Fu Yang
Yao-Hung Hubert Tsai
Wan-Cyuan Fan
Yu-Chiang Frank Wang
Louis-Philippe Morency
Ruslan Salakhutdinov
Venue: ICLR 2022 Submitted (2022)
Keywords: Semi-supervised Image Captioning
Novel Object Captioning

Abstract: Novel object captioning (NOC) learns image captioning models for describing objects or visual concepts which are unseen (i.e., novel) in the training captions. Such captioning models need to sufficiently describe such visual data with fluent and natural language expression. In other words, we expect the produced captions being linguistically fluent, containing novel objects of interest, and fitting the visual concept of the image. The above three aspects thus correspond to fluency, fidelity, and adequacy, respectively. However, most novel object captioning models are not explicitly designed to address the aforementioned properties due to the absence of caption annotations. In this paper, we start by providing an insight into the relationship between the above properties and existing visual/language models. Then, we present VLAF2, for learning Visual-Linguistic Adequacy, Fidelity, and Fluency, which utilizes linguistics observed from captions for describing visual information of images with novel objects. More specifically, we revisit BERT and CLIP, and explain how we leverage the intrinsic language knowledge from such popular models to reward captions with precise and rich visual content associated with novel images. To validate the effectiveness of our framework, we conduct extensive experiments on the nocaps dataset. Our method not only performs favorably against state-of-the-art novel captioning models in all caption evaluation metrics, but also surpasses the SPICE scores of human baseline. We perform quantitative and qualitative analysis to demonstrate how our model generates novel object captions with improved fluency, fidelity, and adequacy. Implementation details and code are available in the supplementary materials.

Under review as a conference paper at ICLR 2022
LEARNING VISUAL- ADEQUACY, FIDELITY, AND FLUENCY FOR NOVEL OBJECT CAPTIONING
Anonymous authors Paper under double-blind review
ABSTRACT
Novel object captioning (NOC) leams image captioning models for describing objects or visual concepts which are unseen (i.e., novel) in the training captions. Such captioning models need to sufficiently describe such visual data with fluent and natural language expression. In other words, we expect the produced captions being linguistically fluent, containing novel objects of interest, and fitting the vi- sual concept of the image. The above three aspects thus correspond to fluency, fidelity, and adequacy, respectively. However, most novel object captioning mod- els are not explicitly designed to address the aforementioned properties due to the absence of caption annotations. In this paper, we start by providing insight into the relationship between the above properties and existing visual/language models. Then, we present VLAF2, for learning Visual-Linguistic Adequacy, Fi- delity, and Fluency, which utilizes linguistics observed from captions for describ- ing visual information of images with novel objects. More specifically, we revisit BERT and CLIP, and explain how we leverage the intrinsic language knowledge from such popular models to reward captions with precise and rich visual con- tent associated with novel images. To validate the effectiveness of our framework, we conduct extensive experiments on the nocaps dataset. Our method not only performs favorably against state-of-the-art novel captioning models in all caption evaluation metrics, but also surpasses the SPICE scores human baseline. We perform quantitative and qualitative analysis to demonstrate how our model gen- crates novel object captions with improved fluency. fidelity, and adequacy. Imple- mentation details and code are available in the supplementary materials.
INTRODUCTION
Novel Object Captioning (NOC) (Agrawal et al. 2019) requires captioning models to accurately
capture images containing novel objects unseen during training captions, and to describe such data with fluent and grammatically correct sentences. Despite impressive benchmark performance
COCO Captions (Chen et al.] 2015) and Flickr (Young et al. 2014), existing image captioning
models (Gao et al.] 2019 Huang et al. 2019 Wang al. 2019 Guo al. 22200) Pan al., 2020 Comia et al. 2020 Zhou et al., 2020) or unpaired image captioning (Gu et al. [019 Feng et al, 2019) generalize poorly to this task since the NOC task covers a larger variety visual concepts,
where nearly 400 object classes barely have any associated training captions.
Existing work of novel object captioning typically rely on the object detection results either fill in the generated slotted sentences (Lu et al. 2018 Wu et al., 2018) or learn the visual vocabulary for novel objects directly (Hu et al. 2020). However, these methods do not consider the semantics and linguistics of the entire sentences comprehensively. Specifically, most existing works do not exhibit the abilities assuring the produced captions with correct and rich visual content, or with sufficient natural and fluent expression. In other words, the three aspects or challenges of NOC still need to be addressed. First all, fluency is expected in the output caption which linguistically natural and grammatically correct. Fidelity reflects the novel objects to be described, while adequacy encourages output captions to properly capture the visual concept of the input image.
Unfortunately, the above properties cannot be casily achieved by standard image captioning mod- els due to the absence of ground truth caption annotations. this paper, we revisit two popular pre-trained visual/language models of BERT (Devlin et al. 2018) and CLIP (Radford et al. 2021)

---

Under review as a conference paper at ICLR 2022
blue shirt with glove and tie on
Captioning Model
a dirt road going through fush grean hilbide.
Captioning Model
two glove hands and red sAm samaing acart blue background
updato
n shot glove and on
cirt road MASK through lush
BERT
dirf road wound through a lush green hilladde
CLIP
hiliido.
glowe hands and blue background.
updatto
(A)
(H)
Figure/ I: Overview VLAF2 captioning model. (a) Leveraging linguistic ability from BERT for describing novel objects. (b) Preserving adequacy and fidelity of novel object captions via CLIP.
and we provide the associated connection fluency, fidelity, and adequacy for NOC. As shown in Fig. 1 we present VLAF2 for learning Visual- Visual-Linguistic Adequacy, Fidelity, and Adequacy, which leverages the intrinsic knowledge of BERT and CLIP for utilizing linguistics observed from cap- tions for describing visual information of images with novel objects. With BERT pre-trained to excel at various linguistic tasks and CLIP to associate visual-language data instance levels, we provide insights to these models and present technical details on how such models can be utilized for performing NOC, while the goals of adequacy, fidelity and fluency can be jointly achieved.
For the evaluation part of our work, conduct extensive experiments benchmark nocaps datasets, confirming that our model able to produce state-of-the-ar results terms metrics linking to adequacy, fidelity, and fluency. In addition, by variety of ablation studies, further provide analysis on how BERT and CLIP are utilized and thus be preferable in solving NOC tasks.
2 CONNECTING FLUENCY, FIDELITY AND ADEQUACY WITH BERT AND CLIP
We first discuss how fluency, fidelity, and adequacy be fundamentally and technically related to the visual/language models of BERT (Devlin et al. (018) and CLIP (Radford et al.] (02T). which will be utilized in our proposed learning framework. For caption fluency, one expects that the im- age caption output to be linguistically natural and grammatically correct. It not only requires to capture the co-occurrence of novel-object vocabularies, the associated collocations such as verbs or modifiers are expected to be properly utilized. We define the context containing novel-object vocabularies as y and the associated collocations as ý. Fluency can be defined as how many col- locations are observed by captioning models given novel image context p()lý). This is exactly the masked language modeling (MIM) objective adopted in BERT, without the log function explic- itly calculated. This is the reason why adopt BERT to learn the co-occurrence of novel-object vocabularies and their associated collocations to improve the linguistic fluency.
We now relate fidelity and adequacy in image caption outputs to the model of CLIP. We start by defining objects appearing in the caption as X, and objects mentioned by the captions as Y. Relevant objects (X. are defined objects that are both included in the images and described by the captions. Fidelity assesses whether the visual content presented in the produced caption is correct, and can be defined as the fraction of relevant objects among objects in captions p(A'( p(X.))
Similarly, adequacy evaluates whether sufficient visual details have been expressed by captions, and we can define it as the fraction of relevant objects among objects in images p() p(x,))
p(x)
Based on the observation of Friedman (2017), these two quantities p(X'(2") and p(){)))) represent the association between X' and y. While CLIP trained to associate image and text data at the instance level, it can be further applied for guiding image captioning models for NOC, boosting the desirable fidelity and adequacy.
3 METHOD
Before presenting the learning framework of VIAFF2 for novel object captioning, we determine the notations and settings for the sake of completeness. Given a small set of caption-labeled images Xt. the corresponding captions Y, as well as a large set of uncaptioned images X,, our goal is to generate the associated captions Y, for X,, using the captioning model Cg, where is the parameters of captioning model C. To achieve this, we propose a two-stage learning framework of VLAF2, guided by two pre-trained vasual-language models of BERT (Devlin et al. (018) and CLIP RRdfford
2

---

Under review as a conference paper at ICLR 2022
Captions
dirt Lahh Nilside.
CLIP
0.7
laptops sitting neat to wach other table next to bunch wine
reñable?
Captioning Model
Captioned image Z7
1/t
masked
BERT
dirt road wound through kuih grewn hillidd..
Ce
road going Nilldde
Cyear
Uncaplioned image Iy.
Figure 2: Learning caption novel objects with linguistic fluency. For caption-labeled image zt, we impose the sequence-tor -eeuueenee objective Lss, for training. For uncaptioned image Za. we exploit BERT improve the wordings the generated caption Du. and the refined caption is denoted as i
et al. 202T). Note that neither BERT CLIP have observed X, during training. An overview of our method shown in Fig. [ While our captioning model determines the labels of novel object images using pretrained VIVO (Hu et al. 2020), our major technical contribution lies in how leverage visual-linguistic information from BERT to CLIP, realizing the goal of producing novel object captions with sufficient fluency, fidelity and adequacy.
Take Fig. a for example, when the captioning model generates a caption for given image, VLAF2 would regularize and substitute the verb wound for going. ensuring the corresponding lin- guistic fluency. Followed by next training stage in which we randomly samples multiple captions from the same image with reward captions properly designed, VLAF2 would further produce cap- tions with improved fidelity and adequacy. For example, in Fig. [b). the caption in the lower box accurately describes the object red scarf and thus receives a higher reward from CLIP. This encour- ages the model to correctly describe the visual content and capture the associated visual concept of the input image. How the pre-trained models of BERT and CLIP would guide the learning of our VLAF2 will be detailed in the following subsections.
3.1 DESCRIBING NOVEL OBJECTS WITH LINGUISTIC FLUENCY
By observing image-caption pairs (Iz, 3t). the image captioning model in Fig. I(a) would learn the visual grounding (i.e., localization of known objects and referring their expressions) as well the linguistics of captions. With uncaptioned images X,, containing novel objects, we adopt BERT to refine the wordings for captions containing novel objects, followed by CLIP assess the quality of the resulting caption outputs.
While initialized by pre-trained models of VIVO (Hu et al. 2020), our captioning model in Fig. Ta) would recognize images with novel objects but lack sufficient ability in "describing" them in terms of captions. To solve this problem, impose conventional which requires supervision of image-caption pairs (Ii, 31). That is, we have
sgquence-to-squenne objective L
L. CrossEntroppy (ût, y1),
where ộ: (xxx)) denotes the predicted caption. As for uncaptioned images X,, while we do not observe ground truth caption for images containing novel objects, the collocations of the asso- ciated novel objects would be discovered via exploiting the intrinsic knowledge of BERT. Thus, the linguistic fluency of resulting captions can be further improved.
We now detail the learning process for the aforementioned uncaptioned image data, As illustrated in Fig. [ given an uncaptioned image T.. the captioning model C generates a caption S = (x,),, Uc wf. ws,..., w where wf denotes the ith word and the caption length. The superscript represents it is generated by our captioning model. We then obtain a masked caption i.rh {wf, wS..., w ....... w with m indicating the mask index, by randomly masking out the words in the caption. We note that, we do not mask nouns in the above process, since they are viewed as relating to objects grounded the visual content, and language models like BERT are not designed to handle such information. Finally, BERT takes the masked sequence 6. input and recovers the masked word conditioned on the semantics of the entire sentence, producing the refined caption b (ERT("")) o [wf, wh.,.., why -...... w
3

---

Under review as a conference paper at ICLR 2022
Captions
FCIDEs
CCLIP
campleng
laptopa sitting on with white wires.
1/
Captioning Model
Iwo lapilope sitinng on table with table
0.6
Captioned image Z
argmax
CLIP
C
a Blue shirt with a glove and le on
0.5
regilarize
Samping
Tpo
hands and scart blue background.
Uncaplioned image IT
Figure 3: Learning caption novel objects with improved visual-linguistic adequacy and fidelity. For caption-labele image ri, we perform SCST training using CIDEr as reward. The sampled cap- tion 12 will be rewarded by CLIP if has higher cross-modal association than the greedy-decoded baseline Va. The superscript d indicates the source of the image. Additionally, we regularize our model with Teep to avoid redundant and repetitive captions.
It is worth pointing out that, however, not every word substitution from BERT is semantically cor- rect. This is the reason why we require CLIP to validate each replacement output. Specifically, if the refined caption comprises a more accurate and associated word that human generally uses to describe the scene, then a higher CLIP score would be obtained (than that of the original caption). Thus, we propose the objective function LBRET for learning wordings, which calculates the cross- entropy loss for the replaced words, gated by the comparison of the CLIP scores of the two captions. More precisely, LERRT is derived as:
O CLIP( vi) CLIP Vū)
LBRRT CrossEntropy whr), g = [1 if CLIPP vd) CLIP (Iu, vi)
(2)
where CLIP(I, y) calculates the association between image and its caption y, indicating how well the captions match the images. More details can be referred to Radford et al. (202T).
3.2 LEARNING NOVEL OBJECT CAPTIONS WITH FIDELITY AND ADEQUACY
Preserving the adequacy and fidelity of captions for images with novel objects is another challenging task. Recall that, as discussed in Sec. fidelity verifies the correctness of visual content presented in the generated caption, while adequacy assesses whether sufficient visual details have been expressed in it. Conventional requence-to-sequenee model training with cross-entropy loss might not reflect the above desirable properties. This is because that, models trained with word-by-word supervision tend to imitate the sentence patterns of the training images instead of relating caption data to the visual content, leading less optimal caption generation.
To tackle the above issues, we utilize the captioning evaluation metric of CIDEr as the reward in learning framework, since it encourages the generated caption be consistent with that of the human annotated ones in the word level. However, while CIDEr can be easily computed for caption- ing labeled images X,, it cannot be explicitly calculated for captioning uncaptioned images X,, due to the absence of ground-truth captions. Instead, we propose to optimize for fidelity and adequacy of the generated captions via the association between images and captions calculated by CLIP,, couraging captions which precisely describe the objects with plentiful visual details, as discussed in Sec. 2 However, we observe that captioning models would achieve improved association by simply repeating the same object that occurs in the image, which undermines the linguistic fluency of the captions. For example, the image caption "a group of cans of soda and other items on a table" " can be replaced by "a pile of cans and bottles of soda on counter with cans of cans" " with higher CLIP score. Therefore, we additionally impose a repetition penalty to avoid such trivial solutions. We now describe how the rewards are calculated and how do update the captioning models.
Rewards for the generated captions. order to realize the above properties, we design the fol- lowing rewards to reflect the quality of the generated captions. For image-caption pairs (æ1, v1), we directly calculate the CIDEr reward for the predicted caption d (i.e., TCIDE CIDEr (j). ().).. To encourage the generated captions for X = X; U X,, with fidelity and adequacy, exploit CLIP to compute the association between X Xt U X,, and Y
(i.e., FCuP CLIP ())

---

Under review as a conference paper at ICLR 2022
As for repetition penalty to preserve linguistic fluency of the generated captions Iu {wi, w2.... } for Zus we formulate it as a linear assignment problem, where every word is assigned to the most similar one in the sentence except for itself. Then, we calculate the similarity between such pairs for each sentence. Intuitively, repetitive captions would have high similarity scores, since the repeated words will be assigned to the exact same words. We define the assignment á as the one that maximizes the average pairwise similarity of a sentence. Thus, we have:
T T Σ
á arg max a
Cwu,, Wo(())
(3)
where (()) is the index of the word assigned to the i-th word in the caption, and C(w;, wy) is the cosine similarity between the GloVe (Pennington et al.] (0114) word representation two words. Since a desirable captioning model would encourage captions with low repetition (i.e., low average pairwise similarity), the reward for repetition penalty defined as follows:
T
1 T
ree = 1
C(wi, Wa())),
(4)
1=1
Note that we do not calculate repetition penalty for Yi since they are regularized by the afore- mentioned CIDEr rewards. With the above discussions, the total reward for caption-labeled data would be r()) = FewD= 31) + TCLIP(TI, 9r), and the total reward for uncaptioned data would be r(0u.) = FCLI (r., 9.) + (9u)
Back-propagatton via reinforce algorithm. Unfortunately, computation of the aforementioned rewards is non-differentiable Thus, adopt reinforce algorithm (Williams) 1992) to optimize the learning of our model. As shown in Fig. [ for an image x we randomly sample the caption j from the word distribution and use greedy decoding obtain the baseline result ÿ". If the sampled captions possess higher linguistic fluency or cross-modal association than the baseline caption, they
will be encouraged by positive rewards and vice versa. We follow Rennie al. (201): Liu et al. 2017 2018) and define the objective function as follows:
VeLRu(() = - (r(())) - r(()) po (FCInE (0a ya) + FCLIP Da)
if E XI
(5)
r(aa)
(rcar(EEd 9a) (Ua)
if E X.,
where d indicates the source of the image, 6 being the parameters of captioning model, and po po()") represents the predicted word logits for the generated captions. With the objective functions defined equations 22 and (5). our captioning model Cp can be trained accordingly.
EXPERIMENT
Datasets & Implementation Details. The training data for the nocaps benchmark comprises the Open Images V4 (Kuznetsova et al.] (2000) object detection training set (1.7M images annotated with bounding boxes for 600 object classes), plus the image-caption pairs from the COCO Captions 2017 (Chen et al. 2015) training set (118K images containing 80 object classes). No additional image-caption pairs are provided for training. We refer the images from the Open Images dataset to the uncaptioned images X,, we define in Sec. and the image-caption pairs from COCO Captions are defined as (X,, Yi). We only use the Open Images datasets during VIVO pre-training but lever- age both datasets for training as described in Sections 3.1 and 3.2 We evaluate our model on the validation and test set of nocaps, which comprises 4500 and 10600 images from the Open Images validation and test sets, respectively. For the architecture of our captioning model, we follow (Hu) jtt al.) 2020; Li et al.] 2020) ZZhang et al., (02T) to use a BERT-base (Devlin al.) (018)) model. As for CLIP and BERT, we directly exploit the pre-trained models released by their authors. The architecture for BERT BERT-Large, and the version for CLIP ViT/B-32. Due to page limits, hyperparameters and other training details can be found in the Appendix A
4.1 EVALUATION METRICS
CIDEr. Similar to evaluation metrics (Papineni et al. 2002 Lin 2004 Banerjee & Lavie 2005)
for NLP tasks, Consensus-based Image Description Evaluation (CIDEr) caleulates the similarity
5

---

Under review as a conference paper at ICLR 2022
Table I: Quantitative results nocaps. The numbers before/after slashes denote scores derived without/wwth Constrained Beam Search (CBS).
m-domain
domain
out-of-domain
overall CIDEr
Method
CIDER
SPICE
CIDEr
SPICE
CIDE
SPICE
SPICE
Validation Set
UpDown
-/79.3
12.4
-773.8
1/11.4
-/71.7
-99.9
-/74.3
-/11.2
Oscara
-/ 83.4
1/.0
-/81.6
-/ 12.0
-/77.6
1/.0
-/ 8/111
-/11.
Oscar
85.4
11.9
84.0
11.7
10.0
13.4
1/.4
Oscarg + VIVO
/92.2
12.9
- /87.8
-/ 12.6
87.5
11.5
- 88.3
- 12.4
VinVL
97.9 /96.8
13.2/ 13.5
89.2 19.2/90.3
12.9/ 13.1
68.4 87.4
10.8 / 11.6
86.1 / 90.9
12.5 12.8
ViaVI VIVO
95.8 /94.8
13.3/
99.5/
12.8/
11.1
88.6/914
12.5/125 12.7
Human
84.4
14.3
85.0
14.3
95.7
14.0
87.1
14.2
102.8 / 101.4
14.8 15.1
97.9 /9688
14.4/ 14.5
86.3/95.4
12.5/12.9
96.3/97.2
14.1
14.2
Test Set
UpDown
- 7/6.0
11.8
-77.22
11.5
-666.7
-19.7
7/3.11
11.2
Oscara
-/31
11.9
79966
11.9
773.6
10.6
- /78.8
- 11.7
Oscar
-/ 84.8
12.1
-/ 82.1
-/ 11.5
-/73.8
-19.7
-/ 80.9
1/11.
Oscar VIVO
89.0
12.9
87.8
12.6
8/0
1111
12.4
VinVL
93.0 /9.8
13.3/13.3
84.7 / 89.0
12.7/
64.0/66.1 66.1
10.9 / 10,9
82.0 85.5
12.4/12.5
VinVL + VIVO
94.5 / 84.4
13.1 12.8
90.9 86.0
12.9 12.6
73.9 77.9
11,2/ 11.3
88.3 / 84,4
12.6/ 12,4
84.6
Ours
101.7 / 90.4 15.0 14.3
95.7 /91.0
14.4/
14.0
78.9 82.5
12.1 / 12.2
93.5 / 89.4
14.1 13.7
Table 2: Quantitative comparisons on caption fluency, fidelity and adequacy. Note that BLEU @4 (B @ and CIDEr (C) are utilize for describing fluency, object precision (P) for fidelity, object recall (R) for adequacy and object Fl scores (FI) for overall cross-modal association.
in-domain
near-domain
out-of-domain
Method
B@4
C
P
F1
B@4
C
P
R
F1
B@4
C
R
Fl
ViaVL
32.6
63.2
59.2
40.8
48.3
30.5
58.3
22.8
32.6
26.8
29.6
48.8
48.4
25.6
33,5
VinVI+VIVO
31.2
59.8
56.0
42.2
48.1
30.1
57.3
28.5
36.3
32 .0
27.3
45.6
49.0
27.3
35.1
Ours
35,9
68.0
58.2
45.6
51.3
32.2
60.8
39,9
41.0
40.4
30.1
50.2
51.3
30.5
38.3
between the reference and generated caption by word n-gram overlap in a rule-based manner. To capture human consensus for image captioning evaluation, it introduces the tf-idf weight to reduce the matching weight of the n-grams that are common in all image captions.
SPICE. Semantic Propositional Image Caption Evaluation (SPICE) (Anderson et al. 2016b) matches the semantics between sentences, such as objects, relations, and attributes of objects.
Specifically, it converts sentences into semantic scene graphs, which allows evaluation to break grammatical constraints and focuses on propositional semantic content. It reflects the accuracy of the visual content and considers less about linguistic properties.
Fluency. To quantitatively evaluate fluency, we remove the effect of the visual information and focus on the quality of linguistic properties in the conventional caption evaluation metrics. Specifically. we remove all the objects and nouns from the captions and report BLEU@4 (Papineni et al.) 2002) and CIDEr scores calculated by the removed version reference and candidate captions. Note that the fluency experiment is conducted on a subset of the nocaps validation set, which contains 1000 images whose caption annotations are available on the official website of the nocaps dataset.
Fidelity & Adequacy. Fidelity and adequacy evaluate how well the captions are associated with images. As defined in Sec. 2 fidelity stands for the fraction of relevant objects described cap- tions among all the objects in captions, and adequacy is the fraction of relevant instances that were retrieved. These two properties are analogous the definition of precision and recall, respectively. Therefore, extract the objects mentioned in the captions and the ground-truth objects in the im- ages and calculate the precision (for fidelity), recall (for adequacy), and Fl (for overall association) scores. The experiment is performed on the validation set of nocaps.
Following Agrawal et al. (201)), we further split the dataset into three subsets for evaluation: in- domain images only contain the objects that have been described in the training captions, our- of-domain images only with unseen (i.c., novel) objects presented, and noma-n ones containing both seen and unseen objects.

---

Under review as a conference paper at ICLR 2022
Table 3: Analyses on BERT, CLIP, and repetition penalty for NOC using nocaps validation set. Note that BERT mainly benefits the linguistic fluency with improved CIDEr, and CLIP is desirable for preserving visual semantics with increased SPICE.
in-domain
near aear-domain
out-of-domain
overall CIDEr
Method
CIDEr
SPICE
CIDEr
SPICE
CIDEr
SPICE
SPICE
Ours
102.77
14.83
97.90
14.40
86.33
12.54
96.25
14.10
Ours w/o BBRR
99.13
14.41
94.72
14.11
84.47
12.38
93.27
13.81
Ours w/o rCuP
101.12
13.80
94.07
13.35
80.54
11.94
92.33
13.14
Ours w/o Tiep
96.73
14.83
89.64
14.12
81.87
12.38
89.08
13.88
Table 4: Analyses on BERT and CLIP for improving caption fluency, fidelity and adequacy. Note
that BERT benefits fluency metries of BLEU@4 (B (B@4) and CIDEr (C), while CLIP focusing on cross-modal association boosts metrics of object precision (P), recall (R), and scores (FI).
in-domain
neur-domain
out-of-domain
Method
B@4
R
F1
B@4
F1
B@4
P
Ours
35.9
68.0
58.2
45.6
51.3
32.2
60.8
39.9
41.0
40.4
30.1
50.2
51.3
30.5
38.3
Ours w/o Cawer
32.8
64.8
58.1
40.9
48
30.4
60.6
39.0
40.0
27.9
49.1
51.6
27.9
36.2
Ours w/o FrsuP
33.2
65.9
58.8
42.1
49.1
31.9
60.7
35.8
37.7
36.8
28.7
49.8
51.6
27.4
35.8
4.2
QUANTITATIVE ANALYSIS
For performance comparisons, we choose UpDown (Agrawal et al. 2019) without SCST optimiza- tion (Rennie et al. 2017) and Oscar (Li et al. 2020) as baselines, as well as VL (Zhang et al.) 2021] that achieves SOTA results on the benchmark of nocaps. In addition, VIVO eHu at. 2020) is a pre-training technique for captioning models, allowing them to recognize the novel objects. Since VL did not report the numbers with Constrained Beam Search (CBS) exploited during infer- ence (CBS is known to improve model performance on out-of-domain data), we reproduce VLn following details stated in the original paper. For more details please refer to Appendix [A] For a comprehensive comparison, we conduct experiments addition, we compare our method with VL. fluency, fidelity, and adequacy to demonstrate the improvement in terms of these properties. We also evaluate the COCO Caption dataset and report in Appendix B
the validation and test set of nocaps. In
The nocaps datasets. The results nocaps are shown in Table [ From this table,
that our model performed favorably against baselines and SOTAs across different metrics. We observe that CBS slightly decreased model performance on seen object captioning, since it forces captioning models to describe the detected objects without considering detection correctness. Nevertheless, for near or out-of-domain images, CBS still benefits the captioning performances. is worth noting that our model largely increased the performance in SPICE score for every data domain, which verifies that our method is able to generate captions with the improved lmage-lagguage association.
we design additional experiments for evaluating fluency, fidelity, and adequacy and report the numbers in Table For fluency, we remove all the objects and nouns in the captions since they relate less to the linguistics of the captions. We then calculate the BLEU@4 (B @4) and CIDEr (C) scores for the captions after removal. For fidelity and adequacy, they indicate that captions should accurately (high precision) describe sufficient (high recall) visual details. Therefore, we report the object precision and recall in this table, and object Fl scores represent the overall association between captions and images. One can see that our method surpass previous methods by a visible margin on all tasks except for in-domain object precision., which further verifies our model improves novel object captioning on fluency, fidelity, and adequacy.
Fluency, fidelity, and adequacy. As described in Sec. 4.
4.3 REMARKS ON BERT AND CLIP FOR CAPTION FIDELITY, ADEQUACY AND FLUENCY
We conduct ablation studies on the nocaps validation set, with the aim to verify the necessity of integrating BERT and CLIP for learning NOC models. Following the same evaluation procedures described in Sec. 4.1 discuss the contributions of these models in terms linguistic and semantic level metrics in Tables and A Detailed ablation analysis of every objective can be further found in Appendix B.2
7

---

Under review as a conference paper at ICLR 2022
Fluency
Fidelity
Adequacy
celling fan room with window
blanket and a glove on blue shirt.
man in suit and tie standing in room.
VinLL
ceiling fan hanging from the ceiling in room.
two glove hands and red scarf on blue background.
man in suit and tie standligg at podium
Ours
Fluency
Fidelity
Adequacy
group of women standing on field with a football.
woman sitting at tennis table with racket.
man and woman playing an accordion.
VieVL
group girls football uniforms posing for picture.
woman wheelchair playing tennis table.
couple of people sitting in church playing accordion.
Ours
Figure 4:
Example results and comparisons for image captions produced by VL and ours in terms of fluency, fidelity and adequacy. Note that both utilize VIVO for novel object detection.
BERT. As shown Table the captioning model without BERT would observe a performance drop in CIDEr for linguistic fluency. but such drops for the metric of SPICE (related to visual content) would be less significant. Similarly, as observed in Table 4 removing BERT would result in the lowest BLEU and CIDEr scores. These results confirm motivation and model design discussed in Sec. since BERT utilized to improve caption quality at the linguistics level.
CLIP. As shown in Table the model without CLIP showed significant drops in captioning metrics of CIDEr and SPICE. However, the performance decrease in SPICE is expected, since CLIP is particularly deployed in our framework for preserving visual content in captions. As for CIDEr, its decrease is mainly due to the deterioration of missing visual content in captions. This is also confirmed by Table in which mainly the metrics reflecting fidelity and adequacy (i.e., at the visual semantics level) would observe significant drops for model trained without CLIP rewards.
Repetition penalty. Recall that, in Sec. 3.2, this penalty and captions with redundant visual information. As seen in Table 3 the model without this penalty observed significant performance drop CIDEr scores. We did not see such trends for SPICE. This is because that, repetitive words in captions mainly violate linguistic structures rather then semantic accuracy, and thus the performance related linguistic fluency would be more sensitive to the deployment of this penalty.
alleviate association between images
4.4 QUALITATIVE ANALYSIS
We now empirically show captions Fig. which are generated by our model and VLn with both pretrained from VIVO for novel object detection. In this figure, wordings that less accurate or incorrectly describe the associated visual content are marked bold. And, our wording improve- that for fluency, model generated
ments are highlighted in red. From this figure,
8

---

Under review as a conference paper at ICLR 2022
vivid captions with more proper wordings. Take the upper-left image for example, our model partic- ularly described "fan hanging from the ceiling in a room" instead of "fan in a room". As for fidelity, model is designed to capture the visual content in an image. Specifically, take the second im- age in the first row for example, we correctly described the number of gloves and the novel object red scarf. while VIn failed to do so. As for adequacy, take the bottom-right image for example, our model was able to recover visual details in the image (i.e., "people playing the accordion" " and "sitting in a church "). For more qualitative examples, please refer to Appendix B.5
5 RELATED WORK
Image captioning. Recent progress of image captioning focuses different model architectures and learning methods. Gao et al.
22019;;
Huang et al. (2019); Wang et al. (2019); Guo et al.] (2020);
Pan et al. (2020): Cormia et al.
2020) design different attention mechanisms for image caption-
i Rennie et al. (2017] the performance. On the other hand, some researchers consider more challenging settings, such as partially supervised (Liu et al.] 2018: Kim et al., 2019) or unpaired image captioning (Gu et al.) 2019 |Fng et al.) 2019). However, these methods restricted the assumption that the unpaired images and captions share the same set of object class, and the number of object class is limited as well, which make them inapplicable to our task.
: Li et al.
2019); Yang et al. (2020) adopt reinforcement learning to improve
Novel object captioning. Previously, novel object captioning approaches (Anderson et al. 20 2018 Hendricks et al., 2016) were only tested on a restrictive dataset with only eight novel object classes held out from the COCO dataset. Their extensions to large-scale image data with various novel objects are not sufficiently studied. Recent studies mainly rely on object detection results to improve the performance on novel object captioning. Lu et al. (2018); ;): Wu et al. (2018) generate slotted caption templates, which are later filled in with visual concepts identified by object detec- tors. Yao al. (0017) exploits copying mechanism to assemble words corresponding to object detector predictions to generate captions. Similarly, Constrained Beam Search (CBS) (Anderson let al. 2016a) is an architecture-agnosticc decoding algorithm that can be exploited during inference to enforce the inclusion of novel object classes in the captions. Instead of explicitly using detection results. Hu et al. (2020) learns the relationship between image and text by aligning object detec- tion tags with their corresponding image region features. Recently. Wang et al. (2021a) indicate that a desirable caption should comprise properties of fluency, fidelity, and adequacy. Nevertheless, most existing NOC approaches are not designed to handle language expression and cross-modal association with the above properties preserved.
Vision and Language Pre-training (VLP). Existing VLP works can be classified into two streams. One is based on the dual-encoder architecture, such as CLIP (Radford al. (02T) and ALIGN (Ta
et al., 2021). utilizing features encoded in each modality followed by contrastive learning (orrd
et al. 2018) for alignment purposes. On the other hand, models like Zhoo et al. (2020): Li et al.
(2020); ; iZhang et al. (202T); Yang et al. (2021) exploit multiple cross-attention layers to learn the relationship between images and text, and show impressive performances on image-text matching tasks. Due to the efficiency of the former models in handling data across modalities, we adopt CLIP for cross-modal association in this paper.
CONCLUSION
A risual-linguiste learning framework with improve d adequacy, fidelity, and fluency (VLAF2) is presented for novel object captioning. We fundamentally quantify the above properties and relate them to the models of BERT and CLIP. We propose objectives and rewards reflecting the desir- able linguistic fluency and visual semantics for NOC. Guided by BERT, our model learns to refine wordings of novel objects. Via reinforce algorithms, we have CLiP-based rewards assess the cor- rectness of visual content described in the generated caption. Empirically, we showed that our model achieved SOTA results on the nocaps benchmark. We further provided analyses on both BERT and CLIP, verifying the necessity of their integration for learning novel object captions. A future di- rection of this work would be utilizing both captioned and uncaptioned images with self-supervised learning strategies for training NOC models.
9

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
We acknowledge that all authors of this work have read and commit to adhering to the ICLR code of ethics. Our work has no ethical concern.
REPRODUCIBILITY STATEMENT
For reproducing our results in the experiment section, codes and training/testing scripts are provided the supplementary materials.
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8948-8957, 2019.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Guided open vocabulary image captioning with constrained beam search. arXiv preprint arXiv:1612.00576 2016a.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propo- sitional image caption evaluation. In European conference computer vision, pp. 382-398. Springer, 2016b.
Peter Anderson. Stephen Gould, and Mark Johnson. Partially-supervised image captioning. arXiv preprint urXiv:/806.06004 2018.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00:: 2015.
Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory trans- former for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10578-10587, 2020.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1 04805, 2018.
Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. Unsupervised image captioning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4125-4134, 2019.
Jerome H Friedman. The elements of statistical learning: Data mining, inference, and prediction. springer open, 2017.
Lianli Gao, Kaixuan Fan, Jingkuan Song, Xianglong Liu, Xing Xu, and Heng Tao Shen. Deliberate attention networks for image captioning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp.
2019.
Jiuxiang Gu, Shafiq Joty, Jianfei Cai, Handong Zhao, Xu Yang, and Gang Wang. Unpaired image captioning via scene graph alignments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10323-10332. 2019.
Longteng Guo, Jing Liu, Xinxin Zhu, Peng Yao, Shichen Lu, and Hanqing Lu. Normalized and geometry-aware self-attention network for image captioning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10327-10336. 2020.
10

---

Under review as a conference paper at ICLR 2022
Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Darrell. Deep compositional captioning: Describing novel object categories without paired training data. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-10, 2016.
Xiaowei Hu, Xi Yin, Kevin Lin, Lijuan Wang, Lei Zhang, Jianfeng Gao, and Zicheng Liu, Vivo: Surpassing human performance in novel object captioning with visual vocabulary pre-training. arXiv e-prints, pp. arXiv-2009, 2020.
Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. Attention on attention for image cap- tioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 4634-4643, 2019.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:
2021.
Dong-Jin Kim, Jinsoo Choi, Tac-Hyun Oh, and In So Kweon. Image captioning with very scarce pervised data: Adversarial supe-super learning approach. arXiv preprint urXiv:1909 02201, 2019.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi bustt, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 128(7):1956-198) 1981, 2020.
Nannan Li, Zhenzhong Chen, and Shan Liu. Meta learning for image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8626-8633, 2019.
Xiujun Li, Xi Yin, Chumyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Semantisemantics aligned pre-training for languula tasks. In European Conference on Computer Vision, pp. 121- Springer, 2020.
Chin-Yes Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74 81, 2004.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image cap- tioning via policy gradient optimization of spider. In Proceedings of the IEEE international conference on computer vision, pp. 873-881, 2017.
Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, and Xiaogang Wang. Show, tell and dis- criminate: Image captioning by self-retrieval with partially labeled data. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 338-354, 2018
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7219-7228, 2018
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1 2018.
Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. X-linear attention networks for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10971-10980, 2020.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
11

---

Under review as a conference paper at ICLR 2022
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint wxXi:2203300000 2021.
Steven Rennie, Etienne Marcheret, Youssef Mrouch, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7008-7024, 2017.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp- 2018
Guanglu Song, Yu Liu, and Xiaogang Wang. Revisiting the sibling head in object detector. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11563- 11572, 2020.
Sijin Wang, Ziwei Yao, Ruiping Wang, Zhongqin Wu, and Xilin Chen. Faier: Fidelity and adequacy ensured image caption evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14050-14059, 2021a.
Weixuan Wang, Zhihong Chen, and Haifeng Hu. Hierarchical attention network for image caption- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8957-8964 8964, 2019.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10900 2021b.
J Williams. Simple statistical folllent- algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-256 256, 1992.
Yu Wu, Linchao Zhu, Lu Jiang. and Yi Yang. Decoupled novel object captioner. In Proceedings of the 26th ACM international conference Multimedia, pp. 1029-1037, 2018.
Xuewen Yang, Heming Zhang, Di Jin, Yingru Liu, Chi-Hao Wu, Jianchao Tan, Dongliang Xie, Jue Wang, and Xin Wang. Fashion captioning: Towards generating accurate descriptions with 2020: 16th European Conference, Glasgow, UK,
semantic rewards. In Computer August 23-28, 2020, Proceedings, Part XIII 16, pp. 1-17. Springer, 2020.
Zhengyuan Yang. Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. Tap: Text-aware pre-training for text-vqa and text-caption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8751-8761, 2021.
Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Incorporating copying mechanism in image cap- tioning for learning novel objects. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6580-6588, 2017.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3579-5588, 2021.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 34, pp. 13041-13049 2020.
12

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Learning to caption novel objects with linguistic fluency
Input: Captioning model Cg(+), Pre-trained BERT, and Pre-trained CLIP.
Data: Captioned image 21, the corresponding GT caption yts uncaptioned image zu. and Ir The
Output: Trained Captioning model Co(+).
Initialize Ca(+);
for it from to num_iters do
0t Cx(x)), Va Carw):
Produce D/ by randomly masking words in the sentence in Kü (except for nouns);
8h + BERT("")
Lezs - CrossEntropy((j y/)
CL, K4) < CLIP (2) then
CRERT t
else
10
LBRET < CrossEntropy(SG..
end
12
C C-2ss- + LBER
Update parameters: 0 A0,m(0,m Mat. VoC)
14
end
Algorithm 2: Learning novel object captions with fidelity and adequacy
Input: Captioning model C-(+) and Pre-trained CLIP.
Data: Captioned image 2t, the corresponding GT caption yı, uncaptioned image x., and Ir the
Output: Trained Captioning model Cg(-).
Initialize Co(-);
for il from to num iters do
of <- Co(x)), if, +- (o(wa) (by sampling);
0f < (ar)), 18 < Co(ar,) ) (by greedy decoding);
Calculate Trep (V and Trep by 4
r(in) VCIDEr (in, m1) VCLIP( 91)
r(ūw)) < TCLPP Ip (Zin Uh) ) Tiep (Ua)
Calculate the gradient VoLRI + --(((())-(()))))) log (og)o(()), d (1,u)
Update parameters: O <- Adam(O, lht. VoLn1)
10 end
A IMPLEMENTATION DETAILS
Following Hu et al. (202) Li et al. (2020) Zhang et al. (2021), we consider a BERT-base (Devlin et al. 2018) architecture for captioning model. Given image, the captioning model jointly takes the image region features and the predicted detection tags to generate the associated caption.
We the same region features VinVL (Zhang al., [221)). which released on their project page. Since the object detection model Omni-detection used in previous works (Hu et al. 2020 (Song
Zhang et al. 202T is not available, we replace it with publicly available model of TSD
et al. 2020 to generate the object detection tag.
Reproducing our method. We perform VIVO (Hu al. 2020) pre-training for 100 epochs with a batch size of 1024 and learning rate of 5 x 10 30 which are exactly the same as the parameters stated in the VIVO paper. After that, we propose train our model following the training process described in Algorithm (T learn to caption novel objects with linguistic fluency. We train our model for 20 epochs with an effective batch size of 512 (256 caption-labeles images and 256 uncaptioned images) and a learning rate of 1.5 x 10 Then, to learn novel object captions with fidelity and adequacy, we train our model as decsribed epochs with effective batch size of 128 (64 Algorithm 2 Specifically, we train our model for
images and 64 uncaptioned images) and a learning rate of 2.5 x 10 We use 8 V100 GPUs to perform the above training algorithms. Codes be found in the supplementary materials.
13

---

Under review as a conference paper at ICLR 2022
Table 5: Image captioning evaluation results COCO "Karpathy" test split.
BLEU@4
METEOR
ROUGE-L
CIDEr
SPICE
VinVL
39.8
29.9
59.6
134.6
23.9
VL
39.7
29.9
59.6
134.5
23.8
Ours
40.0
30.4
60.2
137.3
24.5
Table 6: Ablation studies on nocaps validation set.
in-domain
near-domain
out-of-domain
overall
Method
CIDEr
SPICE
CIDEr
SPICE
CIDEr
SPICE
CIDEr
SPICE
Baseline (Only w/ Le2s)
89.07
13.29
83.29
12.61
68.77
10.59
81.17
12.32
..BERT
92.46
13.40
85.79
12.92
73.21
11.40
84.20
12.69
FCIDEr
101.19
13.84
95.38
13.44
83.24
12.06
93.75
13.23
TCLIP
96.73
14.83
89.64
14.12
81.87
12.38
89.08
13.88
Trep (Ours)
102.77
14.83
97.90
14.40
86.33
12.54
96.25
14.10
Reproducing baseline methods. For VLnVL (Zhang et al. 2021 we leverage the released model on their project page and directly inference on the nocaps dataset. However, for VLn (Zhang et al. 202Ip. since the pre-trate pre-trained model is not publicily available, we reproduce this method using the image region features and object detection tags generated by models mentioned in the beginning of this section to train this model. Specifically, the model is trained for 160K iterations (about 100 epochs) with batch size of 1024 and a learning rate of 5 x 10 5, and fine-tuned for 30 epochs with batch size of 256 and learning rate of 10 using the cross-entropy loss. Last, we perform the SCST optimization (Rennie et al., 2017)) with learning rate of 2 x 10 for 5 epochs to obtain the final model. The numbers reported in Table are derived using this version model.
B ADDITIONAL EXPERIMENTS
B. EXPERIMENTS ON THE COCO CAPTION DATASET
To validate that our method generalize well on the task of describing the seen objects, we conduct experiments the COCO Caption test set and report the numbers in Ta The training data for VIn (Zhagg al.) (021) ) image caption pairs from the COCO (Lin al. (014)) dataset. While for VL+ VIVO and method, additionally leverage the uncaptioned image from the Open Images (Kuznetoova et al. 2020) dataset as extra data. One can see that our method outperforms the other competitive approaches different metrics which verifies the effectiveness of our approach.
B.2 DETAILED ABLATION ANALYSIS
Table lists the performances and compares contributions of the imposed objectives in VLAF2. The baseline model in Table 6 is only trained on the COCO Caption dataset using the sequence-to sequence objective. To confirm our introduction of exploiting BERT to learn the associated wordings of novel images, we apply LEERT to the baseline model, and report the results in the second row of Table 6 The CIDEr scores improve significantly after adopting reinforce algorithm (Williams) 1992) and using CIDEr scores of the generated captions as reward, and the results are in the third row. One can see that the SPICE scores largely increase but the CIDEr scores slightly decrease after the deployment of CLIP. We hypothesize that the captioning model properly captures the visual content images, but describes the scene with poor linguistic fluency. As the discussion in Sec. 3.2. we attribute the performance drop to the degenerate solution of increasing the association between the captions and the corresponding images. Note that we further consider the repetition penalty to regularize the captioning model. The results are shown in the last row of Table One can see that this regularization slightly improves the SPICE scores but significantly increase the CIDEr scores. By comparing the performances listed in Table 6 see that the full version of our VLAF2 achieved the best performance in terms of CIDEr and SPICE. Thus, the design our VLAF2 can be successfully verified.
14

---

Under review as a conference paper at ICLR 2022
Table 7: Quantitative results the nocaps (XD) test set.
overall
Method
CIDEr
SPICE
UpDown Agrawal al. 2019)
73.09
11.20
SimVLM basc (Wang al.
202
94.80
13.10
VIVO (Hu al. 2020)
100.12
14.04
Ours
96.25
14.10
Ours (+CC)
102.39
14.71
Table 8: Ablation studies of the joint-training model on nocaps validation set.
in-domain
near-domain
out-of-domain
overall
Method
CIDEr
SPICE
CIDEr
SPICE
CIDEr
SPICE
CIDEr
SPICE
Baseline (Only w/ Tu2a)
96.1
13.71
90.35
13.41
79.96
11.77
89.07
13.13
+ .LBERT
99.44
13.91
91.13
13.53
81.11
11.82
90.29
13.25
+ -CDDer
109.14
14.52
100.66
14.08
88.61
12.69
99.43
13.87
TCLIP
103.81
15.99
98.91
15.32
93.17
13.67
98.45
15.09
+ TTop (Ours)
110.56
15.23
105.16
14.81
96.22
13.19
104.12
14.55
B.3 EXPERIMENTS ON THE NOCAPS (XD) BENCIIARK
To investigate the limits of performance on nocaps without any restraints on the training datasets, conduct experiments the nocaps (XD) benchmark verify the effectiveness of our method when more image-caption pairs are considered. Specifically, we additionally consider Conceptual Captions (CC) (Sharma al.) 2018) as labeled training samples X, in our learning framework and perform boint-raining to see if extra image-caption pairs benefit the model on novel object captioning. The results are shown in Table Note that both VIVO, Sim VLM and our method adopt BERT-based architecture as the backbones for captioning. As for the training set, since VIVO (Hu ct al. 2020) did not specify the dataset details for their evaluation for the nocaps (XD), we compared method to Sim, which applied much larger web-scale dataset (I.8B image-text pairs) than the CC dataset (3. IM pairs). Yet, our method still performs favorably against SOTAs on the nocaps (XD) protocol and benchmark, verifying the effectiveness of our method even if more image-caption pairs are considered.
In addition, to quantitatively show that the performance gain in Table 7 is not simply contributed by the additional data we considered, we ablate our model on the nocaps validation set and show the results in T We observed similar performance trend as reported in Table where LBERT slightly improves the CIDEr scores, and FCLP siginificantly boost SPICE but slightly deteriorates the CIDEr scores. One can see that the regularization Trep slightly improves the SPICE scores but significantly increase the CIDEr. By comparing the performances listed in Table 6 and Table B we see that our design of distilling BERT enhance fluency (in terms of CIDEr) and the uses of CLIP to encourage captions with sufficient fidelity and adequacy (in terms of SPICE) still function properly when more diverse image-caption pairs are considered, verifying the design of our VLAFF2.
B.4 HUMAN STUDY
To conduct human study, we randomly picked 60 images from the nocaps validation set, and compared the captions generated by our method to those generated by the SOTA of VinVL+VIVO (Zhang et al. (02T), and the human-annotated captions provided by the nocaps dataset. Following the evaluation protocols used in the COCO Captioning Challenge 2015 (Lin ct al. 2014), we designed different metrics and asked individuals evaluate captions from these aspects. The following are the four metrics we used in the experiment: MI: Is the caption generated by human (0: machine, 1: human)? (Percentage of captions that pass the Turing Test.) M2: Rate the correctness of the captions ities are correct. M3: Rate the amount of detail the captions on a scale 1-5 (lack details very detailed): Whether the caption has detailed all the objects and their attributes. M4: Rate the fluency
scale 1-5 (incorrect-correct)) Whether the described objects activ-
15

---

Under review as a conference paper at ICLR 2022
Table 9: Human study the nocaps validation set.
Method
MI (Turing Test)
M2 (Fluency)
M3 (Fidelity)
M4 (Adequacy)
Vin
0.25
3.99
3.70
3.46
Ours
0.43
4.06
4.33
4.24
Human
0.53
4.09
4.44
4.18
of the captions on scale 1-5 (lack of flueney-very fluent): Whether the caption use phrases/words that human generally would use to describe the scene, i.c., the caption is linguistically natural and fluent.
Specifically, M2, M3, M4 correspond to the fidelity, adequacy, and fluency, respectively, which are the particular objectives desired to be achieved. We asked 24 people two answer different questionnaires, and each questionnaire contains 10 captions from each method (i.e., ours, sota, and human caption presented a random order).
report the results Table 9 We see that our method surpassed the SOTA by clear margins, while our performances were comparable to those the human ones across different metrics. This further supports the design of our model for NOC with sufficient fluency, fidelity, and adequacy.
B.5 MORE QUALITATIVE RESULTS
Qualitative comparison on fluency, fidelity and adequacy. In this part, we provide more quali- tative results on the nocaps validation/test set, and the results are shown in Fig. and 6. Note that wordings that are less accurate or incorrectly describe the associated visual content are marked in bold. And, our wording improvements are highlighted in red. Take results in the bottom row of Fig. 5 for example. For the column of fluency, our model particularly described the turtle being "crawl- ing on some rocks" instead of "sitting on the top of a beach". For fidelity, our model predicted the background preferably As for the column of adequacy, though both captions described a young men running, our model successfully captures more details in the image (i.e., "there are number their shirts"), For more qualitative results, please refer to Fig. 6.
"race track" instead of "streer" from the prediction of VinVL model.
Qualitative results of some failure cases. In this part, demonstrate some failure cases of VLAF2 model. We empirically observe that the failure cases mainly come from the wrong/missing detection tags predicted by the pre-trained object detectors. To be more specific, the captioning model largely relies on the detection tags as clues to correctly describe novel objects. Take the result in the left-side of Fig. for example, the detection model falsely recognizes the raccoon squirrel, and this detection result consequently damages the caption prediction. Therefore, how to jointly improve the detection model and captioning model is still open question, and leave this problem for future research. For more failure cases, please refer to Fig. 7.
16

---

Under review as a conference paper at ICLR 2022
Fluency
Fidelity
Adequacy
a group of men standing in a field with watermelon.
woman standing in front of computer screen.
woman sitting on piano.
VinL
group of men cutting up watermelon in a field.
a woman standing in front of projector screen with a presentation.
woman sitting front of piano playing a keyboard.
358
Fluency
Fidelity
Adequacy
tortoiset sitting on top of beach.
group of men running down a street.
group young men running
field.
large turtle crawling on some rocks in the dirt.
group of men running down race track.
group of young men running in the grass with numbers their shirts.
Figure 5:
Example results and comparisons for image captions produced by VL and ours in terms of fluency, fidelity and adequacy. Note that both utilize VIVO for novel object detection.
17

---

Under review as a conference paper at ICLR 2022
Fluency
Fidelity
Adequacy
couple of people sitting on a red bike.
black vase sitting on top table.
man riding pink bike in the street.
VinL
a couple people riding on a bike.
yellow lamp with light bulb on black background.
riding bike street with helmet.
Ours
Fluency
Fidelity
Adequacy
yellow bee sitting on top of blue flowers.
woman holding cell phone in her hand.
VenL
wearing hat and table.
yellow bee flying next bunch of purple flowers.
woman wearing a brown hat and smiling.
woman wearing brown jacket and boots holding a cell phone.
Example results and comparisons for image captions produced by Vin and ours in terms of fluency, fidelity and adequacy. Note that both utilize VIVO novel object detection.
Figure 6:
Detection
Man, Football helmet, Sports uniform, Baseball glove, Baseball bat
Tree, Monkey, squirrel
Dog, Camivore, Lion, Brown bear
tags
Raccoon
lacrosse
Jaguar
brown squirrel sitting tree branch.
man holding tennis racket on a field
collage pictures of dogs and lion.
Figure 7: False captions misled by the wrong object detection tags.
18


Reviews:
[{"review_id": "t9hdm2IGah", "reviewer": "Reviewer_2cGW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros:\n- Leveraging prior visual-linguistic knowledge from pre-trained BERT and CLIP models for the task of novel object captioning. The association between fidelity, fluency and adequacy with the training tasks of BERT and CLIP presents a good motivation for the method adopted. \n\n- The reward based loss functions designed for encouraging caption novelty and fluency and to discourage repetitive captions is interesting and novel. \n\n- The authors design fidelity/fluency and adequacy quantitative evaluation to measure the efficacy of their approach compared to previous work and show improvements. \n\n### Clarifications:\n- Is the BERT model finetuned with the captions from the nocaps dataset in eq 2?  How is the CLIP condition satisfied in eq2, what happens if both the captions are unreliable ? Is there a threshold above which the CLIP scores are compared ?\n\n- The BERT equations also could fulfil the fidelity objective because of the gating function. Is this observed?\n\n- Are the improvements from Eq5. coming because of the r_CIDER term in the labelled caption data? Are the results in Table 1 compared with CIDER optimization for baselines? \n\n- What is observed if the reward is not calculated over the labeled training samples X_l?\n\n- How will  the model perform if Eq 2 is performed during the inference stage only from a pre-trained captioning model?\n\n- In fig 1, it is not very clear where the second caption “two glove hands...” comes from, is one from greedy decoding and other from sampling  ? \n\n- Are the novel words mostly objects/nouns?\n\n### Cons: \n\n- While the method is effective and simple, there are certain ablations which could have been performed qualitatively and quantitatively to draw where the improvements come from. \n\n- The improvements in the method over the SOTA are via CIDER optimization on labeled examples in Eq 5. If the baselines are trained with CIDEr optimization, it might improve overall performance as well. \n\n- The motivation of improving fluency or adequacy is probably not only related to novel image captioning but for image captioning in general. Could this method help to address those problems there? \n\n- As stated in the paper, that previous methods produce grammatically incorrect captions for NOC, that does not seem to be the case (eg in fig 4). There are details missing from baseline captions but they are still fluent. \n", "summary_of_the_paper": "The paper proposes to improve novel object captioning i.e. describing objects/contents that are not seen during training. Specifically, the authors propose to correct the captions generated from captioning models using pre-trained BERT and CLIP models by rewarding the captions with precise and rich visual content. They show qualitative and quantitative improvements on CiDER and SPICE scores compared to previous work and propose to measure the fluency, fidelity and adequacy of the generated captions for a more comprehensive evaluation of novel object captioning.", "summary_of_the_review": "The paper is well written with detailed experimental results to support the claim. The paper proposes interesting ideas to improve NOC and is marginally novel but there are still some concerns as stated in the main review. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bErCDeOTQmR", "reviewer": "Reviewer_1659", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Novel object captioning is one sub-task of image captioning and generating captions that are more fluency, fidelity, and adequacy is one sub-requirement of captioning. However, this method exploits two large-scale models BERT and CLIP to solve this small task, which is not an economic solution. In this way, I think the motivation of the research is not quite significant. Actually, not only novel object captions are not fluency, fidelity, and adequacy, even the captions generated by standard captioning settings (trained by MSCOCO) may also have low fluency, fidelity, and adequacy (Or say that standard captions' fluency, fidelity, and adequacy can be further improved). It would be better if the authors could apply their method to standard captioning or more settings which have limited training data other than only one novel object setting.\n2. In Sec 3.1, the authors mention that they mask certain words instead of nouns since nouns may appear in the image, which should not be generated by BERT. However, the attribute and relation words may also appear in the image, which should not be masked. For example, some horses may be white and some others may be dark. Or in some images, a man ''ride'' a horse while in another image a man ''stand near'' a horse. Then these attributes and relations should also not be masked.\n3. At the end of the first paragraph in Sec 3.2, the author claims that ''word-by-word supervision tend to imitate the sentence patterns of the training images instead of relating caption data to the visual content'' while do not provide suitable evidence. \n", "summary_of_the_paper": "This paper tries to generate novel object captions that meantime satisfy three aspects: fluency, fidelity, and adequacy. To achieve this goal, it exploits two large-scale pertaining models which are BERT and CLIP. Specifically, given a novel object caption, the method first masks some words (expect nouns) in the sentence and then use BERT to generate these masked words. After that, they compare the association degree between two captions (the original caption and BERT re-generated caption) and the image. Such association degree is computed by inputting the image and the caption into CLIP. Based on the association degree, the method decides whether the BERT generated caption should be used as training supervision. ", "summary_of_the_review": "The major concern is about the significance of the motivation and the applied solution is not very economic by using two large scale pre-training models to solve a small task. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MI9lcf1GXxf", "reviewer": "Reviewer_F11g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nS1: This paper pays attention to three important properties of captions -- fluency, fidelity and adequacy and proposes a formwork to generate fluent, faithful and adequate captions to describe images.\n\nS2: Extensive experimental results show that the proposed model outperforms existing models on novel object captioning.\n\nWeaknesses:\n\nW1: The proposed model combines some existing models, like BERT, CLIP, so one of my concerns is the novelty of the proposed framwork. Plus, the reward to train the model is not novel, which is used in Rotian's work [1] and distilling from BERT for text generation was proposed by Chen [2]. Also, the observation that repeating the objects that occur in an image can improve retrieval score is not new [3].\n\nW2: Neigher BERT nor CLIP is trained using the dataset used in the paper, so how to mitigate the problem of domain shift when applying BERT and CLIP? I do not think you can assume that BERT and CLIP are always correct.\n\nW3: It seems that BERT is only used during training. I am wondering whether BERT can be used in the inference stage, for example, using the captioning model to generate a caption with beam search and then employing BERT to generate more captions. In this way, we can see whether BERT can improve fluency.\n\nW4: Lack of human study.\n\n[1] Rotian Luo et al.. Discriminability objective for training descriptive captions. CVPR, 2018.\n\n[2] Y-C Chen et al.. Distilling Knowledge Learned in BERT for Text Generation. ACL, 2020.\n\n[3] Qingzhong Wang et al.. Describing Like Humans: on Diversity in Image Captioning. CVPR, 2019.", "summary_of_the_paper": "This paper proposes a framework that combines masked language models (BERT) and image-text embedding models (CLIP) for novel object captioning. The proposed model significantly improves the fluency, fidelity, and adequacy of the generated images. Extensive experiments are conducted and the comparison between the proposed model and existing models shows that the proposed model performs relatively well.", "summary_of_the_review": "This paper proposed a strong baseline of novel object captioning via combining BERT and CLIP models, however, the novelty is limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "9FNQt5uT23J", "reviewer": "Reviewer_PRhm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n1. Overall, I concur with the key insight from the paper which is that models for NOC should tackle fluency, fidelity and adequacy. It has been observed in Agrawal et al 2019 (nocaps dataset) that methods suffer from these issues. Captions which are fluent often aren't accurate, and captions which are accurate often aren't fluent.\n2. The authors take reasonable steps to fix these issues. Using a LM to fix fluency is a good idea. The authors address one of the main issues with such an approach — BERT like language models can produce fluent captions but it's hard to judge their accuracy. The authors use CLIP to score these and select the best method. Such an approach makes sense.\n3. Another novel contribution in this work is using uncaptioned images (images in the OpenImages dataset which do not have associated training captions) during training. Since these don't have training captions, they modify the reward to use CLIP as well penalize repeated words. Indeed, they show that CLIP + repeated penalty is helpful for overall performance (Table 3).\n\n### Weaknesses\n\n1. One of the biggest and most important weakness in the paper is that the paper doesn't follow the guidelines from the nocaps paper. As mentioned in the nocaps paper (section 3.3), \"Do not use additional paired image-caption data collected from humans. Improving evaluation scores by leveraging additional human-generated paired imagecaption data is antithetical to this benchmark – the only paired image-caption dataset that should be used is the COCO Captions 2017 training split. However, external text corpora, knowledge bases, and object detection datasets may be used during training or inference\". The paper heavily relies on CLIP which has been trained on alt-text data for images (which are human written) for improving the captioning performance. This makes the method proposed in the paper not comparable to existing methods in my opinion. The authors should have used the nocaps (XD) for evaluation.\n2. Related to the second point, I would have liked to see if you can replace CLIP with other approaches (ViLBERT) which also outputs a score indicating how well the caption is aligned with the image. This would form a fair comparison.\n3. Since CLIP is trained on alt-text data for images, it would be interesting to see if joint-training on Conceptual Captions (or a similar web-scale data) works better instead.\n4. CLIP is being used for checking the relevance of the caption with the image. I wonder how will other similar metrics for caption relevance (VIFIDEL) compare against scores from CLIP.", "summary_of_the_paper": "The papers presents an approach for the task of novel object captioning (NOC). NOC requires the model to describe novel objects appearing in test images that were not described in the training captions. The authors claim that existing methods are not good at the task because they don't tackle three concepts required to produce a good caption — \n\n1) Fluency: generating human-like captions, \n\n2) Fidelity: whether the caption is describing the novel object in the image or not, \n\n3) Adequacy: whether the caption covers all the salient visual concepts in the image. \n\nThe authors propose a model Visual-Linguistic Adequacy, Fidelity, and Adequacy (VLAF2) to address these issues. The model utilizes BERT to sample different variations of the captions by substituting random words from the caption. It uses CLIP to assess which of these are more relevant to the image. The model also uses CLIP score and combine it with CIDEr based reward to further train the captioning model via reinforce. \n\nThe model is evaluated on the nocaps benchmark on which it achieves improvements over existing sota methods like VINVL+VIVO. The authors also show that apart from captioning metrics, the model also does better on fluency (B@4, C), Fidelity (Precision), Adequacy(Recall).", "summary_of_the_review": "Overall, I think the paper presents an interesting insight (breaking down caption quality into fluency, adequacy and fidelity) and presents an approach which improves all the three aspects on the nocaps dataset. I raised a couple of concerns (use of CLIP which has been trained on human-generated alt-text data) which goes against the guidelines of the benchmark. It nullifies some of the claims in the paper and I'd be interested in hearing from the authors regarding that. I would have also liked to see comparison with other caption relevance metrics like VIFIDEL which would make the contributions stronger. Overall, I am borderline on this paper and will update my thoughts based on the author response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework for novel object captioning by combining BERT and CLIP.  The model improves fluency, fidelity, and adequacy of generated captions. However, as reviewers mentioned, the novelty is limited, combining large models and big data to solve a downstream task does not make useful insights at this moment.
Decision: Reject