Paraphrasing Is All You Need for Novel Object Captioning
Authors: Cheng-Fu Yang
Yao-Hung Hubert Tsai
Wan-Cyuan Fan
Ruslan Salakhutdinov
Louis-Philippe Morency
Yu-Chiang Frank Wang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Novel object captioning (NOC) aims to describe images containing objects without observing their ground truth captions during training. Due to the absence of caption annotation, captioning models cannot be directly optimized via sequence-to-sequence training or CIDEr optimization. As a result, we present Paraphrasing-to-Captioning (P2C), a two-stage learning framework for NOC, which would heuristically optimize the output captions via paraphrasing. With P2C, the captioning model first learns paraphrasing from a language model pre-trained on text-only corpus, allowing expansion of the word bank for improving linguistic fluency. To further enforce the output caption sufficiently describing the visual content of the input image, we perform self-paraphrasing for the captioning model with fidelity and adequacy objectives introduced. Since no ground truth captions are available for novel object images during training, our P2C leverages cross-modality (image-text) association modules to ensure the above caption characteristics can be properly preserved. In the experiments, we not only show that our P2C achieves state-of-the-art performances on nocaps and COCO Caption datasets, we also verify the effectiveness and flexibility of our learning framework by replacing language and cross-modality association models for NOC. Implementation details and code are available in the supplementary materials.

Paraphrasing Is All You Need for Novel Object Captioning
Cheng-Fu Yang'
Yao-Hung Hubert Tsai?
Wan-Cyuan Fan
Ruslan Salakhutdinoy²
Louis-Philiipe Morency
Yu-Chiang Frank Wang
UCLA
CCarnegie Mellon University
3 National Taiwan University
ANVIDIA
cfyangocs ucla.
Abstract
Novel object captioning (NOC) aims to describe images containing objects with- out observing their ground truth captions during training. Due to the absence of caption annotation, captioning models cannot be directly optimized via sequence- to-sequence training or CIDEr optimization. As result, we present Paraphrasing- to- Capatiininnng (P2C), a two-stage learning framework for NOC, which would heuristically optimize the output captions via paraphrasing. With P2C, the caption- ing model first learns paraphrasing from language model pre-trained on text-only corpus, allowing expansion of the word bank for improving linguistic fluency. To further enforce the output caption sufficiently describing the visual content of the input image, perform self- paraphrasing for the captioning model with fidelity and adequacy objectives introduced. Since no ground truth captions are available for novel object images during training, P2C leverages cross-modas modsltyy (image-text) association modules to ensure the above caption characteristics can be properly preserved. In the experiments, state-of-the-ee performances on nocaps and COCO Caption datasets, we also verify the effectiveness and flexibility of our learning framework by replacing language and cross-modality association models for NOC. Implementation details and code are available in the supplementary materials.
not only show that our P2C achieves
Introduction
Novel Object Captioning (NOC) requires accurately describe images containing novel objects unseen during training captions. Despite impressive benchmark performance on COCO Captions (2] and Flickr30K
unsupervised image captioning works 9 10 cannot generalize well. This is because existing unsupervised captioning models typically assume that training image and caption data share the same visual content (i.e., objects) of interest, which might not be held in practice ( 10).
existing image captioning models 4
5
6 7
Without observing captions describing novel objects during training, number of NOC works choose to rely on object detection results for filling in the generated slotted sentences [T] 12]. Since the object words and the template sentences are generated separately, their relationships might not be well described. Therefore, Hu et al. 13 propose to relate images and the produced captions by aligning the region features of an object and its associated word embedding, aiming at improved description novel objects in scenes which are similar to the training ones. However, a word not presented in the training corpus, might not be properly presented in the predicted caption. That is, the word embedding related to novel objects such as verbs and adjectives might not be fully exploited
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

during inference, resulting in unsatisfactory linguistic fueneyy of output captions. On the other hand, despite the visual feature of a novel object is aligned with textual feature, existing methods generally are not designed to sufficiently caption images containing such objects of interest. In other words, the fidelity and adequacy of the output captions cannot be preserved.
In this paper, we uniquely approach the task of NOC by introducing and learning paraphrasing capabilities into state-of-the-art captioning models. More specifically, we advance pre-trained language models to expand the word bank of a captioning model for NOC, followed by enforcing the paraphaaaggg ability of this NOC model. The goal of the former stage is to preserve the linguistic fluency for NOC models, while that for the latter stage is deployed to exhibit improved fidelity and adequacy of the learned model. Since no ground truth captions of novel object images are available during training, we apply pross-modality association model with objectives/crities particularly designed for NOC.
is worth noting that, for the evaluation part of this work, we not only show that our method achieves state-of-the thrt--rrttt--ttttie performances on the nocaps and COCO Caption datasets, we further assess the metries reflecting the fluency, fidelity, and adequacy of output captions. In addition, via ablation studies, we further confirm the practicality and flexibility of our learning framework, which does not limit to particular language or cross-modality association modules for paraphrasing and image-text alignment.
2
Related Work
Image captioning. Recent progress of image captioning focuses on different model architectures and learning methods. Huang al. [4), Wang et al. (5), Guo et al. (6), Cornia al. 7 design different attention mechanisms for image captioning. Rennie et al. [4] Li et al. [15. Yang et al. [16] adopt reinforcement learning improve the performance. On the other hand, some researchers consider
more challenging settings, such as partially supervised [17] [8] or unpaired image captioning [0] However, these methods are restricted to the assumption that the unpaired images and captions share the same set of object class, and the number of object class is limited as well, which make them inapplicable to our task.
were only tested restrictive dataset with only eight novel object classes held out from the COCO dataset. Their extensions to large-scale image data with various novel objects are not sufficiently studied. Recent studies mainly rely on object detection results improve the performance on novel object captioning. Lu et al. [T]. Wu et al. [12] generate slotted caption templates, which are later filled in with visual concepts identified by object detectors. Yao et al. [22] exploit a copying mechanism to assemble words corresponding to object detector predictions to generate captions. Similarly, Constrained Beam Search (CBS) [19] an iethiitccttur-agnnostt exploited during inference to enforce the inclusion of novel object classes in the captions. Instead of explicitly using detection results, Hu et al. 133 and Vo et al. 23 learn the relationship between image and text by aligning object detection tags with their corresponding image region features. Recently, Wang et al. (24 indicate that desirable caption should comprise properties of fluency, fidelity, and adequacy. Nevertheless, most existing NOC approaches are not designed to handle language expression and cross-modal association with the above properties preserved.
Novel object captioning. Previously, novel object captioning approaches 19
20
21
decoding algorithm that can be
Large-scale Vision and Language Pre-training (VLP).. Recently, researchers discover that scaling up the sizes of both captioning model and training dataset would be effective to improve the perfor- mance on vision and language tasks. Dual-encoder frameworks such as CLIP [25] and ALIGN (26) scale-up contrastive pre-training (27 using 400M and 1.8B image-text pairs for cross-modal align- ment. On the other hand, Transformer-bant
models like [28, 29] not only scale up the training corpus to 5.65M image and caption pairs, but also increase the transformer layers from 12 to 24. More recently, Sim VLM [0 and LEMON [I1 further explore the huge version of Transformer with total of 32 layers, and scale up the pre-training corpus with 1.8B and 200M image-text pairs, respectively. Despite the dataset scale as an important factor in image captioning, we will demonstrate that our model still performs favorably against current large-scale methods under the same model size (i.c., Transformer base version, 12 layers) and with smaller training caption corpus.
2

---

two laptops are sittig next to each other on a noxt to
Paraphrase Model P
dift road wound Bhruugh lush grewn Hilleiie
Captianed image
Captioning Model Co
f
mosked
gotc
Is the semantio of the caption aftered?
ditt road going through lush grean hiltade
Lp
Unceptioned Image.
Figure 1: Learning caption novel objects with linguistic fluency. For caption-labeled image 21, we impose the sequence-to-sequee
objective Ls2. for training. For uncaptioned image Fa. we exploit P to paraphrase the generated caption vu. producing the refined caption S followed by semantic-preserving gate g to verify whether the paraphrased caption has altered the visual content.
3 Method
We first determine the notations and settings for the sake of completeness. Given a small set of images X1 with the corresponding captions Yi, as well as a large set of uncaptioned images X,, containing novel objects, our goal is to generate the captions Y, for X,, via a captioning model Cs (0 describes the parameters for the captioning model C). To achieve this, we propose Puraphrasing-los -oaatgg---ptiininggggg (P2C), which allows to perform paraphrasing to preserve linguistic fluency, and to eeff-parphhrase for boosting the associated fidelity and adequacy. The former is regularized by a pre-trained language model P. while an image-text cross- moddalityy model A is utilized for guiding the unsupervised learning process. We note that the contribution of our proposed P2C lies in how to leverage the paraphrasing mechanism, together with the linguistic and visual information from pre- pre-trained models P and A, for performing NOC with sufficient caption fluency, fidelity and adequacy.
3.1 Describing Novel Objects with Linguistic Fluency
By observing image-caption pairs (z), vr), the captioning model Cg in Fig. would learn the visual grounding (i.c.. localization of known objects and referring their expressions) as well the linguistics information. The training objective of such labelled data is conventional sequence-to-sequee sequeneeece loss Luss, calculated ground-truth caption yr- That is, we have L.2. CrossEntropy (ūx, (ix,)) y)).
However, using the maage-caption pairs (xz, yt) alone is not sufficient to produce fluent captions for uncaptioned images X,, We observe that some commonly-used wording of the associated novel objects, are not presented the training corpus. Therefore, we leverage a language model P, to learn its linguistic knowledge via paraphrasing. That is, for uncaptioned images X,, containing novel objects, we first generate a caption describing X,, Then, we leverage pre-trained language models the paraphrase model P. which replace the generated captions with the most probable wording of given novel object context. A semantic- semantic-preserving gate g is deployed to validate the paraphrased caption. As detailed later, this gate g is realized by cross-modal association model A, which assesses the relationship between the paraphrased caption and the input image, ensuring semantics of the caption is not modified by P.
Learning to paraphrase via language model P. We now present the detailed paraphrasing process- ing for learning Co. As illustrated in Fig. | given uncaptioned image v.. the captioning model Co generates a caption & = Co(x,), DC { wf, wy, w }. where w/f denotes the ith word, and T is the caption length. The superscript represents it is generated by our captioning model. We then obtain a masked caption eh = { [wf. wS w ..... w with m indicating the mask index, by randomly masking out the words in the caption. We note that, we do not mask nouns/objects in the above process, since they are related to objects grounded in the (novel) visual content and thus are not explicitly associated with caption fluency. As result, P takes the masked sequence diM input and predicts the masked word wP with the highest probability conditioned on the context of the entire sentence, producing the paraphrased caption f P(oM), H {wf, w. ..., wh .... w
Semantic-preserving gate g. In the above paraphrasing process, however, not every word substitution from P is guaranteed to be semantically correct. We thus require a proper image-text model (i.e., association model) A to validate the paraphrasing output. That is, if the paraphrased caption comprises more accurate and associated word that human generally uses describe the scene, then higher
3

---

Captinns XP
ICIDEr
P'A
parneloate
two taptopp pitting on tuble with white wires.
5/
0.8
Captioning Model Co
two leptops aitting on tablc with table
Association Model
0.5
Captioeed image 3/1
aigina
blue shirt with a glove and on
A
0.5
reguiariza
Trop
two glove hands and a rod scart on Dlun backkground.
V
27
0.75
regulartre
Uncaptiomed image 2i.
Figure 2: Improving caption fidelity and adequacy via self- salf-paraphr,sing, For caption-labele image Ht, we perform CIDEr optimization. The sampled caption Vă will be rewarded by A if has a higher cross-modal association than the greedy-decoded baseline Vd- The superscript d indicates the source of the image. Additionally, we regularize our model with Trep to avoid repetitive captions.
score would be obtained (than that of the original caption). Thus, we propose the objective function Cp to calculate the loss for the replaced words, gated by the comparison of the association scores of the corresponding captions. More precisely, Lp is derived as:
Cp 9 log(s), g max(0, tanh((((Eu v A( v a))
where g is the gating function preventing Cp from learning from low-quality paraphrased captions, t the one-hot representation of the paraphrased word, and s is the word distribution predicted by the captioning model at the masked timestep m, i.e., we have w arg max(s). We have A(x, y) in to calculate the association between an image x and its caption y, indicating how well the captions match the images, and serves as the margin of the association scores for distinguishing between the two captions.
3.2 Improving Caption Fidelity and Adequacy via Pelf-Paraphrsing
Recall that, discussed in Sec. I caption fidelity verifies whether the details of visual content are presented in the generated caption, while adequacy assesses whether the visual content properly described been expressed in it. While the previous paraphrasing stage increases the linguistic fluency for captions describing images containing novel objects, the novel object word itself still can be incorrectly predicted missing in the output captions. This is because that, the caption corpus Yi does not contain any novel objects, and thus the novel object words have low probability to be captioned.
To tackle the above issues, one possible solution to sala-paaranrasiing for Co with captioning evaluation metric as a critic to reward the paraphrased captions. For labeled images X,, we can exploit the CIDEr score [14, encouraging the generated caption to be consistent with that of the human annotated ones in the word level. However, while CIDEr can be easily computed for captioning labeled images X,, it cannot be explicitly calculated for captioning uncaptioned images X,, due to the absence of ground-truth captions. To address this problem, we discover that cross-modal association is an ideal mcasurement to reflect the fidelity and adequacy for output captions (see utilize association model A to compute the association
remarks in Appendix A). Specifically, score between images and the generated captions, which serves as a label-free critic in our learning framework. Then, when paraphrased caption is rewarded by a higher association score, our P2C would increase the probability of using that word the sentence to describe the image. This learning strategy allows producing captions that precisely describe the objects with plentiful visual details.
Rewarding the generated captions. As a potential challenge in NOC, we observe that captioning models would achieve improved association by simply repeating the same object that occurs in the image, which undermines the linguistic fluency of the captions. For example, the image caption "a group of cans of soda and other items on a table" can be replaced by "a pile of cans and bottles of soda on a counter with cans of cans" " with higher association score.
To tackle this problem, we choose to impose repetition penalty to avoid such trivial solutions. For image-caption pairs (r), M1), we directly calculate the CIDEr reward for the predicted caption ÿr (i.e., rCiDE = CIDEr(ji, 31)). ) To enforce the generated captions for (X,Y) Y) (X,, Y)) U (X,, Y.) with sufficient fidelity and adequacy, we exploit to compute the association reward between X and
Y (i.e., FA = A(r, .))) As for the repetition penalty to preserve linguistic fluency of the generated captions S. = [l1, w2, .... wT } for Zu. we formulate it as linear assignment problem, where every

---

word is assigned to the most similar one in the same sentence except for itself. Then, we calculate the similarity between such pairs for each sentence. Intuitively, repetitive captions would have high similarity scores (with repeating words assigned to the exact same word)s. Thus, we define the assignment á as the one that maximizes the average pairwise similarity of a sentence, i.e.,
T E
â arg max
C(wi, Wo(e))
where a(i) is the index of the word assigned to the i-th word in the caption, and C(w,, wy) is the cosine similarity between the Glo [32) word representation of two words. Since a desirable captioning model would encourage captions with low repetition (i.e., low average pairwise similarity). the reward for repetition penalty is defined as follows:
repp 1
1 T 1=1
Σ
Cww,, wa()))
(3)
Note that we do not calculate repetition penalty for the labeled data Yn since they are regularized by the aforementioned CIDEr rewards.
With the above discussions, the total reward for caption-labeled data would be r(o)) rCIDE= 31 (T (t), and the total reward for uncaptioned data would be r(0.) = ( 0u) + Trep (Bu).
Back-propagation via reinforce algorithm. Unfortunately, computation of the aforementioned rewards is non-differentiible. Thus, we adopt reinforce algorithm (33 to optimize our P2C learning. As shown in Fig. 2 for an image x we use greedy decoding to obtain the baseline result jo, and the paraphrased captions ÿ' are derived from randomly sampling from the word distribution. If the sampled captions possess higher linguistic fluency caption, they will be encouraged by positive rewards and vice versa. We follow Rennie et al. [14], Liu et al. [34 [7 and define the objective function as follows:
modal association than the baseline
V (0) = (r((3)) r( Vo log
(CIDFr(aa Ma) + TA (Taa Da) (ra(Idd Da) (Ua)
a. X1 if X,,
(4)
r r(aa)
where d indicates the source of the image, 0 being the parameters of captioning model, and po (i*) represents the predicted word logits for the generated captions. With the objective functions defined in equations (T. and the NOC model Co can be trained accordingly.
4 Experiments
Implementation details. Following Hu et al. [13. Li et al. 28 Zhang et al. 229, we consider BERT-base (35) architecture for our captioning model Cg. To demonstrate the flexibility of our learning framework, we apply two different versions of BERT, base and large, which are pre- trained on large-scale text-only corpus to model human language as our paraphrase model P. For the association model A, we have three different mross-modal association models, VIFIDEL (36). SR-PL 17 and CLIP (55] with its version being ViT/B-32. VIFIDEL associates image-caption data using word embedding of particular object labels, and SR-PL utilizes the triplet loss to learn the association of image captions, while CLIP is optimized via the contrastive pre-training. For our model reported in the following subsections, we use BERT large for P, and CLIIP for A. Due to page limits, hyperparameters and other training details can be found in Appendix B
Datasets.
The training data for the nocaps benchmark comprises the Open Images V4 [37] object detection training set (1.7M images annotated with bounding boxes for 600 object classes), plus the image-caption pairs from the COCO Captions 2017 [2 training set (0.5M image-
pairs containing 80 object classes). No additional image-caption pairs are provided for training. We evaluate our model on the validation and test set of nocaps, which comprises 4500 and 10600 images from the Open Images validation and test sets, respectively. To compare with current acale methods and investigate our model performance when scaling up the training dataset, we jointly trained our model on the Conceptual Caption dataset (38] and compare our method to other methods that access additional image-text pairs during their training on the nocaps-XD XD [ benchmark (XD
5

---

Table I: Quantitative results nocaps. Note that C and S denote CIDEr and SPICE, respectively.
We highlight the highest score in blue, while the second best scores are marked in bold.
Method
in-domain
near-domain
out-domain
overall
in-domain
near-domain
out nom-domain
overall
S
C
S
C
C
C
C
S
C
S
Validation Set
Test
Set
UpDowa
79.3
12.4
73.8
11.4
71.7
9.9
74.3
11.2
76.0
11.8
74.2
11.5
66.7
9.7
73.1
11.2
Oscarg
83,4
12.0
81.6
12.0
77.6
10.6
81.1
11.7
81.3
11.9
79.6
11.9
73.6
10.6
78.8
11.7
Oscarz
85.4
11.9
84.0
11.7
80.3
10.0
83.4
11.4
84.8
12.1
82.1
11.5
73.8
9.7
80.9
11.3
Oscare
+VIVO
92.2
12.9
87.8
12.6
87.5
11.5
88.3
12.4
89.0
12.9
87.8
12.6
80.1
11.1
86.6
12.4
VinVI.
96.8
13.5
90.7
87.4
11.6
90.9
12.8
93.8
12.7
66.1
10.9
85.5
12.5
VInVI.
VIVO
94.8
13.3
91.4
88.7
11.6
91.4
12.7
94.5
13.1
90.9
12.9
77.9
11.3
88.3
12.6
Human
84.8
14.3
85.0
14.3
95.7
14.0
87.1
14.2
8016
15.0
84.6
14.7
91.6
14.2
85.3
14.6
Ours
101.4
15.1
96.8
14.5
95.4
12.9
97.2
14.2
101.7
15.0
95.7
14.4
82.5
12.2
93.5
14.1
stands for extra data). In addition, to demonstrate that our method also enhances model performance on the general image captioning. we evaluate our P2C on the COCO Captions dataset. Due to page limits, the comparison on COCO Captions can be found in Appendix C.2
4.1 Evaluation metrics
NLP, Consensus-based Image Description Evaluation (CIDEr) calculates the similarity between the reference and generated caption by n-gram overlap in a rule-based manner. To capture human consensus in image captioning, it introduces the tf-idf weight to reduce the matching weight of the n-grams that are common in all image captions.
CIDEr. Like the evaluation metrics 33 40
SPICE. Semantic Propositional Image Caption Evaluation (SPICE) 42 matches the semantics between sentences, such objects, relations, and attributes of objects. Specifically, it converts sentences into semantic scene graphs, which allows evaluation to break grammatical constraints and focuses on propositional semantic content. It reflects the accuracy of the visual content and considers less about linguistic properties.
Fluency. To quantitatively evaluate fluency, we remove the effect of the visual information and focus on the quality of linguistic properties in the conventional caption evaluation metrics. Specifically, we we disregard the n-grams containing the particular object word of interest during the computation BLEU@4 @4 (39] and CIDEr scores. Take the sequence "a b d" for example, when 'b' is the object word, only the unigram "a, c, d" and the bigram "cd" would be taken into account, n-grams such as "ab" or "abe" would be excluded from computation. Note that the fluency experiment is conducted on subset of the nocaps validation set, which contains 1000 images whose caption annotations are available on the official website of the nocaps dataset.
Fidelity & Adequacy. Fidelity and adequacy evaluate how well the captions are associated with images. As defined in Sec. [] fidelity evaluates whether the objects described by the caption are actually presented in the images, while adequacy assesses how many objects in the images are described by the captions. These two properties are analogous the definition of precision and recall, respectively. Therefore, we extract the objects mentioned in the captions and the ground-truth objects in the images and calculate the precision (for fidelity), recall (for adequacy), and Fl (for overall association) scores. The experiment is performed on the validation set of nocaps.
Following Agrawal et al. [1, we split the dataset into three subsets for evaluation: in-domain images only contain the seen objects that have been described during training, out-of-domain images are the unseen/uncaptioned (i.e., novel) ones, while near-domain ones contain both seen and unseen objects.
4.2
Quantitative analysis
For performance comparisons. we choose UpDown (I as baselines, as well as Oscar (28 and VinVL 22 which achieves SOTA results on the benchmark of nocaps. All methods are trained via the SCST optimization [14] except for the UpDown baseline, and Constrained Beam Search (CBS) is exploited during inference. In addition, VIVO [13 is pre-training technique for captioning models, allowing them to recognize the novel objects. Since Vin did not report the numbers with CBS exploited during inference (CBS is known to improve model performance on out-of-domain data), we reproduce VI. following details stated in the original paper. For more details please refer to Appendix B For a comprehensive comparison, we conduct experiments on the validation and test set of nocaps. In addition, we compare our method with Vin and VIVO in fluency, fidelity,

---

Table 2: Quantitative results on the nocaps (XD) benchmark. Note that XD stands for extra data.
Validation set
Test set
Method
Pre-training data
CIDEr
SPICE
CIDEr
SPICE
Encoder- Decoder
43
CCI2M 43
87.4
IL.8
85.3
1L.8
Sncoddr-Deccoddr
CC3M-CCI2M
90.2
12.1
87.3
12.0
Vin
29
5.65M Combined
95.5
13.5
SimVLM V
30
1.8B
94.8
13.1
LEMON
B1
CC3M 38
91.6
13.0
LEMONbaee
CC12M
100.4
13.8
LEMON
AL1200M
106.8
14.1
Ours
COCO Caption 0.5M
97.2
14.2
93.5
14.1
Ours
CC3M
104.1
14.6
102.4
14.7
Table 3: Quantitative comparisons on caption fluency, fidelity and adequacy. Note that BLEU@4 (B@4) and CIDEr (C) are utilized for describing fluency, object precision (P) for fidelity, object recall (R) for adequacy and object Fl scores (F1) for overall cross-modal association.
in domain
near doma-
out out-nf-domain
Method
B@4
C
P
R
F1
B@4
C
P
R
F1
B@4
C
P
F1
ViaVL
21.6
74.8
59,2
40.8
48.3
19.6
73.3
22.8
32.6
26.8
17.9
59.6
48.4
25.6
33,5
VinVL+VIVO
20.6
71.6
56.0
42.2
48.1
19.8
75.4
28.5
36.3
32.0
17.4
59.6
49.0
27.3
35.1
Ours
25,4
87.0
58.2
45.6
51.3
22.1
80.1
39,9
41.0
40.4
19.7
67.7
51.3
30,5
38,3
and adequacy to demonstrate the improvement in terms of these properties. We also evaluate on the COCO Caption dataset and report in Appendix C.2
The nocaps datasets. The results nocaps are shown in Table From this table, we see that our model performed favorably against baselines and SOTAs across different metries. We see that our method substantially improves the CIDEr scores in every domain, which verifies our design to generate more fluent and natural captions. Also, is worth noting that our model largely increased the performance in SPICE score for every data domain, which verifies that method is able to generate captions with the improved image-language association.
The nocaps-XD datasets. To compare with sarge-scale pre-training methods and investigate our model performance when scaling up training dataset, we evaluate our method on the nocaps-XD) benchmark and show the results in Table 2 Note that both VLnVL, SimVLM, LEMON, and our method adopt BERT-based backbones for captioning. From the above table, we see that our method still performs favorably against SOTAs on the benchmark even if we have the fewest training caption samples (3M). Our model performance only slightly below LEMON on the CIDEr score on validation set when it uses larger training corpus (ALT200M), while surpass LEMON by significant margin when the same training corpus (CC3M) used. Thus, the effectiveness of our method when we scale up the training data can be verified. For more detailed discussion and the ablation study on this model, please refer to Appendix
Fluency, fidelity, and adequacy. As described in Sec. we design additional experiments for evaluating fluency, fidelity, and adequacy and report the results in Table 3 For fluency, we remove all the objects and nouns in the captions since they relate less to the linguistics of the captions. We then calculate the BLEU@4 (B @4) and CIDEr (C) scores for the captions after removal. For fidelity and adequacy, they indicate that captions should accurately (high precision) describe sufficient (high recall) visual details. Therefore, we report the object precision and recall in this table, and object F1 scores represent the overall association between captions and images. One can see that our method surpasses previous methods by a visible margin on all tasks except for in-domain object precision, which further verifies model improves novel object captioning fluency, fidelity, and adequacy.
4.3 Ablation studies
Following the same evaluation procedures in Sec. 4.1 we discuss the contributions of the uses of paraphrasing model P and association model A in terms of linguistic and semantic level metrics, and present their results in Table and In addition, to verify the flexibility of our proposed P2C, we replace P and A with different implementations of language models and association models. Then, we evaluate their performance on the nocaps validation set and report the results in Table 6. Detailed ablation analysis of every objective can be further found in Appendix C.I
7

---

Table 4: Analyses on paraphrasing model P. association model A, and repetition penalty for NOC using nocaps validation set. Note that mainly benefits the linguistic fluency with improved CIDEr, and reward from A is desirable for preserving visual semantics with increased SPICE.
in-domain
near-domain
out-domain
overall
Method
CIDEr
SPICE
CIDEr
SPICE
CIDEr
SPICE
CIDEr
SPICE
Ours
102.8
14.8
97.9
14.4
86.3
12.5
96.3
14.1
Ours w/o g
32.8
10.6
21.5
9.5
12.6
8.0
21.3
9.4
Ours w/o Lp
99.1
14.4
94.7
14.1
84.5
12.4
93.3
13.8
Ours w/o PA
101.1
13.8
94.1
13.4
80.5
11.9
92.3
13.1
Ours w/o Tzep
96.7
14.8
89.6
14.1
81.9
12.4
89.1
13.9
Table 5: Analyses on paraphrase model P and association model A for improving caption fluency, fidelity and adequacy. Note that P benefits fluency metrics of BLEU@+ and CIDEr, while A focusing on cross-modal association boosts metries of object precision, recall, and F1 scores.
in-domain
neur-domain
out-of-domain
Method
B@4
R
F1
B@4
P
R
B@4
C
Ours
25,4
87.0
58.2
45.6
51.3
22.1
80.1
39.9
41.0
40.4
19.7
67.7
51.3
30.5
38.3
Ours w/o Lp
21.6
77.1
58.1
40.9
48
19.8
75.5
39.0
40.0
19.1
66.1
51.6
19.1
66.1
Ours w/o PA
23.0
79.3
5888
42.1
49.1
21.2
78.6
35.8
37.7
36.8
18.8
65.7
51.6
27.4
35.8
Paraphrase model P. As shown in Table 4 the captioning model without P would observe performance drop in CIDEr for linguistic fluency, but such drops for the metric of SPICE (related to visual content) would be less significant. Similarly, in the lowest BLEU and CIDEr scores. These results confirm our motivation and model design, since P is utilized to improve caption quality at the linguistics level.
observed in Table removing P would result
To further verify the use and flexibility of paraphrase model P, we replace our paraphrase model P (the large version of BERT) with the base version of BERT. We found out that only slight performance drop is produced. We conjecture that this is because the two versions of BERT are trained on the same text corpus, which means they share the same word bank. As result, model would distill similar linguistic knowledge when using either model to guide the training of our P2C.
Semantic-prestn preserving gate g. From Table 4 we see that captioning model would be misled by the wrong guidance produced by P. if no validation from g to ensure that the semantics of the original captions is not modified by P. This results in significant performance drop on nocaps.
Association model A. As shown in Table the model without the reward calculated by A showed significant drops in captioning metrics of CIDEr and SPICE. However, the performance decrease in SPICE is expected, since A is particularly deployed for preserving visual content in captions. As for CIDEr, its decrease is mainly due to the deterioration of missing visual content in captions. This also confirmed by Table 5 in which mainly the metries reflecting fidelity and adequacy would observe significant drops for model trained without the association rewards.
To further verify the use and flexibility of the association model A, we replace it with different models that also produce association scores between images and captions. Specifically, we consider VIFIDEL [6). SR-PI. [17]. and CLIP 225) As the results shown in Table 6 one can see that the of CLIP as A would achieve the best performance. This is because that, VIFIDEL only associates image-caption data using word embedding of particular object labels, while CLIP assesses such cross-modal data in the instance level, i.e., taking the complete caption of an image into consideration. As for SR-PL, it utilizes the triplet loss learn the association of images and captions, while CLIP is optimized via the contrastive pre-training which fully exploits the negative samples in a mini-batch learn more compact representation space, allowing it to estimate the association more accurately.
Repetition penalty. Recall that, in Sec. Sec.3.2 this penalty is to alleviate the association between images and captions with redundant visual information. As seen in Table the model without this penalty observed a significant performance drop in CIDEr scores. We did not see such trends for SPICE. This is because that, repetitive words in captions mainly violate linguistic structures rather then semantic accuracy, and thus the performance related to linguistic fluency would be more sensitive to the deployment of this penalty.
8

---

Table 6: Uses of different paraphrase models P and association models A for NOC. Results on the nocaps validation set are presented. Note that Stages and are the paraphrasing schemes presented in Sec. 3.1 and Sec. 3.2 respectively.
Overall
Method
Paraphrase Model P
Association Model A
Lp
Twp
CIDEr
SPICE
Baseline
N/A
N/A
K
X
X
81.2
12.3
Ours w/ Stage
BERT base
CLIP 25
X
X
84.1
12.7
Stage
BERT large
CLIP
X
X
84.2
12.7
Ours w/ Stage
BERT Karge
VIFIDEL 36
55.3
10.7
Ours w/ Stage
BERT BERTiarge
SR-PI. 17
84.6
13.2
Ours w/ Stage
BERT Jarge
CI.IP
96.3
14.1
Fluency
Fidelity
Adequacy
group of women standing
blue blanket and glove on blue shirt.
and woman playing accordion.
VinLL
field with football.
group of girls football uniforms
two glove hands and a red scarf on blue background.
couple of people sitting in church playing an accordion.
posing for picture.
Figure 3: Example results and comparisons for image captions produced by Vin and ours in terms of fluency, fidelity and adequacy. Note that both utilize VIVO for novel object detection.
4.4 Qualitative analysis
For qualitative comparisons, we first conduct human study and ask individuals to evaluate a caption from three perspectives: fluency, fidelity, and adequacy. Duc to page limit, the experiment details and the corresponding results are presented in Appendix C.4 For qualitative analysis, we empirically show captions in Fig. which are generated by model and VIn with both pre-trained from VIVO for novel object detection. In this figure, wordings that are less accurate or incorrectly describe the associated visual content marked in bold. And, wording improvements are highlighted in red. From this figure, one can see that for fluency, our model generated vivid captions with more proper wordings. Take the image the left for example, our model particularly described "posing for a picture" instead of "standing on a field". As for fidelity, our model is designed to accurately capture the visual content in an image. Specifically, take the second image for example, we correctly described the number of gloves and the novel object red searf. while VL failed to do so. As for adequacy, take the bottom-right image for example, our model was able recover visual details in the image (i.e.. "people playing the accordion" and "sitting in church"). For more qualitative examples, please refer to Appendix C.5
Conclusion
cattrrrsiinn-t-Capttiiiiii (P2C) for novel object captioning, with particular goals of improving caption fluency, fidelity, and adequacy. In P2C, we advocate the learning of two paraphrasing capabilities for captioning models. The first is language-level paraphrasing, which expands the word bank of a captioning model under the guidance of pre-trained language model, and thus preserves the linguistic fluency during NOC. Secondly, we introduce the paappparaprrsinggg ability for the captioning model to sufficiently describe visual content of the input image, so that both caption fidelity and adequacy can be achieved. Due to the lack of ground truth captions, image-texx -ext association is uniquely exploited for guiding the training process. Empirically, we not only showed that our model achieved SOTA results on benchmark datasets, we also assessed the metrics associated with fluency, fidelity, and adequacy, confirming the effectiveness of our model. Via our ablation studies, we further verified the flexibility of our learning framework by replacing language and cross-modality association modules for paraphrasing and image-text alignment. Finally, since we
In this paper, proposed

---

apply pre- vre-trained language and association models in P2C, their joint optimization would be among our future research directions.
Acknowledgement This work is supported in part by the Ministry of Science and Technology of Taiwan under grant NSTC 11 10026334-F-002-0552. Y.HHH..T., R.S., and I..-P.M. supported in part by the NSF IIS 1635622, NSF Awards # #1750439 #1722822, National Institutes of Health, IARPA D17PC00340, ONR Grant N0001418 12861, and Facebook PhD Fellowship. We also thank to National Center for High-performance Computing (NCHC) for providing computational and storage resources.
References
[1] Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8948-898 8957, 2019.
[2] Xinlei Chen, Hao Fang. Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: /504 00325, 2015.
[3] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[4] Lun Huang, Wenmin Wang, Jie Chen, and Xiao- Yong Wei. Attention on attention for image captioning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4634-4643 2019.
[5] Weixuan Wang, Zhihong Chen, and Haifeng Hu. Hierarchical attention network for image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8957-8964, 2019.
[6] Longteng Guo, Jing Liu, Xinxin Zhu, Peng Yao, Shichen Lu, and Hanqing Lu. Normalized and geometry-aware self-atenntion network for image captioning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10327-10336, 2020.
[7] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memore transformer for image captioning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10578-10587 2020.
[8] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13041-13049, 2020.
[9] Jiuxiang Gu. Shafiq Joty, Jianfei Cai, Handong Zhao, Xu Yang, and Gang Wang. Unpaired image captioning via scene graph alignments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10323-10332, 2019.
[10] Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. Unsupervised image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4125-4134, 2019.
[11] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7219-7228, 2018.
[12] Yu Wu, Linchao Zhu, Lu Jiang, and Yi Yang. Decoupled novel object captioner. In Proceedings of the 26th ACM international conference on Multimedia, pages 1029-1037, 2018.
[13] Xiaowei Hu, Xi Yin, Kevin Lin, Lijuan Wang, Lei Zhang, Jianfeng Gao, and Zicheng Liu. Vivo: Surpassing human performance in novel object captioning with visual vocabulary pre-training. arXiv e-prints, pages arXiv-2009, 2020.
10

---

[14] Steven J Rennie. Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7008-7024, 2017.
[15] Nannan Li, Zhenzhong Chen, and Shan Liu. Meta learning for image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8626-8633, 2019.
[16] Xuewen Yang, Heming Zhang, Di Jin, Yingru Liu, Chi-Hao Wu, Jianchao Tan, Dongliang Xie, Jue Wang, and Xin Wang. Fashion captioning: Towards generating accurate descriptions with semantic rewards. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pages 1-17. Springer, 2020.
[17] Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, and Xiaogang Wang. Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data. In Proceedings of the European Conference on Computer Vision (ECCV), pages 338-354, 2018.
[18] Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, and In So Kweon. Image captioning with very scarce supervised data: Adversarial semi-supervised learning approach. arXiv preprint MXXiv:990.0020000 2019.
[19] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Guided open vocabulary image captioning with constrained beam search. arXiv preprint arXiv:1612.0033 2016.
[20] Peter Anderson, Stephen Gould, and Mark Johnson. Partially-supervieed image captioning. arXiv preprint arXiv:1806.06004, 2018
[21] Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Darrell. Deep compositional captioning: Describing novel object categories without paired training data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-10, 2016.
[22] Ting Yao, Yingwei Pan, Ychao Li, and Tao Mei. Incorporating copying mechanism in image captioning for learning novel objects. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6580-6588, 2017.
[23] Duc Minh Vo, Hong Chen, Akihiro Sugimoto, and Hideki Nakayama. object captioning with retrieved vocabulary from external knowledge. arXiv preprint arXiv:2203. /4499, 2022.
Noc-rek: Novel
[24] Sijin Wang. Ziwei Yao, Ruiping Wang, Zhongqin Wu, and Xilin Chen. Faier: Fidelity and adequacy ensured image caption evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14050-14059, 2021.
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00 000200 2021.
[26] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoe V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.055:
2021.
[27] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 0037403748 2018
[28] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for languael tasks. In European Conference on Computer Vision, pages 121-137. Springer, 2020.
[29] Pengehuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588, 2021.
11

---

[30] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai. Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.1099 2021.
[31] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Li- juan Wang, Scaling up vision-language pre-training for image captioning. arXiv preprint arXiv:2111.12333, 2021.
[32] Jeffrey Pennington. Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[33] Ronald J Williams. Simple statistical ment learning. Machine learning, 8(3):229-255 226,, 1992.
algorithms for connectionist reinforce-
[34] Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image captioning via policy gradient optimization spider. Proceedings of the IEEE international conference computer vixion, pages 873-881, 2017.
[35] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rX0v:1810.04805. 2018.
[36] Pranava Madhyastha, Josiah Wang, and Lucia Specia. Vifidel: Evaluating the visual fidelity of image descriptions. arXiv preprint
999900079440, 2019.
[37] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont- Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 128(7):1956- 1981, 2020.
[38] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018
[39] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 31 311-318, 2002.
[40] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[41] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72, 2005.
[42] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. European conference on computer vision, pages 382-398. Springer, 2016.
[43] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Sorieut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3568, 2021.
[44] Guanglu Song, Yu Liu, and Xiaogang Wang. Revisiting the sibling head in object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11563-11572 2020.
[45] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer. 2014.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In our supplementary materials,
we demonstrate some failure cases of our model. We empirically observe that the
eπrоr mainly comes from the wrong detection tags predicted by the pre-trained object
detectors.
(C) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...fnal
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Please see the supplementary materials.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [Yes] See Section 4 All datasets we used are public.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [Yes] Please see our supplementary materials.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [No]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [No]
13


Reviews:
[{"review_id": "vk0W8G_NZV6", "reviewer": "Reviewer_YAnp", "summary": "This paper proposed an interesting approach for novel object caption generation, where the main idea is to improve the captions via first leveraging a pre-trained language model for paraphrasing to generate more linguistically fluent paraphrases, and then performing self-paraphrasing based on a cross-visual association model. The models are rewarded to generate captions with low repetitions within the framework of the reinforce algorithm to optimize the overall learning process. Experiments and evaluations on nocaps, nocaps (XD), and MSCOCO datasets show the effectiveness of the proposed model. ", "questions": "N/A", "limitations": "See weaknesses above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Overall, the proposed idea appears to be solid and backed by a series of thoroughly carried out experiments. \n\n- Both automatic and human evaluation have been carried out as well as rigorous ablation studies were conducted on multiple different datasets. Results show the effectiveness of the proposed components over the considered baselines and SOTA models by a considerable margin. \n\n- The shared codebase would be beneficial for the research community.\n\n- The paper is very well-written and organized to communicate the key ideas and contributions properly.\n\nWeaknesses:\n\n- The proposed P and A models are built on existing pre-trained language and cross-modal association models. \n\n- I would be interested to see how the proposed models perform in generating captions for out-of-domain (e. g. clinical domain) image captioning datasets (e. g. see https://www.imageclef.org/2022/medical/caption).   ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dx3NEaKIjl", "reviewer": "Reviewer_B4GQ", "summary": "## Summary\nThis paper performs Novel Object Captioning (NOC), which is to describe images containing novel objects that were not seen during the training phase. The data consists of both captioned and un-captioned images. For captioned images, a Captioning model $C_{\\theta}$ generates a caption which is compared against the ground truth caption. For un-captioned images, $C_{\\theta}$ first generates a caption, \\hat{y}\\_u^c, and is masked at a certain word index to produce on \\hat{y}\\_u^m, which is then fed into the paraphrase model, $P$, to perform masked language modeling to predict the masked word. A semantic-preserving gate, $g$ is used to validate \\hat{y}\\_u^m for semantic correctness. $g$ consists of an association model, $A$ which calculates the association between an image and a caption. The caption corpus might not contain words that can be used to describe novel objects, so to improve caption fidelity and adequacy, self-paraphrasing. When the association model, $A$ provides a higher score to a caption, $P2C$ would increase the probability of using that word to describe the image. To avoid repetition, the words in a sentence are paired with all other words and the similarity of a sentence is calculated. Sentences with high repetition would have a high similarity score. A repetition score $r_{rep}$ is calculated using the above similarity scores. Since $r_{rep}$ is not differentiable, a reinforce algorithm is used to reward captions that have higher linguistic fluency or cross modal association than a baseline caption obtained from greedy decoding. Their model beats the existing methods by outperforming on metrics like CIDEr, SPICEm Fluency, Fidelity and Adequacy and achieves results similar to human evaluations. Through ablation studies, the importance of each component in the model is shown.", "questions": "## Questions\n1. Is the captioning model first trained on captioned images before moving on to un-captioned images?\n2. Some details about which pre-trained language models were used as the paraphrase model $P$ would be appreciated.\n3. How many times is masking performed on a single generated caption, $\\hat{y}_u^c$? Do you mask out different indexes and provide each masked caption as a unique item to the model?", "limitations": "## Limitations\nThe authors have mentioned that the core components of their model relies on pre-trained models for performing paraphrasing and image to text association and have not performed a joint optimization of these models.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe authors have identified that the captioning corpus for captioned images does not contain important words that can be used to describe novel objects, thus a lot of the existing methods fail. They overcome this limitation by formulating the problem as a paraphrasing task. They also show effectiveness of existing language and association models as core components of their model.\n\n## Weaknesses\nOne major limitation is that the paraphrasing is highly dependent on the initial caption generated by the captioning model and not much attention has been given on this step. If the initial generated caption performs poorly, the subsequent steps might not produce a high quality caption.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PSSoJX3Rw22", "reviewer": "Reviewer_azLR", "summary": "In this paper, the authors proposed a ‘Paraphrasing-to-Captioning (P2C)’ framework for Novel Object Captioning (NOC), which is an interesting task. Overall, four modules in two stages are proposed to improve the adequacy, fidelity, and fluency of NOC. Specifically, through performing paraphrasing in the first stage, the proposed model is trained to describe novel objects with linguistic fluency. In addition, self-paraphrase is proposed for boosting the associated fidelity and adequacy. Finally, experiments evaluated on the nocaps benchmark are reported. It shows improvements compared to existing methods such as VinVL+VIVO.", "questions": "- In Table 2, the reported results of 'VinVL+VIVO' are different from the results reported in [3]. The authors may explain the possible reason behind this. \n\n[3] NOC-REK: Novel Object Captioning with Retrieved Vocabulary from External Knowledge, CVPR 2022", "limitations": "Yes, limitations have been discussed. I do not find the potential negative societal impact of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- This paper is well-written and easy to follow. Basically, its motivation is reasonable and soundness. After reading this paper, I can basically understand why this model can achieve better performance than existing methods, i.e., the experimental comparisons basically validate the usefulness of the proposed methods. \n\n- The NOC task presented in this paper is interesting to me. I am glad to see the authors can utilize uncaptioned images and unpaired texts in their training.  \n\n- Fluency, fidelity and adequacy are the three most important components in captioning tasks. The authors employ various strategies to ensure the generated captions have these properties.\n\n[Weakness]\n- Important ablation studies are missing. For example, if we hope to justify the effectiveness of paraphrasing, the results of 'Baseline+r_CIDER+r_A' should also be compared. \n\n- I am wondering about the role of the association model in this paper. On the one hand, I am not sure if the comparisons are fair, since the association model employed visual-linguistic pair during their training. On the other hand, employing CLIP provided reward for training captioning models may not be new, [1] and [2] presented similar ideas. \n\n[1] CLIPScore: A Reference-free Evaluation Metric for Image Captioning, emnlp 2021\n[2] Fine-grained Image Captioning with CLIP Reward, CVPR 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KO2QZ392Igf", "reviewer": "Reviewer_HSxb", "summary": "In image captioning, generating expressions that refer to the objects\nthat have never appeared in the training data is one of the important\nissues.  To improve fluency as well as fidelity and adequacy in\ngenerating captions, this paper introduces a two-stage training\nprocess with a support of techniques related to automatic paraphrase\ngeneration in each stage.  In the first stage of training, the\ngenerated raw caption is randomly masked and fed into a BERT model to\npredict the most probable word for it as a paraphrase, and if the\nparaphrased caption has a positive effect according to a cross-modal\nassociation model, the cross-entropy loss between the raw and\nparaphrased captions is back-propagated to update the parameters.  In\nthe second stage of training, the sampled output is regarded as a\nparaphrase of the greedily decoded caption, and their quality gap is\nmeasured by the cross-modal association model so that the difference\nis used to update the parameters in the reinforcement learning\nfashion.  An empirical experiment on the \"nocaps\" dataset confirms\nthat the proposed method can generate better captions than existing\nmethods in terms of two automatic evaluation metrics: CIDEr and SPICE.\nThe ablation studies using the validation set also shows that the\ncomponents of the proposed method function as the authors intended.", "questions": "I have several questions regarding the experiments.\n\n- Among two options for $P$ and three options for $A$, what are chosen?\n\n- What do the blue and bold numbers in Table 1 stand for?\n\n- When computing the fluency scores, the authors \"remove\" all the\nobjects and nouns from the captions.  Did they truly \"remove\" them?\nFor instance, removing \"b\" from \"a b c d\" we can no longer distinguish\nthe adjacency of \"a c\" and \"c d\" which looks problematic.  Removal\nalso leads to shorter sequences, and consequently the significant\nreduction of the total number of longer n-grams will make the scores\nless reliable.\n\n- What does \"-\" in Table 2 indicate?  If the authors copied some\nnumbers from previous publications, they are not comparable with the\nscores computed by the authors themselves, due to the difference of\nthe computational environments, toolkits, etc.", "limitations": "No.  The proposed method has a limitation: it assumes two large pre-trained models and cannot be applied to languages for which such prerequisites are not available.\nThe proposed method exploits neural generative models but their usage is limited to labeling given pairs of sentences.  Therefore, the trained models will not introduce any societal influence nor misleading conclusions.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I acknowledge the following two strengths of this paper.\n\n- The proposed method can seek a wider search space for potential\ncaptions, by utilizing pre-trained language model and sampling-based\ndecoding.  As a result, it can generate captions of higher fluency,\nfidelity, and adequacy, compared to existing methods.\n\n- The paper reports on ablation studies to confirm the contribution of\neach of newly introduced components.  It justifies the proposed\napproach with regard to the motivation, and enables readers to better\nunderstand the proposed method.\n\nThe paper contains several flaws and has some readability issues that\nhinder readers from understanding the proposed method correctly and\neasily, as well as hurts the credibility of the paper.\n\n- There are a couple of notational errors for the proposed method.\nFor instance, in Section 3.1, $w^c_i$ denotes the $i$-th word\n(l.113).  To introduce the masked caption, the masked token is noted\nas $w^m_M$ and $m$ is introduced as the mask index.  However,\n$w^c_M$ should be $w^c_M$ and $m$ should be $M$.  In Eq.(1),\nthe cross entropy is calculated between two words (surface forms),\nwhich is impossible and thus it should be corrected.  In l.160, the\nset of inputs, $X$, and another set of outputs, $\\hat{Y}$, are\ndefined.  However, they should be derived as\n$(X,\\hat Y)=(X_l,\\hat Y_l)\\cup(X_u,\\hat Y_u)$; otherwise,\nthe pairs in the two component sets will be lost.\n\n- The paper is not self-contained.  I appreciate that the authors made\nan enormous effort especially for conducting various experiments.\nHowever, as a result of claiming too many things in the main paper\nwithout describing even minimum explanation for them, the readability\nand clarity are hurt.  I recommend to prune some statements so that\nthe main paper is self-contained and reader-friendly.\n\n- The descriptions in Section 4 are sometimes loose.  Line charts in\nFigures 3 and 4 are inappropriate because they show the results for\nnominal scales: the distance between in and near, and that for near\nand out are not the same.  I ask some questions below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept