Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors proposed a ‘Paraphrasing-to-Captioning (P2C)’ framework for Novel Object Captioning (NOC), which is an interesting task. Overall, four modules in two stages are proposed to improve the adequacy, fidelity, and fluency of NOC. Specifically, through performing paraphrasing in the first stage, the proposed model is trained to describe novel objects with linguistic fluency. In addition, self-paraphrase is proposed for boosting the associated fidelity and adequacy. Finally, experiments evaluated on the nocaps benchmark are reported. It shows improvements compared to existing methods such as VinVL+VIVO.
strengths_and_weaknesses: [Strengths]
- This paper is well-written and easy to follow. Basically, its motivation is reasonable and soundness. After reading this paper, I can basically understand why this model can achieve better performance than existing methods, i.e., the experimental comparisons basically validate the usefulness of the proposed methods. 

- The NOC task presented in this paper is interesting to me. I am glad to see the authors can utilize uncaptioned images and unpaired texts in their training.  

- Fluency, fidelity and adequacy are the three most important components in captioning tasks. The authors employ various strategies to ensure the generated captions have these properties.

[Weakness]
- Important ablation studies are missing. For example, if we hope to justify the effectiveness of paraphrasing, the results of 'Baseline+r_CIDER+r_A' should also be compared. 

- I am wondering about the role of the association model in this paper. On the one hand, I am not sure if the comparisons are fair, since the association model employed visual-linguistic pair during their training. On the other hand, employing CLIP provided reward for training captioning models may not be new, [1] and [2] presented similar ideas. 

[1] CLIPScore: A Reference-free Evaluation Metric for Image Captioning, emnlp 2021
[2] Fine-grained Image Captioning with CLIP Reward, CVPR 2022
questions: - In Table 2, the reported results of 'VinVL+VIVO' are different from the results reported in [3]. The authors may explain the possible reason behind this. 

[3] NOC-REK: Novel Object Captioning with Retrieved Vocabulary from External Knowledge, CVPR 2022
limitations: Yes, limitations have been discussed. I do not find the potential negative societal impact of this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed an interesting approach for novel object caption generation, where the main idea is to improve the captions via first leveraging a pre-trained language model for paraphrasing to generate more linguistically fluent paraphrases, and then performing self-paraphrasing based on a cross-visual association model. The models are rewarded to generate captions with low repetitions within the framework of the reinforce algorithm to optimize the overall learning process. Experiments and evaluations on nocaps, nocaps (XD), and MSCOCO datasets show the effectiveness of the proposed model. 
strengths_and_weaknesses: Strengths:

- Overall, the proposed idea appears to be solid and backed by a series of thoroughly carried out experiments. 

- Both automatic and human evaluation have been carried out as well as rigorous ablation studies were conducted on multiple different datasets. Results show the effectiveness of the proposed components over the considered baselines and SOTA models by a considerable margin. 

- The shared codebase would be beneficial for the research community.

- The paper is very well-written and organized to communicate the key ideas and contributions properly.

Weaknesses:

- The proposed P and A models are built on existing pre-trained language and cross-modal association models. 

- I would be interested to see how the proposed models perform in generating captions for out-of-domain (e. g. clinical domain) image captioning datasets (e. g. see https://www.imageclef.org/2022/medical/caption).   
questions: N/A
limitations: See weaknesses above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In image captioning, generating expressions that refer to the objects
that have never appeared in the training data is one of the important
issues.  To improve fluency as well as fidelity and adequacy in
generating captions, this paper introduces a two-stage training
process with a support of techniques related to automatic paraphrase
generation in each stage.  In the first stage of training, the
generated raw caption is randomly masked and fed into a BERT model to
predict the most probable word for it as a paraphrase, and if the
paraphrased caption has a positive effect according to a cross-modal
association model, the cross-entropy loss between the raw and
paraphrased captions is back-propagated to update the parameters.  In
the second stage of training, the sampled output is regarded as a
paraphrase of the greedily decoded caption, and their quality gap is
measured by the cross-modal association model so that the difference
is used to update the parameters in the reinforcement learning
fashion.  An empirical experiment on the "nocaps" dataset confirms
that the proposed method can generate better captions than existing
methods in terms of two automatic evaluation metrics: CIDEr and SPICE.
The ablation studies using the validation set also shows that the
components of the proposed method function as the authors intended.
strengths_and_weaknesses: I acknowledge the following two strengths of this paper.

- The proposed method can seek a wider search space for potential
captions, by utilizing pre-trained language model and sampling-based
decoding.  As a result, it can generate captions of higher fluency,
fidelity, and adequacy, compared to existing methods.

- The paper reports on ablation studies to confirm the contribution of
each of newly introduced components.  It justifies the proposed
approach with regard to the motivation, and enables readers to better
understand the proposed method.

The paper contains several flaws and has some readability issues that
hinder readers from understanding the proposed method correctly and
easily, as well as hurts the credibility of the paper.

- There are a couple of notational errors for the proposed method.
For instance, in Section 3.1, $w^c_i$ denotes the $i$-th word
(l.113).  To introduce the masked caption, the masked token is noted
as $w^m_M$ and $m$ is introduced as the mask index.  However,
$w^c_M$ should be $w^c_M$ and $m$ should be $M$.  In Eq.(1),
the cross entropy is calculated between two words (surface forms),
which is impossible and thus it should be corrected.  In l.160, the
set of inputs, $X$, and another set of outputs, $\hat{Y}$, are
defined.  However, they should be derived as
$(X,\hat Y)=(X_l,\hat Y_l)\cup(X_u,\hat Y_u)$; otherwise,
the pairs in the two component sets will be lost.

- The paper is not self-contained.  I appreciate that the authors made
an enormous effort especially for conducting various experiments.
However, as a result of claiming too many things in the main paper
without describing even minimum explanation for them, the readability
and clarity are hurt.  I recommend to prune some statements so that
the main paper is self-contained and reader-friendly.

- The descriptions in Section 4 are sometimes loose.  Line charts in
Figures 3 and 4 are inappropriate because they show the results for
nominal scales: the distance between in and near, and that for near
and out are not the same.  I ask some questions below.
questions: I have several questions regarding the experiments.

- Among two options for $P$ and three options for $A$, what are chosen?

- What do the blue and bold numbers in Table 1 stand for?

- When computing the fluency scores, the authors "remove" all the
objects and nouns from the captions.  Did they truly "remove" them?
For instance, removing "b" from "a b c d" we can no longer distinguish
the adjacency of "a c" and "c d" which looks problematic.  Removal
also leads to shorter sequences, and consequently the significant
reduction of the total number of longer n-grams will make the scores
less reliable.

- What does "-" in Table 2 indicate?  If the authors copied some
numbers from previous publications, they are not comparable with the
scores computed by the authors themselves, due to the difference of
the computational environments, toolkits, etc.
limitations: No.  The proposed method has a limitation: it assumes two large pre-trained models and cannot be applied to languages for which such prerequisites are not available.
The proposed method exploits neural generative models but their usage is limited to labeling given pairs of sentences.  Therefore, the trained models will not introduce any societal influence nor misleading conclusions.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: ## Summary
This paper performs Novel Object Captioning (NOC), which is to describe images containing novel objects that were not seen during the training phase. The data consists of both captioned and un-captioned images. For captioned images, a Captioning model $C_{\theta}$ generates a caption which is compared against the ground truth caption. For un-captioned images, $C_{\theta}$ first generates a caption, \hat{y}\_u^c, and is masked at a certain word index to produce on \hat{y}\_u^m, which is then fed into the paraphrase model, $P$, to perform masked language modeling to predict the masked word. A semantic-preserving gate, $g$ is used to validate \hat{y}\_u^m for semantic correctness. $g$ consists of an association model, $A$ which calculates the association between an image and a caption. The caption corpus might not contain words that can be used to describe novel objects, so to improve caption fidelity and adequacy, self-paraphrasing. When the association model, $A$ provides a higher score to a caption, $P2C$ would increase the probability of using that word to describe the image. To avoid repetition, the words in a sentence are paired with all other words and the similarity of a sentence is calculated. Sentences with high repetition would have a high similarity score. A repetition score $r_{rep}$ is calculated using the above similarity scores. Since $r_{rep}$ is not differentiable, a reinforce algorithm is used to reward captions that have higher linguistic fluency or cross modal association than a baseline caption obtained from greedy decoding. Their model beats the existing methods by outperforming on metrics like CIDEr, SPICEm Fluency, Fidelity and Adequacy and achieves results similar to human evaluations. Through ablation studies, the importance of each component in the model is shown.
strengths_and_weaknesses: ## Strengths
The authors have identified that the captioning corpus for captioned images does not contain important words that can be used to describe novel objects, thus a lot of the existing methods fail. They overcome this limitation by formulating the problem as a paraphrasing task. They also show effectiveness of existing language and association models as core components of their model.

## Weaknesses
One major limitation is that the paraphrasing is highly dependent on the initial caption generated by the captioning model and not much attention has been given on this step. If the initial generated caption performs poorly, the subsequent steps might not produce a high quality caption.

questions: ## Questions
1. Is the captioning model first trained on captioned images before moving on to un-captioned images?
2. Some details about which pre-trained language models were used as the paraphrase model $P$ would be appreciated.
3. How many times is masking performed on a single generated caption, $\hat{y}_u^c$? Do you mask out different indexes and provide each masked caption as a unique item to the model?
limitations: ## Limitations
The authors have mentioned that the core components of their model relies on pre-trained models for performing paraphrasing and image to text association and have not performed a joint optimization of these models.
review_title: null

Meta-review: metareview: All reviewers appreciated this paper's simple and intuitive ideas on promoting fluency, fidelity and adequacy in novel-object captioning (NOC) task via paraphrasing modules. They also appreciated the good results on multiple benchmarks and also human evaluation, plus the good writing. The authors also have very detailed useful responses in the rebuttal period. Some suggestions made to the authors to incorporate were clearer ablation tables, out-of-domain task discussion, self-contained and more clear notations and formulations.
confidence: Certain
Decision: Accept