SlowGuess/NAACL_2022_OCR huggingface dataset:
Jaemin Cho<sup>1</sup> Seunghyun Yoon<sup>2</sup> Ajinkya Kale<sup>3</sup> Franck Dernoncourt<sup>2</sup>  
Trung Bui<sup>2</sup> Mohit Bansal<sup>1</sup>
<sup>1</sup>UNC Chapel Hill  <sup>2</sup>Adobe Research  <sup>3</sup>Adobe Inc.
{jmincho,mbansal}@cs.unc.edu {syoon,akale,franck.dernoncourt,bui}@adobe.com
Modern image captioning models are usually trained with text similarity objectives. However, since reference captions in public datasets often describe the most salient common objects, models trained with the text similarity objectives tend to ignore specific and detailed aspects of an image that distinguish it from others. Towards more descriptive and distinctive caption generation, we propose to use CLIP, a multimodal encoder trained on huge image-text pairs from the web, to calculate multi-modal similarity and use it as a reward function. We also propose a simple finetuning strategy of CLIP text encoder to improve grammar that does not require extra text annotation. This completely eliminates the need for reference captions during the reward computation. To comprehensively evaluate descriptive captions, we introduce FineCapEval, a new dataset for caption evaluation with fine-grained criteria: overall, background, object, relations. In our experiments on text-to-image retrieval and FineCapEval, the proposed CLIP-guided model generates more distinctive captions than the CIDEr-optimized model. We also show that our unsupervised grammar finetuning of the CLIP text encoder alleviates the degeneration problem of the naive CLIP reward. Lastly, we show human analysis where the annotators strongly prefer CLIP reward to CIDEr and MLE objectives on diverse criteria.<sup>1</sup>
Describing an image with its detailed, distinguishing aspects is crucial for many applications, such as creating text keys for image search engine and accessibility for the visual impaired. The standard deep learning approaches train an image-conditioned language model by maximizing the textual similarity between generated and reference
Figure 1: Overview of our proposed method. The left side illustrates our image captioning model training with image-text similarity reward based on CLIP (Sec. 3.1). The right side illustrates finetuning CLIP text encoder for improving grammar (Sec. 3.2).
captions (Vinyals et al., 2015; Xu et al., 2015; Rennie et al., 2017; Anderson et al., 2018). However, the reference captions of public datasets often describe only the most salient objects in images. This makes models trained to maximize textual similarity with reference captions tend to generate less distinctive captions that ignore the fine detailed aspects of an image that distinguishes it from others.
To alleviate the problem, we propose to use CLIP (Radford et al., 2021), a multi-modal encoder model trained on large image-text data (mostly English) collected from the web, by using its similarity scores (Hessel et al., 2021) as rewards (Sec. 3.1). In addition, we propose a CLIP text encoder finetuning strategy with synthetic negative caption augmentation to improve the grammar of captioning model, without any extra text annotations (Sec. 3.2). Note that our approach completely eliminates the need for reference captions during reward computation. To comprehensively evaluate descriptive captions, we also introduce FineCapEval, a new dataset that measures captioning in diverse aspects: overall, background, object, and relation between objects (Sec. 4).
In our experiments on MS COCO (Lin et al., 2014) dataset, we show that the captions from models trained with CLIP reward are more distinctive and contain more detailed information compared to
$^{1}$ Code and Data: https://github.com/j-min/CLIP-Caption-Reward
Findings of the Association for Computational Linguistics: NAACL 2022, pages 517 - 527 July 10-15, 2022 ©2022 Association for Computational Linguistics
the captions from CIDEr (Vedantam et al., 2015)-optimized models. The CLIP-guided captions even achieve the higher text-to-image retrieval performance than reference captions that are originally paired with images. We also show that our text encoder finetuning significantly improves caption grammars by removing degeneration artifacts such as word repetition. In fine-grained caption evaluation with FineCapEval and human analysis, we show our CLIP based rewards outperform text similarity objectives by a large margin on all categories.
Image Captioning Metrics. Traditionally, captions have been evaluated with n-gram or scene-graph based similarity metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), CIDEr (Vedantam et al., 2015), and SPICE (Anderson et al., 2016). However, such metrics often fail to capture paraphrased expressions due to the limited number of reference captions or scene-graphs. To tackle the problem, recent works including BERTScore (Zhang et al., 2019), ViLBERTScore (Lee et al., 2020a), UMIC (Lee et al., 2021), and CLIPScore (Hessel et al., 2021) propose to use relevance scores computed by language or multi-modal models pretrained on large data.
Objectives for Image Captioning. Standard deep learning based image captioning approaches train models with maximum likelihood estimation (MLE) objective. Ranzato et al. (2016) point that MLE suffers from exposure bias problem. To tackle exposure bias, Bengio et al. (2015) propose a curriculum learning strategy called scheduled sampling. Ranzato et al. (2016) propose to train models by directly maximizing the textual similarity between generated and reference captions with REINFORCE (Williams, 1992). Rennie et al. (2017); Luo (2020) propose self-critical sequence training (SCST) approach by normalizing rewards to stabilize its high variance.
Recent studies have observed that reference-trained captioning models often neglect important information from images (Dai et al., 2017; Wang et al., 2017). Lee et al. (2020b) use an visual question answering model's accuracy as a reward, encouraging models to generate captions that in-
clude information sufficient to answer a visual question. Dai and Lin (2017); Luo et al. (2018); Liu et al. (2018) use image-text retrieval model's self-retrieval score as a reward and combine them with n-gram based metrics, encouraging captioning models to generate captions that are distinctive to each input image.
Note that these works require careful balancing between self-retrieval and text similarity objectives for stable training. In contrast, by finetuning CLIP text encoder (Sec. 3.2), our approach removes the need of reference caption and text similarity metrics for reward computation.
3.1 CLIP-guided Image Captioning
We propose to use the relevance score between image and text calculated by CLIP (Radford et al., 2021). Following Hessel et al. (2021), we use CLIP-S as our reward:  $\mathsf{CLIP - S}(I,c) = w*\max (\frac{f^I(I)^{\intercal}f^T(c)}{|f^I(I)|\cdot|f^T(c)|},0)$  where  $I$ $c$  are image and caption,  $f^{I}$ $f^{T}$  are CLIP's image and text encoders, and  $w$  is set to 2.5. By maximizing the multimodal similarity of CLIP, which is a contrastively trained model, image captioning models are encouraged to generate captions that contain more distinctive information about the input image. Fig. 1 (a) illustrates this training strategy.
Following Rennie et al. (2017), we optimize our captioning model  $P_{\theta}(c|I)$  with REINFORCE (Williams, 1992) with self-critical baseline. We approximate the gradient of the expected reward for generated caption  $\hat{c}$ , where rewards from beam search are normalized with the baseline rewards  $b$  from the greedy decoding  $\hat{c}_{greedy}$ :  $\nabla_{\theta} \mathbb{E}_{\hat{c} \sim P_{\theta}(c|I)}[R(I,\hat{c})] \approx (R(I,\hat{c}_{beam}) - R(I,\hat{c}_{greedy}))\nabla_{\theta} \log P_{\theta}(\hat{c}_{beam}|I)$  where  $R(I,c) = \mathrm{CLIP - S}(I,c)$ .
3.2 Improving Grammar with CLIP Text Encoder Finetuning
Since CLIP is not trained with a language modeling objective, the captioning model trained with CLIP-S reward often generates grammatically incorrect (e.g., repeated words) captions (See Table 3). We inject grammatical knowledge to CLIP's text encoder with synthetic negative captions, generated by randomly repeating/removing/inserting/swapping/shuffling tokens of the reference captions. We provide the implementation details of such operations in appendix. We introduce a 2-layer per
2While language models are trained with ground-truth previous context, they generate words based on the context words previously generated by themselves during inference.
Image	Criteria	Annotations
(a)	Background	white house, truck digging soil in front of the house, trees and bushes, house surrounded by a small garden, Mini excavator, houses, white and grey building, greenery, two houses, blue and white colored machine
Object	a blue car, a blue car, black car, car, dozer, white and grey building, greenery, black car, green bushes
Relation	parked in the front yard, in front, parked in front of, Parked, car standing on the road
Overall	A blue car parked in the front yard of an off white house with a truck digging soil in front of the house.A blue car in front of a house surrounded by a small garden with trees and bushes in the background.A black car parked in front of a house with a mini excavator behind it with other houses in the background.A car and a dozer parked in front of two white and grey buildings and greenery on both sides.A black car standing on the road surrounded by green bushes on both sides and two houses and a blue and white colored machine in the background.
(b)	Background	velvet carpet stairs, light-brown colored stairs, Off white wall, Cream painted walls, cream wall with straight line light
Object	brown jumpsuit, kid, Toy, black jumpsuit, boy, brown clothes, toy, brown carpet, Little young boy, cotton carpeted stair, dark brown jumper dress, cream wall
Relation	with its head on to, touching, Hiding, Holding, boy holding and playing with the toy, putting, wearing
Overall	A child wearing a brown jumpsuit with its head on to the velvet carpet stairs.A kid is touching their head on a light brown colored stairs.A Kid wearing a black jumpsuit and holding a toy hiding below the stairs with off white wall in the background.A boy wearing brown clothes holding and playing with his toy and playing on a brown carpet on stairs with cream painted walls.Little young boy is putting his forehead on the cotton carpeted stair wearing dark brown jumper dress and background of cream wall with straight line light.
Table 1: FineCapEval examples. For each image, we aggregate the annotations for each criteria from 5 different human annotators. For 'overall' criterion, we evaluate captions with CIDEr. For the rest of criteria, we evaluate captions with word-level recall  $\mathbf{R}_{\text{word}}$ .
ceptron with sigmoid activation to CLIP text encoder's feature  $f^{T}(c)$ , which outputs a grammar score  $g(c) \in [0,1]$ , which is the probability of whether  $c$  is grammatically correct (reference) or not (negative). We train the parameters of the text encoder and grammar score predictor with CLIP's original contrastive objective while fixing the image encoder parameters. Then we train the captioning models with the reward augmented with the grammar score:  $R(I,c) = \mathrm{CLIP - S}(I,c) + \lambda g(c)$ , where  $\lambda = 2.0$ . We illustrate this finetuning strategy in Fig. 1 (b).
4 FineCapEval: Fine-grained Caption Evaluation Dataset
We introduce FineCapEval, a new dataset for caption evaluation in four different aspects. To construct FineCapEval, we collect 500 images from the MS COCO (Lin et al., 2014) test2015 split and Conceptual Caption (Sharma et al., 2018) val split, respectively. Then, for each image, we ask 5 human annotators to write phrases of 1) background, 2) objects (and their attributes; i.e., color, shape, etc.), 3) relation between objects (i.e., spatial relation), and 4) a detailed caption that includes all three aspects. See details of data collection process in appendix. In total, FineCapEval consists of 1,000 images with 5,000 annotations for each of the 4 criteria. In Table 1, we show samples of FineCapEval dataset.
We train CLIP-Res50Transformer captioning model (Shen et al., 2022) with different rewards: MLE, CIDEr, CLIP-S, CIDER+CLIP-S, CLIP
S+Grammar. Following previous works, we conduct experiment on MS COCO (Lin et al., 2014) English captioning dataset with Karpathy split (Karpathy and Fei-Fei, 2015). We evaluate the model with n-gram based metrics, embedding based metrics, text-to-image retrieval scores, and FineCapEval. We also conduct human evaluation with five criteria to understand the human preference of the generated captions in diverse aspects.
Model Architecture and Training. We use the CLIP-Res50Transformer (Shen et al., 2022) as our captioning model architecture. The model consists of CLIP-Res50 for visual feature extraction and a transformer encoder-decoder for conditional language model. We resize images in  $224 \times 224$  to extract 2048-dimensional visual features. The transformer consists of 6-layer encoder and 6-layer decoder. We train our the model with MLE objective for 15 epochs and further train with different rewards for 25 epochs (total 40 epochs), which takes within 1 day with 8 V100 GPUs. We use beam size 5 for beam search decoding. We implement a training pipeline with PyTorch (Paszke et al., 2017), PyTorch Ligthning<sup>3</sup>, and Hugging-Face Transformers (Wolf et al., 2020).
N-gram based Metrics. For N-gram based metrics, we report BLEU-4 (Papineni et al., 2002), CIDEr (Vedantam et al., 2015) METEOR (Banerjee and Lavie, 2005), and ROUGE-L (Lin, 2004).
Embedding-based Metrics. We report BERT-S (Zhang et al., 2019) and CLIP-S/RefCLIP-S (Hes
$^{3}$ https://github.com/PyTorchLightning/pytorch-lightning
Model	Reward	N-gram based	Embed based	Text-to-Image Retrieval	FineCapEval
Text based	Image-Text based	Overall	Bg.	Obj.
BLEU-4	CIDEr	METEOR	ROUGE-L	BERT-S	CLIP-S	RefCLIP-S	R@1	R@5	R@10	CIDEr	Rword	Rword
Ref. captions	29.5*	54.2*	65.0*
CLIP-Res50	MLE	32.5	110.3	27.2	55.2	0.937	0.758	1.12	21.8	45.6	58.0	13.5	11.6	13.0
CLIP-Res50	CIDEr	38.2	124.9	28.7	58.5	0.942	0.759	1.13	20.9	45.6	58.2	12.8	13.1	23.1
CLIP-Res50	CLIP-S	6.2	11.2	18.7	31.6	0.882	0.860	1.17	42.5	71.6	82.2	13.9	20.8	26.4
CLIP-Res50	CIDEr+CLIP-S	37.7	124.6	28.8	58.3	0.941	0.772	1.14	24.4	50.2	63.1	13.0	13.0	23.4
CLIP-Res50	CLIP-S+Grammar	16.9	71.0	24.9	47.3	0.924	0.793	1.15	35.8	64.0	75.8	19.3	21.8	25.5
Table 2: Performance on MS COCO Karpathy test split. \*The first caption out of 5 reference captions are used to calculate retrieval scores. R@K refers to the recall-K of the reference image.  $R_{word}$  refers to the word-level recall for background (Bg.), object (Obj.) and relation (Rel.) criteria (see Sec. 4 for details).
sel et al., 2021).<sup>4</sup> BERT-S measures textual similarity between reference captions and generated captions, CLIP-S measures the image-text similarity between input images and generated captions, and RefCLIP-S averages the textual similarity (with reference captions) and image-text similarity.
Text-to-Image Retrieval. We report the recall of the reference image using a text-to-image retrieval model, to measure the distinctiveness of the generated captions. For the retrieval model, we use pretrained CLIP ViT-B/32 (Radford et al., 2021).
FineCapEval. For background, object, and relation criteria, we measure the captioning performance with word-level recall,  $\mathbf{R}_{word} \in [0,1]$ . See details of  $\mathbf{R}_{word}$  calculation in appendix. For overall caption, we measure the performance with CIDEr.
Human Evaluation. To evaluate captions in terms of human preference, we show a pair of captions from CLIP-S+grammar reward (ours) with CIDEr reward and with MLE baseline to human annotators from Amazon Mechanical Turk<sup>5</sup>. Then we ask them to select a better caption on 5 criteria (overall, background, object, attribute, relation). For each of the 5 criteria, we ask 10 annotators with 50 pairwise selection questions. We use 50 images from FineCapEval for caption generation.
6.1 CLIP Guides Distinctive Captions
In Table 2, the models with CLIP-S and CLIP-S+Grammar rewards achieve higher image-text metrics (CLIP-S / RefCLIP-S) and text-to-image retrieval scores than baselines. Interestingly, their
retrieval scores are even higher than the retrieval score with reference captions. This shows the distinctiveness of their generated captions. For image (a) in Table 3, our model with CLIP-S+Grammar reward describes the rainy weather with 'wet', while the model with CIDEr reward does not describe it.
Our models with CLIP-S and CLIP-S+Grammar rewards score lower text similarity metrics (n-gram based metrics and BERT-S) than the model with CIDEr reward. However, the low scores on these reference-based metrics can be addressed by that the models with CLIP-S and CLIP-S+Grammar rewards often generate captions that include fine-grained information that are not even present in the reference captions. For example, for image (b) in Table 3, CLIP-S+Grammar model describes 'blue sign' of the restaurant, whereas none of the reference captions mentions them.
6.2 Finetuning CLIP Text Encoder Improves Grammar
Table 3 shows that the degeneration (e.g., repeating words) of CLIP-S reward is successfully mitigated by adding the grammar reward (CLIP-S+Grammar). Table 2 shows that adding grammar reward significantly increases all text similarity metrics (e.g., +60 for CIDEr).
6.3 Fine-grained Caption Evaluation
FineCapEval. The rightmost four columns of Table 2 show that the captions with CLIP-S and CLIP-S+Grammar significantly outperforms the captions with CIDEr on all four criteria of FineCapEval: overall, background, object, relation. The gap is smallest in object criterion, which implies MS COCO reference captions describe more object information than background or relation between objects.
Human Evaluation. Table 4 shows human evaluation results on five criteria: overall, background,
$^{4}$ Following the default settings of original papers, BERT-S and CLIP-S/RefCLIP-S are based on RoBERTa-Large (Liu et al., 2019) and CLIP ViT-B/32 (Radford et al., 2021) respectively.
<sup>5</sup>https://www.mturk.com/
Image	Reward	Captions
(a)	CIDEr	a window of an airport with planes on the runway
CLIP-S	several rows of planes parked outside a terminal window area with fog outside a terminal window motion position area motionn
CLIP-S + Grammar	a lot of airplanes parked on a wet airport terminal
Reference Captions	An airport filled with planes sitting on tarmac.
The view of runway from behind the windows of airport.
a truck driving towards some planes parked on the runway
Planes on a wet tarmac unloading at arrival gates.
Window view from the inside of airplanes, baggage carrier and tarmac.
(b)	CIDEr	a group of people riding bikes down a city street
CLIP-S	several cyclists moving and bicycles near a restaurant and a blue advertisement outside a red brick building motion stance p
CLIP-S + Grammar	a group of people riding their bikes on the busy street with a blue sign
Reference Captions	people on bicycles ride down a busy street
A group of people are riding bikes down the street in a bike lane
bike riders passing Burger King in city street
A group of bicyclists are riding in the bike lane.
Bicyclists on a city street, most not using the bike lane
Table 3: Captions generated by models with different rewards on MS COCO Karpathy test split images.
Criteria	CLIP-S + Grammar	Win	Lose	Tie
Overall	v.s. MLE	49.0	41.8	9.2
v.s. CIDEr	51.0	30.8	18.2
Background	v.s. MLE	52.8	35.0	12.2
v.s. CIDEr	53.9	25.4	20.6
Object	v.s. MLE	52.0	36.6	11.4
v.s. CIDEr	55.2	32.8	12.0
Attribute	v.s. MLE	57.2	36.8	6.0
v.s. CIDEr	55.8	37.2	7.0
Relation	v.s. MLE	44.6	44.2	11.2
v.s. CIDEr	49.2	39.6	11.2
Table 4: Human pairwise preference evaluation results.
object, attribute, relation. We sample 50 captions from model trained with CLIP-S+grammar reward (ours), CIDEr reward and MLE baseline using 50 images from Conceptual caption (Sharma et al., 2018) val split. For each of the 5 criteria, we ask 10 human annotators to select a better caption between ours and another method. On all criteria, the human annotators strongly prefer the captions with CLIP-S+Grammar rewards over CIDEr and MLE baseline.
We introduce a novel training strategy for image captioning models by maximizing multimodal similarity score of CLIP and finetuning its text encoder to improve grammar. The use of CLIP reward eliminates the need for reference captions and their bias for reward computation. We also introduce FineCapEval, a dataset for fine-grained caption evaluation. We demonstrate the effectiveness of our proposed method based on improvements in text-to-image retrieval, FineCapEval, and human evaluation on fine-grained criteria along with quali
tative examples. Future works involve finetuning CLIP reward models with desired writing styles for different applications and improving the synthetic augmentation process by using external data suitable for grammars with advanced linguistics expertise.
The CLIP models we used are trained on millions of image-text pairs collected from the web. Birhane et al. (2021) shows that such large-scale datasets often contain problematic and explicit image-text pairs. As the CLIP model card<sup>6</sup> suggests, using CLIP reward for training image captioning models is intended as a research output, and any deployed use case of the models is out of scope.
Our captioning models and CLIP models are trained on English datasets; its use should be limited to English language use cases. As our proposed method is not limited to English and easily extended to other languages, future work will explore the extensions in various languages.
We thank the reviewers for their valuable comments. This work was partially done while JC was interning at Adobe Research and later extended at UNC, where it was supported by ARO Award W911NF2110220, DARPA MCS Grant N66001-19-2-4031, and NSF-CAREER Award 1846185. The views contained in this article are those of the authors and not of the funding agency.
$^{6}$ https://github.com/openai/CLIP/blob/main/model-card.md