SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zihao Yue, Anwen Hu, Liang Zhang, Qin Jin*
{yzihao, anwenhu, zhangliang00, qjin}@ruc.edu.cn
https://github.com/yuezih/SMILE
a woman holding a birthday cake with lit candles. MLE
a woman holding a cake with lit candles. SMILE (ours)
a pretty young lady that has some kind of white frosted birthday cake with lots of lit candles on top of it, surrounded by several other people looking onwardly at something in the distance.
a wire with a street light hanging from it.
a traffic light hanging from the side of a building. SMILE (ours)
a close-up image of two red stoplights hanging from an electrical cable system outside a large brick church building, during the early morning hours.
Figure 1: Descriptive captions generated by our SMILE-optimized captioning model, compared to human annotations and descriptions generated by the MLE-optimized captioning model.
ocean showing a boat sailing on the waters. MLE
a boat floating on top of a large body of water. SMILE (ours)
a lone fishing vessel that appears to be floating peacefully across the vast expanse of crystal blue water near an island in the far distance.
a man in black surfing in high and strong waves. MLE
a man riding a wave on top of a surfboard. SMILE (ours)
a young adult male leans forward as he stands atop an extremely wide red and white striped surfboard in the midst of crashing waves.
Image captioning aims to describe visual content in natural language. As 'a picture is worth a thousand words', there could be various correct descriptions for an image. However, with maximum likelihood estimation as the training objective, the captioning model is penalized whenever its prediction mismatches with the label. For instance, when the model predicts a word expressing richer semantics than the label, it will be penalized and optimized to prefer more concise expressions, referred to as conciseness optimization. In contrast, predictions that are more concise than labels lead to richness optimization. Such conflicting optimization directions could eventually result in the model generating general descriptions. In this work, we introduce Semipermeable MaxImum Likelihood Estimation (SMILE), which allows richness optimization while blocking conciseness optimization, thus encouraging the model to generate longer captions with more details. Extensive experiments on two mainstream image captioning datasets MSCOCO and Flickr30K demonstrate that SMILE significantly enhances the descriptiveness of generated captions. We further provide in-depth investigations to facilitate a better understanding of how SMILE works.
Figure 2: (a) Examples of richness optimization and conciseness optimization. When the label is "pretty" while the model predicts "woman" with less rich semantics, it leads to richness optimization; when the label is "cake" while the model predicts "white", conciseness optimization occurs. (b) Different from MLE, SMILE presents a 'semi-permeability' that accepts richness optimization while blocking conciseness optimization. Best viewed in color.
The task of generating textual descriptions for a given image, commonly referred to as image captioning [47, 54], has long been plagued by the issue of overly-generic outputs [48, 49]. That is, models tend to generate similar descriptions for distinct images using simple concepts, but lack details (as shown in Fig. 1). Such generated captions cannot meet various needs, e.g., fine-grained online image search and recommendation, automatic data annotation for training vision-language models [4, 33], and poses a long-standing challenge in the field of image captioning. Considerable efforts have been devoted to generating more descriptive image captions, including constructing paragraph caption datasets with more details for training models [19], designing additional rewards [26, 28, 29], and prompting [27, 34] vision-language pre-training (VLP) models [59, 60] to enrich captions [55]. While these efforts yield some promising improvements, they seldom try addressing this problem by revisiting the core optimization objective.
Similar to text generation tasks such as machine translation [40], image captioning models are trained to predict the next word given context (the visual content and preceding words). The training is supervised by the maximum likelihood estimation (MLE) objective, which evaluates the model's predictive distribution over the vocabulary, and penalizes the model when it fails to predict the current label. However, unlike machine translation where the target sequence is relatively certain, describing an image can be quite diverse, and limited human annotations may not cover all accurate captions. Therefore, the strict supervision with MLE is not perfectly suitable for image captioning optimization. As shown in Fig. 2, when the model predicts a word (e.g., "woman") that is more 'concise' than the ground truth label (e.g., "pretty"), MLE computes a loss due to the mismatch and encourages the model to assign a higher probability to "pretty". From the model optimization perspective, this optimizes the model to possess the more descriptive captioning ability, a process referred to as richness optimization in this work. In contrast, if the model predicts a word (e.g., "white") expressing more details than the label (e.g., "cake"), MLE also penalizes the model and optimizes it towards a more concise captioning behavior (conciseness optimization). Since ground truth captions in commonly used datasets (e.g., MSCOCO [24]) are mostly simple (around 10 words), conciseness optimization is not rare during training, especially for finetuning VLP models [23, 60, 59]. But for image captioning, descriptions that contain more details rather than just generic concepts are usually preferred so that they can be used for accurately retrieving images or conveying more information for the visually impaired, etc. Therefore, conciseness optimization should be suppressed as much as possible, and richness optimization should be encouraged to incentivize the model to generate more descriptive captions.
In this work, to achieve this goal, we propose a simple but effective training objective for descriptive image captioning, namely Semipermeable MaxImum Likelihood Estimation (SMILE). Unlike MLE, which evaluates the predictive probability distribution over the entire vocabulary, SMILE considers the probability over a vocabulary subset with limited words, i.e., only containing words in the ground truth caption. Under typical conditions (with MLE), when a 'richer' word that conveys additional details gets a high confidence score (e.g., "white" in Fig. 2, and such details generally are not
included in the ground truth caption), it challenges the label word (e.g., "cake") in the competition of probability allocation. This competition consequently leads to a decrease in the probability assigned to "cake", thereby imposing a penalty on the model and thus resulting in conciseness optimization. However, with SMILE, probability allocation is limited to a subset of words that excludes "white". Regardless of the confidence score given to "white", it does not diminish the probability assigned to the label "cake", thus avoiding a loss increase. In this way, we suppress the conciseness optimization that would have been imposed with MLE. In contrast, when the label word expresses details (e.g., "pretty" in Fig. 2) while the model prefers a more 'concise' word "woman", it is highly likely that the more 'concise' word is part of the ground truth caption and therefore, included in the subset. Eventually, even if we only focus on the subset, the word still takes away the probability from the label word "pretty", resulting in model penalization. In this way, richness optimization is maintained. Our objective allows richness optimization but blocks conciseness optimization, similar to a semipermeable membrane. We, therefore, name this objective Semipermeable MLE (SMILE).
To verify the effectiveness of our proposed SMILE, we conduct extensive experiments on the image captioning task. The results demonstrate that SMILE can effectively optimize the model to generate significantly longer and more descriptive captions. Besides, we carry out abundant experiments to analyze how SMILE works, such as ablations on subset selection and descriptiveness origin, etc. Finally, we verify the generalization ability of SMILE on the video captioning task and discuss its limitation when facing other text generation tasks.
2.1 Maximum Likelihood Estimation (MLE)
We first introduce the standard MLE for captioning optimization. As with many text generation tasks [40, 1], the model is trained to maximize the likelihood of the label when predicting the current word  $w$  in the sequence given previous words  $w_{<}$  and visual content  $v$ . The token-level loss function of MLE is defined as:
$$
\mathcal {L} _ {\mathrm {M L E}} = - \sum_ {j} ^ {| \nu |} y _ {j} \log \hat {P} ^ {\nu} (w | w _ {<  }, v; \theta), \tag {1}
$$
where the summation iterates over all words in the vocabulary  $\mathcal{V}$ ,  $y_{j}$  is the  $j$ -th element of the one-hot label vector, and  $\hat{P}^{\mathcal{V}}$  is the predictive probability distribution over  $\mathcal{V}$ .
2.2 Semipermeable MaxImum Likelihood Estimation (SMILE)
Given a target sequence  $D = [w_{1}, w_{2}, \ldots, w_{N}]$ , we form a subset  $\mathcal{V}_{D}$  that only includes the unique words occurred in  $D$ , defined as  $\mathcal{V}_{D} = \{w_{i} \mid w_{i} \in D\}$ . Then, the current word  $w$  prediction is carried out within  $\mathcal{V}_{D}$ , with the loss function of SMILE defined as:
$$
\mathcal {L} _ {\text {S M I L E}} = - \sum_ {j} ^ {| \mathcal {V} _ {D} |} y _ {j} \log \hat {P} ^ {\mathcal {V} _ {D}} (w | w _ {<  }, v; \theta). \tag {2}
$$
Here,  $\hat{P}^{\mathcal{V}_D}$  is the predictive distribution over the subset  $\mathcal{V}_D$ . Specifically, for the  $j$ -th word in  $\mathcal{V}_D$ , the probability  $\hat{p}_j$  is calculated by the relative significance of the confidence score of the word compared to all other words in  $\mathcal{V}_D$ :
$$
\hat {p} _ {j} = \operatorname {s o f t m a x} \left(\mathbf {z} _ {j}\right) = \frac {\exp \left(\mathbf {z} _ {j}\right)}{\sum_ {k \in \mathcal {V} _ {D}} \exp \left(\mathbf {z} _ {k}\right)}. \tag {3}
$$
The probability assigned to the label (the  $j^*$ -th word) determines the prediction loss. When a word  $w^{+}$  expressing additional details beyond the ground truth caption gets a high confidence score (e.g., in Fig. 2, the label is "cake",  $w^{+}$  is "white", and  $\mathcal{V}_D = \{\text{"a","pretty","woman","holding","cake"}\}$ ,  $w^{+} \notin \mathcal{V}_D$ ),  $w^{+}$  does not contribute to the denominator in Eq. (3) since it is not included in  $\mathcal{V}_D$ , and thus does not induce a decrease of the probability assigned to the label word  $\hat{p}_{j^*}$ , which would have otherwise increased the prediction loss. In other words, when the model tends to predict words
like  $w^{+}$ , these words do not participate in the probability allocation and do not take away any of the probability to be assigned to the label word, thus avoiding model penalization which leads to conciseness optimization. In contrast, when the model predicts a more 'concise' word  $w^{-}$  (e.g., the label is "pretty" and  $w^{-}$  is "woman"), such word  $w^{-}$  usually is included in the ground truth caption, i.e.,  $w^{-} \in \mathcal{V}_D$ . When allocating probability over  $\mathcal{V}_D$ ,  $w^{-}$  also participates and takes away some of the probabilities. Therefore, the model penalization is not avoided, and the induced richness optimization is maintained. Blocking conciseness optimization while allowing richness optimization is what SMILE is all about.
SMILE is for further training. Since SMILE only considers the relative probability distribution within a subset, it does not penalize the model to generate detailed words beyond the subset, thus suppressing conciseness optimization. However, it makes sense only if the predicted detail is correct. Therefore, SMILE is applied to further train a model that has already been optimized with MLE, which ensures its fundamental captioning capability. Please note that SMILE is only adopted in the training phase. During inference, the model predicts across the entire vocabulary to determine the current generation.
Initial context restriction It is well known that image captioning models are easy to suffer from exposure bias [2, 10, 36, 61]. When training in a teacher-forcing manner, the model is exposed to the ground truth context; however, during inference, each generation step is conditioned on the previous predictions. This causes a gap between training and inference. As SMILE does not require the model prediction to be consistent with the label as MLE does, the prediction is more prone to deviate from the label. Specifically, generating from the first token of the sequence, a prediction that is inconsistent with the label could lead to a completely unfamiliar context, exacerbating the exposure bias and affecting the autoregressive generation. Fortunately, we find that this issue can be effectively mitigated through some simple strategies, by ensuring that the initial context is correct, i.e., consistent with the label, which we name as initial context restriction and further discuss in Section 3.3.
Basic model and baselines Our proposed SMILE is architecture-agnostic and can be applied to any visual captioning model compatible with MLE optimization. As a representative, we validate SMILE on the base version of BLIP [23], one of the state-of-the-art vision and language models pre-trained with 129M images and paired captions. We first fine-tune BLIP on downstream datasets using MLE as the basic model, and then further optimize it with SMILE. For baselines, we compare our method with the latest descriptive image captioning solution, CapEnrich [55], and another two recent models NliCap [39] and GdisCap [49]. We also provide the performance of human-annotated ground truth captions for reference.
Dataset We evaluate our method on the two most commonly used image captioning benchmarks, MSCOCO [24] and Flickr30K [57]. MSCOCO contains about 120K images, and we adopt the commonly used Karpathy splitting [18] with 5,000 images each for the validation and test sets. Flickr30K contains about 31K images, with 1,000 images each for the test and validation sets. For both datasets, each image has five human-annotated captions.
Evaluation We evaluate the models from three aspects: descriptiveness, accuracy, and fluency. Descriptiveness refers to how detailed the caption describes the image. Following previous works [55], we evaluate descriptiveness by the performance of CLIP self-retrieval, which employs the CLIP model [33] as a retriever to recall the image with its caption from a candidates pool. This is based on the fact that captions with more details can better distinguish different images. The candidates pool is the hard retrieval pool constructed in the CapEnrich work [55], which additionally includes more similar images beyond the test set, placing a higher requirement on the descriptiveness of captions. We also report the average length of the generated captions and the lexical diversity (unique word count of all captions). Accuracy refers to the relevance of the generated caption to the image visual content, which we measure automatically by CLIPScore [13]. It calculates the semantic similarity between the image and the generated caption using a pre-trained CLIP model. For fluency, we report
Table 1: Image captioning performance of different methods on MSCOCO and Flickr30K. For human performance, we randomly select one annotation for each image for comparison with the others.
Dataset	Method	Caption Length	Lexical Diveristy	Self-Retrieval	CLIPScore	PPL
R@1	R@5
MSCOCO	NliCap	9.5	0.8	2.9	9.3	75.5	67.8
GdisCap	9.5	1.0	3.5	10.8	75.8	100.2
CapEnrich	13.3	1.5	9.4	22.6	79.2	63.1
BLIP	10.0	1.4	6.7	16.6	77.2	95.8
BLIP-LSMILE	22.3	4.5	10.0	24.5	75.0	95.6
Human	10.4	4.1	7.6	20.0	77.6	129.1
Flickr30K	CapEnrich	15.2	1.0	29.2	54.9	81.5	67.1
BLIP	11.6	0.8	25.4	46.4	78.8	65.6
BLIP-LSMILE	23.2	2.3	31.2	53.0	78.2	98.0
Human	12.3	2.0	26.2	48.3	79.8	121.2
the language modeling perplexity $^2$  (PPL) of the captions with GPT-2 [32]. However, these automatic metrics have their limitations. For example, CLIP measures the overall semantics of the caption and struggles to focus on too many details. In addition, the evaluation models also suffer from the bias of the training data and thus may have difficulty in handling longer or more complex sentences. Therefore, we also conduct human evaluations, which will be discussed later.
3.2 Main Results
Versus basic model According to Table 1, compared with the basic BLIP model, SMILE significantly increases the average length of the captions (more than doubled on both datasets). SMILE leads to more detailed descriptions as well, greatly enhancing the model's self-retrieval performance. Although SMILE causes a slight decrease in the CLIPScore (-2.2 and -0.6 on MSCOCO and Flickr respectively), we consider this to be a reasonable phenomenon, as 'talking much errs much'. Besides, maintaining a comparable level of perplexity indicates that SMILE can incentivize longer and more detailed captions without compromising fluency.
Versus baselines As shown in Table 1, BLIP significantly outperforms all baselines on lexical diversity with SMILE optimization. Compared to traditional captioning methods that tend to generate dull texts with common words overused, SMILE effectively overcomes the lexical diversity bottleneck and outperforms humans. Our approach also achieves the best performance on self-retrieval. It is worth noting that the most competitive baseline CapEnrich requires automatically constructed data in a specific format, which integrates details from multiple manual captions of each image for training, along with carefully handcrafted or learnable prompts. In contrast, our approach is refreshingly simple, has fewer restrictions, and importantly, exhibits no incompatibility with other methods. Therefore, it can serve as an effective supplement or alternative to the currently employed solutions. Since our vanilla BLIP model with SMILE optimization already surpasses other methods that may require complicated design, we see no strong need for further attempts to combine SMILE with them in this work.
Human evaluation We randomly sample 100 images from the MSCOCO test set for human evaluation. For each image, we collect 4 candidate captions, including three captions generated by CapEnrich, basic model BLIP, and SMILE-optimized BLIP, respectively, and one human-annotated caption. All candidates are randomly shuffled. For each candidate caption, 5 human annotators are asked to independently rate on a scale of 1 to 5 (a higher score indicating better quality) from three aspects, namely descriptiveness, accuracy, and fluency. The 'descriptiveness' refers to the richness of the correct detail relevant to the image. Table 2 shows the average score of each model on the three aspects. SMILE substantially improves the descriptiveness of the captions, with a rate close to 5 (excellent), outperforming all other candidates including human-written captions by a large margin. A score beyond 4 for both accuracy and fluency demonstrates the good quality of captions
$^{2}$ https://huggingface.co/docs/transformers/perplexity
Method	Semantic	Linguistic
Descrip.	Acc.	F1	Flu.
CapEnrich	3.89	4.39	4.12	4.36
BLIP	3.41	4.57	3.91	4.91
BLIP-LSMILE	4.67	4.05	4.34	4.75
Human	3.53	4.53	3.97	4.87
Table 2: Human evaluation results of the captions for the randomly sampled 100 images. (score 1 to 5: terrible, poor, fair, good, and excellent)
Figure 3: Trend of the average length and self-retrieval performance on the MSCOCO test set during training.
Table 3: Performance of different basic and further SMILE-optimized models. (F.S.: model trained From Scratch, i.e., BLIP with a randomly initialized text decoder trained on MSCOCO without pre-training;  $PT$ : pre-trained BLIP without fine-tuning;  $PT + FT$ : our default basic BLIP model with both pre-training and fine-tuning.)
Row	Basic Model	Further Training	Caption Length	Lexical Diveristy	Self-Retrieval	CLIPScore	PPL
R@1	R@5
1	F.S.	-	9.6	0.6	1.9	6.3	73.9	56.2
2	\(\mathcal{L}_{\text{SMILE}}\)	15.4	0.7	0.7	2.5	63.7	86.7
3	PT	-	5.8	1.0	2.7	8.6	74.6	573.4
4	\(\mathcal{L}_{\text{SMILE}}\)	22.5	4.9	8.5	20.1	73.5	113.7
5	PT+FT	-	10.0	1.4	6.7	16.6	77.2	95.8
6	\(\mathcal{L}_{\text{MLE}}\)	10.0	1.4	6.5	16.6	77.2	67.6
7	\(\mathcal{L}_{\text{SMILE}}\)	22.3	4.5	10.0	24.5	75.0	95.6
by our SMILE-optimized captioning model, although it faces a higher risk of 'talking much errs much'. As descriptiveness emphasizes the recall of visual content while accuracy emphasizes description precision, we additionally calculate an F1 value of these two scores as the overall semantic score. As shown in Table 2, SMILE performs best when taking into account both aspects.
Training epochs Fig. 3 presents the impact of training epochs on caption length and self-retrieval performance with SMILE optimization. It shows that SMILE takes only a few training epochs to achieve convergence and does not introduce too much extra training cost. For all of our models optimized with SMILE, we choose the checkpoints according to the best self-retrieval performance on the validation set, which always occurs within 3 epochs.
Basic model As the basic model used for SMILE optimization, BLIP is first pre-trained on a large-scale image-text dataset and then fine-tuned by MLE on downstream datasets. Both the pre-training (PT) and fine-tuning (FT) equip the model with fundamental image captioning capabilities. To investigate how such fundamental captioning ability influences SMILE's effectiveness, we compare the performance of different basic models and their further SMILE-optimized counterparts on MSCOCO. As shown in Table 3, for basic models, both pre-training and fine-tuning improve the model's performance in terms of descriptiveness and accuracy (rows 1, 3, and 5). After subsequent SMILE optimization, each basic model can generate much longer descriptions (rows 2, 4, and 7). Regarding how much SMILE improves descriptiveness, it is notable that when the fundamental captioning ability is poor, SMILE brings no improvement and can further decrease the performance (row 2 vs 1); but once the basic model is able to perform acceptable captioning, SMILE can contribute a lot (rows 4 vs 3 and 7 vs 5). This implies that SMILE requires the fundamental captioning ability of the basic model in order to further correctly generate more details. In addition, as SMILE optimization introduces extra training steps, we further train the default basic model with MLE and validate that simply adding training steps does not bring improvement (row 6 vs 5).
Table 4: Performance of SMILE-optimized models with different initial context restriction strategies.
Initial Context Restriction	Caption Length	Lexical Diveristy	Self-Retrieval	CLIPScore	PPL
R@1	R@5
-	17.0	3.0	6.8	16.2	71.9	82.0
shifting	21.3	4.0	9.6	22.4	75.0	85.2
default	22.3	4.5	10.0	24.5	75.0	95.6
Table 5: Performance comparison with different mixing ratios of MLE and SMILE on MSCOCO.
λ	Caption Length	Lexical Diveristy	Self-Retrieval	CLIPScore	PPL
R@1	R@5
1	10.0	1.4	6.7	16.6	77.2	95.8
0.5	10.8	1.4	6.7	17.9	77.0	67.0
0.1	12.6	1.9	7.6	18.2	76.1	69.3
0.05	14.7	2.3	8.7	20.9	76.4	74.1
0.01	19.8	3.6	10.9	25.1	76.2	79.4
0	22.3	4.5	10.0	24.5	75.0	95.6
Initial context restriction As aforementioned in Section 2.2, we propose initial context restriction to alleviate exposure bias for SMILE optimization. We design two implementations, First-token MLE and First-token Shifting, to achieve such restriction: