SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. First-token MLE adopts MLE loss for the first token of the target sequence, and SMILE for the remaining tokens.
2. First-token Shifting shifts the label of the first token of the target sequence to its alternative. For example, in the vocabulary of the BERT model [7], the word "a" can be replaced with its subword form "#a", which rarely appears to be the first token in the training corpus. By doing so, we replace the familiar word with an unfamiliar one, making it difficult for the model to predict correctly (even within a subset). This often results in model penalization, making the prediction of the first token more likely to be consistent with the ground truth.
Since the shifting approach is designed with certain requirements on the model vocabulary, making it potentially difficult to be applied to all models, we use the First-token MLE by default to address the exposure bias issue. To demonstrate the effectiveness of this strategy, in Table 4, we show that without initial context restriction, SMILE optimization can not achieve promising results. Either the default First-token MLE implementation or the First-token Shifting implementation helps alleviate exposure bias and the former is better.
Mixing  $\mathcal{L}_{\mathrm{MLE}}$  with  $\mathcal{L}_{\mathrm{SMILE}}$  Since SMILE improves descriptiveness at the expense of some accuracy, it could pose a risk in scenarios that demand higher accuracy. To strike a balance, we carry out experiments where the overall learning objective combines both  $\mathcal{L}_{\mathrm{MLE}}$  and  $\mathcal{L}_{\mathrm{SMILE}}$ , defined as:
$$
\mathcal {L} _ {\text {o v e r a l l}} = \lambda \cdot \mathcal {L} _ {\mathrm {M L E}} + (1 - \lambda) \cdot \mathcal {L} _ {\mathrm {S M I L E}}, \lambda \in [ 0, 1 ]. \tag {4}
$$
Table 5 illustrates how the balance between MLE and SMILE impacts the performance in terms of descriptiveness and accuracy. As  $\mathcal{L}_{\mathrm{SMILE}}$  becomes more dominant, the model tends to generate longer outputs with higher lexical diversity and descriptiveness; however, accuracy correspondingly decreases. This demonstrates that a compromise between descriptiveness and accuracy can be achieved by simply combining  $\mathcal{L}_{\mathrm{SMILE}}$  with  $\mathcal{L}_{\mathrm{MLE}}$ . Notably, we find that when  $\mathcal{L}_{\mathrm{MLE}}$  is incorporated at a very low ratio (0.01), the generated captions achieve the best performance in self-retrieval. This suggests that while SMILE facilitates more details, ensuring accuracy is also important for captions to better describe and distinguish images.
Figure 4: Average caption length during training with (a) differentsubseting strategies and (b) on different training corpus.
In this section, we conduct experiments to further analyze how SMILE works. As we hypothesized, SMILE implements a 'semi-permeability' to retain only richness optimization, leading the model to generate more details. Therefore, we first investigate whether such an effect is attributed to our subset selection. Then, we affirm the 'semi-permeability' by flipping it to observe an opposite property of SMILE. Besides, since BLIP-  $\mathcal{L}_{\mathrm{SMILE}}$  surpasses human annotation in descriptiveness as shown in Table 1 and Table 2, we analyze where it gains the capability to generate details. Furthermore, we provide a visualization to illustrate how SMILE exerts its effect from the token-level loss. Finally, we test the generalization ability of SMILE on video captioning.
Subset selection is key in SMILE. We argue that the effectiveness of SMILE is attributed to our subset selection strategy that selects the target sequence words. To verify this, we ablate the SMILEsubseting strategy by replacing it with Randomsubseting, that is, for a given target sequence, we randomly choose 10 words (the average caption length of MSCOCO) from the entire vocabulary to form the subset, ensuring the subset size is relatively consistent with that of SMILE. Fig. 4 (a) illustrates the impact of SMILE and Randomsubseting on the generation length during training. Randomsubseting maintains the describing habits of the basic model, neither lengthening nor shortening the generation length. This is due to that the model's preferred predictions among the entire vocabulary, whether more or less 'rich' than the label, are less likely to be included in the randomly-chosen subset. This leads to a mechanism akin to 'bidirectional impermeability', where both conciseness and richness optimization are weakened. This also demonstrates that it is the subset selection of SMILE that contributes to the increasing length of model generation.
Reverse subsetting flips the 'semi-permeability'. Since SMILE achieves a 'semi-permeability' which allows richness optimization while blocks conciseness optimization, it is natural to assume that such 'semi-permeability' could be flipped with a reversed subset selection. Thus, we conduct experiments by proposing Reverse subsetting, that is, given a target sequence  $D = [w_{1}, w_{2}, \ldots, w_{N}]$ , when predicting each word, we select the complement of the target sequence word set in the entire vocabulary, plus the current label, as the subset. Hence, for the prediction of the  $i$ -th word, the corresponding subset is  $\mathcal{V}_{S_i} = \{w | w \notin \mathcal{V}_D\} \cup \{w_i\}$ . As shown in Fig. 4 (a), with a subset complementary to that of SMILE, Reverse subsetting exhibits the opposite property of SMILE, leading the model to generate increasingly shorter captions. With these observations, we confirm that there is a 'semi-permeability', which is achieved by our subsetting strategy and can be inverted.
Models 'absorb' descriptiveness from the corpus. Although SMILE achieves unidirectional richness optimization, one may still wonder where models with SMILE optimization learn the descriptive captioning ability to surpass human annotators. We suspect that models could learn the usage of details from the whole corpus. To verify this hypothesis, we construct two datasets based on MSCOCO, namely Simplest-COCO, which contains no details, and Simpler-COCO, with only a few details. Concretely, for Simplest-COCO, we extract the subject from the caption and prefix it with a definite article based on grammatical rules, such as "a girl". For Simpler-COCO, we extract the subject-verb-object constituent of the caption as a new caption with few details, such as "a man working". All the mentioned natural language processing (e.g., parsing and part-of-speech tagging)
Figure 5: Token-level model predictive distribution and penalization of (a) MLE and (b) SMILE. The model is less penalized if the label word is assigned a higher probability.
Table 6: Video captioning performance on MSR-VTT.
Method	Caption Length	Lexical Diveristy	EMScore	Self-Retrieval	PPL
P	R	F	R@1	R@5
BLIP4video	8.1	1.1	26.4	28.1	27.2	19.0	38.7	150.1
BLIP4video-LSMILE	20.6	4.5	26.6	28.6	27.5	27.3	50.0	163.0
Human	9.2	3.8	26.6	28.2	27.4	23.0	42.8	561.8
is performed with SpaCy [16]. The average caption length for Simplest-COCO and Simpler-COCO is 2.0 and 2.5, respectively. On these two datasets, we first train the default basic model with MLE until convergence and further optimize it with SMILE. As shown in Fig. 4 (b), only MLE optimization does not achieve longer descriptions than the training corpus on both datasets. On the other hand, with SMILE optimization, the description length does not increase when training on Simplest-COCO; however, on Simpler-COCO, the model generates captions about twice as long as the ground truths. This suggests that richness optimization relies on the details in the whole training corpus, and SMILE helps models 'absorb' the capability to express more details.
Visualization of token-level penalization, MLE versus SMILE Taking a sample from the MSCOCO training set as an example, Fig. 5 shows the predictive distribution and loss value for each word during MLE and SMILE optimization, respectively. Note that the predictions in Fig. 5 (a) and (b) are given by an identical model but the former distribution is calculated over the whole vocabulary, while the latter over the SMILE subset. When predicting the second word ("white" in the target sequence), the model prefers a more 'concise' word "bus" than the label "white". For MLE, "bus" takes away much of the probability, resulting in a significant loss that causes richness optimization. Similar to MLE, SMILE also penalizes such prediction since both "white" and "bus" are included in the subset. When predicting the sixth word (the label word is "and"), the model prefers a more 'detailed' word "parked" to additionally describe the state of the bus. Due to such a mismatch, MLE still imposes a heavy penalty with a loss of 4.51; however, SMILE performs a more lenient optimization with a loss of 0.57. This is because the word "parked" is not contained in the subset, and by allocating probabilities over the subset, the label word "and" obtains the highest predictive probability. Therefore, SMILE circumvents the penalty induced by a more 'detailed' prediction and blocks conciseness optimization.
SMILE generalizes well to video captioning. We conduct experiments on MSR-VTT [53] to test the generalization ability of SMILE on the video captioning task. MSR-VTT consists of 10K videos and each video clip is annotated with 20 English sentences. The BLIP4video [58], a modified version of BLIP designed for video tasks, is chosen as our basic model. As for evaluation, we apply the reference-free metric EMScore [38]. It evaluates embedding matching between video and caption in terms of precision (P), recall (R), and the F1 score (F), to measure the accuracy and descriptiveness of the captions. We also provide the self-retrieval performance, with the F1 score of EMScore for retrieving, and the test set as the candidates pool. Table 6 shows that SMILE also enables the model to generate longer descriptions on video captioning, and brings an overall improvement in both accuracy and descriptiveness.
In this work, we focus on generating descriptive image captions from the perspective of the language modeling objective. Existing works also propose different strategies to increase the descriptiveness of captions. For example, Gu et al. [11] and Liu et al. [26] design two-stage decoding strategies with caption generation and refinement. Liu et al. [26] and Shi et al. [39] integrate descriptive details from multiple-sentence annotations with additional rewards or natural language inference (NLI) relations. In addition to descriptiveness, existing works also employ contrastive learning [3, 6, 29] and self-retrieval strategy [28, 45] to increase the discriminability of captions. Wang et al. [48] reweight the ground truth captions to emphasize more discriminative ones. Wang et al. [49] highlights the uniqueness of each image within groups with similar images. More recently, CapEnrich [55] stimulates VLP models [59, 60] to express more details with prompting strategies [27, 34]. It achieves state-of-the-art performance on the descriptiveness and discriminability of image captioning. However, CapEnrich requires training on automatically constructed data with additional utterances, which integrate details from multiple captions annotated for each image. This leads to unconventional text structure and thus affects the fluency of the generated caption. In contrast to the above-mentioned methods, SMILE does not rely on either multiple annotated captions or the construction of additional data. To the best of our knowledge, SMILE is the first approach to enhance models' descriptive captioning capability by focusing on the language modeling objective itself.
Besides, text generation models trained with MLE tend to generate dull texts with high-frequency words [52, 37]. Attempts for fixing such degeneration issue including variant decoding methods [22, 46, 20, 14, 41] and learning algorithms [12, 9, 15, 51]. In this work, SMILE-optimized models output with much greater lexical diversity, also demonstrating the capacity of mitigation of such problems besides improving descriptiveness.
For descriptive image captioning, we revisit the commonly used maximum likelihood estimation (MLE) training objective. We argue that MLE is not perfectly suitable for image captioning tasks because of two conflicting optimizations: conciseness optimization and richness optimization. To steer the model toward generating more descriptive captions, we propose to mitigate conciseness optimization and maintain richness optimization by revising the objective with a simple but effective vocabulary subset selection strategy, namely Semipermeable MaxImum Likelihood Estimation (SMILE). Extensive experiments validate that SMILE helps models to generate much longer and more descriptive captions without complex architecture design or extra data annotation.
Technically, it is possible to apply SMILE to any autoregressive text generation task with an MLE objective. To investigate this, we conduct a preliminary exploration on the Supervised Fine-tuning [31] (SFT) of Large Language Models [44, 43] (LLMs) – fine-tuning LLMs to follow human instructions, such as writing and answering questions. Unfortunately, comparing SMILE with MLE for SFT, we don’t observe significant improvement in response length or overall quality (see Appendix B). One potential factor is that the target response in the training corpus is often very long (for instance, a paragraph exceeding 100 words). This causes that the constructed subsets are significantly larger than the ones in the image captioning task, and differ less from the entire vocabulary set. Therefore, SMILE may struggle to be directly applied to other text generation tasks. Further exploration of this issue will be reserved for future work.
This work was partially supported by the Beijing Natural Science Foundation (L233008), the National Natural Science Foundation of China (No. 62072462) and the National Key R&D Program of China (No.2020AAA0108600).
A SMILE Implementation
We present pseudocode in Algorithm 1 to better understand SMILE, which can be implemented and applied in such a straightforward manner. Further details including Reverse Subsetting and Random Subsetting can be found in our released code at https://github.com/yuezih/SMILE.
Algorithm 1 Pseudocode of SMILE in a PyTorch-like style.
B: batch size
N: sequence length
V: vocabulary size
logits: language model prediction logits (B x N x V)
labels: ground truth label at each position (B x N)  
if objective == 'MLE':
    loss = CrossEntropyLoss(logits, labels)  
elif objective == 'SMILE':
    # each sequence gets a unique mask to mask words not in the sequence
    logits_mask = zeros(B, V).scatter_(1, labels, True) # B x V
    # expand for every position in the sequence
    logits_mask = logits_mask unsqueeze(1).expand(-1, N, -1).clone() # B x N x V
    # perform first-token MLE (optional)
    logits_mask[:, 0, :] = True # B x N x V
    # apply mask to the logits
    selected_logits = logitsmasked_fill(logits_mask=False, float('-inf')) # B x N x V
    loss = CrossEntropyLossSelected_logits, labels)
scatter_: scatter values into a matrix by indices (https://pytorch.org/docs/stable/generated/torch.Tensor.scatter_.html); CrossEntropyLoss integrates the softmax operation.
Table 7: Performance of LLaMA-13B fine-tuned using MLE and SMILE, respectively, on the Vicuna evaluation framework. Scores are relative values re-scaled by taking the scores of reference answers as full marks (100). Eps.: Training epochs; Obj.: Learning objective; Len.: Average answer length; Lex.: Lexical diversity.
Eps.	Obj.	Len.	Lex.	Gene.	Know.	Role.	Com.Sen.	Fermi	Count.	Writ.	Overall
5	LMLE	94.8	2.4	82.3	88.2	82.9	85.6	77.4	85.3	83.1	83.5
LSMILE	95.8	2.4	92.4	86.2	85.3	90.4	79.6	85.4	85.8	86.4
10	LMLE	89.2	2.3	92.9	88.9	85.8	86.9	78.3	84.4	84.7	86.0
LSMILE	96.1	3.4	97.3	89.0	82.4	87.8	79.2	87.4	75.6	85.5
20	LMLE	90.2	2.3	92.0	88.0	88.2	91.3	75.7	86.5	82.2	86.3
LSMILE	96.4	3.5	93.4	89.2	91.1	90.6	81.6	85.3	62.2	84.8
B SFT of LLMs
Supervised fine-tuning (SFT) aims to enhance the capacity of large language models (LLMs) to follow human instructions better [31]. This section reports our experimental findings comparing SFT tasks carried out with MLE versus SMILE. We choose the widely used LLaMA-13B [44] as the basic model, and fine-tune it on the Alpaca dataset [42], which contains 52,000 instructions and demonstrations generated by the text-davinci-003 engine<sup>3</sup>. To ensure fair comparisons, the pretrained basic model is fine-tuned using both MLE and SMILE, maintaining identical settings for both objectives. Each experiment involves fine-tuning the baseline model using a Low-Rank Adaptation (LoRA) approach [17] for parameter efficiency, employing 4 RTX A6000 nodes with a batch size of 8. The fine-tuning process is conducted using the LMFlow framework [8]. For evaluation, we employ
3https://platform.openai.com/docs/models/gpt-3-5
Table 8: Performance of different basic models before and after SMILE optimization.
Model	\( {\mathcal{L}}_{\text{SMILE }} \)	Caption Length	Lexical Diversity	Self-Retrieval	CLIPScore	PPL
R@1	R@5
\( BLIP_{base} \)[23]	✓	10.0	1.4	6.7	16.6	77.2	95.8
22.3	4.5	10.0	24.5	75.0	95.6
\( BLIP_{large} \)[23]	✓	10.0	1.4	6.7	17.3	77.4	92.3
24.0	4.9	9.2	22.4	74.7	100.0
\( OFA_{base} \)[50]	✓	10.1	1.2	5.9	15.6	76.7	64.6
16.5	4.0	6.9	16.2	71.6	109.7
\( OFA_{large} \)[50]	✓	9.8	1.2	6.4	16.0	77.1	68.6
18.8	5.9	9.3	21.9	73.6	125.9
mPLUG [21]	✓	9.7	1.3	5.7	15.5	76.6	59.6
12.3	2.8	3.9	11.0	71.0	92.3
VinVL [60]	✓	9.9	1.3	5.5	14.4	76.9	60.6
25.2	5.6	5.4	13.8	71.9	113.9
the evaluation framework proposed in Vicuna [5]. It contains eight question categories, including generic (Gene.), knowledge (Know.), roleplay (Role.), common-sense (Com.Sen.), Fermi problems (Fermi), counterfactual (Count.), coding/math tasks, and writing (Writ.). with ten questions for each category. The model's answer is rated by GPT-4 [30] in terms of helpfulness, relevance, accuracy, and detail, by an overall score ranging from 1 to 10. To ensure consistent independent scoring, GPT-4 rates the candidate answer by comparing it with a reference answer (from GPT-3.5-turbo $^3$ ). Given that GPT-4 is not an efficient judge for coding/math tasks [5], and LLaMA-13B also struggles with these tasks, we exclude coding/math tasks in the evaluation.
Table 7 shows the performance of the models after fine-tuned for 5, 10, and 20 epochs using both MLE and SMILE. Compared to MLE, SMILE enables the model to generate slightly longer responses with enhanced lexical diversity. In terms of answer quality, the SMILE-optimized model demonstrates advantages across almost all task types at the early stages of training (after 5 epochs). However, these advantages diminish as the training process continues, particularly in the writing task, where the average quality of answers from the SMILE-optimized model notably declines. In conclusion, the significant effectiveness of SMILE in the SFT task is not evident. As stated in the Limitations section (Section 6) in the main paper, SMILE faces challenges in extending to other text generation tasks.
C Cross-model Generalizability
Since the aforementioned experiments validate the efficacy of SMILE on the base version of BLIP (and BLIP4video), it is crucial to investigate the generalizability of SMILE across different model architectures and sizes. Thus, we apply SMILE to several popular VLP models across scales [23, 50, 21, 60]. The implementation details are consistent with their respective officially released codes. Table 8 displays the performance of these models on MSCOCO. For all models, SMILE notably increases the average length and lexical diversity of the captions, indicating its 'semi-permeability' can generalize well to different model architectures and scales. Besides, for BLIP and OFA, SMILE also significantly enhances their descriptiveness, as evidenced by improved self-retrieval performance. However, such gains are not observed in mPLUG and VinVL. While the generated captions become longer with SMILE optimization, their accuracy is adversely affected, leading to dropped self-retrieval performance and CLIPScore. Regarding this discrepancy on different models, one possible reason might be that mPLUG and VinVL adopt Prefix Language Modeling (PrefixLM) [35] or Masked Language Modeling (MLM) [7] styles pre-training for text generation, whereas BLIP and OFA utilize Causal language modeling (CLM). We hypothesize that CLM is intrinsically more advantageous for text generation tasks, potentially enabling models to develop a generation pattern that is more robust and adaptable in embracing SMILE optimization. We leave further exploration on this matter for future work.
Table 9: Comparison of SMILE-optimized models and MLLMs.
Model	Caption Length	Lexical Diversity	Self-Retrieval	CLIPScore	PPL
R@1	R@5
BLIP-LSMILE	22.3	4.5	10.0	24.5	75.0	95.6
BLIP-LMLE+SMILE	19.8	3.6	10.9	25.1	76.2	79.4
MiniGPT-4	66.3	6.4	10.5	23.8	75.6	14.8
LLaVA	32.5	3.8	9.7	23.4	79.7	20.0
Table 10: Performance of models trained on MSCOCO with and without paraphrases.
Method	Paraphrase	Training Data	Caption Length	Lexical Diversity	Self-Retrieval	CLIPScore	PPL
R@1	R@5
BLIP	✓	567K	10.0	1.4	6.7	16.6	77.2	95.8
BLIP-LSMILE	✓	567K	22.3	4.5	10.0	24.5	75.0	95.6
BLIP-LSMILE	113K	17.9	3.3	8.9	21.3	75.2	90.3
D Comparison with Multimodal LLMs
With the advent of the LLM era, various Multimodal LLMs (MLLMs) [56] are developed to facilitate general multimodal tasks, including descriptive image captioning. Therefore, we conduct a shallow evaluation with two popular MLLMs, MiniGPT-4 (7B) [62] and LLaVA (Lightning-MPT-7B) [25], to compare them with our SMILE-optimized models. Models are prompted to generate captions with an example prompt from the LLaVA project, namely "Describe the following image." As shown in Table 9, BLIP with SMILE achieves comparable self-retrieval performance with MLLMs, while underperforming them obviously in both caption length (MiniGPT-4) and accuracy (LLaVA). However, it is important to note that such a direct comparison might be somewhat unfair. While MLLMs employ additional training data (e.g., detailed description) to learn to describe in detail, SMILE enables models to directly learn from existing data (e.g., short captions in MSCOCO) for more descriptive captioning. Besides, our model size ( $\sim$ 227M) and pre-training data scale are significantly smaller than MLLMs. Despite the strong capabilities demonstrated by LLMs, the heavy computational resources they rely on are worth noting.
E Paraphrases Ablation
Captioning datasets including MSCOCO and Flickr30K usually contain multiple annotations for one image or video [24, 57, 53], a.k.a., paraphrases. This may raise questions about whether the gains in descriptiveness come from combining these paraphrases, rather than 'absorbing' from the entire training dataset. On this topic, we test with a no-paraphrase corpus, keeping just one caption per image for training in MSCOCO. As shown in Table 10, by learning from a corpus without paraphrases, the model can still boost descriptiveness, with remarkable improvement in terms of caption length, lexical diversity, and self-retrieval performance. This suggests that paraphrases in training data are not the key to descriptiveness gains.