Title: Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image Captioning

Learning Objective

Natural Language Processing
Abstract: Image captioning aims to describe visual content in natural language. As 'a picture is worth a thousand words', there could be various correct descriptions for an image. However, with maximum likelihood estimation as the training objective, the captioning model is penalized whenever its prediction mismatches with the label. For instance, when the model predicts a word expressing richer semantics than the label, it will be penalized and optimized to prefer more concise expressions, referred to as *conciseness optimization*. In contrast, predictions that are more concise than labels lead to *richness optimization*. Such conflicting optimization directions could eventually result in the model generating general descriptions. In this work, we introduce Semipermeable MaxImum Likelihood Estimation (SMILE), which allows richness optimization while blocking conciseness optimization, thus encouraging the model to generate longer captions with more details. Extensive experiments on two mainstream image captioning datasets MSCOCO and Flickr30K demonstrate that SMILE significantly enhances the descriptiveness of generated captions. We further provide in-depth investigations to facilitate a better understanding of how SMILE works.

Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation

Abstract

Image captioning aims to describe visual content in natural language. As 'a picture is worth a thousand words', there could be various correct descriptions for an image. However, with maximum likelihood estimation as the training objective, the captioning model is penalized whenever its prediction mismatches with the label. For instance, when the model predicts a word expressing richer semantics than the label, it will be penalized and optimized to prefer more concise expressions, referred to as conciseness optimization . In contrast, predictions that are more concise than labels lead to richness optimization . Such conflicting optimization directions could eventually result in the model generating general descriptions. In this work, we introduce S emipermeable M ax I mum L ikelihood E stimation (SMILE), which allows richness optimization while blocking conciseness optimization, thus encouraging the model to generate longer captions with more details. Extensive experiments on two mainstream image captioning datasets MSCOCO and Flickr30K demonstrate that SMILE significantly enhances the descriptiveness of generated captions. We further provide in-depth investigations to facilitate a better understanding of how SMILE works.

GT: a ___ (pretty) woman holding a cake.

Model: woman (

less rich than label

)

→

GT: a pretty woman holding a ___ (cake).

Model: white (

less concise than label

)

→

exposed context | label | model prediction

Richness

Optimization

permeable

impermeable

Richness

Optimization

MLE

SMILE

Conciseness

Optimization

Conciseness

Optimization

(a)

(b)

1 Introduction

The task of generating textual descriptions for a given image, commonly referred to as image captioning [47, 54], has long been plagued by the issue of overly-generic outputs [48, 49]. That is, models tend to generate similar descriptions for distinct images using simple concepts, but lack details (as shown in Fig. 1). Such generated captions cannot meet various needs, e.g., fine-grained online image search and recommendation, automatic data annotation for training vision-language models [4, 33], and poses a long-standing challenge in the field of image captioning. Considerable efforts have been devoted to generating more descriptive image captions, including constructing paragraph caption datasets with more details for training models [19], designing additional rewards [26, 28, 29], and prompting [27, 34] vision-language pre-training (VLP) models [59, 60] to enrich captions [55]. While these efforts yield some promising improvements, they seldom try addressing this problem by revisiting the core optimization objective.

Similar to text generation tasks such as machine translation [40], image captioning models are trained to predict the next word given context (the visual content and preceding words). The training is supervised by the maximum likelihood estimation (MLE) objective, which evaluates the model's predictive distribution over the vocabulary, and penalizes the model when it fails to predict the current label. However, unlike machine translation where the target sequence is relatively certain, describing an image can be quite diverse, and limited human annotations may not cover all accurate captions. Therefore, the strict supervision with MLE is not perfectly suitable for image captioning optimization. As shown in Fig. 2, when the model predicts a word (e.g., " woman ") that is more 'concise' than the ground truth label (e.g., " pretty "), MLE computes a loss due to the mismatch and encourages the model to assign a higher probability to " pretty ". From the model optimization perspective, this optimizes the model to possess the more descriptive captioning ability, a process referred to as richness optimization in this work. In contrast, if the model predicts a word (e.g., " white ") expressing more details than the label (e.g., " cake "), MLE also penalizes the model and optimizes it towards a more concise captioning behavior ( conciseness optimization ). Since ground truth captions in commonly used datasets (e.g., MSCOCO [24]) are mostly simple (around 10 words), conciseness optimization is not rare during training, especially for finetuning VLP models [23, 60, 59]. But for image captioning, descriptions that contain more details rather than just generic concepts are usually preferred so that they can be used for accurately retrieving images or conveying more information for the visually impaired, etc. Therefore, conciseness optimization should be suppressed as much as possible, and richness optimization should be encouraged to incentivize the model to generate more descriptive captions.

In this work, to achieve this goal, we propose a simple but effective training objective for descriptive image captioning, namely Semipermeable MaxImum Likelihood Estimation (SMILE). Unlike MLE, which evaluates the predictive probability distribution over the entire vocabulary, SMILE considers the probability over a vocabulary subset with limited words, i.e., only containing words in the ground truth caption . Under typical conditions (with MLE), when a 'richer' word that conveys additional details gets a high confidence score (e.g., " white " in Fig. 2, and such details generally are not

included in the ground truth caption), it challenges the label word (e.g., " cake ") in the competition of probability allocation. This competition consequently leads to a decrease in the probability assigned to " cake ", thereby imposing a penalty on the model and thus resulting in conciseness optimization. However, with SMILE, probability allocation is limited to a subset of words that excludes " white ". Regardless of the confidence score given to " white ", it does not diminish the probability assigned to the label " cake ", thus avoiding a loss increase. In this way, we suppress the conciseness optimization that would have been imposed with MLE. In contrast, when the label word expresses details (e.g., " pretty " in Fig. 2) while the model prefers a more 'concise' word " woman ", it is highly likely that the more 'concise' word is part of the ground truth caption and therefore, included in the subset. Eventually, even if we only focus on the subset, the word still takes away the probability from the label word " pretty ", resulting in model penalization. In this way, richness optimization is maintained. Our objective allows richness optimization but blocks conciseness optimization, similar to a semipermeable membrane. We, therefore, name this objective Semipermeable MLE (SMILE).

To verify the effectiveness of our proposed SMILE, we conduct extensive experiments on the image captioning task. The results demonstrate that SMILE can effectively optimize the model to generate significantly longer and more descriptive captions. Besides, we carry out abundant experiments to analyze how SMILE works, such as ablations on subset selection and descriptiveness origin, etc. Finally, we verify the generalization ability of SMILE on the video captioning task and discuss its limitation when facing other text generation tasks.

2 Method

2.1 Maximum Likelihood Estimation (MLE)

We first introduce the standard MLE for captioning optimization. As with many text generation tasks [40, 1], the model is trained to maximize the likelihood of the label when predicting the current word w in the sequence given previous words w < and visual content v . The token-level loss function of MLE is defined as:

where the summation iterates over all words in the vocabulary V , y j is the j -th element of the one-hot label vector, and ˆ P V is the predictive probability distribution over V .

2.2 Semipermeable MaxImum Likelihood Estimation (SMILE)

Given a target sequence D = [ w 1 , w 2 , . . . , w N ] , we form a subset V D that only includes the unique words occurred in D , defined as V D = { w i | w i ∈ D } . Then, the current word w prediction is carried out within V D , with the loss function of SMILE defined as:

Here, ˆ P V D is the predictive distribution over the subset V D . Specifically, for the j -th word in V D , the probability ˆ p j is calculated by the relative significance of the confidence score of the word compared to all other words in V D :

The probability assigned to the label (the j ∗ -th word) determines the prediction loss. When a word w + expressing additional details beyond the ground truth caption gets a high confidence score (e.g., in Fig. 2, the label is " cake ", w + is " white ", and V D = { " a ", " pretty ", " woman ", " holding ", " cake " } , w + / ∈ V D ), w + does not contribute to the denominator in Eq. (3) since it is not included in V D , and thus does not induce a decrease of the probability assigned to the label word ˆ p j ∗ , which would have otherwise increased the prediction loss. In other words, when the model tends to predict words

like w + , these words do not participate in the probability allocation and do not take away any of the probability to be assigned to the label word, thus avoiding model penalization which leads to conciseness optimization. In contrast, when the model predicts a more 'concise' word w -(e.g., the label is " pretty " and w -is " woman "), such word w -usually is included in the ground truth caption, i.e., w -∈ V D . When allocating probability over V D , w -also participates and takes away some of the probabilities. Therefore, the model penalization is not avoided, and the induced richness optimization is maintained. Blocking conciseness optimization while allowing richness optimization is what SMILE is all about.

SMILE is for further training. Since SMILE only considers the relative probability distribution within a subset, it does not penalize the model to generate detailed words beyond the subset, thus suppressing conciseness optimization. However, it makes sense only if the predicted detail is correct. Therefore, SMILE is applied to further train a model that has already been optimized with MLE, which ensures its fundamental captioning capability. Please note that SMILE is only adopted in the training phase. During inference, the model predicts across the entire vocabulary to determine the current generation.

Initial context restriction It is well known that image captioning models are easy to suffer from exposure bias [2, 10, 36, 61]. When training in a teacher-forcing manner, the model is exposed to the ground truth context; however, during inference, each generation step is conditioned on the previous predictions. This causes a gap between training and inference. As SMILE does not require the model prediction to be consistent with the label as MLE does, the prediction is more prone to deviate from the label. Specifically, generating from the first token of the sequence, a prediction that is inconsistent with the label could lead to a completely unfamiliar context, exacerbating the exposure bias and affecting the autoregressive generation. Fortunately, we find that this issue can be effectively mitigated through some simple strategies, by ensuring that the initial context is correct, i.e., consistent with the label, which we name as initial context restriction and further discuss in Section 3.3.

3 Experiments

3.1 Experimental Setup

Basic model and baselines Our proposed SMILE is architecture-agnostic and can be applied to any visual captioning model compatible with MLE optimization. As a representative, we validate SMILE on the base version of BLIP [23], one of the state-of-the-art vision and language models pre-trained with 129M images and paired captions. We first fine-tune BLIP on downstream datasets using MLE as the basic model, and then further optimize it with SMILE. For baselines, we compare our method with the latest descriptive image captioning solution, CapEnrich [55], and another two recent models NliCap [39] and GdisCap [49]. We also provide the performance of human-annotated ground truth captions for reference.

Dataset We evaluate our method on the two most commonly used image captioning benchmarks, MSCOCO [24] and Flickr30K [57]. MSCOCO contains about 120K images, and we adopt the commonly used Karpathy splitting [18] with 5,000 images each for the validation and test sets. Flickr30K contains about 31K images, with 1,000 images each for the test and validation sets. For both datasets, each image has five human-annotated captions.

Evaluation We evaluate the models from three aspects: descriptiveness, accuracy, and fluency. Descriptiveness refers to how detailed the caption describes the image. Following previous works [55], we evaluate descriptiveness by the performance of CLIP self-retrieval, which employs the CLIP model [33] as a retriever to recall the image with its caption from a candidates pool. This is based on the fact that captions with more details can better distinguish different images. The candidates pool is the hard retrieval pool constructed in the CapEnrich work [55], which additionally includes more similar images beyond the test set, placing a higher requirement on the descriptiveness of captions. We also report the average length of the generated captions and the lexical diversity (unique word count of all captions). Accuracy refers to the relevance of the generated caption to the image visual content, which we measure automatically by CLIPScore [13]. It calculates the semantic similarity between the image and the generated caption using a pre-trained CLIP model. For fluency, we report

the language modeling perplexity 2 (PPL) of the captions with GPT-2 [32]. However, these automatic metrics have their limitations. For example, CLIP measures the overall semantics of the caption and struggles to focus on too many details. In addition, the evaluation models also suffer from the bias of the training data and thus may have difficulty in handling longer or more complex sentences. Therefore, we also conduct human evaluations, which will be discussed later.

3.2 Main Results

Versus basic model According to Table 1, compared with the basic BLIP model, SMILE significantly increases the average length of the captions (more than doubled on both datasets). SMILE leads to more detailed descriptions as well, greatly enhancing the model's self-retrieval performance. Although SMILE causes a slight decrease in the CLIPScore (-2.2 and -0.6 on MSCOCO and Flickr respectively), we consider this to be a reasonable phenomenon, as 'talks much errs much'. Besides, maintaining a comparable level of perplexity indicates that SMILE can incentivize longer and more detailed captions without compromising fluency.

Versus baselines As shown in Table 1, BLIP significantly outperforms all baselines on lexical diversity with SMILE optimization. Compared to traditional captioning methods that tend to generate dull texts with common words overused, SMILE effectively overcomes the lexical diversity bottleneck and outperforms humans. Our approach also achieves the best performance on self-retrieval. It is worth noting that the most competitive baseline CapEnrich requires automatically constructed data in a specific format, which integrates details from multiple manual captions of each image for training, along with carefully handcrafted or learnable prompts. In contrast, our approach is refreshingly simple, has fewer restrictions, and importantly, exhibits no incompatibility with other methods. Therefore, it can serve as an effective supplement or alternative to the currently employed solutions. Since our vanilla BLIP model with SMILE optimization already surpasses other methods that may require complicated design, we see no strong need for further attempts to combine SMILE with them in this work.

Human evaluation We randomly sample 100 images from the MSCOCO test set for human evaluation. For each image, we collect 4 candidate captions, including three captions generated by CapEnrich, basic model BLIP, and SMILE-optimized BLIP, respectively, and one human-annotated caption. All candidates are randomly shuffled. For each candidate caption, 5 human annotators are asked to independently rate on a scale of 1 to 5 (a higher score indicating better quality) from three aspects, namely descriptiveness, accuracy, and fluency. The 'descriptiveness' refers to the richness of the correct detail relevant to the image. Table 2 shows the average score of each model on the three aspects. SMILE substantially improves the descriptiveness of the captions, with a rate close to 5 ( excellent ), outperforming all other candidates including human-written captions by a large margin. A score beyond 4 for both accuracy and fluency demonstrates the good quality of captions

25

25

.Caption Length

20

20

Self-retrieval R@1

1

Len. (MLE)

Len. (SMILE)

15

15

R@1 (MLE)

R@1 (SMILE)

10

10

5

5

0

1

2

3

4

5

Epoch

by our SMILE-optimized captioning model, although it faces a higher risk of 'talks much errs much'. As descriptiveness emphasizes the recall of visual content while accuracy emphasizes description precision, we additionally calculate an F1 value of these two scores as the overall semantic score. As shown in Table 2, SMILE performs best when taking into account both aspects.

3.3 Ablation Study

Training epochs Fig. 3 presents the impact of training epochs on caption length and self-retrieval performance with SMILE optimization. It shows that SMILE takes only a few training epochs to achieve convergence and does not introduce too much extra training cost. For all of our models optimized with SMILE, we choose the checkpoints according to the best self-retrieval performance on the validation set, which always occurs within 3 epochs.

Basic model As the basic model used for SMILE optimization, BLIP is first pre-trained on a largescale image-text dataset and then fine-tuned by MLE on downstream datasets. Both the pre-training (PT) and fine-tuning (FT) equip the model with fundamental image captioning capabilities. To investigate how such fundamental captioning ability influences SMILE's effectiveness, we compare the performance of different basic models and their further SMILE-optimized counterparts on MSCOCO. As shown in Table 3, for basic models, both pre-training and fine-tuning improve the model's performance in terms of descriptiveness and accuracy (rows 1, 3, and 5). After subsequent SMILE optimization, each basic model can generate much longer descriptions (rows 2, 4, and 7). Regarding how much SMILE improves descriptiveness, it is notable that when the fundamental captioning ability is poor, SMILE brings no improvement and can further decrease the performance (row 2 vs 1); but once the basic model is able to perform acceptable captioning, SMILE can contributes a lot (rows 4 vs 3 and 7 vs 5). This implies that SMILE requires the fundamental captioning ability of the basic model in order to further correctly generate more details. In addition, as SMILE optimization introduces extra training steps, we further train the default basic model with MLE and validate that simply adding training steps does not bring improvement (row 6 vs 5).

Initial context restriction As aforementioned in Section 2.2, we propose initial context restriction to alleviate exposure bias for SMILE optimization. We design two implementations, First-token MLE and First-token Shifting , to achieve such restriction:

First-token MLE adopts MLE loss for the first token of the target sequence, and SMILE for the remaining tokens.

First-token Shifting shifts the label of the first token of the target sequence to its alternative. For example, in the vocabulary of the BERT model [7], the word " a " can be replaced with its subword form " ##a ", which rarely appears to be the first token in the training corpus. By doing so, we replace the familiar word with an unfamiliar one, making it difficult for the model to predict correctly (even within a subset). This often results in model penalization, making the prediction of the first token more likely to be consistent with the ground truth.

Since the shifting approach is designed with certain requirements on the model vocabulary, making it potentially difficult to be applied to all models, we use the First-token MLE by default to address the exposure bias issue. To demonstrate the effectiveness of this strategy, in Table 4, we show that without initial context restriction, SMILE optimization can not achieve promising results. Either the default First-token MLE implementation or the First-token Shifting implementation helps alleviate exposure bias and the former is better.

Mixing L MLE with L SMILE Since SMILE improves descriptiveness at the expense of some accuracy, it could pose a risk in scenarios that demand higher accuracy. To strike a balance, we carry out experiments where the overall learning objective combines both L MLE and L SMILE, defined as:

Table 5 illustrates how the balance between MLE and SMILE impacts the performance in terms of descriptiveness and accuracy. As L SMILE becomes more dominant, the model tends to generate longer outputs with higher lexical diversity and descriptiveness; however, accuracy correspondingly decreases. This demonstrates that a compromise between descriptiveness and accuracy can be achieved by simply combining L SMILE with L MLE. Notably, we find that when L MLE is incorporated at a very low ratio (0.01), the generated captions achieve the best performance in self-retrieval. This suggests that while SMILE facilitates more details, ensuring accuracy is also important for captions to better describe and distinguish images.

(a)

(b)

25

5

. Caption Length

20

Avg. Caption Length

MLE

MLE (Simplest-COCO)

SMILE

4

SMILE (Simplest-COCO)

15

Random

MLE (Simpler-COCO)

Reverse

SMILE (Simpler-COCO)

3

10

2

5

0

1

2

3

4

5

0

1

2

3

4

5

Epoch

Epoch

3.4 Further Analysis

In this section, we conduct experiments to further analyze how SMILE works. As we hypothesized, SMILE implements a 'semi-permeability' to retain only richness optimization, leading the model to generate more details. Therefore, we first investigate whether such an effect is attributed to our subset selection. Then , we affirm the 'semi-permeability' by flipping it to observe an opposite property of SMILE. Besides , since BLIPL SMILE surpasses human annotation in descriptiveness as shown in Table 1 and Table 2, we analyze where it gains the capability to generate details. Furthermore , we provide a visualization to illustrate how SMILE exerts its effect from the token-level loss. Finally , we test the generalization ability of SMILE on video captioning.

Subset selection is key in SMILE. We argue that the effectiveness of SMILE is attributed to our subset selection strategy that selects the target sequence words. To verify this, we ablate the SMILE subsetting strategy by replacing it with Random subsetting , that is, for a given target sequence, we randomly choose 10 words (the average caption length of MSCOCO) from the entire vocabulary to form the subset, ensuring the subset size is relatively consistent with that of SMILE. Fig. 4 (a) illustrates the impact of SMILE and Random subsetting on the generation length during training. Random subsetting maintains the describing habits of the basic model, neither lengthening nor shortening the generation length. This is due to that the model's preferred predictions among the entire vocabulary, whether more or less 'rich' than the label, are less likely to be included in the randomly-chosen subset. This leads to a mechanism akin to 'bidirectional impermeability', where both conciseness and richness optimization are weakened. This also demonstrates that it is the subset selection of SMILE that contributes to the increasing length of model generation.

Reverse subsetting flips the 'semi-permeability'. Since SMILE achieves a 'semi-permeability' which allows richness optimization while blocks conciseness optimization, it is natural to assume that such 'semi-permeability' could be flipped with a reversed subset selection. Thus, we conduct experiments by proposing Reverse subsetting , that is, given a target sequence D = [ w 1 , w 2 , . . . , w N ] , when predicting each word, we select the complement of the target sequence word set in the entire vocabulary, plus the current label, as the subset. Hence, for the prediction of the i -th word, the corresponding subset is V S i = { w | w / ∈ V D } ∪ { w i } . As shown in Fig. 4 (a), with a subset complementary to that of SMILE, Reverse subsetting exhibits the opposite property of SMILE, leading the model to generate increasingly shorter captions. With these observations, we confirm that there is a 'semi-permeability', which is achieved by our subsetting strategy and can be inverted.

Models 'absorb' descriptiveness from the corpus. Although SMILE achieves unidirectional richness optimization, one may still wonder where models with SMILE optimization learn the descriptive captioning ability to surpass human annotators. We suspect that models could learn the usage of details from the whole corpus. To verify this hypothesis, we construct two datasets based on MSCOCO, namely Simplest-COCO, which contains no details, and Simpler-COCO, with only a few details. Concretely, for Simplest-COCO, we extract the subject from the caption and prefix it with a definite article based on grammatical rules, such as " a girl ". For Simpler-COCO, we extract the subject-verb-object constituent of the caption as a new caption with few details, such as " a man working ". All the mentioned natural language processing (e.g., parsing and part-of-speech tagging)

(a) MLE

(b) SMILE

is performed with SpaCy [16]. The average caption length for Simplest-COCO and Simpler-COCO is 2.0 and 2.5, respectively. On these two datasets, we first train the default basic model with MLE until convergence and further optimize it with SMILE. As shown in Fig. 4 (b), only MLE optimization does not achieve longer descriptions than the training corpus on both datasets. On the other hand, with SMILE optimization, the description length does not increase when training on Simplest-COCO; however, on Simpler-COCO, the model generates captions about twice as long as the ground truths. This suggests that richness optimization relies on the details in the whole training corpus, and SMILE helps models 'absorb' the capability to express more details.

Visualization of token-level penalization, MLE versus SMILE Taking a sample from the MSCOCO training set as an example, Fig. 5 shows the predictive distribution and loss value for each word during MLE and SMILE optimization, respectively. Note that the predictions in Fig. 5 (a) and (b) are given by an identical model but the former distribution is calculated over the whole vocabulary, while the latter over the SMILE subset. When predicting the second word (" white " in the target sequence), the model prefers a more 'concise' word " bus " than the label " white ". For MLE, " bus " takes away much of the probability, resulting in a significant loss that causes richness optimization. Similar to MLE, SMILE also penalizes such prediction since both " white " and " bus " are included in the subset. When predicting the sixth word (the label word is " and "), the model prefers a more 'detailed' word " parked " to additionally describe the state of the bus. Due to such a mismatch, MLE still imposes a heavy penalty with a loss of 4.51; however, SMILE performs a more lenient optimization with a loss of 0.57. This is because the word " parked " is not contained in the subset, and by allocating probabilities over the subset, the label word " and " obtains the highest predictive probability. Therefore, SMILE circumvents the penalty induced by a more 'detailed' prediction and blocks conciseness optimization.

SMILE generalizes well to video captioning. We conduct experiments on MSR-VTT [53] to test the generalization ability of SMILE on the video captioning task. MSR-VTT consists of 10K videos and each video clip is annotated with 20 English sentences. The BLIP4video [58], a modified version of BLIP designed for video tasks, is chosen as our basic model. As for evaluation, we apply the reference-free metric EMScore [38]. It evaluates embedding matching between video and caption in terms of precision (P), recall (R), and the F1 score (F), to measure the accuracy and descriptiveness of the captions. We also provide the self-retrieval performance, with the F1 score of EMScore for retrieving, and the test set as the candidates pool. Table 6 shows that SMILE also enables the model to generate longer descriptions on video captioning, and brings an overall improvement in both accuracy and descriptiveness.

4 Related Works

In this work, we focus on generating descriptive image captions from the perspective of the language modeling objective. Existing works also propose different strategies to increase the descriptiveness of captions. For example, Gu et al. [11] and Liu et al. [26] design two-stage decoding strategies with caption generation and refinement. Liu et al. [26] and Shi et al. [39] integrate descriptive details from multiple-sentence annotations with additional rewards or natural language inference (NLI) relations. In addition to descriptiveness, existing works also employ contrastive learning [3, 6, 29] and self-retrieval strategy [28, 45] to increase the discriminability of captions. Wang et al. [48] reweight the ground truth captions to emphasize more discriminative ones. Wang et al. [49] highlights the uniqueness of each image within groups with similar images. More recently, CapEnrich [55] stimulates VLP models [59, 60] to express more details with prompting strategies [27, 34]. It achieves state-of-the-art performance on the descriptiveness and discriminability of image captioning. However, CapEnrich requires training on automatically constructed data with additional utterances, which integrate details from multiple captions annotated for each image. This leads to unconventional text structure and thus affects the fluency of the generated caption. In contrast to the above-mentioned methods, SMILE does not rely on either multiple annotated captions or the construction of additional data. To the best of our knowledge, SMILE is the first approach to enhance models' descriptive captioning capability by focusing on the language modeling objective itself.

Besides, text generation models trained with MLE tend to generate dull texts with high-frequency words [52, 37]. Attempts for fixing such degeneration issue including variant decoding methods [22, 46, 20, 14, 41] and learning algorithms [12, 9, 15, 51]. In this work, SMILE-optimized models output with much greater lexical diversity, also demonstrating the capacity of mitigation of such problems besides improving descriptiveness.

5 Conclusion

For descriptive image captioning, we revisit the commonly used maximum likelihood estimation (MLE) training objective. We argue that MLE is not perfectly suitable for image captioning tasks because of two conflicting optimizations: conciseness optimization and richness optimization. To steer the model toward generating more descriptive captions, we propose to mitigate conciseness optimization and maintain richness optimization by revising the objective with a simple but effective vocabulary subset selection strategy, namely Semipermeable MaxImum Likelihood Estimation (SMILE). Extensive experiments validate that SMILE helps models to generate much longer and more descriptive captions without complex architecture design or extra data annotation.

6 Limitations

Technically, it is possible to apply SMILE to any autoregressive text generation task with an MLE objective. To investigate this, we conduct a preliminary exploration on the Supervised Fine-tuning [31] (SFT) of Large Language Models [44, 43] (LLMs) - fine-tuning LLMs to follow human instructions, such as writing and answering questions. Unfortunately, comparing SMILE with MLE for SFT, we don't observe significant improvement in response length or overall quality (see Appendix B). One potential factor is that the target response in the training corpus is often very long (for instance, a paragraph exceeding 100 words). This causes that the constructed subsets are significantly larger than the ones in the image captioning task, and differ less from the entire vocabulary set. Therefore, SMILE may struggle to be directly applied to other text generation tasks. Further exploration of this issue will be reserved for future work.

References

Mehdi Allahyari, Seyedamin Pouriyeh, Mehdi Assefi, Saeid Safaei, Elizabeth D Trippe, Juan B Gutierrez, and Krys Kochut. Text summarization techniques: a brief survey. ArXiv preprint , abs/1707.02268, 2017. URL https://arxiv.org/abs/1707.02268 .

Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 1171-1179, 2015. URL https://proceedings. neurips.cc/paper/2015/hash/e995f98d56967d946471af29d7bf99f1-Abstract.html .

Fuhai Chen, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, and Jinsong Su. Groupcap: Group-based image captioning with structured relevance and diversity constraints. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 1345-1353. IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00146. URL http://openaccess.thecvf.com/content_cvpr_2018/ html/Chen_GroupCap_Group-Based_Image_CVPR_2018_paper.html .

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Proceedings of the European conference on computer vision (ECCV) , pages 104-120. Springer, 2020.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/ .

Bo Dai and Dahua Lin. Contrastive learning for image captioning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 898-907, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 46922a0880a8f11f8f69cbb52b1396be-Abstract.html .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423 .

Shizhe Diao, Rui Pan, Hanze Dong, KaShun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. https://optimalscale.github.io/LMFlow/ , 2023.

Adji B. Dieng, Kyunghyun Cho, David M. Blei, and Yann LeCun. Learning with reflective likelihoods, 2019. URL https://openreview.net/forum?id=SJlh2jR9FX .

Anirudh Goyal, Alex Lamb, Ying Zhang, Saizheng Zhang, Aaron C. Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 4601-4609, 2016. URL https://proceedings.neurips.cc/paper/2016/hash/ 16026d60ff9b54410b3435b403afd226-Abstract.html .

Jiuxiang Gu, Jianfei Cai, Gang Wang, and Tsuhan Chen. Stack-captioning: Coarse-to-fine learning for image captioning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , pages 6837-6844, 2018. URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16465 .

Tianxing He and James Glass. Negative training for neural dialogue response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 2044-2058, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. acl-main.185. URL https://aclanthology.org/2020.acl-main.185 .

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7514-7528, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021.emnlp-main.595. URL https://aclanthology.org/2021.emnlp-main.595 .

Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1638-1649, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/ P18-1152. URL https://aclanthology.org/P18-1152 .

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview. net/forum?id=rygGQyrFvH .

Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.

Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.

Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015 , pages 3128-3137. IEEE Computer Society, 2015. doi: 10.1109/CVPR.2015.7298932. URL https://doi.org/10.1109/CVPR.2015.7298932 .

Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages 3337-3345, 2017. doi: 10.1109/CVPR.2017.356. URL https://doi.org/10.1109/CVPR.2017.356 .

Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. Importance of a search strategy in neural dialogue modelling. ArXiv preprint , abs/1811.00907, 2018. URL https://arxiv.org/abs/1811.00907 .

Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision-language learning by cross-modal skip-connections. arXiv preprint arXiv:2205.12005 , 2022.

Jiwei Li, Will Monroe, and Dan Jurafsky. A simple, fast diverse decoding algorithm for neural generation. ArXiv preprint , abs/1611.08562, 2016. URL https://arxiv.org/abs/1611. 08562 .

Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 12888-12900. PMLR, 2022. URL https://proceedings.mlr.press/v162/li22n.html .

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485 , 2023.

Lixin Liu, Jiajun Tang, Xiaojun Wan, and Zongming Guo. Generating diverse and descriptive image captions using visual paraphrases. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 , pages 4239-4248, 2019. doi: 10.1109/ICCV.2019.00434. URL https://doi.org/10.1109/ICCV. 2019.00434 .

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35, 2023.

Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, and Xiaogang Wang. Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data. In Proceedings of the European conference on computer vision (ECCV) , pages 338-354, 2018.

Ruotian Luo, Brian L. Price, Scott Cohen, and Gregory Shakhnarovich. Discriminability objective for training descriptive captions. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 6964-6974, 2018. doi: 10.1109/CVPR.2018.00728. URL http://openaccess.thecvf.com/content_ cvpr_2018/html/Luo_Discriminability_Objective_for_CVPR_2018_paper.html .

OpenAI. Gpt-4 technical report, 2023.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 2021. URL http://proceedings.mlr.press/v139/radford21a.html .

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21:140:1-140:67, 2020. URL http://jmlr. org/papers/v21/20-074.html .

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016. URL http://arxiv.org/abs/1511.06732 .

Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 1702-1723, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1170. URL https://aclanthology.org/N19-1170 .

Yaya Shi, Xu Yang, Haiyang Xu, Chunfeng Yuan, Bing Li, Weiming Hu, and Zheng-Jun Zha. Emscore: Evaluating video captioning via coarse-grained and fine-grained embedding matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17929-17938, 2022.

Zhan Shi, Hui Liu, and Xiaodan Zhu. Enhancing descriptive image captioning with natural language inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , pages 269-277, 2021.

Felix Stahlberg. Neural machine translation: A review. Journal of Artificial Intelligence Research , 69:343-418, 2020.

Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. ArXiv preprint , abs/2202.06417, 2022. URL https://arxiv.org/abs/2202.06417 .

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science, 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.

Gilad Vered, Gal Oren, Yuval Atzmon, and Gal Chechik. Joint optimization for cooperative image captioning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 , pages 8897-8906. IEEE, 2019. doi: 10.1109/ICCV.2019.00899. URL https://doi.org/10.1109/ICCV.2019.00899 .

Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015 , pages 3156-3164. IEEE Computer Society, 2015. doi: 10.1109/CVPR.2015.7298935. URL https://doi.org/10.1109/CVPR.2015. 7298935 .

Jiuniu Wang, Wenjia Xu, Qingzhong Wang, and Antoni B Chan. Compare and reweight: Distinctive image captioning using similar images sets. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16 , pages 370-386. Springer, 2020.

Jiuniu Wang, Wenjia Xu, Qingzhong Wang, and Antoni B Chan. Group-based distinctive image captioning with memory attention. In Proceedings of the 29th ACM International Conference on Multimedia , pages 5020-5028, 2021.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning , pages 23318-23340. PMLR, 2022.

Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=SJeYe0NtvH .

Jason Weston, Emily Dinan, and Alexander Miller. Retrieve and refine: Improved sequence generation models for dialogue. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI , pages 87-92, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-5713. URL https://aclanthology.org/W18-5713 .

Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 5288-5296. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.571. URL https://doi.org/10.1109/ CVPR.2016.571 .

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 , volume 37 of JMLR Workshop and Conference Proceedings , pages 2048-2057. JMLR.org, 2015. URL http://proceedings.mlr.press/ v37/xuc15.html .

Linli Yao, Weijing Chen, and Qin Jin. Capenrich: Enriching caption semantics for web images via cross-modal pre-trained knowledge. ArXiv preprint , abs/2211.09371, 2022. URL https://arxiv.org/abs/2211.09371 .

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 , 2023.

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , 2:67-78, 2014. doi: 10.1162/ tacl_a_00166. URL https://aclanthology.org/Q14-1006 .

Zihao Yue, Yuqi Liu, Liang Zhang, Linli Yao, and Qin Jin. Rucaim3-tencent at trecvid 2022: Video to text description. In Proceedings of TRECVID 2022 . NIST, USA, 2022. URL https://www-nlpir.nist.gov/projects/tvpubs/tv22.papers/rucaim3-tencent.pdf .

Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 25994-26009. PMLR, 2022. URL https://proceedings.mlr.press/v162/ zeng22c.html .

Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in visionlanguage models. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 5579-5588. Computer Vision Foundation / IEEE, 2021. doi: 10.1109/CVPR46437.2021.00553. URL https://openaccess.thecvf.com/content/CVPR2021/html/Zhang_VinVL_Revisiting_ Visual_Representations_in_Vision-Language_Models_CVPR_2021_paper.html .

Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4334-4343, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1426. URL https: //aclanthology.org/P19-1426 .

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023.

A SMILE Implementation

We present pseudocode in Algorithm 1 to better understand SMILE, which can be implemented and applied in such a straightforward manner. Further details including Reverse Subseting and Random Subseting can be found in our released code at https://github.com/yuezih/SMILE .

Algorithm 1 Pseudocode of SMILE in a PyTorch-like style.

scatter_ : scatter values into a matrix by indices ( https://pytorch.org/docs/stable/generated/torch. Tensor.scatter_.html ); CrossEntropyLoss integrates the softmax operation.

B SFT of LLMs

Supervised fine-tuning (SFT) aims to enhance the capacity of large language models (LLMs) to follow human instructions better [31]. This section reports our experimental findings comparing SFT tasks carried out with MLE versus SMILE. We choose the widely used LLaMA-13B [44] as the basic model, and fine-tune it on the Alpaca dataset [42], which contains 52,000 instructions and demonstrations generated by the text-davinci-003 engine 3 . To ensure fair comparisons, the pretrained basic model is fine-tuned using both MLE and SMILE, maintaining identical settings for both objectives. Each experiment involves fine-tuning the baseline model using a Low-Rank Adaptation (LoRA) approach [17] for parameter efficiency, employing 4 RTX A6000 nodes with a batch size of 8. The fine-tuning process is conducted using the LMFlow framework [8]. For evaluation, we employ

the evaluation framework proposed in Vicuna [5]. It contains eight question categories, including generic (Gene.), knowledge (Know.), roleplay (Role.), common-sense (Com.Sen.), Fermi problems (Fermi), counterfactual (Count.), coding/math tasks, and writing (Writ.), with ten questions for each category. The model's answer is rated by GPT-4 [30] in terms of helpfulness, relevance, accuracy, and detail, by an overall score ranging from 1 to 10. To ensure consistent independent scoring, GPT-4 rates the candidate answer by comparing it with a reference answer (from GPT-3.5-turbo 3 ). Given that GPT-4 is not an efficient judge for coding/math tasks [5], and LLaMA-13B also struggles with these tasks, we exclude coding/math tasks in the evaluation.

Table 7 shows the performance of the models after fine-tuned for 5, 10, and 20 epochs using both MLEand SMILE. Compared to MLE, SMILE enables the model to generate slightly longer responses with enhanced lexical diversity. In terms of answer quality, the SMILE-optimized model demonstrates advantages across almost all task types at the early stages of training (after 5 epochs). However, these advantages diminish as the training process continues, particularly in the writing task, where the average quality of answers from the SMILE-optimized model notably declines. In conclusion, the significant effectiveness of SMILE in the SFT task is not evident. As stated in the Limitations section (Section 6) in the main paper, SMILE faces challenges in extending to other text generation tasks.

C Cross-model Generalizability

Since the aforementioned experiments validate the efficacy of SMILE on the base version of BLIP (and BLIP4video), it is crucial to investigate the generalizability of SMILE across different model architectures and sizes. Thus, we apply SMILE to several popular VLP models across scales [23, 50, 21, 60]. The implementation details are consistent with their respective officially released codes. Table 8 displays the performance of these models on MSCOCO. For all models, SMILE notably increases the average length and lexical diversity of the captions, indicating its 'semi-permeability' can generalize well to different model architectures and scales. Besides, for BLIP and OFA, SMILE also significantly enhances their descriptiveness, as evidenced by improved self-retrieval performance. However, such gains are not observed in mPLUG and VinVL. While the generated captions become longer with SMILE optimization, their accuracy is adversely affected, leading to dropped self-retrieval performance and CLIPScore. Regarding this discrepancy on different models, one possible reason might be that mPLUG and VinVL adopt Prefix Language Modeling (PrefixLM) [35] or Masked Language Modeling (MLM) [7] styles pre-training for text generation, whereas BLIP and OFA utilize Causal language modeling (CLM). We hypothesize that CLM is intrinsically more advantageous for text generation tasks, potentially enabling models to develop a generation pattern that is more robust and adaptable in embracing SMILE optimization. We leave further exploration on this matter for future work.

D Comparison with Multimodal LLMs

With the advent of the LLM era, various Multimodal LLMs (MLLMs) [56] are developed to facilitate general multimodal tasks, including descriptive image captioning. Therefore, we conduct a shallow evaluation with two popular MLLMs, MiniGPT-4 (7B) [62] and LLaVA (Lightning-MPT-7B) [25], to compare them with our SMILE-optimized models. Models are prompted to generate captions with an example prompt from the LLaVA project, namely " Describe the following image. ". As shown in Table 9, BLIP with SMILE achieves comparable self-retrieval performance with MLLMs, while underperforming them obviously in both caption length (MiniGPT-4) and accuracy (LLaVA). However, it is important to note that such a direct comparison might be somewhat unfair. While MLLMs employ additional training data (e.g., detailed description) to learn to describe in detail, SMILE enables models to directly learn from existing data (e.g., short captions in MSCOCO) for more descriptive captioning. Besides, our model size ( ∼ 227M) and pre-training data scale are significantly smaller than MLLMs. Despite the strong capabilities demonstrated by LLMs, the heavy computational resources they rely on are worth noting.

E Paraphrases Ablation

Captioning datasets including MSCOCO and Flickr30K usually contain multiple annotations for one image or video [24, 57, 53], a.k.a., paraphrases. This may raise questions about whether the gains in descriptiveness come from combining these paraphrases, rather than 'absorbing' from the entire training dataset. On this topic, we test with a no-paraphrase corpus, keeping just one caption per image for training in MSCOCO. As shown in Table 10, by learning from a corpus without paraphrases, the model can still boost descriptiveness, with remarkable improvement in terms of caption length, lexical diversity, and self-retrieval performance. This suggests that paraphrases in training data are not the key to descriptiveness gains.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n## Summary\n\n- In the paper, the authors introduce a new method for generating descriptive captions for images. They call it SMILE, which stands for \"Semipermeable Maximum Likelihood Estimation.\" Unlike traditional image captioning models that often produce generic descriptions, SMILE allows for more detailed and rich descriptions while still maintaining accuracy.\n- Image captioning differs from Machine Translation in that there are numerous ways to describe an image, and limited human captions cannot encompass the entirety of possible descriptions. To address this issue, SMILE utilizes information from the entire corpus. The model used is BLIP, which is first fine-tuned with standard MLE loss and then further fine-tuned with SMILE loss. The dataset used for this project includes MSCOCO and Flickr30K.\n- SMILE only operates at training time.\n- The key idea in SMILE is restricting the vocabulary to only the unique words present in the target caption during decoding.\n- Since most generation models are trained with teacher forcing, they might suffer from exposure bias. Since SMILE does not require prediction to be consistent with ground truth labels, it is more prone to deviate from the ground truth label. To alleviate this problem, the authors propose two techniques\n\t- First Token MLE\n\t\t- MLE loss for first token, SMILE loss for remaining tokens in the sequence.\n\t- First Token Shifting\n\t\t- Change the label of first token to a similar token \"a\" -> \"##a\" where ## denotes the subword prefixed with a space. This results in model making a mistake on first token and encourages the model to get the first token right during the SMILE training and subsequently during inference.\n- The metrics used are CLIP score, caption length, lexical diversity, self-retrieval@1 and @5, and perplexity for measuring fluency.\n- The baseline models compared against are NliCap, GdiCap and CapEnrich which are quite standard for descriptive image captioning. The authors also do human evaluation on a small subset (50 examples).\n- Results indicate that the SMILE technique is able to generate more descriptive captions.\n\n\n####  nit\n- Typo Line 259 SMLIE -> SMILE\n\nSTRENGTHS:\n- Paper tackles a significant problem in image captioning. \n- Easy to read, prior work is cited well, and results are promising.\n\nWEAKNESSES:\n- Most metrics used are automatic, and the human evaluation set is very small. \n- Paper could also benefit from additional details around human evaluation experiments.\n\nQUESTIONS:\n### Remarks and Questions\n\n- All metrics used in the paper are automatic metrics. Have you considered using RefCLIPScore instead of CLIPScore, which also gives some weight to the reference captions?\n\t- Perhaps another metric to consider could be running an object detector on the image (like RCNN and collecting all objects correctly identified in the caption)\n- Can you provide more details on human evaluation? 50 seems a very small number of samples to justify the results conclusively.  In Table 2, descriptiveness improves to 4.64, but there is also a drop in accuracy compared to the last row.\n- Both MSCOCO And Flickr30K datasets provide multiple captions. It is unclear how the multiple reference captions are used from reading the paper. While constructing the subset vocabulary, do we take a union of all tokens from all the captions for an image? \n\t- Maybe the gains in descriptiveness are coming from combining all captions and the analysis presented in the section \"Models absorb details from the corpus.\"\n- For the video captioning task, the human perplexity is quite high, 561.8, which is significantly higher than other numbers, any idea why this is the case?\n\nLIMITATIONS:\nThe authors have appropriately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel further training approach called SMILE (Submodular Mixup Language Evaluation) for descriptive image captioning, which aims to improve the diversity and quality of generated captions. SMILE introduce to suppress conciseness optimization while encourge richness optimization to incentivize the model to generate more descriptive captions. It is based on submodular optimization and mixup techniques. It is worth mentioning that，the proposed method is architecture-agnostic and can be applied to any captioning model with MLE optimization. Experiments, conducted on two different datasets, show that the proposed approach can achieve competitive performance on three aspects: descriptiveness, accuracy, and fluency.\n\nSTRENGTHS:\n1. The perspective of model optimization about the unbefitting strict supervision with MLE for image captioning is intuitive. It also takes consideration into two optimizations, richness optimization and conciseness optimization.\n2. A flexible and lightweight modification of MLE with a reconducted subset has been developed for further training. This modification is designed to be architecture-agnostic, meaning it can be utilized with any visual captioning model that incorporates MLE optimization.\n\nWEAKNESSES:\n1. If I understand the paper correctly, the SMILE loss suppresses the conciseness optimization and maintains the richness optimization.  The former removes the forcing optimization for descriptive and correct words that are typically not in the subset, while the latter takes into account the concise and correct words that are included in the ground-truth caption and thus in the subset. The criterion used to assess richness and conciseness appears to be relatively simple.\n2. The proposed approach in this paper involves a scheme for further training existing models for image captioning. Nevertheless, several large models, including MiniGPT-4 and LLaVA, have been developed to generate more descriptive captions. Therefore, it is essential to compare the proposed approach with these existing models in the experiment.\n3. During inference, if the target sequence is unknown and the subset is unavailable, does the model output a probability distribution over the whole vocabulary? This can potentially create inconsistencies between the training and testing phases.\n4. In Section 3.4, the authors discussed the \"semi-permeability\" within the subset selection strategy and presented their findings in Figure 4. However, the meaning of \"semi-permeability\" is not clearly defined. Specifically, it is unclear whether \"semi-permeability\" refers to the ability to select and optimize models between the two extremes of simplicity and richness. Furthermore, while Figure 4 shows the relationship between the subset selection strategy and caption length, it does not shed light on the relationship between \"semi-permeability\" and \"caption length.\" \n5. The main text and supplementary material do not provide any qualitative results.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations and, if applicable, potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed the SMILE optimization method to enrich the detailed description of images in Captioning models. The authors analyzed the optimization process in the Image Captioning task and divided the optimization process into conciseness optimization and richness optimization. By suppressing the conciseness optimization process, the authors help the model to enrich the detailed description of the text. Specifically, the authors further train the already trained captioning model and then only calculate the loss of vocabulary on GT during the optimization process, thus avoiding the simple GT suppressing the model from outputting detailed vocabulary.\n\nSTRENGTHS:\n1.\tThe authors proposed the SMILE optimization method, which can significantly improve the model's degree of image description and enrich the vocabulary richness of the output Caption.\n\n2.\tThe ablation study built by the authors provides a detailed discussion of the working principle of the SMILE method. By selecting the right subset selection strategy, the model's output caption's degree of image description can be controlled. Furthermore, the effectiveness of this method in other tasks (Video Captioning) is demonstrated.\n\n3.\tThe SMILE method is a fine-tuning method, so it can help a model to converge quickly and enrich the description of images in Captioning models.\n\nWEAKNESSES:\n1.\tSMILE shows a significant decline in the CLIPScore metric across multiple experiments, and there is also a significant drop in the Acc metric for semantic evaluation in Table 2, indicating that SMILE has some issues concerning semantic correctness.\n\n2.\tThe authors only tested the effect of SMILE on the BLIP model. It would be better to test a few more models to ensure the robustness of this method.\n\n3.\tIn the \"Models 'absorb' details from the corpus\" section, a more refined metric might be needed to highlight the effect. Using only one visualization example lacks persuasiveness.\n\nQUESTIONS:\n1.\tCan it be demonstrated that SMILE does not have a significant loss in semantic correctness?\n2.\tIs there a more specific metric to show why SMILE can help enrich the image description?\n\nLIMITATIONS:\n1.\tFrom the current experimental results, the SMILE method has a significant loss in the correctness of image descriptions. Therefore, how to help the model enrich the image description while ens", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper argues that captioning models with maximum likelihood estimation as the training objective led to conciseness optimization, which limits the model to generate descriptive captions. To this end, they introduce semipermeable maximum likelihood estimation (SMILE), which allows richness optimization for generating longer and more detailed sentences.  As a result, they conduct extensive experiments on MSCOCO and Flickr30K datasets, which demonstrate the SMILE significantly enhances the descriptiveness of generated captions.

STRENGTHS:
1.	**The writing is clear and well-motivated**. It is urgent to explore a new training object function for image captioning optimization. And conciseness optimization is a core problem that needs to be addressed in image/video captioning.
2.	**The proposal SMILE is very subtle**. SMILE considers the probability over a vocabulary subset with limited words (words in the ground truth caption), which is an interesting exploration. The block-level conciseness optimization idea is effective.
3.    **The result is convinced**. The result demonstrates that SMILE can generate significantly longer and more descriptive captions.
4.   **The SMILE is architecture-agnostic and plug-and-play**.  It can generalize to many existing VLP models.

WEAKNESSES:
1. **The SMILE is not generalized to the large language models**. It doesn’t observe significant improvement in response length or overall quality.
2. **Generalization of the video captioning.** Although they conduct experiments on MSR-VTT for video captioning, it will be interesting to extend SMILE to dense video captioning (e.g. Activity-Net).

QUESTIONS:
As shown in Weakness.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10