Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation
Authors: Zihao Yue
Anwen Hu
Liang Zhang
Qin Jin
Venue: NeurIPS 2023 poster (2023)
Keywords: Image Captioning
Learning Objective
Natural Language Processing
TL;DR: We introduce Semipermeable MaxImum Likelihood Estimation (SMILE) for image captioning optimization, which allows richness optimization while blocking conciseness optimization, thus encouraging the model to generate longer captions with more details.

Abstract: Image captioning aims to describe visual content in natural language. As 'a picture is worth a thousand words', there could be various correct descriptions for an image. However, with maximum likelihood estimation as the training objective, the captioning model is penalized whenever its prediction mismatches with the label. For instance, when the model predicts a word expressing richer semantics than the label, it will be penalized and optimized to prefer more concise expressions, referred to as *conciseness optimization*. In contrast, predictions that are more concise than labels lead to *richness optimization*. Such conflicting optimization directions could eventually result in the model generating general descriptions. In this work, we introduce Semipermeable MaxImum Likelihood Estimation (SMILE), which allows richness optimization while blocking conciseness optimization, thus encouraging the model to generate longer captions with more details. Extensive experiments on two mainstream image captioning datasets MSCOCO and Flickr30K demonstrate that SMILE significantly enhances the descriptiveness of generated captions. We further provide in-depth investigations to facilitate a better understanding of how SMILE works.

Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation
Zihao Yue, Anwen Hu, Liang Zhang, Qin Jin"
Renmin University of China
(yzihao, anwenhu, zhangliang00, qjin]@ruc. edu. cn https:://gihhue com http:/ub..coooyueui///SSS/E
Human
Human
woman holding birthday with lit candles.
with light hanging from
MLE
MLE
woman holding cake with candles.
traffic light hanging from the side of building.
SMILE [ours]
SMILE (ours)
pretty young lady that has some kind of white candles
clove-up image of two red stoplights hanging from electrical cable system outside large brick church building, during the early morning
frosted birthday with
top of it, surrounded by several other people distance.
onwardly
Human MLE
Human surfing high strong MLE
floating top large body
riding
surfboard.
SMILE [ours]
SMILE (ours)
a lone fishing vessel that appears to be flosting crystall water near an Island the far distance.
young adult male leans forward as he stands stop extremely red surfboard in the midst crashing waves.
peacefully
striped
Figure 1: Descriptive captions generated by our SMIL.E-ottimieed captioning model, compared to human annotations and descriptions generated by the MI.E-optimized captioning model.
Abstract
Image captioning aims to describe visual content in natural language. As 'a picture is worth thousand words', there could be various correct descriptions for image. However, with maximum likelihood estimation as the training objective, the cap- tioning model is penalized whenever its prediction mismatches with the label. For instance, when the model predicts a word expressing richer semantics than the label, it will be penalized and optimized to prefer more concise expressions, referred as conciseness optimization. In contrast, predictions that are more concise than labels lead richness optimization. Such conflicting optimization directions could eventually result in the model generating general descriptions. In this work, we in- troduce Semipermeable MaxImum Likelihood Estimation (SMILE), which allows richness optimization while blocking conciseness optimization, thus encouraging the model to generate longer captions with more details. Extensive experiments on two mainstream image captioning datasets MSCOCO and Flickr30K demonstrate that SMILE significantly enhances the descriptiveness generated captions. We further provide in-depth investigations to facilitate a better understanding of how SMILE works.
"Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

expased context
label model prediction
GT:
(pretty) holding cake.
Richness Optimization
permeable
Model: woman lless rich than label)
Richness
Richness Optimization
Optimization
GT: a pretty woman holding (cake)
Condiseness
Conciseness Optimization
Optimization
impermeable
Model: white (less concise than label)
SMILE
> Conciseness Optimization
(b)
Figure 2: (a) Examples of richness optimization and conciseness optimization. When the label is "pretty" while the model predicts "woman" with less rich semantics, it leads to richness optimization; when the label is "cake" " while the model predicts "white", conciseness optimization occurs. (b) Different from MLE, SMILE presents permeaabbtbbl that accepts richness optimization while blocking conciseness optimization. Best viewed in color.
Introduction
The task of generating textual descriptions for given image, commonly referred to image captioning [4 [4], has long been plagued by the issuc overly-generic outputs [48] 49). That is,
models tend to generate similar descriptions for distinct images using simple concepts, but lack details (as shown in Fig. D. Such generated captions cannot meet various needs, e.g., eraingr online image search and recommendation, automatic data annotation for training vision-language models [4] 33), and poses a long-standing challenge in the field of image captioning. Considerable efforts have been devoted to generating more descriptive image captions, including constructing paragraph
caption datasets with more details for training models [19]. designing additional rewards [26] 28 29]
and prompting 27 34
lisionnlanguage pre-training (VLP) models [59] 60) to enrich captions (55]
While these efforts yield some promising improvements, they seldom try addressing this problem by
revisiting the core optimization objective.
Similar to text generation tasks such as machine translation 40), image captioning models are trained to predict the next word given context (the visual content and preceding words). The training is supervised by the maximum likelihood estimation (MLE) objective, which evaluates the model's predictive distribution over the vocabulary, and penalizes the model when it fails predict the current label. However, unlike machine translation where the target sequence is relatively certain, describing an image be quite diverse, and limited human annotations may cover all accurate captions. Therefore, the strict supervision with MLE is not perfectly suitable for image captioning optimization. As shown in Fig. 2 when the model predicts word (e.g.. "woman") that more 'concise' than the ground truth label (e.g.. "pretty"), MLE computes loss due to the mismatch and encourages the model to assign higher probability to "pretty". From the model optimization perspective, this optimizes the model to possess the more descriptive captioning ability, a process referred to as richness optimization in this work. In contrast, the model predicts a word (e.g.. 'white") expressing more details than the label (e.g.. "cake"), MLE also penalizes the model and optimizes it towards a more concise captioning behavior (conciseness optimization). Since ground truth captions in commonly used datasets (e.g., MSCOCO (24)) are mostly simple (around 10 words), conciseness optimization is not rare during training, especially for finetuning VLP models [23 60 59
But for image captioning, descriptions that contain more details rather than just generic concepts are usually preferred so that they can be used for accurately retrieving images or conveying more information for the visually impaired, etc. Therefore, conciseness optimization should be suppressed much as possible, and richness optimization should be encouraged to incentivize the model to generate more descriptive captions.
In this work, achieve this goal, we propose simple but effective training objective for descriptive image captioning, namely Semipermeable MaxImum Likelihood Estimation (SMILE). Unlike MLE, which evaluates the predictive probability distribution over the entire vocabulary, SMILE considers the probability over vocabulary subset with limited words, i.e., only containing words in the ground truth caption. Under typical conditions (with MIEE,, when a 'richer' word that conveys additional details gets high confidence score (c.g., "white" in Fig. 2 and such details generally are not

---

included in the ground truth caption), it challenges the label word (e.g., "cake") in the competition of probability allocation. This competition consequently leads decrease in the probability assigned to "cake", thereby imposing penalty on the model and thus resulting in conciseness optimization. However, with SMILE, probability allocation is limited to a subset of words that excludes "white". Regardless of the confidence score given to "white", . it does not diminish the probability assigned to the label "cake", thus avoiding a loss increase. In this way, we suppress the conciseness optimization that would have been imposed with MLE. In contrast, when the label word expresses details (e.g.. "pretty" in Fig. 2) while the model prefers a more 'concise" word "woman", is highly likely that the more 'concise' word is part of the ground truth caption and therefore, included in the subset. Eventually, even if we only focus on the subset, the word still takes away the probability from the label word "pretty". resulting in model penalization. In this way, richness optimization is maintained. Our objective allows richness optimization but blocks conciseness optimization, similar to semipermeable membrane. We, therefore, name this objective Semipermeable MLE (SMILE).
To verify the effectiveness of our proposed SMILE, we conduct extensive experiments on the image captioning task. The results demonstrate that SMILE can effectively optimize the model generate significantly longer and more descriptive captions. Besides, we carry out abundant experiments to analyze how SMILE works, such as ablations subset selection and descriptiveness origin, etc. Finally, we verify the generalization ability of SMILE on the video captioning task and discuss its limitation when facing other text generation tasks.
2 Method
2.1 Maximum Likelihood Estimation (MLE)
We first introduce the standard MLE for captioning optimization. As with many text generation tasks [40 [. the model is trained to maximize the likelihood of the label when predicting the current word w in the sequence given previous words w. and visual content The token-level loss function of MLE is defined as:
[MLE = Σ yı log pV wx vi 0,
(1)
where the summation iterates over all words in the vocabulary V, yj is the j-th element of the one-hot label vector, and pv is the predictive probability distribution over V.
2.2 Semipermeable MaxImum Likelihood Estimation (SMILE)
Given target sequence D [wo W22 ... wN), we form subset VD that only includes the unique words occurred in D, defined as VD = {w; wy E D}. Then, the current word w prediction is carried out within VD, with the loss function of SMILE defined as:
[vl
LSMILE Σ
yj log pVo (w|w v,
Here, PVD is the predictive distribution over the subset Vp. Specifically, for the j-th word in Vp. the probability P3 is calculated by the relative significance of the confidence score of the word compared to all other words in VD:
exp(z,)
p = softmax((z)
ΣKEV, exp(2*)
The probability assigned the label (the j- word) determines the prediction loss. When a word
w expressing additional details beyond the ground truth caption gets high confidence score (e.g.. in
Fig. 2 the label is " "cake", =+ "whit", and Vn = ("a". "pretty", "woman", "holding", "cake") } w VD). w+ does not contribute to the denominator in Eq. (3) since it is not included in Vo. and thus does not induce a decrease of the probability assigned to the label word pj-, which would have otherwise increased the prediction loss. In other words, when the model tends to predict words
3

---

like w*, these words do not participate in the probability allocation and do not take away any of the probability to be assigned the label word, thus avoiding model penalization which leads to conciseness optimization. In contrast, when the model predicts more 'concise' word w- (e.g.. the label is "pretty" and w "woman"), such word w usually is included in the ground truth caption, i.e., =T E VDD When allocating probability over VD. w also participates and takes away some of the probabilities. Therefore, the model penalization is not avoided, and the induced richness optimization is maintained. Blocking conciseness optimization while allowing richness optimization is what SMILE is all about.
SMILE is for further training. Since SMILE only considers the relative probability distribution within a subset, it does not penalize the model to generate detailed words beyond the subset, thus suppressing conciseness optimization. However, it makes sense only if the predicted detail correct. Therefore, SMILE is applied to further train model that has already been optimized with MLE, which ensures its fundamental captioning capability, Please note that SMILE only adopted in the training phase. During inference, the model predicts across the entire vocabulary to determine the current generation.
Initial context restriction It is well known that image captioning models are easy to suffer from exposure bias [2] [0] [6]] When training in a seacher-forcing manner, the model is exposed to the ground truth context; however, during inference, each generation step is conditioned on the previous predictions. This causes gap between training and inference. As SMILE does not require the model prediction to be consistent with the label as MLE does, the prediction is more prone to deviate from the label. Specifically, generating from the first token of the sequence, prediction that inconsistent with the label could lead to a completely unfamiliar context, exacerbating the exposure bias and affecting the autoregressive generation. Fortunately, we find that this issue can be effectively mitigated through some simple strategies, by ensuring that the initial context is correct, i.e., consistent with the label, which we name as initial context restriction and further discuss in Section 3.3
3 Experiments
3.1 Experimental Setup
Basic model and baselines Our proposed SMILE is architecture-agnosiic and can be applied to any visual captioning model compatible with MLE optimization. As a representative, we validate SMILE on the base version of BLIP 233, one of the state-of-the-un vision and language models pre-trained with 129M images and paired captions. We first fine-tune BLIP downstream datasets using MLE as the basic model, and then further optimize it with SMILE. For baselines, we compare our method with the latest descriptive image captioning solution, CapEnrich [55], and another two recent models NliCap [39 and GdisCap [49] We also provide the performance of ground truth captions for reference.
Dataset We evaluate our method on the two most commonly used image captioning benchmarks, MSCOCO (24 and Flickr30K [57]. MSCOCO contains about 120K images, and we adopt the commonly used Karpathy splitting [18] with 5,000 images each for the validation and test sets. Flickr30K contains about 31 K images, with 1,000 images each for the test and validation sets. For both datasets, each image has five human-annotated captions.
Evaluation We evaluate the models from three aspects: descriptiveness, accuracy, and fluency. Descriptiveness refers to how detailed the caption describes the image. Following previous works (55). we evaluate descriptiveness by the performance of CLIP self-retrieval, which employs the CLIP model 33 retriever recall the image with its caption from a candidates pool. This is based on the fact that captions with more details can better distinguish different images. The candidates pool is the hard retrieval pool constructed in the CapEnrich work (55), which additionally includes more similar images beyond the test set, placing a higher requirement on the descriptiveness of captions. We also report the average length of the generated captions and the lexical diversity (unique word count of all captions). Accuracy refers to the relevance of the generated caption to the image visual content, which we measure automatically by CLIPScore [3]. calculates the semantic similarity between the image and the generated caption using pre-trained CLIP model. For fluency, we report

---

Table Image captioning performance of different methods on MSCOCO and Flickr30K, For human performance, we randomly select one annotation for each image for comparison with the others.
Caption Length
Lexical Diveristy
Self-Retrieval
Dataset
Method
CLIPScore
PPL
R@
R@5
NliCap
9.5
0.8
2.9
9.3
75.5
67.8
GdisCap
9.5
1.0
3.5
10.8
75.8
100.2
CapEnrich
13.3
1.5
9.4
22.6
79.2
63.1
MSCOCO
BLIP
10.0
1.4
6.7
16.6
77.2
95.8
BLIP-CSMILE
22.3
4.5
10.0
24.5
75.0
95.6
Human
10.4
4.1
7.6
20.0
77.6
129.1
CapEnrich
15.2
1.0
29.2
54.9
81.5
67.1
BLIP
11.6
0.8
25.4
46.4
78.8
65.6
Flickr30K
BLIP-CsMIEE
23.2
2.3
31.2
53.0
78.2
98.0
Human
12.3
2.0
26.2
48.3
79.8
121.2
the language modeling perplexity (PPL) of the captions with GPT-2 K22 However, these automatic metrics have their limitations. For example, CLIP measures the overall semantics of the caption and struggles to focus on too many details. In addition, the evaluation models also suffer from the bias of the training data and thus may have difficulty in handling longer more complex sentences. Therefore, we also conduct human evaluations, which will be discussed later.
3.2 Main Results
Versus basic model According to Table compared with the basic BLIP model, SMILE signifi- cantly increases the average length the captions (more than doubled on both datasets). SMILE leads to more detailed descriptions well, greatly enhancing the model's self-retrieval performance. Although SMILE causes a slight decrease in the CLIPScore (-2.2 and -0.6 on MSCOCO and Flickr respectively), consider this to be reasonable phenomenon, as 'talks much much'. Besides, maintaining a comparable level of perplexity indicates that SMILE can incentivize longer and more detailed captions without compromising fluency.
Versus baselines As shown in Table I BLIP significantly outperforms all baselines on lexical diversity with SMILE optimization. Compared traditional captioning methods that tend generate dull texts with common words overused, SMILE effectively overcomes the lexical diversity bottleneck and outperforms humans. Our approach also achieves the best performance on self-retrieval. It is worth noting that the most competitive baseline CapEnrich requires automatically constructed data in specific format, which integrates details from multiple manual captions of each image for training, along with carefully handcrafted or learnable prompts. In contrast, our approach is refreshingly simple, has fewer restrictions, and importantly, exhibits no incompatibility with other methods. Therefore, it can serve as effective supplement or alternative the currently employed solutions. Since our vanilla BLIP model with SMILE optimization already surpasses other methods that may require complicated design, we see no strong need for further attempts to combine SMILE with them in this work.
Human evaluation We randomly sample 100 images from the MSCOCO test set for human evaluation. For each image, we collect candidate captions, including three captions generated by CapEnrich, basic model BLIP, and SMILE-optimized BLIP, respectively, and one human-annotuul annotated caption. All candidates are randomly shuffled. For each candidate caption, 5 human annotators are asked to independently rate scale of to 5 (a higher score indicating better quality) from three aspects, namely descriptiveness, accuracy, and fluency. The 'descriptiveness' refers to the richness of the correct detail relevant to the image. Table 2 shows the average score of each model the three aspects. SMILE substantially improves the descriptiveness of the captions, with a rate close 5 (excellent), outperforming all other candidates including numan-writted captions by a large margin. A score beyond 4 for both accuracy and fluency demonstrates the good quality of captions

---

Semantic
Linguistic
Method
Avg Captoon Longth
20
Descrip.
Acc.
F1
Flu.
IMLE: Rig (MLE)
CapEnrich
3.89
4.39
4.12
4.36
15-
BLIP
3.41
4.57
3.91
4.91
Rig2 (SMILE)
BLIP-LSMILE
4.67
4.05
4.34
4.75
Human
3.53
4.53
3.97
4.87
Epoch
Table 2: Human evaluation results of the captions for the randomly sampled 100 images. (score 1 to 5: terrible, poor,fair, good, and excellent)
Figure 3: Trend of the average length and self- retrieval performance on the MSCOCO test set during training.
Table 3: Performance of different basic and further SMILE-optimized models. (F.S.: model trained From Scratch, i.e., BLIP with a randomly initialized text decoder trained on MSCOCO without pre-training; P7: pre-trained BLIP without fine-tuning; PT+ our default basic BLIP model with both pre-training and fine-tuning.)
Further Training
Caption Length
Lexical Diveristy
Self-Retrieval
Row
Basic Model
CLIPScore
PPL
R@1
R@5
1
9.6
0.6
1.9
6.3
73.9
56.2
F.S.
2
LSMILE
15.4
0.7
0.7
2.5
63.7
86.7
3
5.8
1.0
2.7
8.6
74.6
573.4
PT
LSMILE
22.5
4.9
8.5
20.1
73.5
113.7
5
10.0
1.4
6.7
16.6
77.2
95.8
PT+FT
LMLE
1.4
77.2
67.6
7
ESMILE
22.3
4.5
10.0
24.5
75.0
95.6
by our SMILE-optimized captioning model, although it faces higher risk of 'talks much emrs much'. As descriptiveness emphasizes the recall of visual content while accuracy emphasizes description precision, additionally calculate Fl value of these two scores as the overall semantic score. As shown Table 2 SMILE performs best when taking into account both aspects.
3.3 Ablation Study
Training epochs Fig. presents the impact of training epochs caption length and self-retrieval performance with SMILE optimization. It shows that SMILE takes only few training epochs to achieve convergence and does not introduce too much extra training cost. For all of models optimized with SMILE, we choose the checkpoints according to the best self-retrieval performance on the validation set, which always
within epochs.
Basic model As the basic model used for SMILE optimization, BLIP is first pre-trained on a large- scale image-text dataset and then fine-tuned by MLE downstream datasets. Both the pre-training (PT) and fine-tuning (FT) equip the model with fundamental image captioning capabilities. To investigate how such fundamental captioning ability influences SMILE's effectiveness, compare the performance of different basic models and their further SMII.E-opiimized counterparts on MSCOCO. As shown in Table 3 for basic models, both pre-training and fine-tuning improve the model's performance in terms of descriptiveness and accuracy (rows 1, 3, and 5). After subsequent SMILE optimization, each basic model can generate much longer descriptions (rows 2, and 7). Regarding how much SMILE improves descriptiveness, it is notable that when the fundamental captioning ability is poor, SMILE brings no improvement and can further decrease the performance (row 2 vs 1); but once the basic model is able to perform acceptable captioning, SMILE can contributes a (rows
and 7 vs 5). This implies that SMILE requires fundamental captioning ability of the basic model in order to further correctly generate more details. In addition, as SMILE optimization introduces extra training steps, we further train the default basic model with MIK and validate that simply adding training steps does not bring improvement (row 5).

---

Table 4: Performance of SMILE-optimized models with different initial context restriction strategies.
Initial Context Restriction
Caption Length
Lexical Diveristy
Self-Retrieval
CLIPScore
PPL
R@1
R@5
17.0
3.0
6.8
16.2
71.9
82.0
shifting default
21.3
4.0
9.6
22.4
75.0
85,2
22.3
4.5
10.0
24.5
75.0
95.6
Table 5: Performance comparison with different mixing ratios of MLE and SMILE on MSCOCO.
Caption Length
Lexical Diveristy
Self-Retrieval
A
CLIPScore
PPL
R @
R@5
1
10.0
1.4
6.7
16.6
77.2
95.8
0.5
10.8
17.9
77.0
67.0
0.1
12.6
1.9
7.6
18.2
76.1
69.3
0.05
14.7
2.3
8.7
20.9
76.4
74.1
0.01
19.8
3.6
10.9
25.1
76.2
79.4
22.3
10.0
24.5
75.0
95.6
Initial context restriction As aforementioned in Section 2.2 to alleviate exposure bias for SMILE optimization. We design two implementations, First-token MLE and First-token Shifting, achieve such restriction:
propose initial context restriction
1. First-token MLE adopts ML_E loss for the first token of the target sequence, and SMILE for the remaining tokens.
2. First-token Shifting shifts the label of the first token of the target sequence to its alternative. For example, in the vocabulary of the BERT model (7]. the word "a" can be replaced with its subword form "##a", which rarely appears to be the first token in the training corpus. By doing so, replace the familiar word with an unfamiliar one, making it difficult for the model to predict correctly (even within subset). This often results in model penalization, making the prediction of the first token more likely to be consistent with the ground truth.
Since the shifting approach is designed with certain requirements on the model vocabulary, making potentially difficult be applied to all models, we use the First-token MLE by default address the exposure bias issue. To demonstrate the effectiveness of this strategy. in Table we show that without initial context restriction, SMILE optimization default First-token MLE implementation or the First-token Shifting implementation helps alleviate exposure bias and the former better.
achieve promising results. Either the
Mixing fALK with LSMILE Since SMILE improves descriptiveness the expense of some accuracy, it could pose risk in scenarios that demand higher accuracy. To strike balance, carry out experiments where the overall learning objective combines both LMLE and LSMILE, defined as:
Loverall = X CMLE (1 - A) LSMILE, E [0, 1].
Table illustrates how the balance between MLE and SMILE impacts the performance in terms of descriptiveness and accuracy. As LSMILE becomes more dominant, the model tends to generate longer outputs with higher lexical diversity and descriptiveness; however, accuracy correspondingly decreases. This demonstrates that a compromise between descriptiveness and accuracy can be achieved by simply combining LSMILE with LVILE. Notably, we find that when LMILE incorporated very low ratio (0.01), the generated captions achieve the best performance in self-retrieval. This suggests that while SMILE facilitates more details, ensuring accuracy is also important for captions to better describe and distinguish images.

---

Avg. Caption Length
Avg. Caption Length
MLEE Random Reverse
MILE Eimplest-COCO
4-
SNILE (Simplettcoco)
15
(Simlee-Cooo
SMILE (Simple-CoC0)
10
2
Epoch
Epoch
(a)
(b)
Figure 4: Average caption length during training with (a) different subsetting strategies and (b) on
different training corpus.
3.4 Further Analysis
In this section, we conduct experiments to further analyze how SMILE works. As we hypothesized, SMILE implements 'semi-permeability" to retain only richness optimization, leading the model to generate more details. Therefore, we first investigate whether such an effect is attributed to our subset selection. Then, we affirm the pemi--ermeabiility by flipping it to observe an opposite property of SMILE. Besides, since BLIP-CSMILE surpasses human annotation in descriptiveness Table D and T we analyze where it gains the capability to generate details. Furthermore, we provide a visualization to illustrate how SMILE exerts its effect from the token-level loss. Finally, we test the generalization ability of SMILE on video captioning.
shown in
Subset selection is key in SMILE. We argue that the effectiveness of SMILE is attributed to our subset selection strategy that selects the target sequence words. To verify this, we ablate the SMILE subsetting strategy by replacing it with Random subsetting, that is, for given target sequence, we randomly choose 10 words (the average caption length of MSCOCO) from the entire vocabulary to form the subset, ensuring the subset size is relatively consistent with that of SMILE. Fig. 4 (a) illustrates the impact of SMILE and Random subsetting on the generation length during training. Random subsetting maintains the describing habits of the basic model, neither lengthening nor shortening the generation length. This is due to that the model's preferred predictions among the entire vocabulary, whether more or less 'rich' than the label, are less likely to be included in the randomly-chosen subset. This leads to a mechanism akin to 'bidirectional impermeability" where both conciseness and richness optimization weakened. This also demonstrates that it is the subset selection of SMILE that contributes to the increasing length model generation.
Reverse subsetting flips the 'semi peemi--emmaaiiltty... Since SMILE achieves 'semi semi-permeabilin which allows richness optimization while blocks conciseness optimization, natural assume that such 'semi- "semi--prmme cemm--ermeabiltyy could be flipped with reversed subset selection. Thus, we conduct experiments by proposing Reverse subsetting, that is, given target sequence D [w,, W2, w. when predicting each word, we select the complement of the target sequence word set in the entire vocabulary, plus the current label, as the subset. Hence, for the prediction of the i-th word, the corresponding subset is Vs. {wlw & VD} U {w,) As shown in Fig. (a), with subset complementary to that of SMILE, Reverse subsetting exhibits the opposite property of SMILE, leading the model to generate increasingly shorter captions. With these observations, confirm that there is semmi-errccaiility' which is achieved by our subsetting strategy and can be inverted.
Models 'absorb' descriptiveness from the corpus. Although SMILE achieves unidirectional richness optimization, one may still wonder where models with SMILE optimization learn the descriptive captioning ability to surpass human annotators. We suspect that models could learn the usage of details from the whole corpus. To verify this hypothesis, we construct two datasets based on MSCOCO, namely Simplest-COCO, which contains no details, and Simpler-COCO, with only few details. Concretely, for Simplest-COCE we extract the subject from the caption and prefix with definite article based grammatical rules, such "a girl". For Simpler-COCK we extract the vubjeet-verb-object constituent of the caption as a new caption with few details, such as man working" All the mentioned natural language processing (e.g., parsing and pfrt-of-spech tagging)

---

45
tour
FRC
tan
3IM
SOTTO
pEople
DUS
White
[S2P]
Dus
whil
there
big
orange ywlce
coach
di sing
Its
1ightx
WRIM
and
bus
whib
some
Probaci ity
Top 1ee-pobbablityy
the
whbbe
bus
smen 104M
on
street
on
bux
crange
orange
white
orange
prop
white
0.4
arge
red passenger
1s
Wires
and
orange
white
bus
and
some
orange
3nd
5.6
parkod
Sume
buidiiggo
(SEP)
orange
and
poook
Label
whibe
and
orange
bux
and
some
people
[SSP]
Ahile
arange
bux
and
some
people
[SKP]
1.0
2.42
0.14
0.32
4.31
2.46
3.30
3.
0.02
1.23
0.02
0.37
0.25
0.27
0.60
(a) MLE
(b) SMILE
Figure 5: Token-level model predictive distribution and penalization of (a) MLE and (b) SMILE. The model is less penalized if the label word is assigned a higher probability.
Table 6: Video captioning performance MSR-VTT.
Caption Length
Lexical Diveristy
EMScore
Self-Retrieval
Method
PPL
R@D
R@5
BLIP4video
8.1
1.1
26.4
28.1
27.2
19.0
38.7
150.1
BLIP4video- -SMILE
20.6
4.5
26.6
28.6
27.5
27.3
50.0
163.0
Human
9.2
3.8
26.6
28.2
27.4
23.0
42.8
561.8
is performed with SpaCy 16). The average caption length for Simplest-COCO and Simpler-COCO is 2.0 and 2.5, respectively. On these two datasets, first train the default basic model with MILE until convergence and further optimize it with SMILE. As shown in Fig. 4 (b), only MI_E optimization does not achieve longer descriptions than the training corpus both datasets. On the other hand, with SMILE optimization. the description length does not increase when training on Simplest-COCOO however, on Simpler-COCO, the model generates captions about twice as long as the ground truths. This suggests that richness optimization relies on the details in the whole training corpus, and SMILE helps models 'absorb' the capability to express more details.
Visualization of token-level penalization, MLE versus SMILE Taking a sample from the MSCOCO training set as an example, sigos the predictive distribution and loss value for each word during MLE and SMILE optimization, respectively. Note that the predictions in Fig. (a) and (b) are given by an identical model but the former distribution is calculated over the whole vocabulary, while the latter over the SMILE subset. When predicting the second word ("whi te" in the target sequence), the model prefers a more 'concise" word "bus" than the label "white". ". For MLE, "bus" takes away much of the probability, resulting in significant loss that causes richness optimization. Similar to MLE, SMILE also penalizes such prediction since both "white" and "bus" are included in the subset. When predicting the sixth word (the label word is "and"). the model prefers a more "detailed" word "parked" to additionally describe the state of the bus. Due to such mismatch, MLE still imposes heavy penalty with loss of 4.51; however, SMILE performs a more lenient optimization with a loss of 0.57. This is because the word "parked" is not contained in the subset, and by allocating probabilities over the subset, the label word "and" obtains the highest predictive probability. Therefore, SMILE circumvents the penalty induced by more 'detailed' prediction and blocks conciseness optimization.
SMILE generalizes well to video captioning. We conduct experiments on MSR-VTT 53 to
test the generalization ability of SMILE on the video captioning task. MSR-VTT consists of 10K videos and each video clip is annotated with 20 English sentences. The BLIP4video 58 moditied version of BLIP designed for video tasks, is chosen our basic model. As for evaluation, we apply the reference- in terms of precision (P), recall (R), and the F1 score (F), to measure the accuracy and descriptiveness of the captions. We also provide the self-retrieval performance, with the Fl score of EMScore for retrieving, and the test set as the candidates pool. Table that SMILE also enables the model to generate longer descriptions on video captioning, and brings an overall improvement in both accuracy and descriptiveness.
metric EMScore [8 It evaluates embedding matching between video and caption

---

4 Related Works
In this work, we focus on generating descriptive image captions from the perspective of the language modeling objective. Existing works also propose different strategies to increase the descriptiveness of captions. For example, Gu et al. [[]] and Liu et al. [26] design two-stage decoding strategies with caption generation and refinement. Liu et al. [26] and Shi et al. (39 integrate descriptive details
from multiple-sentence annotations with additional rewards or natural language inference (NLI)
relations. In addition to descriptiveness, existing works also employ contrastive learning [3 6 29
and self-retrieval strategy 228 451 to increase the discriminability of captions. Wang et al. 48 re-
weight the ground truth captions to emphasize more discriminative ones. Wang al. [49] highlights
the uniqueness of each image within groups with similar images. More recently, CapEnrich [55]
stimulates VLP models (59] 60] to express more details with prompting strategies [27 [4] It
achieves state-of-the-art performance on the descriptiveness and discriminability of image captioning. However, CapEnrich requires training on automatically constructed data with additional utterances, which integrate details from multiple captions annotated for each image. This leads to unconventional text structure and thus affects the fluency of the generated caption. In contrast to the above-ment mentioned methods, SMILE does not rely on either multiple annotated captions or the construction of additional data. To the best of our knowledge, SMILE is the first approach enhance models' descriptive captioning capability by focusing on the language modeling objective itself.
Besides, text generation models trained with MLE tend to generate dull texts with high- gee-ffreuency words 52 37 Attempts for fixing such degeneration issue including variant decoding methods [22] 46 20
44
and learning algorithms [22 9 5T). In this work, SMILE-optinize, models output
with much greater lexical diversity, also demonstrating the capacity of mitigation of such problems besides improving descriptiveness.
5
Conclusion
For descriptive image captioning, we revisit the commonly used maximum likelihood estimation (MIE) training objective. We argue that MLE is not perfectly suitable for image captioning tasks because of two conflicting optimizations: conciseness optimization and richness optimization. To steer the model toward generating more descriptive captions, we propose to mitigate conciseness optimization and maintain richness optimization by revising the objective with a simple but effective vocabulary subset selection strategy, namely Semipermeable MaxImum Likelihood Estimation (SMILE). Extensive experiments validate that SMILE helps models to generate much longer and more descriptive captions without complex architecture design or extra data annotation.
6 Limitations
Technically, is possible to apply SMILE to any autoregressive text generation task with an MLE objective. To investigate this, conduct preliminary exploration on the Supervised Fine-tuning 31
(LLMs) - fine-tuning LLMs to follow human instructions, such writing and answering questions. Unfortunately, comparing SMILE with MLE for SFT, we don't 't observe significant improvement in response length or overall quality (see Appendix B). One potential factor that the target response in the training corpus is often very long (for instance, paragraph exceeding 100 words). This causes that the constructed subsets are significantly larger than the ones the image captioning task, and differ less from the entire vocabulary set. Therefore, SMILE may struggle to be directly applied to other text generation tasks. Further exploration of this issue will be reserved for future work.
(SFT) of Large Language Models 44 43
Acknowledgements
This work was partially supported by the Beijing Natural Science Foundation (1.233008), the National Natural Science Foundation of China (No. 62072462) and the National Key R&D Program of China (No.2020AAA0108660),,
10

---

References
[1] Mehdi Allahyari, Seyedamin Pouriyeh, Mehdi Assefi, Saeid Safaci, Elizabeth D Trippe, Juan B Gutierrez, and Krys Kochut. Text summarization techniques: brief survey, ArXiv preprint, abs/1707.02268, 2017. URL IRLttts:///axxv htt.:/orgg/bb////////// 62268
[2] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Process- ing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1171-1179, 2015. htLphtts:://p// httpttpps//prroeedigggs neurips. wttppppe.ccpppprrre/co 20 /7tpa/////9919999977777777777770 htt.:///ttt..a-abbtatatttttttiia
[3] Fuhai Chen, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, and Jinsong Su. Groupcap: Group-hased image captioning with structured relevance and diversity constraints. 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 1345-1353. IEEE Computer Society, 2018. doi: thecvf comm/content_cypr_220 htm1/Chen_ Sirieen_GrouCppp__Gr hrtp:ccapp_oooo-----peaa Baapeeedd__aaee__ee www.e_aeeeee_eeeeeeeeeee__ _ap0111ppppppp___ html
10.1109/CVPR.22088.014444
[4] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Proceedings of the European conference on computer vision (ECCV), pages 104-120. Springer, 2020.
[5] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL
https://lmsys. http://ssssssrr////////22/33 -0- vicuna/
[6] Bo Dai and Dahua Lin. Contrastive learning for image captioning. Advances in Neural Information Processing Systems 30: Annual Conference on Neural In- formation Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA.
pages 898-907, 2017. URL https: /pppceddprssssdingss nww..ne..ii..... cc/paper/2017/hash
1199220880a88111886666888 396bee Abstract.html http
[7] Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-41 Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N1991444 org/N19-1
URL
[8] Shizhe Diao, Rui Pan, Hanze Dong, KaShun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. https://optimalscalee gitb. 1t/LM//LLM////// 2023.
[9] Adji B. Dieng, Kyunghyun Cho, David M. Blei, and Yann LeCun. Learning with reflective likelihoods, 2019. URL /tttps://openreiieeee http:ppnwwvieeeet.ooormaiaiaaaaa net/Forumiid--11122jF7
[10] Anirudh Goyal, Alex Lamb, Ying Zhang, Saizheng Zhang, Aaron C. Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, neur ips. httpppappr221///aaaa pages 4601-4604 2016. URL Awstssstb.ttttttt..e html
[11] Jiuxiang Gu, Jianfei Cai, Gang Wang, and Tsuhan Chen. Stack-captioning: Coarse-te corrse-to-fine so-fine learning for image captioning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (1AAI-18), and the 8th AAAI Symposium (EA New Orleans, Louisiana, USA, February 2-7, 2018, pages 6837-6834848 6844, 2018. URI. https: www. aaai. www.orgg//scco. htt///idexx.../
Educational Advunces in A Artificial Intelligence
http:///s..nne..ppp////////////a http:ppp///..........// 8/ paper waaper/viww///eee 16465

---

[12] Tianxing He and James Glass, Negative training for neural dialogue response generation. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 2044-2058, Online, 2020. Association for Computational Linguistics. doi: 10. 00.18633/1/2020. main. 185. URI. htttps://acllnntoooogy.oom
wtw.:008...1.1... acl-main. 185.
[13] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7528, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguisties. doi: 10. v1/2021 1e000221emmlpmmii.55500 595. URL http:ttst///aaaanooolgge. orgg 222222211122 www.e--nnniii main. 595
[14]
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1638-1649, Mclbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/ P18-1 1152. URL
[15] Ari Holtzman, Jan Buys. Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICL.R 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htt.ttts////opeeeiiiee net/forum?idrryg@@yrrn
[16] Matthew Honnibal and Ines Montani. spaCy 2:
Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
[17] Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[18] Andrej Karpathy and Fei-Fei Li. Deep descriptions. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, Jume 7-12, 2015, pages 3128-3 3137. IEEE Computer Society, 2015. doi:
alignments for generating image
10.1 00.1099/CVR..2015.72999
URL WtL/hhtps://ddot org/ 10. 1111/1199/2/01111 2015. 7298932
[19] Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hicrarchical approach for generating descriptive image paragraphs. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA July 21-26, 2017, pages 3337-3345, URL / //oi///ooiioo org/ 10. 109/CVPR. 2017. 356
2017. doi: N0.1109/CVR..2017..33
[20] Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. Importance of search strategy in neural dialogue modelling. ArXiv preprint, abs/1811.009011 2018. URL
http://aaxxiv..g//bbb//// 00907
[21] Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hchong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision- sainuala learning by cross-modal skip-connections. arXiv preprins Xii::2205.1200) 2022.
[22] Jiwei Li, Will Monroe, and Dan Jurafsky. simple, fast diverse decoding algorithm for neural generation. ArXiv preprint, abs/1611 1.85662, 2016. URL https::/aarxvvorrgaaab//// 08562
[23] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages URL httpa///pppoedennnggg.g.errrrr// wttpppeees/e///ee2ee2pp htmlll
PMLR, 2022.
[24] Tsung- Yi Lin, Michael Maire. Serge Belongic. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[25] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arX1v:2304.08485, 2023.
12

---

[26] Lixin Liu, Jiajun Tang, Xiaojun Wan, and Zongming Guo. Generating diverse and descriptive image captions using visual paraphrases. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 4239 4248, 2019. doi: 10. 1A11109C.222210000055. 2019.00433
URI. httttt::/ddo...rgg///// 11.00...000 1109//CCV
[27] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys, 55(9):1-35, 2023.
[28] Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, and Xiaogang Wang. Show, tell and discriminate: Image captioning by self-retrieval with partially laheled data. In Proceedings of the European conference on computer vision (ECCV), pages 338-354, 2018.
[29] Ruotian Luo, Brian I.. Price, Scott Cohen, and Gregory Shakhnarovich. Discriminability objective for training descriptive captions. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22 2018, pages 6964-6974, 2018. doi: 10.1 109.1//9.///00111 2018800020 URL / thecvf. cwm/contem cvpr_ 2tt_22018hhtm/mmm Bttp:/LLoooooiirrrieeee Discr http:///////ttttuuooooiiiiininnn
for
cVPR__cp__________ wap.rr__ppppp_____ html
[30] OpenAI. Gpt-4 technical report, 2023.
[31] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[32] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18))9, 2019.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. html
PMLR, 2021. URL httt:::/pprcceeiingg. mlr.
[34] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Mach. Learn. Res., 21:140:1 -140:67, 2020. URL http::/jml.. brgg/papers//21/28-074. html
[35] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 218555555,, 2020.
[36] Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
06732
[37] Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1702- 1723, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.11855//1/N119-1199 U http:/::////aaaaaooooo.o.ooo
[38] Yaya Shi, Xu Yang, Haiyang Xu, Chunfeng Yuan, Bing Li, Weiming Hu, and Zheng-Jun Zha. Emscore: Evaluating video captioning via coarse-grained and fine-grained embedding matching. In Proceedings of the IEFE//VV Conference on Computer Vision and Pattern Recognition, pages 17929-17933 2022.
13

---

[39] Zhan Shi, Hui Liu, and Xiaodan Zhu. Enhancing descriptive image captioning with natural language inference. In Proceedings of the 59th Annual Meeting of the Association for Computa- lional Linguistics and the 1th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 269-277, 2021.
[40] Felix Stahlberg. Neural machine translation: A review. Journal of Artificial Intelligence Research, 69:343-418, 2020.
[41] Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. ArXiv preprint, abs/2202.064 06417, 2022. URL
https://arxiv http://aaxiiioorrra///////////// http://abb/22..6644/0 06417
[42] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An inslrwction-follloinng llama model. https://githh ctt wawatsssaaassstaorddedoooo
2023.
[43] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science, 2022.
[44] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[45] Gilad Vered, Gal Oren, Yuval Atzmon, and Gal Chechik. Joint optimization for cooperative image captioning. In 2019 IEEE/CVF International Conference Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2. 2019, pages 8897-8906. IEEE, 2019. doi: 10.1109/CCC.2211......
URL httphhttt///ooiiio hrwpd/oo..go///oo 1109/11190CCCC 2019.00099.0090
[46] Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[47] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 31 3156-3164, IEEE Computer Society, 2015. doi: 10.1109CCVPR.2015..2235 7298935
URL
wr/..//..........1////......... 1009//VPR. 2015
[48] Jiuniu Wang, Wenjia Xu, Qingzhong Wang, and Antoni B Chan. Compare and reweight: Distinctive image captioning using similar images sets. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 370-386. Springer, 2020.
[49] Jiuniu Wang, Wenjia Xu, Qingzhong Wang, and Antoni Chan. Group-based distinctive image captioning with memory attention. In Proceedings of the 29th ACM International Conference on Multimedia, pages 5020-5028, 2021.
[50] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai. Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple saweence-to-sequene learning framework. In International Conference Machine Learning, pages 23318 2-3400. PMLR, 2022.
[51] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. TtLL/tttss///peeneeeee
[52] Jason Weston, Emily Dinan, and Alexander Miller. Retrieve and refine: Improved sequence generation models for dialogue. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Belgium, 2018. Association for Computational Linguisties. doi: 10. 16555/111118851133 URL httpps://cclanthooog.. .rg W11--/0511000/111111333
Oriented Conversational AI, pages 87-92, Brussels,
14

---

[53] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTI VTT: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages w0tgg//0..000 1109/ Computer Society, 2016. doi: 10.1109/CVPR.2016.55 URL https: h org/10 CVPR. NTR.20066.57 571.
IEFF
[54] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings the 32nd International Conference Machine Learning, ICML 2015, Lille, France, 6-11 1 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 337/xuc15.htm
Jog,. 2015. URL http://proceeddings.. mar.ssmml...em press/
[55] Linli Yao, Weijing Chen, and Qin Jin. Capenrich: Enriching caption semantics for web images via cross-modal pre- pre-trained knowledge. ArXiv preprint, abs/2211.0937 2022. URL
https://arxiv http://wxiv.oo.raaaaa abs// 2211.09371
[56] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
[57] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014. doi: 10.1162/
tacl 00166. URL htttss://aclanthooogg o01
[58] Zihao Yue, Yuqi Liu, Liang Zhang, Linli Yao, and Qin Jin. Rucaim3-tencen at trecvid 2022: Video to text description. In Proceedings of TRECVID 2022. NIST, USA, 2022. URL pdf
https://wwwrnlprr.niss. wisp::pppi.iii..oogg.....e...e.e gov. http://ii.......o.oopp/////////p http:///wtte//ttp//////////
[59] Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 25994-26009. PMLR, 2022. zeng22c.html html
mlr press/v162/ 6
[60] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision- language models. In IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2021, virtual, June 19.25, 2021, pages 5579-5588. Computer Vi-
sion Foundation IEEF, 2021. doi: 10. 10.1109/CVPR444437. 000.//446647.2221000505555 1005500 URL https://openaccess thecvf.
BiiinnggViiiiiiiii WinVVL_Revisitiii wtt.nteent/cypr221//tmmm////////
[61] Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4334 4343, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10. 10.18663/11/P11-14422 URL
[62] Deyao Zhu. Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-lanne language understanding with advanced large language models. arXiv preprint 10552, 2023.
15

---

A SMILE Implementation
We present pseudocode Algorithm to better understand SMILE, which can be implemented and applied in such straightforward manner. Further details including Reverse Subseting and Random Subseting can be found in our released code at htthttss:/ /agp::://gitubb wttp:hww...mooueeih/SSLLLL
Algorithm Pseudocode of SMILE in a PyTorch-like style.
# B: batch size
# N: sequence length
# V: vocabulary size
# logits: language model prediction logits (B x N V)
# labels: ground truth label at each position (B x N)
if objective 'MLE': prossEntropyyLoss(lggiti labels)
elif objective 'SMILE':
each sequence gets unique mask to mask words not in the sequence logits_mask = zeros(B, V) scatterer_) labels, True) # B
expand for every position in the sequence logits_mast = logits_mask. unsqueezequeeee11 (x). N, -1). clone( #BXNXV
perform first-token MLE (optional) oogits_mask[:: :] True # B
apply mask to the logits selected_logits logi ts. masked_ (lggiiilllooiitssssssaalllaaall float('-inf))) # N
loss Trossfntrroppoossceelleeeeeeells labels)
scatter_: scatter values matrix indices (https://pytorch.. http:///wtoorrhorroddsssssasaaae http://www........../////a/////e
Tensor. swatterr...ttt CrossEntropyLoss integrates softmax operation.
Table 7: Performance of LLaMA-13B fine-tuned using MLE and SMILE, respectively, on the Vicuna evaluation framework. Scores relative values re-scaled by taking the scores of reference answers as full marks (100). Eps.: Training epochs; Obj.: Learning objective; Len.: Average answer length; Lex.: Lexical diversity.
Eps.
Ohj.
Len.
Lex.
Gene.
Know.
Role.
Com.Sen.
Fermi
Count.
Writ.
Overall
LALE
94.8
2.4
82,3
88.2
82.9
85.6
77.4
85.3
83.1
83.5
5
ESMILE
95.8
2.4
92.4
86.2
85.3
90.4
79.6
85.4
85.8
86.4
LALE
89.2
2.3
92.9
88.9
85.8
86.9
78.3
84.4
84.7
86.0
LSMILE
96.1
3.4
97.3
89.0
82.4
87.8
79.2
87.4
75.6
85.5
LMLE
90.2
2.3
92.0
88.0
88.2
91.3
75.7
86.5
82.2
86.3
ESMILE
96.4
3.5
93.4
89.2
91.1
90.6
81.6
85.3
62.2
84.8
B SFT of LLMS
Supervised fine-tuning (SFT) aims to enhance the capacity of large language models (LLMs) to follow human instructions better (3]. This section reports our experimental findings comparing SFT tasks carried out with MLE versus SMILE. We choose the widely used LLaMA- 44 as the basic model. and fine-tune it on the Alpaca dataset (42 which contains 52,000 instructions and demonstrations generated by the text-davinci-003 engind To ensure fair comparisons, the pre- trained basic model is fine-tuned using both MLEE and SMILE,, maintaining identical settings for both objectives. Each experiment involves fine-tuning the baseline model using a Low-Rank Adaptation (LoRA) approach [17] for parameter efficiency, employing 4 RTX A6000 nodes with batch size of 8. The fine-tuning process is conducted using the LMFlow framework [8) For evaluation, we employ
Ihttps://platfoo wttp://w...commoo//////aa///////
16

---

Table Performance of different basic models before and after SMILE optimization.
Caption Length
Lexical Diversity
Self-Retrieval
Model
LSMILE
CLIPScore
PPL
R@
R@5
10.0
1.4
6.7
16.6
77.2
95.8
BLI huse
2
22.3
4.5
10.0
24.5
75.0
95.6
10.0
1.4
6.7
17.3
77.4
92.3
BLIP lage
24.0
4.9
9.2
22.4
74.7
100.0
10.1
1.2
5,9
15.6
76.7
64.6
OFhsse
16.5
4.0
6.9
16.2
71.6
109.7
9.8
1.2
6.4
16.0
77.1
68.6
OFae 50
18.8
5.9
9.3
21.9
73.6
125.9
9.7
1.3
5.7
15.5
76.6
59.6
mPLUG 2
12.3
2.8
3.9
11.0
71.0
92.3
9.9
1.3
5.5
14.4
76.9
60.6
VLnVL 60
25.2
5.6
5.4
13.8
71.9
113.9
the evaluation framework proposed in Vicuna [5]. It contains eight question categories, including generic (Gene.), knowledge (Know.), roleplay (Role.), common-sensc (Com.Sen.), Fermi problems (Fermi), counterfactual (Count.), coding/math tasks, and writing (Writ.), with ten questions for each category. The model's answer is rated by GPT-4 30 in terms of helpfulness, relevance, accuracy, and detail, by an overall score ranging from 1 to 10. To ensure consistent independent scoring, GPT-4 rates the candidate answer by comparing with reference answer (from GPT--35-turbd Given that GPT-4 not an efficient judge for coding/math tasks [5]. and LLaMA- 13B also struggles with these tasks, we exclude coding/math tasks in the evaluation.
Table 7 shows the performance the models after fine-tuned for 5, 10, and 20 epochs using both MLE and SMILE. Compared to MLE, SMILE enables the model to generate slightly longer responses with enhanced lexical diversity. In terms of answer quality, the SMILE-optimizer model demonstrates advantages across almost all task types the early stages of training (after 5 epochs). However, these advantages diminish as the training process continues, particularly in the writing task, where the average quality of answers from the SMILE-optimized model notably declines. In conclusion, the significant effectiveness of SMILE in the SFT task is not evident. As stated the Limitations section (Section 6 in the main paper, SMILE faces challenges in extending other text generation tasks.
C Cross-model Generalizability
Since the aforementioned experiments validate the efficacy of SMILE on the base version of BLIP (and BLIP4video), it is crucial to investigate the generalizability of SMILE across different model architectures and sizes. Thus, we apply SMILE to several popular VLP models across scales 23 The implementation details are consistent with their respective officially released codes. Table displays the performance of these models on MSCOCO. For all models, SMILE notably increases the average length and lexical diversity of the captions, indicating its bemiiperreabile can generalize well to different model architectures and scales. Besides, for BLIIP and OFA, SMILE also significantly enhances their descriptiveness, evidenced by improved self-retrieval performance. However, such gains are not observed in mPLUG and VL While the generated captions become longer with SMILE optimization, their accuracy is adversely affected, leading dropped self-retrieval performance and CLIPScore. Regarding this discrepancy might be that mPLUG and VinVL adopt Prefix Language Modeling (PrefiXLM) 35) or Masked Language Modeling (MLM) [7] styles pre- pre-training for text generation, whereas BLIP and OFA utilize Causal language modeling (CLM). We hypothesize that CLM intrinsically more advantageous for text generation tasks, potentially enabling models to develop generation pattern that is more robust and adaptable in embracing SMILE optimization. We leave further exploration on this matter for future work.
50 21 60]
different models, one possible reason
17

---

Table 9: Comparison of SMILE-optimized models and MLLMs.
Caption Length
Lexical Diversity
Self-Retrieval
Model
CLIPScore
PPL
R@D
R@5
BLIP-CsMII
22.3
4.5
10.0
24.5
75.0
95.6
BLIP- MLLEASMILE
19.8
3.6
10.9
25.1
76.2
79.4
MiniGPT-4
66.3
6.4
10.5
23.8
75.6
14.8
LLaVA
32.5
3.8
23.4
79.7
20.0
Table 10: Performance of models trained on MSCOCO with and without paraphrases.
Training Data
Caption Length
Lexical Diversity
Self-Retrieval
Method
Paraphrase
CLIPScore
PPL.
R@D
R@5
BLIP
567K
10.0
1.4
6.7
16.6
77.2
95.8
BLIP-CSMILE BLIP-ESMILE
S67K
22.3
4.5
10.0
24.5
75.0
95.6
113K
17.9
3.3
8.9
21.3
75.2
90.3
D
Comparison with Multimodal LLMs
With the advent of the LLM era, various Multimodal LLMs (MLLMs) [56] developed facilitate general multimodal tasks, including descriptive image captioning. Therefore, we conduct a shallow evaluation with two popular MLLMs. MiniGPT-4 (7B) [62] and LLaVA (Lightning-MPTT MPT-7B) [25]. to compare them with our SMILE-optimized models. Models are prompted to generate captions with an example prompt from the LLaVA project, namely "Describe the following image. As shown in Table 9 BLIP with SMILE achieves comparable self-retrieval performance with MLLMs. while underpertorming them obviously in both caption length (MiniGPT-4) and accuracy (LLaVA). However, it is important to note that such direct comparison might be somewhat unfair. While MLLMs employ additional training data (e.g., detailed description) to learn to describe detail, SMILE enables models to directly learn from existing data (e.g.. short captions in MSCOCO) for more descriptive captioning. Besides, our model size (~227M) and pre-training data scale are significantly smaller than MLLMs. Despite the strong capabilities demonstrated by LLMs, the heavy computational resources they rely on are worth noting.
E
Paraphrases Ablation
Captioning datasets including MSCOCO and Flickr30K usually contain multiple annotations for one image or video [24] 57 53). a.k.a., paraphrases. This may raise questions about whether the gains in descriptiveness come from combining these paraphrases, rather than 'absorbing' from the entire training dataset. On this topic, we test with a no-paraphrase corpus, keeping just one caption per image for training in MSCOCO. As shown in Table TO] by learning from corpus without paraphrases, the model can still boost descriptiveness, with remarkable improvement in terms of caption length, lexical diversity, and self-retrieval performance. This suggests that paraphrases in training data are not the key to descriptiveness gains.
18


Reviews:
[{"review_id": "axrn6SznSU", "reviewer": "Reviewer_cENC", "summary": "This paper argues that captioning models with maximum likelihood estimation as the training objective led to conciseness optimization, which limits the model to generate descriptive captions. To this end, they introduce semipermeable maximum likelihood estimation (SMILE), which allows richness optimization for generating longer and more detailed sentences.  As a result, they conduct extensive experiments on MSCOCO and Flickr30K datasets, which demonstrate the SMILE significantly enhances the descriptiveness of generated captions. ", "questions": "As shown in Weakness.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\t**The writing is clear and well-motivated**. It is urgent to explore a new training object function for image captioning optimization. And conciseness optimization is a core problem that needs to be addressed in image/video captioning.\n2.\t**The proposal SMILE is very subtle**. SMILE considers the probability over a vocabulary subset with limited words (words in the ground truth caption), which is an interesting exploration. The block-level conciseness optimization idea is effective.\n3.    **The result is convinced**. The result demonstrates that SMILE can generate significantly longer and more descriptive captions.\n4.   **The SMILE is architecture-agnostic and plug-and-play**.  It can generalize to many existing VLP models.", "weaknesses": "1. **The SMILE is not generalized to the large language models**. It doesn’t observe significant improvement in response length or overall quality.\n2. **Generalization of the video captioning.** Although they conduct experiments on MSR-VTT for video captioning, it will be interesting to extend SMILE to dense video captioning (e.g. Activity-Net).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the thoughtful comments and valuable suggestions. Our responses to the specific questions are as follows:\n\n> **Q1: The SMILE is not generalized to the large language models.** It doesn’t observe significant improvement in response length or overall quality.\n\n**A1:** The initial motivation for this paper was based on the idea that MLE is not perfectly suitable for **image captioning**, as \"a picture is worth a thousand words\". Hence, we introduced SMILE to encourage models to generate longer captions with more details. In addition, we have conducted **preliminary verification** on the Supervised Fine-Tuning (SFT) of Large Language Models (LLMs) as **an intriguing exploration in the general question-answering domain**. We have demonstrated in *Limitation* and *Supplementary* that SMILE may not be directly generalizable to other tasks, and we analyzed the possible reasons, hoping to offer some empirical experiences for future research. However, it's essential to note that, as a method specifically designed for the captioning task, the generalizability of SMILE for broader tasks is beyond the scope of this paper. We leave it to our future studies.\n\n> **Q2: Generalization of the video captioning.** Although they conduct experiments on MSR-VTT for video captioning, it will be interesting to extend SMILE to dense video captioning (e.g. Activity-Net).\n\n**A2:** Dense video captioning requires the model to describe multiple events from an untrimmed video, where a more detailed description would be beneficial. Thus, extending SMILE to this domain is indeed exciting. However, being a combination of two sub-tasks, namely **event detection** and **event captioning**, the mainstream solutions for dense video captioning either adopt a two-stage approach, where the second stage captioning **boils down to a typical video captioning task**, or they jointly address both sub-tasks, resulting in **the language generation being influenced by inter-subtask interaction**. For example, the state-of-the-art method Vid2Seq [1] augments a language model with special time tokens, incorporating event localization into sequence generation. Therefore, applying SMILE to dense video captioning is not a straightforward and simple task and requires specific and meticulous design for different models. Additionally, given the length and rich semantics of dense video captions, evaluating the generated results presents its challenges.\n\nWe appreciate this valuable suggestion. We will explore the performance of SMILE in dense video captioning in future work.\n\n[1] Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning, CVPR 2023.\n\nOnce again, we sincerely appreciate your comments and suggestions. Please let us know if there's more feedback."}, {"review_id": "UOYp5FGmfR", "reviewer": "Reviewer_5fp8", "summary": "The authors proposed the SMILE optimization method to enrich the detailed description of images in Captioning models. The authors analyzed the optimization process in the Image Captioning task and divided the optimization process into conciseness optimization and richness optimization. By suppressing the conciseness optimization process, the authors help the model to enrich the detailed description of the text. Specifically, the authors further train the already trained captioning model and then only calculate the loss of vocabulary on GT during the optimization process, thus avoiding the simple GT suppressing the model from outputting detailed vocabulary.", "questions": "1.\tCan it be demonstrated that SMILE does not have a significant loss in semantic correctness?\n2.\tIs there a more specific metric to show why SMILE can help enrich the image description?\n\n\n\n", "limitations": "1.\tFrom the current experimental results, the SMILE method has a significant loss in the correctness of image descriptions. Therefore, how to help the model enrich the image description while ens\n", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe authors proposed the SMILE optimization method, which can significantly improve the model's degree of image description and enrich the vocabulary richness of the output Caption.\n\n2.\tThe ablation study built by the authors provides a detailed discussion of the working principle of the SMILE method. By selecting the right subset selection strategy, the model's output caption's degree of image description can be controlled. Furthermore, the effectiveness of this method in other tasks (Video Captioning) is demonstrated.\n\n3.\tThe SMILE method is a fine-tuning method, so it can help a model to converge quickly and enrich the description of images in Captioning models.\n", "weaknesses": "1.\tSMILE shows a significant decline in the CLIPScore metric across multiple experiments, and there is also a significant drop in the Acc metric for semantic evaluation in Table 2, indicating that SMILE has some issues concerning semantic correctness.\n\n2.\tThe authors only tested the effect of SMILE on the BLIP model. It would be better to test a few more models to ensure the robustness of this method.\n\n3.\tIn the \"Models 'absorb' details from the corpus\" section, a more refined metric might be needed to highlight the effect. Using only one visualization example lacks persuasiveness.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the thorough understanding and insightful comments. We address the specific concerns and questions as follows:\n\n> **Q1: (a)** SMILE shows a significant decline in the CLIPScore metric across multiple experiments, and there is also a significant drop in the Acc metric for semantic evaluation in Table 2, indicating that SMILE has some issues concerning semantic correctness. **(b)** Can it be demonstrated that SMILE does not have a significant loss in semantic correctness?\n\n**A1 (a):** As highlighted in our paper, \"**Talks much errs much.**\" The MLE-optimized model tends to output generic captions of about 10 words (e.g., \"a woman holding a cake with lit candles\"), which usually contain simple concepts to describe the main features in images, and thus are less prone to errors. In contrast, SMILE captions, with an average length exceeding 20 words, contain rich additional details, amplifying the risk of errors, especially when these details are inherently harder to identify (e.g., small objects) and express (e.g., less frequent words).\n\nTherefore, we believe the observed decline in CLIPScore (-2.2 on MSCOCO and -0.6 on Flickr30K) is reasonable and acceptable. Also, it's important to note that since the CLIP model isn't impeccable at comprehending all details in images and captions (especially richer SMILE captions with more low-frequency words), we believe that the reliability of CLIPScore needs to be viewed with caution.\n\nIn human evaluations, we noticed that the annotators often base accuracy assessment on **error spotting**, meaning simple captions easily get perfect scores (i.e., 5). However, for very detailed captions, even a slight inaccuracy can reduce scores (to 4 or less), greatly disadvantaging SMILE in human evaluations. To illustrate this better, we provide some real examples from our human evaluations in ***Global Response***.\n\n**A1 (b):**\n- Firstly, SMILE has shown significant improvement in **self-retrieval performance**, suggesting that the generated captions are fundamentally correct and more discriminative for retrieving images. An imprecise caption would struggle to retrieve the original image, especially at Recall@1.\n- Secondly, in human evaluations, an accuracy score beyond **4 (*good*)** demonstrates the good quality of SMILE captions, even though they confront the risk of \"talks much errs much\".\n- Lastly, our experiments in *Supplementary* show that a **compromise** between descriptiveness and accuracy can be achieved by simply combining SMILE with MLE, and such a compromise can lead to **enhanced accuracy and self-retrieval performance**.\n\nBeyond the above responses, we'd also like to mention that while SMILE helps models in altering their **generation habits** (i.e., producing longer and more detailed captions), the quality of the caption still relies a lot on the basic model's **inherent capability**. We view SMILE as a preliminary exploration and leave further accuracy enhancements to our future endeavors.\n\n> **Q2:** The authors only tested the effect of SMILE on the BLIP model. It would be better to test a few more models to ensure the robustness of this method.\n\n**A2:** Thank you for pointing this out. In this paper, we validated only on BLIP-base and its video task variant. However, ensuring robustness across different models and model scales is essential. As a result, we applied SMILE to **BLIP-large** and another popular VLP model, **OFA**, on MSCOCO. As shown in the table, SMILE consistently improves the generation length, lexical diversity, and descriptiveness of the models. In the revised manuscript, we will include a new section in the appendix with more models and analyses.\n\nMethod|Cap. Len.|Lex. Div.|R@1|R@5|CLIPScore|PPL\n-|:-:|:-:|:-:|:-:|:-:|:-:\nBLIP-large|10.0|1.4|6.7|17.3|**77.4**|92.3\nBLIP-large-SMILE|**24.0**|**4.9**|**9.2**|**22.4**|74.7|100.0\nOFA-base|10.1|1.2|5.9|15.6|**76.7**|64.6\nOFA-base-SMILE|**16.5**|**4.0**|**6.9**|**16.2**|71.6|109.7\nOFA-large|9.8|1.2|6.4|16.0|**77.1**|68.6\nOFA-large-SMILE|**18.8**|**5.9**|**9.3**|**21.9**|73.6|125.9\n\n> **Q3:** In the \"Models 'absorb' details from the corpus\" section, a more refined metric might be needed to highlight the effect. Using only one visualization example lacks persuasiveness.\n\n**A3:** As introduced in the preamble of Section 3.4 *Further Analysis*, this section digs into **where the model learns the usage of detailed words**. To answer the question, we conducted experiments to explore how the presence of detail in the corpus impacts the efficacy of SMILE. We constructed two datasets for this purpose: Simplest-COCO, which contains no details, and Simpler-COCO, with only a few details. The results demonstrate that on Simplest-COCO, SMILE does not induce the model to produce longer outputs. However, even with minimal details present in the corpus (Simpler-COCO), SMILE is able to incentivize the model to express more details. This suggests that the model's capability to express details **stems from the training corpus**. If we understand your inquiry correctly, the metric we employ to highlight the effect is the caption length variation, i.e., as the caption becomes progressively longer, it indicates the model is \"absorbing\" the capacity to generate details. We will refine this expression for better clarity in the revised version. Lastly, as for our visualization, it's primarily meant to showcase **how SMILE exerts its effect from token-level loss**, helping better understand our methodology; it should not serve as the visualization of the \"absorb\" section.\n\nOnce again, we sincerely appreciate your concerns and suggestions and hope our responses address them. Please let us know if there's more feedback."}, {"review_id": "j3EWqDPFXl", "reviewer": "Reviewer_XLEL", "summary": "This paper proposes a novel further training approach called SMILE (Submodular Mixup Language Evaluation) for descriptive image captioning, which aims to improve the diversity and quality of generated captions. SMILE introduce to suppress conciseness optimization while encourge richness optimization to incentivize the model to generate more descriptive captions. It is based on submodular optimization and mixup techniques. It is worth mentioning that，the proposed method is architecture-agnostic and can be applied to any captioning model with MLE optimization. Experiments, conducted on two different datasets, show that the proposed approach can achieve competitive performance on three aspects: descriptiveness, accuracy, and fluency.", "questions": "See Weaknesses.", "limitations": "Yes, the authors have adequately addressed the limitations and, if applicable, potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The perspective of model optimization about the unbefitting strict supervision with MLE for image captioning is intuitive. It also takes consideration into two optimizations, richness optimization and conciseness optimization.\n2. A flexible and lightweight modification of MLE with a reconducted subset has been developed for further training. This modification is designed to be architecture-agnostic, meaning it can be utilized with any visual captioning model that incorporates MLE optimization.\n", "weaknesses": "1. If I understand the paper correctly, the SMILE loss suppresses the conciseness optimization and maintains the richness optimization.  The former removes the forcing optimization for descriptive and correct words that are typically not in the subset, while the latter takes into account the concise and correct words that are included in the ground-truth caption and thus in the subset. The criterion used to assess richness and conciseness appears to be relatively simple.\n2. The proposed approach in this paper involves a scheme for further training existing models for image captioning. Nevertheless, several large models, including MiniGPT-4 and LLaVA, have been developed to generate more descriptive captions. Therefore, it is essential to compare the proposed approach with these existing models in the experiment.\n3. During inference, if the target sequence is unknown and the subset is unavailable, does the model output a probability distribution over the whole vocabulary? This can potentially create inconsistencies between the training and testing phases.\n4. In Section 3.4, the authors discussed the \"semi-permeability\" within the subset selection strategy and presented their findings in Figure 4. However, the meaning of \"semi-permeability\" is not clearly defined. Specifically, it is unclear whether \"semi-permeability\" refers to the ability to select and optimize models between the two extremes of simplicity and richness. Furthermore, while Figure 4 shows the relationship between the subset selection strategy and caption length, it does not shed light on the relationship between \"semi-permeability\" and \"caption length.\" \n5. The main text and supplementary material do not provide any qualitative results.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the thoughtful comments and valuable feedback. We address the specific concerns as follows:\n\n> **Q1:** The criterion used to assess richness and conciseness appears to be relatively simple.\n\n**A1:** We understand your concern, but we have to clarify that what we refer to as richness and conciseness are **relative concepts**. For example, conciseness optimization is attributed to the **MORE descriptive** prediction than the ground truth, and these \"more descriptive\" words are typically outside the ground truth caption. From another perspective, since we would like the captions to convey **additional details** beyond the ground truths, we need the **new words outside the sentence**. The same logic applies to richness optimization.\n\n> **Q2:** It is essential to compare the proposed approach with existing multimodal LLMs.\n\n**A2:** Thank you for the suggestion. While multimodal LLMs employ **additional training data** (e.g., detailed description) for supervised fine-tuning to align the model with the capability to describe in detail, our exploration focuses on leveraging existing data (e.g., short captions in MSCOCO) to learn more descriptive captioning. Given that our method **does not rely on extra training data**, and the model size (~227M) and pretraining data scale are **significantly smaller** than multimodal LLMs, a direct comparison might be somewhat unfair. However, we believe such a comparison remains interesting and crucial to understanding and contrasting the performance of models under different paradigms. Therefore, we've added evaluations for two popular multimodal LLMs, MiniGPT-4 and LLaVA, on the MSCOCO test set. As shown in the table, BLIP with SMILE achieves comparable self-retrieval performance with LLMs, while underperforms them obviously in both caption length (MiniGPT-4) and accuracy (LLaVA). However, despite the strong capabilities demonstrated by LLMs, the heavy computational resources they rely on are worth noting.\n\nMethod|Cap. Len.|Lex. Div.|R@1|R@5|CLIPScore|PPL\n-|:-:|:-:|:-:|:-:|:-:|:-:\nBLIP (SMILE)|22.3|4.5|10.0|24.5|75.0|95.6\nBLIP (MLE+SMILE)|19.8|3.6|**10.9**|**25.1**|76.2|79.4\nMiniGPT-4-7B|**66.3**|**6.4**|10.5|23.8|75.6|14.8\nLLaVA-Lightning-MPT-7B|32.5|3.8|9.7|23.4|**79.7**|20.0\n\n> **Q3:** During inference, if the target sequence is unknown and the subset is unavailable, does the model output a probability distribution over the whole vocabulary? This can potentially create inconsistencies between the training and testing phases.\n\n**A3:** As stated in the *Method* (line 93), during inference, the model predicts across the entire vocabulary to determine the current generation. In fact, **during both training and inference, the model calculates logits over the whole vocabulary**. During training, SMILE selects part of the logits to calculate a subset probability distribution (using Softmax) for **loss calculation**, which is a post-processing of the model output and is independent of the predicting process. In other words, the **prediction behavior** of the model remains consistent during training and inference; the subset selection is only for the **evaluation** of the model's prediction (i.e., loss calculation). Hence, we believe SMILE does not introduce an inconsistency between training and inference.\n\n> **Q4: (a)** The meaning of \"semi-permeability\" is not clearly defined. **(b)** While Figure 4 shows the relationship between the subset selection strategy and caption length, it does not shed light on the relationship between \"semi-permeability\" and \"caption length.\" \n\n**A4 (a):** As we mentioned in the *Introduction* (line 62), \"semi-permeability\" refers to the nature of SMILE that **allows richness optimization but blocks conciseness optimization**, similar to a semipermeable membrane. We will describe the meaning more clearly in the revised version.\n\n**A4 (b):** By designing different subset selection strategies, different **types** of \"semi-permeability\" are achieved, which in turn affects the caption length. Specifically, *SMILE subsetting* creates a type of \"semi-permeability\" that permits richness optimization while blocking conciseness optimization, leading to gradually increasing caption lengths. In contrast, *reverse subsetting* establishes an opposite type of \"semi-permeability\" that only permits conciseness optimization, resulting in decreasing caption lengths. Here, caption length variations serve as indicators of the presence of \"semi-permeability\". We will enhance this explanation in the revised version. Moreover, our *Supplementary* provides more analysis about how the **degree** of \"semi-permeability\" affects caption length. We present experiments that combine SMILE with MLE, illustrating the significant correlation between caption length and the proportion of SMILE loss.\n\n> **Q5:** The main text and supplementary material do not provide any qualitative results.\n\n**A5:** We've included some qualitative results in Fig. 1. To show more results of SMILE's effects, we've also provided a PDF file with additional qualitative results in ***Global Response***. These results are from our randomly selected human evaluation set, accompanied by their obtained scores.\n\nOnce again, we sincerely appreciate your concerns and suggestions and hope our responses address them. Please let us know if there's more feedback."}, {"review_id": "RVtFs4FniG", "reviewer": "Reviewer_nk4n", "summary": "## Summary\n\n- In the paper, the authors introduce a new method for generating descriptive captions for images. They call it SMILE, which stands for \"Semipermeable Maximum Likelihood Estimation.\" Unlike traditional image captioning models that often produce generic descriptions, SMILE allows for more detailed and rich descriptions while still maintaining accuracy.\n- Image captioning differs from Machine Translation in that there are numerous ways to describe an image, and limited human captions cannot encompass the entirety of possible descriptions. To address this issue, SMILE utilizes information from the entire corpus. The model used is BLIP, which is first fine-tuned with standard MLE loss and then further fine-tuned with SMILE loss. The dataset used for this project includes MSCOCO and Flickr30K.\n- SMILE only operates at training time.\n- The key idea in SMILE is restricting the vocabulary to only the unique words present in the target caption during decoding.\n- Since most generation models are trained with teacher forcing, they might suffer from exposure bias. Since SMILE does not require prediction to be consistent with ground truth labels, it is more prone to deviate from the ground truth label. To alleviate this problem, the authors propose two techniques\n\t- First Token MLE\n\t\t- MLE loss for first token, SMILE loss for remaining tokens in the sequence.\n\t- First Token Shifting\n\t\t- Change the label of first token to a similar token \"a\" -> \"##a\" where ## denotes the subword prefixed with a space. This results in model making a mistake on first token and encourages the model to get the first token right during the SMILE training and subsequently during inference.\n- The metrics used are CLIP score, caption length, lexical diversity, self-retrieval@1 and @5, and perplexity for measuring fluency.\n- The baseline models compared against are NliCap, GdiCap and CapEnrich which are quite standard for descriptive image captioning. The authors also do human evaluation on a small subset (50 examples).\n- Results indicate that the SMILE technique is able to generate more descriptive captions.\n\n\n####  nit\n- Typo Line 259 SMLIE -> SMILE", "questions": "### Remarks and Questions\n\n- All metrics used in the paper are automatic metrics. Have you considered using RefCLIPScore instead of CLIPScore, which also gives some weight to the reference captions?\n\t- Perhaps another metric to consider could be running an object detector on the image (like RCNN and collecting all objects correctly identified in the caption)\n- Can you provide more details on human evaluation? 50 seems a very small number of samples to justify the results conclusively.  In Table 2, descriptiveness improves to 4.64, but there is also a drop in accuracy compared to the last row.\n- Both MSCOCO And Flickr30K datasets provide multiple captions. It is unclear how the multiple reference captions are used from reading the paper. While constructing the subset vocabulary, do we take a union of all tokens from all the captions for an image? \n\t- Maybe the gains in descriptiveness are coming from combining all captions and the analysis presented in the section \"Models absorb details from the corpus.\"\n- For the video captioning task, the human perplexity is quite high, 561.8, which is significantly higher than other numbers, any idea why this is the case?", "limitations": "The authors have appropriately addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Paper tackles a significant problem in image captioning. \n- Easy to read, prior work is cited well, and results are promising.", "weaknesses": "- Most metrics used are automatic, and the human evaluation set is very small. \n- Paper could also benefit from additional details around human evaluation experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful understanding and valuable feedback. We address each of your concerns and questions as follows:\n\n> **Q1: (a)** All metrics are automatic metrics. Have you considered RefCLIPScore instead of CLIPScore? **(b)** Perhaps another metric could be running an object detector.\n\n**A1 (a):** As shown in Fig. 1, SMILE encourages much richer captions than the ground truth, and such **mismatches** could lead to **unfair penalization** by reference-based metrics, including RefCLIPScore. Thus, we did not consider these metrics. We concur with your concerns regarding automatic metrics, which prompt us to conduct additional human evaluations.\n\n**A1 (b):** Thank you for proposing detection-based metrics. We recognize that the details in captions might be multifaceted, e.g., object attributes and states, and it is difficult for object detection methods to fully reflect these details. However, we agree that object information is an important aspect of descriptiveness, so we added a corresponding evaluation with **UMIC** [1], where the image encoder takes the Faster R-CNN features as input, enabling a better grasp of objects in images. The table below shows UMIC evaluation on MSCOCO, where SMILE notably boosts the basic model's performance. It's worth noting that the best-performing method, CapEnrich, introduces auto-constructed training data by extracting scene graphs from multiple reference captions, which could benefit object recall.\n\nMethod|UMIC\n-|:-:\nNliCap|65.5\nGdisCap|67.4\nCapEnrich|**71.5**\nBLIP|68.2\nBLIP-SMILE|69.9\nHuman|71.1\n\n[1] UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning, ACL 2021.\n\n> **Q2: (a)** Can you provide more details on human evaluation? **(b)** 50 seems to be a very small number of samples. **(c)** In Table 2, descriptiveness improves but accuracy drops.\n\n**A2 (a):** Our human evaluation was a blind review process, with each caption candidate scored by five annotators on three dimensions: descriptiveness, accuracy, and fluency. A score of 5 indicates \"*excellent*\", e.g., no errors or inaccuracies (in accuracy), or including rich details (in descriptiveness). Conversely, a score of 1 suggests that the caption is \"*terrible*\", e.g., entirely wrong or describes minimal visual content. To help better understand the human evaluation results, we've provided some real examples in ***Global Response***. We will enhance the description of human evaluations in our revision.\n\n**A2 (b):** We agree that 50 cases may not sufficiently justify our results. Thus, we've conducted a new round of human evaluation with **another 50 random cases**. While 100 cases is still a modest sample set, it entails a substantial workload of 1200 ratings for each annotator. The updated results (100 cases) shown in the following table align with our initial findings. \n\nMethod|Descrip.|Acc.|F1|Flu.\n-|:-:|:-:|:-:|:-:\nCapEnrich|3.89|4.39|4.12|4.36\nBLIP|3.41|**4.57**|3.91|**4.91**\nBLIP-SMILE|**4.67**|4.05|**4.34**|4.75\nHuman|3.53|4.53|3.97|4.87\n\n**A2 (c):** As highlighted in our paper, \"**Talks much errs much.**\" Generic captions (~10 words long) often contain simple concepts to describe the main features in images, and thus are less prone to errors. In contrast, SMILE captions (over 20 words on average) contains much more details, amplifying the risk of errors, especially when these details are inherently harder to identify (e.g., small objects). We noticed that the annotators often base accuracy assessment on **error spotting**, meaning simple captions easily get perfect scores (i.e., 5). However, for very detailed captions, even a slight inaccuracy can reduce scores (to 4 or less), greatly disadvantaging SMILE in human evaluations. We've illustrated this with examples in ***Global Response***. Nevertheless, it's worth noting that SMILE captions still scored above 4 on average.\n\nFinally, if accuracy is still a concern, our *Supplementary* experiments show that descriptiveness and accuracy can be **balanced** by combining SMILE with MLE, and such a trade-off can lead to enhanced accuracy and self-retrieval performance.\n\n> **Q3: (a)** It is unclear how the multiple reference captions are used. **(b)** Maybe the gains in descriptiveness are coming from combining all captions and the paper analysis.\n\n**A3 (a):** Thank you for your in-depth understanding. We didn't have any specific scheme to leverage multiple reference captions (paraphrases). Unlike some methods that necessitate paraphrases for learning descriptive captioning, SMILE naturally integrates such capability from the training corpus.\n\n**A3 (b):** We totally agree with you that the descriptiveness gains might partly come from the paraphrases. However, our work didn't utilize these paraphrases but treated them as distinct training samples, hence we didn't mention them specifically. On this topic, we tested with **a no-paraphrase corpus**, keeping just one caption per image for training in MSCOCO. Subsequent table results confirm that the model could still boost descriptiveness without paraphrases.\n\nMethod|Paraphrase|Train. Data|Cap. Len.|Lex. Div.|R@1|R@5|CLIPScore|PPL\n-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nBLIP|√|567K|10.0|1.4|6.7|16.6|77.2|95.8\nBLIP-SMILE|√|567K|22.3|4.5|10.0|24.5|75.0|95.6\nBLIP-SMILE|×|113K|17.9|3.3|8.9|21.3|75.2|90.3\n\n> **Q4:** In video captioning, human perplexity is significantly higher.\n\n**A4:** Since it's common for human language perplexity to be higher than that of language models, especially in video descriptions where more intricate semantics are conveyed (each MSR-VTT video is annotated with 20 captions, indicating great diversity), we believe such relatively high perplexity is reasonable. While perplexity isn't a high-precision metric, we use it to demonstrate that SMILE captions don't have serious fluency issues, rather than comparing different models.\n\nOnce again, we sincerely appreciate your meticulous feedback. Please let us know if you have further inquiries or remarks."}]

Author rebuttal:
We sincerely thank all the reviewers for their valuable feedback, which is crucial for further refining our work. After the initial submission, we dedicated considerable time to improving the paper's structure and writing. Consequently, some of the potentially confusing and ambiguous areas highlighted in the reviews should now be enhanced. Moving forward, we will thoroughly consider the reviewers' comments and suggestions to make further revisions to our manuscript.

Furthermore, to better address the reviewers' questions, we provide supplementary materials as an attachment, including some qualitative results from our human evaluations. We hope these results will assist the reviewers in gaining a deeper understanding of our work.

Meta-review: The paper presents a novel training technique called SMILE, which is specifically designed for captioning. By blocking conciseness optimization of MLE through subsetting vocabulary, SMILE effectively allows the final trained model to generate more descriptive captions than the ground-truth captions themselves. The paper also convincingly shows the effectiveness of the method and valid analyses.

As Reviewers cENC and nk4n mentioned in their reviews, conciseness optimization is a core problem that needs to be addressed in image/video captioning. Also as pointed out by Reviewer 5fp8, the ablation study and analyses provide a sound discussion of the working principle in the SMILE optimization, and demonstrate its effectiveness.

Overall, the work is timely and will definitely draw attention from the community. Based on the positive consensus made across all the reviewers, the AC also recommends the acceptance of the paper.
Decision: Accept (poster)