Models See Hallucinations: Evaluating the Factuality in Video Captioning
Authors: Hui Liu
Xiaojun Wan
Venue: EMNLP 2023 Main (2023)
Keywords: Hallucination
Factuality Evaluation
Video Captioning

Abstract: Video captioning aims to describe events in a video with natural language. In recent years, many works have focused on improving captioning models' performance. However, like other text generation tasks, it risks introducing factual errors not supported by the input video. Factual errors can seriously affect the quality of the generated text, sometimes making it completely unusable. Although factual consistency has received much research attention in text-to-text tasks (e.g., summarization), it is less studied in vision-based text generation. In this work, we conduct the first human evaluation of the factuality in video captioning and annotate two factuality datasets. We find that 56\% of the model-generated sentences have factual errors, indicating it is a severe problem in this field, but existing evaluation metrics show little correlation with human factuality annotation. We further propose a weakly-supervised, model-based factuality metric FactVC, which outperforms previous metrics on factuality evaluation of video captioning.

Models See Hallucinations: Evaluating the Factuality in Video Captioning
Hui Liu and Xiaojun Wan
Wangxuan Institute of Computer Technology, Peking University (xinkeliuhui, wanxiaojun edu. cn
Abstract
Video content:
Video captioning aims to describe events in video with natural language. recent years, many works have focused on improving cap- tioning models' performance. However, like other text generation tasks, risks introducing factual errors not supported by the imput video. Factual errors can seriously affect the quality of the generated text, sometimes making it com- pletely unusable. Although factual consistency has received much research attention in text- to-text tasks (e.g., summarization), is less studied in vision-based text generation. In this work, we conduct the first human evaluation of the factuality in video captioning and annotate two factuality datasets. We find that 56% of the model- generated model-generated sentences have factual errors, indicating it is a severe problem in this field, but existing evaluation metrics show little cor- relation with human factuality annotation. We further propose eeakly-supervieed model- based factuality metric FactVC, which outper- forms previous metrics on factuality evaluation of video captioning.'
Caption 1:
A woman is throwing darts at a board. She throws them at board. She jumps off into the distance and smiles.
Caption 2:
A man is seen standing in room and leads into a man speaking to the camera.
The man is throwing darts at a dart board. The man then throws the dart board and then goes back to the camera.
Caption 3:
man in a white shirt standing at a dart board. He throws a dart at the end.
Table I: The factual emors video captioning. show video with three captions from different captioning models. Factual errors are marked in red.
1 Introduction
Video captioning is a challenging cross-modal task that aims describe videos with natural language sentences. In recent years, video captioning has received much attention in computer vision and nat- ural language processing communities. Substantial progress has been made to generate descriptions for videos that contain single event (Venugopalan et al., 2015; Pan et al., 2020) or multiple events (Zhou et al., 2018b; Lei al., 2020). However, like other text generation tasks, video captioning mod- els risk introducing factual errors not supported by the input video. Examples are shown in Table 1. This paper defines factual errors (or hallucinations) as follows: span of caption text that contradicts the video or describes something not appearing in
the video. Factual errors can cause misunderstand- ings of the video content, sometimes making the generated captions completely unusable.
Factual consistency evaluation has received much research attention in text-to-text tasks, in- cluding summarization (Maynez et al., 2020; Krys- ciński et al., 2020), knowledge-gge -aguudde dialogue (Honovich et al., 2021), text simplification (Devaraj et al., 2022), and large language models(Bang et al., 2023). Nevertheless, it is less studied in vision-to- text tasks, especially video captioning. Therefore, this work focuses the research gap in the factu- ality evaluation of video captioning.
Recently, more works have focused on videos with multiple events (Wang et al., 2021; Yamazaki et al., 2022), and it may bring more factual errors.
'Datasets and code will be released at https: //g:::///iitn thub. com/PKULiuHuii/ee /FactVC

---

So we choose multiple events video captioning for our evaluation. We use ActivityNet Captions (Krishna et al., 2017) and YouCook2 (Zhou et al., 2018a) as our video datasets, for they are the most common datasets for this task. Then we carefully select six recent models on each dataset to gener- ate video captions. The models differ in model framework, pretrained features, and imput signals. After collecting the videos and captions, we design a factuality annotation protocol and conduct hu- man annotation. In the end, we obtain two human- annotated factuality datasets ActivityNet-Fact (200 videos, 3,834 sentences) and YouCook2-Face (100 videos, 4,080 sentences). By analyzing the human annotation, we find that factual error (hallucina- tion) is severe problem in video captioning. To sum up, there are 87% of the paragraphs, 56% of the sentences, and 15% of the words have factual errors. There are different types of factual errors, including person-related errors, action-related er- rors, object-related errors and so on.
2 Related Work
Factuality evaluation is first proposed in the field of document summarization. Maynez et al. (2020) conducted a human annotation on the XSUM dataset (Narayan et al., 2018) and found that more than 70% of summaries generated by summariza- tion models have factual errors. Other human an- notations( Wang et al., 2020; Pagnoni al., 2021) reach similar conclusions. To measure the factual consistency, researchers proposed different met- rics, which can be roughly divided into Entailment- based metrics (Falke et al.. 2019; Kryšciński et al., 2020) and QA-based metrics (Durmus et al., 2020; Wang et al., 2020; Scialom et al., 2021). Inspired by the works in summarization, factuality evaluation is studied for other tasks, including knowledge- grounded dialogue (Honovich et al., 2021), text simplification (Devaraj et al., 2022) and large lan- guage models (Bang et al., 2023).
For vision-based text generation tasks, the most widely used metrics are based on n-gram match- ing between references and generated captions, in- cluding BLEU(Papineni et al., 2002), ROUGE(Lin, 2004), METEOR CIDEr(Vedantam al., 2015). They cannot match deeper semanties between captions and vision in- puts. Recently, there are model-based metrics such as BERTSccrreZhanng et al., 2019), CLIP- Score(Hessel et al.. 2021), EMScore(Shi et al.. 2022), PAC-S(Sarto et al., 2023). They leverage large-scale pretrained models to compute match- ing score, even do not requiring reference captions. A little work pays attention the hallucination problem in vision-based text generation. CHAIR (Rohrbach et al., 2018) proposes an image rele- vance metric to evaluate the object hallucination in image captioning. They restrict their evalu- ation to 80 MSCOCO objects. COAHA(Ullah and Mohanta, 2022) uses object and action words matching to asses hallucination in video caption- ing. EMScore(Shi et al., 2022) and PAC-S(Sarto et al., 2023), designed for the overall evaluation of video captioning, also shows the potential iden- tify hallucinating captions. However, there is still no related human-annotated dataset and complete factuality evaluation work.
Since hallucination is severe problem in video captioning, test to what extent existing auto- matic evaluation metrics can measure the factuality of video captions. We find that most existing met- ries correlate poorly with human judgment. There- fore we propose a new metric VCctVC (Factual consistency for Video Captioning). We use the CLIP model (Radford et al., 2021) to encode video frames and captions. Considering the CLIP model is trained image-text pairs, it may have gap transferring to video factuality evaluation. So automatically construct a training set using text augmentation skills and finetune CLIP on Our VCctVCC metric achieves a higher correlation with human factuality annotation. The main contribu- tions of this work are as follows:
and Lavie, 2005),
We conduct the first thorough factuality eval- uation on video captioning. We find that hal- lucination is a severe problem while existing evaluation metrics can hardly measure it.
We design factuality annotation protocol factuality
and construct two datasets for video captioning.
We propose new factuality metric Ca which achieves much higher correlation with human annotation on video captioning, and can be further transferred to evaluate the factuality of image captioning.
3 Human Annotation
Considering there does not exist factuality anno- tation of video captioning, we decide to construct our own datasets. We use ActivityNet Captions

---

(Krishna et al., 2017) and YouCook2 (Zhou al., 2018a) as our source video datasets and select six recent models for each dataset to generate captions. Then we design a factuality annotation protocol and conduct our human annotation.
label 1 if it has factual errors else label O; Word- level: Within each sentence, annotators need mark phrases and words that have factual errors. It is worth noting that we focus on whether the cap- tion has factual errors given the video (Precision) and do not care whether the caption describes the video completely (Recall). Refer to Appendix A for the complete annotation protocol and examples.
3.1 Datasets and Models
ActivityNet Captions(Krishna et al., 2017) contains 20k untrimmed videos of various human activi- ties. Previous works(Lei et al., 2020; Yamazaki et al., 2022) report results the ae-test split (2,457 videos). We randomly sample 200 videos from the ae-test split for human annotation. YouCook2 (Zhou et al., 201 (0) contains 2,000 long untrimmed videos from 89 cooking recipes. Previous works report results on the val split (457 videos). We randomly sample 100 videos from the val split for human annotation.
3.3 Annotation Procedure
According to previous works (Kryšciński et al., 2020; Wang et al., 2020), the inter-annotator agree- ment through the crowdsourcing platform is rel- atively low. To make annotation more reliable, hire three graduate students as our annota- tors. We provide them with a detailed instruc- tion document and several annotation examples so that they can fully understand the annotation protocol. The annotations are checked multiple times during the annotation process. The anno- tations will be adopted only when an annotator completes all videos and passes every check. We collect three annotations for each video caption and combine them get the final annotation. For paragraph-leve' annotation, we use the median score as the final score. For sentence-level and word-level annotation, we use the majority label as the final label. We quantified the degree of inter-annotator agreement using Krippendorff's 's al- pha coefficient (Krippendorff, 2011). On the Ac- tivityNet videos, the inter-annotator interval met- rics are 0.750, 0.674, and 0.583 for paragraph- level, sentence-level, and word-level annotations respectively. On the YouCook2 videos, the inter- annotator interval metries are 0.781, 0.774, 0.710 for paragraph- sentence-level, and word-level annotations respectively. The metrics show a sub- stantial agreement between annotators. The agree- ment for word-level annotation is relatively low because it has more uncertainty and ambiguity.
We select six recent captioning models for each dataset and obtain the output captions the sampled videos. For ActivityNet Captions, the selected models include: MART(Lei et al., 2020), COOT(Ging et al., 2020), PDVC-gt(Wang et al., 2021), PDVC-pred(Wang et al., 2021), Song(Song et al., 2021), VLTinT(Yamazaki et al., 2022). For YouCook2, the selected models in- clude: VTrans(Zhou et al., 2018b), MART(Lei et al., 2020), COOT(Ging et al., 2020); COOT- 100n et al., 2020), UniVL(Luo al., 2020). VLTinT((Ymmzzak) al., 2022). The above models are different in model framework, input signals, pretrained features, and pretraining scales.
3.2 Annotation Protocol
We design an annotation protocol to instruct an- notators on how to measure and label the factu- ality of video captions. For the factuality anno- tation in summarization, annotators often give binary label 0/1 for each summary sentence, in- dicating whether the sentence is factual or not (Maynez al., 2020; Kryšciñíski et al., 2020). How- ever, the video captions have hierarchical structures pparagraph-seetence--vve
Annotation Analysis
and we want to obtain the factuality annotation for different granularity. So we design new annotation protocol. Anno- tators are asked to give three levels of factuality annotation for each video caption. Paragraph- level: For each paragraph, annotators need to give a factuality Likert scale from to 5, where 1 means the paragraph has many severe factual errors, and 5 means there are no obvious factual errors; Sentence-level: For each sentence, annotators give
4.1 Datasets Statisties
Based on sampled ActivityNet and YouCook2 videos, we collect two annotated factuality datasets ActivityNet-Fact and YouCook2-Fact The ActivityNet- dataset contains 1,200 paragraphs, 3,834 sentences, and 48,235 words, among which 81.9% of the paragraphs, 51.4% of the sentences, and 13.5% of the words have factual errors. The YouCook2-Far dataset contains 600 paragraphs,

---

Category
Description
Example
Ratio
Person-related factual errors, e.g. gender, age, pronoun errors
A woman throwing darts at a board.
Person
25.5%
Action-related factual errors, not consistent with the video
The woman then begins dancing with the dog...
Action
38.1%
Object-related factual errors, not consistent with the video
She then shows off rag and speaking to the camera.
Object
19.9%
Adjective-related factual errors, e.g. color, numerical errors
A person in red shirt is walking towards the camera.
Adjective
6.6%
Poor
Poor-generatee sentence so that contains factual errors
the bull is UNK and the bull UNK
5.1%
Generation
Other factual errors, e.g. relation errors, preposition errors
the person riding the horses the air.
Other
4.8%
Table Typology of factual errors in video captioning. Examples of each category are shown with a related video frame. Factual errors marked in red italics. Ratios shown the ActivityNet- dataset.
4,080 sentences, and 29,879 words, among which 98.3% of the paragraphs, 59.6% of the sentences, and 16.5% of the words have factual errors. This indicates that factual error is severe problem in video captioning and should attract more rescarch attention.
level annotation, we use the ratio of factual sen- tences and words as annotation score). We test model-free metrics BLEU(Papineni et al.. 2002), ROUGE(Lin, 2004), METEOR(Banerjee and Lavie, 2005), CIDEr Vedantam et al., 2015). COAHA(Ullah and Mohanta, 2022), and model- based metrics BERT (Zhang al., 2019), EM- Score(Shi al., 2022) 2, The results are shown
4.2 Factual Error Type Analysis
Factual errors have different categories and dis- tributions in summarization task (Pagnoni al.. 2021). To analyze factual error types in video cap- tioning, we conduct a post-analysis with our anno- tated datasets. We collect phrases/words marked as factual errors appearing at least twice and classify them into different ситог categories. The results are shown in Table 2. We can see that the factual errors in video captioning Fact, the most common factual error categories Person, Action, and Object, which count for 83.5% the total factual errors. For YouCook2-Fact,, Ob- ject is the dominant category, which counts for 92.3% of the total factual errors.
Table and Table 4.
Metric
ref
Para
Sent
Word
Bleu4
T
0.178
0.173
0.174
METEOR
T
0.204
0.196
0.229
Rouge-L
T
0.170
0.151
0.185
CIDEr
T
0.151
0.141
0.133
COAHA
T
0.176
0.174
0.179
various. For ActivityNet-
BERTScore
T
0.243
0.196
0.198
EMScore
V
0.305
0.242
0.341
EMScore
T
0.452
0.389
0.447
EMScore
VT
0.458
0.388
0.464
Table 3: Pearson correlation between automatic evalua- tion metrics and human annotation on ActivityNet- Fact dataset. "ref" means the metric reference is human- written caption (TT), input video (V), or both (VT).
5 Metric Analysis
Now that factual errors broadly exist in video cap- tions, we
know what extent existing metrics can measure the factuality of video cap- tions. We test the correlation between automatic metrics and human annotation (for sentence/word-
To surprise, the most commonly used met-
Foor FMScore, we use the VIT-B/16 CLIP model, which performs better than the default ViT-B/32 CLIP model.

---

Metric
ref
Para
Sent
Word
tomatically using text augmentation skills. Given vidco V together with human -anaan-nn cap- tion sentence T. we use a set of text transformation functions to augment the dataset. The transfor- mation functions include positive transformations (T*) which ensure the new sentence is factually
Blcu4
T
0.197
0.237
0.250
METEOR
T
0.411
0.371
0.415
Rouge-L.
T
0.361
0.333
0.372
CIDEr
T
0.150
0.123
0.179
COAHA
T
0.193
0.236
0.286
correct and negative transformations (T~) which introduce factual errors into the sentence.
BERTScore
T
0.426
0.403
0.434
EMScore
V
0.346
0.372
0.350
The positive transformations include: 1)Para- phrasing: we generate paraphrases using the back- translation method. We use the Google Translation API and use German and French as middle lan- guage; 2) Simplification: we use a tool to simplify complex and compound sentences into simple sen- tences. The negative transformations include: 1) Person Swap: design a set of rules to change person words' gender, age, and pronoun; 2) Action Swap: we collect common action set and apply deletion and insertion; 3) Object Swap: we collect a common object set and apply object substitution; 4) Adjective Swap: we swap adjectives (color, nu- merical words, etc.) in original sentences; 5) Poor Generation: we simulate the poor generation sen- tences by inserting "UNK" word and redundancy phrases. We design the negative transformations according to the factual errors shown in Table 2. We first apply positive transformations to obtain fact-consistent sentences and then apply negative transformations to them to get sentences with fac- tual errors. Finally, we collect a set of data samples (V.T*, T.), where V means the input video, T means a fact-consistent sentence, and T means the corresponding fact-inconsistent sentence. A de- tailed description of the data generation process is in Appendix B.
EMScore
0.524
0.501
0.537
EMScore
VT
0.543
0.530
0.555
Table 4: Pearson correlation between automatic evalu- ation metrics and human annotation on the YouCook2- Fact dataset.
rics for video captioning, such as Bleu4 and CIDEr, correlate poorly with factuality annotation. ME- TEOR performs relatively better but still show a weak correlation with factuality annotation. As for model-based metrics, BERTSccree shows little su- perior to METEOR two datasets, indicating that just introducing large-scale text-pretrained model is not enough. EMScore leverages the image-text pretrained model CLiP(Radford et al., 2021) and shows a higher correlation with human annotation. addition, it can evaluate video captions without human-writte captions.
6 VC Metric
Although EMScore achieves a good correlation with human factuality annotation, has two draw- backs: 1) EMScore uses the pretrained model CLIP, which is trained on image-text pairs from the In- ternet, and it may not transfer well to the video captioning data; 2) EMScore is designed for evalu- ating the overall quality of the video caption, specifically designed for factuality evaluation. As result, we propose a new metric FactVC (Factual consistency for Vidco Captioning). We first auto- matically construct a factuality training set using text augmentation skills and then finetune the CLIP model. We also improve the calculation of the sim- ilarity score so that it is more suitable for factuality evaluation.
6.2 CLIP Finetuning
We only finetune the projection layers of the pretrained CLIP model. Given batch of data {VV, T D with a batch size of B, we first use the CLIP model to calculate the similarities between each video and text:
I Bal(a))
1
$5 cos
(1)
VV.
6.1 Training Data
where s means the similarity score between V, and T,t. cos means cosine similarity, fik the
Collecting a large-scale training dataset through hu- man annotation is expensive and time-consuming. Inspired by (Kryšciński et al., 2020; Gokhale et al., 2022), we decide to construct our training set au-
'https://translat" www.ge..oo.oooooooooooooo...oo hhtps:://gi thub. com/garaing Pintence-Simplific.

---

k-th frame of video V,, Vi| is the sampled frame number, Eg and E, are the vision encoder and text
7 Experiments
7.1 Comparison with Other Metrics
encoder of CLIP model. The similarity score sig between V2 and 7 is computed similarly. Then we finetune the CLIP model using the fol- lowing loss function:
We compare the FactVC metric to other au- tomatic metrics including Bleu(Papineni al., 2002). ROUGE(Lin, 2004), METEOR[Banerjee and Lavie, 2005), CIDEr(Vedantam et al., 2015), COAHA(Ullah and Mohanta, 2022), BERa et al., 2019), EMScore(Shi et al., 2022), PAC-S(Sarto et al., 2023), The results are shown in Table 5 and 6. We omit the metries (3) with worse correlation here (Bleu, ROUGE, ME- TEOR, CIDEr, COAHA), and you check them in Tables 3 and 4. From the table, EMScore and PAC-S perform better than BERTScore, indicating the usefulness of the CLIP model. However, they perform relatively poorly using video as the refer- ence. Our VactVCC metric, on the other hand, shows much better performance in this setting. Com- pared to other metrics, VactVC shows the highest correlation with human annotation in all settings.
B
exp(st
Leoarse
(2)
(cxp(((T))
B
Liie max (0, M - ,, + + sT,))
1=1 C - Leoaree + fine
(4)
where Lecarse a cross-entropy loss to learn whether the video content and text are matched, fine is a hinge loss to learn to assign a higher score to the fact-consistent text. M and are hyper- parameters.
6.3 Score Calculation
7.2 Ablation Study
With the finetuned CLIP model, we can calculate the factuality score FactVC as follows:
We conduct an ablation study to test the effective- ness of each component FactVC. The results are shown in Table 7. FactVC(no finetune) re- moves the finetuning process and shows an obvious (6) performance degradation. FactaC(Cenarse) only uses Lecarse to finetune CLIP and FactVC((Cine)) only uses Lime to finetune CLIP. From the sults, we find that Lecorse can ensure stable fine- tuning, only using fiine is not a good choice, but can help video encoding together with Looarse. FactVC(F-vaaluu uses the F-value-base score in- stead of the precision-based score, showing per- formance degradation. This proves that factual con- sistency is more related to the precision of video captions. FactVC(( 0.5) sets to 0.5 in eq (5) and it is inferior to FactVC with 0.75. This shows that the fine- ane-grained score is more important in Factuality evaluation.
FactVC(V)) - (1 - a.S. + asy
(5)
V
1
S. =cc(
(fk(), (T)
1
99
max cos (E,(f),
[T)
SEEV
JJET
where is a balance factor, S. is the coarse-grained similarity score between video V and sentence T. SP is the precision-based fine-grained similarity score computed between each frame f and each word x.
Similar to EMScore, FCctVC can use video V. human-written caption T", or both (V,T*) as ref- erence. When using human-written caption T** replace the video frame f in Eq (6) and Eq (7) with the caption word and use the CLIP text encoder to compute FactV (TT)) F FacIVC(V T*) is the average of (actVC(() and C(T*).
7.3 Model Ranking
Evaluation metrics are often reported at the system level to compare the performance of different mod- els, and a reliable metric should be consistent with human judgment. We test the performance of the six models on the ActivityNet-Fact dataset using PAC-S and VactVC and report the average scores. The results are shown in Table 8. All the metries are scaled to [o, 1]. Compared to human factual- ity annotation, PAC-S ranks the COOT, PDVC-gt,
We improve the calculation of FactVC metric in two aspects: 1) we use the precision-basee based score instead of the F-value-based score, for factuality is more related to the precision of video captions; 2) we introduce a parameter a to balance the coarse- grained score and the fine-grained score, we set to 0.75 to favor more on fine- -nnagied score.

---

Video ref
Text as ref
Video & Text ref
Metrics
Para
Sent
Word
Para
Sent
Word
Para
Sent
Word
BERTScore
0.243
0.196
0.198
EMScore (ViT-B/32)
0.253
0.190
0.300
0,425
0.356
0.432
0.427
0.352
0,446
EMScore (ViT-B/16)
0.305
0.242
0.341
0.452
0.389
0.447
0.458
0.388
0.464
PAC-S
0.332
0.271
0.384
0.467
0.374
0.478
0.470
0.378
0.495
FactVC
0.462
0.371
0.480
0.511
0.438
0.498
0.551
0.465
0.545
Table 5: Pearson correlation between automatic metrics and human factuality annotation the ActivityNet-Facc dataset. We test each metric in three settings: video as the reference, text as the reference, video and text as the reference. BERTScore only work with text as the reference. Metrics with worse correlation omitted.
Video ref
Text as ref
Video & Text ref
Metrics
Para
Sent
Word
Para
Sent
Word
Para
Sent
Word
BERTScore
0.426
0.403
0.434
EMScore (ViT-B/32)
0.337
0.353
0.361
0.518
0.482
0.523
0.543
0.514
0.553
EMScore (ViT-B/16)
0.346
0.372
0.350
0.524
0.501
0.537
0.543
0.530
0.555
PAC-S
0.312
0.335
0.331
0.543
0,516
0.546
0.562
0.544
0.570
FactVC
0.408
0.410
0.420
0.584
0.558
0.592
0.606
0.583
0.615
Table 6: Pearson correlation between automatic metrics and human factuality annotation the YouCook2-Fact dataset. Metrics with worse correlation are omitted.
Video as ref
Text as ref
Video & Text as ref
Metrics
Para
Sent
Word
Para
Sent
Word
Para
Sent
Word
FactVC
0.462
0.371
0.480
0.511
0.438
0.498
0.551
0.465
0.545
FactVC(no finetune)
0.349
0.281
0.388
0.497
0.425
0.483
0.512
0.433
0.510
FactVC
0.427
0.348
0.466
0.508
0.435
0.492
0.537
0.456
0.532
FactVC(L fine)
0.239
0.166
0.221
0.446
0.394
0.456
0.406
0.336
0.404
FactVC(F-value)
0.442
0.352
0.454
0.474
0.403
0.472
0.512
0.428
0.514
FactVC(( = 0.5)
0.444
0.366
0.458
0.485
0.423
0.474
0.523
0.450
0.518
Table 7: Ablation study on the ActivityNet-Fact dataset. The pearson correlation between each metric and human factuality annotation.
and PDVC-pred models differently. In contrast, FactVC ranks them consistently with human anno- tation.
performance. Even training on a different dataset, the FactVC metric still performs well. Considering the huge domain gap between ActivityNet (ANet) and YouCook2 (You2) datasets, our FactVC metric has good generalizability different video cate- gories and textual styles.
7.4 Cross-Dataset Experiments
We conduct cross-dataset experiment to test the generalizability of the FactVC metric. We use different datasets to finetune CLIP model and test them ActivityNet-Faet and YouCook2-Fact datasets. The results are shown in Table Com- pared to the CLIP model without finetuning, our finetuned method can obviously improve the metric
7.5 Transferring to Image Captioning
According to the above experiments, FactVCC per- forms well evaluating the factuality of video cap- tioning. We want to know whether our method can transfer to image captioning. So we ad-

---

Models
Annotation
PAC-S
FactVC
Metric
ref
Likert
Binary
Word
MART
0.413(6)
0.602(6)
(.569(6)
Bleu4
T
0.266
0.230
0.252
COOT
0.446(4)
0.649(3)
0.615(4)
METEOR
0.308
0.239
0.295
PDVC-gt
0.553(2)
0.647(4)
0.633(2)
Rouge-L
0.364
0.289
0.361
PDVC-pred
0.513(3)
0.652(2)
0.631(3)
CIDEr
0.375
0.300
0.336
Song
0.443(5)
0.641(5)
0.597(5)
CLIPScore RefCLIPScore
0.359
0.220
0.297
VLTinT
0.558(1)
0.678(1)
0.648(1)
VT
0.457
0.298
0.398
CLIPScore"
0.364
0.238
0.309
Table Model performance with ranking on Sentence- level annotation and automatic metrics. We use video and text the reference to compute PAC-S and FactVC metrics. The ranking of each model is shown in paren- theses, and the rankings that are inconsistent with hu- man annotation are marked in red.
RefCLIPScore"
VT 0.466
0.315
0.409
CLIPScore*> **
0.398
0.249
0.367
RefCLIPScore**
VT
0.513
0.341
0.478
Table 10: Pearson correlation between automatic rics and human annotation on MSCOCO-Faas dataset. * and ** mean using indo-ffinetnne, and image-finetuned CLIP model respectively.
ActivityNNt- Fact Dataset
Finetune Data
ref
Para
Sent
Word
None
0.512
0.433
0.510
sults are shown in Table 10. CLIPScore and Re- FCLIPSccoreHessell et al., 2021) are CLIP-based metrics for image captioning. CLIPScore uses im- age as reference and RefCLIPScore additionally uses human captions as reference. CLIPScore* and RefCLIPScore" use video-finetuned CLIP model. We also construct training from im- age caption data using the same text augmenta- tion skills, and finetune the CLIP model to get CLIPScore** and RefCLIPScore** From Table 10, we can see that with our finetuned method, CLIP- Score and RefCLIPScore can better measure the factuality of image captions.
ANet
0.545
0.460
0.544
VT
You2
0.540
0.457
0.533
ANet + You2
0.551
0.465
0.545
YouCook2-Facc Dataset
None
0.572
0.555
0.587
ANet
0.584
0.569
0.604
VT
You2
0.611
0.582
0.613
ANet + You2
0.606
0.583
0.615
Table 9: FactVC metric cross-dataset experiments. We show the Pearson correlation between metrics and hu- man annotation, using input video and human-written caption (VT) as reference.
For more implementation details, experiments and qualitative analysis, please refer Appendix C and D.
ditionally collect an annotated factuality dataset MSCOCO-Fact based on 200 MSCOCO(Lin et al.. 2014) test images and five recent image caption- ing models' outputs. The selected models include: BUTD(Anderson et al., 2018), BUTD-sc(Anderes) et al., 2018), VinVLnZhang et al.. 2021), OFA- base(Wang et al., 2022), OFA-huge(Wann et al., 2022). Unlike video captioning where caption is multi-sentence paragraph, image caption is single sentence. We collect three kinds of factual- ity annotation for each image caption: Likert (1-5 factuality score), Binary (0 or 1, indicating whether the sentence has a factual error), Word (whether each word has factual error, and we use the ratio of factual words as word-level annotation score).
Conclusion
In this work, we focus on the factuality evalua- tion in video captioning. We collect two human- annotated factuality datasets for video captioning and find that hallucination is severe problem, with 56% of the model-generated sentences having different kinds of factual errors. However, most existing metrics show little correlation with hu- man annotation. So we propose a new factuality metric FactVC. It is trained on an automatically- constructed training set and correlates much better with the factuality annotation. Experiments also show the potential of our method in evaluating im- age captions. Although factual consistency a hot research topic in NLP tasks, it is less studied video captioning. We hope our work can fill this
We test the correlation between image caption- ing metrics and human factuality annotation. Re-

---

research gap and promote further research in video captioning.
text simplification. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7331- 7345.
Limitations
Due time and financial constraints, we con- duct our human annotation on the AetivityNet, YouCookII and MSCOCO dataset. In future, we hope to continue our evaluation on more differ- ent datasets (c.g. Charades Captions(Wang et al., 2018), MSR-VTT(Xu et al., 2016)).
Esin Durmus, He He, and Mona Diab. 2020. Feqa: A question answering evaluation framework for faith- fulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5055- 5070.
Tobias Falke. Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Rank- ing generated summaries by correctness: An interest- ing but challenging application for natural language inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages
Ethies Statement
For human annotation, annotators are treated fairly and friendly. We paid them 12 dollars per hour, more than the local average minimum wage. We removed all content in the dataset that contains personal information about the annotators.
Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsi- avash, and Thomas Brox. 2020. Coot: Cooperative
hierarchical transformer for video-text representation learning, Advances in neural information processing systems, 33:22605-2261
Acknowledgments
This work was supported by National Key R&D Program of China (2021 YFF0901502), National Science Foundation of China (No. 62161 160631,, State Key Laboratory of Media Convergence Pro- duction Technology and Systems and Key Lab- oratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We appreciate the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.
Tejas Gokhale, Abhishek Chaudhary. Pratyay Baner- jec, Chitta Baral, and Yezhou Yang. 2022. Semanti- cally distributed robust optimization for vision-and- language inference. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1493-1513.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7514-7528.
References
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. Q2:: Evaluating factual consistency in knowledge- grounded dialogues via question generation and ques- tion answering. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 7856-7870.
Peter Anderson, Xiaodong He, Chris Buehler. Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the act workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65-72.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. In Proceedings of the IEEE in- ternational conference on computer vision, pages 706-715.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt on reasoning, ballucination, and interactivity. arXiy preprint ar 2302.000
Wojeiech Kryšcinski. Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 9332-9346.
Ashwin Devaraj. William Sheffield, Byron Wallace, and Junyi Jessy Li. 2022. Evaluating factuality in

---

Jie Lei, Liwei Wang. Yelong Shen, Dong Yu. Tamara Berg, and Mohit Bansal. 2020. Mart: Memory- augmented recurrent transformer for coherent video paragraph captioning. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 2614-2614.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallu- cination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035-4045.
Sara Sarto, Manuele Barraco, Marcella Comia, Lorenzo Baraldi, and Rita Cucchiara. 2023. Positive- augmented contrastive learning for image and video captioning evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6914-6924.
Chin-Yew, Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization
branches out, pages 74-81.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays. Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European confer- ence on computer vision. pages 740-755. Springer.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. Questeval: Summariza- tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594-6604.
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. 2020. Univl: unified video and language pre-training model for multimodal understanding and generation. arXiv preprint ar 200ivv002.00333..
Yaya Shi, Xu Yang, Haiyang Xu, Chunfeng Yuan, Bing Li, Weiming Hu, and Zheng-Jun Zha. 2022. Emscore: Evaluating video captioning via coarse- moasee-rained and fine aue-grained embedding matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 17929-17938
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919.
Yuqing Song, Shizhe Chen, and Qin Jin. 2021. To- wards diverse paragraph captioning for untrimmed videos. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11245- 11254.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empiricul Methods in Natural Lan- guage Processing, pages 1797-1807
Nasib Ullah and Partha Pratim Mohanta. 2022. Think- ing hallucination for video captioning. In Proceed- ings of the Asian Conference on Computer Vision pages 3654-3671.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstrac- tive summarization with frank: A benchmark for factuality metrics. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 12299
Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image de- scription evaluation. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4566-4575.
Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee. Adrien Gaidon, Ebsan Adeli, and Juan Carlos Niebles. 2020. Spatio-tempon graph for video cap- tioning with Enowledge distillation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10870-10879.
Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. 2015. Sequence to sequence-video to text. In Proceedings of the IEEE international conference on computer vision, pages
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th anmual meeting of the Association for Computa- tional Linguistics, pages 311-318.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 5008-5020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- ur, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR.
Peng Wang, An Yang, Rui Men, Junyang Lin. Shuai Bai, Zhikang Li. Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. Ofa: Unifying ar- chitectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In Inter- national Conference on Machine Learning, pages 23318-23340. PMLR

---

Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng. Ran Cheng, and Ping Luo. 2021. End-to-end dense video captioning with parallel decoding. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 6847-6857.
Algorithm 1 The algorithm to generate dataset
Require:
S set of videos V and captions T
T set of positive transformations
T set of negative transformations
Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang. 2018. Video captioning via hierarchical reinforcement learning, In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4213-4222.
function GENERATE_DATAAS,TT
P <- Ø
> set of positive data
for (V,T) in S do
p t- PU ((V,)) for fn in T+ do
Jun Xu, Tao Mei, Ting Yao. and Yong Rui. 2016. Msr- vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE con- ference computer vision and pattern recognition, pages 5288 5296.
T <- [n(T) P P-PU((V,TT)
end for end for D < Ø for (V,T+) in P do
Kashu Yamazaki, Khoa Vo, Sang Truong, Bhiksha Raj, and Ngan Le. 2022. Vltint: Visual-linguisti wransformerrin--rranttrnnee for coherent video para- graph captioning. arXiv preprint arXiv:2211.15103
b set of data pairs
for fn in T do
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jian- feng Gao. 2021. Vinvl: Revisiting visual representa- tions in vision-language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588.
T + [n(T+)
D - DU ((VT(,
end for end for end function return D
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- herger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Confer- ence on Learning Representations.
44,820 and 5,180 for ActivityNet and 18,029 and 1,971 for YouCook2. For CLIP finetuning, we start with the pretrained ViT-B/16 CLIP model. We sam- ple three frames from each video clip uniformly. We set the margin M in Eq (3) to 5.0 and the loss weight À in Eq (4) 0.1. We finetune the projec- tion layers of the CLIP model for three epochs with batch size of 256 and learning rate of 5e 5e-5. 5. Dur- ing score calculation, we set the balance factor a in Eq (5) to 0.75, favoring fine-grained scores more than coarse-grained ones. Regarding the complex- ity cost, we finetune CLIP 3 epochs, which cost 4-6 hours on single 2080Ti GPU card. We will keep the above settings unless otherwise stated.
Luowei Zhou, Chenliang Xu, and Jason J Corso. 2018a. Towards automatic learning of procedures from web instructional videos. In Second AAAI Confer- ence on Artificial Intelligence.
Luowei Zhou, Yingbo Zhou, Jason Corso, Richard Socher, and Caiming Xiong. 2018b. End-to-end dense video captioning with masked transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8739-8744
A Annotation Protocol and examples
The detailed annotation instructions and protocol provided to the annotators are shown in Table 11. Two annotation examples are provided in Table 12.
C.2 Extra ablation test
B Training Dataset Generation
We explore the impact of the value in Eq (4). The results are shown in Table 13. Note that when À = 0, we only use Locarse to finetune the CLIP model. From the table, FC performs better when choosing À between 0.3). It makes the CLIP model make use of both Lenarse and fjnne- Considering FactVC is based on the image-text pretrained model CLIPPRadford et al., 2021), we want to explore the impact of using different CLIP models. The results are shown in Table 14. We
A detailed description of the data generation pro- cess shown in Algorithm 1.
Experiments
C.1 Implementation details
We use the training split of ActivityNet Captions and YouCook2 to construct our training and vali- dation set. The training and validation set size are

---

Annotation Instructions
About the task:
Our task is video captioning, which uses Al models to generate text descriptions about the video content automatically. Currently, Al models often generate descriptions with factual errors that contradict the video or describe something appearing in the video. Your task annotate the factual consistency between Al-generated captions and video content.
About the annotation method:
You need annotate 200(100) videos in total. Each video contains six paragraph captions from different Al models, and each paragraph contains several sentences. You need to watch each video and then label the six captions. For each sentence, you need to judge whether is fact-consistent with the video content. If the sentence is fact-inconsistent, mark the words/phrases with factual errors. For the whole paragraph, you need to give a factuality Likert scale(1-5, the higher, the better):
5: There are no obvious factual errors.
4: There are few minor factual errors. Most parts are fact-consistent,
3: There are factual errors, but the fact-consistent contents more.
2: There are more factual errors, and the fact-inconsistent contents are more.
There are a lot severe factual errors. It can hardly describe the video content.
Other tips:
AI models sometimes generate poor-quality descriptions, which may affect your annotation. If the sentence has minor grammar errors, you need label it according to the corrected sentence. Otherwise, you need to label it as fact-inconsistent. There is special word "UNK" (unknown word) which you should label as inconsistent.
About the relationship between multiple sentences in a paragraph. The sentences are not in strict time order. The first sentence may describe the second half the video, while the second may describe the first half. As a result, you should annotate each sentence separately and not consider the relation between multiple sentences.
About the completeness of captions. You should only focus on the captions' correctness and not consider whether the caption describes the video completely. For example, caption only describes part of the video without factual errors, while caption 2 describes most video content with factual errors. The factuality of caption is better than caption 2.
About the commonsense. You can use commonsense during annotation. For example, the video content is "a person laying on the bed with eyes closed", then the caption "a person is sleeping on the bed" is correct.
About the annotation of phrases/words. You should mark as few words as possible there are factual errors. For example, if the video content about "A person lays on the bed", and the caption is "A person sits on the bed", you should mark "sits". If the video content is about "A person lays on the ground", and the caption is "A person sits on the bed",, you should mark the whole phrase "sits on the bed".
The order of models shuffled. Do not assume the first caption comes from model 1, and the second caption comes from model 2, etc. After completing video annotation, you should back and check it and ensure your standard is consistent.
We provide you with several annotation examples. Please read them before starting your annotation.
This is very helpful for understanding the annotation method.
Table 11: The detailed annotation instructions and protocol that we provided to the annotators.

---

Video content:
Video content:
A man is seen speaking the camera and leads into him holding up pair of tools. The man then begins ironing the shirt while speaking to the camera. He continues to iron the iron and ends by showing off the finished product.
Add chickpeas lemon juice and lemon juice bowl.
Add flour salt pepper and a spoon of chicken breast and mix.
Add chopped tomatoes chopped spring onions and little salt and pepper.
Paragraph score: 2
Toss and mix everything together.
Paragraph score: 2
A woman is seen ironing a pair of pants on an ironing board while speaking to the camera. She continues ironing the pants and ends by show- ing off the shirt.
Pour macaroni and milk on the pasta.
Add pasta sugar salt pepper and vinegar to the salad.
Paragraph score: 2
Add salt and pepper and mix.
Mix the salad.
A woman is ironing shirt ironing board.
Paragraph score: 4
She shows off pair of pants.
She then irons the shirt on the ironing board. Paragraph score: 4
Pour boiled macaroni and boiled macaroni in bowl.
Add some mayonnaise and blend until smooth.
A woman is seen speaking to the camera and leads into large iron of large iron.
Add diced celery and minced garlic to bowl. Mix everything together.
The woman then begins ironing the shirt and irons the iron.
Paragraph score: 4
The woman continues to iron the iron and shows off the iron.
Add cup of chopped green onions and 1 cup of chopped green onions.
Paragraph score: 3
Plate the meat with the sauce and bread crumbs.
Add diced onion celery celery and mint the food processor.
She then shows the iron the iron and continues to use the iron.
Mix the ingredients in the bowl.
She then irons the ironing the shirt and begins ironing the pants.
Paragraph score: 2
A woman is standing kitchen talking the camera.
Add pasta bowl. Mix mayonnaise mayonnaise mayonnaise salt and pepper.
Paragraph score: 3
Add the cabbage celery and red bell pepper to the cabbage.
Toss the salad. Paragraph score: 4
Table 12: Annotation examples. For each example, we show the video content, five paragraph captions, and the paragraph factuality scores. The phrases/words that are not factual are marked red.

---

FactVC
Video as ref
Text as ref
Video & Text as ref
À value
Para
Sent
Word
Para
Sent
Word
Para
Sent
Word
= 0.0
0.421
0.343
0.448
0.508
0.435
0,490
0.537
0.455
0.528
= 0.1
0.444
0.363
0.460
0.515
0.441
0.502
0.547
0.465
0.541
À = 0.2
0.438
0.355
0.462
0.514
0.443
0.499
0.543
0.461
0.536
À = 0.3
0.436
0.353
0.463
0.514
0.443
0.500
0,541
0.460
0.536
0.5
0.427
0.346
0.459
0.513
0.442
0.500
0.537
0.457
0.534
1.0
0.402
0.325
0.439
0.509
0.439
0.499
0.525
0.446
0.527
Table 13: Pearson correlation between FactVC(using different À value) and human factuality annotation ActivityNet-Fan Fact. We sample one frame from each video clip in this experiment. The best performance in each column is marked in bold.
Video as ref
Text as ref
Video & Text as ref
CLIP Model
Size
Para
Sent
Word
Para
Sent
Word
Para
Sent
Word
RN50
102M
0.317
0.265
0.357
0.478
0.412
0.480
0.488
0.418
0.497
RN101
120M
0.289
0.239
0.316
0.486
0.410
0.491
0,490
0.412
0.500
RN50x4
178M
0.282
0.230
0.328
0.478
0.403
0.497
0,480
0.403
0.507
RN50x16
291M
0,296
0.230
0.333
0.503
0.435
0.510
0.502
0.428
0.517
RN50x64
623M
0.276
0.216
0.320
0.470
0.401
0.466
0.467
0.394
0.474
ViT-B/32
151M
0.321
0.253
0.369
0.472
0.396
0.466
0.483
0.402
0.489
VIT-B/16
150M
0.349
0.281
0.388
0.497
0.425
0.483
0.512
0.433
0.510
ViT-L/14
428M
0.332
0.254
0.356
0.456
0.401
0.442
0.469
0.404
0.462
ViT-L/14-336px
428M
0.340
0.268
0.364
0.454
0.400
0.441
0.468
0.405
0.463
Table 14: Pearson correlation between FactVC (using different CLIP models, no finetuning) and human factuality annotation on ActivityNet-Facc We show each model's size (number of parameters). The best ResNet(RN) CLIP model and Vision- (ransformer(ViI) CLIP model are marked in bold.
test VactVCC performance on ActivityNet-Facc with different CLIP models without finetuning. The table shows that among ResNet-based CLIP mod- cls, RN50 and R650x16 perform best; among ViT- based CLIP models, ViT-B/16 performs best. This leads us the conclusion that larger CLIP model does not necessarily perform better on factuality evaluation. As a result, we use the relatively small ViT-B/16 CLIP model this work.
Table 15. We can see that when just using video as the reference, FactVC is better than previous metrics. When using both video and text as the reference, VactVC achieves the highest accuracy of 94.3%.
Metric
Acc(%)
Metric
Acc(%)
BLEU1
60.1
EMScore
89.5
BLEU4
66.1
PAC-S
90.1
C.3 Experiments on FetivItyNet-FOIL
Rouge-L.
56.7
FactVC
91.0
METEOR
72.9
EMScore*
92.4
Shi et al. (2022) introduced the ActivityNet-FOIL dataset by automatically injecting foil visual con- cepts into the original captions from ActivityNet Captions ac-test split. It contains 1,900 correct-foil paragraph pairs, and least one sentence in the foil paragraph contains foil visual concept. This experiment uses different metrics to evaluate the correct-foil paragraph pairs and compute the pair- wise ranking accuracy. The results are shown in
CIDEr
77.9
PAC-S*
93.5
BERTScore
86.7
FactVC"
94.3
Table 15: FOIL dataset. EMScore, PAC-S and FactVC use video as the reference. EMScore". PAC-S* and FactVC* use both video and text the reference.
Pairwise ranking accuracy on ActivityNet-

---

Video 1:
Generated caption:
A person skiing down a hill of snow. They go over a hill of snow. They continue skiing down the hill together.
Reference captions:
A group of people are on a snowy mountain top. They are skiing down the numerous hills together. We see them flip and turn sharply in the driven snow. A man is crouched down in the snow looking at the camera. He is then seen skiing through the snow.
He is also seen riding the lifts before skiing again.
Paragraph: 1.0
Sentence: 1.0
Word: 1.0
Bleu2: 0.258
METEOR: 0.320
CIDEr: 0.238
BERTScore: 0.610
EMScore(V): 0.649
EMScore(VT): 0.805
FactVC(V): 0.838
FactVC(VT): 0.905
Video 2:
Generated caption:
He does a gymnastics routine on the bars. He does a gymnastics routine on the bars. He dismounts and lands on the bars.
Reference captions:
A man is seen standing before a set of uneven bars and begins inching himself forward. He raises his legs up when he stops and continues inches forward. He moves down all the way to end and jumps off into the mats in the end.
A man stands on front the parallel bars holding it. The man starts to advance holding his hands. The man stops in the middle of the parallel bars, raise his legs and after continues advancing. Then, the man stops the end of the bars, again he raises his legs, then exercises up and down. Next, the man jumps on the mat.
Paragraph: 0.75
Sentence: 0.667
Word: 0.957
Bleu2: 0.144
METEOR: 0.082
CIDEr: 0.0
BERTScore: 0.243
EMScore(V): 0.613
EMScorc(VT)): 0.582
FactVC(V): 0.711 FactVC(VT); 0.671
Table 16: Video captioning evaluation examples. For each example, we show the three-level factuality annotation scores and different metric scores. All scores are scaled in j0, 1]. For EMScore and FactVC, "V" means using video as reference and "VT" means using video and text as the reference. The factual emrors are marked in red in the generated captions.

---

Video 3:
Generated caption:
A young boy is seen sitting behind sink with woman standing behind him. The boy then begins washing dishes while the boy watches from the side. The boy continues to brush his face and ends by turning off the camera.
Reference captions:
A toddler washes dishes in sink while stand on chair. The boy washes cup, sip cup and dish.
After, the boy jumps on the chair and then takes the dish again.
A small child is seen standing before sink washing dishes. He wipes around the sink and continues washing dishes. He puts the clean dishes next to him.
Paragraph: 0.25
Sentence: 0.0
Word: 0.643
Bleu2: 0.379
METEOR: 0.448
CIDEr: 0.121
BERTScore: 0.724
EMScore V): 0.723
EMScore(VTT) 0.630
FactVC(V): 0.497
FactVC(VT): 0.522
Video 4:
Generated caption:
A man is seen standing on track with stick. The man then begins playing with another person on rope. The man continues playing and walks away.
Reference captions:
A woman is standing a picnic table outside an RV. She using water from a dispenser, pouring it onto material. She cleans the pink item with the water.
A lady is outside wringing cloth on a bench. The lady places the pink cloth down on the bench. The lady removes green hose from the brown bench. The lady press the cloth down on the bench and water drains.
Paragraph: 0.0 Sentence: 0.0
Word: 0.310
Bleu2: 0.180
METEOR: 0.072
CIDEr: 0.008
BERTScore: 0.373
EMScore(V) (V): 0.152
EMScore(VT)): 0. 129
FactVC(V): 0.033 FactVC(VT): 0.072
Table 17: Continued with Table 16. Two more video captioning evaluation examples.

---

D CC Qualitative Analysis
We show several video captioning evaluation exam- ples in Table 16 and 17. The annotation scores and metric scores are scaled in [o, 1] for comparison. In video 1, the generated caption has no obvious factual errors. BERTScore, EMScore, and CC assign relatively high scores, while VactVC shows the most confidence. In video 2, the generated cap- tion has minor factual error, but has a poor over- lap with the references. All text-reference-bass metrics give low scores, while FactVC gives a more reasonable factual score. In video 3, the generated caption has many different kinds of factual errors. However, probably because of the semantic over- lap, BERTScore and EMScore give it high scores. Our FactVC gives relatively low score. In video the generated caption full of severe factual errors, and Fa correctly gives a very low score. The examples show that our FactVC metric performs best measuring the factuality of video captions.


Reviews:
[{"review_id": "AChdyzvwVg", "reviewer": "Reviewer_1XK7", "summary": "", "questions": "1. While FactVC demonstrates better correlations in factuality, it is important to consider other evaluation standards such as accuracy, naturalness, and so on. In other words, can we solely rely on FactVC for evaluating the quality of captions?\n2. In Table 7, why does FactVC (L_fine) exhibit a significant drop compared to L_coarse? (L433)\n3. The paper states, \"Similar to EMScore, FactVC can utilize video V, human-written caption T*, or both (V, T*) as reference\" (L394-L396). However, the equations provided only demonstrate the usage of video V. How can we incorporate the usage of T* and (V, T*)?\n4. In Section 7.3, the authors employ an average score and rank the models. However, would it be more appropriate to assign a separate rank to each caption and then calculate a rank correlation (such as Spearman's rho) across all captions?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your review! Below are our responses to your questions.**\n\n1. **Can we solely rely on FactVC for evaluating the quality of captions?**\n\n   FactVC is designed and trained to evaluate the factuality of video captions. We did not test it for other evaluation standards. We recommend using FactVC and other metrics together to evaluate the overall quality of captions. \n\n2. **Why does FactVC (L_fine) exhibit a significant drop compared to L_coarse in Table 7?**\n\n   L_coarse is similar to the original CLIP loss function that trains video-text matching. L_fine trains the model to assign a higher score to fact-consistent text. Only using L_fine to finetune the CLIP model will destroy the video-text matching ability, causing a performance drop.\n\n3. **How to use T\\* and (V, T\\*) to compute FactVC score?**\n\n   When using human-written caption T\\*, we replace the video frame $f$ in Eq (6) and Eq (7) with the caption word $w$ and use the CLIP text encoder to compute FactVC(T\\*). FactVC(V, T\\*) is the average of FactVC(V) and FactVC(T\\*). We will improve the equations in the revised version.\n\n4. **Would it be more appropriate to conduct a caption-level ranking experiment?**\n\n   Section 7.3 conducts a model-level (system-level) ranking experiment. We design this experiment mainly because researchers often use system-level metric scores to compare different models. Caption-level ranking is also a good supplemental experiment.\n\n5. **Are there other types of errors other than factual errors in video caption?**\n\n   Video caption errors contain content-unrelated errors (e.g., grammar errors) and content-related errors (e.g., factual errors, content-missing errors). Factual errors are the main part of content-related errors. Besides, many video captions fall into the \"right but not good\" category. We treat these captions as fact-consistent in this study.\n\n6. **Are FactVC only involves fine-tuning CLIP from image to video? Why not using video-text models like VideoCLIP?**\n\n   The training of FactVC is not simply fine-tuning CLIP from image to video. We use text augmentation skills to construct the training set and leverage different loss functions. Besides, we design a scoring method to make it more appropriate for factuality evaluation. The reason for using CLIP is that it can be used to evaluate video and image captions (Section 7.5). VideoCLIP may be suitable for video caption and needs experimental verification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Hallucination poses as a fundamental issue in LLM (or the entire deep learning-based field, broadly speaking). The paper specifically addresses factual errors in video captions, which, according to their annotations, prove to be a significant problem (L73).\n2. FactVC is a more effective metric for detecting factual errors in captions.\n3. The paper is well-organized and provides a clear explanation of the topic.", "reasons_to_reject": "1. The definition of factual errors or hallucinations provided in the paper (L40-L42) appears to be vague and general. It seems that most caption errors can be classified as hallucinations since content words form the main part of the sentences. However, it is unclear what other types of errors exist if they are not factual. Grammar errors come to mind, but it is unlikely that they constitute the primary type of error in generated captions. Therefore, the high number of hallucinations raises some suspicion.\n2. It seems that FactVC only involves fine-tuning CLIP from image to video, which, if true, may not be a significant contribution. Additionally, instead of using CLIP for videos, why not utilize VideoCLIP [1], which is specifically designed for zero-shot video-text understanding?\n\n[1] \"VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding\"", "questions_to_authors": "1. While FactVC demonstrates better correlations in factuality, it is important to consider other evaluation standards such as accuracy, naturalness, and so on. In other words, can we solely rely on FactVC for evaluating the quality of captions?\n2. In Table 7, why does FactVC (L_fine) exhibit a significant drop compared to L_coarse? (L433)\n3. The paper states, \"Similar to EMScore, FactVC can utilize video V, human-written caption T*, or both (V, T*) as reference\" (L394-L396). However, the equations provided only demonstrate the usage of video V. How can we incorporate the usage of T* and (V, T*)?\n4. In Section 7.3, the authors employ an average score and rank the models. However, would it be more appropriate to assign a separate rank to each caption and then calculate a rank correlation (such as Spearman's rho) across all captions?", "paper_topic_and_main_contributions": "The paper constructs a dataset to analyze and measure factual errors in video captioning. Then, the authors propose FactVC, a weakly-supervised, model-based factuality metric. This metric is developed by finetuning the CLIP model, which is an image-based model. Experimental results demonstrate that FactVC achieves a higher correlation compared to other metrics.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "1. L138: \"even do not requiring\" -> \"even without requiring\"\n2. Please ensure consistent while space before the left parenthesis throughout the text.", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "7Qu1LVjrl5", "reviewer": "Reviewer_joFz", "summary": "", "questions": "1.\tIs there any reason to set the number of 200 and 100 for two newly human-annotated factuality datasets?\n2.\tDo the authors think about the possible solutions to address the problem of factual errors in video captioning?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your review! Below are our responses to your questions.**\n\n1. **Is there any reason to set the number of 200 and 100 for two newly human-annotated factuality datasets? Are the datasets relatively small?**\n\n   In this study, we use six models to generate paragraph-level captions for each video. Considering the number of models and caption length, annotation is time-consuming. We collect three annotations for every caption, further increasing the labor cost. As a result, we set the number of 200 and 100 for two datasets so that the scales of the two datasets are comparable, containing 3834 and 4080 sentences, respectively. Compared to factuality annotation in Text Summarization [1, 2], the scale of our datasets is comparable or larger.\n\n2. **Possible solutions to address the problem of factual errors in video captioning?**\n\n   Reducing the factual errors in video captioning is challenging. Although the solution is beyond the topic of this paper, we can share some possible ways. First, we can adjust the training target of the caption model. For example, we can use contrastive learning or reinforcement learning skills to punish the generation of factual errors. Second, we can extract additional information from video content or external knowledge base and restrict the generated caption consistent with the information. We hope researchers propose more effective methods in the future.\n\n3. **Are the used models of video paragraph captioning relatively old?**\n\n   We choose the representative models in the last few years. We did not choose VLCap[3] because we have chosen VLTinT[4]. They are done by the same authors and VLTinT[4] is an improved version of VLCap[3].\n\n4. **Regarding the caption of Table 3, is there a mistake?**\n\n   Yes, we made a mistake in writing here. The human-written caption should be represented as T, as you pointed out. We will fix it in the revised version.\n\n**References**\n\n[1] On Faithfulness and Factuality in Abstractive Summarization. ACL 2020\n\n[2] Evaluating the Factual Consistency of Abstractive Text Summarization. EMNLP 2020\n\n[3] VLCap: Vision-Language with Contrastive Learning for Coherent Video Paragraph Captioning. ICIP 2022\n\n[4] VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioning. AAAI 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis paper is well-motivated, which aims to provide a thorough factuality evaluation for video captioning.\n2.\tThis paper provides two human-annotated factuality datasets and a factuality metric for the task of video captioning, which could benefit the subsequent exploring of this task.\n3.\tThe experiments are thorough.\n4.\tThe writing is good and easy to follow.\n", "reasons_to_reject": "1.\tThe scale of two human-annotated factuality datasets is relatively small, i.e., 200 videos and 100 videos.  \n2.\tThe used models of video paragraph captioning are relatively old, i.e., there is only one model proposed after 2022. The authors might considering test more recently designed models, such as VLCap [1].\n3.\tIn the caption of Table 3: \"ref\" means the metric reference is human written caption (V), input video (V), or both (VT). I think that human written caption should be T.\n", "questions_to_authors": "1.\tIs there any reason to set the number of 200 and 100 for two newly human-annotated factuality datasets?\n2.\tDo the authors think about the possible solutions to address the problem of factual errors in video captioning?\n", "paper_topic_and_main_contributions": "This paper first conducts the first human evaluation of the factuality in video captioning and annotate two factuality datasets. The authors find that 56% of the current model-generated sentences have factual errors. To this end, they further propose a new factuality metric FactVC for factuality evaluation of video captioning.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] vlcap: vision-language with contrastive learning for coherent video paragraph captioning. ICIP 2022.", "Typos_Grammar_Style_and_Presentation_Improvements": "None."}, "extra_text": {}}, {"review_id": "N1yNAUDBW8", "reviewer": "Reviewer_nfxG", "summary": "", "questions": "(1) Have you considered using the original captions, without any positive transformation, as the positive samples?\n\n(2) It is surprising that FactVC aligns exactly with the human ranking annotation in Table 8, as previous generation evaluation papers [2,3] have shown that ranking through many candidates (e.g., 6 in this table) can be very challenging. In such cases, model-generated texts should have equivalent qualities if they are all state-of-the-art models. I am wondering if this annotation was still conducted by the three annotators, and what is the inter-agreement of this analysis. If so, it might be a bit unconvincing for me. It would be ideal if the authors could provide some examples that include all the model-generated captions and human annotations.\n\n(3) Have the authors considered pair-wise ranking for Table 8? Some papers [3,4] have shown that pair-wise ranking is more stable than ranking through many candidates at once, which might provide a more reliable evaluation for the model-generated captions.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your review! Below are our responses to your questions.**\n\n1. **Have you considered using the original captions, without any positive transformation, as the positive samples?**\n\n   Yes. The direct way to collect positive samples is by using the original captions. However, the original captions' scale is small (36k for ActivityNet, 10k for YouCook2). In the initial research, we tried different positive and negative transformations. Positive transformations can enhance the diversity of samples and improve performance.\n\n2. **Regarding the model ranking experiment in Table 8. How can the metric rank the models correctly? How to get the human annotation for model ranking?**\n\n   The chosen caption models are representative models in the last few years. Because they were proposed at different times (for now, the SOTA model is VLTinT), their performances are different. So a good metric can rank the models correctly. \n\n   We did not perform additional human annotation for the model ranking experiment. We directly use the human annotation in Section 3.3. We compute the average scores of captions from different models and use them to rank the models. All the annotators do not know which caption comes from which model.\n\n3. **Have the authors considered pair-wise ranking for Table 8?**\n\n   When hiring annotators to compare different models, pairwise ranking can provide a more reliable evaluation. However, in our experiment, we did not hire annotators to rank the models. We use the annotation score in Section 3.3 and compute the average score to rank the models. A similar experiment is done in [1]\n\n4. **Regarding the human annotation settings, why use relatively few annotators? Are the datasets relatively small?**\n\n   According to the previous work[2], factuality annotation through the crowdsourcing platform is not a good choice. We hire only three annotators so that we can train them well and check the annotations conveniently. Each model-generated caption is judged by three annotators, and we use the majority as the final annotation. Compared to factuality annotation in Text Summarization [2,3], the scale of our datasets is comparable or larger.\n\n5. **Regarding the missing references, what are the distinctions between our text transformations and the transformations in [4]?**\n\n   Thank you for supplementing the references! Because our focus is video captioning (plus page limit), some work of other related tasks is not included. We will fully check the references and add the related work. \n\n   As for the distinctions between our work and [4], they are different in (1) we use paraphrasing and simplification as positive transformations while [4] use original data as positive samples; (2) our negative transformations are based on our analysis in Section 4.2, covering most factual error types while the negative transformations in [4] are designed for story generation (e.g., introducing grammar and logical errors.)\n\n**References**\n\n[1] EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching. CVPR 2022\n\n[2] Evaluating the Factual Consistency of Abstractive Text Summarization. EMNLP 2020\n\n[3] On Faithfulness and Factuality in Abstractive Summarization. ACL 2020\n\n[4] UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation, EMNLP 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper is very well-written and easy to follow. The organization of the manuscript effectively follows their contributions.\n\n(2) This paper's contributions, including both the dataset and the metric, will significantly benefit the development of video captioning.\n\n(3) The authors conduct extensive experiments to demonstrate the improvements of this new evaluation method.", "reasons_to_reject": "(1) Some relevant references are missing (see missing references section).\n\n(2) The human analyses were conducted with relatively few annotators compared to other metric or evaluation papers, which could potentially lead to biased results due to the limited number of annotations. While Section 4 provides some fascinating analyses, the small number of annotations might make the conclusions less convincing. However, I personally find the analysis in Section 4.2 particularly intriguing, as it highlights the high error rates in recognizing objects and actions, indicating that current models still lack the ability to capture these crucial factors.\n\n(3) The approach of creating positive and negative samples in this paper is similar to a previous study [1]. It would be beneficial to cite and discuss the distinctions between your work and their method in the paper.", "questions_to_authors": "(1) Have you considered using the original captions, without any positive transformation, as the positive samples?\n\n(2) It is surprising that FactVC aligns exactly with the human ranking annotation in Table 8, as previous generation evaluation papers [2,3] have shown that ranking through many candidates (e.g., 6 in this table) can be very challenging. In such cases, model-generated texts should have equivalent qualities if they are all state-of-the-art models. I am wondering if this annotation was still conducted by the three annotators, and what is the inter-agreement of this analysis. If so, it might be a bit unconvincing for me. It would be ideal if the authors could provide some examples that include all the model-generated captions and human annotations.\n\n(3) Have the authors considered pair-wise ranking for Table 8? Some papers [3,4] have shown that pair-wise ranking is more stable than ranking through many candidates at once, which might provide a more reliable evaluation for the model-generated captions.\n", "paper_topic_and_main_contributions": "This paper investigates factual error issues in video captioning. The authors begin by conducting human analysis and annotation to identify factual problems. They then propose a weakly-supervised factuality metric, FactVC, to address these issues. Extensive experiments show that FactVC outperforms other baselines, demonstrating its superiority.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Missing space after 1) in line 334.\n"}, "extra_text": {"Missing_References": "In the related work section, the paper should mention some relevant vision-based generation tasks and evaluation methods, such as (a) comprehensive analysis on multimodal generation [5], (b) a challenging image-based generation task: Visual storytelling [6,7,8,2,4], (c) reference-free metric for image-based generation [3], and (d) some other reference-free text-based metrics: [9].\n\n-----\n[1] UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation, EMNLP 2020.\n\n[2] No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling, ACL 2018.\n\n[3] Learning to Rank Visual Stories from Human Ranking Data, ACL 2022.\n\n[4] Storytelling from an image stream using scene graphs, AAAI 2020.\n\n[5] Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations, EMNLP 2020.\n\n[6] Visual storytelling, NAACL 2016.\n\n[7] Show, Reward and Tell: Automatic Generation of Narrative Paragraph from Photo Stream by Adversarial Training, AAAI 2018.\n\n[8] Plot and Rework: Modeling Storylines for Visual Storytelling, ACL 2021.\n\n[9] BLEURT: Learning Robust Metrics for Text Generation, ACL 2020."}}]

Author rebuttal:

Meta-review: This paper studies an interesting topic of hallucination in video captioning models. All reviewers positively rated the work and agreed that contributions of the paper, namely two datasets and a novel metric, are significant and exciting to the field. In addition, all reviewers also commented high on the quality of the paper.
Decision: Accept-Main