A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video
Authors: Keito Kudo
Haruki Nagasawa
Jun Suzuki
Nobuyuki Shimizu
Venue: EMNLP 2023 Main (2023)
Keywords: Video Summarization
Multimodality
TL;DR: Proposal of practical and challenging multimodal video summarization task.

Abstract: This paper proposes a practical multimodal video summarization task setting and a dataset to train and evaluate the task. The target task involves summarizing a given video into a predefined number of keyframe-caption pairs and displaying them in a listable format to grasp the video content quickly. This task aims to extract crucial scenes from the video in the form of images (keyframes) and generate corresponding captions explaining each keyframe's situation. This task is useful as a practical application and presents a highly challenging problem worthy of study. Specifically, achieving simultaneous optimization of the keyframe selection performance and caption quality necessitates careful consideration of the mutual dependence on both preceding and subsequent keyframes and captions. To facilitate subsequent research in this field, we also construct a dataset by expanding upon existing datasets and propose an evaluation framework.
Furthermore, we develop two baseline systems and report their respective performance.

A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Ceyyfrme-Caapiie Pairs from Video
Keito Kudo' Haruki Nagasawa' Jun Suzuki' Nobuyuki Shimizu"
'Tohoku University "LY Corporation
(keito to. kudo. q4, haruki.nagasan s8 jddc. tohoku. ac. jun. wun.suzukketoookuuaac
nobushimelycopp.com
Abstract
poirs Keytrame Capilea
"A logo slown."
This paper proposes a practical multimodal video summarization task setting and dataset train and evaluate the task. The target task involves summarizing a given video into pre- defined number of keyframe-caption pairs and displaying them in listable format grasp the video content quickly. This task aims to extract crucial scenes from the video in the form of im- ages (keyframes) and generate corresponding captions explaining each keyframe's 's situation. This task useful a practical application and presents a highly challenging problem worthy study. Specifically, achieving simultancous optimization of the keyframe selection perfor- mance and caption quality necessitates care- ful consideration of the mutual dependence on both preceding and subsequent keyframes and captions. To facilitate subsequent research in this field, also construct dataset by expand- ing upon existing datasets and propose an eval- uation framework. Furthermore, we develop baseline systems and report their respective performance.
number
"Aa athlen is mady start.
Multi- lebbom
Sp stum
jompligg
astudiuns
Video
janqer conglene long Janp and thon nuss
flag"
Figure 1: Overview of the Multi- VidSum task. We assume the number of the output keyframe-caption pairs is given (e.g.. 4) depending summary display space and user preference.
has shifted to multimodal video summarization, which requires joint outputs of both image and text pairs, e.g., video captioning (Sun et al., 2022; Koh et al., 2023), dense video captioning (Krishna et al., 2017; Zhou et al., 2018b; Yang et al., 2023b), and video storytelling (Li et al., 2020).
Multimodal video summarization tasks are high-demand practical application and challeng- ing research task. Thus, this paper, we propose a multimodal video summarization task, which we re- fer to Vidti- task. This task considers more practical situations and challenging perspectives. The Vidli- task requires us to summarize a given video into predefined number of keyframe- caption pairs and display them in a listable format to grasp the video content efficiently.
1 Introduction
The popularity of video sharing platforms has in- creased, which has resulted in a substantial increase in daily video-watching by individuals. As re- sult, there is increasing interest in practical video summarization systems that can comprehend video content efficiently, and many previous studies have proposed different automatic video summarization methods to address this demand.
We first formulate the task definition and de- velop evaluation metrics to assess the task appro- priately. Then, we construct training and evaluation datasets to process this task. In addition, we de- velop two baseline systems using the constructed data, thereby providing effective benchmark for future research and offering insights into handling the target task based on performance comparisons.
Most early attempts only considered video and image data, and these methods were developed in the vision and image processing community (Apos- tolidis et al., 2021), e.g., keyframe detection (Wolf, 1996; Kulhare et al., 2016; Yan et al., 2018; Khu- rana and Deshpande, 2023) and video storyboard- ing (Zhang et al., 2). However, the recent trend
The challenge of this task lies in the simultane- ous exccution of keyframe selection and caption generation while maintaining sufficient consider-
'This datasct and our systems (code) are publicly available at https: wwwb. com/ kei tokudo/ /Multt -i-Vi.uum.

---

ation of their relative importance. Specifically, to generate appropriate captions, we attempt to select keyframes that are well-aligned with the desired content (and vice versa). Thus, the system must consider these interdependencies and make opti- mal choices for keyframes and captions, which is challenging. Therefore, the proposed task provides a practical setting that bridges the gap with real- world applications.
Dense video captioning In the dense video cap- tioning task, events in video segments are detected and captions are provided. For example, a previ- ous study Krishna et al. (2017) introduced the first dataset for this task, i.e., the ActivityNet Captions, and proposed corresponding baseline system. We extend this task to the proposed Aulti-VidSum task. where keyframes serve compressed segments. In dataset
addition, we create distinct by adding keyframe annotations the existing Ac- tivityNet Captions. The Vullti-VdSSum task poses a more significant challenge as requires a precise selection N cnaframe-captiion pairs as represen- tative summaries.
Our primary contributions are summarized as follows: (1) We propose the challenging and prac- tical Multi-VidSum task. (2) We generate a dataset for this task to facilitate effective training and eval- uation of relevant systems. (3) We develop new evaluation metric for this task. (4) We develop and analyze baseline systems, thereby providing benchmark and insights for future research.
Visual storytelling Similar to the Multi-Vine VidSum task, visual storytelling (Huang et al., 2016; Wang et al., 2018c) has been proposed previously. The VisStory task involves generating story text that is relevant to the input images. Although the concept of the VisStory task is similar to that of the Multi- VidSum task, the Multi- VidSum takes the vidco data as the input; thus, it is a more challenging setting than the visual storytelling task. In addition, this dataset to pretrain baseline model (see Section 5.3.1 for details).
2
Related Work
Here, we introduce several related studies. Note that other related works (e.g., multinodal gener- ation and keyframe detection) are listed in Ap- pendix A.
Multimodal summarization Multimodal sum- marization studies have explored incorporating im- ages in document summarization (Zhu et al., 2018) 8). Similarly, in terms of video summarization tasks, previous studies have proposed methods gener- ate summary results using both images and text (Fu et al., 2021; He et al., 2023). However, in these methods, the text output by the model is extracted from the imput automatic speech recognition results or documents; thus, such methods do not gener- ate summaries by considering visual information explicitly. In our research, our goal is to generate summaries by considering visual information and selecting appropriate frames simultaneously.
Proposed Multi- VidSum task
In this section, we define the proposed Multi- VidSum task. The multimodal attempts to present users with a set of key teyframes from the input video with relevant captions.
task
3.1 Summary Length
First, we discuss the summary length, which refers to the number of keyframes and their caption pairs to summarize the video data. Determining the ap- propriate summary length is heavily dependent on factors, e.g., video length and the number of scenes, However, other considerations, e.g., the space avail- able to display the summary, layout constraints, and user preferences to comprehend the summary, also play significant roles. Thus, it is necessary to incor- porate situational information alongside the video to estimate the suitable summary length. However, preparing such situational information is difficult; therefore, in this study, we assume that the appro- priate summary length N is always given as the input together with the video data (similar to pro- viding a preferred summary length from a user in
Video storytelling Video storytelling (Li et al., 2020), which is promising video summariza- tion task, has limitations in terms of its dataset, model, and evaluation criteria. For example, the relevant dataset is small and inadequate for train- ing and evaluation. In addition, this method relies on gold data to derive the number of keyframe- caption pairs, which results in unrealistic task set- tings. Also, keyframe detection was not assessed due to an absence of keyframe annotations. There- fore, in the current study, we constructed a large dataset with corrected keyframe annotations from the open domain to address these limitations.

---

an on-the-fly manner.
M coptions
3.2 Task Definition
atiiee ready tost
Annotated
In this section, we define the proposed Multi- VidSum task. Here, let x - (zi,...,, Er) be a video, where T is the video length, y (y1,..., yN) ) is series of keyframes selected from video x, and yi is the i-th keyframe when viewed from the beginning of the video. Note that the keyframe is a single frame selected from the video. In addition, z = (z1,..., 2N) ) a series of explana- tory text corresponding to the series of keyframes y. Thus, 2 is the caption corresponding to y:- Then, the proposed Multi- VidSum task can be defined as follows:
captian correpanding candidana
Video
3.5
5.1%
3:00
4.2%
Figure 2: Overview of the Multi-VidSum dataset. Each video comprises M (reference) captions, and each cap- tion has K (reference) keyframe candidates.
3.3.2 Evaluation difficulty
The dataset structure for this task (Figure 2) in- cludes vidcos with M z N captions, where each caption has K keyframes. These keyframe-caption pairs are basic building blocks used to construct the video summaries, thereby resulting in multiple valid summarization options for each video. Thus, evaluating the performance of video summariza- tion systems is challenging because the general approach of comparing system outputs single reference is feasible. To address these difficulties, we establish an evaluation framework that accounts for these issues (Section 3.4). As a result, we can create an environment that is conducive to assess- ing the quality and effectiveness of video summa- rization systems.
T: (x + (y: z).
(1)
As shown in Figure 1, the system output can also be interpreted as a sequence of koyframe-caption pairs, i.e., ((11, 21),... (UN,,
3.3 Task Difficulty (Challenge)
There are two primary challenges associated with the proposed task. The first challenge is related to the task's inherent complexity, and the second challenge is related to the difficulty associated with evaluating performance. These challenges are dis- cussed in the following subsection.
3.4 Evaluation Criteria
3.3.1 Task complexity
The following are descriptions of the evaluation cri- teria for each type of output; keyframe and caption.
The greatest challenge involved in solving Multi- VidSum is how we select and generate a set of keyframe-caption pairs simultancously in an effi- cient and effective manner. Specifically, the proce- dures used to select keyframes and generate cap- tions are interdependent. Here, the keyframe in- formation is required to generate appropriate cap- tions, and the caption information indispensable in terms of selecting appropriate keyframes. This implies that determining one without using the other one may induce a suboptimal solution at a high probability. To the best of our knowledge, the interdependence among the different modali- ties (i.e., image and text) is unique and does not appear in cxisting tasks. Thus, we must consider new method solve this property effectively and efficiently when developing related video summa- rization methods.
3.4.1 Evaluation criterion for keyframe detection
Here, we assume that each video has M reference captions, with each caption having K keyframe candidates (see Figure 2). Multiple keyframe candi- dates are used to capture different scenes that align with each caption. We calculate a matching score based on exact matching to evaluate the predicted keyframe list against the reference keyframe list. This involves determining whether the predicted keyframes match the answer keyframes precisely. We define the aligned keyframe matching (AKM) score the maximum matching score over all pos- sible sub-lists of the reference keyframes. However, exact matching (AKMex) can be strict; thus, we also introduce flexible matching score (AKMcos) based on the cosine similarity between the predicted and reference keyframe feature vectors. Note that both
"Similar discussions and solutions have been made relative to the document summarization task (Over and Yen, 2003)).

---

Number of videos
12,009
Average Number key-frames per cuption
14.72
Average Number of captions per video
4.8
Average
per sentence
Table Statistics of Multi-VidSum dataset
Activily Captions
The riding
persum
stop momert and begin to pose for the
catiooing on person wrist while the camera
metrics range from 0-1, where higher values indi- cate better keyframe selection performance."
Ours
girl and bey riding horses for
A pensom billoving somoons's wrist.
pooc.
3.4.2 Evaluation criterion for caption generation
Figure 3: Examples of captions in ActivityNet Captions dataset that mention the cameraman and camera zoom, which are unrelated to the content shown in the given keyframe.
The generated captions are evaluated in terms of two distinct metrics, i.c., METEOR (Lavic and Agarwal, 2007) and BLEURT (Sellam et al., 2020). In this evaluation, the top N keyframes predicted by the model and their corresponding captions based on the highest AKMcos scores are selected. Then, the selected keyframes and captions are used as references to calculate the AKMex, BLEURT, and METEOR values.
With the crowdsourced keyframe annotations, we obtained an average of 14.72 keyframe annotations per caption.
4.2 Reliability Check
We asked trained annotator (rather than the crowd workers) to assess the matching of the annotated keyframes and the corresponding captions in ran- dom sample of 196 videos. The trained annotator found that 3.87% of the keyframe annotations rep- resented annotation errors. However, we consider that this error rate is sufficient; thus, the generated dataset is sufficiently reliable to train and evaluate systems designed for the Multi-VidSumm task.
4 Multi-VidSum Dataset
We constructed a dataset for training and eval- uating the Multi-VidSumi task in machine learn- ing approaches. To impact the community more, constructed dataset for the Multi-VidSum task by expanding ActivityNet Captions (Krishna al., 2017), which commonly used, includ- ing the shared task of Dense- CranseCCptioninng Events in Videos.5 5 We tasked crowd workers to add keyframe annotations that align with the captions in the Ac- Net Captions." 6 Note that the original Activi- tyNet Captions dataset has no keyframe annotation; thus, including the additional annotations to handle the proposed Multi- VidSum task is considered unique contribution to the field.
We also attempted to increase the reliability through automatic annotation filtering. Here, calculated the image features for all frames using the pretrained model (Zhou al., 2018b) and then we eliminated annotated keyframes the distance between the centroid of the image features for all annotated keyframes assigned to a single caption and the image feature of cach annotated keyframe was relatively large. For example, we reduced the variance of the image features for the annotated keyframes from 0.0536-0.0166 In this case, the error rate of the remaining annotated keyframes evaluated by the expert annotator was reduced from 3.87%-1.76%.
4.1 Statistics
The ActivityNet Captions dataset contains approxi- mately 15,000 videos. We excluded some videos to annotate for several reasons, e.g., unavailable videos. Thus, the resultant annotated video be- comes about 12,009. Table shows the statistics of task.?
the dataset generated for the
4.3 Test Set Refinement
RRefer to Appendix B.1 precise definition addi- tional details.
We constructed our dataset using the caption anno- tations provided in the original ActivityNet Cap- tions dataset. However, some of the captions in the original dataset are of low quality; thus, there was concern that an effective and reliable eval- uation of the system's performance would not be possible. For example, it refers to the behavior
*For detailed information about METEOR and BLEURT, please refer Appendix B.2.
/http:// ///ctivciiiiii net.et..ooo ctg//challenges/201//
tasks/ casks/anet_ct ceret_captionnne html
"Refer to Fthics Statement for information regarding
wages paid for this annotation.
"The detailed process is described in Appendix C.

---

Finally, the summarization module selects N pairs as the system's 's final of the cameraman outside of the frame, e.g., the zoom of the camera. In addition, discussed in (Otani al., 2020), this issue also linked to the limited variety of actions present in ActivityNet Captions. To address these issues, we engaged a reliable data creation company with native English speakers to re-annotate the correct captions for subset of the original validation set to create test set with high-quality captions for random sam- ple of 439 videos. Detailed information about the instructions provided the annotators is given in Appendix C.2.8
Surocatiation modaa
Gearladd suplin
Yeagy aepliaueg rocal
Strend Inaeo
Koraas slacsss Fodak
A comparison of the captions before and after the re-amnotation process is shown in Figure 3. Through the re- e--anotation process, qualitatively confirmed reduction in terms of the references to content not observable in the video, e.g., the cameraman. Thus, we believe that we can perform more accurate evaluations by using this newly an- notated test set. For a more detailed analysis of the test set (i.e., action diversity, vocabulary diver- sity and caption information), please refer to Ap- pendix C.4. Note that the newly annotated dataset used the test set in all experiments diseussed in this paper.
Figure 4: System overview model.
Iterative refinement
refinement model.?
5.1.1 System descriptions
Overview Figure 4 shows an overview of the It- erative refinement model, which comprises four modules, i.e., the segment extraction, image cap- tioning, keyframe selection, and summarization modules. The system takes a video as input and produces N keyframes and captions according to the following process.
5 Baseline Systems
Here, we propose two baseline systems to tackle the proposed task. To address the challenges discussed in Section 3.3.1, present two different baseline systems: Iterative refinement model, which iter- atively selects keyframes and generates captions alternately and Simul-determination model that selects and generates all keyframes and captions collectively. Simul-determination model, given the anticipated difficulty of simultaneous selection and generation of all keyframes and captions, we implement method that incorporates pretrained image captioning model. By comparing the out- comes of these two baseline systems, we seek to provide insights researchers interested in under- taking future investigations related tasks.
1.
The segment extraction module divides the video into segments, where segment refers to semantically similar and contiguous times- pan within the video.
2. The image captioning module generates cap- tion for the median frame of each segment. The keyframe selection module selects a frame that best matches the generated caption for each segment.
4. The image captioning module then gener- ates another caption for each selected frame. Note that steps and 4 are repeated / times (iterative refinement. 10) The parameter is fixed at for all settings in this study.
5.1 Iterative refinement model
As our first baseline model, propose Iterative refinement model that divides the task into multiple subtasks and processes them individually and itera- tively using different experts. In the following, we provide a comprehensive overview of the Iterative
output.
In the following, we describe each module in detail.
FFor detailed experimental configurations information on reproducing our experiments, refer to Appendix D. "Refer to Appendix F. for information regarding the im- pact of this approach.
*See Ethics Statement about the wages paid for this work.

---

Keyframe selection module The keyframe se- lection module takes a segment and a caption as Segment extraction module The segment ex- traction module divides the input video into seg- ments. We extract segments first to prevent redun- dant computations to reduce computational costs because many frames are very similar, especially if the time of their appearance is close. We also use the segment information in the final keyframe- caption pair selection phase.
Trandiener Encedor
Transformse Decadir
forn frem
reyy
Leah
Figure 5: Overview our Simul-determinatioon model
We use PySceneDetect (Castellano, 2014) and Masked Transformer Model (MTM) (Zhou al., 2018b) to extract segments. PySceneDetect is video processing tool to detect scene changes, and the MTM includes the functionality to extract seg- ments. Using these tools, we prepare N or more segments by combining the 20 segments extracted by the MTM with those created by PySceneDe- tect." In addition, the MTM calculates score that gauges the quality of each segment." This score is then used in the summarization module.
its input to predict which frames in the segment match the caption. Specifically, this module uses a model that performs binary classification to de- termine whether each frame in the segment is a keyframe or not. The architecture of this model is based on a bidirectional LSTM. Additional details about this model are given in Appendix D.2.
Summarization module The summarization module selects N keyframe-caption pairs from the candidates for each segment. To determine the N key fayframm-caption pairs, we consider the sum of the segment score (computed by the segment ex- traction module) and the caption score (computed by the image captioning module). We apply dy- namic programming technique to select the opti- mal N deyframe-caption pairs while minimizing time overlap among the segments. The selected keyframes and captions then used the final output of the system.
Image captioning module To generate cap- tions for the candidate frames, we use pretrained image captioning models, i.e., Fairseq-image- captioning (Krasser, 2020), ClipCap (Mokady et al., 2021), Clip-Reward (Cho et al., 2022), and InstructBLIP (Dai et al., 2023). We also use Vid2Seq (Yang al., 2023b), which is dense video captioning model, generate captions. 13 Note that Vid2Seq is employed not for the direct generation of summaries but only for generating captions for the selected frames. The notable differ- ence between Vid2Seq and the other image caption- ing models is that Vid2Seq takes the entire video and automatic speech recognition (ASR) result as the input to generate caption, whereas the image captioning models only take images the input. Here, Vid2Seq is used to investigate the effective- ness of audio information and the context of the video for the caption generation process. These models also calculate a score that reflects the qual- ity of the generated captions, and this score is used in the subsequent summarization module. Refer to Appendix F for detailed information about these models and scores.
5.2 Simul-determinatioon model
The Iterative refinement model generates captions and selects keyframes separately, which means that does not consider the context of the caption or keyframe. Thus, we also propose a model that generates captions and keyframes simultaneously to address this limitation.
5.2.1 Overview
This model is based on the transformer (Vaswani et al., 2017) encoder and decoder architecture with several modifications to make it suitable for the tar- get task. Figure 5 shows an overview the model. The encoder takes as input the feature vectors of all frames in the video (obtained by down- down-sampling every 0.5 seconds) as the input, where the image encoder of CLIP (Radford et al., 2021a) is used to create the feature vectors. Then, the decoder produces sequence of N frame indices and their captions, thereby predicting a keyframe and gen- crating a caption N times. Although the keyframe and caption are different modalities, this model
We exclude overly long segments (longer than 75% of the entire video) and reduce the score for segments that overlap with other segments.
12 As for PySceneDetect scores, they are uniform fixed at 1.
15 We used the Vid2Seq model reimplemented in Pytorch
(Yang at al., 2023a).

---

generates and selects them as single sequence. which allows the model to consider the context while generating captions and selecting keyframes, If the input a text token, the decoder receives the embedding corresponding to the token, and if is a frame, the decoder receives the features created by CLIP. We modify the architecture to fit the target task, which is described in the following.
MS COCO (Chen al., 2015) and Visual Story- telling (Huang et al., 2016)). Each training instance is created according to the following procedure.
We select N sets of images and captions from the dataset." 14
2. We randomly divide the input sequence length of the encoder into N spans.
3. We then select a single index at random from each span. Note that these indices are defined the indices of the N keyframes.
5.3 Architecture Changes
Gate mechanism This model follows a repetitive process whereby a keyframe is predicted for each caption, and corresponding text is then generated. This cycle is repeated for total of N iterations. Thus, the model should predict a keyframe when the input is the <bos> token, but text token oth- erwise. To enable this, we introduce gate mech- anism that determines whether to predict a frame index or a text token according to the input token. The formulation of the gate mechanism is given in Appendix G.4.2.
For each span, the input to the encoder comprises the same image features. However, except for the image feature that corresponds to the index of the keyframe (selected in the above procedure), we apply noise to the features. Additional details about this process are given in Appendix G.3
The model goes through two pretraining phases, where it initially uses a pseudo dataset based on MS COCO and then Visual Storytelling. This two- phase pretraining strategy is employed because Vi- sual Storytelling includes stories that connect each image and caption pair, thus, the content of the dataset is similar to video data.
Pointer mechanism In the proposed method, it is necessary to select a keyframe from the input frames in the decoder. However, applying cross- attention only allows the model to use the encoder input information (i.e., frames) indirectly for pre- diction. Thus, we also apply pointer mechanism inspired by CopyNet (Gu et al., 2016). Here, when predicting a keyframe, we calculate the cosine sim- ilarity between the last hidden state of the decoder and each hidden state of the encoder, and the soft- max function is applied to obtain the probability distribution that the frame the keyframe. The cor- responding formulation given in Appendix G.4.2.
After the pretraining phase is com- plete, we proceed to fine-tune the model using our Multi VidSum dataset. We sample eight patterns of N keyframes and the corresponding captions from each video with N more captions. 15
Fine-tuning
Inference In the inference process, we first use the pretrained image captioning model described in Section 5.1 to generate captions for all sampled frames. Then, all the sampled frames are input to the model's encoder, while the frame and gen- erated caption pairs are provided input the model's decoder to calculate score based on the likelihood- (refer to Appendix G.2 for addi- tional details). Finally, the system determines the final output by identifying the list of N keyframes and captions scored highly by the model. Note that there are numerous combinations of N keyframe- caption pairs; thus, calculating the likelihood for all possible combinations is computationally ex- pensive. In addition, the ideal keyframes and cap- tions to be selected depend on the preceding and subsequent keyframes and captions. To address
5.3.1 Training and inference strategy
Loss function In lecting keyframes and the quality of generated cap- tions are equally important, and we cannot prior- itize one over the other. Thus, we optimize the keyframe prediction loss and generated captions loss with equal importance. To achieve this, we calculate the cross-entropy loss independently for both the keyframes and captions, and we minimize their sum. The corresponding formulation is given in Appendix G.4.3
accuracy in se-
Pseudo video dataset pretraining To improve the caption quality and adapt the model the video inputs, we perform pseudo video dataset pre- training. We use two image caption datasets, i.e.,
'FFor the MS COCO dataset, we sample images their accompanying captions. For the Visual Storytelling dataset, we use the first N images and captions of a story. Multiplc keyframes are annotated to a single caption (Sec- tion 4.)

---

Keyframe ASM AKMoas
Caption BLEURT METEOR
Irenative
Fairee-imagee-ae image-captionnng
38.1
80.14
31,66
9.88
Benahh ribaI
refnement
ClipCap
38.95
79.32
30,61
10.12
Amar aivollarre ckep
Amarsnng dest Ares dazg
sung tona diss
CIJI--Reward
58.10
80.20
37,91
10.66
sit sipple
InstructBLIP izero shot)
37.13
78.17
35.15
12.69
InstructBLIP (low-sbot)
38.27
Vid2Seq
42.35
81.20
45.16
15.17
The wannay scund aeaeevati anddiedaallle gtll.
The Dea scppo beb amoodigg aen paocraarits
Eiely sorp oe be (woin ga valking woaddi cents
Simul
self
37.93
79.22
34,30
9.22
determination fainsce-image-aapioonnnng
40.21
32.44
ClipCap
42.03
81.80
30.84
10.36
CLIP-Rewand
42.43
82.01
35.28
10.49
InstrucBBL. IP (xcro-shot)
43.22
36.94
13.42
InstruciBLIPP (few shot)
41.86
82.04
37.35
13.75
Vid2Seg
81.30
44.85
Steal Courakadin
Ana ma adkatoee
Artier le poing hill
dii lanking
ddia kiilig
Table 2: Performance of baseline models on the test set of the the performance when the Simul-determinationn model generates captions itself without using pretrained image captioning model results. "Fairseq-image image---aaaaaae -itge-cappiionnnng"" "ClipCap", "CLIP-Reward" "InstructBLIIP, and "Vid- Seq" are the type of pretrained image captioning models. For readability purposes, all values are displayed multi- plied by 100. See Section 5.1, 5.3.1. for more details.
dana sanmy sapo dass alpe
dataset. Here, "self" means
Axaroding
sking dien Asin aoklig
deera saany dape snoall lel.
skpc
thec mikee onllacko
Figure 6: Comparison of summary generated by base- line models for video.
ability (BLEURT and METEOR) compared to Iter- ative refinement model. This suggests that Simul- determination model had an advantage over Iter- ative refinement model in terms of considering the preceding and succeeding keyframes and cap- tions. By comparing the performance of Simul- determination model relative to self- self-generation ver- sus using captions from a pretrained model, it is clear that both keyframe selection and caption qual- ity realize higher performance when leveraging a pretrained model, which highlights the effective- ness of the proposed approach. Note that the per- formance of Simulldeterminationn model is heav- ily dependent on the selection of the pretrained model, and using a superior pretrained model re- sults in higher-quality captions. In addition, when using audio information (i.c., Vid2Seq), the quality of the generated captions was even higher. How- ever, the keyframe selection performance remained relatively consistent, even when the evaluation of generated captions was low. These findings high- light the strengths of Simul-determination model in keyframe selection and caption generation and they underscore the importance of using pretrained model to realize performance improvements.
these issues, we introduce a beam search algorithm, where the calculation of the likelihood for a single keyframe-caption pair is performed in single step. The details of the beam search algorithm and the corresponding score are given in Appendix G.1.
5.3.2 Model Architecture
As an implementation the Transformer, we adopt the architecture of Flan-T5 architecture (Chung et al., 2022). Additionally, we integrate the pointer and gate mechanism, as described in Sections 5.3. Additional details and the hyperparameters used during training are described in Appendix G.4.
6 Experiments and Results
Settings As described in Section 3.4, we em- ployed multiple evaluation metrics to assess the keyframes and captions. Specifically, in this ex- perimental evaluation, we used AKMex and AKMcos for the keyframes and the METEOR (Lavie and Agarwal, 2007) and BLEURT (Sellam et al., 2020) for the captions. For the test set, we constructed a dataset consisting of 439 videos. In addition, to ensure accuracy and consistency, we performed a re-captioning process on the videos (Section 4.3). Regarding the training data, the specifics differ among the baseline models discussed in Section 5.
Discussion
7.1 Effect of Image Captioning Models
Results Table compares the performance of the baseline models. As can be seen, the Simul- determination model tends to have higher keyframe selection ability (AKM) and caption generation
Figure 6 shows the keyframes and captions gener- ated by the Iterative refinement model and Simul- determination model. By comparing the results obtained by the image captioning models, i.e.,

---

Nofneard
7.3 Effect of Audio Information
As shown in Table 2, using Vid2Seq as an im- age captioning model demonstrated superior per- formance for the generated captions compared to models that rely solely on image inputs, e.g., In- StructBLIP. This improvement can be attributed to Vid2Seq's utilization of audio information during the caption generation process. Figure shows output examples obtained by Iterative refinement model using Vid2Seq and InstructBLIPP (few-shot), along with the ASR results that are expected to have influenced the captions generated by Vid2Seq. For example, the video in Figure 7 shows the shoe polishing process. Recognizing the act of shoe pol- ishing requires attention to specific details in the image; thus, it is considered challenging for im- age captioning model that solely takes the image as the input to grasp the situation accurately. contrast, by leveraging audio information, Vid2Seq excels at grasping the nuances of shoe polishing accurately, thereby enabling the generation of cap- tions that incorporate more detailed information. In fact, found that captions for Keyframes 2 and 3 generated by Vid2Seq include precise details by utilizing specific words extracted from the ASR results taken as the input, e.g., "wipe," reg. and "spray".
naking exhapping
Autun se
alinng locandingg tohin
mes noateg eoeding building
nn inoo into oor canmendedd by obor poppi.
The raa prothe Olee wifi
pidoss dooepoiddaod aproo Rin..
The sulls inks roon Imaico polistas.
poing
geny warrppatte and promeos
ASH coudi
tn le oak
wipe
ogais
Figure 7: Summaries generated by Iterative refine- ment model using Vid2Seq and InstructBLIP (few-shot), along with the ASR outputs that appear to influence captions generated by Vid2Seq.
the ClipCap and InstructBLIP, it can be seen that captions generated by InstructBLIP, which is high-ppefformance image captioning model, pro- vide more informative summaries by describing finer details within the images effectively. For ex- ample, in Figure 6, when comparing the results for Keyframe generated by the Simul-dctermination model using ClipCap and InstructBLIP, we can ob- serve that the latter caption includes the specific information, e.g.. "a blue flag in front of him." In addition, the captions generated by using the pre- more natu-
trained models (e.g., InstructBLIPP) ral and accurate than the self- celf-geeerated captions. In contrast, the self-gennran generation approach incorporates both definite articles and pronouns for coherence and storytelling, as evident the use "A man" and "The man" in consecutive captions (Figure 6).
Conclusion
In this paper, we have proposed practical and chal- lenging task setting of multimodal video summa- rization, Multi-VidSum. We also extended the Ac- tivityNet Captions with the keyframe annotations obtained by the human annotators (crowd workers), To the best of our knowledge, our dataset is the first large video dataset with keyframe annotation, and its quality was assured by human evaluations. We also proposed evaluation criteria of Multi- task to appropriately evaluate this task. Extensive experiments were conducted our dataset using two baseline methods, and we provided bench- mark and relevant insights. We hope our dataset will make new movements in vision-and -nd----nn-lnggge research and accelerate them.
7.2 Qualitative Differences between Baseline Models
Iterative refinement model, the keyframes are predicted based on the segments identified by the segment extraction module as described Sec- tion 5.1.1. I. Thus, we observed a tendency for var- ious keyframes to be selected compared to the simul-determination model. However, the varia- tion in keyframes does not necessarily correlate with the quality of the summaries. For exam- ple, some frames might be less significant, con- taining only text as titles that are dissimilar to other frames. As result, the Iterative refinement model exhibits inferior performance compared to the dimul-determinationn model.

---

Limitations
are not included in our dataset as well as the initial dataset of our dataset, and thus, owners of the video immediately delete their videos at any time they want.
this work, we proposed a video summariza- tion task based on an actual use case and pro- posed a corresponding dataset and bascline sys- tems for it. However, our proposed the Simul- determination model does not consider the entire keyframe-caption pairs to generate the optimal sum- mary (despite being somewhat alleviated by the beam search algorithm). To address this issue, will be necessary to construct a system that gener- ates a summary that considers future information using reinforcement learning.
Acknowledgements
We thank four anonymous reviewers who provided valuable feedback. We would like to also ap- preciate the member of Tohoku NLP Group for their cooperation in conducting this research. We would like to especially thank Kotaro Kitayama. Shun Sato, Tasuku Sato, and Yuki Nakamura for their contributions to the predecessor of this re- search. This research was conducted as joint research between LY Research and Tohoku Uni- versity. Also, part of this research (dataset con- struction) was condueted with the support of JST Moonshot R&D Grant Number JPMJMS2011 (fun- damental research).
In addition, as discussed in Section 5.2.1, the proposed Simul-determinationn model takes every frame downsampled every 0.5 from the video as input. However, this configuration raises scala- bility issue because the number of frames taken as input increases with the length of the video, thereby leading to high computational costs. To address this scalability issue, potential solution involves integrating model that identifies salient frames from the video. However, to maintain simplicity in the baseline system, this development is left future task in this work.
References
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. Analyzing the Behavior of Visual Question Answering Models. Proceedings of the 2016 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1955-1960.
Ethics Statement
The initial state of our dataset, which we created to facilitate this study, is the existing dataset, Activi- tyNet Captions dataset, mentioned in Section 4. In addition, our models were trained only the constructed dataset and published datasets that have been used in many previous studies. Thus, our task, dataset, and methods do not further umplify biases and privacy for any other ethical concerns that im- plicitly exist in these datasets and tasks.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question An- swering. In Proceedings of the IEEF International Conference on Computer Vision (ICCV).
Evlampios Apostolidis, E. Adamantidou, Alexandros Metsai, Vasileios Mezaris, and I. Patras. 2021. Video summarization using deep neural networks: A survey. Proceedings of the IEEF, 18388838
Lorenzo Baraldi, Costantino Grana, and Rita Cucchiara. 2017. Hierarchical Boundary-A Award Neural Encoder for Video Captioning, In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 3185-3194.
In addition, we used crowd-sourcing system to annotate the keyframes for the videos in the initial dataset (Section 4). Here, we paid total of ap- proximately $30,000 USD worth of Japanese yen for this keyframe annotation work. Moreover, as described in Section 4.3, we reannotated the cap- tions in the test data. Here, we additionally paid approximately $10,000 USD worth of Japanese yen for this caption reannotation process, representing $22.8 per video. We believe that we paid a suf- ficient amount to the workers for this annotation labor.
Reshmi Bhooshan and Suresh 2022. Multi- modal Framework for Video Caption Generation. IEEE Access. 10:92166-92177
Brandon Castellano. 2014. pyscenedetect.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015. Microsoft COCO Cap- tions: Data Collection and Evaluation Server. arXiv preprint, 203/150
We disclose some privacy risks and biases in the video because the source of our dataset is the videos submitted to a public video-sharing plat- form. However, we emphasize that these videos
Yangyu Chen, Shuhui Wang, Weigang Zhang, and Qing- ming Huang. 2018. Less Is More: Picking Informa- tive Frames for Video Captioning. In 15th Earopean

---

Conference on Computer Vision (ECCV), pages 367- 384.
toonist: A Comic-Style Video Summarization Sys- tem for Anime Films. In 22nd International Confer- ence Multimedia Modeling 2016 (MMM), volume 9516.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open- Source Chatbot Impressing GPT-4 with 90%* * Chat- GPT Quality.
Zhe Gan, Chuang Gan, Xiaodong Hc, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. 2017. Semantic Compositional Networks for Visual Captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1141-1150.
Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Demoncourt, Trung Bui, and Mohit Bansal. 2022. Fine-grained Image Captioning with CLIP Reward. In Findings of the Association for Computational Linguistics (NAACL), pages 517-527.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating Copying Mechanism in Sequenceeto-Seqqenee Learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1631-1640.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu. Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sha- ran Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny 7hou, Quoc V. Le, and Jason Wei. 2022. Scaling Instruction- Faneettion-Fineuneed Language Models. arXiv preprint, cs. A.LGG2210.114160
Michael Gygli, Yale Song, and Liangliang Cao. 2016. Video2gif:: Automatic Generation of Animated GIFs from Video. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pages 1001-1009.
Bo He, Jun Wang, Jielin Qiu, Trung Bui, Abhinav Shri- vastava, and Zhaowen Wang. 2023. Align and Attend: Multimodal Summarization With Dual Contrastive Losses. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 14867-14878.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. InstructBLIP: Towards General-purpose Vision-language Models with instruction tuning. CoRR, arviv preprint, cs. CwC.aaXii:230550650000
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Technologies (NAACL), pages 4171-4186.
Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, and Radu Soricut. 2020. Multimodal Pretraining for Dense Video Captioning. Proceedings of the Ist Conference of the Asia-Pacific Chapter of the Asso- ciation for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing (AACI-IJCNLP) pages 470-490.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image Worth 16x16 Words: Transformers for Image Recog- nition at Scale. In 9th International Conference on Learning Representations (ICLR).
Ting-Hao Kenneth Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin. Ross Girshick, Xiaodong He, Push- meet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Mar- garet Mitchell. 2016. Visual Storytelling, In Proceed- ings of the 2016 Conference of the North American Chapter of the Association for Computational Lin- guistics; Human Language Technologies (NAACL), pages 1233-1239.
Liana Ermakova, vale-e Cossu, and Josiane Mothe. 2019. A survey on evaluation of summarization meth- ods. Inf. Process. Manag., 1695):199411814.
Sergey loffe and Christian Szegedy. 2015. Batch Nor- malization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference Machine Learning (ICML), volume 37, pages 448-456.
Xiyan Fu, Jun Wang, and Zhenglu Yang. 2021. MM- AVS: A Full-Scale Dataset for multi-mode modal summa- rization. In Proceedings of the Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies (NAACL), pages 5922-5926.
Khushboo Khurana and Umesh Deshpande. 2023. Two stream multi-layer convolutional network for deyframe-based video summarization. Multimedia Tools and Applications, pages 1-42.
Tsukasa Fukusato, Tatsunori Hirai, Shunaya Kawamura, and Shigeo Morishima. 2016. Computational Car-

---

Diederik P. Kingma and Jimmy Ba. 2015. Adam: Method for Stochastic optimization. In 3rd Inter- national Conference Learning Representations (ICLR).
Medhini Narasimhan, Anna Rohrbach, and Trevor Dar- rell. 2021. CLIP-it! Language-Guu Guided Video Sum- marization. In Advances in Neural Information Pro- cessing Systems, pages 13988-14000.
Jamie Kiros, William Chan, and Geoffrey Hinton, 2018. Mlustrativee Language Understanding: Large-Scale Vi- sual Grounding with Image Search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 922-933.
Mayu Otani, Yuta Nakahima, Esa Rahtu,, . and Janne Heikkilä. 2020. Uncovering Hidden Challenges in Query-based Video Moment Retrieval. In The British Machine Vision Conference (BMVC).
Myle Oit, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL (Demonstrations), pages 48 53.
Ryan Kiros, Ruslan Salakhurdinov, and Richard S Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv: 14111213325
Over and J. Yen. 2003, An Introduction to DUC 2003: Intrinsic Evaluation of Generic News Text Summarization Systems.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. 2023. Grounding Language Models to Images for Multimodal Inputs and Outputs. 40th International Conference on Machine Learning (ICML).
Pingbo Pan, Zhongwen Xu, Yi Yang, Fei Wu, and Yuet- ing Zhuang. 2016. Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning. In 2016 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 1038.
Martin Krasser. 2020.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense- Captionnn Events in Videos. In International Conference on Computer Vision (ICCV).
Yingwei Pan, Ting Yao, Houqiang Li, and Tao Mei. 2017. Video Captioning with Transferred Semantic Attributes. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 984- 992.
Sourabh Kulhare, S. Sah, Suhas Pillai, and Ptucha. 2016. Key frame extraction for salient activity recog- nition. In 23rd International Conference on Pattern Recognition (ICPR), pages
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πlya Sutskever. 202 la. Learn ing Transferable Visual Models From Natural Lan- guage Supervision. In Proceedings of the 38th Inter- national Conference on Machine Learning (ICML), volume 139, pages 3748-8763.
Alon Lavie and Abhaya Agarwal. 2007. METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments. In Proceedings of the Second Workshop on Statistical Machine Translation (WMT), pages 228-231.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. 2020. Video storytelling: Textual sum- maries for events. IEEE Transactions on Multimedia, 22(2):554-565.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin. Jack Clark, Gretchen Krueger, and llya Sutskever. 2021b. Learn- ing Transferable Visual Models From Natural Lan- guage Supervision. In Proceedings of the 38th Inter- national Conference on Machine Learning (ICML). volume 139, pages 8748-8763.
Shanhao Li, Bang Yang, and Yuexian Zou. 2022. Adap- tive Curriculum Learning for Video Captioning. IEEE Access, 10:31751-31759.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Con- ference on Learning Representations (ICIR).
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners.
Xiaopeng La, Tiancheng Zhao, and Kyusong Lee. 2021. VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to- Image Search with Weighted Bag-of-words In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the I 1tth International Joint Conference on Natural Language Processing (ACL), pages 5020- 5029.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning Robust Metrics for Text Genera- tion. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics (ACL). pages 7881 1-8922.
Paul Hongsuck Sco, Arsha Nagrani, Anurag Amab, and Cordelia Schmid, 2022. End-to to-End Generative Pre- training for Multimodal Video Captioning. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 17959- 17968.
Ron Mokady, Amir Hertz, and Amit H. Bermano. 2021. ClipCap: CLIP Prefix for Image Captioning. arXiv preprint, es. CV7311..973341..

---

Qingfeng Sun, Yujing Wang, Can Xu, Kai Zheng, Yam- ing Yang, Huang Hu, Fei Xu, Jessica Zhang, Xiubo Geng, and Daxin Jiang. 2022. Multimodal Dialogue Response Generation. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (ACL), pages 2854-2866.
Sivic, and Cordelia Schmid. 2023b. Vid2Seq: Large- Scale Pretraining of a Visual Language Model for Dense Video Captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- lern Recognition (CVPR), pages 10714-10726.
Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Bal- las, Christopher Pal, Hugo Larochelle, and Aaron Courville. 2015. Describing Videos by Exploiting Temporal Structure. In 2015 IEEE International Conference Computer Vision (ICCV), pages 4507- 4515.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe. Jonathon Shlens, and Zbigniew Wojna. 2016. Re- thinking the Inception Architecture for Computer Vision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition CVPR, pages 2818-28266
Yitian Yuan, Xiaohan Lan, Xin Wang. Long Chen, Zhi Wang, and Wenwu Zhu. 2021. A Closer Look at Temporal Sentence Grounding in Videos: Dataset and Metrie. In Proceedings of the 2nd International Workshop on Human-centric Multimedia Analysis (HUMA), pages 13-21. ACM.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. 2018a. Reconstruction Network for Video Captioning. In 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 7622-7631.
Ke Zhang, Wei-Lum Chao, Fei Sha, and Kristen Grau- man. 2016. Video Summarization with Long Short- Term Memory. In Computer Vision - ECCV 2016 14th European Conference, Amsterdam, The Nether- lands, October 11-14, 2016, Proceedings, Part VII. volume 9911 of Lecture Notes in Computer Science, pages 766-782. Springer.
Junbo Wang, Wei Wang, Yan Huang, Liang Wang, and Tieniu Tan. 2018b. M3: Multimodal Memory Mod- elling for Video Captioning, In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 7512-7520.
Luowei Zhou, Chenliang Xu, and Jason Corso. 2018a. Towards Automatic Learning of Procedures From Web Instructional Videos. In AAAI Conference on Artificial Intelligence, pages 7590-7598.
Xin Wang, Wenhu Chen, Yuan-Fang Wang. and William Yang Wang. 2018c. No Metries Are Per- feet: Adversarial Reward Learning for Visual Story- telling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 899-909.
Luowei Zhou, Yingbo Zhou, Jason J. Corso, Richard Socher, and Caiming Xiong. 2018b. End-to-End Dense Video Captioning With Masked Transforme.. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR).
Ronald J. Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):299-256.
Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Jiajun Zhang, and Chengqing Zong 2018. MSMO: Multi- modal Summarization with Multimodal Output. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 4154-4164,
W. Wolf. 1996. Key frame selection by motion analysis. In 1996 IEEE International Conference Acoustics, Speech, and Signal Processing Conference Proceed- ings (ICASSP), volume 2, pages 1228-1231.
Chenggang Yan, Yunbin Tu, Xingzheng Wang, Yong- bing Zhang, Xinhong Hao, Yongdong Zhang, and Qionghai Dai. 2020. STAT: Opatial-Temport Atten- tion Mechanism for Video Captioning, IEEE Trans- actions on Multimedia, 22(1))229-22
X. Yan. Syed Zulqarnain Gilani, Hanlin Qin, Ming- tao Feng. L. Zhang, and A. Mian. 2018. Deep Keyframe Detection in Human Action Videos. ArXiv, 100/1804..0021.
Antoine Yang, Arsha Nagrani, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023a. Vidchapters- TM: Video Chapters at Scale. In Advances in Neural Information Processing Systems (NeurIPS).
Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef'

---

A Other related work
by adding captions.
Multimodal Generation
Several models for multimodal generation tasks have been proposed based on a causal language model (Sun et al., 2022; Koh et al., 2023). The study by (Sun et al., 2022) is notable for its feature of generating both text and images. Additionally, (Koh et al., 2023) introduced model that takes text and images as input dur- ing language model inference, retrieving images that align with the generated text and producing outputs simultancously. Our task shares similari- ties with this study regarding selecting (retrieving) images and generating text simultaneously. How- ever, a notable difference between our task and this task is the visual similarities among candidates of images (frames), where many frames exhibit min- imal visual differences in video. Consequently, considering these subtle distinctions and selecting important frames becomes challenging for our task.
Video Captioning Video Captioning aims to gen- erate sentence (or a few sentences) describing the overview of given video. Later, Video Caption- ing is extended to Dense Video Captioning, which simultaneously detects event locations video segments and captions for each segment. Note that Dense Video Captioning essentially matches Video Captioning if the videos only consist of one segment (one event) or the videos were split into segments beforchand.
A typical approach to Video Captioning is to use transformer (Vaswani et al., 2017) that consists of the video encoder and caption decoder (Seo et al., 2022; Li et al., 2022). which can be seen as exten- sions of image captioning. such as Mokady et al. (2021); Dai et al. (2023). Currently, many studies have proposed techniques to improve the quality of generated captions, e.g., (Yao et al., 2015; Baraldi et al., 2017; Pan et al., 2016; Gan et al., 2017; Pan et al., 2017; Wang al., 2018b, Yan et al., 2020; Bhooshan and K., 2022; Li al., 2022).
Keyframe Detection Keyframe Detection is task to select series of salient frames from videos. This is one of the fundamental and most straight- forward video summarizations. The word "salient" here means something conspicuous or representa- tive of the entire video. For example, Yan et al. (2018); Khurana and Deshpande (2023) proposed a method to summarize videos in multiple important images (keyframes). Wolf (1996); Kulhare et al. (2016) proposed keyframe detection method uti- lizing optical flow features. Similarly, Fukusato al. (2016) detected keyframes from anime films, Chen et al. (2018) proposed a video captioning method that can show informative keyframes in video using reinforcement learning.
Video2GIF Video2GIF is the task that converts input video into GIF animation. Gygli et al. (2016) proposed the large-scale dataset for the task and show the baseline performance on the dataset. Un- like the Multi-VidSum, Video2GIF is the summa- rization without using any caption information, so the dataset cannot be diverted the Multi- VidSum. However, showing the GIF animation as a video summary is an attractive option instead of a keyframe in the Multi-VidSuni
A.1 Other tasks
Other than the aforementioned tasks, various vision and language tasks such Visual Question An- swering (Antol et al., 2015; Agrawal et al., 2016). Text-Image Retrieval (Kiros et al., 2014, 2018; Lu et al., 2021; Radford et al., 2021b).
Narasimhan al. (2021) proposed method for query-foeused video summarization that summa- rizes a video into set of keyframes.
These works take an unsupervised approach to detect keyframes because there are no large datasets available for training Keyframe Detection models in supervised approach. Despite their ap- proach, we provide a relatively large dataset with keyframe annotations by hand that enables us to train (and evaluate) Keyframe Detection models in a supervised manner.
B Evaulation criteria details B.1
AKM Definition
We assume each video has M (reference) captions, and each caption has the K (reference) keyframe candidates, where M > N and K 2 1. Fig- ure 2 shows an example. The reason to incor- porate multiple keyframe candidates for the refer- ence that each caption can explain several similar scenes. For evaluating the model of the predicted N keyframe list with the M reference keyframe
Displaying selected keyframes a single page can be an excellent approach for video summary at glance. The Multi-Vine VidSum task further promising task since it can offer users an easier and quicker understanding of the target video contents

---

set list (M 2 N), we calculate the matching score between the two lists. Let p (PI..... Pv) be a predicted keyframe list by a model. Moreover, A - .d....,Ay. denotes a list of reference keyframe sets. We assume that the sequences in p and A are sorted under the chronological order of the given video. Similarly, A (A AN) is a sub-list of A, whose length is identical to the system output N. Note that the meaning of the sub- list here is eliminating M N reference keyframe sets from A. Then, we introduce a matching func- tion mex (---) that receives predicted and reference keyframes and returns a matching score I if the in- putted two keyframes correctly match and other- wise. Finally, we define aligned keyframe match- ing (AKM) score with exact matching, AKMex. as follows:
metric within the realm of caption generation tasks, and is precisely the metric we have chosen adopt our investigation. BLEURT neural language moodel-based evaluation metric assessing semantic similarity between generated captions and references.
C Detail for proposed dataset
C.1 Data construction procedure
Figure 2 shows the overview single data in our dataset. Each data consists of three parts; video, a set of captions, and sets of keyframe candidates. Each caption has several keyframes that can match the corresponding caption. The reason for the mul- tiple keyframes for each caption is that the video has the same very similar frames in one video to consider the multiple candidates. We created the data by assigning caption text to the videos and selecting the keyframes that best match the caption. In more detail, we added the keyframe annotations to the existing captioned movic dataset created by Krishna et al. (2017). We added about 10 keyframe annotations per caption. Specifically, define the task as selecting the most appropriate time by looking at the video and the caption. The actual working platform used by the cloud worker is shown in Figure 8. On the working platform, the vidco is placed on top and shows segments by the color bar. The bottom box contains the cap- tions. Each caption has an "add" button to record the keyframe's time. Annotators are required to explore frames (can record multiple candidates) that match the corresponding caption. We set the minimum time unit of the keyframe caption 0.5 seconds. All cloud workers have to annotate at least one time for each caption.
Σ max m [mectenna)] (2) aeAi
AKMex
max AEA
N
where Á E A means taking an N sub-list from A. In Eq. 2, the meaning of computing max Á, (Mxx a)} to check the matching be- tween predicted keyframe p; and answer keyframe a in the candidate set A.. We will obtain 1 if Pt a, where a E A; satisfies, and 0 otherwise. The meaning of summation from i = to N di- vided by N is straightforward, just taking the aver- age of inner matching counts over the system out- put. Note that, in implementation, we can compute Eq. using the dynamic programming technique. The evaluation by Eq. 2 might be too strict be- cause it is a challenging problem to find exact matching keyframes. To relax the evaluation, we introduce relaxed matching score AKMcas by sub- stituting mex (...) as (:):
To extend the dataset, we used Yahoo Crowdsourcing service https: / https:///rowt::: htt::///ooodsurriingo yahoo. co. jp/. We carried out the tasks six days week, which took two months. 352 people joined in total. Each crowd-sourcing task consists of five videos and ten individual workers annotated for each task. On average, about 160 tasks are included in one batch, and three batches are carried out daily.
Meus(p, a) = max (0, Cos(VP: Va)),
(3)
where Vp = vv- and Va V. Moreover, Vp and v, are the image feature vectors for predicted and reference keyframes. v is the mean vector of image feature vectors of a video. Cos(x, y) is a function that returns the cosine similarity between given two vectors, x and y.
B.2 Caption evaluation criteria
C.2 Test set refinement instructions
Our evaluation of the generated captions uses two metrics: METEOR (Lavie and Agarwal, 2007) and BLEURT (Sellam et al., 2020). METEOR stands as a prevalent and extensively employed evaluation
As detailed in Section 4.3, we conducted re- annotation of low-quality captions by reliable an- notators to ensure a proper evaluation of the task. We asked the annotators to modify the captions so

---

Rank
2
3
5
Original
500 7.7%
do 3.7%
hold 3,5%
7.9%
3.5%
Refined
play
stand 6.2%
hold 5.4%
speak 5.3%
show 5,1%
6.7%
Table 3: Distribution of the top-5 most frequent verbs in the captions before and after test set refinement. Here, "Original" and "Refined" indicate the distribution of the verbs in the captions before and after refinement, respectively.
Amasiaa blard
ta doww
owny
Peopic booddig bewe hepetic
Figure 8 Web application for annotators: annotators can add annotations by pushing buttons.
video
unigue
unique
average words per caption
Steeyrelling
YouCook2 (val)
457
126
954
8.7
1094
that they meet the following instructions.
Ours
439
324
1528
11.6
Each caption should describe the given scene, which consists of a set of images.
Table 4: Vocabulary diversity and caption length in existing video summarization datasets (Video Story- telling (Li et al., 2020), YouCook2 (Zhou et al., 201 8a). and ViTT (Huang et al., 2020)). These datasets have no keyframe annotations. For the YouCook2 dataset, the test set is not publicly available, and previous work (Yang al., 2023b) used the validation set as the evaluation data. Thus, used the validation set for this analysis.
The caption should be single sentence.
The caption should be free of spelling and grammar errors.
The caption should focus on describing the actions of people.
The caption should contain information that is naturally inferred from the given scene.
Yuan et al., 2021), and we compared the results before and after re-annotation (Table 3). As a re- sult, we confirmed that the frequency of verbs that do not represent actions (e.g., show) has reduced. Thus, we believe that we perform more accu- rate evaluations by using this newly annotated test set.
The caption should describe the common el- ements shared by the majority (more than half) of the 10 or so images extracted for each scene.
To confirm the validity of the test set, we also compared vocabulary diversity with existing datasets. Table 4 shows that the newly annotated dataset is inferior in terms of vocabulary diversity compared to VITT (Huang et al., 2020) because the ViTT is large-scale dataset and includes greater number of test samples than ours. In con- trast, compared to YouCook2 (Zhou al., 20)8a), which is a dataset of the same scale, our dataset has higher vocabulary diversity. Ideally, it is desir- able to increase the number of videos to increase diversity; however, due to budget constraints, we needed to restrict the number of videos used in the test set. Nonetheless, contains sufficient diversity in terms of actions and situations for dataset of this size.
C.3 Train and test split
According to the data split defined both Activi- tyNet and ActivityNet Captions, the training data consists of 7,727 samples. We use these 7,727 samples as the training dataset. As described in Scction 4.3, the test set comprises 439 videos sam- pled from the validation dataset, which consists of 4,282 videos defined by ActivityNet. 16
C.4 Test Set Analysis
Action and vocabulary diversity In video- related tasks, action diversity in the video content is important to realize an effective evaluation (Otani et al., 2020; Yuan et al., 2021). Here, to evaluate the diversity of actions, we investigated the distri- bution of verbs in the captions (Otani et al., 2020;
believe that dataset
Caption informativeness In the context the video summarization task, the informativeness of a generated summary is a crucial factor (Ermakova
16 As the original test set of ActivityNet is not publicly available, we created the test set of our by sampling from the validation set of ActivityNct.

---

In Zhou et al. (2018b), they introduced the fol- lowing four loss function to train the MTM.
The regression loss Lr
Inpwar
The event classification loss Le
Excde
ounde
The binary cross entropy mask prediction loss L
prbiiio
The captioning model loss Li
Figure 9: Overview of the MTM model.
The regression loss L, The regression loss is the loss function in the proposal decoder to learn the potions of segments:
al., 2019) in terms of realizing high-quality sum- marizations. In this task, the summary con- structed from captions assigned to keyframes; thus, the informativeness of each caption in the test set is crucial in terms of achieving valid task evalua- tion. Here, we consider that the informativeness of each caption corresponds to the number of words in each caption (i.e., the length of the caption) be- cause, if the number of words is small, the caption will be insufficient constitute an effective sum- mary, e.g., omitting information about the subject (i.e., the actor). Thus, we confirmed the validity of the test set by comparing the number of words in the captions between existing datasets. As shown in Table 4, VITT (Huang et al., 2020), which is large-scale video summarization dataset, contains many short captions with average of 3.1 words per caption. In fact, after checking the dataset, found that it primarily comprises words or phrases, e.g., "Introduction" and "Adding water". con- trast, the captions in our dataset are sufficiently long, with an average of 11.6 words per caption, which is sufficiently informative. Thus, our dataset is considered effective for the proper evaluation of the VidSum task.
L, Smoothy (6e, 0o) Smoothn (Ō,, 0t). (4)
The MTM model predicts the segments using two variables 0g and O,. Here, 0 refers to the cen- ter offset of the segment, and 6) refers length offset of the segment in the video. If we decide these two variables, regression loss is defined as the smooth L1 loss summation between the predictions (6er O:) and ref- erences (0c, 0z).
determine segment. The
The event classification loss L.: Same as the re- gression loss, the event classification loss is the loss in the proposal decoder to learn proposal score:
Le BCF((P, Pa).
(5)
P. and P. donate the predicted and reference pro- posal score, and BCE is the binary cross entropy loss. The proposal score is treated as confidence score for prediction.
The binary cross entropy mask prediction loss L The binary cross entropy mask prediction loss is the loss for connecting the three components, vidco encoder, proposal decoder, and caption de- coder. The loss is defined as:
D Details of Iterative refinement model
T = BCF BCF(BBa Ep, i), SeM Ep. Sa, Eu,
D.1 Model detail for Segment Extraction Module
(6)
where S, and Sp indicate the start position of ref- erence and predicted segments and E, and Ep in- dicate the end position of reference and predicted segments. i indicates that the frame in the video now model pays attention.
As described in Section 5.1, we use Masked Transformer Model (MTM) (Zhou et al., 2018b) the segment extraction module. MTM is Transformer-based model for the Dense Video Cap- tioning task proposed in (Zhou et 2018b). When we train MTM, follow the instruction written in Zhou al. (2018b), including the configurations and hyperparameters. Figure shows the overview the MTM. Here, we only use part of the video encoder and proposal decoder to extract segments.
BM (Sp. Ep. i) is the binary mask function that outputs if and only if the i-th frame is included in the reference segments, namely,
BM Ep. = {
1 if i [Sa. Ea) otherwise
(7)

---

fCM is the smoothed gated mask function be- tween BM and fas, that is,
Enyluer
[cp Ep, San L)
Siquase yrou byyr
= (Pp Ep. + +1 (S) Eps Su. Ea, i))
(8)
where Pe is the proposal score.
Werne lelling
Next, JM is the mask function that converts the discrete position information described by Eq. 7 learn. fM be written
Figure 10: Overview of the Keyframe Detection model.
into continuous as:
Keyframe selection model
Ep. So: Ea, =0((V
Optimizer Learning Rate
Adam (Kingma and 2015)
0.0001 16
Batch Size
where (-) indicates the logit function, and is the multi-layer perceptron. Moreover, VM is defined follows:
Number of Epochs
20
Table 3: The hyperparameters settings of the Keyframe selection Model
VM ((p)) p(Ee). (Sp, Ep. t)],
(10)
is 1024, and the hidden size of the feed-forward layer is 2048. The head size of the transformer is eight, and the number of layers is two. The paraemeter in Eq. 12, we set d = 10,
where p is the positional encoding function, and [t] is the concatenation function. Finally, the fCM output is passed to the caption decoder. Additionally, thanks to this masking ar- chitecture, the caption decoder's ' loss is passed to both the proposal decoder and the video encoder continuously.
1.0, As 1.0, A4 0.25. We create input image features using both ResNer200 (He al., 2016) and BN-Inception (loffe and Szegedy, 2015). Note that we do not use the validation data for the model selection. We trained the models by the constant iterations and used the models provided by the last iteration. We use the code provided at https://ggt thub. cwm/ http:/alllffeeeeeeeeee Also, we used TITAN X (Pascal, Intel (R) Xeon (R) for training this model.
The captioning model loss T The captioning model loss the loss in the caption decoder to learn the caption generation. The loss L is described as:
L = BCE((,,,(u,,)) X,Y 1)).
(11)
X and Y indicate the encoded feature vectors of each frame and caption text, respectively. wy is the reference caption at the time step t. p(w,/ XX Y donates the predicted probability the f-th word of the reference caption.
D.2 Model detail for Keyframe selection module
Figure 10 shows the overview of the model for the Keyframe selection module (Keyframe selection model). In training for the Keyframe selection model, we use the binary cross-entropy loss as a loss function, namely,
Overall loss function Finally, the total loss func- tion L for MTM is defined as
L Jap + Aot Aa
Σ T
Σ D (12)
LBCB(() Σ wn ymlog a (hhm)
D.1.1 Hyperparameters and Training Configurations
+ (1 - Um log(1 - o (lmm))
The setting of hyperparameters of the MTM fol- lows the Zhou et al. (2018b). We set the kernel size of the temporal convolution in the Caption Decoder from to 251 and the stride factor to 50. The model dimension of the transformer model
(13)
Here, a (x:) indicates the logit function. Also, hm is the output of the linear layer connected after the sequence process layer (Bidirectional LSTM) for

---

Keyfrume
Caption
Keyframe AKML, AKMeza BLEURT METHOR
Caption
AKMac
AMMo
BLEURT
METEOR
Roop
33.37
78.32
36.13
13.03
Random
37.98
78.47
36.63
13.39
Ruop +4.33
+0.45
-0.55
+0.12
Using key srymeeme selection module 38.27
78.55
36.67
13.46
Roop 44.33
-0.16
+0.35
40.37
Rnop 3
+4.61
+0.32
+0.38
+0.35
Roop
Table 7: Comparison hetween the case where the keyframe selection module is used (Using keyframe selection module) and the case where it is not used (Random) in Iterative refinement model. This result is obtained when InstructBLIP (few-shot) is used as the caption generation model. For readability purposes, all values are displayed multiplied by 100.
Table 6: Ablation study results demonstrate the effec- tiveness of iterative refinement. The score increment is indicated based on the baseline score from roop (when the center frame of the segment is selected). This result is for the case where InstructBI.IP (few-shot) is used as the image captioning model. Iterations roopl roop4 correspond to the first to fourth refinements, respectively. For readability purposes, all values are displayed multiplied by 100.
all evaluation metrics, validating the effectiveness of the keyframe selection module.
F Image captioning models detail
each frame, Ym is the binary reference label that represents whether the frame is the keyframe or not, and Wm is the weight. M is the total number of keyframe captions of the training data. Table 5 shows the model hyperparameter settings of these components. As a pre-processing, the image fea- tures from ResNet200 (He et al., 2016) and BN- inception (Ioffe and Szegedy, 2015)) are reduced to 512 dimensions for each by using linear layer, and then concatenated with text features generated by BERT (Devlin et al., 2019)), whose dimension is 768. Thus, the input dimension becomes 1792 (512 512 + 768). We used TITAN X (Pascal, Intel (R) Xeon (R) for training this model.
F1 pretrained image captioning models
We use pretrained image captioning models for both Iterative refinement model and Simul- determination model. Here, we show the details of each model.
Fairseq-image-cappioonn captioning (Krasser, 2020) is fairseq (Ott et al.. 2019) based image captioning library. It converts images into feature vectors using Inception V3 (Szegedy et al., 2016) and generates captions using Transformer. We fine-tuned the model pretrained on MS COCO (Chen et al., 2015) using the keyframe-caption pairs included in our Multi- VidSum dataset.
Fairseq-image-
Ablation study for Iterative refinement model
ClipCap ClipCap (Mokady et al., 2021) is a model that aims to generate high-quality image cap- tions by combining pretrained CLIP (Radford al., 2021a) and GPT-2 (Radford et al., 2019). ClipCap first converts the input image into a sequence feature vectors using the visual encoder of CLIP. Then, the sequence of feature vectors is converted by the Mapping Network and used the prefix of the input to GPT-2 to generate captions. We also fine-tuned this model using the keyframe-caption pairs included in our VideoStory dataset.
E.1 Impact of iterative refinement
Table 6 illustrates the performance change resulting from iterative refinement, where keyframe detec- tion and caption generation are performed alter- nately. It can be observed that the scores, except for AKMcos, exhibited tendency improve with each iteration. However, due to AKMcos being soft evaluation metric, is considered that there was no significant change even after the iterations.
E.2 Impact keyframe selection module
CLIP-Reward CLIP-Reward (Cho et al., 2022)
To assess the effectiveness of the keyframe selec- tion module (Section 5.1.1), we conduct a com- parison between the system using the module and system where the keyframe selection module is omitted, and keyframes is randomly selected. Ta- ble shows the results. The experiment results con- firmed that the use of the keyframe selection mod- ule led to an improvement in performance across
is model that proposes to solve the image caption- ing task in the framework of reinforcement learn- ing, which enables fine-grained image captioning. The model trained using the REINFORCE Algo- rithm (Williams, 1992) with the similarity between the image and the caption calculated by CLIP as the reward, and can be trained without reference cap- tions. We finettned the image captioning model

---

trained using CLIP and CLIP as the reward func- tion using the keyframc-caption pairs included in our dataset.
3. If there no span that time, use the caption for the nearest span in time.
In this paper, we use the PyTorch re- implementation of the Vid2Seq code provided by Yang et al. (2023a) and conduct fine-tuning with ActivityNet Captions.
InstructBLIP InStructBLIP (Dai et al., 2023) is also an image captioning model that combines pre- trained ViT (Dosovitskiy et al., 2021) and Flan- T5 (Chung et al., 2022) or Vicuna (Chiang et al., 2023) and instruction-tuning was conducted using 26 datasets. In this study, we use the model based on Flan-T5-xxl. Since this model has been reported to perform well on held-out datasets, even in zero- shot, we generated captions with zero-shot and few-shot prompts without fine-tuning. We feed the prompt "Generate a simple caption for the input image." to the model for zero-shot and "Generate a simple caption for the input image like the fol- lowing examples: <Example: He fights the man in the red suit.> <Example: men are swimming on the lake by the kayak and walking in the lake- side holding kayaks.> Do not copy the examples." for few-shot (two-shot). The few-shot prompt is composed of randomly selected captions from the training dataset.
F.2
Caption score for Iterative refinement model
In the Summarization module of Iterative refine- ment model, score reflecting the validity qual- ity of the captions is used to select list of N keyframe-caption pairs. This score is primarily based on the likelihood of the captions generated by each model, excluding CLIP-Reward. The CLIP- Reward, on the other hand, uses the similarity between the image and the caption calculated by CLIP (Radford et al., 2021a).
G Simul-determination model details
G.1 Beam Search Algorithm
As described in Section 5.3.1, we introduce the beam search algorithm to global optimization. In this process, we repeat the beam search process N times, retaining only the top W (beam size) re- sults at each step and moving on to the next step. For each step, the model calculates the score based on likelihood (see Appendix G.2 for more details about this score) for lists of less than or equal to keyframes and captions pairs. Finally, among the W remaining results, we select the N pairs with the score based on the highest likelihood as the final output of the model. Algorithm 1 presents the pseudo-code for the beam search algorithm used in demul-determination model. In the experi- ment, a beam width of 8 is employed for the Simul- determination model.
Vid2Seq Vid2Seq is pretrained model proposed for the dense video captioning task in Yang al. (2023b). takes as input a feature vector for each frame, along with the automatic speech recogni- tion results extracted by Whisper (Dai et al., 2023) and generates tokens that represent the span in the video and captions for that particular span. In this study, we use this model one of the variants of the image captioning model. Specifically, pre-generate spans and captions for all videos. Dur- ing the inference of Iterative refinement model and Simulldetermination model, we select a span con- taining the position (time) of the required frame and use the caption associated with that span. How- ever, the span generated by Vid2Seq does not cover the entire video and there are overlapping spans. Hence, when choosing caption corresponding to frame at particular time, the following algorithm is applied:
G.2 Score for inference process
To score the selected keyframes and generated cap- tions, we used a score based on the likelihood calcu- lated by Simul-determination model. Specifically, we independently calculated the likelihood of the model selecting the keyframe and the likelihood of the generated caption and used the sum of the normalized values as the score. This is because we consider that the performance of both keyframe selection and caption generation is equally impor- tant in this task. The score for N candidates for keyframe and caption is formulated as follows.
1. If there is only one span at that time, use the caption for that span.
2. If there are overlapping spans at that time, pri- oritize the shorter span (as shorter spans are presumed to contain more localized informa- tion).

---

Ccore(ycande Zeand)
N
anorm (fglameeee(lllllii..eee
N
(1 normenption of (f (
(14)
where ycand is the candidate keyframes, cand is the candidate captions f is the model trained to calculate the likelihood. is hyperparameter that controls the balance between the score of the keyframe and caption. We specified a as 0,5 this study because we considered that keyframe selection and caption generation performance are equally important in this task. normframe and nor caption are the min- max normalization for all frames and corresponding captions likelihoods, spectively.
Algorithm 1 Beam search algorithm for Simul-
determination model
Input Num KeyFrame E N: Number key-frame Input BeamW W idth N Beam width
Input Video List of frames in a video
Captions
2: for each frame Videos do aagiiissppodddnnngggggoiiggggaar
3:
4: end for Candidates
Bewins +- []. "emmeeFramec do
for
G.3 Definition of noise for pseudo Multi- VidSundatsse pretraining
each Beamnaamss Beasicaption Beams
9:
LastT ime Beamtrnneel time
10:
for each frame, coption V ideos, Captions
The formulation of the noise for pseudo Multi- VidSumdataset pretraining is as follows:
do
11:
if frome <= Last imm then continue
12:
Noise EBx,
(15)
13:
end if Impaltames
14:
15:
Beamtisaamss [fame)]
where e is scalar value obtained by averaging all the values of the image feature in the mini-batch. a parameter that controls the magnitude of the noise, and in this experiment, ,3 was unified to 0.05. x is sampled from a normal distribution A((0 1) for each frame.
16:
Inpuleaptions
17:
Beameaptions + [caption]
18:
score = Trons =Tronsformert
19:
Inputiramee Imputcaptions
20:
21:
Candidates. oppendd (score, Inputtiammes Imputeapsions)
22:
23:
24:
end for end for
G.4 Model and Training Configurations
25:
G.4.1 Hyperparameters and Training Configurations
26: Beams +-
27:
CooreTopK
(eaamWidth)
28:
end for
Table 8 presents the hyperparameters we use dur- ing training. For the parts where the architecture remains unchanged, the initial parameters are taken from the pretrained Flan-T5.17 This is expected transfer the language generation ability acquired through pretraining on large amounts of solve this task.
29:
return Bcams[0]
Also, as mentioned in Section 5.3.1, we cre- ate the pseudo Multi- VidSum dataset and the fine- tuning dataset through random sampling. Conse- quently, we use different random seeds to construct
We use the published available model on Hug- gingface Transformers, https: pettps:/hhuggingfaae co/ooogle/ 1lan-55-bal

---

Pseudo Pre-training (MS COCO)
G.4.2 Model Architecture details
Number of Training Data
100,000
Gate mechanism The formulation of the gate mechanism as follows:
Number of Validation Data Optimizer
1,000
AdamW
81 0.9
pl (w0++1 ( = Gate(wr (tato(ws)Ptramal Pframe (1t
82 0.999
== x 10
+ Gatc(wi)) (wi+1 (wss)
Learning Rate Schedule
Cosine decay
Warmup Steps Max Learning Rate
1000
{
if ut <bos>
Gate(w))
0.00001
Dropout
otherwise
Batch Size
192
(16)
Number of Epochs
30
Pseudo Pre-training (Visual Storytelling)
where wy is the input token (frame or text token) at time step i, Pframe and Ptext are the probabili- ties of predicting frame index and text token, respectively, and Gate(w;) is the gate function that controls the prediction modalities.
Number of Training Data
39,553
Number of Validation Data
4931
Optimizer
AdamW
81 0.9
Ba 0.999
x
Learning Rate Schedule
Cosine decay 1000
Pointer mechanism The formulation the
Warmup Steps Max Learning Rate
pointer mechanism is as follows:
0.00001
Dropout
0.1
Batch Size
192
Ptrame (t)
(17)
Number of Epochs
30
= sootmax (oooocCCossheaaa Wdec: Hene W Wene));
Fine-tuning
Number of Training Data
23,216
where (tramo(t) is the probability distribution of the keyframe at time step hdee, is the last hidden state of the decoder at time step I, Hene is the all last hidden states of the encoder, and W Wdees Wene are the weight matrices.
Number of Validation Data Optimizer
2902
AdamW
81 0.9
8g 0.999
6 x 10
Learning Rate Schedule
Cosine decay 500
Warmup Steps Max Learning Rate
Other changes In addition to integrating the pointer mechanism and gate mechanism, several minor architcctural changes are made. Specifically, We set the maximum input sequence length of the model to 2048 to handle long videos. 18 Also, to convert the dimension of the input image feature, include an additional linear layer. Moreover, indicated in Eq. 17, when predicting the keyframe, the model computes the cosine similarity between the hidden states of the encoder and the decoder. During this process, the hidden state undergoes L2 normalization. Similarly, when predicting the text token, the hidden state is also subject to L2 normalization. Specifically, this is formulated as follows:
0.001
Dropout
0.1
Batch Size
192
Number of Epoccs
200
Table 8: Hyperparameters and training configurations Simul-determination model. AdamW is a optiomizer proposed in Loshchilov and Hutter (2019)
the validation set. For the pseudo Multi-VidSun dataset constructed from the Visual Storytelling dataset, we employ the validation set images and captions defined in (Huang et al., 2016). During training, we select the model with the lowest loss the validation set and proceed to the next train- ing phase. Similarly, during evaluation, we chose the model with the lowest loss on the validation set obtained during fine-tuning. We conduct train- ing of the models using NVIDIA A6000 (48GB memory) and A100 (80GB memory).
Proken (1) = softmax(norm(h WT (T...))))))) (18)
where (t) is the probability distribution of text tokens at time step t, h4 is the hidden state of the decoder at time i, and Wvocab is the weight matrix at the decoder head.
The T5 model urchitecture does not have limit on the input sequence length, hut is necessary to set limit for the pointer mechanism and keyframe prediction.

---

Keyfrume A0 AKMes
Cuption
performance was extremely degraded, we adopted
BLEUKT METEOR
all the changes.
vanilla.
37.59
79.40
37.74
14.14
vanilla Juss function
36.79
78.95
38.00
13.97
vanilla pointer mechanism
39.24
79.
36,88
vanilla gate mechanism
36.73
79.56
37.69
14.10
Table 9: Results of the ablation study for Simul- determination model. "vanilla" refers to model trained using the architecture of Flan-T5 (Chung et al., 2022), which is the base architecture of Simul- determination model, with a pross-entropy loss. + loss function indicates the performance when the loss func- tion changed as described Section G.4.3. + pointer mechamism and gate mechanism indicates the per- formance when the model's architecture is changed as described in Section G.4.2. This result is for case where InstructBIIIP (few-shot) is used the image cap- tioning model. For readability purposes, all values are displayed multiplied by 100.
G.4.3 Loss function
The loss function, denoted as L, can be formulated follows:
C(,, x, Yigh, Zigt) aCTrramm (0, Vigi) (19)
+ (1 CL)p (0, 2ogt),
where CL is cross-entropyyloss, 0 is the model parameters, x is a input video, Yigt is the target keyframe, and Zigt is the target caption. a is a hy- perparameter that balances the keyframe and cap- tion importance. We specified as 0.5 in this study because we considered that keyframe selection and caption generation performance are equally impor- tant in this task.
G.4.4
Ablaition study for architecture and loss function changes
We conducted an ablation study to examine how the changes in the architecture and loss function of Simul-determination modelaffect the performance. Table shows the results of the ablation study. Note that these models are trained only on the Multi- VidSumdataset and not on the pseudo video dataset. As a result of the experiment, the change in the loss function described in Section G.4.3 improved the performance by 0.3 in BLEURT, the introduction of the pointer mechanism described in Section G.4.2 improved the performance by 1.7 in AKMes. and the introduction of the gate mechanism improved the performance by 0.16 in AKMex. However, there was no architecture change that resulted significant improvement in performance across all metrics. On the other hand, since there was no case where the


Reviews:
[{"review_id": "4T82Qbl3Hb", "reviewer": "Reviewer_ZpMr", "summary": "", "questions": "1. Nice work. Have you thought about how the new task you propose might vary across various content genres? For example, a sitcom often has little visual variation but has high textual variation while a sportscast might have very high visual variation. Just a question, not a requirement.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback.\n\n## Question about the influence of the video genre\nThank you for your insightful question.\nWhile we have not conducted a quantitative analysis, there were noticeable qualitative differences in the task's difficulty depending on the video genre. \nFor instance, as you pointed out, in genres like sitcoms, the model frequently generated nearly identical captions repeatedly. \nWe think this difficulty arises from the limited visual variations within the video, making it challenging for the model to interpret the contextual situation accurately (l.1037-1039).\nConversely, we observed a tendency for the model to generate appropriate captions in genres like sportscasts.\nWe expect that in genres characterized by limited visual variation, such as sitcoms, there is a potential for generating more appropriate captions using the audio information embedded in the video. This is our next direction for enhancing our video summarization system, but it exceeds the focus of the current paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Thorough literature review including papers from pre-deep learning days.\n2. Convincing new task developed through clever exploitation of existing dataset.\n3. Solid and insightful baseline methods.\n", "reasons_to_reject": "1. No significant weaknesses to report.", "questions_to_authors": "1. Nice work. Have you thought about how the new task you propose might vary across various content genres? For example, a sitcom often has little visual variation but has high textual variation while a sportscast might have very high visual variation. Just a question, not a requirement.", "paper_topic_and_main_contributions": "This paper proposes a new video summarization task consists of jointly generating keyframes and appropriate captions so as to create a good summary consumable by a human being. The proposed approach cleverly exploits an existing dataset by adding additional attributes to the keyframes. The proposed baselines are sound and insightful. The proposed new task and the dataset are convincing, and set up a rewarding avenue for future research.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing references.", "Typos_Grammar_Style_and_Presentation_Improvements": "No typos that I noticed."}, "extra_text": {}}, {"review_id": "RQUBwEoIGp", "reviewer": "Reviewer_wL5Q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments.\n\n## Validity of keyframe selection method \nThank you for your feedback.\nIn the iterative refinement model (sec 5.1), **the median frame is used only in the initial step** of the iteration (l.342-343).\nIn the following steps, the keyframe selection module predicts a keyframe for generating captions (l.344-346).\nThe keyframe selection module, an LSTM-based model, takes a segment (a portion of a video) and a corresponding caption pair as inputs and predicts one single keyframe (l.386-393).\nThrough an iterative process of predicting keyframes by the keyframe selection module and re-captioning by the image caption module (l.375-385), we expect a convergence toward more appropriate selections of keyframes.\nIndeed, through the four cycles of this iterative refinement process, we observed enhancements of $+0.54$ points in the caption score (BLEURT) and $+4.90$ points in the keyframe score ($\\texttt{AKM}_{\\texttt{ex}}$). See Table 5 in Appendix F for more details.\nWe hope our answer dispels your concern about not validly selecting keyframes in the iterative method (our baseline system).\n\n\n## Is it difficult to scale up the model?\nWe appreciate your insightful feedback.\nRecent models developed for addressing video-related tasks require the use of all frames from the video as inputs[1, 2, 3]. \nFollowing these previous studies, we have adopted the format of taking all frames as inputs to maintain the simplicity of our proposed model as a baseline system.\nTherefore, as you pointed out, we acknowledge the potential emergence of scalability issues.\n(Note that this challenge is not unique to the model for our task but rather a common concern in video-related tasks.)\nHowever, the proposed Simul-determination model (sec. 5.2) can work in the same algorithm without taking all frames.\nHence, the scalability issue could be mitigated by incorporating a model that identifies salient frames from the video, as you suggested. \nAlternatively, we can use an existing method that selects distinct frames[4] to alleviate this issue.\nWe hope the reviewer will accept that we keep this scalability issue as a future issue, as it is beyond the difficulty level to solve as the first attempt at our proposed video summarization task since it is not a specific issue of our task, but a common issue for all video-related tasks. \nIn the camera-ready version, we will discuss this scalability issue in the limitation section. \n\n## Missing ablation study\nWe appreciate your perceptive comment. In a preliminary experiment, we conducted an ablation study regarding the architectural modifications.\nAs a result, for instance, we confirmed the effectiveness of employing the Pointer mechanism (l.445-457) by observing $1.7$ points improvement in keyframe selection score($\\texttt{AKM}_{\\texttt{ex}}$).\nAlso, through the modification of the loss function (l.459-467), we observed an improvement of $0.3$ points in the caption score (BLEURT).\nWe will include the details of this ablation study in the camera-ready version.\n\n\n\n[1] Khushboo Khurana and Umesh Deshpande. 2023. Two stream multi-layer convolutional network for keyframe-based video summarization. Multimedia Tools and Applications, pages 1–42.\n\n[2] Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mohit Bansal. 2023. Hierarchical Video-Moment Retrieval and Step-Captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition\n(CVPR), pages 23056–2306.\n\n[3] Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. 2022. End-to-end Generative Pretraining for Multimodal Video Captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17959–17968.\n\n[4] Elyas Rashno and Farhana Zulkernine. 2023. Efficient Video Captioning with Frame Similarity-Based Filtering. In Database and Expert Systems Applications, pages 98–112.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. paper is well-written and chronologically significant.\n2. The author collected the dataset for further research in the community.\n3. Idea of the Simul-determination model is novel due to its simultaneous generation and performance.", "reasons_to_reject": "1. For iterative refinement, there is no evidence why only median frame caption works better for selecting keyframes. probably, a better way\nwould be sampling based on weights (i.e. importance sampling)\n2. section 5.2.1, feeding all frames of a video to the encoder may cause an issue in scaling up the model. it would be beneficial if there is a single model to select frames from the video for generating salient keyframes which can be used in both baselines.\n3. ablation study for the architectural changes is missing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a framework for selecting relevant frames and corresponding caption generation from videos that can generate a summary of the video. Two main challenges the author considers for this task are the method of selecting the frame from video segments(iterative and simultaneous approach), caption pairs, and evaluation of the task (aligned keyframe matching score for frames and  METEOR for caption). For this task, the authors also contribute to a new dataset.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NxHYCah7J5", "reviewer": "Reviewer_nTTm", "summary": "", "questions": "- Selected keyframes may be subjective to annotators. \n- Can authors check the performance with previous dense captioning models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. \n\n## Importance and difficulty of this task\n**Difficulty:**\nThe primary difference between the Multi-VidSum task and the Dense Video Captioning task is that the main objective of the Multi-VidSum task includes summarization.\nIn the Dense Video Captioning task, the model is expected to generate captions for **every** event within the video.\nIn contrast, the Multi-VidSum task requires **selecting** scenes that are important for comprehending the content of the video.\nTherefore, the Multi-VidSum task presents a **different type of difficulty** from the Dense Video Captioning task.\nA similar relation exists between text summarization tasks and event extraction tasks developed in the NLP field.\nWe believe the reviewer will agree that these tasks (text summarization task and event extraction task from text) have essential differences and difficulties, and we hope this statement dispels the concern of the equivalent difficulty between our task (Multi-VidSum task) and Dense Vide Captioning task.\n\n\nAs the Multi-VidSum task mainly focuses on summarization, we also mention the difference between this task and existing video summarization tasks.\nIn this task, the systems are required to perform summarization through two modalities, which sets it apart from the other video summarization tasks.\nThis is a more challenging task setting than unimodal summarization tasks such as[1, 2].\nSpecifically, to generate appropriate captions, selecting keyframes well-aligned with the desired content (and vice versa) (l.67-69).\nConsequently, the system must consider these interdependencies and make optimal choices for keyframes and captions, making it challenging (l.69-72).\n\n\n**Necessity, importance, and novelty:**\nThe necessity, importance, and novelty of the Multi-VidSum task stem from its orientation towards **more practical** scenarios compared to the Dense Video Captioning task.\nFundamentally, the objective of the video summarization task is to enable users to comprehend the video's content within a short time.\nTherefore, video summarization is foreseen to have practical utility within video streaming services, and these summaries are expected to be consumed on devices with constrained display areas like smartphones and tablets (l.144-147).\nGiven the screen size constraints on these devices, displaying comprehensive video descriptions as required in the Dense Video Captioning task is redundant. \nSimilar discussions and solutions have been made in the document summarization task[1] (footnote 2).\n\nAlso, the current video summarization tasks[2, 3] often deal with outputs of a single modality.\nThis is expected to have low informativeness, making it unsuitable for practical applications.\nIndeed, previous research has reported that multimodal summarization yields higher user satisfaction with the summaries compared to unimodal summarization[4].\nTherefore, in the Multi-VidSum task, the resulting summary will be more easily understandable for users by providing both keyframes and captions as outputs.\nOther video-related tasks[5,6] also require multimodal output.\nHowever, these tasks are considered insufficient for practical scenarios since the sentences in the summary are created by retrieval-based methods.\nOn the other hand, the input of the Multi-VidSum task is only the video, and the captions of the output are generated by the model, making it a more practical task setting.\n\n\n\n\n## Effect of subjectivity on annotation\nAs mentioned in section 4.2, we conducted a reliability check as a post-evaluation of the annotation results.\nIn the reliability check, a trusted **third-party** annotator (not crowd workers) evaluated whether each annotated keyframe corresponds to the caption (l.266-269).\nAs a result, we found that only $1.76$% of the captions were inappropriate (l.286), indicating that the **impact of subjectivity on the keyframe annotation is limited**.\n\n\n\n## Performance check with existing dense captioning models\nWe greatly appreciate your suggestion.\nUnfortunately, we cannot simply apply existing dense captioning models to our task, so the performance of existing dense captioning models is unavailable.\nThis limitation is because the current dense captioning models[7, 8] are designed to generate captions for all events in a video comprehensively.\nThey **lack functionalities** to **select $N$ important scenes (keyframes)** and to **extract keyframes**.\n\n## Reproducibility\nUpon acceptance, we will **provide our data and code as stated in footnote 1**.\nThis allows all researchers to validate our paper's experiments using our code.\nWe believe many papers guarantee the reproducibility viewpoint in the same manner (by releasing the corresponding source code and data).\nTherefore, our paper should be equal to others in the reproducibility viewpoint and not extraordinarily low.\nWe kindly request that reviewers reconsider this point and grade our papers appropriately.\n\n\n\n[1] P. Over and J. Yen. 2003. An Introduction to DUC2003: Intrinsic Evaluation of Generic News Text Summarization Systems.\n\n[2] Evlampios Apostolidis, Georgios Balaouras, Vasileios Mezaris, and Ioannis Patras. 2021. Combining Global and Local Attention with Positional Encoding for Video Summarization. In  IEEE International Symposium on Multimedia (ISM), pages 226–234.\n\n[3] Pinelopi Papalampidi and Mirella Lapata. 2023. Hierarchical3D Adapters for Long Video-to-text Summarization. In Findings of the Association for Computational Linguistics (EACL), pages 1297–1320.\n\n[4] Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2018. MSMO: Multimodal Summarization with Multimodal Output. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4154–4164. \n\n[5] Bo He, Jun Wang, Jielin Qiu, Trung Bui, Abhinav Shri- vastava, and Zhaowen Wang. 2023. Align and Attend: Multimodal Summarization With Dual Contrastive Losses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14867–14878.\n\n[6] Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S.Kankanhalli. 2020. Video storytelling: Textual summaries for events. IEEE Transactions on Multimedia, 22(2), pages 554–565.\n\n[7] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-Captioning Events in Videos. In IEEE International Conference on Computer Vision (ICCV), pages 706–715.\n\n[8] Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023. Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10714–10726.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors introduced a multimodal video summarization task and dataset contributing to multimodal research.\n- The manuscript released a new benchmark dataset for the task of Multi-VidSum with carefully designed refinement and refinement processes. Furthermore, the evaluation criterion for Multi-VidSum is straightforward and simple.", "reasons_to_reject": "For now, it doesn't seem clear that a multimodal video summarization is more difficult and novel than dense video captioning. The authors should explain the necessity and importance of their proposed task and datasets.", "questions_to_authors": "- Selected keyframes may be subjective to annotators. \n- Can authors check the performance with previous dense captioning models?", "paper_topic_and_main_contributions": "This paper proposes a multimodal video summarization task and dataset. They extended Activitynet captions to allow existing models to perform multi-modal video summarization. The authors preserve the dataset's quality and show the proposed task's evaluation process with baseline methods.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a framework for selecting relevant (key-)frames and corresponding captions, generated from videos that will constitute a summary of the video. The authors formulate the task as two main challenges: (1) the method of selecting the frames from video segments (solved using iterative and simultaneous approaches), and (2) evaluation of the task (aligned keyframe matching score for frames and METEOR for caption). The authors also contribute a convincing new dataset by adding additional attributes to the keyframes of existing data. The proposed baselines are sound and insightful, setting up a rewarding avenue for future research. The paper is overall well written, and authors commit to updating the paper in response to a meaningful discussion with reviewers.
Decision: Accept-Main