Sparse Frame Grouping Network with Action Centered for Untrimmed Video Paragraph Captioning
Authors: Guorui Yu
Yimin Hu
Yuejie Zhang
Rui Feng
Tao Zhang
Shang Gao
Venue: EMNLP 2023 Findings (2023)
Keywords: video paragraph captioning
transformer
grouping
action centered
contrastive learning

Abstract: Generating paragraph captions for untrimmed videos without event annotations is challenging, especially when aiming to enhance precision and minimize repetition at the same time. To address this challenge, we propose a module called Sparse Frame Grouping (SFG). It dynamically groups event information with the help of action information for the entire video and excludes redundant frames within pre-defined clips. To enhance the performance, an Intra Contrastive Learning technique is designed to align the SFG module with the core event content in the paragraph, and an Inter Contrastive Learning technique is employed to learn action-guided context with reduced static noise simultaneously. Extensive experiments are conducted on two benchmark datasets (ActivityNet Captions and YouCook2). Results demonstrate that SFG outperforms the state-of-the-art methods on all metrics.

Sparse Frame Grouping Network with Action Centered for Untrimmed Video Paragraph Captioning
Guorui Yu!, Yimin Hu', Yuejie Zhang', Rui Feng', Tao Zhang", Shang Gao Schh. of Comp. Sci, Shanghai Key Laboratory of Intelligent Information Processing,
Fudan University
Sch. of Inf. Manag. & Eng, Shanghai Key Laboratory of Financial Information Technology, Shanghai University of Finance and Economics, 'Sch. of Inf. Tech, Deakin University 212102404096m. fudan. edu. cn, ( yjzhang, fengrui J@fudan, edu. cn, taozhang@maiil shufe. edu. cn, shang. gaoedeakin. edu. au
Abstract
Captions
Captions
Generating paragraph captions for untrimmed videos without event annotations challenging, especially when aiming to enhance precision and minimize repetition at the same time. To address this challenge, we propose module called Sparse Frame Grouping (SFG). It dynam- ically groups event information with the help of action information for the entire video and excludes redundant frames within pre-defined clips. To enhance the performance, Intra- Contrastive Learning technique is designed to align the SFG module with the core event con- tent in the paragraph, and Inter-Contrastive Learning method is employed to learn action- guided context with reduced static noise si- multaneously. Extensive experiments con- ducted on two benchmark datasets (ActivityNet Captions and YouCook2). Results demonstrate that SFG outperforms the state-of-tha of-the-art art meth- ods on metrics.
Caption
Gonerator
Castion Generator
Sparse Frame Grouping
Event Annotattons
Frames
Frames
Our Mathod
Traditional Methods
Figure 1: Comparison of SFG with traditional methods.
As a common frame sampling strategy in most existing works, random sampling method randomly selects one frame every one two seconds. The hypothesis of previous works is that the frames in one second are highly similar, so random sampling does not result in significant information loss. How- ever, actions often occur rapidly, such as within a one-second time frame, and usually lead changes to events. Random sampling may miss crucial action frames that occur quickly and retain many similar frames as noise. According to (J. Lei and Mohit, 2022), or two key frames are sufficient cap- ture important content, and the "redundant" frames can be neglected in an action clip. In this way, we can learn the visual content from a small number of frames that contain critical information.
1 Introduction
Video Paragraph Captioning for untrimmed video aims to describe multiple events in the video with three or four sentences. Initially, the focus was on describing each event with single sentence in long video (J. Lei and Mohit, 2020) (J. Mun and Han, 2019), until (Y. Song and Jin, 2021) pro- posed to directly generate paragraph captions with- out event ground truths. This refined approach to paragraph captioning is more practical but also more challenging. Firstly, without event labels, becomes difficult to precisely locate the transitions between different events and to summarize the com- plete story from holistic perspective. Secondly, most prior studies generated one sentence for each event, whereas paragraph captioning without event ground truths requires generating sentences that are two or three times longer. This naturally exac- erbates issues of redundancy and repetition. Our work focuses on video paragraph captioning with- out event annotations as shown in Fig. I.
Therefore, our objective is to learn event distribu- tions guided by action frames, and utilize them as estimations for event locations, enabling us to con- struct comprehensive paragraph stories by incorpo- rating these momentary yet important actions. Con- sidering the impracticality of including all frames (often averaging hundreds) into the analysis, it be- comes essential find an effective way for frame selection and information grouping. Our work specifically focuses on action-driven video para- graph captioning. It tackles the captioning task by utilizing sparse frames and refined video an-

---

chors with action visual representation. No event annotation or any other additional information is used except for the video and paragraph captions, thereby ensuring the practicality of our method.
paraal-p visual-triplets, while the neg- ative pairs consist of unmatched pairs within the batch. For each sentence in the paragraph, triplet (s, v, o) (Sec. 3.2) is extracted and the embeddings of all triplets are averaged. This contrastive learn- ing approach is referred to as Intra-Contrastive Learning, and its purpose is to minimize the dis- tance between the overall video anchors and the paragraphs.
We propose our vanilla model based on the Transformer architect (J. Lei and Mohit, 2020). A new grouping module, named Sparse Frame Grouping (SFG), is introduced. It uses designed anchors as queries to select and group useful frames. The objective is to extract two types of content from the video: event distributions and action changes between adjacent frames. The idea is to progressively estimate the actual event distri- butions by grouping visually similar video frames using these designed anchors.
of
Besides, crucial to differentiate the video anchors from other visual contents in the video. The Inter-Contrastive Learning method takes the left video representation as a negative visual part. It requires the distance between the video anchors and the triplet embeddings is at least smaller than the distance from the negative visual part to the triplet embeddings. We conduct experiments on ActivityNet Caption and YouCook2 datasets. The results demonstrate that our model outperforms the state-of-the (SOTA) methods when evaluated using the event ground truths on both datasets.
At each encoder layer, SFG is first applied to ev- ery two frames and selects one informative frame, where anchors are random frames. As a result, a se- quence of sparse and important video frames are ob- tained, instead of using the randomly sampled ones. This process is referred to as clip-level SFG. Next, SFG is employed to group event context across the entire video. Video anchors are initialized using triplets of ground truths during the training phase. The objective is to ensure that the anchors capture the action-drived event distributions in the video, while the triplets provide a concise representation of the main action content in the sentences. This process is referred to as video-level SFG. At this stage, the video anchors are updated by grouping the newly selected frames. The remaining steps follow the standard transformer computation. In the decoder, we modify the cross-attention mecha- nism by attending video anchors in the video-level SFG additionally. The weighted anchors, along with the weighted video frames, guide the decoder to generate each word at each decoding step.
The contributions of this work are follows: (1) We propose Sparse Frame Grouping (SFG) module to estimate event locations and catch action- driven visual content. It operates at both the short clips and the whole video levels without relying on event annotations, thereby enhancing the practical- ity of video paragraph captioning.
(2) We build an Intra- and Intro-contrastive loss for SFG training. It helps the model focus more on learning event distribution rather than other noisy contextual information, reducing redundancy and improving precision of the captions.
(3) Our model achieves SOTA results on bench- mark datasets ActivityNet and YouCook2, outper- forming even the methods using event annotations.
2 Related work
To train SFG without using event annotations, we introduce an Intra- and Inter-Contrastive Learn- ing technique. The objective is to ensure that the video anchors are close to their relevant event contents in the video. This process naturally in- volves contrastive learning between anchors and paragraph. The challenge is that the embedding of the entire paragraph contains excessive noise for the video anchors which are intended to rep- resent the center of events. What aim learn is a summary of the event context within the para- graph, while disregarding those static descriptions that are not relevant to the actions in our video an- chors. We modify the positive contrastive pair from
Dense Video Captioning As a multi-modal task, has attracted much attention. Popular catego- rization of dense video captioning falls into three: predicting event location and sentences, solely pre- dicting sentences with event annotations, and pre- dicting paragraphs without event annotations. In this paper, our focus falls into the third category. (L. Zhou and Xiong, 2018) is the first adopting transformer decode sentences and create differ- entiable masks for the consistency between propos- als and captions. (J. Mun and Han, 2019) used a pointer network to select possible event proposals from candidates and to recurrently generate sen-

---

lence for each event. (T. Wang and Luo, 2021) firstly tackled this task as parallel multitask. Lo- calization and caption generation are predicted at the same time using a DETR scheme. (P. Jae Sung and Anna, 2019) adopted adversarial inference to evaluate the paragraph quality. (J. Lei and Mohit, 2020) proposed a memory module in decoder to remove redundant visual and text information dy- namically. These prior studies all rely on event annotations. (Y. Song and Jin, 2021) focused on reducing redundancy using an adding and erasing module at each decoding step without event anno- tations. However, little attention have been paid to video redundancy itself at the encoder stage. In our work, we aim to address this issue by tackling it from the initial frame selection stage and capturing the core action content of the video.
j-th word in the length paragraph, and dw is the dimension of word embeddings. Here we build our vanilla model based on the transformer struc- ture, following the approach described in (L. Zhou and Xiong, 2018) and (J. Lei and Mohit, 2020). The encoder transformer E' is used to learn the video representation V. from video input V and the decoder transformer D generates the captions conditioned on Ve. Specifically, first adopt N- layer transformer encoder to model the long range interactions:
V++ = FFN(V, + MultiHead( (v, V3, Vi)) (1)
where V, is the output of i-th transformer layer, FFN( and M Multillea(() denote feed-forward network and multi-heas attention as in (L. Zhou and Xiong. 2018). For the decoder, we first use the masked multi-head self attention (L. Zhou and Xiong, 2018) 88 to model the hidden states of the last layer, then we introduce cross attention to fuse the text modality and the visual modality. For cross attention, we use the text embeddings at each de- coding step as the query, and Vn from the last layer of the encoder as key and value, so that each word
Video Action Understanding Action localiza- tion video is an important task for video under- standing. (Yeung et al., 2016) and (Zhou et al., 2018) utilized reinforce learning to dynamically determine the next frame. (Z. Yuan and Gangshan, 2021) proposed to sample motion from cumula- tive distribution. (B. Korbar and Torresami, 2019) proposed to understand video only through some salient clips which were ranked and selected for further understanding. (Gowda et al., 2021) se- lected single frame and utilized high level feature for video classification. The previous works sel- dom combine the two modalities (visual and text) together. Our work utilizes both to better select key frames.
can be generated based on the previously predicted words and the attended vidco contents. The vanilla model is trained typically by the Maximum Likeli- hood Estimation (MLE) loss:
T Σ 1=0
Linle
log Dogp(uf We V.)
(2)
T
3 Methods
The proposed Sparse Frame Grouping Network is shown in Fig. 2. It consists of three modules: clip- level SFG, video-level SFG, and Intra- and Inter- Contrastive Learning. Together with the vanilla transformer, these modules generate action-driven and event annotation-free paragraphs. this sec- tion, the vanilla transformer first described, fol- lowed by the introduction to the three modules.
T
1
T'mle
log log 12 Vo)
(3)
T
1=
where Ve is conditioned video representation. yč is the predicted word at the t-th step and is the length of caption. Without event segment annota- tions, there are three challenges in video paragraph captioning. Firstly, directly decoding the entire video usually leads to confusion in recognizing clear contexts and decoding each into word out- put. Secondly, the random frame sampling strategy results in the loss of essential key information, af- fecting the decoder's ability catch action changes and decode precise words. Lastly, the MLE loss tends to generate high frequency and repetitive words in previous steps.
3.1 Vanilla Model
Given untrimmed video V = ("",,.., Un), ), U/ E Rd. where & is the i-th video frame representa- tion (referred to frame hereafter), is the total number of frames in V. and d, is the dimension of video frame. The video paragraph captioning task aims to describe the content V with para- graph Y = (,),..., yk), Uj E Rdx, where yj is the

---

nowing ho lawn
Vib PHEAO
CA min
nowig.
Leal
Lins
ArgMax
SPa
Sc9n0 hune Dwarr
ORO CO CBBCBORCOOO 18
Ieee
Free
Frars Groging
SAdr Head Alperion
Video anthoss
Vilro take..
Tran'lare Slace
& Hars
12ed
Spara Frane Groging
Peed Powwer
10 03 0 20
Masi
ArgMMx
Clip level
Test arbectting
Tran Blax
Soow
Freer
Irew
Yr-1
esee
'bdd
boniy
Linear
REDGERS DOD DEGEEDUOR 20nd
every
Edl
Figure 2: Overview of SFG Network. (a) Architect of the network. SFG is applied at both the clip and video level in every encoder layer. The decoder dynamically generates words based on previous video anchors and selected video frames. Intra- and Inter-Contrastive Learning is introduced for video anchor training. (b) Detail of SFG.
3.2 Progressive Grouping Transformer
be embedded as {ys: yo Yoke 3o ys is subject, M/l1 ...y Yokk one verb or verb phrase. and yo is object. We sum the word embeddings in the triplet and project them to the video representation space: af - Linear(E Mtriplet), af E Rd is one anchor among the video anchors. By initializing the left vidco anchors and setting the number of anchors to be equal to the number of sentences in the captions, the anchors can effectively group the content with action-related visual features and learn the event distributions through different anchors. We get the updated video anchors A and the selected video frames via:
As shown in Fig. 2(a), based on the vanilla trans- former (Sec.3.1), instead of directly feeding all the video frames into the transformer encoder, we incorporate the clip-level sparse frame group- ing (SFG) module and video-level grouping mod- ule prior to each transformer layer, denoted as SFG and SiCvio We employ dynamic to- kens, known as anchors, to selectively choose use- ful frames from the input or the previous trans- former layer. The objective of these modules is to obtain set of sparse video frames and a collection of video anchors that can learn event distributions dynamically.
1 viden SFGrideo Abridoo (4)
Formally, we take the I-th transformer layer as an example and have the video frames yl as the input. Firstly, we randomly select one frame from every two frames as anchors for SFG p4 SFG (Aspp)))) where N'sp rep- resents anchors at the clip level. p is the half length of the input frames. To help SFG learn the action-driven event context, we initialize the video level frames via triplet embeddings. Assum- ing that triplets highly summarize the main con- tents of the sentence in captions, we extract one triplet (L. Wang and Svetlana, 2016) from each sen- tence and project to the same latent space as the video frames through linear projection. E.g., one triplet from a sentence is < 8,U,0 >, which can
v41 Trans fromer Layer(Vh)), (5)
where I' is the outcome of ArgMar, which is the index of center frames at this stage. We adopt the video frames selected by SFG and forward them to 1-th transformer layer. With the stacked encoder transformer layers and SFG modules, we get set of sparse video frames with fewer redun- dant frames and the most representative anchors. These anchors already group the core content of the events in the video.
3.3 Sparse Frame Grouping Block
As shown in Fig. 2(b), SFG utilizes the dynamic anchors to select the center frames for each anchor

---

and group action-driven event content around the updated anchors.
batch are negative pairs. The Intra-contrastive loss
Lintra La-t1 Li-ba as follows:
Formally, we take anchors
{a1,...,mml am),a. E Rd as the query, and the video frames V as the key and value pairs to compute the weights of video frames. To select target frames, we change the original SoftMax to ArgMar and weight matrix to one-hot matrix. Then we select the center frames and obtain the updated anchors Å through the ArgMaz ar function:
Th eapl A T.)/T
Lo-st =
log
(8)
Tb
j=1 exp(A, -T)/T
==1
Tb
exp(T; Ai)TT Σ (T; Aj)/T
Le-
=
Σ Tb :=1
log
(9)
where To is the batch size. We use dot produc- tion to measure the similarity between A and T. The loss aims to ensure that all the anchors cap- ture similar contextual information the possi- ble events described in the captions. However, reduce redundancy in the generated captions, we also need to secure that the video anchors learn less static noise. Therefore. we further apply Inter- Contrastive Learning to guarantee that the video anchors are at least closer to the triplet embeddings compared to the surrounding visual features. For negative pairs, we select the frames that are not cho- sen by SFGridee in the last transformer encoder layer, which is A, (1 14) x V. Thus the negative visual pair becomes An, T >. The pos- itive pair the Contt-oonte Learning is the same as that the Mara--ontraative Learning. The Inter-contrastive loss is as below:
Á = A+ ArgMar(Ax x)
(6)
where A represents the anchors discussed in Sec. 3.2. However, the onc-hot matrix from ArgMax is not differentiable. We further use Gumbel SoftMax (E. Jang and Ben, 2016) and transform Eq. 6 as follows:
A A+ cxp(A x V B) Σ crp A X B)
x V,
(7)
where Å represents the updated anchors. 3 sam- pled from the Gumble(0, 1) distribution to ensure that SFG can merge the video event information progressively during the training phase.
3.4 Intra- and Inter- Video Contrastive Learning
We obtain sparse video representation Ve with sig- nificantly fewer frames compared to the raw vidco and set of event context anchors after encoding. To effectively train the proposed video anchors, es- pecially in the absence of event annotations, we introduce an Intra- and Video Contrastive Learning technique. This technique enables the learning of event context, which is mainly action context-driven. Specially, we first employ intra- video contrastive learning to minimize the distance between the representation of dynamic anchors A and the text representation. However, the ground truth information may contain noise. For example, grouping frames should not include an excessive number of static or descriptive words such as color, position and background. We aim to extract some confined context from the captions that includes action description while excluding the "unimpor- tant" words. Therefore, triplets used (Scc.3.2). Assuming that short triplets highly summarize the main contents of the video, we denote the posi- tive pair as A, >, where 7 is the average of anchor representation and is the average of triplet embeddings. The unmatched pairs in the
Linter mar A o (10)
where a margin constant. This inter-video loss helps distinguish the representation of video anchors from the other parts of the video. The total training loss is:
Loss Lmite + X1 Lintra + XaLinters
(11)
where A and X2 are loss weights over zero.
Experiments
4.1 Datasets and Implementation
Datasets and Evaluation Metries: All the exper- iments are conducted on two widely used bench- mark datasets, i.e., ActivityNet Captions(K. Ran- jay and Juan, 2017) and YouCook2(L.. Zhou and J, 2018). For fair comparison, we use four mainstream evaluation metries, including BLEU-4 (B @4), METEOR, CIDER and R@4. A higher score for any metric indicates better performance in captioning. Besides, we evaluate repetition using R @4, following the previous work in (P. Jae Sung and Anna, 2019). lower R@4 score indicates

---

better performance, suggesting less repetition in the paragraph caption.
video. TDPC may learn the fragmented informa- tion scattered throughout the frames. In contrast, our model makes more efforts to address this prob- lem, resulting in more action-driven and organized descriptions. The uplift in METEOR score serves as evidence of a preference for longer sentences in our approach.
Implementation Details: For visual features, we utilize the video features extracted by ResNet-200 and BNInception, as provided by (L. Zhou and Xiong, 2018). The length of video anchors is set to 10 for ActivityNet Captions and 14 for YouCook2. In the inference phase, random frames are used to initialize the video anchors for Fiidoo For text features, words are initialized using Glove embed- dings. Text limited to 60 words. Empirically, the loss weight è is set to and A2 to 0.5. The batch size is 32. Greedy decoding is adopted which is comparable to beam search.
Tab. shows the performance on YouCook2 validation set. Our method achieves the SOTA performance, demonstrating a significant improve- ment over the other methods across all the metrics, Also, YouCook2 contains large number of events, which highlights the importance of event distribu- tions for caption generation.
4.2 Comparison with SOTA Methods
4.3 Ablation Studies
In Tab. 1, we compare model with SOTA methods on ActivityNet Captions. These meth- ods can be divided into two categories: (1) infer- ring without event annotations, including TDPC (CVPR21) (Y. Song and Jin, 2021); (2) with an- notations, including VTrans (CVPR18) 11PRR8)(C) (L. Zhou and Xiong, 2018), MFT (ECCV18) (Y. Xiong and Lin, 2018), GVD (CVPR19) 19) (L. Zhou and Mar- cus, 2019), GVDsup (CVPR19) (L. Zhou and Mar- cus, 2019), AdvInf (CVPR19) (P. Jae Sung and Anna, 2019), Transformer-XL (ACL19) (Z. Dai and Ruslan, 2019), MART (ACL20) (J. Lei and Mohit, 2020), PDVC (ICCV21) (T. Wang and Luo, 2021). It is worth noting that GVD, GVDsup and AdvInf adopt additional object features to align the descriptions and objects in the video. Naturally stronger feature leads to impressive performance. However, model outperformas them, regard- less of whether the object features are integrated or event annotations are used.
To verify the effectiveness of the proposed group- ing module and loss, we conduct ablation studies on ActivityNet.
Effects of SFG As shown in Tab. 3, Vanilla is the baseline with MLE loss. Vanilla + SFG refers to only SFG is used in every layer of the encoder transformer. Vanilla + SFGoodoo + Lintra + Linter is the complete module for our video-level SFG. The last row is the full model. The experiments report that: (1) SFG con- tributes to a reduction in repetition, as indicated by the improvement in R @4. R @@4 is improved by 38% and by 59% in rows 1 & and rows & 6. This suggests that the dynamic frame sampling per- formed by SFG not only benefits the selection of informative frame, but also results in sparse video representation, reducing the redundancy from the video side; (2) for SFGridoo in row 5, there are considerable improvements in the precision and repetition. This indicates the video anchors indeed learn to estimate the video content, reducing the static noise and, consequently the redundancy.
Without the event ground truths, our model still achieves the best performance @@4, Meteor and CIDER with relative improvement of 3.7%. 6.5% and 5.9% respectively compared to PDVC (on ActivityNet valv.l). The uplift achieved even with fewer frames and no event annotations. While TDPC also utilizes key frames, it relies on the pre- trained model to reconstruct the common represen- tation space between text and video. It should be the strong prior knowledge in our model that makes the generation more precise. Additionally, TDPC tends to generate shorter sentences, which may re- sult in lower METEOR score for AdvInf. This could be attributed to TDPC encoding all frames directly and overlooking event distributions in the
Effects of Intra- and Inter-Contrastive Loss In Tab. 3, Vanilla FF Lintra refers to only SFGvideo is used in every encoder layer with Lossmara- Vanilla + FFuddeo + Linter is to verify whether the distinction of anchors alone is sufficient to summarize the event distribution in the video, without the supervision of triplets. From the results, it can be concluded that Loss has greater impact on improving the model precision (B 4, METEOR and R @@4), while Lossinter sig- nificantly reduces @4. Specifically, when com- paring with row 2, row (Lossintra + (Sidde))) has improvements B @4 METEOR and CIDER

---

Table I: Performance Comparison on ActivityNec. ae-val. V. F. O: visual, flow and obj features. * utilizing stronger features. The italic number means they significant in hypothesis testing,
Event Annotation
ae-va!
Gentest
Model
Train
Infer
B&44
METEDR?
CIDER?
R.444
B841
METEORT
CIDER?
RR41
Zhang
VTranst... Zhou and Xiong, 2018)
Y
Y
V+F
9.75
15.64
22.16
7.79
9.31
15.54
21.33
7.45
Tansformer-XLL Dai Ruslan, 2019)
Y
V+F
10.39
15.09
21.67
8.54
10.25
14.91
21.71
8.79
MARTU. and Mohit, 2020)
Y
Y
V+F
10.33
15.68
23.42
5.18
9.78
15.57
22.16
5.44
POVOCT Wang and Luo, 2021)
Y
VoP
11.00
15.93
27.27
GVDLL
Marcus,
Y
GVDsupiL. Zhou and Marcus, 2019)
Y
Y
V-F+O
11.30
16.41
22.94
7.04
10.70
16.10
22.20
Advlof(P. Song and 2019)
Y
20.97
MPT(Y. Xiong and Lin, 2018)
Y
V+F
8.45
14.75
14.15
17.59
PDVOCT, Wang and Luo, 2021)
Y
VoF
10:24
15,80
20.45
TDPC" (V. Song 2021)
Vanilla Ours
N
N
V+F
11.23
15.34
25.23
7.38
10.87
15.21
25.12
8.23
N
V+F
12.24
I6.97
28.89
2.51
12.21
16.62
28.54
2.53
Table 2: Performance Comparison on YouCook2.
Table 4: Effects of Anchor Position Testing Phase.
Model
B04:
METEOR:
CIDER1
anchor position
B@41
METEOR!
CIDERT
R@41
VTrame7: Dai Ruclan, 2019)
33.26
Tandlomme-- XL/Z Dai and Roslan, 2019)
6.36
14.76
26.35
6.30
random sample
10.3
28.56
3.12
MARTO Lei and Mobi,, 2020)
8.00
15.00
35.74
4.39
uniform sample
12.24
16.97
28.89
2.51
Vanillla Ours
5.00
833
16.24
39.27
4.12
Table Effects of Frame Num. at Clip- and Video- levels.
Table 3: Ablation Studies.
Indos
Mosel
504
METBOR
CIDER
R.04
Clip Level
Video Level
B@4-
METEOR1
CIDER
R@4,
Yandltc
11.23
15:34
25:23
7.38
2
YardteeSSPGas
11.39
15:09
25
5.34
2
11.69
15.45
27.35
4,93
3
FandltonS SealltssSG-aaaeee Restree
11.42
15.90
26.76
3.89
3
8
11.54
15:23
27.21
4.12
FazullioSSPP SPC
11.58
16.00
27
4.67
2
10
12.24
16.97
28.89
2.51
VandltinSS SFG Hosate
hite VnnillonS Genge Cronn Fipera
3
10
11.86
16.01
27.54
2.49
baaeY anillas
8
VasiltesS Saalle-SPG.acee P.aaye Syy-SS D.Gasoo Tagtee Teate
setting 8/10 anchors in vidco-level SFG, It can be seen that selecting one video frame from every 2 frames for clip-level SFG and using 10 anchors in video-level SFG yield the best performance. When comparing 3 frames to frames at the clip-level, the precision decreases, but R @@4 is improved by 20% (in rows & 2). This represents a trade-off be- tween repetition and precision. For video anchors, using 10 anchors can cover most events in Activ- ityNet Captions while anchors may miss some. Also, we test different ways of select anchors in the testing phase, with randomly selecting frames as anchors using a random seed or uniformly selecting over total frames. The results are as Tab. and it can be observed that the random sampling method is not as effective as uniform sampling.
by 0.3%, 3.0% and 3.3% respectively, except for R @4 which is increased. We speculate that the increase in R@4 may be due the absence of SFGaip but the precision benefits from the su- pervision of Lossmtra. which is similar to the representation of triplets in the captions. As for Loss it boosts all the metrics, especially R@4, suggesting that Loss helps video anchors ex- clude many static visual content and reduce repeti- tion.
Effect of Anchor Number During the training phase, the initialization of the video anchors are only related to triplets from captions. However, dur- ing the testing phase, we currently select one frame at regular intervals from the video as an anchor. The difference is that the tokens in SFG are prede- fined in terms of quantity, and random frame is used the initial value. We conduct further analy- sis determine the optimal number of anchors for the clip-level and video-level SFGs. As shown in Tab. 5, clip-level refers to setting one anchor for every 2/3 video frames, while video-level refers to
Table 6: Effects of Anchors & Fusion in Decoder.
Decoder
B@41
METEOR;
CIDER1
R@ 4
w/o anchors
11.95
16.34
28.56
3.12
add concatenate
12.12
16.01
28.23
3.09
12.24
16.97
28.89
2.51

---

wit me moco and lomo toat vergt ont
dirt bunl
tt baand placee acquzze purtar slcand stage durnga party.. donn wontyy man fightn fight with the crnwd.
para for uoL. to wordh Than te shua lovid and cow.
Vonell del ovent annotuionas man plays quitar on stage hike band play music the mas teahis playing and worri shale handts The mun plas guitar thei people play guiter
by
speaiing
pawningg
andlandy ynu Borliig
Ours (wll ovent annokatonns A mon is proseing guitar with group of poople stand rg on stage, A mon give hise pultartin a rann The rain beginn to play the quitar in the crowd. i exanen claps her hards. The man catitens for the wornat.
Vark event walid back and with Mort vala sour the trek
group piople
jumpingg
hours
camens More
aoverdl other intardiona laiee place.
Violla (ol movet aemetetiond: A begg group of people are owen aitingg anound an indior poot with one man spesking to unothor
evar coee peop take bore Seed Dentict uvend unndelice perk Oummanr Geirninng
followee womnn apeaiding
the parningg She
horseff around and onds by jumpping down.
Figure 4: Qualitative Results2.
Gure twll word ammotetendd a Large group on people sttting and pecatle roaling lines, The junpinng
cameess womon and inseviowedd the
witching
decodar
decoder
GlIAA gymmiat seen stund before unonan baro while many wazdinng seaca The gir jumpa up and bogana perforiin a routric on the burs thi Hipa foreel iround and wide jumping down.
TuRINren thik
City Eipil
Yeler desuping
creetational
standing mota bike shee and standing the
Trenforer Plck
right
Ours lels ownnt annotations: go moso stands under inowon bard with ceople watching Sho jumps and po around jumds stands
Spars frims Greating
Treefrrec
Figure 3: Qualitative R.sultsl.
fone liruping
Soare Frane Grouping
Oe Levt
Copt
5 Qualitative results
Video frames
Vidco frames
Fig. 4 displays example captions produced by our model on ActivityNet Captions. We compare them with GT (the Ground Truth) and those generated by the Vanilla models. The triplets in all para- graphs are highlighted for comparison. Compared Vanilla, our model tends to generate more pre- cise and action-driven descriptions, as evident in the examples. Particularly in the third case, where the surroundings are relatively static and only the athlete is performing different actions, our model is able to differentiate between these actions in the neighbouring frames.
Figure 5: Different Model Structures.
step is used query attend to the anchors, and the weighted anchors are fused by concatenating them with the visual representation as V. This V is then utilized to guide the decoding of the next step word in the sequence. We verify the effectiveness of anchors in the decoder and compare different fusion ways including concatenating and adding. As shown in Tab. 6, attending to the anchors indced improves the relevance of the generated captions. Additionally, the results indicate that concatenation is more effective compared to addition.
For event distributions, in the first case, there are three events, and the second is momentary com- pared with the other two. The vanilla model fails to describe the second event even with annotation information. Our model, however, captures the ac- tion "is taken by" and decodes it to "give" in the generated description.
Effects of Different Model Structures We also adjust the placements of the clip-level and video- level SFG modules. In the left encoder struc- ture shown in Fig. 5, the order of the modules is SF SFG Gelip. SFGvideo and transformer layer. The block (SFGdip, SFGvideo transformer layer) is stacked twice in the structure. The right structure in Fig. 5 considers the information lost in the second SFG puts SFG first in the second layer
Effects of Fusion Ways in Decoder To utilize the dynamic anchors for generation guidance, we use the anchor attention in the decoder cross at- tention. Formally, the word embedding at the t-th

---

Performance Structures
is especially true when there multiple related yet distinct actions in the video, which may result in less accurate or consistent captions terms of semantic coherence.
Acknowledgements
This work supported by National Science and Technology Innovation 2030 Major Project (No. 2021ZD01 14001; No. 2021ZD01 14000), Na- tional Natural Science Foundation of China (No. 61976057; No. 62172101), the Science and Tech- nology Commission of Shanghai Municipality (No. 2151 I 101000; No. 22DZ1 100101), and the Funda- mental Research Funds for the Central Universities of China (No. 2023 I 10139).. Yuejie Zhang, Rui Feng and Tao Zhang are corresponding authors.
Figure 6: Performance of Model Structures. a and represent the left and right structures in Fig. 5, respec- tively.
transformer, and conducts FFGe after the last layer of transformer.
Comparing the two structures, the left outper- forms the right. This suggests that even though SFGeip discards some frames before SFGuideo. the key information is retained, allowing SFGeideo learn the event contributions. Also frame sam- pling helps remove noise, which benefits SFGvidoo grouping center context of events. Fig. 6 shows the performance of the two structures.
6 Conclusions
In this work, we present a network with Sparse Frame Grouping (SFG) enhanced Transformer via Contrastive Learning for event annotation-fiee video paragraph captioning. The SFG module is designed select and group informative frames and learn event distributions without relying on event annotations. Additionally, an Intra- and Inter- Contrastive Learning technique is proposed to train the SFG, enabling it to learn the overall event context in the video and distinguish it from the static context. Experimental results two bench- mark datasets demonstrate that SFGTransformer achieves SOTA performance. In the future, we plan extend the method longer video data such as movie data to further investigate the influence of key action visual information. And we also inves- tigate training method without triplets to simplify the pre-process of data.
Limitations
Video paragraph captioning requires semantic con- sistency between the generated captions and the video content. Although the "action-guided frame grouping" method provides some guidance for frame selection and grouping, there are still po- tential instances of semantic inconsistency. This

---

References
Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei. 2016. End-to-end learning action detec- tion from frame glimpses in videos. In Proceedings of the IEEE conference computer vision and pat- tern recognition, pages 2678-2687.
D. Tran B. Korbar and L.. Torresani. 2019. Scsampler: Sampling salient clips from video for efficient action recognition. In ICCV, pages 6232-622-220 6242.
Hu B. Zhang and F. Sha. 2018. Cross-modal and hicrarchical modeling of video and text. In ECCV, pages 374-390.
Y. Yang C. Jaime G L. Quoe Z. Dai, Z. Yang and S. Rus- lan. 2019, Transformer-xl: Attentive language mod- beyond fixed-length context. In ACL, pages 2978-29888
S. Gu E. Jang and P. Ben. 2016. Categorical reparame- terization with gumbel-softmax. ICLR.
W. Limin Z. Yuan, T. Zhan and W. Gangshan. 2021. Mgsampler: An explainable sampling strategy for video action recognition. In CVPR, pages 1513- 1522.
Shreyank Gowda, Mareus Rohrbach, and Laura Sevilla-Lara 2021. Smart frame selection for ac- tion recognition. In Proceedings the AAAI Con- ference on Artificial Intelligence, volume 35. pages 1451-1459.
Kaiyang Zhou, Yu Qiao, and Tao Xiang. 2018. Deep reinforcement learning for unsupervised video sum- marization with diversity-representaaie representativeness reward. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
B. Tamara L J. Lei and B. Mohit. 2022. Revealing single frame bias for video-and-languag learning. arXiv preprint arXiv:2206.03425
Y. Shen Yu B. Tamara J. Lei, L. Wang and B. Mohit. 2020. Mart: Memory-augmentt recurrent trans- former for coherent video paragraph captioning. In ACL, pages 2603-2614.
Z. Ren N. Xu J. Mun, L. Yang and B. Han. 2019. Streamlined dense video captioning. In CVPR, pages 6588-6597.
Frederic F. K. Ranjay, H. Kenji and C. Juan. 2017. Dense-captioning events in videos. In ICCV, pages 706-715.
Y. Li L. Wang and L. Svetlana. 2016. Learning deep muneture-preserving image-text embeddings. In CVPR, pages 5005-5013.
C. L. Zhou and C. Jason 2018. Towards automatic learning of procedures from web instructional vidcos. AAAI.
J. Corso Socher I., Zhou, Zhou Xiong. 2018. End-to-end dense video captioning with masked transformer. In CVPR, pages 8739-8748.
X. Chen C. Jason L. Zhou, K. Yannis and R. Marcus. 2019, Grounded video description. In CVPR, pages
D. Trevor P. Jae Sung, R. Marcus and R. Anna. 2019. Adversarial inference for multi-sentence video de- scription. In CVPR, pages 6598-666 6608.
Z. Lu F. Zheng R. Cheng T. Wang, R. Zhang and Luo. 2021. End-to-end dense video captioning with paral- lel decoding. In ICCV, pages 6847-6857.
S. Chen Song and Q. Jin. 2021. Towards diverse para- graph captioning for untrimmed videos. In CVPR, pages 11245-11254.
B. Dai Y. Xiong and D. Lin. 2018. Move forward and tell: progressive generator video descriptions. In ECCV, pages 468-483


Reviews:
[{"review_id": "0CPPCfBlBv", "reviewer": "Reviewer_tkpR", "summary": "", "questions": "please see the weaknesses", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Answer1: We validated the effectiveness of selected video frames in generating descriptions by conducting ablation experiments comparing metrics such as CIDEr and BLEU. Therefore, in rows 3-5 of Table 3, we performed the ablation study of our $SFG_{video}$ module. In this module, video features learned through video anchors and the original video frames were inputted together into the model, resulting in a significant improvement compared to the vanilla model.\n\nAnswer2: In fact, we have designed two selection strategies. One is to randomly sample from every 2 frames, and the other is to design several anchors to search for key frames throughout the entire video. The one you mentioned is our first strategy, which is designed to reduce redundancy between frames. The second strategy is actually aimed at selecting key frames.\n\nAnswer3: We use the tool provided by[1] to extract the triplets in the captions, as cited in L285\n[1] Yang X, Tang K, Zhang H, et al. Auto-encoding scene graphs for image captioning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 10685-10694.\n\nAnswer4: During the training phase, the initialization  of the video anchors are only related to triplets from captions. However, during the testing phase, we currently select one frame at regular intervals from the video as an anchor. We assume that the actions are distributed relatively evenly throughout the entire video. In addition to that, we add an experiment with randomly selecting frames as anchors using a random seed. The results are as follows:\n |  anchor position |  B@4  |   METEOR  |   CIDER  |   R@4 |\n |  ----  | ----  |----  |----  |----  |\n | random sample  | 10.3  | 15.8  | 28.56  | 3.12  |\n | uniform sample  | 12.24 | 16.97  | 28.89  | 2.51|\n  \nIt can be observed that the random sampling method is not as effective as uniform sampling.\n\nAnswer5: At the end of our paper, we conducted a comparison of model performance and provided some actual generated examples. In fact, we did not select examples based on having the best metrics, but rather chose examples with a higher frequency of events or similarities between events. We believe that annotations are also a form of subjective expression, and from the generated results, our method tends to focus more on describing actions, resulting in more concentrated expressions.\n\nAdditionally, there are two aspects to how we understand noise: 1. Redundant frames in the video are considered noise for generation purposes. 2. We aim to extract information from captions that can identify transitions between events, and we consider action information to be a strong signal while non-action information is considered noise for differentiating events. The noise you mentioned actually refers to the second aspect.\n\nAnswer6: Thank you for your suggestion. The papers we compared with included state-of-the-art (SOTA) models without pre-training models. In 2022, many SOTA models started utilizing large-scale or pre-training models, which we felt would not be a fair comparison. However, thanks for your advice and we will add these works in the regular paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n1)The proposed approach achieves state-of-the-art.\n2) The problem is well-motivated.\n3) The paper is very well written and clear.\n", "reasons_to_reject": "Weaknesses:\n\n1)\tOn line 268 you mentioned using dynamic tokens to select useful video frames. How did you verify that the selected video frames are useful?\n2)\tOn line 276 you mentioned selecting one frame from every two frames as the anchor. Therefore, I think your method is only for sparse sampling, and it doesn't mean that your way is helpful for selecting useful video frames.\n3)\tHow to get y_{triplet} in line 293.\n4)\tYou need to add some ablation experiments. For, example, does the initialization position of this video anchor affect the performance of the model?\n5)\tWhy do you think the ground truth information may contain noise? You should use the results of the ground truth information against the results of the main content to verify the main content is helpful to improve performance.\n6)\tThe methods you compared are outdated, you should add some more recent methods.\n", "questions_to_authors": "please see the weaknesses", "paper_topic_and_main_contributions": "This paper proposed the Sparse Frame Grouping module to dynamically group event information. To enhance the performance, the Intra-Contrastive Learning and Intra-Contrastive Learning method are proposed to align the SFG module and learn action-guided context, respectively.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dCMDtyRZTA", "reviewer": "Reviewer_71yn", "summary": "", "questions": "Question A: Does access to triplets of the ground truth during training provide shortcut learning?\n\nQuestion B: How is the SFG network evaluated in testing when the ground truth is not available?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your concerns!\nAnswer1: The ground truth is used as label and guidance of anchors during the training phase but not during the testing phase.\nAnswer2:  The difference is that the tokens in SFG are predefined in terms of quantity, and a random frame is used as the initial value.\nand thanks a lot for pointing out our mistakes. We will make the necessary corrections in the main text. We appreciate your time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. As far as the reviewer's knowledge, the proposed approach is novel and the problem is well-motivated;\n2. The experiments and analysis are reasonable and provide insights into the strength of the proposed approach over existing models, making the usefulness further convincing;\n3. The paper is relatively well-written.", "reasons_to_reject": "1. It is unclear whether the access to triplets of the ground truth during the training phase provides shortcut learning for the model. If other models in Table 1 do not need this information, this may cause unfair model comparisons. Furthermore, it is also unclear how the model is evaluated when the ground truth is unavailable during the testing phase (Question A, B).", "questions_to_authors": "Question A: Does access to triplets of the ground truth during training provide shortcut learning?\n\nQuestion B: How is the SFG network evaluated in testing when the ground truth is not available?", "paper_topic_and_main_contributions": "The paper proposed a novel Sparse Frame Grouping module for video paragraph captioning without event annotations. It also proposed Intra- and Inter-Contrastive Learning methods to improve model precision and reduce repetition. The network is evaluated on ActivityNet Captions and YouCook2 and demonstrates outstanding performance than the current state-of-the-art. Detailed analyses further consolidate claims about the utility of the proposed network sub-structures.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. \"Inter-contrastive\" instead of \"Intro-\" on line 149;\n2. Figure 5 could be better presented by showing the number directly in a table format: for example, the difference in METEOR is not very obvious in the current plot;\n3. More intuition about the design of the clip- and video-level SFG can be mentioned earlier in the paper to better clarify the different use of these two modules; the current description in section 3.2 is a little hard to follow given very similar notations of different anchors and video frames selected by the two SFG modules."}}, {"review_id": "TpsJ1wzQ2j", "reviewer": "Reviewer_UvCg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestion. In fact, we are indeed considering whether our method can also be helpful for video localization or grounding tasks, meaning that our approach can learn the positional information of video events in an unsupervised manner. However, proving the effectiveness of localization is a complex task that requires further exploration in the context of localization tasks. This goes beyond the scope of this paper, and we are currently working on it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes the novel task of video captioning for without video enent segmentation and annotations. To tackle the task, this paper proposes the Sparse Frame Grouping module to estimate event locaitons, and build an Intra- and Intro-contrastive loss for training. Experiments on two datasets show that the proposed method outperforms even the methods using event annotations. This paper is interesting and easy to read. ", "reasons_to_reject": "The issue I concern about is about the experiments. I suggest the author to evaluate the event segmentation created by the proposed method. This is significant to explain why the proposed method outperform the methods using event annotations.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The topic of this paper is NLP engineering experiment. The paper aims to generate caption paragraph for video without enent annotations. The contributions of this paper are the proposing of the Sparse Frame Grouping module and the proposing of Intra- and Intro contrastive loss for training.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z8M0uUNw5h", "reviewer": "Reviewer_UFoE", "summary": "", "questions": "See reasons to reject.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions. \n\nFor A.1, Figure 1 is a brief illustration and details can be seen in Figure 2, including the shape of the input and the output. \n\nFor A.2, We defined it in L285-L293 when first proposed it. We also explained the composition of triplets and the embedding method. In L414 of Implementation Details, we mentioned that 'In the inference phase, random frames are used to initialize the video anchors for $SFG_{video}$'. So we believe that the GroundTruth information is not revealed in the testing phase.\n\nFor B.1, we believe this is a matter of understanding. We devised two methods for frame selection. One involves choosing between two frames, but even this selection process follows our designed grouping approach rather than being random. This design aims to eliminate redundancy between frames. The other method involves utilizing several anchors to select key frames across the entire video scope. Thus, your remarks pertain solely to the first approach. However, the design purpose you described is targeted at the second approach.\n\nFor B.2, The SFG video does not generate new video frames. The four legends on the right side of the video frame in Figure 2 represent tokens of the SFG video.\n\nFor B.3, We use 4 layers and every 2 layer of the encoder is illustrated as Figure 4.We try the clip level selection in the first layer and the last layer.\n\nFor C.1, thank you for your suggestion. Indeed, we had not encountered significant testing in the previous work of paragraph captioning task. Therefore, during the rebuttal phase, we specifically researched how to conduct significance testing for this task. Below are the findings from our testing on ActivityNet Captions ae-val, where * means the uplift compared to Vanilla model is significant.\n | model | B@4 | METEOR | CIDER |R@4 |\n|  ----  | ----  |----  |----  |----  |\n| Vanilla | 11.23 |15.34 |25.23 |7.38  |\n| Ours | 12.24 |16.97*  |28.89*  |2.51* | \n\nFor C.2, on the one hand, recent works including the one you cited used the pre training models, which is not a fair comparison. On the other hand, even in comparison, our metrics are directly comparable to those in [1].\n\nFor C.3, the number of anchors does not equal the number of triplets, as during the testing phase, we do not know the exact count of triplets in the captions. Therefore, the number of anchors is predefined. We have mentioned that in L414\n\nFor C.4, in the ablation experiment of Table 4, we conducted a comparison between clip levels 2 and 3, which corresponds to comparing video lengths divided by 4 and video lengths divided by 9. This aligns with what you requested regarding the number of frames in the decoding stage.\n\nAs of now, there has been no comparison in the SOTA literature concerning speed. Therefore, this aspect has not been addressed in our current work. We appreciate your suggestion and plan to explore comparisons with larger models in future endeavors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper tackles a more challenging video paragraph generation problem that does not require event annotations.\n\n- The proposed method improves accuracy and diversity of video paragraph captions on two datasets.", "reasons_to_reject": "A. The presentation of the introduction should be largely improved.\n\n1) Figure 1 is inaccurate and hard to understand. First, not all existing methods rely on event annotations to generate captions. Second, it is unclear what the outputs of the sparse frame grouping are.\n\n2) The method description in the introduction is hard to understand. \n\nL93: What are the triplets? Please define the term when it first appears.\n\nL103: “In the decoder, … attending video anchors in the video-level SFG additionally” However, the video anchors are initialized by the groundtruth triplets as said in L93. Do you leak the groundtruth information to the model?\n\n\nB. The SFG module is not described clearly and the design might be problematic.\n\n1) The SFG_{clip} uses eq5&6 to generate the output frame features. The main idea is to select the frame that is most similar to the anchor. According to L276-280, the SFG_{clip} randomly selects one frame as the anchor. If I understand correctly, this approach is just randomly downsampling the input video sequence by a factor of 2, which is against the motivation of select action-centric frames.\n\n2) It seems that the SFG_{video} mainly learns the video anchors (eq 3&4), but does not affect the update of the video frame features. But in Figure 2, the SFG_{video} block seems to generate a new sequence of video frame features, which is confusing.\n\n3) Is it true that every transformer layer in the encoder downsamples the video frames by a factor of 2? How many layers do you use?\n\nC. There are quite some problems in the experiments.\n\n1) The performance improvements are marginal and a significant test is necessary. \n\n2) It did not compare with more recent works such as [1]. \n\n3) In Table 4, the number of video anchors should be the same as the number of triplets in the caption, right? Why here the numbers are 8 and 10?\n\n4) More ablations are required to understand the model. For example, how many video frames are left to decode the sentence? How much does it affect the performance? What is the speed compared to other methods?\n\n[1] Yang, Antoine, et al. \"Vid2seq: Large-scale pretraining of a visual language model for dense video captioning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "questions_to_authors": "See reasons to reject.", "paper_topic_and_main_contributions": "The paper aims to improve untrimmed video paragraph captioning. It proposes a sparse frame grouping (SFG) module which selects frames in the video for decoding, and also a intra- and inter-contrastive learning method to improve the learning of the SFG module. Experimental results demonstrate the effectiveness of the proposed approach. The proposed method achieves state-of-the-art performance on two datasets.\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 3: typos in row 6 and 7.\n\nFigure 4: missing connections between blocks in the right figure.\n\nThe reference format is incorrect.\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for video paragraph captioning without event annotations called Sparse Frame Grouping, using modifications of contrastive learning methods to improve model precision and reduce repetition.

Reviewers generally found the problem important and that the proposed method improves performance with sound experimental evaluation. The main concerns revolved around more ablation experiments, analysis of learned model's segmentation capabilities, more comparisons, and scale of improvements. The authors clarified the majority of the problems. Reviewer UFoE had some remaining concerns regarding writing and presentation, especially on describing the method and triplet selection in contrastive learning, which I encourage the authors to carefully revise.
Decision: Accept-Findings