SynopGround: A Large-Scale Dataset for Multi-Paragraph Video Grounding from TV Dramas and Synopses

Authors: chaolei  tan
zihang  lin
junfu  pu
zhongang  qi
wei-yi  pei
zhi  qu
yexin  wang
ying  shan
wei-shi  zheng
jian-fang  hu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-09 05:04:22

Abstract: Video grounding is a fundamental problem in multimodal content understanding, aiming to localize specific natural language queries in an untrimmed video. However, current video grounding datasets merely focus on simple events and are either limited to shorter videos or brief sentences, which hinders the model from evolving toward stronger multimodal understanding capabilities. To address these limitations, we present a large-scale video grounding dataset named SynopGround, in which more than 2800 hours of videos are sourced from popular TV dramas and are paired with accurately localized human-written synopses. Each paragraph in the synopsis serves as a language query and is manually annotated with precise temporal boundaries in the long video. These paragraph queries are tightly correlated to each other and contain a wealth of abstract expressions summarizing video storylines and specific descriptions portraying event details, which enables the model to learn multimodal perception on more intricate concepts over longer context dependencies. Based on the dataset, we further introduce a more complex setting of video grounding dubbed Multi-Paragraph Video Grounding (MPVG), which takes as input multiple paragraphs and a long video for grounding each paragraph query to its temporal interval. In addition, we propose a novel Local-Global Multimodal Reasoner (LGMR) to explicitly model the local-global structures of long-term multimodal inputs for MPVG. Our method provides an effective baseline solution to the multi-paragraph video grounding problem. Extensive experiments verify the proposed model's effectiveness as well as its superiority in long-term multi-paragraph video grounding over prior state-of-the-arts. Dataset and code are publicly available. Project page: https://synopground.github.io/.

Main paper: 
 introduction: 
 As a crucial problem in multimodal understanding, video grounding aims at linking semantically relevant temporal intervals in an untrimmed video with specific natural language queries. Recently, video grounding has received increasing attention since a wide range of downstream applications can be promoted by it, such as improving the searching granularity of video retrieval [4, 13, 16, 22, 77] , enabling language-aware scenarios of video editing [6, 20, 21, 36] ,Figure 1 : Comparison of video grounding datasets. Our Syn-opGround is the first dataset to introduce the challenges of both long videos and long queries into video grounding. and making video question answering [32, 39, 41, 44, 45, 78, 83] more evidence-based. Up to now, a large number of datasets and methods have been established to advance this line of research.However, considerable drawbacks still exist in previous video grounding datasets [2, 23, 24, 37, 60, 66] . First of all, as presented in Figure 1 , most of commonly-used datasets are constructed upon short videos and brief sentence queries. This setup limits the model in developing stronger abilities that can model and bridge the longform videos [27, 57] and long-text queries [23, 81] . Besides, shorter queries that describe detailed events (as shown in Table 1 ), are more prone to causing the risk of semantic ambiguity in referring expressions [52, 56] , i.e., the occurrence of one-to-many correspondence between queries and moments, which will adversely affect the model learning. In particular, this ambiguity issue is more prominent for the recently proposed MAD [66] dataset which features long input videos but short general descriptions. For example, it is highly likely to find cases where multiple moments are semantically corresponding to the same short description like "She steps closer. " (shown in Table 1 ), especially when searching content in a long video. Furthermore, as listed in Table 1 , existing benchmarks are tailored for language queries referring to low-level visible activities, while all of them overlook the importance of more complex events and abstract concepts. Such drawbacks actually limit the applications of video grounding in scenarios where complex descriptions with abstract concepts should be associated with long-term videos. For example, accelerating the movie post-production by automatically integrating raw footage into a coherent story based on the plot scripts is a practical need, but it cannot be satisfied by the current video grounding techniques developed from existing datasets.In this work, we curate and present a large-scale dataset called SynopGround to address the current limitations of video grounding datasets. We collect and manually annotate episodes from popular TV dramas of various genres, yielding a large-scale video grounding dataset consisting of over 2800 hours of fully-annotated videos. Specifically, for each video, we crawl its human-written synopsis consisting of multiple paragraphs from the Internet, and further annotate the precise temporal boundaries for each paragraph in the given synopsis. As demonstrated in Figure 1 , our dataset has both Table 1 : Comparison of queries in different datasets. The red-bold text is a detailed description, while the blue-italic text is an abstract and concise expression.Query Charades [23] A person runs to the window then looks out. DiDeMo [2] The little girl jumps back up after falling. TACoS [60] He flips the eggs, making an omelet. ActivityNet [37] A woman walks to the piano and briefly talks to the elder man. Ego4d-NLQ [24] What did I pick from the fridge? MAD [66] She steps closer. SynopGround (Ours) . . . Stefan and Elena decided to go to the cabin left by Elena's parents, where they spent a happy time. . . . significantly longer average video length and average query length than most existing ones. It is the first video grounding dataset that can support the research on long-term contextual video grounding with complex queries. Moreover, compared to the short sentence queries in the existing datasets, our long paragraph queries can unambiguously indicate one-to-one correspondence between language queries and target moments, which is crucial for learning accurate cross-modal alignment. Furthermore, as shown in Table 1 , there are very concrete descriptions for visible activity concepts like "go to the cabin", as well as extremely concise and abstract expressions like "spent a happy time" in the query from our dataset. This enables to learn and evaluate the comprehensive understanding of semantic concepts at diverse abstraction levels.Based on our dataset, we pioneer to introduce and explore a more challenging and complex setting of video grounding called Multi-Paragraph Video Grounding (MPVG). The MPVG task receives a multi-paragraph synopsis and a long narrative video as inputs to localize the temporal interval of each synopsis paragraph from the video. To promote and inspire future research, we further propose a novel Local-Global Multimodal Reasoner (LGMR) to explicitly model the local-global structures of long-term multimodal inputs and conduct iterative cross-modal reasoning within and across the two levels of structures for effectively tackling the multi-paragraph video grounding problem. Extensive experiments demonstrate the effectiveness of our baseline in the proposed research direction.The main contributions of this work are summarized as follows:• We present SynopGround, a large-scale video grounding dataset consisting of over 2800 hours of TV drama videos with manual temporally-annotated professional synopses.  
related work: 
 In this section, we aim to review and discuss the existing works in the video grounding and narrative video understanding areas.Datasets. In video grounding, Charades-STA [23] , DiDeMo [2] , ActivityNet-Captions [37] and TACoS [60] are the four most commonly used datasets for model training and evaluation. However, a majority of these datasets [23, 37, 60] are adapted from pre-existing video datasets tailored for closed-set recognition or localization tasks [7, 62, 64] , which makes them severely limited to a pre-defined set of visual and linguistic concepts. DiDeMo [2] is a customized video grounding dataset. However, it overly simplifies the annotation and only supports the model to select from 5 evenly-divided segments of the video. In addition, shortcut learning issues caused by distribution biases in previous datasets have been reported [38, 56] , which could adversely affect the benchmark reliability. Moreover, the above datasets [2, 23, 37, 60] are all constructed on a relatively small-scale collection of short videos and simple sentence descriptions, which cannot support the need of large-scale model training for long-term contextual video-language understanding that incorporates complex language queries. Recently, the Ego4d-NLQ [24] and MAD [66] datasets are introduced. Nevertheless, both of them still focus on the simple visible activities and short-term temporal events. Specifically, Ego4d-NLQ contains egocentric videos and adopts brief interrogative queries asking about simple visible fact grounded on a short video interval. The MAD dataset is semiautomatically constructed on movies with audio descriptions and its average video length is significantly longer compared to the other existing datasets. However, the language queries of MAD are still brief sentences that individually describe short-term events in the long video. Different from all of the prior works, our proposed SynopGround is the first video grounding dataset that considers both long-form videos and long-text queries. Additionally, we adopt narrative videos conveying storylines and tightly correlated synopsis paragraphs as inputs, which poses more challenges for the video grounding model to understand high-level story plots and invisible abstract concepts over a longer context. Tasks and settings. Early research of video grounding has largely focused on grounding single sentences in videos, i.e., the Video Sentence Grounding (VSG) task introduced by Gao et al. [23] and Hendricks et al. [2] . Afterwards, a series of extended tasks [5, 18, 40, 42] have been proposed. Escorcia et al. [18] first introduced the task of Video Moment Corpus Retrieval (VCMR) for combining video retrieval and moment localization, and Lei et al. [42] curated the TVR dataset to incorporate multi-modal information into VCMR. Lei et al. [40] proposed QVHighlights dataset along with a new direction combining highlight detection and moment retrieval. To reduce ambiguity by exploring inter-query context, some recent works [1, 5, 12, 34, 63, 71] have shifted to a multi-query version of video sentence grounding, where the model is required to understand several temporally ordered sentences and localize each sentence in a richer context. Specifically, Bao et al. [5] first studied multi-sentence video grounding in a fully-supervised setting, and the semi-supervised setting [34] as well as weakly-supervised setting [69] have also been investigated after that. These prior works have shown the great potential of contextually understanding multimodal content in untrimmed videos and language descriptions. In this work, we take a step further to introduce a more challenging setting of contextual video grounding called Multi-Paragraph Video Grounding (MPVG). It requires to understand both short-term intraparagraph semantics and long-term inter-paragraph dependencies, which connects the complex temporal structures of long videos with the complicated semantics of long paragraphs. Methods. A lot of approaches [11, 14, 28, 31, 43, 47-49, 51, 53, 55, 68, 70, 74-76, 79, 82, 84-87, 89, 90] have been developed for video grounding over recent years. As summarized in [88] , these methods can be roughly categorized into proposal-based and proposal-free methods. Proposal-based methods typically involve a two-stage process of generating moment proposals for relevance score ranking, which often leads to issues like inefficiency and limited adaptability.In contrast, proposal-free methods tend to have better efficiency and they directly predict the temporal boundaries based on cross-modal interactions, which is more suitable for various real-world scenarios.Considering the long-term characteristics in our dataset, we choose to design our Local-Global Multimodal Reasoner (LGMR) in the more efficient proposal-free fashion. Different from the previous approaches that only consider modeling the cross-modal correspondence between a single paragraph and the video, our method is constructed by reasoning through the local and global structures of multiple paragraphs and the video.Understanding visual content presented in narrative videos is an important area with many works [3, 15, 25, 26, 29, 30, 33, 54, 59, 65-67, 72, 80] proposed accordingly. Most of these prior works neglect to model and understand the content of narrative videos based on their high-level storylines while focusing on specific downstream applications, such as movie genre classification [65] , character identification [26, 29, 54] , action localization [25] , scene segmentation [15, 30] , and shot classification [59] . In addition to that, some works have started pursuing story-level understanding in many different ways. For instance, Tapaswi et al. [72] proposed MovieQA dataset to comprehend movie stories by questionanswering. MSA [80] , CMD [3] , and SyMoN [67] datasets utilized synopses as language queries and formulated movie understanding as text-to-video retrieval. This line of research is closely related to ours. However, we focus on a more challenging video grounding task with long contexts and complex queries, in which the model should understand the long-range cross-modal dependencies so as to reason about the video grounding results at story level. 
synopground dataset: 
 In this section, our goal is to give a formal definition of our introduced multi-paragraph video grounding problem and illustrate the details of the data collection, annotation, statistics, and processing.Considering video paragraph grounding [5] 𝑒 for the 𝑖-th paragraph query in the target video. In our dataset, the video V is an episode from a TV drama, and Q is the corresponding human-written synopsis that contains 𝑁 paragraphs, with 𝑄 𝑖 indicating the 𝑖-th paragraph in the synopsis Q. Illustration of our MPVG is in Figure 2 .We collect all the TV drama episodes from a leading online platform Tencent Video with official acknowledgement and permission. The plot synopsis for each episode of the TV dramas is scraped from Table 2 : Detailed comparison with existing video grounding datasets. Our SynopGround is at a larger scale in terms of the total duration of videos and it contains precise temporal annotations generated by human annotators. It is also the first large-scale dataset that considers both long-form videos and long-text queries for multi-paragraph video grounding.Charades-STA [23] ANet-Captions [37] DiDeMo [2] TACoS [60] Ego4d-NLQ [24] MAD [ Synopses that are too long or too short are discarded to ensure an adequate number of paragraphs in each synopsis. A total of 520 licensed and high-viewership TV dramas with textual synopses are finally selected to constitute our dataset, and we randomly sample several episodes from each selected TV drama to further annotate. Specifically, annotators are asked to read and understand the synopsis in advance. They then thoroughly watch the corresponding TV drama episode to determine the starting and ending time of the video content depicted by each synopsis paragraph.Our data annotation pipeline is organized into multiple rounds to ensure the annotation quality. Specifically, all collected videos are first divided into numerous disjoint subsets of videos. In each annotation round, synopsis paragraphs for videos in one subset will be annotated with timestamps and each annotator is told to provide a score to indicate level of confidence in the annotated results. Afterwards, we first discard samples with low confidence as an initial cleanup, and then some of the remaining samples are selected to be manually checked in terms of quality. If the annotation quality is thought of as satisfactory, the annotation process will move on to another unlabeled subset of video data. Otherwise, the current batch of data would be re-annotated. The above procedures are repeated by tens of annotators until we finish the annotation of all candidate samples. For post-annotation assessment, we randomly select a proportion of the annotated data to be re-annotated by other annotators. Concretely, we calculate the temporal IoU (Intersection over Union) between the two results from different annotators, which reaches a value of about 85%. This assessment result is much better than those of other datasets, such as the Activ-ityNet [37] , where different annotators only achieve an agreement degree around 70%. The higher degree of agreement across different 1 URL: https://www.tvmao.com. All texts are translated into English using Tencent Cloud Translation with full permission and compliance. annotators in our dataset verifies the effectiveness of our designed pipeline for data annotation and quality control.Data distribution. We first illustrate some statistical distributions of our dataset in Figure 3 . As shown in Figure 3 (a), the TV dramas used in our dataset cover a wide spectrum of genres, which demonstrates the diversity of the collected data. In Figure 3 (b), we show the normalized duration of the target video segments. Most of the target video segments cover less than 20% of the full episode, which can be challenging for the model to correctly localize. In Figure 3 (c), we visualize the distribution of the number of queries/paragraphs in each synopsis, and most synopses are composed of 5-13 paragraphs. Exploring the contextual information among these paragraphs is important for achieving promising performance in our multi-paragraph video grounding task. In Figure 3 (d) and (e), we visualize the temporal distributions of the starting timestamps and ending timestamps of the target video segments. Both of them approximately present a uniform distribution, which ensures the model cannot benefit much from the distribution bias. Detailed comparison with other datasets. In Table 2 , we compare our dataset with other existing datasets in detail. As suggested, our videos are much longer in duration than those of Charades-STA [23] , ActivityNet-Captions [37] , DiDeMo [2] , TACoS [60] and Ego4d-NLQ [24] . Although the average video duration in our dataset is shorter than that of MAD, our total duration of videos is more than twice that of MAD, showing that our dataset is at a larger scale. Furthermore, the duration of target segments in our dataset is significantly longer while the normalized target span is still short, making our target moments challenging to be localized. Note that some datasets like MAD have shorter normalized target span than ours, but their short and general queries bring the harmful and undesirable semantic ambiguity issue as mentioned before. In addition, our dataset is the first to incorporate paragraph queries, and the average number of words in each query is significantly larger than those of other datasets, which greatly reduces the semantic ambiguity of the queries. Moreover, our synopsis queries involve both abstract expressions and concrete descriptions, enabling the model to learn semantic concepts at more diverse abstraction levels. Data splits. As shown by the statistics in Table 3 , we carefully divide the entire data into three non-overlapping splits for training, validation, and testing. Our training, validation, and testing sets consist of 3,187, 400, and 400 videos, respectively. It is worth noting that each video is an episode from a TV drama. Additionally, we manually guarantee half of the videos for validation/testing are sourced from TV dramas that are not selected for the training set.To promote data utilization, we provide pre-extracted features for public release. Specifically, pre-trained CLIP [58] ViT-L/14 model is adopted to extract frame features for videos using a sampling rate of 3 FPS. Additionally, we extract segment features for videos with SlowFast [19] network, which is pre-trained on Kinectics-600 [9, 10] and AVA [25] datasets. To capture the character and dialogue information related to the storylines, we extract embedded subtitles using OCR models DBNet [46] and SVTR [17] . Then, we encode each extracted subtitle to a feature representation using a pre-trained RoBERTa [50] model. The pre-extracted CLIP, SlowFast, and OCR features describe the video from different aspects. They provide complementary information of the video and are beneficial for aligning synopsis with the video. Due to copyright restrictions, we cannot release the raw video frames but we will provide URL links where researchers can access and view the original videos. 
method: 
 In this section, we illustrate the details of our proposed baseline method to tackle the multi-paragraph video grounding problem.As shown in Figure 4 Given the long-form video inputs, we design a local-global attentive encoder to capture the evolving temporal dynamics of long-term video content, which exploits more structural temporal information than straightforward full attention. Specifically, we first project the video features of length 𝑇 into a hidden dimension of 𝐷, then for each video encoder layer, we split the input video feature sequence into non-overlapping temporal windows of length 𝑀, resulting in the intra-window video representations 𝐹 𝑤 ∈ R 𝐾 ×𝑀 ×𝐷 , where 𝐾 is the total number of the temporal windows. For the video features in the 𝑖-th window, i.e., 𝐹 𝑤 𝑖 ∈ R 𝑀 ×𝐷 , we first encode the detailed local information by performing temporal self-attention within the scope of that window as follows:EQUATIONwhere 𝐹 ℓ 𝑖 ∈ R 𝑀 ×𝐷 is the encoded video features with rich local contexts. Based on the local video features, we further apply a global-level self-attention on the global window features to connect different local contexts. Instead of using a simple pooling layer to aggregate the local window features, we exploit an attention-based method similar to the attention pooling in CLIP [58] to dynamically gather important local contexts for global interaction as:EQUATIONwhere 𝐹 𝑔 ∈ R 𝐾 ×𝐷 represents the global window features across the entire video, and Avg (•) is an average operation across temporal dimension for the local window features. Then the global window features are interacted with each other by a self-attention as:EQUATIONNext, we merge the information from the local and global contexts of the intra-window and inter-window features as:EQUATIONwhere Rep (•) and Flatten (•) respectively indicate repeating the global window features by 𝑀 times in its corresponding window and unfolding the window-level representations into a feature sequence. LN (•) and FFN (•) denote the layer normalization operation and feed-forward network, respectively. 𝐹 𝑉 ∈ R 𝑇 ×𝐷 denotes the output features of a video encoder layer, and the output of each former layer will be further fed to the next layer for encoding.Existing methods developed for single-paragraph video grounding [5, 63, 71] either encode the language query into a single global embedding [63] causing too much information loss, or employ selfattention on the complete multimodal sequence of all text features and video features [71] , which incurs prohibitive resource cost thus is unsuitable in the multi-paragraph scenario. In this work, we explore a novel way to model the local-global query structures and cross-modal correspondences by iteratively reasoning about the local subparagraph features and global paragraph features.To begin with, we first utilize a pre-trained RoBERTa [50] model to obtain the token-level language features from the 𝑖-th input paragraph, i.e., 𝐹 𝐿 𝑖 ∈ R 𝑁 𝐿 𝑖 ×𝐷 , where 𝑁 𝐿 𝑖 is the total number of language tokens in the 𝑖-th input paragraph. Afterwards, we jointly utilize a paragraph encoder and a subparagraph extractor to efficiently model the intrinsic local and global structures of the long text inputs, as shown in Figure 4 . Concretely, we first embed all the token-level features within a paragraph into a global query feature 𝑄 𝑔 𝑖 ∈ R 𝐷 by an average-pooling operation. Then, we exploit 𝐸 learnable vectors 𝑂 𝑆 ∈ R 𝐸×𝐷 to extract the important subparagraph representations under the semantic guidance of 𝑄 𝑔 𝑖 as follows:EQUATIONwhere 𝑊 1 ∈ R 𝐷 ×𝐷 and 𝑊 2 ∈ R 𝐷 ×𝐷 are learnable projection matrices and LN (•) is the layer normalization operation. 𝑄 ℓ 𝑖 ∈ R 𝐸×𝐷 is the extracted subparagraph features that can be adaptively learned to represent meaningful local semantics specific to each paragraph for enhancing the cross-modal reasoning abilities within the paragraphs. Note that the number of extracted subparagraph features is typically small and the computation process will be efficient.After obtaining the local subparagraph features and global paragraph features, we construct an iterative local-global reasoning process where each iteration involves intra-level reasoning, crossmodal reasoning and cross-level reasoning. Firstly, we conduct intra-level reasoning by employing two self-attention layers respectively within each window of local queries 𝑄 ℓ 𝑖 and within all global queries 𝑄 𝑔 . Afterwards, we achieve cross-modal reasoning by extracting relevant information from 𝐹 𝑉 to 𝑄 ℓ and 𝑄 𝑔 by crossattention layers. Then, we conduct cross-level reasoning also by two cross-attention layers, i.e., one is for extracting information from 𝑄 𝑔 to 𝑄 ℓ and the other one is for extracting information from a window of local queries 𝑄 ℓ 𝑖 to the corresponding global query 𝑄 𝑔 𝑖 . The cross-interacted features will serve as the output features of each decoder layer and are fed to the next layer for iterative decoding. Finally, the local and global output features, i.e., 𝑄 ℓ 𝑖 and 𝑄 𝑔 𝑖 of the last decoder layer are concatenated and fed to an MLP predictor to obtain the central timestamp t𝑖 𝑐 and duration Δ t𝑖 of the target interval for the 𝑖-th paragraph query. Then the temporal boundaries T𝑖 = ( t𝑖 𝑠 , t𝑖 𝑒 ) can be calculated as t𝑖𝑠 = t𝑖 𝑐 -Δt 𝑖 2 , t𝑖 𝑒 = t𝑖 𝑐 + Δt 𝑖 2 .We train our model with a localization loss L 𝑙𝑜𝑐 and an attention loss L 𝑎𝑡𝑡 which are formulated as follows:EQUATIONEQUATIONwhere L 𝑙1 and L 𝐺𝐼𝑜𝑈 are L1 and GIoU [61] losses, respectively. T𝑖 is the predicted time span for the 𝑖-th query and T 𝑖 is the groundtruth. L 𝑎𝑡𝑡 is an attention loss on the global query features. The term 𝑎 𝑖 𝑗 indicates the attention weights between the 𝑖-th global query feature and the 𝑗-th video feature, while 𝑚 𝑖 𝑗 is an indicator that takes 1 if the 𝑗-th video feature is inside the ground-truth interval of the 𝑖-th query, and 0 otherwise. This loss explicitly encourages the model to learn higher attention weights between text queries and visual elements that are correlated. In total, the training loss is defined as the weighted sum of the above two losses as L = 𝜆 1 L 𝑙𝑜𝑐 + 𝜆 2 L 𝑎𝑡𝑡 , where 𝜆 1 and 𝜆 2 are the hyper-parameters to balance these two different kinds of losses. 
experiments: 
 In this section, we illustrate our experimental setup and results for verifying and analyzing the effectiveness of our proposed method.Evaluation metrics. For each query in the synopsis, we calculate the temporal Intersection over Union (IoU) between the predicted time span t𝑠 , t𝑒 and the ground-truth time span [𝑡 𝑠 , 𝑡 𝑒 ]. Following previous video grounding methods [2, 23] , we adopt two kinds of metrics to evaluate the performance: 1) mean IoU (mIoU) metric: average temporal IoU score calculated over all queries in the dataset; 2) IoU@𝜃 metric: the proportion of queries with a temporal IoU score higher than 𝜃 , here we use 𝜃 ∈ {0.3, 0.5, 0.7}. Implementation details. Our proposed method is implemented by PyTorch. The pre-extracted SlowFast, CLIP, and OCR features are aligned at sequence dimension and concatenated at channel dimension as the video feature input. We use a pre-trained RoBERTa [50] model to extract OCR features at each timestamp. The loss weights are set as 𝜆 1 = 1, 𝜆 2 = 0.2. For data augmentation, we choose to randomly shuffle the order of paragraphs in the same synopsis by a probability of 𝑝 during training and 𝑝 = max(0, 1 -𝑇 𝑖 𝑇 𝑚𝑎𝑥 ), where We adopt a local window length 𝑀 of 25 for our video encoder. The number of layers for video encoder and query decoder are set as 2 and 3, respectively. Our model is trained on 4 NVIDIA Tesla V100 GPUs by Adam [35] optimizer using a learning rate of 0.0001 and batch size of 16 for a total of 50 epochs within one day.Performance Comparison. As shown in Table 4 , we evaluate the performance of our proposed LGMR on the challenging multiparagraph video grounding task and compare it with the existing state-of-the-art methods DepNet [5] and PRVG [63] . DepNet is the baseline method proposed for multi-sentence video grounding, and PRVG is a concise and effective method based on DETR-like architectures [8] . For a fair comparison, all reported methods employ the same features as ours. The comparison results in Table 4 demonstrate that our proposed model achieves the best performance and outperforms others by a significant margin, which validates the effectiveness of the proposed LGMR method for addressing MPVG.Impact of different features. To investigate the effect of different features, we conduct experiments with various combinations of SlowFast, CLIP, and OCR features. As shown in Table 5 , we observe that using a single kind of features already yields satisfactory performance. Specifically, using the SlowFast, CLIP, or OCR features alone is able to produce an mIoU of 39.1%, 39.8%, and 41.7%, respectively. We notice that the CLIP features and OCR features are more helpful than the SlowFast features, which might be because 1) CLIP is pre-trained on large-scale image-text pairs, which makes it generalize better to the downstream task of video-language grounding. 2)The OCR features encode rich character-related and dialogue information, which is important for understanding the story plots in the narrative video. Additionally, we can see that the model using all three features together achieves the best performance on all metrics, showing that different kinds of features convey complementary information of the video content for language grounding. Ablation studies on design choices. To further validate the rationality of our proposed model, we conduct ablation experiments on the designs of local-global temporal attention and cross-modal attention loss, as shown in Table 7 . Firstly, we compare our model performance with that of a variant model where the local-global encoder is replaced by a vanilla full attention encoder [73] . The result suggests that our local-global encoder performs better in both accuracy and efficiency for long video inputs. Besides, we remove L 𝑎𝑡𝑡 and observe severe degradation in model performance. This highlights the importance of explicitly guiding the model to associate and align correlated visual and textual features. 
conclusion: 
 In this work, we present a large-scale dataset for video-language grounding called SynopGround, which consists of over 2800 hours of long narrative videos with human-written synopses and manually annotated timestamps. It is the first video grounding dataset considering both long-form videos and long-text queries, and contains query descriptions conveying both low-level events as well as high-level plots for learning more complex and abstract concepts. We further introduce a challenging Multi-Paragraph Video Grounding (MPVG) task which incorporates long paragraph queries into multi-query video grounding. In addition, we propose a novel Local-Global Multimodal Reasoner (LGMR) to explicitly model the local-global structures of long-term inputs and conduct iterative reasoning within and across the two levels of structures, which can serve as a good starting point to inspire future research. 
 
 Appendix:  


Decision: This paper introduces a large-scale video grounding dataset for multi-paragraph video grounding, and a strong baseline model for multi-paragraph video grounding is proposed by modeling the local-global information in the long video and paragraphs. The contribution of building such a dataset is solid and the baseline model also makes sense, which could be interesting to the research community on video grounding. The author also did a good rebuttal, addressing the major concerns from the reviewers, despite some remaining concerns regarding how to ensure the dataset quality with 80 annotators, comparing the normalized span with MAD dataset. The AC urges that the authors should revise the manuscript according to the reviewers' feedback and address all these remaining concerns. Meanwhile, the authors should also release the dataset, code and baseline models as they promise in the paper.

Reviews: review: paper_summary: paper_summary: This work proposes a new video-grounding dataset, SynopGround. Unlike previous datasets in this domain, SynopGround consists of long paragraph queries mapped to a segment of the respective video. These long queries reduce the ambiguity for possible one-to-many mapping for query and video segments. The data is collected from the online platform- Tencent Video, with proper acknowledgement and permissions. This work includes a handcrafted train-val-test split. Each video (sample) is an episode from a TV drama, and authors make sure that ~50% of the samples in the test/val set do not belong to the train set. The annotation process is explained well where annotators attain an 85% agreement score (IoU over individual's annotations). Despite proper data collection permissions, authors acknowledge the copyright restrictions and inability to provide the raw video frames. They assure only to share the pre-extracted features for reproducibility and enable future research over the shared features. Along with this dataset contribution, this paper presents a novel baseline: Long-Global Multimodal Resasoner (LGMR). The model is well explained and includes the implementation details, enough to replicate the work. The architecture design choices, modality selection, and loss function selection are justified by appropriate ablation studies.
strength_weakness: strength_weakness: 1. Sample real-world use cases were provided where SynopGround can be utilised.
2. SynopGround dataset can support long-term textual video grounding research with complex queries.
3. Mutual agreement among the annotators, based on IoU of time-segment annotations, is claimed to be near 85%, which shows high agreement among the annotators.
4. Fine details about the novel Long-Global Multimodal Reasoner (LGMR) model are provided.
5. Sufficient experiments and ablation studies are conducted to justify the design choices behind the proposed LGMR model.
6. This submission includes detailed supplementary material with additional details about LGMR implementation, additional qualitative analysis with attention weights, and some visual results.
7. This submission contains good-quality plots and figures.
limitations: limitations: 1. In Line 579, the authors mention that they will provide the pre-extracted features for public release and that, due to copyright restrictions, they will not be able to release the raw video frames. Public URLs will be provided. This restricts the research in the future because if the URLs get updated, the videos will be lost, and the mapping of the current timestamp annotations may not match with another version of the same video. Moreover, there may not be any way to extract video features from another model.
2. Line 573, How did authors extract subtitles via OCR? Subtitles are supposed to be extracted from audio transcriptions? Did the videos include embedded subtitles over frames?
3. I encourage authors to evaluate LGMR over other datasets, such as MAD, for completeness. Does it claim SoTA over other benchmarks? The results may boost the contribution or reveal some shortcomings of LGMR architecture for shorter queries.
4. The details about human evaluation are missing in Table 4. Please share details on how the human evaluation is conducted. How was the agreement among the annotators?

Minor concern
5. The task of Multi-Paragraph Video Grounding (MPVG) seems a special case of multi-sentence video grounding. Authors acknowledge this, so I suggest not emphasising this as a new task but building over an existing one.
paper_summary: paper_summary: This paper proposes a SynopGround dataset, a large-scale video grounding dataset. Based on the dataset, propose an MPVG task aimed at multi-paragraph video grounding. At the same time, an LGMR architecture is proposed; combining the two features of local and global to enhance the robustness of the model.
strength_weakness: strength_weakness: The strengths of this paper include:
1. Novelty: The paper introduces a dataset (SynopGround), a task (MPVG), and a model (LGMR).
2. Methodology: The authors propose a Local-Global Multimodal Reasoner model that includes a local-global temporal encoder and a local-global iterative decoder. This technique demonstrates promising potential in the task of MPVG.
3. Paper drawings look better.
4. Comprehensive Evaluation: The paper provides a rigorous experimental evaluation of the proposed model against strong baselines. This provides credible evidence of the model's effectiveness, offering a solid foundation for future improvement.
limitations: limitations: Review and Evaluation of Weaknesses:
1. Dataset construction: The dataset is manually annotated. How to ensure the balance of the data? How are data labeling rules designed? Is the synopsis data handwritten by humans? How to ensure the dataset quality?
2. Task definition: What is the difference between multi-paragraph video grounding (L328) and multi-sentence video grounding (L321)? What is the difference between one-time multi-paragraph video grounding and multiple one-paragraph video grounding?
3. In this way, what the MPVG task realistic application is? The significance and application of the MPVG task is not clear.
4. What do (d) and (e) in Figure 3 mean? unclear expression.
5. Unclear definition: Are Q_1, Q_2,...Q_N in the proposed method LGMR (Figure 4) continuous? What is the relationship between the corresponding V and Q? What is V, is a long video? 
6. L560-L562, it seems that the training set, validation set, and test set are not completely separated.
7. Typo errors: e.g. (1) L621'figure 4'. (2)  'an' mIoU.
8. L856 ‘Specifically, using the SlowFast, CLIP, or OCR features alone is able to produce an mIoU of 39.1%, 39.8%, and 42.7%,’， ‘42.7%’ means? 
9. How is the loss parameter lambda (lambda_loc and lambda_att) of L805 determined? Are there any other settings for comparison?
10. Why do some formulas use ‘=’ and some use ‘<-’? Does it have any special meaning?
11. There is no statement about publicly releasing the dataset and code of this method, which makes future re-production of the reported results a really challenging (if not impossible) task.
paper_summary: paper_summary: This paper proposes a novel dataset for multi-paragraph video grounding from tv dramas and synopses. Different from existing datasets that focuses on short descriptions, the queries in this dataset are from synopses which are more detailed and challenging. The authors also propose a Local-Global Multimodal Reasoner method for this task. Extensive experiments show the proposed methods are effective.
strength_weakness: strength_weakness: - The proposed dataset and task are novel for video grounding tasks.
- The paper is well written and clear to understand.
- The proposed method is novel and specifically designed for long video grounding.
limitations: limitations: - Although the task proposed in the paper is multi-paragraph video grounding, I understand that it can also be applied to single-query scenarios. Adding some baselines for single-query video grounding, especially focusing on long video grounding (referring to the papers listed below), can make the experiments more comprehensive.

RGNet: A Unified Retrieval and Grounding Network for Long Videos.

Localizing Moments in Long Video Via Multimodal Guidance.

Cone: An efficient coarse-to-fine alignment framework for long video temporal grounding.
paper_summary: paper_summary: This paper proposed a large-scale multi-paragraph video grounding dataset which contains over 2800 hours TV show videos. This long paragraph query annotation requires method to model long range dependencies. Besides, it proposed a LGMR model to explicitly model local-global content in multimodal.
strength_weakness: strength_weakness: This paper is well written and clearly presented. The SynopGround dataset has longer videos and queries than existing video grounding dataset. The paragraph has both abstract expressions summarizing video storylines and specific descriptions portraying event details, involving multi-level semantics in video understanding. It proposed a baseline model that solves local and global semantics and achieves effectiveness in multi paragraph video grounding setting.
limitations: limitations: 1.	The setting proposed by this paper is not challenging enough, since many works have focused on video paragraph grounding, i.e. giving a paragraph but localizing moments for each sentence in given paragraph. (i) The ground truth moments in old paragraph grounding setting have much shorter normalized temporal span than your setting, so it is more difficult to localize a very short moment other than localize a long paragraph. (ii) if many sentences are correctly localized in a paragraph, the paragraph then can be localized. Understanding the contextual correlations of a sentence and it belonged paragraph is more difficult than only understanding the paragraph. So your setting is not so challenging.
2.	The TV show have subtitles and the paragraphs annotations are mainly correlated to subtitles especially the high-level semantics, so a nlp method can solve it. The visual information contributes little. It does not encourage models to learn multimodal.