Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper unified the generic video summarization method and the query-based video summarization method. Specifically, the paper proposed a language guided video summarzation approach that takes both video frames and text as input and summarizes the video frames under the influence of the text. For generic video summarization, the model relies on a video captioning method to provide the text guidance. The proposed approach achieves superior performance on both generic video summarization datasets and the query-based video summarization datasets.
main_review: Method:

1. I have concerns with the generic video summarzation part. The proposed approach relies on video captioning method to provide a text that describe the event in the video. This seems pretty straightforward conceptually. However, in practical, it relies on a very good video captioning system. 

  1.a The videos in summarzation datasets are very long. I wonder how many sentences are generated by the video captioning method? How to generate a caption for a day-long video? Did you sub-sample the frames?

  1.b The genres of videos in the summarzation dataset might be differ to the videos for training the video captioning model. How to handle the domain gap?

  1.c It seems the main reasons for the improvement in Table 1 and Table 2 are 1. Using Transformer. 2. Using better image feature (CLIP). The improvement brought by the caption is very limited (0.1 in Table1 under the setting of CLIP-Image + bi-LSTM. 0.7 in Table 1 under the setting of CLIP-Image + Transformer). The caption part only brought marginally improvement. It seems suggest the Caption is not useful? 

  1.d If the caption is helpful for video summarization.  I wonder what property would the caption need to improve the performance? Does the caption need to be concise or complicated? There are multiple off-the-shelf video captioning system. Why would the paper use the approach proposed by Iashin et. al?

  1.e It would be better to show the example of captions in SumMe and TVSum.

---

Rating:

My main concerns are with the captioning part and the ablation study in Table 1. Conceptually I think the proposed approach makes sense and the direction is very interesting. However, I have doubt on whether the experiments support the claim. I would happy to raise the score to 7 if the concerns are answered.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper describes a video summarization model that is trained to use either user provided query or automatically generated dense  captioning to select key-frames as summaries.

At the core of the approach is an attention mechanism that computes similarity between frame embeddings and text embeddings. Subsequently, each frame representation is replaced by a weighted-average (softmaxed) of the language embeddings; key and value embeddings are from the text model while query embeddings are from the image model. Finally, a score is assigned to each frame via a standard transformer based model. A post-processing step converts the key frame scores to key-shot level summaries such that only 15% of the videos are selected. The overall model is trained with a weighted Binary cross-entropy loss.

main_review:  
Strength: 
 - The idea of guiding visual summarization with language models is very nice. Intuitively, one would expect the caption/query of a video to represent the summary of visual concepts in it. I encourage the authors to explore the idea further.
- The paper is clearly written and well organized.
- The overall architecture (Clip Image + visual captioning + score transformers) is better than earlier approaches.

Weakness: 
- Some important details about the model are not discussed (see main questions below)
- The motivation for guiding video summarization with language model, as far as my understanding goes, is to train systems that can learn visual representation in correlation with high-level concepts encoded in text-embeddings. The paper did not show, empirically, that the introduced attention led to better visual representation learning. If this is shown it would make the paper a strong submission.
  - I recommend to design an experiment where the system is trained with language-guided attention, and evaluated without generating captions or the attention mechanism. If such a system proves to be better, then it would be a strong validation for the very nice motivation.

Main questions:
- How are N frames selected? The sampling method is not discussed, however, the sampling could have huge difference which might lead to significant misalignment with respect to a query.
- In the case of the generated captions, are the videos subsampled for the caption generation? if so is the sampling for caption generation and visual embedding the same?
- The paper discusses using global attention (i.e., computing similarity between each frame and textual embedding). Does this not lead to a high-entropy distribution? I am referring to the distribution defined by the softmax of the similarity scores. This can become particular problematic for captions/queries that are very long.
- Do you think a (dense) video captioning model (like the one used here) along with the scoring transformer would suffice to achieve the results? In other words, to what extent does the language-guided attention architecture plays a role in getting the results?
  - Table 1. Shows the important components of the model, with respect to performance, are the quality of the embeddings and the transformer architecture. For instance, in case of [Clip Image + video caption + bi-LSTM] the approach is comparable or worse, in most cases, than related works. 
- Can you say something about choosing the key and value embeddings to be from the text model rather than the video? Is this to avoid the the high-entropy problem in long sequences? ( see Question 3)

Significance: 
- The work address an important problem in a research application that is early on its development. 
limitations_and_societal_impact: Concerns are addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new and uniform method for generic and query-focused video summarization. In addition to typical video input, the method also takes caption or user query as input, and thus learns a language-guided representation to help video summarization. 
main_review: Strength: 

- Applying transformers to model different modalities and improve video summarization.
- The proposed method outperforms state-of-the-art generic summarization methods on both benchmark dataset by 5% and 3%, respectively.

Weakness: 

- The reconstruction loss: training the model to reconstruct keyframes is not very intuitive to the reviewer.  Any frame could be leant to reconstructed from the transformers, which does not help summarization, in particular for the unsupervised setting(s).
- The diversity loss only considers pairwise similarities among keyframes. This may be insufficient : as the major event may take a few frames to summarize, they are more similar to each other than a random frame (noise).
- There are manually annotated captions as ground truth for UT Egocentric dataset as shown in [A].
    - It would be more insightful to check the ground truth captions for video summarization.
    - [26] in this case could also be fair comparison.
- [26] can also use captions as input, the reviewer would encourage the authors to compare and report the results.
- The clarification of the paper could be improved:
    - Implementation: Are the weights on each loss summed up to 1? This is not claimed in the paper and chosen by heuristics.
    - Experiments: is the summary taking at most 15% of the whole video? This is very important for fair comparison. Also TVSum has a commonly used train/test split, as shown in [1, 2, 9, 25, 29, 42].
    - Typo: line 43, In case of generic; line 61, comprising of all; line 84, has lead to; line 258 Fig. 4 shows compares summaries; line 266, its hard to tell;

Additional comments: 

There are many objects and actions in a video, but most can be  not relevant to the major activities  and considered as 'background'. It would be great to visualize the attention weights among video frames to captions/queries. 

[A] Videoset: Video summary evaluation through text. Serena Yeung, Alireza Fathi, Li Fei-Fei. ArXiv 2014.
limitations_and_societal_impact: Yes. The authors discussed the impacts of the proposed method in Sec. 5.
review_title: null

Meta-review: This submission received the following final ratings: 6, 6, 7.

Reviewer X5gK had initially expressed concerns about the effectiveness of the approach and the value of captions for summarization. The responses on these points provided by the authors convinced the Reviewer to raise the rating to 7.

Reviewer Uj81 gives a rating of 6, but recommends adding details about the sampling of the generated captions to the paper.

Reviewer tjo5 appreciates the clarifications given in the author response and confirms the original rating.

The ACs agree with the recommendation of acceptance.
Decision: Accept (Poster)