Text-to-video generation helps video-language alignment
Authors: Luca Zanella, Massimiliano Mancini, Willi Menapace, Sergey Tulyakov, Yiming Wang, Elisa Ricci


Reviewer paper summaries: The paper presents an innovative method for enhancing Video-language alignment. This is achieved by leveraging video generators to create videos from contrast captions, which incorporate action and event order flips. To overcome the disparity between real and synthetic videos, a novel learning objective is introduced. This objective promotes alignment for both real and synthetic video-caption pairs, regularized by a consistency loss on scene semantics independent of temporal descriptions. They demonstrate their approach improves over 4 benchmarks:  VideoCon test sets, SSv2-Temporal, SSv2 Events, and ATP-Hard.

The paper introduces SynViTA, a novel framework designed to enhance temporal understanding in video-language alignment by leveraging synthetic videos. SynViTA uses a state-of-the-art text-to-video generator to create videos from contrast captions, which include action and event order flips. To address the alignment challenges of synthetic videos, the framework employs a mixture of experts to evaluate alignment scores and a consistency loss to focus on shared semantic content. SynViTA demonstrates significant improvements over existing methods on various benchmarks, highlighting its potential for enhancing video-language alignment.
Summary of strengths: The primary advantages of the paper are as follows:

- The authors introduced a novel approach to address the challenges associated with generating videos that do not strictly adhere to the provided textual description. The weighting strategy of the synthetic videos seems to be important and also enforcing consistency between the synthetic and real videos to prevent domain gap is useful.

- The authors demonstrated the superiority of their proposed method over other methods using only textual contrastive captions on two types of models, VIDEO-LLAVA and MPLUG-OWL 7B.

- The authors conducted comprehensive ablation studies to evaluate the impact of various design choices within their work, including the number of frames, the loss components and the weighting strategy of the synthetic videos.

* SynViTA is claimed to be the first method to leverage synthetic videos for video-language alignment, providing a novel approach to address temporal understanding challenges.
* The framework’s use of alignment scores to weigh the contribution of synthetic videos ensures that only well-aligned synthetic data influences the training process, improving the overall model performance.
* By introducing a consistency loss, SynViTA effectively reduces the domain gap between synthetic and real videos, ensuring the model focuses on shared semantic content rather than appearance, which enhances temporal alignment.
Summary of weaknesses: The primary weakness of the paper are listed below:

- The method proposed in the paper is based on generating synthetic videos. Nonetheless, there isn’t a figure that shows how these videos look like compared to the original video. TIt would be beneficial to include a figure showcasing an example of frames from both a real video with its original caption and a synthetic video with its contrastive caption, along with the weight score it received.

- The authors show that using an ensemble of models that evaluate the video-text alignment based on sampled frames obtains the best performance for the weight strategy. The authors also mentions that VideoCon which uses the video is not as good, as the image-based ensemble. It would be interesting to include experiment of using Gemini-Pro and GPT-4v on the 4 benchmarks, to show that both SynViTA (and VideoCon) training approach is better than these SOTA models in zero-shot.

- In Table 1, it seems that the improvement of the method is very minor compared to VideoCon, especially for ATP-hard. For ATP-hard it seems that using VideoCon causes degradation, and using SynViTA is comparable, although it uses more data compared to the vanilla models.

- Since the authors used synthetic videos, I would expect that it would improve the classifier’s ability to measure video-text alignment for both real and synthetic videos. However, the authors didn’t show any results for synthetic videos. Showing some results on datasets with synthetic video would show another benefit of using synthetic videos during training.

- The authors use only one type of video generator, there are two things that are not clear: 1) if the generated videos are realistic enough and how it will effect the results by using other video generation models, 2) are the synthetic videos have the same properties as the original videos, the same length, resolution, sample rate, etc.

*  The effectiveness of SynViTA heavily relies on the quality of synthetic videos generated. Any misalignment or artifacts in these videos could negatively impact the model’s performance. Could the authors provide some insights on how to mitigate this weakness?

* The process of generating synthetic videos and evaluating their alignment scores adds computational overhead, which may limit the scalability and efficiency of the framework in large-scale applications. Could the authors provide a computational complexity comparison with the other SOTA baselines?

* Since the alignment score between synthetic videos and their captions is central to the scoring criterion, could the authors show some ablation study on replacing the current implementation of alignment score with other criteria to show the effectiveness of this design?
* Could the authors provide some examples to show the synthetic videos generated from contrast captions while conditioning on the real positive videos?
Comments, suggestions and typos: It would be helpful to add to Table 1 the size of the training data of VideoCon vs. SynViTA. It is not clear if the improvement is because of the use of synthetic videos or by training with a larger training set.

The title should follow the Research Paper Title Capitalization Rules.
Overall assessment scores: 2.5

2.5