Adaptively Building a Video-language Model for Video Captioning and Retrieval without Massive Video Pretraining

Authors: xiaoyu  wu
shengjin  wang
jiayao  qian
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 08:42:58

Abstract: Large-scale pretrained image-language models have shown remarkable performance recently. However, building a video-language model is more challenging due to the complexity of video and the difficulty of collecting high-quality data. This paper builds a video-language model in an adaptive manner, which transfers the knowledge from the image domain and can achieve state-of-the-art performance without any further massive video pretraining. The main contributions include a Visual Perception Adapter that seamlessly and efficiently adapts a pretrained image-language model to the video domain and a fine-grained contrastive learning with Inter-modal Token Alignment that bridges semantic gaps between vision, audio, and language with less data. The proposed model is evaluated on video captioning and retrieval. Experiments demonstrate that the proposed model exhibits competitive performance compared to models pretrained on millions of video-text pairs. Notably, our model's CIDEr and R@1 scores on the MSR-VTT dataset exceed the existing state-of-the-art by 6.3\% and 1.3\%.

Main paper: 
 introduction: 
 Recently, there has been a growing interest in video-language multimodal learning. Among various tasks, video captioning and video retrieval stand out as two quintessential tasks, demanding models with robust generative and representational capabilities, respectively. While traditional approaches [11, 29, 33, 35, 61] study different video multimodal tasks independently, recent works [7, 23, 25, 48, 60] unify representational and generative tasks through flexible architectures and scale up the model and dataset. While these methods improve the model's generalization, their significant training costs pose challenges for research and application.As shown in Figure 1 , current Vision-Language Models (VLMs) struggle to achieve both low training costs and high generalization. Traditional VLMs [11, 29, 33, 35, 61] are trained on datasets corresponding to different downstream tasks, which are usually manually annotated and high in quality but small in scale. These models lack multi-task capability (including generative and representational tasks) and are insufficiently generalizable. Recently, with the rise of the Big Convergence concept [3, 15] , large-scale pretrained VLMs that unify various multimodal tasks have emerged. Some [9, 42] are trained on large-scale video-text datasets [36] , achieving fair generalization performance. However, the difficulty of collecting high-quality video-text data results in unsatisfactory performance and incurs high training costs [15] . Some [7, 48, 53] go further by unifying video and image with a flexible architecture, leveraging higher-quality image datasets. They are trained on mixed datasets or through multiple stages, thus achieving excellent performance on multiple downstream tasks. However, the computational costs sharply increase, making training such models a severe problem in scenarios with limited resources. Some methods [30, 31] heavy pre-training but mostly focus on representational tasks like retrieval or classification. Our method skips video-text pre-training and utilizes an existing image-language model to achieve high generalizability, low training costs and strong multi-task capacity.To perform multitasking, a unified architecture is crucial. In addition, the strong generalization ability of large-scale pretrained VLMs is largely derived from large-scale image-text datasets, so models trained on such datasets contain valuable knowledge. Employing an image-text pretrained Multiway Transformer [23] as baseline, this paper proposes an adaptive methodology to build a robust, multi-channel, low-training-cost model with satisfactory generalization performance in generative and representational tasks.We propose two modules to achieve the goal. Firstly, we seamlessly adapt the baseline to the video domain with our proposed Visual Perception Adapter (VPA), which improves the original attention mechanism and enables to model videos with lower costs. Inspired by parameter-efficient fine-tuning (PEFT) methods [19] , VPA is designed to adapt the model to data with different distributions while minimizing disturbance to the original structure as much as possible. It is worth noting that most PEFT methods focus on the same tasks and the number of learnable parameters, while ours empower the model to conduct new tasks. VPA comprises a Grouped Temporal Attention (GTA) module and a Patch Feature Dropout (PFD) module. GTA actively reduces complexity by constraining the attention range in cross-attention, which is easy to implement and does not introduce new parameters or alter the entire attention mechanism like other variants with linear complexity [39, 50, 52, 62] . PFD passively reduces complexity by randomly dropping keys/values at each layer. While some works explore token masking during the input stage [17] , our PFD works more like dropout, functioning at the feature level and varying at each layer. In addition, to utilize the audio channel in videos, we add audio modality experts to the baseline as in [4] , which only increase a small number of Feed Forward Network (FFN) parameters.Secondly, we propose a fine-grained trimodal contrastive learning with Inter-modal Token Alignment (ITA). To avoid massive video-text pretraining, we directly train the adapted model using high-quality downstream datasets, inevitably facing the semantic gap between images and videos. To address this issue, we better utilize these high-quality data by employing a top-k strategy for finer-grained semantic alignment between different modalities instead of traditional coarse-grained alignment at the sentence/video level. In summary, our contributions are as follows:• We establish a robust multi-channel video-language model in an adaptive way that maximally leverages the knowledge learned by existing image-language models from large-scale datasets. It seamlessly transfers the knowledge to the video domain without the need for massive video pretraining and is able to perform both representative and generative tasks. • We propose two components for the adaptive building method:VPA and ITA. VPA optimizes the patch-level spatio-temporal attention mechanism while keeping minor disturbance to the inherited knowledge. ITA establishes token-level finegrained semantic alignment across three modalities to efficiently utilize downstream datasets.• Experiments demonstrate the state-of-the-art performance of our model. We take video captioning and retrieval as examples of representative and generative tasks. In particular, our model's CIDEr and R@1 scores on the MSR-VTT dataset exceed the existing state-of-the-art by 6.3% and 1.3%. 
related works 2.1 vision-language pretraining: 
 Vision-language pretraining aims at acquiring powerful visionlanguage representations through large-scale training of noisy image/video-text pairs collected from the Web. These models [21, 25, 40, 60] are usually trained by the contrastive loss or the language modeling loss. BLIP-2 [23] is a state-of-the-art image-language model which combines the advantages of [1, 4, 20, 24] from both architectural and data perspective, however, it cannot handle video tasks. Due to difficulties in utilizing videos effectively, video-text pretraining is more challenging. Videos are usually modeled by temporal encoding modules [27, 43] , 3D local attention features [27, 29, 42] or global patch-level self-attention [9] . Furthermore, many video-language models [27, 29, 42] are trained solely on videotext datasets, neglecting the use of high-quality image data, while some also exploit image-text data [9, 48, 53, 56] and achieve better performance. However, these methods incur higher training overhead, and this paper proposes a method that achieves comparable results without extensive video-text pretraining by adapting image models to video tasks.Instead of combining image-text data with video-text data for largescale pretraining, transferring a pretrained image model to the video domain is more efficient. Several approaches improve the effectiveness of video recognition by extending pretrained image models [2, 5, 37, 59 ]. In addition to these unimodal methods, VideoOFA [9] and VideoCoCa [56] are the closest works to the idea proposed in this paper. They transfer image-text models OFA [49] and CoCa [60] , [4, 23] and our improved encoder (c). We first extract visual and audio features using pretrained models and then feed them along with text tokens and learnable queries into a Multiway Transformer Encoder. Finally, the model is supervised by multiple losses, including our proposed ITA loss. The baseline encoder consists of a shared self-attention layer, a visual cross-attention layer, and two modality-specific FFNs. Our improved encoder adds a audio pathway and replaces the cross-attention layer with the proposed VPA, which is further composed of PFD and GTA.respectively. However, these approaches still require video-text pretraining and suffer from the inefficiency of processing patch-level features. This paper demonstrates that our adapted model achieves better results without further video pretraining. Furthermore, we reduce the computational cost of processing patch-level features through dimension reduction and local attention.Video captioning and video retrieval are two downstream tasks performed in this paper to validate the adapted model. Most of the previous approaches improve the performance by adopting new network architectures, such as multi-way feature fusion [11, 33, 44] , attention mechanisms [11, 58, 61] , multimodality [10, 33, 61] , special text generation mechanisms [61] . Although these approaches enhance performance in diverse aspects, they are constrained to smallscale datasets with careful labeling. This paper focuses on adapting a pretrained image-language model and transferring knowledge from large-scale datasets to downstream tasks. 
method: 
 Our workflow constructs a Multiway Transformer [4, 23] as baseline network first, followed by adaptive enhancements. This section begins with the constructed overall framework (Figure 2 ), followed by individual sections on the proposed VPA and ITA modules.We start by uniformly sampling 𝐿 𝑣 frames from the video to form the keyframes set F. Each frame undergoes individual encoding utilizing a pretrained vision encoder (E 𝑣 ) based on Vision Transformer [14] , resulting in patch-level features (𝒗 𝑝 ), supplemented with temporal encoding (𝜺) [46] .EQUATIONwhere 𝑖 represents the 𝑖-th frame in 𝐿 𝑣 frames and 𝐷 𝑣 denotes visual feature dimension. The audio is partitioned into 𝐿 𝑎 non-overlapping segments A. And a Transformer-based pretrained encoder (E 𝑎 ) [16] and a linear layer is then applied to extract segment-level features (𝒂), which are also supplemented with temporal encoding (𝜉 𝑗 ).EQUATIONwhere 𝑗 represents the 𝑗-th segment in 𝐿 𝑎 segments, 𝑾 𝑖𝑛𝑝 is the learnable weight matrix, and 𝐷 𝑚 denotes model dimension. For text, we utilize tokenizing and embedding methods identical to [13] , obtaining 𝐿 𝑡 token-level embedding vectors.EQUATIONwhere T is the tokenized one-hot vector of text, and 𝑾 𝑒𝑚𝑏 is the learnable embedding matrix. The pretrained visual encoder and audio encoder are kept frozen through the whole process.Next, these features are fed into a Multiway Transformer encoder [4, 23] comprising 𝑀 blocks. For simplicity, we will omit the block index and incorporate pre-layer normalization and residual connection within the layer function. Several learnable queries and text tokens are concatenated in a token sequence and fed into a shared self-attention layer. Additionally, by controlling attention mask m, we can either block information interaction between different modalities or implement causal self-attention. Formally,EQUATIONwhere Φ 𝜃 𝑠 denotes the self-attention layer. Then, the encoded queries and video patch features are fed into cross-attention to obtain visual information. Furthermore, the outputs of self-attention and cross-attention are encoded by modality-independent FFNs:EQUATIONwhere Φ 𝜃 𝑐 denotes the cross-attention layer, F denotes the FFN layer with parameters of different modalities. Finally, after 𝑀 blocks of encoding, the outputs of the last block are denoted [ q, t] for loss function. The baseline employs three loss functions described in [23] , namely, language modeling loss, contrastive loss, and matching loss:EQUATIONThese losses cover generative and encoding objectives, which can facilitate both video captioning and retrieval tasks. For more details, we recommend readers to refer to [23] .Our improvements to the baseline are threefold: Firstly, we add input for audio features and its corresponding modality FFN. Secondly, we replaced the cross-attention module with the VPA module, which will be detailed in Section 3.2. Thirdly, we enhanced the contrastive loss to the fine-grained ITA loss, which will be detailed in Section 3.3. It is important to note that the improved model inherits all parameters except for the Audio FFN, thus preventing loss of learned knowledge. Formally, Equations (4)(5)(6) are improved as:EQUATIONEQUATIONEQUATIONwhere Φ ′ 𝜃 𝑐 denotes VPA and L 𝑖𝑡𝑎 , 𝜆 𝑖𝑡𝑎 denote ITA loss and its weight.VPA is employed to enable a fix number of learnable vectors to perceive video patch-level features through enhanced cross-attention mechanism while maintaining low complexity. As shown in Figure 2(c ), VPA replaces the cross-attention module in the baseline and consists of two parts: PFD and GTA. Video features first undergo random dropout through PFD. Subsequently, they serve as keys/values for grouped local attention with perceptual vectors.Feature Dropout. This module directly reduce the number of patches for efficiency during training. As shown in Figure 3 , 𝑃% of patches are randomly dropped before cross-attention:EQUATION)where ℎ 𝑖 is the index of the randomly selected patches and 𝒗 - 𝑝 is the features after PFD, whose length is decreased from 𝐿 𝑣 to 𝑛 = ⌊𝐿 𝑣 • (1 -𝑃%)⌋. Our method implements this specific dropout at the feature level rather than the pixel level, so that the patch features of the video can be pre-extracted, which also expedites the training process. Moreover, dropping on patch level allows the model to reason through partial information, facilitating the model's understanding of videos and suppressing overfitting. Compared to BERT [13] or BEiT [3] , our method directly drops the tokens instead of replacing them with [mask] tokens, which could lower the computational costs. Compared to MAE [17] , our approach drops differently at each layer and refrains from reconstructing the dropped (or masked) tokens, which gets rid of an extra decoder.Temporal Attention. This module further alleviates the increasing complexity of cross-attention with the growing number of video patches while introducing no extra training parameters.As shown in Table 1 , there are various works aiming to reduce the quadratic complexity of self-attention. However, they either involve additional training parameters (row 3), offer limited reduction in complexity (row 4), cannot be applied to cross-attention (row 5), or significantly changes the original attention calculation (row 6).It is crucial to note that such works mainly focus on scaling up language models or handling long sequence time-series data, which differs from the case in this paper.Our approach targets a visual perception mechanism with learnable vectors as queries and video patch features as keys/values. Therefore, prior methods like CosFormer [39] , which enhances the weights of the diagonal region in the attention matrix, are not applicable. However, our situation allows for the introduction of a different prior knowledge: the high correlation between spatially and temporally adjacent patches in videos. This correlation enables the extraction of local structures and motion information, sharing similarities with the benefits of using CNNs to process images.Specifically, we first reshape 𝒗 𝑝 of dimensions 𝐿 𝑣 × 𝐷 𝑣 into a tensor of dimensions 𝑇 × 𝐻 × 𝑊 × 𝐷 𝑣 , which can be viewed as a cube where each small block is a vector of dimension 𝐷 𝑣 . Then, as shown in Figure 4 , this cube is divided into 𝐺 non-overlapping windows of size 𝑡 × ℎ × 𝑤, and queries (𝒒) are also divided into 𝐺 groups, each with a length of 𝑙 𝑞 . The size and shape of the windows, as well as the number of queries, can be adjusted arbitrarily while ensuring that 𝑡, ℎ, 𝑤 are positives. The upper part of Figure 4 shows a couple of examples. Next, queries and the windowed patches from the same group perform cross-attention with a time complexity of O (𝑡ℎ𝑤𝑙 𝑞 𝐷 𝑚 ) = O (𝑇 𝐻𝑊 𝐿 𝑞 𝐷 𝑚 /𝐺 2 ):EQUATIONwhere G(•) 𝑔 denotes the grouping operation as described above, the subscript 𝑔 denotes the 𝑔-th group, and 𝑾 𝑞 , 𝑾 𝑘 , 𝑾 𝑣 are learnable matrices. Equation ( 11) is repeated 𝐺 times, which is implemented by batching and results in a complexity of O ( 𝑇 𝐻𝑊 𝐺 𝐿 𝑞 𝐷 𝑚 ) for GTA. In Figure 4 , Example (a) represents the vanilla approach of full attention (see Figure 2(b) ). This method applies global spatiotemporal attention, exhibiting high complexity and the possibility of introducing noise. Example (b) illustrates the case with a window size of 1×4×4, involving global spatial attention and local temporal attention modeling, similar to the approach in [23] for image processing. Example (c) depicts the scenario of 2 × 2 × 2, incorporating local spatiotemporal attention, allowing each query to focus on a small spatiotemporal region. Example (d) showcases the situation a of 2 × 4 × 4 window, featuring global spatial attention and local temporal attention modeling. This aligns with the pretrained model we used, mitigating the discrepancy between pretraining and downstream tasks. Additionally, this method facilitates modeling of short-term temporal information, avoiding introducing significant noise when directly modeling long-term temporal information.To accelerate training using mini-batches and when simultaneously using GTA and PFD, it is necessary to keep the number of dropped patches within each window to be the same. Therefore, Equation ( 11) is rewritten as:EQUATIONwhere the superscript minus symbol denotes the PFD operation as in Equation (10) . Therefore the Φ ′ 𝜃 𝑐 in Equation ( 8) is rewritten as:EQUATIONwhere Concat(•) denotes concatenation. As shown in Figure 2 , the [𝒒 𝑓 , 𝒂 𝑓 , 𝒕 𝑓 ] of the last block are denoted as [ q, ã, t]. Contrastive learning has been demonstrated in a number of works to be beneficial in bridging the semantic gap between different modalities [7, 23, 40, 58] . To enable the adapted model to fully exploit the information from multichannel videos, we propose a fine-grained vision-audio-text contrastive loss to align multimodal features. The vanilla contrastive loss based on InfoNCE [45] is as follows:EQUATIONwhere 𝒙 and 𝒚 are positive pairs from two modalities (e.g., visual queries and audio tokens), N 𝑦 is the set of negative samples of 𝑦 modality from the rest of the batch, 𝜂 is the temperature coefficient, and 𝑆 (•, •) is the similarity function, typically using cosine similarity.In the context of this paper, both 𝒙 and 𝒚 are feature sequences. To obtain the similarity between them, common approaches include using pooling or additional encoding layers [34] to get the overall features of the sequences, followed by calculating cosine similarity. We argue that such methods either fail to utilize fine-grained relations between modalities or introduce additional parameters. Another model [23] , which is also based on the Perceiver architecture [20] , computes the similarity between each query and the text's [CLS] feature, and then selects the maximum value as the final similarity. However, in our case, each query only represents local visual information from the video, and combining information from multiple queries is necessary for better alignment with the text. Therefore, we generalize the calculation of 𝑆 using a top-K strategy as illustrated in Figure 5 and following equations:EQUATION)where TopK(•) indicates taking the largest 𝐾 value from the matrix. The outputs of the last block of the backbone are denoted [ q; ã; t], and we calculate the bidirectional contrastive losses between these three modalities: EQUATIONEQUATIONEQUATIONThe grey item of MV-GPT is re-evaluated by [57] . The grey parameters count of mPLUG-2-base is estimated by us, and the grey results on MSR-VTT indicate they use a different split, leading to an unfair comparison.EQUATION 
experiments 4.1 experiment setup: 
 We perform a comprehensive evaluation of our proposed method on six benchmarks covering two downstream tasks: video captioning and video retrieval.For video captioning, we evaluate on three widely used datasets (MSVD [6] , MSR-VTT [54] , VATEX [51] ), as well as one novel dataset (VALOR-32k [7] ). We utilize the following metrics to evaluate the performance of our model: Bleu@4 (B@4) [38] , METEOR (M) [12] , ROUGE-L (R) [28] , and CIDEr (C) [47] . During the inference process, we employ beam search with a beam size of 3, which is generally used in many video captioning methods. Video retrieval performance is evaluated using the MSR-VTT dataset [54] and the DiDeMo dataset [18] . We use text-to-video recall at rank K (R@K, where K=1, 5, 10) as our evaluation metrics. And we follow the same evaluation process as in [23] . We use CLIP-ViT-L/14 [40] and AST [16] finetuned on AudioSet as the visual encoder and the audio encoder. We also inherit the weights of BLIP-2 [23] . By default we empirically extract 8 frames at equal intervals for datasets (MSR-VTT, MSVD, VATEX, VALOR) with shorter video lengths, and 16 frames for DiDeMo dataset. Please refer to the supplementary material for detailed experimental setups.We compare the proposed model with other state-of-the-art models in various aspects. As shown in Figure 6 , our model achieves leading results both in trainable parameters and the video-text pairs used, demonstrating the superiority of our adaptive approach. In detail, as shown in Table 2 and Table 3 , compared to models without video-text pretraining, our model basically achieves state-of-theart results on all five benchmarks. One exception is the GIT [48] on video captioning, which utilizes up to 800M higher-resolution image-text pairs for pretraining. In the captioning task, our model outperforms methods with parameters below 1B and training pairs below 10M. Compared to methods with higher training cost, our approach still outperforms methods with pretraining data exceeding 100M (CLIP4Cap [43] , MV-GPT [42] ) and the model with 5 times the parameters (VideoCoCa [56] ). It even outperforms methods with training parameters more than 20 times larger than ours (GIT2 [48] ) on the MSR-VTT dataset. It's important to note that the GIT2 model, which achieved the best comprehensive metrics, constructs a model with 5.1B parameters and uses up to 12.9B image-text pairs for training, and thus exhibits a significant improvement compared to [11] .Vision Encoder PT Video-Text Learnable Params Extra Modalities MSR-VTT DiDeMo R@1 R@5 R@10 R@1 R@5 R@10 all other methods. In contrast, our method focuses on constructing a model with reasonable parameters, leveraging existing visuallanguage models, and achieving superior performance without additional video-text pretraining. In the retrieval task, compared to methods with video-text pretraining, our approach achieves competitive results, surpassing CLIP4Clip [34] and CLIP-ViP [55] , which utilize over 100 million video-text pairs. Additionally, our [26] and VALOR 𝐿 [7] , which employed video datasets exceeding 10 million instances.For models with audio, we surpass them with R@1 scores of 60.7 and 65.3 in retrieval and a CIDEr score of 80.7 in captioning. We also achieve the best results in B@4 and METEOR metrics on the VALOR dataset, which has a higher demand for audio understanding. It is important to note that VALOR and VAST are trained on much larger and audio-focused corpus. However, our method achieves competitive results with much less audio for training. VideoOFA [9] and VideoCoCa [56] are the closest works to our approach, both aiming at enhancing video-language learning using existing image-language models. Compared to them, our model has fewer trainable parameters and achieves better results without millions of videos for pretraining. gradually adding modules to the baseline (Figure 2 (b) and Equation ( 6)). We found that the audio modality, GTA module, PFD module, and ITA loss all effectively enhance the model's performance.The best results are obtained by employing all four strategies. More ablations of GTA and ITA are in Section C of the supplementary.Mechanisms. We compare several linear-complexity attention mechanisms mentioned in Table 1 , and the results are presented in Table 6 . We replace the cross-attention with their improved attention mechanism according to their open-source codes. We found that introducing new learnable parameters (LinFormer [50] ) or changing the operation of attention mechanism (FlowFormer [52] ) can disrupt the model's original knowledge, leading to unsatisfactory performance. InFormer [62] performs slightly better than others, but in the context of this paper, limiting the length of the query has minimal impact on efficiency improvement. Our proposed GTA involves minimal modifications to cross-attention and is tailored for scenarios that require adapting an image-text pretrained cross-attention to the video domain while minimizing disruption to features. We attribute this to an excessive restriction on the model. The best results are obtained when the dropout rate was reduced to 30%.Visualization. Figure 7 is a sample from the dataset. In this example, where other models predict similar simple descriptions, our model predicts more fine-grained descriptions, which we attribute to our fine-grained features and ITA loss function. More qualitative results can be found in the supplementary. 
conclusions: 
 This paper presents a novel and efficient approach to adaptively construct a video-language model. The proposed method transfers knowledge from the image-language domain to the video domain, efficiently processes the abundant information in videos through a Visual Perception Adapter, and narrows the semantic gaps between vision, audio, and text modalities by employing a fine-grained trimodal contrastive learning with Inter-modal Token Alignment.Extensive experiments validate that the constructed model attains satisfactory performance without the need for large-scale videotext pretraining. Limitation: Our model focuses on processing short videos and is not able to handle longer videos. To address this issue, we plan to explore efficient and low-data-demand methods for modeling longer videos. Broad Impact: The outstanding performance of this model demonstrates its potential to be applied to multimodal large language models, where it can serve as an efficient video feature extractor. 
 
 Appendix:  


Decision: Initially, the paper received mixed reviews, with ratings oscillating between "Borderline Reject" and "Borderline Accept." Reviewers appreciated the clarity of the presentation and the novelty of the proposed methods but raised concerns about the over-claims regarding the novelty, lack of sufficient comparison with state-of-the-art methods, and generalization evidence.

After the rebuttal, the authors addressed several concerns, particularly around the novelty and comparison with existing works. However, some issues such as the generalization of the model across different datasets and tasks, and the dependence on the choice of the pretrained image-text model remained partially addressed. The final ratings leaned more towards acceptance after considering the rebuttal and additional clarifications provided by the authors.

Reviews: review: paper_summary: paper_summary: This paper addresses the challenges of building video-language models by transferring knowledge from large-scale pretrained image-language models, thus avoiding the need for extensive video pretraining. The key contributions include a Visual Perception Adapter that adapts image-language models to the video domain efficiently and a fine-grained contrastive learning approach with Inter-modal Token Alignment to bridge semantic gaps between vision, audio, and language. The model is evaluated on video captioning and retrieval tasks, demonstrating competitive performance compared to models pretrained on millions of video-text pairs.
strength_weakness: strength_weakness: 1. The tackled problem is important.
2. The proposed method is intuitive.
3. The paper is overall easy to follow.
limitations: limitations: 1. The current manuscript over-claims the difference. For example, in Figure 1, the message audience probably receives is that this is the first work leverages existing pretrained model to avoid pretraining on video. But there has been an existing extensive effort along this line [a,b,c]. However, the current manuscript fails to discuss and acknowledge these existing work, and thus leave the audience a wrong impression about the position of the work. It is better to compare with at least a few of these work on multi-channel inputs, which were also shown to achieve SOTA performance [a,b,c] when without any pretraining on videos.

2. The authors stressed on the generalization but there is no evidence on any common ways of showing generalization, such as cross-dataset evaluation or out of distribution evaluation.

[a] VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs

[b] Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

[c] Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval
paper_summary: paper_summary: Recently, large-scale pretrained image-language models have shown remarkable performance recently. However, how to train a strong video-language model under training costs is more challenging due to the complexity of video and the difficulty of collecting high-quality data. This paper proposes two modules called VPA and ITA to build a video-language model in an adaptive manner, which transfers the knowledge from the image domain and can achieve state-of-the-art performance without any further massive video pretraining. However, there are still some issues in the paper that need to be addressed.
strength_weakness: strength_weakness: 1.	The pipeline proposed in this paper is particularly clear and easy to understand, which makes it accessible even to those who may not be experts in the field. This straightforward approach ensures that researchers can understand and apply the methods discussed.
2.	The paper writing is commendable for its clarity and straightforwardness. The authors use simple and direct language, making complex concepts more understandable.
3.	The paper conducts sufficient experiments to verify the effectiveness of the proposed method for video captioning and retrieval.
4.	The proposed method has the potential for further improvement.
limitations: limitations: 1.	Performing Parameter-Efficient Fine-Tuning (PEFT) on downstream tasks based on a general large model is a common approach. However, you should compare your proposed VPA with some mainstream PEFT methods to demonstrate its advantages, such as LoRA, adapters, soft prompts, and so on.
2.	Can the method you proposed be used in other downstream tasks, such as video question answering?  Other works, such as mPLUG-2, have been validated on more downstream tasks.
3. Your study improves upon the Q-former of BLIP-2, incorporating new components. However, as demonstrated in Table 5, the proposed GTA and PFD do not markedly boost performance. Does this mean that even without using GTA and PFD, the performance is already quite good with the original setup?
4. If addressing the aforementioned issues, it would greatly benefit your work.
paper_summary: paper_summary: This paper presents an innovative approach to adapting an image-text pretrained model, BLIP-2, into an efficient video-text model. The resulting model demonstrates impressive performance on both video captioning and video retrieval tasks across various benchmarks, achieving these results without the need for extensive video-text pretraining. The authors highlight the high performance and reduced training costs associated with their method.
strength_weakness: strength_weakness: 1. The paper is clearly written, well-structured, and easy to follow.

2. The experiments are thorough, demonstrating the proposed model's competitive performance in both video captioning and retrieval tasks without relying on video-text pretraining.

3. The Visual Perception Adapter (VPA) is a novel contribution. Ablation studies validate its effectiveness in reducing computational costs while maintaining high performance in video modeling.
limitations: limitations: 1. Further Potential with Video-Text Pretraining: Although the authors emphasize that their method does not require video-text pretraining, it raises curiosity about whether incorporating video-text pretraining could further enhance the model's performance.

2. Dependence on Image-Text Pretrained Model: The method relies on a pretrained image-text model as the foundation. There is a lack of discussion on how the choice of this pretrained model and the effectiveness of the image-text pretraining might influence the proposed method's performance. Currently, experiments are conducted using only a single baseline, BLIP2.

3. Insufficient Citation and Discussion of Stronger SOTA Methods: The paper lacks sufficient references and discussions of stronger SOTA methods, such as "VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset" (NeurIPS 2024). Including such comparisons would provide a more comprehensive understanding of the model's relative performance.
paper_summary: paper_summary: The paper introduces a framework for adapting an image-language model for video-language tasks. The methods consist of a novel Visual Perception Adapter that encodes videos at a lower cost, and a fine-grained trimodal contrastive learning with Inter-modal Token Alignment. The paper includes extensive experiments and analyses to support the effectiveness of the proposed methods.
strength_weakness: strength_weakness: - The paper is easy to follow.
- The proposed methods are novel and effective in adapting an image-text model for video-language tasks.
- The experiments are extensive.
limitations: limitations: - Adding some Video QA tasks can make the experiments more solid.

- Some suggestions:

1) There is insufficient information on ITA in the introduction section.
2) Figure 1 could be considered for placement on the first page.
3) Figure 2 might be improved by including some brief information about modules like PFD, GTA, ITA, etc., and emphasizing the design of the method in the main figure.