Question: Maybe it doesn't look novel enough? It's an extension of CLIP that already has a notion of video and text (e.g., title). The extension is in terms of user comments, which is another form of the text. Though seemingly useful, the step to include comments as another form of text is not that surprising technically.
Evidence: We use CLIP as the initialisation for the backbone. Our concrete implementation of the CAM g is a 2-layer transformer, consisting of two residual multihead self-attention blocks.
The input consists of M + 1 input embeddings (for the M comments and title/video embedding φ oi ) having 512 dimensions each. Each block performs 8-head self-attention on the inputs, followed by two linear layers with output size 2048 and 512 respectively. LayerNorm normalisation is used, along with GELU activation following the first linear layer. From the M + 1 outputs of the transformer, we take only the output at index 1 (the index of the text/video embedding), and pass it through a final 512 × 512 linear layer. We use learning rate 0.001 for training the CAM, and 1 × 10 −6 when training the entire model, using the Adam Kingma & Ba (2015) optimizer.
All implementation and architecture details can be found in the appendix.
In this work, we focus on multi-modal learning with a particular focus on learning video-text encoders for retrieval by proposing a novel, multi-modal adaptation module.
Video-text pretraining. Originating from the NLP domain, where the transformer architectures has been a key ingredient and subject to optimization in a multitude of ways, it has recently found applications in the vision-language domain. For example, recent works have leveraged transformers to learn generalizeable image or multi-modal image-text and video-multilingual text representations. A few works combine visual and text modalities as inputs to a BERT model to simultaneously learn semantic video and text representations. For representation learning, the availability of large-scale datasets such as HowTo100M has enabled more effective pretraining of video-text representations for multiple downstream tasks. More recently, show that adding a generative objective to contrastive pretraining can yield gains in video-text downstream tasks. Based on the CLIP model, which works well even without finetuning for some retrieval tasks, train video-text CLIP-initialized models by gradually scaling up video training from image training and a custom dataset. While we also start with a CLIP initialization as in, the focus of our paper lies in developing a novel method for leveraging user comments, a modality that has previously been overlooked in the text-video retrieval literature, as a valuable source of information. As is standard, the pretrained representations are subsequently evaluated on smaller datasets such as MSVD and MSR-VTT.
Multi-modal domain adaptation. While residual adapters for domain adaptation have been explored for uni-modal models such as CNNs, e.g. in, there are no works that translate this concept to the multi-modal domain, where cross-modal learning dominates.
There is little prior work using user comments as additional context in multimodal settings. The use of user comments and reactions to refine predictions has been discussed by for minimising harms on facebook.com. Overall, we are the first to demonstrate that user comments can be used as a complementary modality when learning video-text representations.
We use CLIP (Radford et al., 2021) as the initialisation for the backbone.****While we also start with a CLIP initialization as in Luo et al. (2021), the focus of our paper lies in developing a novel method for leveraging user comments, a modality that has previously been overlooked in the text-video retrieval literature, as a valuable source of information.
Answer: Although our method uses CLIP as a backbone since it is a natural fit for this task, it is not exclusive to CLIP and should be easily applicable to other models as well.We agree that leveraging comments is an intuitive idea, however, training with comments in practice is not straightforward since they’re often only weakly correlated with the video and can even hurt performance if used naively. Hence, the technical novelty is in introducing a method that is able to make use of the valuable information present in comments, while discarding the irrelevant ones.To summarise, our main technical contribution is two-fold: 1) introducing our proposed context-adapter module that is able to learn from noisy modalities, and 2) demonstrating for the first time that user comments can help video-text retrieval.