Learning Context-Adapted Video-Text Retrieval by Attending to User Comments

Learning strong representations for multi-modal retrieval is an important problem for many applications , such as recommendation and search . Current benchmarks and even datasets are often manually constructed and consist of mostly clean samples where all modalities are well-correlated with the content . Thus , current video-text retrieval literature largely focuses on video titles or audio transcripts , while ignoring user comments , since users often tend to discuss topics only vaguely related to the video . In this paper we present a novel method that learns meaningful representations from videos , titles and comments , which are abundant on the internet . Due to the nature of user comments , we introduce an attention-based mechanism that allows the model to disregard text with irrelevant content . In our experiments , we demonstrate that , by using comments , our method is able to learn better , more contextualised , representations , while also achieving competitive results on standard video-text retrieval benchmarks . 1 INTRODUCTION . Training large scale multi-modal models from paired visual/text data from the web has seen great success in video understanding and retrieval . However , typically only the caption or “ alt text ” is used during training , ignoring potentially relevant text present on the web page , such as user comments , tags , descriptions and other metadata . The challenge in using this extra context is that it may often be not directly relevant to the video ( a comment “ cool video ! ” ) , or it may be relevant but non-distinctive ( a tag “ cat ” could apply to many videos ) . Similarly , other modalities can also exhibit this behavior . For example , current work that learns from audio-visual correspondence ( Asano et al. , 2020 ; Alwassel et al. , 2020 ; Morgado et al. , 2020 ) need clean datasets such as VGGSound ( Chen et al. , 2020a ) to learn meaningful correspondence between videos and sound , whereas online videos tend to have for example background music that replaces the actual sounds happening in the video . In this paper , we propose a method that can take advantage of this auxiliary context while simultaneously filtering it for meaningful information . Most current models enforce a strict correlation between the different input modalities under the assumption that all are informative of the content . The main intuition of our work is that when training a model on partially unrelated data , we need to introduce a mechanism that allows the model to discount auxiliary data when it is not helpful for the task . To this end , we build a model with a hierarchical attention structure . Current representation learning models that are based on transformer architectures already exploit the idea of an attention mechanism to model the correlation between different parts of an input signal . For example in text understanding , the attention mechanism is applied per word , allowing the model to understand the structure of natural language . Even though in principle one could use the same scheme to model the importance of different text inputs on a per-word basis , we find that this makes it difficult to learn the individual importance of inputs . Moreover , due to the computational complexity of current transformers ( squared with sequence length ) this approach would only work for a small number of short comments . We thus add a second layer of attention per processed input that allows the model to assess the amount of information at a higher level of features . We find that this mechanism aligns well with the intuition that some inputs are very relevant to the problem and others can be disregarded . In our experiments we show that we can indeed learn meaningful information from user comments that translates to good zero-shot retrieval performance on common benchmarks . Additionally , we can show that the model can correctly identify whether auxiliary information is informative of the content of a video or not . The ability to incorporate auxiliary contextual information also opens up possibilities for useful applications . In the video retrieval setting our method can be used to iteratively refine a text descriptor with new inputs as shown in Fig . 1 , allowing incremental searching . In the zero-shot video classification setting ( i.e. , “ retrieving ” the correct class description prompt ) the prediction for an ambiguous video can be steered towards the correct class using surrounding text from a webpage or user hints ( Fig . 4 ) . 2 RELATED WORK . In this work , we focus on multi-modal learning with a particular focus on learning video-text encoders for retrieval by proposing a novel , multi-modal adaptation module . Video-text pretraining . Originating from the NLP domain , where the transformer architectures has been a key ingredient and subject to optimization in a multitude of ways ( Vaswani et al. , 2017 ; Devlin et al. , 2019 ; Radford et al. , 2019 ; Raffel et al. , 2019 ; Lewis et al. , 2020b ; Clark et al. , 2020 ; Lewis et al. , 2020a ) , it has recently found applications in the vision-language domain . For example , recent works have leveraged transformers to learn generalizeable image ( Desai & Johnson , 2020 ; Sariyildiz et al. , 2020 ; Chen et al. , 2020b ) or multi-modal image-text ( Li et al. , 2019 ; Lu et al. , 2019 ; Tan & Bansal , 2019 ; Su et al. , 2019 ; Li et al. , 2020 ; Chen et al. , 2019 ) and video-multilingual text ( Huang et al. , 2021 ) representations . A few works ( Sun et al. , 2019b ; a ; Zhu & Yang , 2020 ; Luo et al. , 2020 ) combine visual and text modalities as inputs to a BERT model to simultaneously learn semantic video and text representations . For representation learning , the availability of large-scale datasets such as HowTo100M ( Miech et al. , 2019 ) has enabled more effective pretraining of video-text representations for multiple downstream tasks . More recently , Patrick et al . ( 2020b ) show that adding a generative objective to contrastive pretraining can yield gains in video-text downstream tasks . Based on the CLIP model ( Radford et al. , 2021 ) , which works well even without finetuning for some retrieval tasks ( Luo et al. , 2021 ) , Bain et al . ( 2021 ) train video-text CLIP-initialized models by gradually scaling up video training from image training and a custom dataset . While we also start with a CLIP initialization as in Luo et al . ( 2021 ) , the focus of our paper lies in developing a novel method for leveraging user comments , a modality that has previously been overlooked in the text-video retrieval literature , as a valuable source of information . As is standard , the pretrained representations are subsequently evaluated on smaller datasets such as MSVD ( Venugopalan et al. , 2015 ) and MSR-VTT ( Xu et al. , 2016a ) . Multi-modal domain adaptation . While residual adapters for domain adaptation have been explored for uni-modal models such as CNNs , e.g . in Rebuffi et al . ( 2017 ) , there are no works that translate this concept to the multi-modal domain , where cross-modal learning dominates ( Alwassel et al. , 2020 ; Asano et al. , 2020 ; Morgado et al. , 2020 ) . Multi-modal pretraining . There is little prior work using user comments as additional context in multimodal settings . The use of user comments and reactions to refine predictions has been discussed by Halevy et al . ( 2020 ) for minimising harms on facebook.com . Overall , we are the first to demonstrate that user comments can be used as a complementary modality when learning video-text representations . 3 METHODS . In this section we will first recap the mechanism behind current contrastive , multi-modal representation learning methods that rely on clean data . We will then introduce our Context Adapter Module that allows learning from the auxiliary modality through an attention mechanism . Finally , we will describe how we can extend an existing backbone for images to videos , to be able to leverage large , pretrained models . 3.1 BACKGROUND . In multi-modal representation learning we are given a datasetX ofN samples xi ∈ X , i ∈ { 1 , . . . N } that individually consists of different signals . Most previous work focuses on two modalities and we will—for now—also adhere to this standard to simplify the notation . This means that each input sample xi = ( vi , ti ) is a pair of—in our case—a visual input vi ∈ V and its associated text , often the title , ti ∈ T , 1 ≤ i ≤ N . The goal is now to learn mappings fv : V 7→ Y , fv ( vi ) = φv , i and ft : T 7→ Y , ft ( ti ) = φt , i from each of the modalities to a d-dimensional , joint embedding space Y = Rd . Recent methods , such as Radford et al . ( 2021 ) , learn the mapping ( in their case from images and their captions ) to the embedding space with a double contrastive loss over a mini-batch B ⊂ X using an affinity matrix A computed between all pairs of samples in the batch : Aij = 〈 φv , i√ τ‖φv , i‖ , φt , j√ τ‖φt , j‖ 〉 ( 1 ) An entry Aij measures the similarity between the embeddings φv ( vi ) and φt ( ti ) via cosine similarity that is scaled by a temperature parameter τ . The idea is now to maximize the similarity between the embeddings from the same sample , i.e . the diagonal of A and minimize all non-diagonal entries . This can be achieved efficiently using a double-contrastive formulation that operates across columns and rows of A , L ( A ) = 1 2 |B|∑ i=1 Aii log ∑|B| j=1 expAij + Aii log ∑|B| j=1 expAji . ( 2 ) This formulation has the neat effect that it accomplishes maximizing the diagonal entries and minimizing all other entries of A in one self-balancing formulation . However , it makes the critical assumption that both modalities are equally informative of each other . In the case of sometimes irrelevant data , or when one modality has much less information content than the other ( e.g . “ nice video ! ” ) , this assumption does not hold and training with this objective will result in a very volatile learning objective and thus a sub-optimal joint embedding . In the next section we will introduce our Context Adapter Module that is able to deal with this type of inputs by allowing it to discount information when it is not relevant for the context . 3.2 CONTEXT ADAPTER MODULE . In order to capture and filter the relevant information from the comments , we propose a transformerbased Context Adapter Module ( CAM ) which operates in a residual fashion , additively adapting either the visual or text branch of CLIP with contextual information obtained from the comments ( see Fig . 2 ) . Formally , we are now adding another modality—the comments—to the input which extends it to xi = ( vi , ti , ci,1 , . . . , ci , M ) with ci , k ∈ T . To reduce clutter in the notation , we have defined a fixed number of comments M for each sample . Since both , title and comments , share the same underlying modality , namely text , we can leverage the same encoder to transform comments to embeddings ft ( cik ) = φc , ik . As we expect the comments to be sometimes unrelated , our Context Adapter Module needs a mechanism to discount off-topic comments and update the primary modality φv ( vi ) or φt ( ti ) , steering it in the most informative direction . We introduce this mechanism as a function of both the primary modality and the comment embeddings φc , ik , as we want to compare the informativeness of all these inputs at a high level . To this end , we design adapter modules gv and gt that extract information from the comments in the form of a residual : φ̂oi = φoi + go ( φoi , φc , i,1 , . . . , φc , i , M ) , o ∈ { v , t } ( 3 ) With the adapted embeddings φ̂vi and φ̂ti we recompute the affinity matrix ( now Â ) ( Eq . 1 ) and use it for the loss L ( Â ) . This design has several advantages . On one hand , extracting “ only ” a residual from the auxiliary inputs cik means that the model is easily able to ignore them by predicting g ( · ) = 0 . On the other hand , this effectively allows us to skip the adapter module when we benchmark the model on a dataset that does not have comments , while still learning the joint embedding from richer data . In practice , we implement g as a small transformer architecture . Rather than operating on tokenised words , this transformer operates on embeddings ( φvi and φti ) themselves , taking as input the encoded feature from the branch to be adapted , along with comment features φc , ik . By treating embeddings as tokens in their own right , we allow the embeddings to attend to each other and learn what combinations of the inputs should be used to update the original feature through the residual connection . Additionally , to avoid bleeding information between the two modalities through the Context Adapter Module , during training , we only adapt either the video embedding with gv or the text embedding with gt . If we would use both adapters simultaneously , there is a trivial solution that minimizes the loss L : when the adapters learn to remove the original embedding through the residual , e.g . go ( φoi , { φc , i , k , } ) = −φoi + φc , i,1 both adapted embeddings become the same φ̂vi = φ̂ti which trivially maximizes their similarity , thus preventing the model to learn a meaningful modality alignment . To prevent the model from learning a transformation of the embedding space through the residual , we train only one adapter at a time . We also consider the case where gv = gt and choose a random modality to adapt for each minibatch , leading to an adapter that is agnostic to the source of the primary embedding .

Summary: This paper proposes the use of user comments in addition to videos and titles to learn better representations for retrieval. Since user comments may be loosely related to the video, they use an attention-based mechanism to ignore irrelevant user comments. Their experiments show that using comments using this mechanism leads to better contextualized representations, which lead to competitive results on standard benchmarks.