Question: Can you illustrate the structure of the network?
Evidence: In order to capture and filter the relevant information from the comments, we propose a transformerbased Context Adapter Module (CAM) which operates in a residual fashion, additively adapting either the visual or text branch of CLIP with contextual information obtained from the comments (see Fig.). Formally, we are now adding another modality-the comments-to the input which   extends it to x i = (v i , t i , c i,1 , . . . , c i,M ) with c i,k ∈ T . To reduce clutter in the notation, we have defined a fixed number of comments M for each sample. Since both, title and comments, share the same underlying modality, namely text, we can leverage the same encoder to transform comments to embeddings f t (c ik ) = φ c,ik .
As we expect the comments to be sometimes unrelated, our Context Adapter Module needs a mechanism to discount off-topic comments and update the primary modality φ v (v i ) or φ t (t i ), steering it in the most informative direction.
We introduce this mechanism as a function of both the primary modality and the comment embeddings φ c,ik , as we want to compare the informativeness of all these inputs at a high level. To this end, we design adapter modules g v and g t that extract information from the comments in the form of a residual:
With the adapted embeddings φvi and φti we recompute the affinity matrix (now Â) (Eq. 1) and use it for the loss L( Â). This design has several advantages. On one hand, extracting "only" a residual from the auxiliary inputs c ik means that the model is easily able to ignore them by predicting g(•) = 0. On the other hand, this effectively allows us to skip the adapter module when we benchmark the model on a dataset that does not have comments, while still learning the joint embedding from richer data.
In practice, we implement g as a small transformer architecture. Rather than operating on tokenised words, this transformer operates on embeddings (φ vi and φ ti ) themselves, taking as input the encoded feature from the branch to be adapted, along with comment features φ c,ik . By treating embeddings as tokens in their own right, we allow the embeddings to attend to each other and learn what combinations of the inputs should be used to update the original feature through the residual connection.
Additionally, to avoid bleeding information between the two modalities through the Context Adapter Module, during training, we only adapt either the video embedding with g v or the text embedding with g t . If we would use both adapters simultaneously, there is a trivial solution that minimizes the loss L: when the adapters learn to remove the original embedding through the residual, e.g.
both adapted embeddings become the same φvi = φti which trivially maximizes their similarity, thus preventing the model to learn a meaningful modality alignment. To prevent the model from learning a transformation of the embedding space through the residual, we train only one adapter at a time. We also consider the case where g v = g t and choose a random modality to adapt for each minibatch, leading to an adapter that is agnostic to the source of the primary embedding.
We evaluate our Context Adapter Module on the above described datasets with comments in Table. We find that on both datasets adding comments boosts the retrieval performance significantly, confirming the value of the modality. Further, moving from images to videos also improves the performance. Since KineticsComments (human actions) is quite different to RedditVC (broad range of videos, games, etc.) combining both datasets bridges the domain gap and improves the evaluation performance on KineticsComments.
Baseline Comparisons. Since the main purpose of the Context Adapter Module is to extract meaningful information from sometimes unrelated auxiliary data, we perform most of the experiments in this section with an image/text backbone using a single video frame instead of a video/text backbone to reduce the computational burden of the evaluation.
In Table we evaluate the efficacy of the proposed Context Adapter Module against various baselines in various settings on our RedditVC dataset. The most basic baseline is achieved by not using any form of context adaption, which degenerates the model to the one proposed by.
Across all experimental settings, we find that finetuning the backbone architecture helps to improve the performance, while using the proposed CAM with frozen backbones (rows h,k) even outperforms the naive baseline of row (a). We also compare to two baselines. The first baseline is randomly replacing the title features with comments during training (Table(c-d)), however, this does not yield any improvement over no adaptation. In the second baseline we average the title features φ t and the . Surprisingly, we find that training with comments performs best when they are added to the image branch (rows j-l), as opposed to the text-branch (rows g-i). We hypothesize that this is because the comments can better adapt the semantically richer feature of the visual modality as opposed to the text, which can often be as short as a single word.
Adapting Different Modalities. Since both the visual embedding φ v and the text embedding φ t , come from the same embedding space and should be similar for the same sample, we perform an experiment where we swap the embeddings during evaluation time. In Table we find that there are still differences between φ v and φ t and swapping results in decreased performance. When we train both adapters we can achieve good performance in both cases.  Auxiliary Information. Additionally, Table shows how varying the source of the text at train and test time impacts the retrieval results. Whilst the performance of the model trained with image labels degrades when given comments at evaluation time, a model trained with comments still benefits from image labels at test time. As expected, the performance of both models is negatively impacted when tested with uncorrelated random words as additional text. However, the model trained with image labels does noticeably worse in this setting. This suggests that it has become overly reliant on image labels and easily fooled by irrelevant information, which overrides the title feature.
Varying the number of comments. In Fig. we vary the number of comments during training and evaluation time. Training on more comments results in better retrieval results when evaluated on the same number of comments. However, while there is little difference for models trained with only a few comments, we found that increasing the number of comments at train time leads to models that are more robust to different number of comments at test time, indicating that training with more comments learns an overall better context adapter module that is then able to work efficiently in various settings.
Figure 2: Method Overview. We introduce a context adapter module that uses inputs of the auxiliary modality to adapt the embedding of another branch. With this module the model is able to accept or discount information.
Answer: The main architecture of our context adapter model is shown in Fig. 2. The backend is CLIP extended with the TimeSformer principle [1] for which we will add a diagram to the appendix.[1] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding?
arXiv preprint arXiv:2102.05095, 2021.