Question: The paper emphasises heavily on the video element, but why does page 6 state that experiments are conducted on a single still frame?
Evidence: RedditVC. We collect a dataset "RedditVC" of videos along with their titles and comment threads from social news site reddit.com, using their provided API. The videos are collected and used in a manner compatible with national regulations on usage of data for research. Unlike most curated video datasets, this data is more representative of the types of videos shared "in the wild", containing a large proportion of videogames, screenshots and memes.
Using a classifier trained on a small amount of labelled data, we estimate that videogame footage makes up 25% of examples, other screenshots, memes and comics make up 24%, live action footage is 49% and artistic styled content (such as drawn animation) is 2%. The average video length is 33s.
From 1 million raw videos collected, we perform deduplication and filtering, ending up with a training set of 461k videos, and validation and test sets of 66k videos each. For the video evaluation results in Table, we use a subset of the test set, consisting of 5000 videos with at least three comments each. In addition to comments, for one experiment in Table we also obtain textual labels of objects in the video thumbnails, using the Google Vision API. We obtain labels for 10,212 different classes, from "Abacus" to "Zwiebelkuchen". This serves as a useful comparison, giving textual metadata that can be expected to be more directly related to the visual video contents than user comments, and can be seen as a proxy for user-generated "hashtags". It also illustrates how, by using text as an intermediary, we can integrate predictions from any black-box classifier in the same framework.
KineticsComments. As an additional video dataset with comments, we construct a dataset based on Kinetics-700, for which we download the videos along with associated YouTube metadata including title, description and comments. We translate non-English titles and descriptions into English using a commercial translation API. We use the title as the primary text modality, and for auxiliary context we use comments and (when training) sentences from the description. This leaves us with 484,914 videos for training, each being around 10s. We also construct a test set, consisting of videos from the Kinetics test set for which we have at least 3 comments, giving a set of 2700 videos which we use to evaluate our method in Table.
We evaluate our Context Adapter Module on the above described datasets with comments in Table. We find that on both datasets adding comments boosts the retrieval performance significantly, confirming the value of the modality. Further, moving from images to videos also improves the performance. Since KineticsComments (human actions) is quite different to RedditVC (broad range of videos, games, etc.) combining both datasets bridges the domain gap and improves the evaluation performance on KineticsComments.
Baseline Comparisons. Since the main purpose of the Context Adapter Module is to extract meaningful information from sometimes unrelated auxiliary data, we perform most of the experiments in this section with an image/text backbone using a single video frame instead of a video/text backbone to reduce the computational burden of the evaluation.
In Table we evaluate the efficacy of the proposed Context Adapter Module against various baselines in various settings on our RedditVC dataset. The most basic baseline is achieved by not using any form of context adaption, which degenerates the model to the one proposed by.
Across all experimental settings, we find that finetuning the backbone architecture helps to improve the performance, while using the proposed CAM with frozen backbones (rows h,k) even outperforms the naive baseline of row (a). We also compare to two baselines. The first baseline is randomly replacing the title features with comments during training (Table(c-d)), however, this does not yield any improvement over no adaptation. In the second baseline we average the title features φ t and the . Surprisingly, we find that training with comments performs best when they are added to the image branch (rows j-l), as opposed to the text-branch (rows g-i). We hypothesize that this is because the comments can better adapt the semantically richer feature of the visual modality as opposed to the text, which can often be as short as a single word.
Adapting Different Modalities. Since both the visual embedding φ v and the text embedding φ t , come from the same embedding space and should be similar for the same sample, we perform an experiment where we swap the embeddings during evaluation time. In Table we find that there are still differences between φ v and φ t and swapping results in decreased performance. When we train both adapters we can achieve good performance in both cases.  Auxiliary Information. Additionally, Table shows how varying the source of the text at train and test time impacts the retrieval results. Whilst the performance of the model trained with image labels degrades when given comments at evaluation time, a model trained with comments still benefits from image labels at test time. As expected, the performance of both models is negatively impacted when tested with uncorrelated random words as additional text. However, the model trained with image labels does noticeably worse in this setting. This suggests that it has become overly reliant on image labels and easily fooled by irrelevant information, which overrides the title feature.
Varying the number of comments. In Fig. we vary the number of comments during training and evaluation time. Training on more comments results in better retrieval results when evaluated on the same number of comments. However, while there is little difference for models trained with only a few comments, we found that increasing the number of comments at train time leads to models that are more robust to different number of comments at test time, indicating that training with more comments learns an overall better context adapter module that is then able to work efficiently in various settings.
We evaluate our Context Adapter Module on the above described datasets with comments in Table. We find that on both datasets adding comments boosts the retrieval performance significantly, confirming the value of the modality. Further, moving from images to videos also improves the performance. Since KineticsComments (human actions) is quite different to RedditVC (broad range of videos, games, etc.) combining both datasets bridges the domain gap and improves the evaluation performance on KineticsComments.
Baseline Comparisons. Since the main purpose of the Context Adapter Module is to extract meaningful information from sometimes unrelated auxiliary data, we perform most of the experiments in this section with an image/text backbone using a single video frame instead of a video/text backbone to reduce the computational burden of the evaluation.
In Table we evaluate the efficacy of the proposed Context Adapter Module against various baselines in various settings on our RedditVC dataset. The most basic baseline is achieved by not using any form of context adaption, which degenerates the model to the one proposed by.
Across all experimental settings, we find that finetuning the backbone architecture helps to improve the performance, while using the proposed CAM with frozen backbones (rows h,k) even outperforms the naive baseline of row (a). We also compare to two baselines. The first baseline is randomly replacing the title features with comments during training (Table(c-d)), however, this does not yield any improvement over no adaptation. In the second baseline we average the title features φ t and the . Surprisingly, we find that training with comments performs best when they are added to the image branch (rows j-l), as opposed to the text-branch (rows g-i). We hypothesize that this is because the comments can better adapt the semantically richer feature of the visual modality as opposed to the text, which can often be as short as a single word.
Adapting Different Modalities. Since both the visual embedding φ v and the text embedding φ t , come from the same embedding space and should be similar for the same sample, we perform an experiment where we swap the embeddings during evaluation time. In Table we find that there are still differences between φ v and φ t and swapping results in decreased performance. When we train both adapters we can achieve good performance in both cases.  Auxiliary Information. Additionally, Table shows how varying the source of the text at train and test time impacts the retrieval results. Whilst the performance of the model trained with image labels degrades when given comments at evaluation time, a model trained with comments still benefits from image labels at test time. As expected, the performance of both models is negatively impacted when tested with uncorrelated random words as additional text. However, the model trained with image labels does noticeably worse in this setting. This suggests that it has become overly reliant on image labels and easily fooled by irrelevant information, which overrides the title feature.
Varying the number of comments. In Fig. we vary the number of comments during training and evaluation time. Training on more comments results in better retrieval results when evaluated on the same number of comments. However, while there is little difference for models trained with only a few comments, we found that increasing the number of comments at train time leads to models that are more robust to different number of comments at test time, indicating that training with more comments learns an overall better context adapter module that is then able to work efficiently in various settings.
Table 2: Ablation Study. Comparing Text-to-Video and Video-to-Text retrieval results between different baselines and our method.****Table 3: Adapting different heads. Text-to-video R@10 results on RedditVC-5. We either adapt the text or image features.****Table 5: MSVD (Chen & Dolan, 2011) Zero-shot text-to-video retrieval. CLIP is an important baseline since we use it as our backbone initialisation. Even though comments are not available for this benchmark the model trained with comments has learned better representations.****Table 1: Comments as a modality. Treating comments as an auxiliary modality to a video improves Text-to-Video retrieval for both RedditVC and KineticsComments test sets. As a baseline, we use the CLIP model on just the first frame. We then train just our CAM on top of CLIP, adapting the visual embedding with information from the comments, which improves retrieval metrics. Training our full model, which includes temporal attention, on RedditVC again validates that the inclusion of comments helps, with a further jump in Kinetics performance when using the Kinetics training set.****We find that the choice of frame does indeed matter, with the middle frame in a video performing better than the first frame. This result is relatively intuitive since the actual content of a video usually happens in the middle of the video which is similar to the photographer bias (pictures tend to display an object in the middle of the frame).
Answer: We do use video. The statement on page 6 refers only to the ablation experiments in Tables 2, 3 and 5. In Table 1 we show that using video does indeed help to learn better representations and Tables 5,6,7,8 and the supplement evaluate our video models. For the single image experiments, we are now running a comparison between evaluating on the first vs. the middle frame of a video and will post the results as soon as they are ready.