Learning Context-Adapted Video-Text Retrieval by Attending to User Comments
Authors: Laura Hanu
Yuki M Asano
James Thewlis
Christian Rupprecht
Venue: ICLR 2022 Submitted (2022)
Keywords: Multimodal Representation Learning
Video
Text
Retrieval
User Comments

Abstract: Learning strong representations for multi-modal retrieval is an important problem for many applications, such as recommendation and search. Current benchmarks and even datasets are often manually constructed and consist of mostly clean samples where all modalities are well-correlated with the content. Thus, current video-text retrieval literature largely focuses on video titles or audio transcripts, while ignoring user comments, since users often tend to discuss topics only vaguely related to the video.
In this paper we present a novel method that learns meaningful representations from videos, titles and comments, which are abundant on the internet. Due to the nature of user comments, we introduce an attention-based mechanism that allows the model to disregard text with irrelevant content. 
In our experiments, we demonstrate that, by using comments, our method is able to learn better, more contextualised, representations, while also achieving competitive results on standard video-text retrieval benchmarks.


Under review as a conference paper at ICLR 2022
LEARNING CONTEXT-ADAPTER VIDEO-TEXT RETRIEVAL BY ATTENDING TO USER COMMENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Learning strong representations for multi-modal retrieval is important problem for many applications, such as recommendation and search. Current benchmarks and even datasets are often manually constructed and consist of mostly clean samples where all modalities are well-correlatd with the content. Thus, current video-text retrieval literature largely focuses video titles audio transcripts, while ignoring user comments, since users often tend to discuss topics only vaguely related the video. In this paper we present novel method that learns meaningful representations from videos, titles and comments, which are abundant on the internet. Due to the nature of user comments, we introduce an bbased mechanism that allows the model to disregard text with irrelevant content. In our experiments, we demonstrate that, by using comments, our method is able learn better, more contextualised, representations, while also achieving competitive results on standard video-text retrieval benchmarks.
INTRODUCTION
Training large scale multi-modal models from paired visual/text data from the web has seen great success in video understanding and retrieval. However, typically only the caption or "alt text" is used during training, ignoring potentially relevant text present on the web page, such as user comments, tags, descriptions and other metadata.
The challenge in using this extra context is that may often not directly relevant to the video (a comment "cool video!"), or it may be relevant but non-distinctive (a tag "cat" could apply to many videos). Similarly, other modalities can also exhibit this behavior. For example, current work that learns from audio-visual correspondence (Asano et al., 2020; Alwassel et al. 2020; Morgado et al. 2020) need clean datasets such as VGGSound (Chen et al. 2020a) to learn meaningfll correspondence between videos and sound, whereas online videos tend have for example background music that replaces the actual sounds happening in the video.
In this paper, we propose a method that take advantage of this auxiliary context while simultane- ously filtering it for meaningful information. Most current models enforce a strict correlation between the different input modalities under the assumption that all are informative of the content. The main intuition of our work is that when training a model on partially unrelated data, we need to introduce a mechanism that allows the model discount auxiliary data when it is not helpful for the task.
To this end, we build model with a hierarchical attention structure. Current representation learning models that are based on transformer architectures already exploit the idea of an attention mechanism to model the correlation between different parts of an input signal. For example in text understanding. the attention mechanism is applied per word, allowing the model understand the structure of natural language. Even though in principle one could use the same scheme to model the importance of different text inputs on a per-word basis, we find that this makes difficult to learn the individual importance of inputs. Moreover, due to the computational complexity of current transformers (squared with sequence length) this approach would only work for a small number of short comments. We thus add second layer of attention per processed input that allows the model to assess the amount of information at a higher level of features. We find that this mechanism aligns well with the intuition that some inputs very relevant to the problem and others be disregarded.

---

Under review as a conference paper at ICLR 2022
Look what I found! 7h
Look found! 1h
Look found! 7h C Comments stare
Look what Found! (hh
ta Comments
Stre
[ Comment
Stare
bi
conmenss Share
sho i drinking friet a dloet watte??
who it siing froo cfec wotw??
why in drink in froma dercce wite
nic Pur
aute by my wouldh do that nice hur
2Xp
Figure I: Video Retrieval from Title and comments. We show the top 4 videos retrieved for the ambiguous title "Look what found!", and from left to right we progressively add more comments which our model uses to refine the results.
In our experiments we show that we can indeed learn meaningful information from user comments that translates to good zero-shot retrieval performance common benchmarks. Additionally, can show that the model can correctly identify whether auxiliary information is informative of the content of a video or not.
The ability to incorporate auxiliary contextual information also opens up possibilities for useful applications. In the video retrieval setting our method can be used to iteratively refine text descriptor with new inputs as shown in Fig. D allowing incremental searching. In the zero-shot video classification setting (i.e., "retrieving" the correct class description prompt) the prediction for an ambiguous video can be steered towards the correct class using surrounding text from a webpage or user hints (Fig.
RELATED WORK
In this work, we focus on multi-modas learning with a particular focus on learning video-text encoders for retrieval by proposing a novel, multi-modal adaptation module.
Video-text pretraining. Originating from the NLP domain, where the transformer architectures has been a key ingredient and subject to optimization in a multitude of ways (Vaswani et al.,
2017] Devlin
al. |
Radford et al., 2019 Raffel al., 2019 [ewis al., [0206] Clark al.,
2020 Lewis
al.
2020a), it has recently found applications in the vision-language domain. For example, recent
works have leveraged transformers to learn generalizeable image (Desai & Johnson
2020; Sariyildiz
ct al. 2020; Chen et al. (2200b) or multi-modal image-text (Li al. |ll.||2119) [219:] Lu et al. 2019: Tan &
Bansal 2019; Su et al. 2019; Li et al. 2020; Chen et al. 2019) 9 and video-
vidco-multiinguur. multilingual text (Huang
al. 2021) representations. A few works (Sun et al. 201 9b a Zhu & Yang 2020: Luo al. (020)
combine visual and text modalities as inputs to a BERT model to simultaneously learn semantic video
and text representations. For representation learning, the availability of large-scale datasets such as
HowTo100M (Miech et al. (2019) has enabled more effective pretraining of video-text representations for multiple downstream tasks. More recently, Patrick al. (2020b) show that adding generative objective to contrastive pretraining can yield gains in video-text downstream tasks. Based on the CLIP model (Radford al.) (20T). which works well even without finetuning for some retrieval tasks (Luo) et al. 2021), Bain et al. (2021) train video-text CL.P-initialized models by gradually scaling up video training from image training and a custom dataset. While we also start with a CLIPP initialization as in Luo et al. (2021), the focus of our paper lies in developing novel method for leveraging user comments, modality that has previously been overlooked in the text-video retrieval litcrature, as subsequently evaluated on smaller datasets such as MSVD /Venugopalan et al. 2015) and MSR- VTT (Xu et al.) 2016a).
valuable source of information. As is standard, the pretrained representations
Multi-modal domain adaptation. While residual adapters for domain adaptation have been explored for uni-modal models such as CNNs, e.g. in Rebuffi et al. (2017), there are works that translate
2

---

Under review as a conference paper at ICLR 2022
this concept to the multi- modal domain, where cross-moda learning dominates (Awwassel al. 2020 Asano et al. 2020 Morgado et al. 2020).
Multi-modal pretraining. There is little prior work using comments additional context in multimodal settings. The use of user comments and reactions to refine predictions has been discussed by Hallevy al. (2020) for minimising harms facehook.com. Overall, are the first to demonstrate that user comments can be used as a complementary modality when learning video-text representations.
METHODS
In this section we will first recap the mechanism behind current contrastive, multi-modal representa- tion learning methods that rely clean data. We will then introduce our Context Adapter Module that allows learning from the auxiliary modality through an attention mechanism. Finally, we will describe how we can extend an existing backbone for images to vidcos, to be able to leverage large, pretrained models.
3.1 BACKGROUND
In multi modli- representation learning we are given a dataset X' N samples x, E X {l.... N that individually consists of different signals. Most previous work focuses we will-for for now also adhere to this standard to simplify the notation. This means that each input sample v; (vi, 4.) is pair of-in our case-a visual input oz E V and its associated text, often the title, 4 E TT, iiss N.
modalities and
The goal is each of the modalities as RRadford et al. (222)) learn the mapping (in their case from images and their captions) to the embedding space with double contrastive loss over a mini-batch B X using affinity matrix A computed between all pairs of samples in the batch:
learn mappings f. VV- (+(r) Φu,i and ft )i مr. from
d-dimensional, joint embedding space ) Rd. Recent methods, such
Dur
Ø
Aij =
(1)
Villoulll' Villoual1
An entry Ay measures the similarity between the embeddings (v:) and oi (t:) via cosine similarity that is scaled by temperature parameter т. The idea is now to maximize the similarity between the embeddings from the same sample, i.e. the diagonal of A and minimize all non-diagonal entries. This can be achieved efficiently using a double-contrastive formulation that operates across columns and rows of A,
((A) = -
Σ
A log Σ/ 2j=1 exp A.j log Σ exp Aji
A
(2)
This formulation has the neat effect that it accomplishes maximizing the diagonal entries and minimizing all other entries of A one self-balancing formulation. However, makes the critical assumption that both modalities are equally informative of each other. In the case of sometimes irrelevant data, or when one modality has much less information content than the other (e.g. "nice video!"), this assumption does not hold and training with this objective will result in very volatile learning objective and thus a sub-optimal joint embedding.
In the next section will introduce Context Adapter Module that is able deal with this type of inputs by allowing to discount information when it is not relevant for the context.
3.2 CONTEXT ADAPTER MODULE
In order to capture and filter the relevant information from the comments, we propose transformer- based Context Adapter Module (CAM) which operates in residual fashion, additively adapting either the visual or text branch of CLIP with contextual information obtained from the comments adding another modality- comments- the input which
(see Fig. 2. Formally, we
3

---

Under review as a conference paper at ICLR 2022
Virual du
Vidko
Backhone
Conteet Adapter Module
Commenta
Similarity
Cormmnnts
Textual
Looks skips
Testual Bucidbons
What Iot
Aduptise Module
Sindlatty
Title Chrrpe
Backbhon
Title
Toxtual Barkhuue
Ctt
(a) Adapting the text branch.
(b) Adapting the visual branch.
Figure Method Overview. We introduce context adapter module that uses inputs of the auxiliary modality to adapt the embedding of another branch. With this module the model is able to accept or discount information.
extends it to 2; (Eg, 1,, Cr,1,.... C.MM) with G. E T. To reduce clutter in the notation, have defined fixed number of comments M for each sample. Since both, title and comments, share the same underlying modality, namely text, can leverage the same encoder transform comments embeddings [e(ca) = &n.ik-
As expect the comments be sometimes unrelated, our Context Adapter Module needs mechanism to discount off-topic comments and update the primary modality (vs))) or (ett), steering in the most informative direction.
We introduce this mechanism as a function of both the primary modality and the comment embeddings Pe.ik: as we want to compare the informativeness of all these inputs at a high level. To this end, we design adapter modules Ir and that extract information from the comments in the form of residual:
2or = Dot + goldoi, 0e,1,1 Be.,,M) o E (v.
With the adapted embeddings Por and du we recompute the affinity matrix (now Á) (Eq. I and use for the loss L( (A). This design has several advantages. On one hand, extracting "only" a residual from the auxiliary inputs Cat. means that the model is easily able ignore them by predicting g(.) = 0. On the other hand, this effectively allows us to skip the adapter module when we benchmark the model on dataset that does not have comments, while still learning the joint embedding from richer data.
In practice, we implement as small transformer architecture. Rather than operating on tokenised words, this transformer operates on embeddings (dvv and Φv)) themselves, taking as input the encoded feature from the branch to be adapted, along with comment features Φe. By treating embeddings as tokens their own right, we allow the embeddings attend to each other and learn what combinations of the inputs should be used to update the original feature through the residual connection.
Additionally, to avoid bleeding information between the two modalities through the Context Adapter Module, during training, we only adapt either the video embedding with or the text embedding with gt- we would use both adapters simultaneously, there is a trivial solution that minimizes the loss L: when the adapters learn to remove the original embedding through the residual, e.g. go (0.... ( }) - Φc.i,1 both adapted embeddings become the same d.. = 6.s which trivially maximizes their similarity, thus preventing the model to learn a meaningful modality alignment. To prevent the model from learning transformation of the embedding space through the residual, we train only one adapter at time, We also consider the case where = and choose a random modality adapt for each minibatch, leading to adapter that is agnostic the source of the primary embedding.
3.3 FROM IMAGES TO VIDEOS
To leverage the capacity of large pre-trained computer vision models, we adopt the architecture by Radford et al. (2021) our backbone models fo and ft. While this transformer trained on huge volume of image-text data, it cannot be applied directly to videos since is built for images and has temporal extent. To take advantage of the temporal information present in video
4

---

Under review as a conference paper at ICLR 2022
Table Comments modality. Treating comments an auxiliary modality video improves Text-to-Vice retrieval for both RedditVC and KineticsCommente test sets. As baseline, we use the CLIP model on just the first frame. We then train just our CAM on top of CLIP, adapting the visual embedding with information from the comments, which improves retrieval metrics. Training our full model, which includes temporal attention, on RedditVC again validates that the inclusion of comments helps, with a further jump in Kinetics performance when using the Kinetics training set.
Reddit Eval.
Kinetics Eval.
Method
#Frames Train w/ Comm. Test w/ Comm.
R@5
R@ 10
R@5
R@ 10
CLIP
no
no
7.29
9.47
36.6
47.0
Ours + Our CAM (RedditVC)
yes
no
7.41
9.72
Ours + Our CAM (RedditVC)
yes
yes
13.95
18.24
42.4
53.3
Ours (RedditVC)
yes
7.73
10.28
43.4
55.6
Ours (RedditVC)
yes
yes
11.15
14.89
46.3
58.6
Ours SorddivCC*Knettccsca
yes
no
7.11
9.38
50.3
63.7
Ours (Reddit SeaddiVVCKKinetieeCmmss
yes
yes
10.93
14.58
52.9
67.4
data, we use the Divided Space- Time attention mechanism recently introduced in the TimeSformer architecture (Bertasius et al. 2021). We modify the image transformer architecture by adding patchwise self-attention across frames time each of the 12 residual attention blocks, followed in each case by a zero-initialised linear layer. We also add a learned temporal position embedding which is summed to the input and again zero-initialised. The initialisation transparent, such that when loading pretrained weights trained from images, at initialization time, the modifications do not affect the inference of the model. During training, the model can then gradually activate the additional temporal components to learn from the temporal information of video. Full details on the architecture are provided in the appendix.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We use CLIP (Radford et al. 2021) as the initialisation for the backbone. Our concrete implementation of the CAM g is a 2-layer transformer, consisting of two residual multihead self-attention blocks. The input consists of M + 1 input embeddings (for the M comments and title/video embedding doi) having 512 dimensions each. Each block performs 8-head self-attention on the inputs, followed by two linear layers with output size 2048 and 512 respectively. LayerNorm normalisation is used, along with GELU activation following the first linear layer. From the M - outputs of the transformer, we take only the output index (the index of the text/video embedding), and pass it through a final 512 x 512 linear layer. We use learning rate 0.001 for training the CAM, and 1 x 10-6 when training the entire model, using the Adam Kingma & Ba (0155) optimizer.
All implementation and architecture details can be found in the appendix.
4.2 PRE-TRAINING DATASETS
RedditVC. We collect dataset "RedditVC" of videos along with their titles and comment threads from social news site reddit.com, using their provided API. The videos are collected and used in manner compatible with national regulations on usage of data for research. Unlike most curated video datasets, this data is more representative of the types of videos shared "in the wild", containing large proportion of videogames, screenshots and memes.
Using a classifier trained on a small amount labelled data, we estimate that videogame footage makes up 25% of examples, other screenshots, memes and comics make up 24%, live action footage is 49% and artistic styled content (such as drawn animation) is 2%. The average video length is 33s.
From 1 million raw videos collected, we perform deduplication and filtering, ending up with training set of 461k videos, and validation and test sets of 66k videos each. For the video evaluation results in Table 1 use subset of the test set, consisting of 5000 videos with least three comments each.
5

---

Under review as a conference paper at ICLR 2022
Table 2: Ablation Study. Comparing Text-to-Video and Vidco-to-Text retrieval results between
different baselines and our method.
Finetune Backhone
Train.
Eval. CAM
Text + R@1
Video R@ 10
Video + Text R@1
CAM
R@ 10
(a)
X
1.70
7.30
1.57
6.86
(b)
2.25
9.85
2.22
9.28
(c)
rand Øook) rand(de: Oock)
2.06
9.40
2.04
8.83
(d)
avg(de, Φot)
1.64
7.99
1.96
9.49
(c)
avg(de, dck) (vg Pck)
1.67
7.79
1.97
9.01
(n)
avg(de, (ot)
2.49
11.75
2.63
12.06
(g)
2.28
9.70
2.14
9.22
(h)
X
gr
1.98
9.93
1.70
8.87
()
g+
gr
2.06
10.49
1.96
9.98
①
go
2.22
9.67
2.11
9.11
88
X
9v
go
2.71
12.22
2.86
12.44
9v
Jo
2.86
13.07
2.93
13.07
In addition comments, for one experiment in Table we also obtain textual labels of objects in the video thumbnails, using the Google Vision API. We obtain labels for 10,212 different classes, from "Abacus" "Zwiebelkuchen". This serves useful comparison, giving textual metadata that can be expected to be more directly related to the visual video contents than user comments, and can be seen as a proxy for user-generated "hashtags". also illustrates how, by using text as an intermediary, we can integrate predictions from any black-box classifier in the same framework.
CeneticsComments As an additional video dataset with comments, we construct a dataset based on Kinetics-700 (Kay al. 2017 Carreira et al. 2019), for which we download the videos along with associated YouTube metadata including title, description and comments. We translate non-English titles and descriptions into English using a commercial translation API. We use the title as the primary text modality, and for auxiliary context use comments and (when training) sentences from the description. This leaves us with 484,914 videos for training, each being around 10s. We also construct test set, consisting of videos from the Kinetics test set for which have at least 3 comments, giving a set 2700 vidcos which we use to evaluate our method in Table
4.3 EVALUATING THE CONTEXT ADAPTER MODULE
We evaluate our Context Adapter Module on the above described datasets with comments in Ta- [ We find that on both datasets adding comments boosts the retrieval performance significantly, confirming the value of the modality. Further, moving from images to videos also improves the performance. Since KinetiesComments (human actions) is quite different RedditVC (broad range of videos, games, etc.) combining both datasets bridges the domain gap and improves the evaluation performance on KineticsComments.
Baseline Comparisons. Since the main purpose of the Context Adapter Module to extract meaningful information from sometimes unrelated auxiliary data, we perform most of the experiments this section with an image/text backbone using a single video frame instead of a video/text backbone to reduce the computational burden of the evaluation.
In Table we evaluate the efficacy of the proposed Context Adapter Module against various baselines in various settings form of context adaption, which degenerates the model to the one proposed by Radford et al. 2021
RedditVC dataset. The most basic baseline is achieved by not using any
Across all experimental settings, we find that finetuning the backbone architecture helps to improve the performance, while using the proposed CAM with frozen backbones (rows h,k) even outperforms the naive baseline of (a). We also compare to two baselines. The first baseline is randomly replacing the title features with comments during training (Table cc-b)....))...........))))>>)) however, this does not yield any improvement adaptation. In the second baseline average the title features oi and the

---

Under review as a conference paper at ICLR 2022
Table 3: Adapting different heads. Text-to-video @@ 10 re- sults on RedditVC-5. We either adapt the text or image features.
Table 4: Influence of Auxiliary Data. Text-to-video retrieval
results on RedditVC-5. We vary the auxiliary text source during training and testing. Using images labels from a classifier during training works well but does not generalize.
train CAM
test @v
test tz
R@ 10
Aux. Train Data
Im. Labels
Comments
Rand. Words
gv
11.48
2.98
0.50
9.48
Image labels Comments
22.20
6.91
3.06
go &
10.08
9.16
10.45
9.48
6.67
comment features de which can be seen as a very simple adapter module (Table(Zje-)), This yields a small benefit when both training and evaluation use averaging, but falls behind utilizing proposed CAM on the visual modality (Table ()). Surprisingly, we find that training with comments performs best when they are added to the image branch (rows j-l), opposed to the text-branch (rows g-i). We hypothesize that this because the comments can better adapt the semantically richer feature of the visual modality opposed to the text, which often be as short single word.
Adapting Different Modalities. Since both the visual embedding 6, and the text embedding dr. come from the same embedding space and should be similar for the same sample, we perform an experiment where we swap the embeddings during evaluation time. In Table B we find that there are still differences between & and d and swapping results in decreased performance. When we train both adapters we can achieve good performance in both cases.
Table 5: MSVD (Chen & Dolan 2011) Zero-shot texl-to-video retrieval. CLIP an important base- line since use it as backbone initialisation. Even though comments are not available for this benchmark the model trained with comments has learned better representations.
185
Text-to-vile to-vidoooeooooooooooooo R@10
1550
35
Train with 1 comms
Train with comms
Train with 5 comms
Method
R@1 @5 @ @10 Rk
Train with 7 comms
CLIP-Straigh Dattill-Qeinerro et all 2021
37.0 64.1 73.8
3
Train with 9 comms
CLIP44lip Luo et al. 2021
38.5 66.9 76.8
2
#comments at eval
Ours (no comments)
39.5 68.4 78.3
2
Ours (no TimeSformer)
37.7 66.0 76.5
3
Figure 3: Varying Number of Comments. The model can effectively benefit from additional aux- iliary information and models trained with more comments become better at extracting informa- tion even when fewer are available during testing.
Ours
39.5 68.6 78.4
2
Ours (with Kinetics)
40.9 69.5 78.7
2
Auxiliary Information. Additionally. Table shows how varying the source of the text train and test time impacts the retrieval results. Whilst the performance of the model trained with image labels degrades when given comments at evaluation time, model trained with comments still benefits from image labels at test time. As expected, the performance both models is negatively impacted when tested with uncorrelated random words additional text. However, the model trained with image labels does noticeably worse in this setting. This suggests that has become overly reliant on image labels and easily fooled by irrelevant information, which overrides the title feature.
Varying the number of comments. In Fig. we vary the number of comments during training and evaluation time. Training on more comments results in better retrieval results when evaluated on the same number of comments. However, while there is little difference for models trained with only a few comments, we found that increasing the number of comments at train time leads to models that are more robust to different number of comments at test time, indicating that training with more

---

Under review as a conference paper at ICLR 2022
comments learns overall better context adapter module that is then able to work efficiently in various settings.
4.4 COMPARISON THE STATE OF THE ART
In order to compare to other methods, use the text-to-video retrieval benchmarks MSR-VTT Xu et al. 2016a) and MSVD (Venugopalan et al., 2015). While these benchmarks do not directly take into account the novel ability of method to consider contextual information, they give measure of the transferability of the learned text and video features. For evaluation, we skip the CAM and dui and Φы directly since these benchmarks do not involve comments. Fine-tuning a learned representation only yields an indirect measure of the original model's 's quality. Since we are interested in the immediate quality of the learned representations, focus this evaluation on zero-shot performance on the datasets, without fine-tuning, Nonetheless, fine-tuning results can be found in the appendix.
Datasets. For MSVD (Table 5). we use the test split consisting of 670 videos and approximately 40 captions each.
For MSR-VTI (Table we use the 1k-A or "jsfusion" evaluation split of the data, consisting of 1000 (video,texi) pairs and 20 captions per video.
Moreover, we show the zero-shot performance of our method on 2 additional datasets. Table 7 we report the Text-to-Vide retrieval results on the ActivityNet (Fabian Caba Heilbron & Niebles 2015 dataset. We evaluate on the "val 1" split, as in SSB (Patrick et (000bb) who we compare to. In Table 8 show our Text-to- Video retrieval performance the LSMDC Rohrbach et al. (2017) dataset, where we use the test set comprising of 1000 videos.
Finally, report the Video-to- Text retrieval results for MSVD and MSR-VTT, which follow similar patterns, in the Appendix.
Results. Across all benchmarks we improve over all other methods in zero-shot retrieval perfor- mance. On MSR-VTT our performance even slightly better than CLIP4Clip (Luo et (021). that uses the same backbone initialization as us but additionally trains on How To100M which is more than 100 times larger than our dataset.
is interesting to note, that we are the only method that evaluates zero-shot performance on such wide spectrum of datasets which shows good generalization performance of the learned representations.
Using comments, videos and more data. In Tables 6 9 and 10 we also show the performance of our model different variants.
Across all three benchmarks, our model trained with comments is consistently better than the variant trained without comments. This means that even though the evaluation performed without comments (none of these datasets provides user comments), the additional context during training has improved the underlying video-text representations.
Similarly, using video instead of single image improves the results as well. The main difference here, however, is that the video also available during evaluation. Thus, the observation is that video provides more or better context for retrieval than single images.
Finally, we find that adding Kinetics to the training set has only small effect on the quality of the leamed representation and can depending the domain gap between Kinetics and the benchmark dataset- even affect the results slightly negatively.
Thus, while achieving fftte-of-the-ee performance, signal to improve multi-modal representation learning, even when comments are not available during test time.
also show that comments provide a strong
8

---

Under review as a conference paper at ICLR 2022
Table Text-to-video retrieval on MSR-VTT. Vis. Enc. Init.: Datasets used for pretraining visual encoders for tasks other than visual-text retrieval, eg object classification. pre-training: Visual-text pretraining data. Object, Motion, Face, Scene, Speech, OCR and Sound classification features. #samp. PT: size of the pre-training dataset in millions of samples.
Method
Vis Enc. Init.
pre-training
#samp. PT R @!
R@S
R@
10 Rk
HT MIL.-NCE et
2019
How 100
100M
21.2
29.6
SupportSct Paniik et 1020h
IG65M, ImageNet How T0M
100M
8.7
23.0
31.1
31
FiT Bain
2021
ImageNet
CC3M, WehVid-2M
5.5M
18.7
39.5
51.6
10
CLIP4CIp
CLIP. HowTo100M
500M
32.0
57.0
66.9
4
VideoCLIP Xa 2021
1o00MM00M
100M
10.4
22.2
30.0
Ours (no comments)
CLIP, RedditVC
401M
31.5
54.3
64.5
Ours (no TimeSformer)
CLIP, RedditVC
401M
31.9
53.5
65.3
Ours
CLIP, RedditVC
401M
32.2
54.9
65.4
4
Ours (with K700)
CLIP, RedditVC, K700
402M 30.8
56.1
67.1
4
Table 8: LSMDC (Rohrbach et al. 2017 Text-to- Video retrieval.
Table 7: ActivityNet (Fabian Caba Heilbron &
Niebles 2015) Text-to- Text-to-Video retrieval.
Method
R
R@5
@ 10
MedR
Method
R@I R@5 R@ MedR
JSFusion
Yu 2018
9.1
21.2
34,1
36.0
CE
2019
11.2
26.9
34.8
25.3
SSB
20206
0.0
0.2
0.3
2238
MMT
12.9
29.2
38.8
19.3
Ours
22.7
33.0
26
RT Bai
2021
15.0
30.8
39.8
20.0
Ours
24.0
48.0
60.1
4.5
LIMITATIONS
dog
cookio
We find that the context adapter be led to override the information in a title if we adversar- ially craft comments that all point different content. Qualitative examples of this can be seen in Fig. 4 The model without comments, cor- rectly associates the image with a cookie (jar), however when adding a comment about "dog" the model prefers the dog label over the cookie label. More examples can be found in ap- pendix (Appendix A.5 and the results browser in the supplementary material).
NIA
0,37
0
0.63
"daggie"
"yummy", "looks delicious"
0.76
0,017
cool"
CONCLUSION
Figure 4: Failure Case. A heatmap showing the similarities between the image adapted with dif- ferent comments (rows), and captions (columns). The adapter can steer away the embedding from the right association "cookie jar" depending on the comment input.
We have presented the Context Adapter Mod- ule, which is able to extract information from auxiliary input sources for learning a joint, multi- modal embedding. In experiments, are able to show that the model improves the rep- resentation quality of video-text embedding learning when adapting the representation with user comments. Moreover, it is able to identify whether an auxiliary input is relevant to the content in the other modalities or not. This mechanism could, for example, be used filter datasets for meaningful auxiliary content. The model can be initialized from existing, large-scale image-text backbones and the context adapter learns from comparatively small number of examples (<500k). The module itself is modality-agnosti and can potentially be used in large variety of multi-mon multi-modality representation learning tasks such as audio-visual learning, or to combine the outputs of unreliable expert models.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY AND BROADER IMPACT
Online data collected from user genereeeeraed content are subject to all sorts of biases and potentially problematic content. We have carefully selected the discussion topics from which we source the data, to include topics with a lower chance of problematic topics such pets, art, jokes and games. On top of that we filter text content with Detoxify (Hanu & Unitary team 2020) for harmful content. The use of content filtering text model such as Detoxify can introduce its own biases. Although we have used the 'unbiased' version of Detoxify to minimise these, there might still be particular words that associated with offensive or hateful language that will likely be strongly correlated with high toxicity scores, regardless of the intent of the author. This could lead to overfiltering non-toxic text that contains particular identity terms, which might result in discrimination against already vulnerable groups. Moreover, the user-base of Reddit is predominantly young, male and western (Barthel et al. 2016) and thus inherently there will be certain biases present regardless of how thoroughly the dataset is filtered. The use of a primarily English language data source and pretrained model means learned concepts will be far from globally representative. The social biases discussed in Section 7.1 of Radford et al. (202T) all still apply. Additionally, as the data is downloaded from public website, content creators and users could not be asked for consent to be included in the dataset.
The presented context adapter mechanism has wide applicability across different multi-modal tasks where one input source expected to sometimes uninformative. However, as detailed in the previous section, when trained online data, the model has mechanism distinguish between true and false statements and will simply learn from correlations. It has been repeatedly shown that algorithms trained on online user content are often unfit for any kind of production system and pose risks (Bender et al.) 2021). Our work and analysis is for research purposes only and we strongly advise against using models trained on the RedditVC dataset in non-research applications.
Given the above discussion on impact and biases, we refrain for now from publicly releasing the RedditVC dataset or models trained it, as their intent is for research proposes only and not fit for real-world applications. However, we are planning to release the comments collected for the KinetiesComment dataset as the content of the videos much more constrained, which results in easier filtering of the data.
For further reproducibility of the general approach will release the code. Additionally, models trained on datasets other than RedditVC will be released as well.
REFERENCES
Humam Alwassel, Bruno Korbar, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised lcarning by gross-moda audio-vides clustering. In NeurlPS, 2020.
Yuki M. Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi modal self-supervision. In NeurlPS, 2020.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. arXiv preprint arXiv:2104.006500 2021.
Michael Barthel, Galen Stocking, Jess Holcomb, and Amy Mitchell. Reddit news users more likely to be male, young and digital in their news preferences. Pew Research Center, 2016.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Conference on Fairness, Accountability, and Transparency (FAceT '21). ACM, New York, NY, USA, 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understand- ing? arXiv preprint arXiv:2102.05009, 2021.
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 700 human action dataset. arXiv preprint arXiv:1907.06995 06987, 2019.
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 190-200, 2011
10

---

Under review as a conference paper at ICLR 2022
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual datasct. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020a.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, and Heewoo Jun. Generative pretraining from pixels. In ICML, 2020b.
Yen-Chun Chen, Linjie Li. Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740 2019.
Kevin Clark, Minh-Thang Luong, Quoe V. Le, and Christopher D. Manning. ELECTRA: Pre-training text
encoders as discriminators rather than generators. In ICLR, 2020.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. arXiv preprint arXiv:2006. 06666, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In ACI, 2019.
Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961-970, 2015.
Alon Halevy, Cristian Canton Ferrer, Hao Ma, Umut Ozertem, Patrick Pantel, Marzich Sacidi, Fabrizio Silvestri, and Ves Stoyanov. Preserving integrity in online social networks. arXiv preprint arXiv:2009.10311 2020.
Laura Hanu and Unitary team. Detoxify. Githuh. http:///jhhh...cmmmmiiaaaaaalatf 2020.
Po-Yao Yao Huang, Mandela Patrick, Junjie Hu, Graham Neubig, Florian Meize, and Alexander Hauptmann. Multilingual multimodal pre-traine pre-training for zero-shot cross longullngual transfer of lasgun-language models. In Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL), June 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. arXiv preprint arXiv:170.00877 2017.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: fttt::/705.09550 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. Pre-training via paraphrasing. arXiv preprint arXiv:2006.15002 2020a.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Ahdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-scquence pre-training for natural language generation, translation, and comprehension. In ACL, 2020b.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. Unicoder-vi: universal encoder for vision and language by cross- modal pre-training. In AAAI, 2020.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint 190v:1998.03357, 2019.
Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. arXiv preprint arXiv:1907. 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task agnosage visiolinguistic represen- tations for vision-an and languaee tasks. Neurlps, 2019.
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. UniVI:: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002. 06353, 2020.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2 088600 2021.
Shuming Ma, Lei Cui, Damai Dai, Furu Wei, and Xu Sun. Livehot: Generating live video comments based on visual and textual contexts. In AAA/ 2019, 2019.
11

---

Under review as a conference paper at ICLR 2022
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivie. Howtol00m: Learning a text-video embedding by watching hundred million narrated video clips. ICCV, 2019.
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra, Audio-visual instance discrimination with cross-modal agreement. arXiv preprint arXiv:2004.12943, 2020.
Mandela Patrick, Yuki M Asano, Polina Kuznetsova, Ruth Fong. Joao F Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transformations. arXiv preprint a:Xiv2203.004298, 2020a.
Mandela Patrick, Po- Yao Huang, Yuki Asano, Florian Metze, Alexander Hauptmann, João Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. arXiv preprint arXiv:2010.02824 2020b.
Mandela Patrick. Yuki M. Asano, Polina Kuznetsova, Ruth Fong, João Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transformations, 2021.
Jesús Andrés Portillo-Quitten José Carlos Ortiz- Bayliss, and Hugo Terashima-Marin A straightforward framework for video retrieval using clip. arXiv preprint arXiv:2 12443, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Iyya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog. 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2 0000000 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.1 10683, 2019.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In NeurlPS, 2017.
Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Chris Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. International Journal of Computer Vision, 2017.
Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In ECCV, pp. 153-170. Springer, 2020.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BBETT Pre-training of generic visual-linguistic representations. arXiv preprint arXiv: 0000/90888880 085300 2019.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using contrastive bidirectional transformer. arXiv preprint arXiv:1906.00 05743, 2019a.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. /CCV, 2019b.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakoh Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, and Kate Sacnko. Sequence to sequence - video to text. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
Hu Xu, Gargi Ghosh, Pa-Yao Huang, Dmytro Okhonko, Armen Aghajanyan. and Florian Metze Luke Zettle- moyer Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot vidco-text understanding. arXiv preprint arXiv:2109. 14084, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MTT: A large video description dataset for bridging video and language. In CVPR, 2016a.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296. 2016b.
12

---

Under review as a conference paper at ICLR 2022
Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In ECCV, 2018.
Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In CVPR, 2020.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 QUALITATIVE EXAMPLES
In Fig. we adapt the text branch, similar to Fig. of the main paper. The example in the second row of Fig. shows how our Context Adapter Module can leverage the comments to learn that the content is indeed about parrots, as opposed to dogs. The fourth row shows that without comments, the title alone can be extremely ambiguous while comments can again guide the model to retrieve relevant videos of drumming.
We provide examples where the video branch has been adapted in Fig. In most cases, the retrieved titles are broadly related to the video thumbnails. However, when provided with the comments, the retrieved titles become more specific to the videos. For example, in the example from the second row of a screenshot from Mario Kart, the retrieved titles are generally about games e.g. The Castle or Sun Haven, whereas when adapting the video with the comments, the model retrieves titles specifically about Mario Kart. Similarly, in the example from the last row, the model seems to get confused about the content of the video when deprived of the comments, which provide the necessary context about feeding fish.
Finally, in Fig. 7 we show the saliency of comments with regards to a given video and title. For this, we use the approach of masking out each comment in turn, allowing us to visualise the effect of each individual comment on the network output. We compare the output descriptor when including all comments to the descriptors with a comment masked, using the inner product as score of similarity, and present the comments sorted from lowest to high, the expectation being that an uninformative comment will not cause large shift in the descriptor (so will still have high similarity when excluded) whereas a salient comment will cause a larger shift (and so lower similarity when excluded). We show results for adapting both the text branch (left) and visual branch (right). and observe that, as expected, uninformative comments such as "That was great!" and "Possibly?!?!??? Lol" cause little change the descriptor, whereas comments related to objects in the video cause larger shift. This demonstrates that the method is able to pick out and filter the relevant information.
A.2 TRAINING DETAILS
The majority of experiments were conducted on rented 4xA 400000 GPU server costing ap- proximately 170USD per day, over the course of three months. Image models (using batch size 128) and video models without comments (using batch size 50) could train on a single 40GB GPU. For TimeSformer models the visual branch was processed on a separate GPU (when training with CAM and batch size 50) or pair of GPUs (for finetuning video benchmarks with batch size 128). Pretraining the adapter on images takes approximately one hour per epoch. Training the full video model with CAM takes approximately 6 hours per epoch. For the video experiments, we first train the CAM for epochs with the backbone frozen, and then train the rest of the network for one epoch, with the backbone modified to have temporal attention as described in Section 3.3 with 5 comments, and adapt the visual branch of the model.
We use the CAM
Table 9: MSVD Chen & Dolan 2011 Video-to- Table 10: MSRVTT Xu et al. Text retrieval.
2016a) Video-to-
Text retrieval.
Method
R@1 1 R@5 R@ 10 MedR Method
R@1 1
R@S R 10 MedR
SSB Patrick et
2000
21.4
46.2
57.7
6.0
SSB Patrick et al. 2000
12.7
27.5
36.2
24.0
CLIP (Purtilo Quintere al.
2021
59.9
85.2
90.7
CLIP (Porailo-Quinter et al, 2021
27.2
51.7
62.6
Ours (no comments)
62.5
87.0
92.8
Ours (no comments)
33.0
57.9
66.9
Ours (no TimeStormer)
59.1
85.1
91.0
Ours (no TimeSformer)
31.8
56.9
66.7
Ours
61.6
87.9
93.4
Ours
33.2
58.5
68.1
3
Ours (with Kinetics)
61.3
87.0
92.4
Ours (with Kinetics)
33.0
58.3
68.2
We use both photometric and temporal data augmentation. For photometric augmentation we employ random crops (0.5 1.0 scale), random horizontal flipping, and colour jitter (brightness, contrast, saturation, huc). For temporal augmentation, we first temporally subsample the input frames (which often 30fps) according to random stride selected uniformly from (4, 8, 16, 32) and then choose
14

---

Under review as a conference paper at ICLR 2022
drag stop! Any very appreciated!!
with balance
that
my
datee thoure lie my erestent
ligg for coouaance and thi sands of years
lemo, bm making all bat keop in mand
Look dad, my first
Commmnee
They snépward that eary He. watchin ri
Than gicel
Livkk the persen x veeeeer and know what
onsh sarret inter
to
esarth, they don't have huan without
neaded.
cenee heat and tody osspen contrer
My tribute Original Trilogy just 6,000 Preque's next they will eventually all join video in comments
Comments
Trekedfy Unterarntah, that's
Very athoug? sudget
Thanka the comment
This Iiss neade be turther up The thart
smoning dreat 30b.
dude
Experimenting splitting
Cerment
Super focused tome on those duns
Tranket
I think my buddy swings a little too hard. Any advice for how to calm him down?
[X] oCmmmnnts
Moobe pure dothes and flat cap at
Andi drodaaaton bull sl lndo litll too and
Have him that
Figure 5: Examples of retrieved video thumbnails when adapting the text branch.
a random 8-frame segment uniformly. We normalise inputs using the same preprocessing as Clip (ImageNet mean and standard deviation, 224 x 224 input size).
At evaluation time we use a temporal stride of 16 and split the video into 8-frame chunks, taking the average of the descriptors of the chunks.
15

---

Under review as a conference paper at ICLR 2022
Comments
Video thumbnail
Retrieved titles w/o adapting
Retrieved titles with adapting
stggle snggle
Inday out adles up recre loniin then caee (verry
viste
Stund ent Lane the to part on tnd the marninge
kineading and face smookiing
InOU CANT ESCAPS THE CUDDLES
beal pay
anty Mordecst wan tucking te in when dia't host und
Nile, aleo deay
Jui mat belenc
roah, ne aus erseyit eruch in (2p0 eneveer
likes
mesh
neet, out he curprics dotintory hel easth my doul.
inloo witl
troveling
Ho seers rellos We
Soundd ant lusa purt in the mocminng
labay Hardeca
when clavt
Comments
Video thumbnail
Retrieved titles w/o adapting
Retrieved titles with adapting
yoark
wny
ceased inarteing
Made another notess spe hall [ec]
ercatr do light reflecingg Porert and of
anyaa
aver
well eventudly
add or the Veder caaway. wall
Fall dil conmrnnee
cansuiting project pes Siccy"
wertad tea koft dissparnted
TDCS" areaar canaan eipeda TRO2. Paal rise diotes
warted
mone accter locia and oee
live to
more canann expecta (Ri2)
piral aill 5,200
Frover Deez Vader
Full des commenta
den Wan Barth thoept Certh Jest Rige willke intrisaaaa
Comments
Video thumbnail
Retrieved titles w/o adapting
Retrieved titles with adapting
Full bour my Auly de desocsed mecinaal islind
(Meaaia so Scivernn the stoue sholl can realy help
Iecond
started poing Drowe This 2nd soar
(reaicag is my capy mavi darlcctvv, what?
leec intuutted of tome ven whee nos
pres, setwens the pero and tne
ausk Maris kart doubls dain but sputiaa
shit
(recos) saue sawks jns doi't
wunder wheee Morvetous Morchent
(reaaca thua the
Hario kann mamant
Theme sheil
Duen the of The Caudle me with the D.
b. after ih marie 201
Hoven- The Wainig
Comments
Video thumbnail
Retrieved titles w/o adapting
Retrieved titles with adapting
Apprec viea roll
kiday then months fire
working called
Bounda bec, keep going Huske
Sae oo work on old projet.. andec as
Your'
San to spri an projects 12 sendidd up as 7Rittti Sliving
Culf Waat
your
roddy Thonks
The it firs toin pucinn whar pile on partw de
sadieer and sannow an rusedaai (us ineens) aw thend
woaeh yroufl yot binds srng junt
play
quar kedit of myy dinging. My roke
aoet ou my sinding My tako Ranovie klaee
reconting down owly wnll
Sonteria suniie
Sweeet man thonks
folar, fresabhy wrong and in ciftirers kay but
pritty raw to sargerninng Rendzack wake mit
Comments
Video thumbnail
Retrieved titles w/o adapting
Retrieved titles with adapting
Awn
part frund cheeamet eg.
Thrr pook chth scropf co neno placo polits
red,
ani thank
The the Weit dinci
looked
autaslying out sstaiing at maxt?
The
Fllned the perocur trotaties tareeedd
glasa mato ohee lo 14.2 Brdyy bedi but
uctuclly oatch coull
replec
Eass cnpare erew why ldd showirg thoouhh my
we agin kycks cettng and veree resvinr in 5o Ea out
den't
ou cash
downlatt ment
Figure 6: Examples of retrieved titles when adapting the visual branch.
We randomly mask out comments with probability 0.5. We randomly skip adding the residual from the adapter with probability 0.5, which ensures that unadapted descriptors are also used in the loss and so the backbone network can still be used without the adapter.
16

---

Under review as a conference paper at ICLR 2022
Title:
She does this every time we feed her.
Lo when caM the Bow kod like course
Wiich broed?
+ight me the food muster
exctted
I'm fre got broed
sure she can
reaction
what dog? all see is kongaros
FoOD DANCE
That arodigg!
That- aroziig!!
Pahaha
like sppath! I'm uncilid
Lmno, She's OEPMITELY dog plasse out slovee iilvv me the food master
Wh breed?
in my head can heur the itlle Mario spri ng every time she
my head can hear the little Moy spring every time she gaing
*sproing foins going
Dhrkks the inspo!
inspot
Title: Here's Giorno's theme from JJBA.
Bre, plano great
Fie. piano has grouk fome Ico
HOW BIG ARE UK HANDS? Thise octaves soom like rothing
aro HOW BIG ARE LA HANKST Those octaver seem lire nothing atunggement
Sooutiuully played
Dude law key thard actually b stand Would
damni trying learn lost mat
Shoots?
SICC DTD
damtn was trying leam it but nox lost all motivation
the fond orangement you car find cover on
Beautifull played
Yiu plecas &
Hove itt Experity played, poople will the tam yes glaase!!
Artully gental
That was great
Actusily genial
Same yes please!!
That was great! please
Hove Expertly playss A lot people will get the tem
Title: Car battery problem? No Problem!
diettic man. Zap
Car bamtedee ane vut direct current Noth happent
Car batteree are direct currer Weetiin happene we culture
Elestrsboam already deturiess video ago Se can someaine tel me ihhils healthy, safe, or cainfull Augare's
ssee
Bectrobosm ready debun koed this sideo a long ago
seee yours man
This mode me laugh
This made me tough
Plaase eniighton us. What artic would Nlw this work
You know volts primary concerme right..
There abockusety repercasaions
This mases fll hurt
primary
This makes my filings hurt
There are abso unely repertundors trylng. Laf unserstand principle butling how this worlo..
pcsabbe.
Lol unndu Soadd the priceple behind how this works... but
Figure 7: Visualising comment saliency. We show the title and thumbnail for three vidcos, and show the ranked saliency of comments when adapting using the Text branch (left) and Image branch (right). Comments mentioning topics relevant the title or image are ranked highly, while irrelevant comments are lower.
All retrieval experiments are GPU accelerated using the FAISS library.
A.3 VIDEO-TO-TEXT RETRIEVAL
We report our zero-shot Video-to-Text Retrieval evaluation results on MSVD and MSR-VTV in Tad Table IQ
A.4 KINETICS COMMENTS
In this section we will describe the details for the additional comments we retrieve for the Kinectics- 700 dataset. In Table [ and Fig. g we show the distribution of the number of comments in the dataset. We collect a maximum of 10 comments and exclude videos without comments, which leaves
17

---

Under review as a conference paper at ICLR 2022
Table 11: Comments per video statistics for the KineticsComments dataset.
#comments
2
3
6
7
9
10
#videos
50322
21847
11946
7960
5596
4311
3220
2671
2245
1852
us with 111 920 videos of the originally 650 000 video clips. The majority of videos has one two comments available. Nonetheless, our experiments show (Table that our method can make use of this information to learn better representations.
50000
# Videos
30000
20000
10000
2
4
5
7
9
10
# Comments
Figure We show a histogram comment statistics on KineticsComments.
A.5 ADDITIONAL FAILURE CASES
In Fig. 9 we show additional failure cases. We find that vary vague comments "Why" generic expressions "Ain't his fault" can distract the model from the title. In the last example, the model does not capture the concept of sad dog due the mention of "happy" in the comments.
A.6 DATASET DETAILS
We report additional details on the benchmark datasets used in the paper.
ActivityNet ActivityNet Fabian Caba Heilbron & Niebles (2015) a video dataset with 200 classes, 100 videos per class and 1.5 activity instances per video. The dataset contains 648h of video. There are 10,024 training videos (15,410 instances) and 4,926 validation videos (7,654 instances). The dataset itself has license and is provided as annotated list of youtube links. The copyright of the individual videos remains with their creators.
MSR-VTT MSR-VTT Xu et al. (2016b) a video dataset with 10 categories, and 10,00 clips from 7180 videos. For training we use the 6.5k video training set with 20 captions each. It is annotated with 200,000 sentences video descriptions created by crowd workers. The dataset itself has no license and the copyright of the individual videos remains with their creators.
MSVD MSVD Chen Dolan (201T) consists 1200/20/670 videos for training/validaaiionnssiinn The dataset itself has no license and the copyright of the individual videos remains with their creators.
LSMDC Large Scale Movie Description Challenge (LSMDC) Rohrbach et al. (017)) consists of 118K video clips from 202 movies. We evaluate test of 1,000 videos from movies disjoint
18

---

Under review as a conference paper at ICLR 2022
Photographer had photo shoot with adorable client
Photographer had photo shoot with adorable client
Commeetts
Thread
Thread
When revoking downvate is even mor...
When revoking downvote is even mor...
Boomed
Boomed
identity
gives iointity
Why?
Why?
Dragon
Dragon
Use Comments
Use Commentss
his fault, It's sped up damn
Ain't fauit. It's sped up
guy_caim
Holy shit, guy _calm down.
down..
My little sad bai
My little sad boi Cormnenio
Use Comments
happy,
happy,
Maybe It's the coller
Maybe It's the collr
Figure 9: Examples of failure cases where using comments confounds the model and leads more mismatched retrieved thumbnail.
from the train and val sets, as done in Bain et al. 2021 The dataset itself has no license and the copyright of the individual videos remains with their creators and the dataset is solely to be used for research purposes.
A.7 DATASET CURATION
Table 12: Prevalence of toxic text in the dataset. We report the proportion of posts, titles, and comments that are flagged as hav- ing potentially offensive content by the open- source library Detoxify. We use a threshold of 0.9.
Table Bootstrapping confidence interval estimates. Showing mean and 95% confidence intervals obtained by bootstrapping with 80% of the test sets. Recall in units of percentage points. Using models trained on RedditVC and finetuned the respective benchmark training sets.
Detoxify label
% titles % comments
toxicity
2.32
5.62
Method
R@S
MedR
severe toxicity
0.00
0.00
MSR-VTI 42.6+1.5 70.0+0.9 81.6+0.7 2.0=0.0
obscenity
1.23
3.73
MSVD 48.5+1.1 79.2+0.7 87.540.6 2.0=0.0
identity attack
0.00
0.00
insult
0.82
1,95
threat
0.05
0.07
sexually explicit
0.09
0.22
19

---

Under review as a conference paper at ICLR 2022
Table 14: Mean recall performance and 95th percentile confidence intervals across five experiments trained with different random seeds for the best two models from Table in the main paper.
Finetune Backbone
Train. CAM
Eval. CAM
Text - Video
Video - Text
R@1
R@ 10
R@1
R@10
9r
9r
2.65:10.09
12.03.1..14
2.79110.16 2.9110.12
12.33.10.10 12.9411.13
9e
ge
2.9510.110
13.0810.12
Table 15: We experiment with an additional modality: Audio. Our model can learn from audio as well as comments and it is even possible to combine both modalities in the same adapter module.
Aux. modality
TVR R@1
TVR R@ 10
VTR R@1
VTR R@ 10
2.25
9.85
2.22
audio
2.08
10.02
1.97
9.60
comments
2.24
10.71
2.01
10.24
comments audio
2.62
12.75
2.30
12.23
We use the GPU implementation of the FAISS similarity search toolkit (Johnson et al., 2017) to efficiently deduplicate the dataset by indexing the video thumbnail embeddings obtained from ResNet18. These indices are then used to discard video entries with high similarity to other posts.
A.8 CONFIDENCE INTERVALS
We report the mean retrieval results in Table 14 and the confidence intervals of the best two image models from the ablation study (Table 1, main paper), namely the models adapting the visual branch, with and without finetuning the backhone. These are computed across five independent training runs with different random seeds. Since training the video model takes considerable amount of time, we estimate the variance of our method through bootstrapping the validation dataset. We generate validation datasets with 80% of the full size by sampling with replacement and use the quantitative results to estimate the standard deviation, reported in Table 13.
B ADDITIONAL EXPERIMENTS AND RESULTS
During the discussion period we have performed additional experiments based on the reviewers' suggestions.
Additional Modality: Audio As context adapter module is general in terms of modality, have performed an additional experiment by using the audio signal that comes with the video. In this experiment (see Table 15p we use the audio encoder of GDT Patrick et al. (021) encode five two second audio clips and add an MLP to map the audio embedding to the CLIP feature space.
We find that using comments alone is a stronger learning signal than audio alone. Exploiting both, comments and audio, improves the performance over using each auxiliary modality alone. Using audio alone only slightly improves R10 numbers upon using any auxiliary modality. We hypothesise this is because the embedding learned by GDT's audio encoder network is considerably different the embedding learned by our pre-trained CLIP model. This insight is supported by the fact that fine-tuning the CLIP network together with the audio MLP is essential to learn anything in this setting.
Ablation Image Model: Frame Time For our image model, that operates on single frames only, ablate the choice of frame within the video in [G
We find that the choice of frame does indeed matter, with the middle frame in a video performing better than the first frame. This result is relatively intuitive since the actual content of a video usually happens in the middle the video which is similar to the photographer bias (pictures tend to display an object in the middle of the frame).
20

---

Under review as a conference paper at ICLR 2022
Table 16: For the experiments using images instead of videos, ablate the position of the frame in the video. Extracting the frame from the middle of the video is better, likely due to the fact that the important action video usually occurs the middle or end of the clip.
Model
Frame
TVR R@I
TVR R@10
VTR R@1
VTR R@ 10
adapting text
first
1.36
7.20
1.45
7.67
adapting text
middle
1.50
7.46
1.50
7.97
Table 17: We test the robustness of the model to distractor comments (randomly sampled from the remaining dataset). We find that context adapter module is able deal with with bad comments better than the averaging baseline without having been trained with distractor comments.
Model
#Distr.
TVR R@1
TVR R@10 10
VTR R@
VTR R @@) 10
averaging
1
2.18 (-12.52%)
10.50 (-10.63%)
2.38 (-9.33%)
11.12 (-7.85%)
averaging
3
1.82 (-26.70%)
8.83 (-24.80%)
1,91 (-27.20%)
9,47 (-21.47%)
averaging
5
1.41 (-43.30%)
7.64 (-34.96%)
1.63 (-37.98%)
8.01 (-33.60%)
averaging
7
1.41 (-43.18%)
6.95 (-40.83%)
1.32 (-49.91%)
6.66 (-44.83%)
adapt. title
1
1.94 (-5.66%)
9.76 (-6.92%)
1.82 (-7.48%)
9.28 (-7.02%)
adapt. title 3
1.67 (-18.83%)
8.81 (-15.96%)
1.61 (-18.05%)
8.42 (-15.59%)
adapt. title
5
1.59 (-22.81%)
8.13 (-22.46%)
1.49 (-24.30%)
7,81 (-21.77%)
adapt. title 7
1.50 (-27.15%)
7.66 (-26.95%)
1,40 (-28.85%)
7.19 (-27.949%
adapt. img 1
2.63 (-8.24%)
12.19 (-6.71%)
2.53 (-13.78%)
11.87 (-9.15%)
adapt. img
3
2.25 (-21.25%)
10.62 (-18.71%)
2.25 (-23.34%)
10.62 (-18.72%)
adapt. img 5
2.02 (-29.34%)
9.78 (-25.13%)
2.00 (-31.81%
9.53 (-27.05%)
adapt. img
7
1.82 (-36.59%)
8.98 (-31.29%)
1.81 (-38.37%)
8.83 (-32.38%)
Distractor Comments In order to test the ability our adapter to discard irrelevant information, evaluated how models perform when progressively add distractor comments from other videos compared to the averaging baseline from Table We find that, while both methods get gradually worse with a slower rate, as it can be seen when looking at the percentage decrease from the original score, showing that it is better equipped to discard irrelevant information. It is important to note that the model was not trained with explicit distractors and has learned this capability through the builtin attention mechanism in the context adapter model.
method's 's retrieval performance decreases distractor comments,
To test the performance of our model on another publicly available dataset, we perform experiments using the LiveBot Dataset [Ma al. 2019). As our pre-trained backbone is trained only in English, but the dataset is collected from a Chinese website, we translate all title and comments English before we run our method (see Table 18
Additional Dataset
Consistent with the results in the main paper, the retrieval performance of CLIP is surpassed by our model trained with comments. Using comments during retrieval further improves the results, confirming that comments are a useful modality and that our approach can successfully leverage this auxiliary information.
Table 18: LiveBot Dataset results. We show that even on a dataset such a LiveBt which is live comments on video feed, our CAM can extract meaningful information from the comments.
Model
Comments TVR R@5
TVR R@
VTR R@5
VTR R @ 10
CLIP
42.0
50.0
36.0
44.0
CLIP + CAM
no
45.0
50.0
39.0
46.0
CLIP + CAM
yes
53.0
64.0
53.0
66.0
21

---

Under review as a conference paper at ICLR 2022
Table 19: Bascline similarity thresholding. We show that the similarity score alone cannot identify distracting comments and discarding low similarity comments decreases performance.
Model
Sim. tresh.
TVR R@1
TVR WRR@II 10
VTR R@1
VTR R@ 10
averaging averaging
2.33
11.46
2.59
11.86
0.7
2.11
10.11
2.35
10.52
averaging
0.8
2.16
9.96
2.20
9.78
Filtering Comments by Similarity To understand the importance of learning to filter comments using simple thresholding technique the similarly returned by comparing embeddings, conduct the following experiment in [9] We use the similarity between a title and a comment as computed by the encoder model and remove comments that differ from the title more than a certain threshold. Since this experiments evaluates a non-learned filtering procedure we use the averaging baseline from Table 2 la reference model.
We find that removing comments generally reduces the retrieval performance as comments can indeed contain additional information. Simply thresholding the similarity metric cannot distinguish between comments that have other but useful information and comments that do not contain information related the video. Our model learns filter comments and can thus make use this information.
C MODEL DIAGRAM
22

---

Under review as a conference paper at ICLR 2022
[*comment 1", "oomment 2", "comment 3*]
Textual Backbone
512
Commant
Input Tokens
Comment
Comment
Video Feature
TimeSformer
(x2) Residual Attention
LayerNorm
Multi-Head Attention
LayerNorm
Linear + GELU
Linear
Comment
Output Tokens
Comment
Comment
Video Feature
Adapted
Figure 10: We show a diagram of the feature extraction and Context Adapter Module for the case of
adapting the Video Feature. Multi-Head Self-Attention is performed on the input tokens (which are themselves video or textual features) as part of transformer architecture consisting two Residual Attention blocks. Finally the output token corresponding to the Video Feature is passed through final linear layer and added to the original feature in a residual fashion.
23


Reviews:
[{"review_id": "me08J42G9Ox", "reviewer": "Reviewer_PRNC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n1. The paper is well written, and the problem is well motivated as well as clearly formulated.  The authors have made a strong case for using comments to learn better representations. They have adequately explained the challenges of using this modality, which explains why it was not used in earlier work. Although I was not aware of research in this area, I was able to follow the paper clearly. \n2. As far as I can tell, the idea of using comments while ignoring irrelevant comments has not been tried before. Hence, having an attention mechanism that can actually ignore irrelevant comments can aid many machine learning tasks, including representation learning.\n3. The experiments are carried out in diverse settings, which is impressive to carry out. However, I would expect this method to be tried on a public video dataset that already has comments. The dataset released by [1] can be a good fit.\n4. I appreciate that the limitations of this work are properly addressed. However, as shown in Figure 4, it concerns me that most real-world videos will have “distractor” comments which will be relevant to some part of the images in the video and may result in noisy representations. \n\n# Weaknesses\n\n1. It is not clear if something complicated like the Context Adapter Module is required to discard v_i and t_i pairs, where v_i and t_i are not equally informative of each other. Can a simple heuristic be used instead? For instance, if the initial value of A_{ij} does not qualify a certain threshold, then they are probably NOT equally informative of each other and are worth discarding.\n2. Similarly, can a heuristic be used to discount off-topic comments that have a low similarity score with the title? It would be great if the authors can compare their method with a heuristic like this.\n3. In the experimental section under 4.3, the authors show that varying the number of comments leads to improvement in performance. However, to simulate a realistic scenario, a subset of these comments should be unrelated or ambiguous. As an ablation study, the ratio of these unrelated comments should also be varied. It is difficult to conclude if the attention mechanism is working in absence of these experiments.\n4. The authors mention that they collected comments from the Kinetics dataset. For realistic use cases, it would be great to see experiments on a dataset which has video lengths larger than 10 seconds. Moreover, it would be important to see some statistics on the Kinetics dataset (particularly a histogram for number of comments on different videos). The authors mention that there are at least 3 comments, but more statistics would be appreciated. This is important because this is the only video dataset with comments that will be publicly released by the authors.\n\n# Other Concerns\n1. The majority of the experiments involve using the RedditVC dataset. The authors mention that they do not plan to release this dataset due to dataset bias issues. This makes reproducibility a major concern. It would be ideal if the authors can evaluate their method on a publicly available video dataset with comments [1].\n\n2. It would be important to add a significance test for these experiments. There are no error bars reported in the experiments. Were the experiments run multiple times to estimate errors in retrieval performance (particularly in Table 1)?\n\n3. While referring to the Appendix, it would be great if the authors can link to a particular section. For example, in Sec 4.5, Limitations, the reader would want to know where exactly they can look for additional examples, rather than simply mentioning appendix. I spent quite some time scrolling through the appendix to find more examples of limitations but could not find any.\n\n# References\n[1] Ma, Shuming, et al. \"Livebot: Generating live video comments based on visual and textual contexts.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n\nEdit after response:\nThank you for your detailed response.\nI would like to agree again that the proposed method has some merit. However, my main concern about the applicability of this module for realistic scenarios where the number of comments is very large still remains. I can see that this method will work when there are only a handful of diverse comments. It is unclear if it would work in a setting with large number of comments, where the number of distractor comments is larger than the non-distractor ones.", "summary_of_the_paper": "This paper proposes the use of user comments in addition to videos and titles to learn better representations for retrieval. Since user comments may be loosely related to the video, they use an attention-based mechanism to ignore irrelevant user comments. Their experiments show that using comments using this mechanism leads to better contextualized representations, which lead to competitive results on standard benchmarks.", "summary_of_the_review": "Please refer to the discussion in the Main Review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PRjjVeVzgu0", "reviewer": "Reviewer_599p", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The main idea of this paper is taking users' comments as a noisy signal to train multimodal (visual-text) video representation learning. Since the comments are not always strongly correlated to the contents that we are trying to learn, they proposed the CAM to selectively reflect the patterns discovered from the comments. Specifically, they utilize skip connections that are used in ResNet, so the model is forced to reflect patterns only if they are really necessary. This is an interesting idea and seems to work well in practice, according to the experiments.\n\n2. This paper focuses mainly on the effect of \"comments\", but I think the proposed idea may be applicable more broadly. For instance, multimodal video models using audio and visual features might take advantage of similar approaches. Some videos convey more information with visual signals (e.g., the sound is edited with a background music) while some other videos convey more information with audio (e.g., a music video with the visual is the album jacket all the time). Selectively reflecting signals from each mode may be applicable to any mode, even including visual or audio, not just text (comments).\n\n3. In this sense, this paper uses an idea which can be more generalized only within the narrow scope of \"comments\", and the experiments are verifying the efficacy of comments, not that of the model. For instance, the conclusion we can learn from Table 1 is that training and testing with comments help visual-text retrieval. However, this is already known. I believe the real value of this paper is coming from the model design, but that part is not well reflected in this table.\n\n4. The authors constructed two new datasets instead of using existing publicly available ones. This significantly limits opportunities of replication by others, so I am curious if the authors are willing to publicly share the dataset. Given the scale of the two datasets, they should be valuable sources to advance this field.\n\n5. Regarding comparison with SOTA models (in Section 4.4): The authors chose two datasets, MSVD and MSR-VTT. Although these are widely used datasets, there are many other datasets for this retrieval task. As it is known that retrieval performance highly depends on the training datasets, I'd recommend the authors to compare on other datasets like YouCook2, TVR, ActivityNet Captions, and more.\n\n6. In Table 6, the authors are comparing the proposed models trained on CLIP + RedditVC (+K700) against baselines trained on other pre-training datasets, mostly on HowTo100M. Related to my comments #5 above, the final performance can be significantly dependent on the pre-training datasets. For instance, we observe CLIP4Clip performs comparably with the proposed model, and it looks reasonable to guess it is because all 5 rows (CLIP4Clip and 4 Ours) were pre-trained on CLIP dataset. To be comparable, pre-training datasets should be the same for all models, either on HowTo100M or CLIP, or both. It is unclear now if the improvement is coming from the novel model or from more informative pre-training datasets. (Note that the number of samples is not the only factor determining the quality of datasets. HowTo100M is relatively weaker, for example, as most clips are short and the associated text sentences are very short and less informative for many clips.) As it is hard to control all conditions across datasets, it is recommended to (pre-)train on the same datasets.", "summary_of_the_paper": "In this paper, the authors proposed a multi-modal video representation learning and retrieval method based on it, taking advantage of visual and text embeddings, particularly users' comments on video sharing platforms. For this, the authors present a new component called Context Adapter Module (CAM), applying skip connections to selectively reflect user comments in the model. According to their experiments, the authors claim that the proposed method outperforms baseline models, by providing more contextualized representations of videos.\n", "summary_of_the_review": "Overall, the paper proposes an interesting idea, which I believe to have potentially broader impact than what the current manuscript claims. I encourage the authors to conduct additional experiments to apply the idea to other modes, as well as comparing with other methods on more datasets in a fairer way.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EketF6f50PG", "reviewer": "Reviewer_SsvR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Writing]\n- First of all, what is \"video-text retrieval\"? I think this could be comprehended as a video OCR method, which apparently is not what the paper is about. According to what I have understood from the paper, I think the authors are performing \"text-based video retrieval\". Please reconsider the vocabulary, and also neatly define the task at the beginning of the paper.\n\n- Table 1: What datasets do CLIP and CLIP + Our CAM use?\n\n- References: Most references miss detailed bibliographic information. Please provide complete bibliographic information for all references.\n\n[Evaluation]\n- I am not convinced that the proposed method is superior to CLIP4Clip. Although CLIP4Clip seems to require larger-scale training, since it has already been trained and published, there should be no demerit for that. Please explain in more detail the merit of the proposed method compared to CLIP4Clip. In addition, the proposed method requires user comments, which can not always be expected in large quantity. This could actually be a demerit in some situations.\n\n[Presentation]\n- Section 4.1: Please illustrate the structure of the network.\n- Fig. 3: The background color makes it difficult to see the lines, so please remove the background color, and also broaden the line widths.", "summary_of_the_paper": "The authors propose a text-based video retrieval method based on context from weakly related user comments. Evaluation shows that considering user comments improves the retrieval performance. Compared to conventional methods, the proposed method shows similar performance with CLIP4Cilip. However, CLIP4clip needs a much larger-scale training, so the superiority of the proposed method is claimed.", "summary_of_the_review": "The proposed method works well in case user comments are available, but even so, compared to a conventional method; CLIP4Clip, it performs similarly, so the technical merit is limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7Ecf4o9Bvb", "reviewer": "Reviewer_2J7x", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper:\n\nS1. The idea is intuitive.  Indeed a lot of videos now have user comments.  This is especially true for popular videos.  Having said that, popular videos probably are already getting a lot of signals from the popularity alone (likes, votes, etc.).  I'd be curious how the method could help retrieve less popular but relevant videos, where there may be some but not a lot of user comments in the first place (as it's a less popular video).\n\nS2. The method itself is straightforward.\n\nS3. Experiments are conducted on real datasets.  There is also ablation type of experiments looking into various components of the model. However, it is not immediately clear if there's a baseline that just throws all the comments as a single piece of text (just like a title would be), without attention to filter them out.  If so, that would be a more comparable baseline as it has the same information, but with a simpler processing.\n\nWeaknesses of the paper:\n\nW1. The novelty is rather incremental.  It's an extension of CLIP that already has a notion of video and text (e.g., title). The extension is in terms of user comments, which is another form of the text.  Though seemingly useful, the step to include comments as another form of text is not that surprising technically.\n\nW2. Though the paper emphasises heavily on the video element, page 6 states that experiments are conducted on a single still frame.  There needs to be more discussion on the generalisability of the result.  I wonder if the choice of frame may affect the overall results.\n\nW3.  The treatment of user comments as another piece of text is rather simplistic.  There could be richer structures to exploit, for instance the response/reply or discussion thread nature of some of these comments, which would be more interesting given the nature of this data.\n", "summary_of_the_paper": "The paper looks into video-text retrieval problem.  The claim is that existing works mostly rely on titles and captions.  The paper argues for using user comments.  The challenge is that not all user comments are meaningful or relevant.  Therefore, the paper looks into attention mechanism to filter out the irrelevant content. The main contribution is a context adapter module based on transformer that allows relating visual input and textual input. The base architecture seems to have been derived from CLIP.", "summary_of_the_review": "The paper proposes an intuitive idea of leveraging on user comments to improve video-text retrieval, but the technical novelty is rather incremental.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on how to improve video-text retrieval via using additional user comments, and uses an attention mechanism to filter out the irrelevant comments. The main contribution is a context adapter module that allows learning from the auxiliary modality through an attention mechanism. The reviewers appreciated the overall idea's intuition and well-written paper, but they also felt that the technical novelty is incremental, and that the treatment of user comments should be more intuitive via the dialogue thread structure. There were also concerns about the applicability of the context adapter module to more realistic scenarios with much longer videos, where the number of comments is very large, and where number of distractor comments is larger than the non-distractor ones.
Decision: Reject