wMAN: WEAKLY-SUPERVISED MOMENT ALIGNMENT NETWORK FOR TEXT-BASED VIDEO SEGMENT RETRIEVAL
Authors: Reuben Tan
Huijuan Xu
Kate Saenko
Bryan A. Plummer
Venue:  (2020)
Keywords: vision
language
video moment retrieval

Abstract: Given a video and a sentence, the goal of weakly-supervised video moment retrieval is to locate the video segment which is described by the sentence without having access to temporal annotations during training.  Instead, a model must learn how to identify the correct segment (i.e. moment) when only being provided with video-sentence pairs.  Thus, an inherent challenge is automatically inferring the latent correspondence between visual and language representations. To facilitate this alignment, we propose our Weakly-supervised Moment Alignment Network (wMAN) which exploits a multi-level co-attention mechanism to learn richer multimodal representations. The aforementioned mechanism is comprised of a Frame-By-Word interaction module as well as a novel Word-Conditioned Visual Graph (WCVG). Our approach also incorporates a novel application of positional encodings, commonly used in Transformers, to learn visual-semantic representations that contain contextual information of their relative positions in the temporal sequence through iterative message-passing. Comprehensive experiments on the DiDeMo and Charades-STA datasets demonstrate the effectiveness of our learned representations: our combined wMAN model not only outperforms the state-of-the-art weakly-supervised method by a significant margin but also does better than strongly-supervised state-of-the-art methods on some metrics.

Under review as a conference paper at ICLR 2020
WMAN: WEAKLY-SUPERVISED MOMENT ALIGNMENT NETWORK FOR TEXT-BASED VIDEO SEGMENT RE- TRIEVAL
Anonymous authors Paper under double-blind review
ABSTRACT
Given video and a sentence, the goal of weakly-supervised video moment re- trieval is to locate the video segment which described by the sentence without having access to temporal annotations during training. Instead, model must learn how to identify the correct segment (i.e. moment) when only being provided with video-sentence pairs. Thus, an inherent challenge is automatically infer- ring the latent correspondence between visual and language representations. To facilitate this alignment, propose our Weakly-supervised Moment Alignment Network (wMAN) which exploits a multi-level co-attention mechanism to learn richer multimodal representations. The aforementioned mechanism comprised of Hoane-By-Worts interaction module as well as novel Word-Conditions Visual Graph (WCVG). Our approach also incorporates novel application of positional encodings. commonly used in Transformers, to learn visual-semantic representations that contain contextual information their relative positions the temporal sequence through iterative pesssag-paassing Comprehensive exper- iments on the DiDeMo and Charades- STA datasets demonstrate the effectiveness of our learned representations: our combined wMAN model not only outperforms the state-of-the-ar weakly-supervised method by a significant margin but also obtains improvement of 10% for the Recall @ accuracy metric strongly- supervised state-of-the-ar methods on the DiDeMo dataset.
INTRODUCTION
Video understanding has been mainstay of artificial intelligence research. Recent work has sought to better reason about videos by learning more effective spatio-temporal representations (Tran al.) 2015 Carreira & Zisserman, 2017). The video moment retrieval task, also known as text-to-clip retrieval, combines language and video understanding to find activities described by natural lan- guage sentence. The main objective of the task is to identify the video segment within a longer video that is most relevant sentence. This requires model to learn the mapping of correspondences (alignment) between the visual and natural language modalities.
In the strongly-supervve Ghosh et al., 2019) generally learn joint visual-semantic representations by projecting video and language representations into common embedding space and leverage provided temporal tations to learn regressive functions (Gao et al.] (017) for localization. However, such temporal annotations are often ambiguous and expensive collect. MMthhun al. (2019) seeks to circumvent these problems by proposing to address this task in the weakly-supervised setting where only full video-sentence pairs are provided as weak labels. However, the lack of temporal annotations ren- ders the aforementioned approaches infeasible. In their approach (Figure [p), Mithun et al. (2019) proposes Text-Guided Attention (TGA) mechanism to attend on segment-level features w.r.t. the sentence-level representations. However, such an approach treats the segment-level visual represen- tations as independent inputs and ignores the contextual information derived from other segments in the video. More importantly, it does not exploit the fine-gra grained semantics of each word in the sentence. Consequently, existing methods are not able to reason about the latent alignment between the visual and language representations comprehensively.
setting, existing methods (Hendricks al. 2017 Chen al. 2018

---

Under review as a conference paper at ICLR 2020
(8)
Existing Methods
Sentence Representation
Information From Other Framas
(b)
Our
Proposed
Approach
Word-Level
Representation
Figure 1: Given video and sentence, aim is to retrieve the most relevant segment (the red bounding box in this example). Existing methods consider video frames independent inputs and ignore the contextual information derived from other frames in video. They compute similarity score between segment and the entire sentence to determine their relevance to each other. In contrast, our proposed approach aggregates contextual information from all the frames using graph propagation and leverages finc-graines frame-by- interactions for more accurate retrieval. (Only some interactions shown prevent overcrowding the figure.)
In this paper, we take another step towards addressing the limitations of current weakly-supervise video moment retrieval methods by exploiting the fine-grained temporal and visual relevance each video frame to each word (Figure [p). Our approach is built two core insights: 1) The temporal occurrence of frames or segments in video provides vital visual information required to reason about the presence of event; 2) The semantics of the query integral reasoning about the relationships between entities in the video. With this mind, we propose our Weakly-Supervisee Moment Alignment Network (wMAN), An illustrative overview our model shown in Figure 2 The key component of wMAN is multi-level co-attention mechanism that is encapsulated by Frame-by-Wand Word (FBW) interaction module as well as Word-Conditioned Visual Graph (WCVG). To begin, we exploit the similarity scores of all possible pairs of visual frame and word features create frame-specific sentence representations and word-specifie video representations. The in- tuition is that frames relevant word should have higher measure of similarity compared to the rest. The word representations are updated by their word-specific video representations to create sesuatisemanti representations. Then graph (WCVG) is built upon the frame and visual- semantic representations as nodes and introduces another level attention between them. During the message- passing process, the frame nodes iteratively updated with relational information from the visual-semantic nodes to create the final remporally-awart multimodal representations. The contribution of each visual-semantic node a frame node dynamically weighted based on their similarity. To learn such representations, wMAN also incorporates positional encodings (Vaswani al. 2017 into the visual representations integrate contextual information about their relative positions. Such contextual information encourages the learning of temporally-awee -wware multimodal representations.
To learn these representations, we use Multiple Instance Learning (MIL) framework that is similar in nature the Stacked Cross Attention Network (SCAN) model (Lee et al., 2018). The SCAN model leverages image region-by-word interactions to learn better representations for image-text matching. addition, the WCVG module draws inspiration from the Language-Condinn Conditioned Graph Network (LCGN) by Hu et al. (2019) which seeks to create context-aware object features in im- age. However, the LCGN model works with sentence-lewe representations, which does not account for the semantics of each word to each visual node comprehensively. wMAN also distinguishes itself from the bobove-mentione models by extracting lemporally-wwar multimodal representations from videos and their corresponding descriptions, whereas SCAN and LCGN only work on images.
2

---

Under review as a conference paper at ICLR 2020
Visual-Semantic
swinging
Representation
Weungge-assinng
10
Mescage-pssingg
GRU
EN
Frame-By Brame-By-Weil By-
1+T
Interactions
explely
W
Frame-Specific Sentence
E expls
Representation
PE
CNN
LSTM
UN
ly
Sim
3,
Q) log cop oRll
Figure 2: An overview of our combined wMAN model which is trained end-to-end. We use the outputs of the GRU word representations where its inputs are word embeddings. The visual representations are the outputs of the LSTM unit where its inputs are the extracted features from a pretrained CNN. The visual representations contextual information about their relative positions in the sequence. Our model consists of a two- stage multimodal interaction mechanism Frame-By- By-nedd Interactions and the WCVG.
concatenated with positional encodings to integrate
The contributions of our paper are summarized below:
We propose a simple yet intuitive MIL approach for weakly-supervised video moment retrieval from language queries by exploiting fine-grained frame-by-wond alignment.
Our novel Word-Conditioned Visual Graph learns richer visual-semantic context through multi-level co-attention mechanism.
We introduce novel application of positional embeddings in video representations to learn temporally-awwe aware multimodal representations.
To demonstrate the effectiveness of our learned temporally-aware multimodal representations, perform extensive experiments over two datasets, Didemo (Hendricks et al.) 2017) and Charades- STA (Gao et al. 2017), where we outperform the state-of-the-art weakly supervised model by significant margin and strongly-supervised state-of-the-an models on some metrics.
2 RELATED WORK
Most of the recent works in video moment retrieval based natural language queries (Hendricks
ctt al. 2017] Ghosh et al. 2019 Xu et al. 2019)) Zhang et al. 2019 Chen et al.] 2018 Yuan et al. 2019 Chen & Jiang, 2019: Chen al. 2019 Ge et al. 2019) in the strongly-supervisee set- ting, where the provided temporal annotations can be used to improve the alignment between the visual and language modalities. Among them, the Moment Alignment Network (MAN) intro- duced by Zhhang et al. (2019) utilizes a structured graph network to model temporal relationships between candidate moments, but of the distinguishing factors with our wMAN is that our it- message-passing
erative message- passing process is conditioned on the multimodal interactions between frame and word representations. The TGN (Chen et 2018) model bears some resemblance to ours by lever- aging frame-by- -word interactions to improve performance. However, it only uses single level of attention which not able to infer the correspondence between the visual and language modalities comprchensively. In addition, we reiterate that all these methods train their models using strong supervision, whereas we address the weakly supervised setting this task.
There are also number of closely-related tasks to video moment retrieval such temporal activity detection in videos. general pipeline of proposal and classification is adopted by various temporal activity detection models (Xu et al. 2017 Zhao al. 2017; Shou et al. 2016) with the temporal proposals learnt by temporal coordinate regression. However, these approaches assume you are provided with predefined list of activities, rather than open-ended list provided via natural
3

---

Under review as a conference paper at ICLR 2020
language queries test time. Methods for visual phrase grounding also tend to be provided with natural language queries as input (Chen et al., 2017 7 Liu et al. 2017; Faghri et al. 2018 Nam et al.)
2017| Karpathy & Fei-Fei |015| Plummer et al. 2018), but the task 18 performed over image regions to locate a related bounding box rather than video segments to locate the correct moment.
3 WEAKLY-SUPERVISE MOMENT ALIGNMENT NETWORK
In the video moment retrieval task, given a ground truth video-sentence pair, the goal is to retrieve the most relevant video moment related the description. The weakly-supervised version of this task we address can be formulated under the multiple instance learning (MIL) paradigm. When training using MIL, one receives a bag of items, where the bag is labeled as a positive if at least one item in the bag is a positive, and is labeled as negative otherwise. In weakly-supervised moment retrieval, we are provided with a video-sentence pair (i.e., a bag) and the video segments are the items that we must learn to correctly label as relevant to the sentence (i.e., positive) or not. Follow- ing Mithun et al. (2019), we assume sentences are only associated with their ground truth video, and any other videos are negative examples. To build good video-sentence representation, we introduce Moment Alignment Network (wMAN), which learns context-aware visual- semantic representations from fine-grained frame-by- interactions. As seen in Figure our network has two major components - (1) representation learning constructed from the Frame-By- Word attention and Positional Embeddings (Vaswani al. 2017), described in Section 3.1 and (2) a Bord-Conditioned Visual Graph where we update video segment representations based on context from the rest of the video, described in Section 3.2 These learned video segment representations are used to determine their relevance to their corresponding attended sentence representations using a LogSumExp (LSS)) pooling similarity metric, described Section
our
3.1 LEARNING TIGHTLY COUPLED MULTIMODAL REPRESENTATIONS
In this section we discuss our initial video and sentence representations which are updated with contextual information in Section 3.2 Each word in an input sentence is encoded using GloVe embeddings (Pennington et al. 2014) and then fed into a Gated Recurrent Unit (GRU) (Cho al. 2014]. The output of this GRU is denoted as W = {wi, W2, '", wQ} where Q is the number of words in the sentence. Each frame in the input video is encoded using a pretrained Convolutional Neural Network (CNN). In the case of 3D CNN this actually corresponds to small chunk of sequential frames, but we shall refer to this as a frame representation throughout this paper for simplicity. To capture long-range dependencies, we feed the frame features into a Long Thort-Term Memory (LSTM) (Hochreiter & Schnidhuber [199).. The latent hidden state output from the LSTM are concatenated with positional encodings (described below) form the initial video representations, denoted as V = [11, U2, ..... UN) } where N is the number of frame features for video V.
Positional Encodings (PE). To provide some notion of the relative position of each frame we in- clude the PE features which have been used in language tasks like learning language representations using BERT (Devlin et al. 2018: Vaswani al., 2017). These PE features can be thought of similar to the temporal endpoint features (TEF) used in prior work for strongly supervised moment retrieval task (e.g.. Hendricks et al. (2017), but the PE features provide information about the temporal po- sition of each frame rather than the rough position at the segment level. For the desired PE features of dimension d, let pos indicates the temporal position of each frame, is the index of the feature being encoded, and M is a scalar constant, then the PE features are defined as:
{
sin(pos/ /M/d) if i is even cos(pos/MMMd) otherwise.
Through experiments, we found the hyper- prper-parameter M = 10,000 works well for all videos. These PE features concatenated with the LSTM encoded frame features at corresponding frame position before going to the cross-modal interaction layers.
3.1.1 FRAME--By WORD INTERACTION
Rather than relating a sentence-leve representation with each frame as done in prior work (Mithun et 2019, we aggregate similarity scores between all frame and word combinations from the input
4

---

Under review as a conference paper at ICLR 2020
video and sentence. These Frame-By- Word (FBW) similarity scores used to compute attention weights to identify which frame and word combinations are important for retrieving the correct video segment. More formally, for N video frames and Q words in the input, compute:
of w where E [1, NJ and j [1.
8ij
|0 meyl
Note that e now represents the concatenation the video frame features and the PE features. F Frame-Specific Sentence Representations. We obtain the normalized relevance of cach word w.r.. to each frame from the FBW similarity matrix, and use it to compute attention for each word:
exp(s.)
0i = Σラ exp(s.))
(3)
Using the above-mentioned attention weights, a weighted combination of all the words are created, with correlated words the frame gaining high attention. Intuitively, word-frame pair should have a high similarity score if the frame contains a reference to the word. Then the frame- specifcc sentence representation emphasizes words relevant to the frame and defined as:
1
(4)
Note that these frame-specific sentence representations don't participate in the iterative message- passing process (Section 3.2) Instead, they are used to infer the final similarity score between video segment and the query (Section 3.3).
Word-Specific Video Representations. To determine the normalized relevance of each frame w.r.t. to each word, we compute the attention weights of each frame:
exp(si))
(5)
exp(sig)
Similarly, we attend to the visual frame features with respect each word by creating weighted combination of visual frame features determined by the relevance each frame to the word. The formulation of each word-specific video-representation is defined as:
N 4.y Us.
f,
(6)
These word-specific video representations are used in our Word-Conditioned Visual Graph, which will discuss in the next section.
3.2
WOND-CONDI VISUAL GRAPH NETWORK
Given the sets of visual representations, word representations and their corresponding word-specific video representations, the WCVG aims to learn temporally-awan multimodal representations by integrating lisual-smmattc and contextual information into the visual features. To begin, the word representations are updated with their corresponding video representations to create new visual- semantic representation w/is by concatenating each word wy and the word-specific video represen- tation Jj. Intuitively, the visual-semantic representations not only contain the semantic context of each word but also summary of the video with respect to each word. A fully connected graph then constructed with the visual features v, and the embedded attention of visual-semantic represen- tations w as nodes.
Keessage-Passing The iterative pneessage-passingg process introduces second round of FBW interaction similar to that in Section 3.1.I to infer the latent temporal corre- spondence between each frame 2 and visual-semantic representation w The goal is to update the representation of each frame v; with the video context information from each word-specific video representation whis To realize this, first learn a projection W, followed by ReLU of wieee to
Iterative
5

---

Under review as a conference paper at ICLR 2020
obtain a new word representation compute similarity matrix 5'y every Passsage-passingg iteration, namely, we obtain a replacement for wy in Eq. (2) via wrj ReLU (Wi(w)))))
Updates of Visual Representations During the update process, each visual-semantic node sends its message (represented by representation) each visual node weighted by their edge weights. The representations of the visual nodes at the t-th iteration are updated by summing up the incoming messages as follows:
Q 0 1)
y W2(concat of
to the newly computed FBW similarity matrix s/j. and W2 a learned projection make of the same dimensions the frame-specific sentence representation 4 (refer to Eq. which are finally used to compute sentence-segment similarity score.
where aij is obtained by applying Eq. 3
3.3 MULTIMODAL SIMILARITY INFERENCE
The final updated visual representations 1 = ".......... ,..,"()))) are used to compute the relevance of each frame its attended sentence-rerresentationn.. A segment is defined any arbitrary tinuous sequence of visual features. We denote segment as S {of u where K is the number of frame features contained within the segment S. We adopt the Exp (LSE) pooling similarity metric used in SCAN (Lee et al. 2018), to determine the relevance each proposal segment has to the query:
LA 1 4
SimLsE(S,Q) log( exp(AR(U 4)) where R(vx.4))
(8)
is a hyperparameter that weighs the relevance of the most salient parts of the video segment to the corresponding frame-specific sentence representations. Finally, following Mithun et al.) (2019). given a triplet (X+, y+, Y-), where (X+, .Y+) is a positive pair and (X+, Y-) negative pair, we based ranking loss LT train model which ensures the positive pair's similarity score is better than the negative pair's by at least a margin. Our model's loss is then defined as:
Trtaall E
Σ TT V+.V")
(V+
Sim LSE is used the similarity metric between all pairs. At test time, SémLsE is also used to rank the candidate temporal segments generated by sliding windows, and the top scoring segments will the localized segments corresponding the input query sentence.
EXPERIMENTS
We evaluate the capability of wMAN to accurately localize video moments based natural lan- guage queries without temporal annotations on two datasets DiDeMo and Charades-STr On the DiDeMo dataset, old = 0. Recall@N N represents the percentage of the test sliding window samples which have a overlap at least 0 with the ground-truth segments, mIOU the average IOU with the ground- truth segments for the highest ranking segment to each query input. On the Charades-STr. dataset, only the Recall @ N metric is used for evaluation.
adopt the mean
-irrecttion-Ov--nniiiii (IOU) and Recall at IOU thresh-
4.1 DATASETS
Charades-STA The Charades-STA dataset is built upon the original Charades (Sigurdsson et al. [2066]] dataset which contains video-level paragraph descriptions and temporal annotations for ac- tivities. Charades-STA is created by breaking down the paragraphs to generate sentence-level anno- tations and aligning the sentences with corresponding video segments. In total, it contains 12,408 and 3,720 query-moment pairs in the training and test sets respectively. For fair comparison with the weakly model TGA (Mithun et al.) 2019), we use the same non-overlapping sliding windows of sizes 128 and 256 frames to generate candidate temporal segments.

---

Under review as a conference paper at ICLR 2020
Table 1: Moment retrieval performance comparison the Charades-STA test (a) contains representative results of strongly-supervised methods reported in prior works while (b) reports the performance of weakly-supervised methods including our approach.
Training
iou 0.3
iou 0.5
iou = 0..
Method
Supervision R Φ
R@5 R@ 10 R@ 1 R@5 R 10
R@
R@5 R@ TO
(a) CTRL (Gao et
2017
Strong
23.63
58.92
8.89
29.52
MLVI (Xu et al. 2019)
Strong
54.7 95.6 99.2
35.6 79.4 93.9
15.8
45,4 62.2
MAN Zhang et al.
2019)
Strong
46.53 86.23
22.72
53.72
(b) TGA (Mithun et al.
2019)
Weak
29.68
83.87
98.41
17.04
58.17
83.44
6.93
26.80
44.06
wMAN (ours)
Weak
48.04 89.01
99.57
31.74
72.17
86.58
13.71
37.58
45.16
Upper Bound
99.84
88.17
46.80
DiDeMo The videos in the Distinct Describable Moments (DiDeMo) dataset are collected from Flickr. The training, validation and test sets contain 8395, 1065 and 1004 videos respectively. Each query contains the temporal annotations from at least 4 different annotators. Each video is limited to a maximum duration of 30 seconds and equally divided into six segments with five seconds each. With the five-second segment as basic temporal unit, there are 21 possible candidate temporal segments for each video. These 21 segments will used to compute the similarities with the input query and the top scored segment will be returned as the localization result.
4.2 IMPLEMENTATION DETAILS
For fair comparison, we utilize the same input features as the state-of-the-art method (Mithun et al, 2019). Specifically, the word representations are initialized with GloVe embeddings and fine-tuned during the training process. For the experiments on DiDeMo, we use the provided mean-poole mean-pooled visual frame and optical flow features. The visual frame features are extracted from the fc7 layer of VGG-16 Simonyya & Zisserman (2014)) pretrained on ImageNet [Deng et al. (2009) The input visual features for our experiments on Charades-S"
are C3D (Tran et al. (2015) features. We adopt an initial learning rate of le 5 and a margin= 0.5 used in our model's 's triplet loss (Eq.9). In addition, we use three iterations for the message-passing process. Our model is trained end-to-end using the ADAM optimizer.
4.3
RESULTS
4.3.1
CHARADES-STD
The results in Table shoow that our full model outperforms the TGA model by significant margin on all metrics. In particular, the Recall@ accuracy when IOU 0.7 obtained by our model is almost doubled that of TGA. It is notable that observe consistent trend of the Recall @ accuracies improving the most across all IOU values. This not only demonstrates the importance of richer joint visual-semantic representations for accurate localization but also the superior capability of our model to learn them. Our model also performs comparably to the strongly-superviee MAN model on several metrics.
To better understand the contributions of each component of our model, we present a comprehensive set of ablation experiments Table Note that our combined wMAN model comprised of the FBW and WCVG components as well as the incorporation of PEs. The results obtained by our FBW variant demonstrate that capturing mneigrai frame-by-word interactions essential to inferring the latent temporal alignment between these two modalities. More importantly, the results in the second row (FBW-WCVG) show that the second stage of multimodal attention, introduced by the WCVG module, encourages the augmented learning of intermodal relationships. Finally, we also observe that incorporating positional encodings into the visual representations (FBW-WCVG: + PE) especially helpful in improving Recall@ @ accuracies for all IOU values. We provide results for a model variant that include TEFs which encode the location of each video segment. In Table 2 our experiments show that TEFs actually hurt performance slightly. Our model variant with PEs (FBW.

---

Under review as a conference paper at ICLR 2020
Table 2:
Charades-STA ablation experiment results held-out validation set. In the table, indicates the same number of model parameters as the combined wMAN model.
iou
0.3
1ou = 0.5
1ou = 0.7
Method
R@1
R@5
R@
R@
R@5
R@
R@1
R@5
R@ 10
FBW
41.41
93.79
99.23
26.91
72.19
85.97
10.83
34.85
45.20
43.99
90.85
99.19
28.24
70.70
86.14
11.64
34.85
45.20
FBW-WCVG + TEF
43.99
88.03
98.99
28.01
69.19
86.01
11.20
35.29
44.45
FBW-WCVG *
42.18
91.09
99.19
27.47
70.58
86.09
11.52
34.97
45.11
FBW-WCVG + PE (WMAN)
46.05
91.25
99.19
29.00
69.46
86.26
13.30
36.99
45.32
Table 3: Moment retrieval performance comparison on the DiDeMo test set. (a) contains representä- tive results of
methods reported in prior works while (b) reports the performance of weakly-supervised methods including our approach.
Method
Training Supervision
R@1
R@5
mIOU
(a) MCN Hendricks et (2017)
Strong
28.10
78.21
41.08
TGN (Chen ct al. 2018
Strong
28.23
79.26
42.97
(b) TGA Mithhnn et al. 2019
Weak
12.19
39.74
24.92
wMAN
Weak
38.07
63.94
38.37
Upper Bound
74.75
100.00
96.05
WCVG + PE) outperforms the model variant with TEFs (FBW-WCVG+ TEF) on all the metrics. We theorize that the positional encodings aid in integrating temporal context and relative positions into the learned visual-semantic representations. This makes particularly useful for Charades-STA since its videos are generally much longer.
To gain insights into the fine-graines interactions between frames and words, we provide visual- izations in Figure 3 Our model is able to determine the most salient frames with respect to each word relatively well. both examples, we observe that the top three salient frames with respect to each word are generally distributed over the same subset of frames. This seems to be indicative of the fact that our model leverages contextual information from all video frames as well as words in determining the salience of each frame to specific word.
4.3.2 DIDEMO
Table reports the results on the DiDeMo dataset. In addition to reporting the state-of-the-art weakly supervised results, also include the results obtained by groongl-suuerviss methods. It can be observed that our model outperforms the TGA model by a significant margin, even tripling the Recall @ accuracy achieved by them. This demonstrates the effect of learning richer joint visual- semantic representations on the accurate localization of video moments. In fact, our full model outperforms the strongly-supervised TGN and MCN models the Recall@) 1 metric by approxi- mately 10%.
We observe consistent trend in the ablation studies (Table 4 as with those of Charades-STA. In particular, through comparing the ablation models FBW and FBW-WCVG, we demonstrate the effectiveness of our multi-level co-attention mechanism in WCVG where improves the Recall accuracy by a significant margin. Similar to our observations in Table [ PEs help to encourage accurate latent alignment between the visual and language modalities, while TEFs fail in this aspect.
5 CONCLUSION
In this work, we propose our weakly-supervised Moment Alignment Network with Word- Conditioned Visual Graph which exploits multi-level co-attention mechanism to infer the latent alignment between visual and language representations at fine- grained word and frame level. Learn ing antext--wwre visual- semantic representations helps our model to reason about the temporal oc- currence of event well as the relationships of entities described in the natural language query.
8

---

Under review as a conference paper at ICLR 2020
Table 4: DiDeMo ablation experiment results the validation In the table, indicates the same number of model parameters as the combined wMAN model.
Method
R@D
R@5
MIOU
FBW
30.19
66.74
39.06
FBW WCVG
39.93 66.53
39.19
FBW-WCVG TEF
37.55
66.36
39.11
FBW-WCVG
39.06
66.31
39.05
FBW-WCVG PE (wMAN)
41.62 66.57
39.20
(a)
GT:
0.0
9.9
Predicted:
0.0
1,1,3,
2, 1,
3
3, 2,
Query:
A person is holding a glass cup.
Lb)
GT:
13.2
20.8
Predicted:
10.7
21.3
2
3, 2
2, 3,
2,
1, 1, 1,
Query:
A person sits down on a chair.
Figure 3: Visualization of the final relevance weights of each word in the query with respect to each frame. Here, we display the top three weights assigned the frames for each phrase. The colors of the three numbers (1,2.3) indicate the correspondence to the words in the query sentence. We also show the ground truth (GT) temporal annotation well our predicted weakly localized temporal segments in seconds. The highly correlated frames to each query word generally fall into the GT temporal segment both examples.
Finally, our experimental results empirically demonstrate the effectiveness of such representations on the accurate localization of video moments.
REFERENCES
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua. Temporally grounding nat- ural sentence in video. In Proceedings of the 2018 Conference on Empirical Methods in Natural

---

Under review as a conference paper at ICLR 2020
Language Processing, pp. 2018.
Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. Localizing natural language in videos. AAAI, 2019.
Kan Chen, Rama Kovvuri, and Ram Nevatia. Query-guided regression network with context policy for phrase grounding. In Proceedings of the IEEE International Conference on Computer Vision, pp. 824 832, 2017.
Shaoxiang Chen and Yu-Gang Jiang. Semantic proposal for activity localization in videos via sen- tence query. 2019.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406. 1078, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-2552 Icee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0444 04805, 2018.
Fartash Faghri, David Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual-
semantic embeddings with hard negatives. In Proceedings of the British Machine Vision Confer- ence (BMVC), 2018. URL httLattts///ttttia
com/ Partashff / vsepp
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE International Conference on Computer Vision, pp- 5267-5275, 2017.
Runzhou Ge, Jiyang Gao, Kan Chen, and Ram Nevatia. Mac: Mining activity concepts for language- based temporal localization. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp.
IEEE, 2019.
Soham Ghosh, Anuva Agarwal, Zarana Parekh, and Alexander Hauptmann. Excl: Extractive clip localization using natural language descriptions. arXiv preprint arXiv: 19Xx::994400255. 2019.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In International Conference on Computer Vision (ICCV), 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Ronghang Hu, Anna Rohrbach, Trevor Darrell, and Kate Saenko. Longuage-conditionee graph net- works for relational reasoning. In Proceedings of the IEEE International Conference Computer Vision (ICCV), 2019.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 3128-3137, 2015.
Kuang- Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 201-216, 2018.
Jingyu Liu, Liang Wang, and Ming-Hsuan Yang. Referring expression generation and comprehen- sion via attributes. Proceedings of the IEEE International Conference on Computer Vision, pp- 4856-4864, 2017.
Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Koy-Chowdhury. Weakly supervised video moment retrieval from text queries. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11592-11600 1101, 2019.
10

---

Under review as a conference paper at ICLR 2020
Hyeonseob Nam, Jung-Wor Woo Ha, and Jeonghee Kim. Dual attention networks for multimodal rea- soning and matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 299-307, 2017.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1543, 2014.
Bryan Plummer, Paige Kordas, M Hadi Kiapour, Shuai Zheng, Robinson Piramuthu, and Svet- lana Lazebnik. Conditional image-text embedding networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 249-264, 2018.
Zheng Shou, Dongang Wang. and Shih-Fu Chang. Temporal action localization in untrimmed videos via multi-stage cnns. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1049-1058, 2016.
Gunnar A. Sigurdsson, Gil Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.15555 2014.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings the IEFE international conference on computer vision, pp. 1489-4497,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Huijuan Xu, Abir Das, and Kate Saenko. R-c3d: Region convolutional 3d network for temporal activity detection. In Proceedings of the IEEE international conference on computer vision, pp- 5783-5792. 2017.
Huijuan Xu, Kun He, Bryan A Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko. Multilevel language and vision integration for text-to-clip retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9062-9069, 2019.
Yitian Yuan, Tao Mei, and Wenwu Zhu. To find where you talk: Temporal sentence localization in video with attention based location regression. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 9159- 9166, 2019.
Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis. Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1247-1257, 2019.
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2914-2923, 2017.
11

---

Under review as a conference paper at ICLR 2020
A EXPERIMENTS FOR NUMBER OF MODEL PARAMETERS ON CHARADES-SD -TA
Table 5: Moment retrieval performance comparison on the Charades-STA test set. In this table, display the number of model parameters as well as the results achieved on the Charades-Sta Test Set.
Number of
1ou = 0.3
iou 0.5
0.7
Method
Model Parameters
R
R@5 R@ 10 R@I
R@5 R 10 R
R@5
R@ 10
TGA (Mithun et al.
2019)
3M
29.68 83.87 98,41 17.04 58.17 83.44
6.93
26.80
44.06
TGA (Mithun ct al.
2019
19M
27.36 77.58 99.03 14.38 59.97 85,83
5.24 30.40 44.67
LCGN (Hu et al.
2019)
152M
35.81 82.93 99.09 19.25 65.11 85.19
7.12 32.90 43.63
FBW
3M
38.13 90.49 99.48 24.73 69.42 85.29
9.91 34.32 43.94
FBW
20M
38.33 91.10 99.43 24.51 69.18 86.31
10:24
33.17 44.40
FBW + WCVG
18M
42.84 88.05 99.54 27.60 70.00 86.61
11.47 34.30 44.73
wMAN (ours)
18M
48.04 89.01 99.57 31.74 72.17 86.58 13.71 37.58 45.16
we show the comparisons of the different methods with different number of model parameters. While wMAN has 18M parameters as compared to 3M parameters in TGA, the per- formance gains are not simply attributed to the number of model parameters. We increase the dimensions of visual and semantic representations ers in the TGA model which leads to a total of 19M parameters. Despite having more parameters than wMAN, it still does significantly worse all metrics. We also provide results obtained by a direct adaptation of the work the image level for VQA as well. While LCGN leverages attention over the words in the natural language query, the computed attention is only conditioned on the entire sentence without contextual information derived from the objects' visual representations. contrast, the co-attention mechanism in our combined wMAN model is conditioned on both semantic and contextual visual information derived from words and video frames respectively. LCGN also lot more compli- cated and requires significantly more computing resources than wMAN. Despite possessing much more parameters than wMAN, is still not able to achieve comparable results to ours.
In Table
well as corresponding fully-connected lay-
Graph Network (LCGN), which is designed to
B ABLATION ON NUMBER OF MESSAGE PASSING ROUNDS
Table 6: Charades-STD ablation experiment results on the held-put pul validation set.
Number of Message
iou = 0.3
iou = 0.5
iou = 0.7
Passing Rounds
R@l
R@5
R@ 10
R@
R@5
R@1
10
R@1
R@5
R@10
43.39
86.14
99.13
15.18
68.89
86.14
13.10
36.14
45.18
3
46.05
91.25
99.19
29.00
69.46
86.26
13,30
36.99
45.32
43.71
88.07
99.15
15.31
68.94
86.53
13.10
36.50
45.24
Table 7: DiDeMo ablation experiment results on the validation set.
Number of Message Passing Rounds R@1 1 R@5 MIOU
2
40.04 66.57
39.14
3
41.62
66.57
39.20
4
40.11 66.36 39.06
In this section, we include ablation results on the number of message-passing rounds required to leam effective visual-semantic representations. In our experiments, we have found that three rounds work best on both Charades-Sta and DiDeMo.
12

---

Under review as a conference paper at ICLR 2020
C
ABLATION ON TEST SETS
Table 8: Charades-STA ablation experiment results on the test set.
iou = 0.3
iou = 0.5
iou = 0.7
Method
R@1
R@5
R@
10
R@
R@5
R@ 10
R@l
R@5
R 10
FBW
41.47
95.99
99.56
26.74
72.68
86.50
10.37
34.97
44.89
FBW PE
45.77
95.59
99.41
29.68
74.92
86.23
12.71
37.47
45.16
FBW-WCVG
42.95
90.16
99.35
27.79
70.53
86.66
11.26
34.83
45.00
FBW-WCVG + PE (wMAN)
48.04
89.01
99.57
31.74
72.17
86.58
13.71
37.58
45.16
Table 9: DiDeMo ablation experiment results on the test set.
Method
R 1 R@5
MIOU
FBW FBW + PE
26.43
63.88
38.49
20.67
63.44
38.58
FBW-WCVG
36.47
63.83
38.36
FBW-WCVG PE (wMAN) 38.07
63.94
38.37
The results in Tables 8 and emphasize the importance of both PEs and WCVG in our approach. We observe the best performances when PEs are integrated into visual-semantic representa- tions learned by the WCVG. The consistency of these performance gains across both datasets seems indicative of the increased capability of our model to learn aemporall--aware visual-semann semantic repre- sentations.
13


Reviews:
[{"review_id": "rJxsIwXpcB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: \nThe authors proposed a weakly-supervised method to localize video moments given text queries.  The model builds multi-level relational graphs among pairs of word and video frame, and the graph is used to aggregate visual-semantic feature for each word and each frame. Then the attentive features are used to localize the sentence query in videos by calculating the similarity of words and frames. In summary, the proposed weakly-supervised Moment Alignment Network (wMAN) utilizes a multi-level co-attention mechanism to learn richer multimodal representations for language based video retrieval..\n\nPros:\n1. Significant performance improvement on Didemo and Charades-STA datasets. The authors achieved very good performance on both dataset, even higher than some of the full-supervision methods, such as CTRL and MLVI.\n\nCons:\n1. The overall novelty of the proposed methods is limited. Essentially, the key points of the model is hierarchical visual semantic co-attention.,which is proposed originally in [Hierarchical Question-Image Co-Attention\nfor Visual Question Answering], although the original application is VQA in image domain. So in this way, the novelty is only marginal.\n2. Paper writing can be improved. Figure 2 shows the overall structure of the model, however, the caption doesn't explain all the notations in the figure, such as WCVG, and the equations. Additionally, the reference is very far away from Figure 2, which makes the whole paper hard to read.\n3. For evaluation part, one important ablation study is missing: the number of steps T for message passing. This eval is important, as it shows the necessity of using \"multi-level\" attention.\n\nMinor comments:\n1. Make the caption of Figure 2 self-explainable, e.g. the meaning of LSE.\n2. There is a \"word-conditioned\" visual graph network, why not the other way, \"frame-conditioned\" semantic graph net and iterate over it?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Skeukb_ncH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a model for text based video clip (video moments or text-to-clip) retrieval. The goal is to identify a video segment within a longer video that is most relevant to an input sentence. Authors propose a new model based on a weakly-supervised training approach. This model does not require explicit temporal annotations to align text and video, but it only needs as an input the full video and sentence pairs.\n\nKey aspects of the model are: i) A coattention step frame-by-word and word-by-frame that produces the basic embeddings of the model, which is enriched with positional information, and ii) A contextual step that aggregates contextual information from all the frames using graph propagation. Afterwards, they use a  LogSumExp pooling strategy to score similarity among the input sentence and video frame.  \n\nThe main contribution of the paper is incremental (specially respect to Mithun et al., 2019), I do not see a ground-breaking contribution. One of the main novelties with respect to previous text-to-clip models is the use of co-attention schemes at the level of words and frames. However, the idea of co-attention at different grain-levels have been proposed before. Actually, while the model makes an extensive use of frame-to-word encoding, it is not clear to me what is the role of the word-to-video representation in Eqs. 5 and 6. \n\nIn general, the paper is well written. The experimental evaluation is convincing. However, it is not clear why authors change the structure of the evaluation among the experiments. As an example, for the experiments in Charades-STA dataset, they include scores for different IOUs levels, but they do not repeat this for DiDeMo dataset. Similarly, for DiDeMo dataset, results in Table 3 are for the test set, while the ablation study in Table 4 is for the validation set. I will recommend to standardize the evaluations. \n\nAnother comment is that in several experiment best performance is obtained using just the FBW module, it will be interesting to further analyze why the contextual cues hurt performance in some cases, maybe at least a qualitative analysis. Also, in some part of the papers, authors state that the proposed model does better than strongly-supervised state-of-the-art methods on some metrics, looking all the reported tables, I do not think that this is the case. Authors show qualitative results about cases where the model perform well, it will be good to also analyze failure cases, actually, according to the final scores, there is still lot of cases that the model can't handle properly.\n\nI rate the paper as borderline, but there is not such a rating at ICLR 2020, so I will lean to weak reject.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "ByxSBaCJqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\nThis paper proposes a method for aligning an input text with the frames in a video that correspond to what the text describes in a weakly supervised way. The authors propose a combination of a “Frame-By-Word” (FBW) representation and a Word-Conditioned Visual Graph (WCVG). The proposed method outperforms the weakly supervised baseline presented in the paper in experiments by a large margin. In addition, it quantitatively performs close to previous strongly supervised methods.\n\n\nPros:\n+ New Word-Conditioned Visual Graph representation\n+ Outperforms weakly supervised baseline\n+ Ablation study of the moving parts\n+ Interesting use of positional embeddings for multi-modal learning\n\nWeaknesses / comments:\n- What is the processing speed of the method compared to the baseline?\nThe proposed method makes multiple comparisons while computing the attention weights over all words and frames. Does this cause the method to be slower than the baseline? If so, how much slower is it?\nAnswers to these questions can help readers to keep in mind the trade-off of the proposed method for achieving the accuracy presented in the paper.\n\n\n- Number of parameters comparison with baseline:\nDid the authors make sure to have similar number of model parameters for the baselines and the proposed method? Maybe I missed it, but I couldn’t see a mention of this anywhere. It would be useful to state this so that readers are sure that it’s not the number of parameters that is helping the method.\n\n\n- Assumption that sentences are only associated with its ground truth video:\nThe authors mention that they have the same assumption as Mithun et al., 2019. Can this assumption be detrimental if the dataset does not follow it? Say there are sentences in the dataset that could describe segments in multiple videos. Could this assumption lead to suboptimal representation learning / relationship learning for words / video frames?\n\n- Determining the size of the sliding window:\nFrom reading the paper, it looks like the sliding window used for computing the word / frame relationships has to be manually defined. This seems a bit suboptimal for the generalizability of this method. Do the authors have any comments on this?\n\n- Can this model be supervised? If so, how does it compare to the supervised baselines?\nThe authors point out that their weakly supervised method performs close to the strongly supervised previously proposed. This is a nice finding, however, have the authors try to answer the question of what would happen if the proposed model is supervised? Will the proposed model outperform the strongly supervised baselines? Or at least perform the same?\n\nConclusion:\nIn conclusion, the proposed method makes sense and it has been shown to empirically outperforms a previous weakly supervised baseline. The authors also provide an ablation study of the moving parts to show that the entire pipeline is important to achieve the highest performance in the hardest setting. It would be nice if the authors successfully answer / address the questions / concerns mentioned above in the rebuttal.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byx0x0jDcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a weakly-supervised wMAN model for moment localization in untrimmed videos. Only the video-level annotation is available for training, and the goal is retrieving the video segment described by the sentence. The proposed model explored to utilize better context information and captured the relation between video and sentence/word via graph neural networks. In particular, instead of modeling the context information between the sentence and each video frame, wMAN tried to learn the representation with multi-level and co-attention, which considers all possible pairs between the word and the frame. The proposed model was evaluated on two publicly-available dataset and achieved reasonable results.\n\nPros:\n- Weakly-supervised method for video moment localization is a reasonable and important direction.\n- wMAN explicitly utilized multi-level context information between the sentence and the video frame, and used the graph neural network and the message passing to model the representation. I think this is a reasonable direction.\n- wMAN is evaluated with two publicly available datasets, and is compared with state-of-the-art methods and other \"oracle\" baselines. The performance is impressive and could be a better baseline for the future work.\n\nCons:\n- wMAN model the relation for all possible pairs of the word and the video frame. However, if the video is quite long, say 10 minutes, 30 minutes, or even few hours, will the method still be efficient and effective?\n- When building the relation between the word and the frame, is there any emphasis on verb, some particular word, or self-learned attention? For some particular word, say \"people\" and \"cup\", won't it have strong connection with many frames? But for some of the words, say \"hold\" and \"sits\", could it play a more important role?\n- Followed by previous question, in the qualitative results, it seems the boundary parts of the predicted video segments are less accurate. Is it because some of the words case these false positive results? What do you think the reason is?\n- Experimental results: I suggest the author to provide more ablation analysis to the experiment section. For example, the full model of wMAN works better than FBW on R@1, but worse on R@5 and R@10. Is there a particular reason about this? PE seems to be important for wMAN, and the authors provides few sentences analysis about this, but I don't think I fully understand this part. Another problem is that there is only few qualitative results, and in both these two examples, predicted results cover the GT segments. Is this always the case for wMAN? Why? Some failure cases could also be very helpful.\n- Less technical comments: The paper writing is fine to me, but I don't like the typesetting. I suggest to put the model figure more close to the methodology section and the qualitative results on page 8.\n\nOverall, I think the paper is marginal above the accept line.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for aligning an input text with the frames in a video that correspond to what the text describes in a weakly supervised way. The main technical contribution of the paper is the use of co-attention at different abstraction levels.

Among the four reviewers, one reviewer advocates for the paper while the others find this paper to be a borderline reject paper. Reviewer3 who was initially positive about the paper, during the discussion period, expressed that he/she wants to downgrade his/her rating to weak reject after reading the other reviewers' comments and concerns. The main concern of the reviewers is that the contribution of the paper incremental, particularly since the idea of co-attention has been used in many different area in other context. The authors responded to this in the rebuttal that the proposed approach incorporate different components such as Positional Encodings and is different from prior work, and that they experimentally perform superior compared to other co-attention usages such as LCGN. Although the AC understands the authors response, the majority of the reviewers are still not fully convinced about the contribution and their opinion stay opposed to the paper.
Decision: Reject