Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Overview: 
The authors proposed a weakly-supervised method to localize video moments given text queries.  The model builds multi-level relational graphs among pairs of word and video frame, and the graph is used to aggregate visual-semantic feature for each word and each frame. Then the attentive features are used to localize the sentence query in videos by calculating the similarity of words and frames. In summary, the proposed weakly-supervised Moment Alignment Network (wMAN) utilizes a multi-level co-attention mechanism to learn richer multimodal representations for language based video retrieval..

Pros:
1. Significant performance improvement on Didemo and Charades-STA datasets. The authors achieved very good performance on both dataset, even higher than some of the full-supervision methods, such as CTRL and MLVI.

Cons:
1. The overall novelty of the proposed methods is limited. Essentially, the key points of the model is hierarchical visual semantic co-attention.,which is proposed originally in [Hierarchical Question-Image Co-Attention
for Visual Question Answering], although the original application is VQA in image domain. So in this way, the novelty is only marginal.
2. Paper writing can be improved. Figure 2 shows the overall structure of the model, however, the caption doesn't explain all the notations in the figure, such as WCVG, and the equations. Additionally, the reference is very far away from Figure 2, which makes the whole paper hard to read.
3. For evaluation part, one important ablation study is missing: the number of steps T for message passing. This eval is important, as it shows the necessity of using "multi-level" attention.

Minor comments:
1. Make the caption of Figure 2 self-explainable, e.g. the meaning of LSE.
2. There is a "word-conditioned" visual graph network, why not the other way, "frame-conditioned" semantic graph net and iterate over it?

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This work presents a model for text based video clip (video moments or text-to-clip) retrieval. The goal is to identify a video segment within a longer video that is most relevant to an input sentence. Authors propose a new model based on a weakly-supervised training approach. This model does not require explicit temporal annotations to align text and video, but it only needs as an input the full video and sentence pairs.

Key aspects of the model are: i) A coattention step frame-by-word and word-by-frame that produces the basic embeddings of the model, which is enriched with positional information, and ii) A contextual step that aggregates contextual information from all the frames using graph propagation. Afterwards, they use a  LogSumExp pooling strategy to score similarity among the input sentence and video frame.  

The main contribution of the paper is incremental (specially respect to Mithun et al., 2019), I do not see a ground-breaking contribution. One of the main novelties with respect to previous text-to-clip models is the use of co-attention schemes at the level of words and frames. However, the idea of co-attention at different grain-levels have been proposed before. Actually, while the model makes an extensive use of frame-to-word encoding, it is not clear to me what is the role of the word-to-video representation in Eqs. 5 and 6. 

In general, the paper is well written. The experimental evaluation is convincing. However, it is not clear why authors change the structure of the evaluation among the experiments. As an example, for the experiments in Charades-STA dataset, they include scores for different IOUs levels, but they do not repeat this for DiDeMo dataset. Similarly, for DiDeMo dataset, results in Table 3 are for the test set, while the ablation study in Table 4 is for the validation set. I will recommend to standardize the evaluations. 

Another comment is that in several experiment best performance is obtained using just the FBW module, it will be interesting to further analyze why the contextual cues hurt performance in some cases, maybe at least a qualitative analysis. Also, in some part of the papers, authors state that the proposed model does better than strongly-supervised state-of-the-art methods on some metrics, looking all the reported tables, I do not think that this is the case. Authors show qualitative results about cases where the model perform well, it will be good to also analyze failure cases, actually, according to the final scores, there is still lot of cases that the model can't handle properly.

I rate the paper as borderline, but there is not such a rating at ICLR 2020, so I will lean to weak reject.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposed a weakly-supervised wMAN model for moment localization in untrimmed videos. Only the video-level annotation is available for training, and the goal is retrieving the video segment described by the sentence. The proposed model explored to utilize better context information and captured the relation between video and sentence/word via graph neural networks. In particular, instead of modeling the context information between the sentence and each video frame, wMAN tried to learn the representation with multi-level and co-attention, which considers all possible pairs between the word and the frame. The proposed model was evaluated on two publicly-available dataset and achieved reasonable results.

Pros:
- Weakly-supervised method for video moment localization is a reasonable and important direction.
- wMAN explicitly utilized multi-level context information between the sentence and the video frame, and used the graph neural network and the message passing to model the representation. I think this is a reasonable direction.
- wMAN is evaluated with two publicly available datasets, and is compared with state-of-the-art methods and other "oracle" baselines. The performance is impressive and could be a better baseline for the future work.

Cons:
- wMAN model the relation for all possible pairs of the word and the video frame. However, if the video is quite long, say 10 minutes, 30 minutes, or even few hours, will the method still be efficient and effective?
- When building the relation between the word and the frame, is there any emphasis on verb, some particular word, or self-learned attention? For some particular word, say "people" and "cup", won't it have strong connection with many frames? But for some of the words, say "hold" and "sits", could it play a more important role?
- Followed by previous question, in the qualitative results, it seems the boundary parts of the predicted video segments are less accurate. Is it because some of the words case these false positive results? What do you think the reason is?
- Experimental results: I suggest the author to provide more ablation analysis to the experiment section. For example, the full model of wMAN works better than FBW on R@1, but worse on R@5 and R@10. Is there a particular reason about this? PE seems to be important for wMAN, and the authors provides few sentences analysis about this, but I don't think I fully understand this part. Another problem is that there is only few qualitative results, and in both these two examples, predicted results cover the GT segments. Is this always the case for wMAN? Why? Some failure cases could also be very helpful.
- Less technical comments: The paper writing is fine to me, but I don't like the typesetting. I suggest to put the model figure more close to the methodology section and the qualitative results on page 8.

Overall, I think the paper is marginal above the accept line.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: 
Summary:
This paper proposes a method for aligning an input text with the frames in a video that correspond to what the text describes in a weakly supervised way. The authors propose a combination of a “Frame-By-Word” (FBW) representation and a Word-Conditioned Visual Graph (WCVG). The proposed method outperforms the weakly supervised baseline presented in the paper in experiments by a large margin. In addition, it quantitatively performs close to previous strongly supervised methods.


Pros:
+ New Word-Conditioned Visual Graph representation
+ Outperforms weakly supervised baseline
+ Ablation study of the moving parts
+ Interesting use of positional embeddings for multi-modal learning

Weaknesses / comments:
- What is the processing speed of the method compared to the baseline?
The proposed method makes multiple comparisons while computing the attention weights over all words and frames. Does this cause the method to be slower than the baseline? If so, how much slower is it?
Answers to these questions can help readers to keep in mind the trade-off of the proposed method for achieving the accuracy presented in the paper.


- Number of parameters comparison with baseline:
Did the authors make sure to have similar number of model parameters for the baselines and the proposed method? Maybe I missed it, but I couldn’t see a mention of this anywhere. It would be useful to state this so that readers are sure that it’s not the number of parameters that is helping the method.


- Assumption that sentences are only associated with its ground truth video:
The authors mention that they have the same assumption as Mithun et al., 2019. Can this assumption be detrimental if the dataset does not follow it? Say there are sentences in the dataset that could describe segments in multiple videos. Could this assumption lead to suboptimal representation learning / relationship learning for words / video frames?

- Determining the size of the sliding window:
From reading the paper, it looks like the sliding window used for computing the word / frame relationships has to be manually defined. This seems a bit suboptimal for the generalizability of this method. Do the authors have any comments on this?

- Can this model be supervised? If so, how does it compare to the supervised baselines?
The authors point out that their weakly supervised method performs close to the strongly supervised previously proposed. This is a nice finding, however, have the authors try to answer the question of what would happen if the proposed model is supervised? Will the proposed model outperform the strongly supervised baselines? Or at least perform the same?

Conclusion:
In conclusion, the proposed method makes sense and it has been shown to empirically outperforms a previous weakly supervised baseline. The authors also provide an ablation study of the moving parts to show that the entire pipeline is important to achieve the highest performance in the hardest setting. It would be nice if the authors successfully answer / address the questions / concerns mentioned above in the rebuttal.
review_title: Official Blind Review #1

Meta-review: This paper proposes a method for aligning an input text with the frames in a video that correspond to what the text describes in a weakly supervised way. The main technical contribution of the paper is the use of co-attention at different abstraction levels.

Among the four reviewers, one reviewer advocates for the paper while the others find this paper to be a borderline reject paper. Reviewer3 who was initially positive about the paper, during the discussion period, expressed that he/she wants to downgrade his/her rating to weak reject after reading the other reviewers' comments and concerns. The main concern of the reviewers is that the contribution of the paper incremental, particularly since the idea of co-attention has been used in many different area in other context. The authors responded to this in the rebuttal that the proposed approach incorporate different components such as Positional Encodings and is different from prior work, and that they experimentally perform superior compared to other co-attention usages such as LCGN. Although the AC understands the authors response, the majority of the reviewers are still not fully convinced about the contribution and their opinion stay opposed to the paper.
Decision: Reject