Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a novel approach, Siamese Sampling and Reasoning (SiaSamRea), to give self-taught guidance information as soft-labels to train the models via sampling several similar short-clips within the same video for the task of video QA. The authors provide experimental results to validate the effectiveness with new state-of-the-art performances on five benchmark datasets: MSRVTT-QA, MSVD-QA, How2QA, ActivityNet-QA and TGIF-QA.
main_review: The task of video QA is one of important and interesting problems in visual-linguistic AI. Since computational cost can be large for video understanding to handle arbitrary questions related to the video contents, it is inevitable to take certain sampling strategies. For example, one of recent works, ClipBERT, provides good approaches with sparse clip sampling. On the other hand, the proposed approach utilizes more sample clips that are similar but not same. With the aggregated features of the samples, the authors propose the methods to improve the models without supervision. The inductive bias of this work seems that “clips with high visual similarities are expected to have more consistent predictions on their class probabilities.” It can be achieved via Siamese sampling, sampling more short-clips near the anchor clip. From the assumption, the experimental results seem to show new state-of-the-art performances on some benchmark datatsets such as MSRVTT-QA, MSVD-QA, How2QA and ActivityNet-QA.

However, this paper has some weak points. 
Firstly, the proposed methods seem to be based on ClipBERT [22] mostly. Following the manuscript, most parts of the architectures come from ClipBERT including video encoder, text encoder, and multi-modal transformer as a model of initial weights. I think the result on ClipBERT should be reported on all of the tested benchmark datasets. But, the results on MSRVTT-QA only are reported in Table 3. In this case, it is not clearly validated that the proposed method gives positive impacts. Moreover, the result on ClipBERT on TGIF-QA was reported in [22], and they are better than the result of the proposed approach.
Secondly, it is not clear that all of experiments performed fairly on all of comparative methods. The models in this paper are pretrained with TGIF-QA dataset from ClipBERTs as initial models, and the fine-tuning is done for the downstream tasks. On the other hands, it seems that the other methods were trained with their datasets and the strategies. If it is not easy to be performed properly, it would be better to clarify their experimental settings briefly in the manuscript.
Thirdly, it would be more persuasive to survey self-supervised learning methods in related work section.

Personally, I think the term Siamese is not good choice on this concept. It might lead to confusion.
As a suggestion, it would be better to clearly formulate and explain the proposed methods if finding and utilizing analogous machine learning frameworks, e.g., multi-view learning [1*,2*]. Probably, it will help the authors figure out pros and cons, the potentiality and the limitation, and find better directions to improve.



* Minor
   -	line 139, spare -> sparse


[1*] H. Su et al., Multi-view Convolutional Neural Networks for 3D Shape Recognition, ICCV 2015.

[2*] B. Xiong et al., Multiview Pseudo-Labeling for Semi-supervised Learning from Video, Technical Report, https://arxiv.org/abs/2104.00682.

----------------------------
### Post Rebuttal

I have read all of other reviews and the authors' response. 
The rebuttal resolves most of my concerns. 
However, I think that there is still remained necessity to improve the manuscript in presentation, discussion and reproducibility.
I keep my score as it was.
limitations_and_societal_impact: There is no limitations and societal impact in the manuscript.
How about mentioning the potential risks of the proposed video QA systems being trained with social biases on image data from the public or the industry?
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a Siamese Sampling and Reasoning network (SiaSamRea) to model the interdependent semantic across clips in a video. The contribution of the paper is to employ Siamese sampling and calculate the similarities between the multimodal features of clip-text pairs. The Siamese knowledge is then used to generate soft labels which guide the learning process. Experimental results on five Video QA datasets showed the effectiveness of the proposed method against existing works.  
main_review: Strengths:

(1) I like the idea of modeling the interdependent relationship of internal clips. The proposed method is well-motivated and straightforward for implementation.

(2) The proposed method shows improvements across multiple Video QA datasets

Weaknesses:

(1) Although the idea is interesting, the current implementation appears to be an extended version of ClipBERT with Siamese network.

(2) There are several missing details about the model that the authors should explain more clearly:

•	How many anchor clips are used in the proposed model? How does the number of anchor clips affect the performance?
•	In Eq. 5, what is the soft learning target Q used for? 

(3) Experiments:
- Why didn’t the authors report the performance of ClipBERT on TGIF-QA dataset in Table 5. As per results from ClipBERT paper, it outperforms the proposed method on most tasks, as result for the Count task is not available. It is important to compare the proposed method with ClipBERT on the TGIF-QA as it is the most prominent dataset for Video QA. 

- As SiaSamRea shares some common components with ClipBERT, please explain the performance degrade on the TGIF-QA against ClipBERT.  

- The examples in Figure 4 do not show the interdependent relationship between internal clips in the video. Please provide more diverse showcases.

- Ablation studies are not sufficient. Please justify your choices of hyperparameters such as the number anchor clips, the number frames in each clip, the value of the balance term, and the integrating weight.

 - One of the downsides of the Siamese networks is that it is quadratic in time. Please provide more analysis on the memory usage and computation cost of the proposed model.

Minor comment: Page 6, Line 190: multi-label classification -> multi-class classification

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors propose a network architecture to improve the Video QA task. The model learns and aggregates knowledge from siamese (nearby) frames and develop a reasoning strategy for integrating the inter-dependent knowledge between these contextual clips to make the final prediction. Their two modules, first, the siamese knowledge generation first learns inter-relationships among clips (anchor and siamese) and the second module, siamese knowledge reasoning that produces soft labels for self-supervised refinement during the training. They perform extensive experiments on five Video QA datasets to show the ability of their proposed SiaSamRea on both open ended QA and Multiple Choice QA and improve over the state-of-the-art models. 

main_review: The proposed method, Siamese Sampling and Reasoning (SiaSamRea), has three main components. First, the sparse sampling strategy to get sparse frames and the siamese sampling strategy to get the siamese frames. Then, these frames are encoded using video encoders and a multimodal transformer generates multimodal features which are used to produce contextual relationships between the anchor frames and the siamese frames from the same video. Finally, the knowledge reasoning module generates soft labels based on the similarities between frames acquired in the generation module. The model is trained using cross entropy loss or the multi-margin ranking loss based on the task at hand. The proposed method is novel and the paper is well written. 

1. Are the weights for the Video Encode /Text Encoder and Multimodal transformer all initialized using the ClipBert model and then fine-tuned on the respective datasets ? It is good to mention what these encoders are eg. resnet 50, 12 layer transformer etc. 

2. In section 3.3.1, the matrix A in line 161 is NxN. Is N the total number of anchor and siamese samples in the full video ? f_anchor_1, f_siam_11, f_siam_12, f_siam_13, f_anchor_2, f_siam_21, f_siam_22, f_siam_23 ……. And so on? 

3. Did the authors try using the anchor as the center frame and having siamese frames near to both sides of the center frame? 
Currently, from the figure and description it seems the siamese frames are samples only at (t_anchor + t_s) locations. 

4. During training, as there are more clips/frames for each video sample, how much does it increase the training time and memory usage compared to, for eg.  ClipBERT? 

5. Will your method underperform/overperform if attention over frames is done in dense sampling or following methods similar to Long term feature banks [1]?

[1] Wu, Chao-Yuan, et al. "Long-term feature banks for detailed video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.


##Post Rebuttal

I have read the authors' response and they address my concerns. I maintain my original rating.
limitations_and_societal_impact: The authors have not discussed any limitations or the social impact of their method or the domain of VideoQA in general. There are some things that could be important to consider while designing models for these datasets and problems. Are the datasets biased towards some specific answers given the questions (for eg, if there’s perching in the question, would the answer always be bird?). This is an important thing to consider if the training and test distributions are different and to reduce bias in learning. Is the model looking at the image to answer these questions or just overfitting to the textual bias? Also, due to pre-training on large scale datasets which are also potentially biased (racial, gender etc.), this might get propagated in the downstream task training and  hence not suitable if such models are to be deployed in real settings. The authors should consider these limitations and discuss them in the paper.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper addresses VideoQA and proposes a new approach: "Siamese Sampling and Reasoning". The method samples multiple sparse clips per video, an anchor clip, and several siamese clips with the same length but different starting frames. Then, pairwise similarities between these clips are computed and used to produce soft learning targets: Intuitively, all clips should produce similar predictions of the answer class. This self-supervised cross-entropy loss is combined with standard cross-entropy between predictions of the anchor clip and the ground truth labels. The method is evaluated on 5 VideoQA benchmarks and shows significant improvements.
main_review: ## Strength

- The method is novel in that it extends ClipBERTs sampling strategy of using several sparse clips with a new loss and new modules over ClipBERTs simple averaging strategy.

- Related work has been analyzed and described thoroughly.

- Strong results in combination with ablations suggest that the proposed method could be helpful for video understanding tasks.


## Weaknesses

### Major points:

1- **Confusing description**: 
Authors claim to learn "inter-relationship among clips" (line 12) and state that other methods "consider each clip-text pair separately" (line 36), "each clip-text pair is independently encoded" (line 107). This sounds to me like they would combine several clip-text data points in some way and integrate knowledge between data points. However, the method samples multiple combinations of frames for the same data point, therefore I find these main claims confusing. E.g. ClipBERT simply states in its abstract that "a single or a few sparsely sampled short clips from a video are used at each training step" which I find to be a much more fitting description.

2- **Effect of pre-training**: Authors load pre-trained weights from ClipBERT and then pre-train again on TGIF-QA dataset. They claim that this has advantages (line 335), however, there is no ablation to support this claim: It would be helpful to see the results on MSRVTT-QA and MSVD-QA without pre-training on TGIF-QA dataset.

3- **No comparisons to ClipBERT**: 
- ClipBERT results are missing from Table 5 (Performance on TGIF-QA), while they are added in Table 3 (Performance on MSRVTT-QA and MSVD-QA) and no reason is given on why these results are missing. This is important as ClipBERT reports better performance on TGIF-QA than the paper and has a very similar setup (same model, no more than 1 clip during testing, see ClipBERT paper table 8 second-last row.)

- The ablations in Table 1 should be additionally done either on TGIF-QA or on MSRVTT-QA and compared to the ClipBERT result described above.

4- **Hyper-parameters**: Authors use dataset-specific hyperparameters tuning (line 276) which could be another reason for some of the improvements over the ClipBERT method. What was the hyper-parameter tuning approach?

5- **Table 1**: This table reports the ablation study on the effect of SKG and SKR components. However, there is no result with SKG alone. Also, the authors don't report the mean and std of multiple experiments. Would be great if the authors report mean/std and also results with only SKG. 


*Given the above concerns, it seems that the main improvement in this paper comes from taking ClipBERT and pretraining it on TGIF-QA before applying it to other datasets. This is contradicting to the story which reasons that the improvements are due to the proposed new modules.*



### Minor points:

1- Motivation is missing as to why the method is evaluated exclusively on VideoQA and not on other tasks like e.g. Video Classification. 

2- There are no details on pre-training and fine-tuning time complexity.

3- Architecture details are missing: "multimodal transformer is applied to produce the clip-text feature" (line 133). I would assume the output is a single feature vector obtained from the CLS token, however, this should be specified directly. Otherwise, it would be very hard to reproduce the paper.

**Typos**:
The paper is difficult to follow. The structure of the paper needs to be improved and also there are lots of typos and grammar errors:
- 78 "Not" should be "Note"
- -Figure 3 sentence 1 "to be extracted" sentence is missing a proper verb.
- 129 "all clips number" is not a proper term, should be "number of all clips".
- 129: "Each clip ... is uniformly sampled ... frames" should be rephrased to "frames are sampled per clip" or similar.
- 136 "dimension" should be "dimensions"
- 144 "including" is confusing here. Does this mean: siamese clips have semantics that are similar "to" the anchor clip?
- 153: "predictions on their predicted class probabilities": Duplicate "predict" is incorrect
- 225, 278: "excavate" means physically digging (e.g. with a shovel), would be better to replace it with "use" here. 
- 364 "Boarder" should be "Broader"
- 375 "look" should be "looking"
limitations_and_societal_impact: No
review_title: null

Meta-review: After the authors’ rebuttal, intensive discussion followed between reviewers, and finally the reviewers’ consensus has reached to the reject opinion for this submission at this time. All agree that the main idea of this work is interesting, and they do not want to be strongly against this work. However, the reviewers also acknowledged that there is much room for improvement to be published. For example, the paper needs more clarifications and details about where the improvements come from, ClipBERT should be compared in different scenarios, and better benchmark datasets may be necessary. 
Decision: Accept (Poster)