Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aggregates (a) contrastive learning, (b) background mixing and (c) target pseudo-labels on video domain adaptation and achieves excellent performance (outperforming SOTA). The paper is written well and easy to follow.
main_review: The paper has solid results and ablations on several key datasets in video domain adaptation. The paper reads well.

However, the novelty is limited. Using contrastive learning to leverage labeled and unlabeled data (UDA is a special case) is not new. Sampling at different speeds and background mixing can be considered as data augmentation (SimCLR already showed that data augmentation is crucial for contrastive learning). In addition, using pseudo-labels in the target domain is not new either.

The idea of background mix is interesting, but it is not clear WHY it is more effective than Mixup or CutMix? Is it more effective in video domain adaptation? or it is a general approach for video recognition tasks.

Good performance is crucial for a paper, but I would like to see more insightful understanding in a NeurIPS paper. e.g. why sampling at different speeds is optimal for video domain adaptation, or it is just one of many working sampling strategies? is it dataset specific? why background mix is crucial for video domain adaptation? does background encode shortcut in the source domain?
limitations_and_societal_impact: Please see limitations in the main review

Minor comments:
Figure 1 is too small to see how background is mixed

Figure 2 and 3 are not easy to follow.

Figure 5: please add more explanation on "become progressively discriminative".
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a video-based contrastive learning framework by maximizing the similarity between encoded representations of the same video at two different speeds as well as minimizing the similarity between different videos played at different speeds. Results on UCF-HMDB, Jester, and Epic-Kitchens validate its effectiveness. 
main_review: [Merits]

1. No adversarial loss is needed, which helps to stabilize the training.  

2. Formulating background mixing into the CL framework is novel. 

3. Results on UCF-HMDB, Jester, and Epic-Kitchens as shown in Table 1-5 are good.

4. Code is provided. 


[Improvement Suggestions]

1. How the background information is extracted?  Why In Line 65 "Importantly, since mixing background doesn’t change the temporal dynamics". Why? e.g. when the synthetic videos are generated via background mixing, what if it becomes false positive when the background information is added to the original frame?

2. This is an application of contrastive learning (Formula 3) with mixup (Formula 2) and pseudo labeling (Formula 4) in the scope of UDA for action recognition. Except for the second point (BG part in Formula 3) mentioned above, all components have been proposed in the previous work. There is no specific new method in terms of hard negative selection for the background. Could the author[s] justify the novelty of the proposed method precisely? 

3. How long does it take for the SSL convergence? If it is stable across multiple seeds? Why only a single number is reported for all experiments? Table numbers look good, but the current form seems to lack ablation illustrations. 

4. Could author[s] add discussions or results with the baseline mixup UDA method https://arxiv.org/pdf/2001.00677.pdf?
limitations_and_societal_impact: This work has no potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new contrastive method for unsupervised video domain adaptation. This method employs temporal contrastive learning based on graph convolution and back ground mixing approach for enhancing action similarity, so called CoMix. 
Additionally, for tagert domain, CoMix uses a pseudo label-based target domain contrastive learning. The authors evaluate CoMix on UFC-HMDB, Epic-Kitchen, and Jester datasets with intensive comparison and ablation studies. 
main_review: ### Strength
- Unsupervised video domain adaption is a challenging but significant research topic. 
- Leveraging contrastive learning, graph representation for temporal relations, and background mixing in video domain adaption is an interesting idea and seems novel (considering NeurIPS deadline). 
- It is well-written, well-organized, and easy to read.
- Including ablation study, experiments seem thorough except for the results on Epic-Kitchen dataset.
- Source code was submitted.

### Weakness
- Some important previous references were missed in related work such as 
  - For domain adaption: MCD [Saito et al. 2018]
  - For Image Mixture: PuzzleMix [Kim et al. 2020] and CoMixup [Kim et al. 2021]
- Also, the comparison with MM-SADA [Munro and Damen 2020] was missed even if it was referred to in the paper as [51]. In particular, the comparison is important because MM-SADA proposes Epic-Kitchen dataset and MM-SADA with RGB only not using optical flow shows comparable or better performances than the proposed CoMix. This should be clarified.
- The authors employs graph convolution for temporal contrastive learning. In this study, because the number of clips are not large (=16), the fully connected graph is not a problem. However, when the number of clips increases (for longer clip sequence), the connections also exponentially increases, which might restrict scale-up of this method. 
-  For background mixing, the authors uses a very simple image processing method. How about using more advanced pretrained foreground-background decomposition method? How large can the background mixing method influence on the performance?
- How sensitive are the results by target pseudo-label confidence threshold? Any results?
- For GCN ablation, did the author try to use MLP, LSTM, or Transformer instead of GCN? Because removing GCN makes the total parameter reduced, other NN can be helpful to validate the effect of GCN.  
- Minor comments
  - L276 and L308 in P7, SAVA was wrongly cited. [10] --> [12]
  - Overall, there is some redundancy in Introduction, Proposed Method and captions in Figures 2 and 3.
  - Figure 3 can be more detailed. If the page limitation is an issue, More detailed figure can be located at supplementary material. 
  - Instead of naive video cutmix, VideoMix [Yun et al. 2020] can be used. 
  - [Song et al. 2021] can be added in related work although it is not published until the NeurIPS deadline. 

#### References
- [Saito et al. 2018] Maximum classifier discrepancy for unsupervised domain adaptation. CVPR 2018.
- [Kim et al. 2020] Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup. ICML 2020.
- [Kim et al. 2021] Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity. ICLR 2021.
- [Yun et al. 2020] VideoMix: Rethinking Data Augmentation for Video Classification. arXiv 2020. 
- [Song et al. 2021] Spatio-temporal Contrastive Domain Adaptation for Action Recognition. CVPR 2021.


=========================== After rebuttal =======================
I read the other reviewers' comments and author response. Beacuse the authors addressed most of my concerns, I decided to raise my score. 

limitations_and_societal_impact: #### No negative societal impact
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors tackle the problem of video domain adaptation for the task of action recognition. To align source and target domain features, they propose to use contrastive learning instead of domain adversarial learning. For contrastive learning, two instances of different speeds from the same video constitute a positive pair. Two instances of different speed from the different video constitute a negative pair. In addition to the contrastive learning, they propose the background mixup in order to bridge the domain gap between source background and target background. They also incorporate target pseudo-labels to learn more discriminative target classifier. They validate the efficacy of the model on the publicly available benchmarks: UCF-HMDB, Jester and Epic-Kitchens.

The claimed contribution of the paper are as follows.
 1) Introducing a new UDA framework, Contrast and Mix (CoMix): contrastive learning with speed, and background mixup for bridging the domain gap 
2) Integrate the pseudo labeling for UDA 
3) Extensive experiments on three datasets, and achieving state-of-the-art performance on UCF-HMDB and Jester.

main_review: This paper has the following strengths.

+ The problem addressed, unsupervised domain adaptation for video recognition, is an interesting, but unsolved problem.
+ The proposed method has a good motivation and sensible design choices. Speed is an almost invariant feature for recognizing actions in many cases (except a few examples such as walk vs. jog/run, slow dance vs. fast dance). Background shift between source videos and target videos is a known problem. Pseudo-labeling is shown to be helpful for the UDA in image domain [1].
+ The paper is structured well. It is easy for readers to follow except for a few confusing sentences.

[1] Unsupervised Domain Adaptation for Distance Metric Learning, Sohn et al., ICLR 2019.


However, this paper has weaknesses as well.
- The comparison with adversarial domain adaptation methods (ADDA, DANN, TA$^3$N, SAVA or even the source only baseline) is unfair in Table 1. CoMix is based on the video-level representations from GCN, while the compared methods are not using GCN representations. The authors claim the proposed method is more effective than domain adversarial methods based on this result. However, we do not know where the improvement comes from. Does it come from GCN or CoMix (contrastive learning with background mixup)? The ablation on the GCN shows CoMix w/o GCN gives 88.1% on the UCF-HMDB which is close to SAVA (86.7%, note that SAVA does not have pseudo-labeling) than CoMix with GCN. Therefore, it is not straightforward to draw conclusion that CoMix is better than adversarial DA methods. It would be better to see the results from the source only, DANN and ADDA baselines with GCN equipped. 
- There is no comparison with MM-SADA [51] on the Epic-Kitchens dataset. MM-SADA is a multi-modal method. Therefore, it is unfair to compare CoMix (43.2% on avg) with RGB+Flow version of MM-SADA (50.3% on avg). However, comparison with RGB only MM-SADA (43.9% on avg) should be there. RGB only MM-SADA (43.9%) shows slightly higher performance than CoMix (43.2%). This result might suggest adversarial learning is still valuable for the domain adaptation problem.
- All ablation experiments are conducted on UCF-HMDB which is a quite small (only 3K videos with 12 classes), and performance-saturated dataset. It is hard to draw concrete conclusions from UCF-HMDB experiments. It is better to conduct ablation experiments on the Epic-Kitchens or Jester dataset which are larger and less performance-saturated.

If the majority of my concerns are resolved by the rebuttal, I will consider increasing my rating.

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The initial reviews raised several concerns about novelty, missing experiments, and analyses that provide insights. The rebuttal addressed most of the concerns; some reviewers raised their ratings after the rebuttal. Overall, the paper shows how to combine existing ideas to tackle a challenging problem in a simple and effective manner. The reviews are generally positive, mainly due to the good empirical performance, but they are not overwhelmingly enthusiastic about the technical novelty. I carefully read all the reviews, rebuttal, and discussion, and also read the paper in detail. I tend to agree that the paper lacks novelty but must say the proposed approach is simple, effective, and performs well. I am recommending an acceptance, but wouldn't mind if it is rejected.
Decision: Accept (Poster)