Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper tackles unsupervised domain adaptation for the video recognition task. The paper raises concerns with the use of contrastive methods for aligning source and target domain features. Those concerns involve intra-domain positives, false positives in a cross-domain matches that hinder the contrastive learning process and negatively affect the adaptation. The proposed solution is to utilize intra-domain positives and mine robust cross-domain positives that reduces likelihood of pseudo-label noise. The proposed solution is shown to be superior than the state-of-the-art methods, by experiments on several cross-domain action recognition benchmarks.
strength_and_weaknesses: **Strengths:**
- The paper raises some interesting points on the way state-of-the art models are trained for domain adaptive action recognition in a contrastive learning. The paper suggests that impurity of the pseudo-label and cross-domain FP/FN hinder contrastive learning, that results in a sub-optimal domain alignment.
- The proposed solution moves away from the contrastive learning and modifies it to include intra-domain positives and reduce reliance on negative samples. Furthermore, rather than following existing strategy of creating cross-domain positives, a new strategy is proposed that synthesizes positives for a source-domain anchor that likely reduces the pseudo-label noise.
- There are more minor observations for improvements such as value of MLP in contrastive learning for intra-domain and cross-domain optimization.
- Experimental analysis is exhaustive, includes state-of-the art methods for comparison and shows reasonable improvements with proposed strategy.
- Ablation studies also provide interesting insights into the individual component of the method and effects of hyper-parameters used on the final performance.


**Concerns:**
- One of the aspect of the method includes the use of fitting a Gaussian distribution on the target domain features, which is more or less similar to the strategy used here [1]. I would like to know more about how proposed method is different/better than [1] both intuitively and empirically. This would provide more information on the overall contribution of the work.
- The proposed solution revolves around reducing the FN/FP influence of the cross-domain alignment, but this intuition is never verified directly empirically. It would be beneficial to have FN/FP numbers before and after the use of proposed approach to further solidify the points raised in the introduction.
- Similarly, there should be an empirical analysis to show how the proposed synthesis strategy is reducing pseudo-label noise. The paper claims that such synthesis will weakens the noise, but is never empirically verified.

**Typos:**
- "robust intra-domain and cross-domain positives without *replying* on negative samples
for video DA problem." --> relying

**References:**
[1] Ding, Ning, et al. "Source-Free Domain Adaptation via Distribution Estimation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.



clarity,_quality,_novelty_and_reproducibility: Overall the paper has explained the proposed idea with good clarity and performed extensive evaluation to show efficacy of the proposed approach.
summary_of_the_review: Overall, the paper provides good contribution to the domain adaptation for the task of video action recognition. There are some concerns raised in the review, addressing them should provide more clarity on the final decision.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work is the first to strengthen that limited variations in intra- domain positives, pseudo-label noise in inter-domain positives, and false negatives are the three under-explored key problems. Motivated by these concerns, this paper proposes a unified solution by introducing more informative and robust intra-domain and cross-domain positives without relying on negative samples for video DA. This model achieves state-of-the-art performance on several challenging cross-domain benchmarks for video DA. 
strength_and_weaknesses: Strength:
- This paper has several interesting findings like "MLP head is crucial for intra-domain positives optimization but largely hampers the convergence for cross-domain positives". The proposed method is with adequate insights from these findings. 

- This work introduces the bottlenecks of existing contrastive-based video DA methods.

Weakness:
- It seems that this work can be applied to existing video da models. I suggest the author should do further ablation study to testify whether this unified model can be applied to other video da models like TA3N. 
- This work needs more ablation studies to support the proposed findings. Are these findings similar in different video DA benchmarks? 
- This work only conducts experiments on two small cross-domain datasets UCF-HMDB and Kitchens. These two benchmarks only have a few classes. Validating the method in large-scale cross-domain datasets with more classes will make your conclusions more convincing. 
clarity,_quality,_novelty_and_reproducibility: The reviewer agrees that the work tackles an essential problem for video domain adaptation. However, several parts need further clarification by the authors to conclude the contributions of the work.
summary_of_the_review: This paper has proposed a novel contrastive-based unified video DA method without relying on negatives by mining informative and robust intra-domain positives and cross-domain positives.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper points out the existing limitations of current contrastive-based VDA methods, including limited variations of positives within the same domain, inaccurate cross-domain positive pairs, and inevitable false negatives. To address the above limitations, the authors introduce target-domain nearest neighbors and synthetic features from the target domain based on class centroids to perform VUDA in a non-contrastive manner. The proposed method is well-motivated, and its experimental results demonstrate its performance.
strength_and_weaknesses: Strength:
This paper begins with an easy-to-follow analysis of what hinders existing contrastive-based methods, which motivates the following proposed methods. Overall, the proposed modifications are simple-yet-effective and well-motivated. The experimental results can further justify the effectiveness of the proposed methods.

Weakness:
Some details need to be further discussed. 
a)	Authors claim that “intra-domain positives provide the least performance gains” indicates "their limitation in variations", while this causation is not convincing purely based on the comparison in Figure 1 (a). One could argue that such inferior performance is simply due to other reasons, e.g., the lack of minimizing domain discrepancy compared to cross-domain methods. It would be better for authors to provide more justification for this statement. 
b)	While Figure 1 presents an intuitive analysis of the limitations of existing contrastive-based methods based on HMDB->UCF, I wonder whether these empirical observations also exist in other adaptation scenarios. More specifically, for Figure 1 (b), the higher purity of vanilla NN can be due to the fact that UCF is a relatively less challenging dataset (i.e., source only can already achieve 88.8). It is also known that multiple samples from UCF101 are clipped from the same raw video, which could also be the main reason why NN here is more accurate. Therefore, such observation on UCF101 may not work for other more challenging datasets. 
c)	In Table 2, authors present the performance of their method utilizing RGB and RGB+Flow for comparison, while only introducing RGB+Flow results from previous methods. To make a fair comparison, authors should also, from my perspective, include the RGB results from previous methods in Table 2. 
d)	Some minor formatting issues can be observed (e.g., Table 2 appears earlier than Table 1, and a citation error in Table 1 where MSTN^* Ganin et al. (2016) should be MSTN^* Xie et al. (2018)).
clarity,_quality,_novelty_and_reproducibility: Overall, the quality of this paper is good. The proposed methods are well-motivated in general, while part of the motivation requires a more detailed justification. The paper is well-written and easy to follow with certain novelty. Sufficient technical details are provided for future reproduction.
summary_of_the_review: This paper presents a simple-yet-effective method for VDA, which leverage some machine-learning techniques to address the limitations of previous methods. The paper is presented clearly with sufficient details and analysis, while part of the motivation requires further justifications. Based on the quality of this paper, I would recommend a “marginally above the acceptance threshold”.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper handles noisy pseudo-labeling in domain adaptation for action recognition. The paper shows that the previous contrastive learning-based methods can be noisy and improves the model performance with robust cross-domain positives. Experiments are conducted on two benchmarks.

strength_and_weaknesses: 
Strength
---
- The paper analyzes the noisy pseudo labeling for domain adaptation in action recognition and proposes a way to solve this with Gaussian data augmentations and target nearest neighbors.
- From a simple gaussian distribution, the method synthesizes the target features and uses these for cross-domain alignment.

Weakness
--- 
- The novelty is somewhat limited as the proposed components are borrowed from previous works (e.g., SimSiam). And the neighborhood clustering concept is also presented in [1], which is not referenced in this paper.
- Extensive comparison is needed. For example,  the benefit of synthesizing features is not very clear. How are the proposed cross-domain components compared to the previous cross-domain alignment methods? In addition, Instead of using the cross-domain alignment in this paper, how does the classical pseudo-labeling performs? How does the pseudo labeling in Sahoo et al. 2021 work compared to the cross-domain component in this paper? It is not clear that the proposed components are better than previous alignment methods.
-  As mentioned, the paper is missing some references for relevant works (e.g., feature generation [1], contrastive alignments for DA [2, 3, 4]).

[1] Li, Shuang, et al. "Transferable semantic augmentation for domain adaptation." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*
. 2021.
[2] Saito, Kuniaki, et al. "Universal domain adaptation through self supervision." *Advances in neural information processing systems*
 33 (2020): 16282-16292.
[3] Kim, Donghyun, et al. "Cds: Cross-domain self-supervised pre-training." *Proceedings of the IEEE/CVF International Conference on Computer Vision*
. 2021.
[4] Harary, Sivan, et al. "Unsupervised Domain Generalization by Learning a Bridge Across Domains." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*
. 2022.
clarity,_quality,_novelty_and_reproducibility: It is easy to read and understand. However, novelty is somewhat limited.
summary_of_the_review: While the method outperforms existing SOTA methods, the novelty is somewhat limited since the proposed components are from previous works. And more extensive experiments are needed.

-- POST REBUTTAL --
Thank the authors for the response. My concerns are partially addressed. I still believe that the idea of neighborhood clustering is not new to domain adaptation. Even though there is a difference from NC[2], the proposed clustering does not seem very novel. Other implementation details seem to be engineering optimization with limited novelty. However, this work is meaningful as it provides extensive experimental results and analysis, which can contribute to the research community. Therefore, I increase my score to 6
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed a solid engineering solution to video domain adaptation, a relatively under-explored problem compared to the image counterpart. The approach is built upon SimSiam but introduces several beneficial engineering improvements, based on exploring the informative and robust positives. The most important insights from this paper is that the video domain adaptation performance is hindered by "the limited variations in intra-domain positives, pseudo-label noise in cross-domain positives and false negatives". Through extensive evaluation with diverse baselines in video domain adaptation, domain alignment, self-training etc, as well as the extensive ablations of the proposed approach, the empirical benefits and claimed insights have been justified sufficiently.

The paper received mixed ratings. Reviewers pointed out essential issues which are with reasonable concerns in the AC's opinion. Authors made a great effort during the rebuttal, by providing detailed and relevant responses as well as a moderately revised version of the paper. After discussion (though after several rounds of expedition), two reviewers increased the overall score (5->6 and 6->8), while some other reviewer acknowledged the author response.

While the current version is acceptable, especially that the implementation details are included in the Appendix, AC personally has some further comments for the authors to address in a future version:

- The presentation is not nice enough, leaving with the impression of low-quality writing. For example, the citation format looks weird, in the form of Method Author (Year) throughout the paper. Authors shall read the guideline of ICLR and comply with the writing requirement. The correct form shall be Method (Author, Year) if the citation is not used as the subject of the sentence.

- Evaluation is only performed on UCF and HMDB, two obsolete datasets in the video domains. While I understand that in the DA community, previous work generally followed these two datasets, however, it is extremely easy to construct new datasets (even with identical source-target classes) from the big video datasets such as ActivityNet, Kinetics, and something-something. Since this is not done by the authors, we shall expect more sufficient evaluation on the UCF and HMDB. For example, as a video-oriented paper, we cannot see any showcases of video clips/snapshots regarding the insights of the algorithm, e.g. the discovered intra-domain positives, inter-domain positives, and etc.

- The quality of visualization in Fig. 3 is low-quality. We cannot observe any meaningful things from it. I suggest removing it directly. The vacated room can be used for a more in-depth case study. Fig. 1(b) is rasterized, why not use the vector format with higher quality. Fig. 1(a), text is overlapped with the bars...

- Discussion on Xie et al. (2018). This section shall be placed in the related work section or in the Appendix. Currently, it is placed in the body of the paper, making the presentation non-smooth.

- Necessary citations. For example, Eq. (9)--(12) seems to follow that in [Regularizing Deep Networks with Semantic Data Augmentation] or more older denoising auto-encoders.

Overall, authors have been very active in the rebuttal, by providing very detailed responses that overturn the reviewers' opinions. But as the AC, I consider the quality of the paper itself as the most crucial thing, and the current version is not satisfactory. Please make sure to improve and polish the paper significantly.
justification_for_why_not_higher_score: The whole approach is solid but more an engineering solution. The empirical benefit is sufficient for an ICLR poster, but the technical part is relatively weaker. In addition, the current presentation quality is not above the bar of ICLR and requires another revision.
justification_for_why_not_lower_score: Reviewers are in unanimous agreement of accepting the paper, especially after rebuttal and discussion. While I can find some flaws that need the authors to further address, they are not serious enough to result in a rejection.
Decision: Accept: poster