Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose to solve the problem of unsupervised model adaptation. In this task, the aim is that given a classifier trained on a source dataset and a set of unlabelled target examples, an unsupervised model adaptation is performed to improve the performance of the model on the target dataset without requiring access to source examples. The method is evaluated on standard domain adaptation datasets for classification, detection and segmentation. The approach is related to EM and some theoretical insights provided.

main_review: Main review
The main insight adopted in this paper is that as the source samples are not provided, the model embeddings for the target dataset for the initial model (before adaptation) could serve to provide embeddings that are closer to the source domain. Then using these embeddings in a contrastive learning setting, instance-wise and class-wise contrastive learning with these set of models is used to guide the model adaptation procedure. 

Pros:
1) The proposed approach is interesting
 2) It is thoroughly  validated with state of the art results for this relatively new task of model adaptation for classification, detection and segmentation. 
3) Thorough analysis is provided for the method along with theoretical insights by relating it to EM algorithm


Cons:

While the paper is well written, thoroughly evaluated and with good insights, it has a few issues. 

Several aspects of the paper are not very clear and require clarification
1) How are the negative and positive sets identified? One possibility is that for an instance, all historical embeddings for that particular instance form the positive and all other samples would be negative.
2)  If this is the case, negative samples would far outweigh positive samples. How is this imbalance handled.
3) Around line 148 it is mentioned: "For HisNCE, we assign each key a reliability score to encourage to memorize the well-learnt historical embeddings only.” How is this reliability score computed. Later on, for the categorical case, it is mentioned that the samples that are consistent in prediction across current and historical models are termed as well-learnt sample. Is this same criterion used for assigning a reliability score?
4) Similar issue with positive and negative sets for the HCCD learning is also present.

On the whole, the main issues with the paper relate to clarity in writing. There are minor grammatical mistakes also that could be addressed. To conclude, based on the method, analysis and results I retain a positive view on the paper. 

Post rebuttal

The rebuttal addresses the concerns raised and provides the clarifications required. I continue to retain my original rating of accept.
limitations_and_societal_impact: Yes, the limitations and impact are discussed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors address unsupervised model adaptation (UMA). While unsupervised domain adaptation (UDA) lets the model access the source domain data during an adaptation to a target domain, such access could raise multiple concerns such as privacy and portability. UMA does not allow the model to access the source domain data during adaptation. The proposed method, historical contrastive learning (HCL), utilizes the historical source hypothesis to remedy the lack of source data. The experimental results show that the proposed method achieves state-of-the-art performance for multiple visual recognition tasks.
main_review: Originality:
- UMA has two original components; one is historical contrastive instance discrimination (HCID), and the other is historical contrastive category discrimination (HCCD).
- The idea of contrastive learning between two models can be found in another paper for continual learning [a]. It is not a problem since [a] is released after the deadline of NeurIPS 2021.

[a] Cha et al., Co2L: Contrastive Continual Learning. arXiv, 2021.

Quality: 
- While there are already many references, the authors should refer to the following papers.
- [b] addresses source-free domain adaptation under universal domain adaptation settings, which means there is no guarantee that the classes in the target domain are the same as in the source domain. Since UMA in this paper is a variant of closed domain adaptation, [b] can be considered a more complex problem.
- [c] and [d] address source free domain adaptation for image classification. [c] does not compare the performance to those of existing UDA methods. [d] reports some results on VisDA, Office-31, and Office-Home. [d] uses AlexNet for Office-31; therefore, the performances are worse than those in Table 7.
- By the way, [37] is already published in CVPR 2021.

[b] Kundu et al., Universal Source-Free Domain Adaptation. CVPR, 2020.

[c] Kurmi et al., Domain Impression: A Source Data Free Domain Adaptation Method. WACV, 2021.

[d] Yeh et al., SoFA: Source-data-free Feature Alignment for Unsupervised Domain Adaptation. WACV, 2021.

Clarity: 
- The motivation and formulation are clearly described.
- The theoretical insights in Section 3.5 clarifies the use of the proposed method for UMA.

Significance:
- This paper's major strength is throughout experiments for various visual recognition tasks such as segmentation, detection, and classification. HCL itself shows the state-of-the-art performance; moreover, the combination with other state-of-the-art methods shows that HCL is complementary to each.

Post-rebuttal:
- I have read the other reviews and all responses from the authors. There are no concerns to recommend this paper for acceptance. Therefore, I would like to keep my first score.
limitations_and_societal_impact: In the checklists, there are descriptions of the limitations and the societal impacts. The societal impact here is a positive one, but the reviewer agrees with the authors.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a generic method HCL to solve the source data-free domain adaptation problem across various computer vision tasks. Based on the memory mechanism, the method involves two designs to learn discriminative features while preserving the source hypothesis. The first design HCID leverages the contrastive instance discrimination where positive pairs are made of the feature from the current model and features from historical models including the original source model and model in previous epochs. The second design HCCD leverages the consistency between the current model and previous models to select confident pseudo labels for task-specific self-training. Experiments across various vision tasks and domain adaptation settings validate the effectiveness of both designs and the generic method.
main_review: ## Pros
1. Originality: The HCID design and the HCL method are novel for source data-free domain adaptation. 
2. Quality: The proposed method is technically sound and supported by theoretical analysis. 
3. Clarity: The writing is clear and easy to follow. The paper is well-organized.
4. Significance: This paper targets the generic unsupervised model adaptation (UMA) problem, which is significant. Demonstratable results are achieved by HCL on various UMA benchmarks.

## Cons
1. Originality: Missing related works [A, B] on the recent popular UMA problem, although some results in these works may be better than the results in this paper.
2. Clarity: How HCL works with the two designs (HCID and HCCD) is unclear. $L_{HCL} = L_{HisNCE} + L_{HisST}$? Is there any tradeoff hyper-parameter between two objectives or do the two designs work in the meantime or in a two-stage fashion? If there is a tradeoff, how does the tradeoff hyper-parameter affect the adaptation performance? If there are other hyper-parameters, please clarify their effects similarly. In addition, the implementation or training details of the HCL applied to various computer vision tasks or domain adaptation settings are not provided adequately.

References

[A] Kundu et al., Universal source-free domain adaptation, CVPR'20 

[B] Liang et al., Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer, arxiv'20

-----------------------------------------------------------------------------------
## Post-rebuttal
The response has addressed most of my concerns on the training details of the proposed method. I decided to keep my original score of 7 because of the strong performance and extensive experiments in the submission.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the unsupervised model adaptation problem, where only the trained model, but not the source data, is available to use for target domain adaptation. The authors propose historical contrastive learning (HCL) technique to help this source-free setting. HCL is composed of two components, HCID and HCCD. HCID learns from target instances by contrasting the features generated from the adapted model and historical models. HCCD explores self-training to pseudo label target instances to learn category discriminative representations. Experiments on three different vision applications are conducted. The paper is generally easy to follow, despite some unclear points. The major concern is the novelty.  
main_review: Strong points
1.	Model adaptation without access to source data is an interesting and practical problem.
2.	The experiments include three computer vision applications, segmentation, detection and classification. The performances on different datasets are consistently better than baselines, although some improvements are marginal (see week points for detail)

Weak points
1.	Limited novelty is my major concern. I recognize the authors apply contrastive learning to this new problem, and I also recognize the proposed modifications, e.g. the reliability scaler in Eq 3 compared to the traditional InfoNCE. But these do not meet the NeruIPS bar, especially the proposed modifications are not fully justified by experiments. (see below)
2.	For $r$, to support the claim that this soft version (compared to traditional $r_i^{t-m} =1$), the analytical experiment of $r=1$ is missing. Similarly, in Eq 4, a baseline by setting $h_{con} =1$ (no reweighting) is missing.
3.	Missing key related references [ref-1]. 
[ref-1]Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri. Domain impression: A source data free domain adaptation method. WACV 2021
[ref-2] Jogendra Nath Kundu, et al., Towards Inheritable Models for Open-Set Domain Adaptation, CVPR 2020
Please consider making a discussion/comparison.
4.	The equation above line 164, $p^t$ is a scalar, l-1 norm on a scalar does not seem correct to me. One more right bracket.
5.	A completed ablation study on different applications is highly recommended. 

Other minors: 
6.     For the baselines, I suggest removing all the non-source-free ones, which are out of this paper’s focus. 
7.	Line 121, the reference [20] seems wrong.

limitations_and_societal_impact: No known concern on limitations and potential negative societal impact.
review_title: null

Meta-review: The paper tackles a challenging problem of unsupervised domain adaptation (UDA) without access to source data while assuming access to a pretrained source model. The proposed method takes inspiration from contrastive losses (such as InfoNCE) and tailors these for the UDA setting. Reviewers appreciated the significance of the problem tackled by the paper (source-free UDA) and the experiments on a variety of vision tasks that show clear improvement over existing methods. Authors should revise the paper taking into account the reviewers' comments, including a discussion of the prior work pointed out in the reviews on the source-free UDA and universal DA problems. 
Decision: Accept (Poster)