Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a clustering-based method for the Source-Free Unsupervised Domain Adaptation (SFDA) problem. Considering the semantic information embedded in the feature space of the source model, the authors proposed an AaD clustering objective, aiming to simultaneously encourage the model’s discriminability and diversity during the adaptation. Besides, this paper also compared and analogized their method with several previous SFDA methods and gave some insightful analysis. Finally, experimental results on several domain adaptation benchmarks under different SFDA settings verified the efficacy of the proposed method.
strengths_and_weaknesses: Strengths

1. Overall, this paper is well-written and easy to follow.
2. The proposed method is simple but effective, and the analysis between the proposed AaD objective and the previous SFDA methods in section 3.2 is insightful.
3. The experiment part is rich, which covers a wide range of SFDA SOTA methods and different SFDA settings (e.g., close-set, open-set, and partial set). The experimental results can generally confirm the effectiveness of the proposed method.

Weakness

1. Novelty: Compared with the previous neighborhood clustering-based method and the contrastive learning method, the proposed approach seems to be an incremental work, and the novelty is relatively limited.
2. The description of the algorithm and the training process is not completely clear. Please refer to the Questions section.
questions: 1. For the description of Algorithm 1 in line 5, it is written, “Update model by minimizing Eq.5”. Does this mean only updating the feature extractor f as implemented in [1], or updating the feature extractor and the classifier g simultaneously?
2. The choice and the analysis of $\lambda$: according to the simplified upper-bound of the proposed AaD objective, theoretically, $\lambda$ = $\frac{N_{C_i}}{N_{B_i}}$. Although the authors have explained the treatment of $\lambda$ and the introduction of decay exponent $\beta$, I am still curious about the numerical connection between the truly utilized $\frac{N_{C_i}}{N_{B_i}}$ (the theoretical value) and the evolving $\lambda$.
3. As we know, pseudo labeling is another widely leveraged technique for the SFDA problem, and it is usually combined with the hard or soft neighborhood clustering methods. I am curious whether the authors have considered combining pseudo labeling methods with the proposed AaD approach.

Other comments:

Typos:
In line 298, “Soruce-free” → “Source-free”.
limitations: The authors discussed the limitations in section 2.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper approaches the source-free domain adaptation problem through the lens of clustering, which forces similar features to cluster and dissimilar features to disperse. A simple upper bound for the objective is introduced and optimized. On VisDA and Office-31, the proposed method achieves state-of-the-art performance.
strengths_and_weaknesses: ### Strengths
- The paper is well written and easy to read. 
- Impressive performance improvement in VisDA over previous SOTA. 

### Weaknesses
- The proposed method's concept and effect are very similar to previous contrastive learning methods, such as InfoNCE. InfoNCE or SimCLR also set positive/negative pairs and force the positive pairs to attract while the negative pairs disperse. The difference between SimCLR and the proposed method appears to be marginal; SimCLR applies the loss to the projection layer, whereas the proposed method applies the loss to the output layer. 
- Using the nearest neighbor as a positive pair in contrastive learning is not a new discovery for the deep learning community. [1] 
- The experimental results are not promising. The proposed method performs well on VisDA, but it only outperforms HCL by 0.1 percent on Office-31 and fails to achieve state-of-the-art performance on Office-Home. 
- Storing all target features in a memory bank and retrieving K-nearest neighbors from the memory bank for each batch appears to be a significant computational burden. Have the authors thought about this and computed the computational complexity of the proposed algorithm? Is this method capable of handling large target datasets? In comparison to existing methods, how much time or FLOPs does the proposed method take? 

### Overall 
The proposed loss objective is straightforward, and the concept is simple to grasp. The theoretical novelty and significance of the contributions, however, appear to be insufficient for this venue.

[1] Dwibedi, Debidatta, et al. "With a little help from my friends: Nearest-neighbor contrastive learning of visual representations." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

questions: - The legends in Figure 2 are too small to read.
limitations: I couldn't find any discussion of the work's limitations or potential negative societal impact by the authors.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed an efficient and effective approach for source-free domain adaptation, which encourages nearest neighbors in the feature space to be similar, while the rest of the samples within the minibatch are dissimilar.
strengths_and_weaknesses: A well-written paper that has clear math notations and solid derivations. And the results on both DA dataset and Source-free open-set DA datasets looks promising. However, although the intuition for the Attracting-and-Dispersing strategy is super clear but not much theoretical support is provided. Also, some details of the model are not discussed in detail and the claim on the efficiency needs to be validated.
questions: 1. This paper claims the search of nearest neighbors within the mini-batch is efficient but has not been validated. Presenting the complexity analysis and the compute time would help to validate this claim.

2. Some details of the model are not discussed in detail. For example, it is not clearly presented how to calculate the probability that the feature, $p_{ij}$, and how it has been involved in the back-propagation.

3. What's the memory requirement for building up two memory banks that store all the target features?

4. The experiment results on DA datasets show that the proposed method has higher average accuracy than the traditional DA approaches. How does the proposed method perform better than those algorithms without accessing the source samples and labels?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a simple SFDA method that aims to adapt the model to the target data in a self-supervised manner. The adaptation is based on contrastive learning; positive pairs are constructed by searching kNN of the target sample from memory bank, while all sample pairs within the mini-batch is considered as negative pairs. Differently from popular contrastive learning methods, the output space is used to compute loss in the proposed method, which strongly encourages the model to assign each target data into predefined class labels. The experimental results validate the advantage of the proposed method. 

strengths_and_weaknesses: I will describe strengths and weaknesses from four perspectives: originality, quality, clarity and significance.

<originality>

Not highly novel. Methodologically, the proposed method is similar with contrastive learning with InfoNCE loss, but it is different in how to compute similarity scores in the loss function. Specifically, while the positive and negative pairs are constructed at the feature space in a similar way with [R1], the similarity scores in the loss function are computed at the output space. The authors provides some theretical justification on the formulation of the proposed method, though I have a concern on its derivation, which will be described in <quality>.   
[R1] "Mean Shift for Self-Supervised Learning," ICCV 2021.

<quality>

Marginal. I have two concerns mainly on the derivation of the proposed method.
- In line 144-145, the authors state "we cannot get this upper-bound without P(B_i)," but I respectfully disagree with it. Using exactly the same deformations with Eq. (3) and (4), we can also have the same results by starting from -log P(C_i) except for the difference of the coefficients of the second and third terms at the third line of Eq. (4), which will be ignored in Eq. (5). This is due to the approximation adopted at the second line of Eq. (4), in which the sum over the target data is replaced with that within a mini-batch. Since the derivation of the proposed loss is quite similar with that of InfoNCE loss if we ignore P(B_i), showing the necessity of P(B_i) should be important to validate the novelty as well as the advantage of the proposed method. 
- Due to the approximation mentioned above, it should be important to use a sufficiently large mini-batch as in contrastive learning with InfoNCE loss, which leads to a concern on sensitivity of the performance to batch size. However, the batch size seems to be fixed in all experiments.

<clarity>

Great. This paper is well-written and also well-organized. Someone might think that Section 2 and Section 3.2 should be integrated, but I think it depends on preference.

<significance>

Good. This paper provides some significant results in terms of showing that such a simple method can perform on par or better than other existing methods in several benchmark datasets. It is also a good news to be able to effectively tune the hyperparameter beta in an unsupervised way.

questions: Related to the concerns described at <quality>:
- Do we really need P(B_i) to derive the loss function of the proposed method? What is the specific advange of considering P(B_i) here?
- How is the sensitivity of the proposed method to batch size?

Minor things:
- Is N_{C_i} different from K?

---

<After reading authors' rebuttal>

Since my concerns on <quality> have been resolved in the rebuttal, I updated my score from 4 to 6.

limitations: I think the proposed method does not perform well when batch size or the number of target data is too small.
review_title: null

Meta-review: metareview: This paper proposes a source-free domain adaptation method based on unsupervised clustering. The main assumption is that the source-trained model could generate target domain features that have smooth predictions in a neighbourhood. The proposed method optimizes the upper bound of the objective of prediction consistency. Experimental results show that the proposed method outperforms pseudo label and neighbourhood clustering methods.

While the main idea is not significantly novel, the effectiveness of the proposed algorithm is demonstrated by solid experimental studies. This is again a simple and efficient deep learning method designed with intuitions and without strong theoretical evidence. I would recommend acceptance of this paper given its impressive performance and solidness.

confidence: Certain
Decision: Accept