Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a neighborhood affinity based consistency approach for source-free domain adaptation. While some prior adaptation methods use nearest neighbors to obtain pseudo labels for the unlabeled target samples, this work proposes the use of reciprocal nearest neighbors (RNN). The observation is that RNNs show a higher degree of label consistency than nearest neighbors. Accordingly, the adaptation is formulated as improving the label consistency with the Reciprocal Nearest Neighbors of a given target instance. The method yields excellent performance over 4 datasets.
main_review: Strengths
========

a. The motivation for various nearest neighbor strategies is clear (L42-51). The proposed reciprocal nearest neighbor approach is intuitive and makes sense. The simplicity of the approach is a key highlight of this work. 

b. Good work on the analysis. The figures are helpful in understanding the effectiveness of the approach. Figure 4 (Left) presents another interesting observation where per-shared fraction (measured without using target labels) converges with target accuracy (more details in the next section). The ablation study is detailed and supports the claims made in the paper. Moreover, the insights presented in this work would be highly useful for research in Source-Free Domain Adaptation.

c. The manuscript is mostly easy to follow (though some clarifications and typographical fixes are required). Overall, it is a good read.

d. The inclusion of code is appreciated.


Scope of improvement
=================

a. Scalability: The requirement of a memory bank and nearest neighbor computation seems like a large overhead (e.g. consider large scale datasets like ImageNet). While I understand that features $z_i$ and prediction scores $p_i$ are cheaper to memoize than the input data itself, it is imperative to investigate the effect of reducing the space and computational overhead. 

* A runtime analysis (training time, FLOPS / epoch etc.) is required to understand the scalability of the proposed framework. Could the authors provide a runtime comparison against the prior state-of-the-art methods? 

* Could the authors provide more details on how the nearest neighbors are computed? A typical nearest neighbor computation would entail $O(n_t * n_t)$  operations (quadratic) for each epoch (every target feature is compared with every other feature to obtain the list of nearest neighbors). In my opinion, this is a huge overhead considering that most adaptation algorithms require $O(n_t)$ operations (linear). 

* Consider the following approach: Suppose the target training dataset is split into, say, 10 mutually exclusive and exhaustive folds. Each fold pertains to 10% of the data (call this, one mini-epoch) and is trained upon sequentially, i.e., we reduce the memory bank to 10% of the data, and ensure that samples in this mini-epoch pertain to the corresponding fold. After one epoch on the entire dataset is completed (i.e. after 10 mini-epochs), the data is randomly shuffled to create the next 10 folds. In this manner, the model is trained on the entire dataset, however, would require 10 times less overhead for memoization and NN-computation. Furthermore, this could reveal that perhaps only a fraction of the dataset in-memory is sufficient and the performance gain saturates with an increase in samples. If this works, then one of a major drawbacks of the proposed framework could be solved. Could the authors verify this and comment on the efficacy of this approach?

b. Writing:

* L147-149: This part is confusing. My understanding is that $\mathcal{S}\_{i}^{T}$ is a constant vector (not “scalar”, L147) containing the prediction scores of $x\_i^t$. The loss $\mathcal{L}\_{self}$ essentially computes a weighted sum (weights = $\mathcal{S}\_i$) of the class probabilities ($p\_i$). The expression is written like so (and not $p\_i^T \cdot p\_i$) because the loss $\mathcal{L}\_{self}$ is assumed to back-propagate only through the variable $p\_i$ and not through the weights ($\mathcal{S}\_i$). If this is the case, please consider paraphrasing this part to clarify the same.

* Some additional improvements are suggested in Minor Comments below.

c. Affinity measure: In the current formulation, affinity is a hyperparameter ($A = 1$ or $r<=1$). Thus, affinity is assigning discrete weights to the similarity between pairs of samples in Eq. 1. Did the authors consider using continuous affinity values to obtain a “degree of connectivity” (L55), for e.g. using feature distance $D_{dis} = || z_i - z_j ||$)?

d. The observation in Figure 4 (left) could potentially lead to a reliable convergence criterion. The usual practice in Unsupervised DA is to consider a labeled validation set for determining training convergence / selecting maximum epochs etc. A recent work [P1] presents an approach to determine training convergence without using target labels. In the proposed framework, the per-shared plot in Figure 4 (left) seems to correlate with the target accuracy. This could potentially be used as a convergence criterion. Could the authors provide more insights on this aspect?

[P1] Venkat et al., "Your Classifier can Secretly Suffice Multi-Source Domain Adaptation", NeurIPS 2020.


Minor Comments
=============

* A recent work by Wang et al., “Tent: Fully Test-Time Adaptation by Entropy Minimization”, ICLR 2021 also performs source-free adaptation using entropy minimization which results in a simple approach. It would be interesting to compare the runtime vs. performance trade-off between Tent and the proposed neighborhood approach. 

* Typos: L273 (consistence -> consistency), L217 (date -> data), L145 (simple -> simply), L115 (weight -> weigh), L103, 232 (discusses -> discussed), Fig. 1 caption (pushed -> pushing), L34 (vary -> very).

* L90, L149, L270: Avoid 1-3 worded lines to save space (paraphrase the sentence).

* L45-51: Here, it would be a good idea to introduce / define RNN and nRNN so that the figure 1(c) is comprehensible. Also, in Fig. 1(c) left, consider showing nearest neighbors connectivity on the left and the RNN/nRNN plot on the right (i.e. swap the two diagrams). This would enhance readability.


I would be happy to increase the score if my concerns are addressed.
limitations_and_societal_impact: I could not find a discussion on potential limitations and societal impact. I would suggest the authors add a discussion on these aspects. As an example, the nearest neighbor calculation and memoization operations are compute-heavy, which might put organizations with access to large computational resources at a higher advantage than others.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper addresses the problem of Source Free Domain Adaptation for the classification task. SFDA is a challenging DA scenario where instead of source data only the source pretrained model is available for adaptation to unlabeled target. 

This work focuses on closed-set domain adaptation task. The approach is motivated from an interesting observation that though targe samples do not align with the source classifier, these samples still form clusters with consistent class semantics. Following this, the authors aim to leverage this local neighborhood structure by encouraging label consistency among samples with high neighborhood affinity. The affinity resembles the degree of connectivity which is defined separately for general neighbors (nRNN), reciprocal neighbors (RNN), and extended neighbors (i.e., neighbor of neighbor).

The approach involves maintaining two memory banks, a) features and b) prediction scores for all the target samples. While neighbors are selected based on feature space similarity, the adaptation losses aim to maximize the affinity weighted prediction score similarity among the neighbors.

main_review: Strengths:

+The authors introduce a source-free domain adaptation method by using the intrinsic data structures. They propose a method for adaptation by encouraging label consistency among local target features by differentiating between nearest neighbors, reciprocal neighbors, and expanded neighbors, and further exploiting that information.

+The paper is well-written with clear justification and ablation analysis of the various learning components. The approach achieves significant improvement on VisDA-C (3%)  and marginal improvement (0.4%) over the prior-art [21] on the Office-Home dataset.


Weaknesses:

1) The minibatch update of $p_i$’s coupled with the self-regularization loss $L_{self}$ seems similar to an mommnetum update of $S=[p_1, p_2,..., p_{n_t}]$. While $p_i$ is computed based on the current model weights, $L_{self}$ does not allow drastic changes in $p_i$, thus acting as a regularizer. I am curious wether $L_{self}$ can be replaced by a simple momentum update of $S$. 

2) Following the previous, any comments on what would be the behavior if such momentum update is applied while updating $F$. As I see, all the affinity-based losses and self-regularization is applied on $p$ while $z$ is used only for neighborhood selection (i.e. for assigning affinities). What if we apply similar regularization on $z$.

3) The proposed approach does not employ any pseudo-label-based loss as used in other prior arts. I am curious whether such a loss would further improve the adaptation performance. I observe that all the losses are defined on $p$, i.e., a softer form of pseudo-label.

4) I am curious about the importance of the diversity loss, $L_{div}$. I would like to see the performance where all other losses are active but without the diversity loss, $L_{div}$ (in the Left and middle panel of Table 5). This would give us some idea regarding robustness of the proposed approach against imbalanced target.

5) The authors claim that “source-free methods outperform methods that have access to source data during adaptation” (L215). They should have compared the same method with supervision from source data during adaptation. I would like them to show comparisons with other relevant non-SF prior-arts like A1. Also, can the performance of the proposed approach be improved further in the presence of source samples?
6) The authors have set the affinity value for the expanded neighbors to an arbitrary value of 0.1 (L254). And, the authors stated that expanded neighbors should have less affinity than the nearest neighbors (L165), yet they are using the same affinity, 0.1 for both expanded neighbors and nRNNs (L143, L254). Please clarify.

References:
[A1] Na et al, FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation [CVPR’21]


**** Post-rebuttal ****

I thank the authors for the detailed rebuttal. The rebuttal addresses most of my concerns regarding the limitation which were not discussed before. Below I restate some of these:
a) In presence of source data the performance of the method degrades.
b) Without diversity loss performance degrades significantly. This raises concerns about whether this method is at all suitable for open-set, partial, or universal DA settings.
c) The significance of setting different affinities for the expanded neighbors and nRNNs must be clarified thoroughly.

I hope that the authors would include a thorough discussion on the limitations of this approach in their revised draft. I'd like to raise the score to '6: Marginally above the acceptance threshold'.


limitations_and_societal_impact: No. The authors have not discussed any limitations of the approach. Certain limitations would be a) robustness to larger domain shifts, b) robustness to class-imbalanced target, etc. Further, the authors’ response of No to whether this approach could have potential negative societal impacts is a major oversight.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose a source-free domain adaptation (SFDA) method by exploiting the intrinsic target neighborhood structure. Extensive experiments conducted on both 2D image and 3D point cloud datasets demonstrate the effectiveness of the proposed method to a certain extent.
main_review: 1.	Please give more explanations and discussions about the motivation of utilizing the memory bank to retrieve nearest neighbors.
2.	Is the hyperparameter r used in Eq.5 and Eq.9 the same? Why not choose different hyperparameters for these two parts? 
3.	Please provide comparative experiments with more source-free domain adaptation (SFDA) methods, such as [1] [2] [3].
4.	It’s recommended to provide visualization of target features before adaptation in Fig.5 for better comparison.
5.	Does the performance improve significantly at the expense of consuming large computational time cost? Please provide time complexity analysis of the proposed method.
6.    The current manuscript should be carefully polished, e.g., some grammar errors, the layout of Figures and Tables.
7.    The submitted code cannot re-implement the results in this paper, and there is a large performance gap. So I recommend to reject this paper.

[1] Kurmi V K, Subramanian V K, Namboodiri V P. Domain Impression: A Source Data Free Domain Adaptation Method[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021: 615-625.
[2] Tian J, Zhang J, Li W, et al. VDM-DA: Virtual Domain Modeling for Source Data-free Domain Adaptation[J]. arXiv preprint arXiv:2103.14357, 2021.
[3] Yeh H W, Yang B, Yuen P C, et al. SoFA: Source-data-free Feature Alignment for Unsupervised Domain Adaptation[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021: 474-483.
limitations_and_societal_impact: None
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a framework to perform Source-free Domain Adaptation by fully exploiting the local representation structure in the latent space. Specifically, it maintains two memory banks for logits and representations, respectively. Using the a sample in the mini-batch as query, it retrieve semantically similar neighborhoods from the bank, and constrains their logits to be similar. In addition, a self-regularization loss is applied to mitigate noisy neighbors, and a diversity loss is used to balance the sizes among different clusters. Experimental results on four benchmark datasets verify the superiority of the proposed method. Also, thorough ablation studies are conducted to analyze the effect of different model components. 
main_review: Strength:

1. The proposed framework is technically sound, which can exploit the local representation structure from various granularities.
2. The paper is well written, and it's easy to capture the gist of this work.
3. Sufficient experiments are conducted to evaluate the whole model and various model components.

Weakness and question:

1. Technically, I'm convinced that the proposed method is able to promote the feature compactness within some feature cluster. However, except for the feature compactness, it is also important to constrain the inter-cluster feature separability so as to guarantee the good performance on target domain. How can the proposed model achieves this important property?
2. In Eq. 1, authors use the distance towards query sample to depict semantics. Considering the representation bias of each specific instance, why not use the distance towards the cluster center associated to the query sample? This scheme may capture semantics more precisely.
3. The objectives in Eq. 9 are composed of a diversity term, two neighborhood semantic similarity terms and a self-regularization term. There should be a trade-off among these three kinds of objectives, such that some trade-off parameters might further promote the performance of this model.

Justification of rating:

I am convinced by the technical merit of this paper, and think it as a well-suited framework for Source-free Domain Adaptation. Though there are some points needed to be further clarified, this work is decent enough as a whole.


**Post Rebuttal**

I have read the response from authors and other reviews. In general, I admire authors' efforts on exploring the local and global neighborhoods structures in the representation space of multiple domains. Most of my concerns are addressed in the response, and thus I keep my positive rating.
limitations_and_societal_impact: The limitations of the proposed method are less discussed in the current version. Also, the potential social impact is lacked in the paper. Authors are encouraged to analyze how their method can potentially benefit and also hamper the related real-world applications.
review_title: null

Meta-review: This work proposes a framework to perform Source-free Domain Adaptation by fully exploiting the local representation structure in the latent space. Specifically, it maintains two memory banks for logits and representations, respectively. Using the a sample in the mini-batch as query, it retrieve semantically similar neighborhoods from the bank, and constrains their logits to be similar. In addition, a self-regularization loss is applied to mitigate noisy neighbors, and a diversity loss is used to balance the sizes among different clusters. The reviewers found the paper to be technically sound and the approach interesting. There were some concerns about the technical novelty and the strength of the results, for which the paper can be argued from both a positive and negative perspective, since it builds on previous work (reciprocal NNs) and thus have some incremental flavor. Nevertheless, the positive reviewers highlighted the novelty of the method as a whole. After discussion it was concluded that the paper merits publication.  
Decision: Accept (Poster)