Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper addresses the multi-source free domain adaptation problem where multiple pretrained source models and unlabeled target data are given. Theoretical analysis aims at proving that multiple source domains improve the probability to ensure a generalization bound. The method is developed based on mutual information maximization, pseudo labeling and consistency losses. Experiments show that the proposed method achieves slightly better results. 
main_review: This paper addresses a new problem but with limited technical novelty. The proposed method combines a number of well-known techniques. The two loss terms in Eq.(2) and (3) in Section 5.1 form components of the standard mutual information criterion used for semi-supervised learning but without providing the similar insights.  The linear combination of multiple source models is also a very common integration strategy.  The pseudo-label generator in Section 5.2 follows the most common confident-based pseudo label determination strategy but with a bit tedious process. It lacks any more insightful principle. 

Moreover, the techniques adopted are mainly semi-supervised learning techniques and there is not really any domain adaptation factor. Does this mean source free domain adaptation can only be solved from the semi-supervised learning perspective?


The related work section is insufficient. It only very briefly mentions the source-free domain adaptation methods. Are there any major technical differences between this proposed method and the methods in the literature? It fails to mention and compare their method with the existing work [1] for the same multi-source free domain adaptation problem in the related work section even though this work [1] also uses information maximization, prototype-base pseudo labeling and meta-learning to learn the weighting coefficients for multiple source domains.

The experiments are not very convincing with slight performance gains. Are all the comparison methods used the same feature extraction networks?  The feature extraction network can lead to performance differences.

Other questions: 
1. For the Source-Specific Transferability Perception, in line 180, why are the weight vectors taken as prototypes if the classifiers compute inner-product instead of Euclidean distance?


[1] Sk Miraj Ahmed, et al. Unsupervised multi-source domain adaptation without access to source data. CVPR 2021. 

limitations_and_societal_impact: see main review
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work considers unsupervised domain adaptation issue. A novel framework called CAiDA is proposed which uses the pre-trained source models rather than source data which are commonly used in traditional methods. To demonstrate the effectiveness and superiority of the proposed method, the authors did theoretical analyses as well as numerical experiments. The numerical experimental results show the proposed model can do better than state of art methods.
main_review: Pros:
This paper is well organized, the motivation is clear and logical, and it is easy to follow the major idea.

The results are valid. Extensive experimental results are provided to show the effectiveness of the proposed method. Besides, some theoretical analyses are also provided to provide insights behind the proposed model.

Cons:
The writing of this paper can be further polished, especially the theoretical part. The theorems should be stated in a clearer and more formal way. The theorem and assumptions should use the italic form, but the proofs should not. Besides, there are too many steps in the proof of the two theorems, these should be simplified.

There are also some typos in the paper, these should be revised in the final version.

The authors only discussed the limitations about the assumptions in theoretical analysis. The limitations of this model and why it does not work very well on some specific datasets should also be included.

limitations_and_societal_impact: See Cons above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work presents CAiDA for Multi Source-Free Domain Adaptation in the absence of source data. The core idea is to define confident anchors to obtain pseudo labels for target domains. In the presence of multiple source domains, the contribution of each source domain for the target sample is measured using source-specific transferability perception modules. Further a class-relationship aware consistency loss is proposed to maintain semantic consistency across domains. The proposed method consistently performs better than prior arts on several datasets.
main_review: Overall, the novelty is limited as the approach ties multiple existing ideas together. However, this work can be improved by focusing more on the distinguishing elements.

1. Novelty: 

One of the major drawbacks is that the novelty is limited in the proposed framework, especially when the improvement over prior work DECISION [1] is only marginal (e.g. 0.4% overall improvement on Office-Caltech dataset in Table 1). Specifically, there are two major components which are employed in previous works:

    - The conditional entropy minimization ($\mathcal{L}_{ent}$) and diversity promoting ($\mathcal{L}_{div}$) losses are quite common in the Domain Adaptation literature. For instance, DECISION [1] employs both these losses.

    - While the proposed pseudo-labeling approach is based on the nearest confident anchor (high confidence target sample), DECISION [1] labels target samples based on the nearest cluster centroid. These two methods are notionally similar - both assign target pseudo labels based on the nearest high confidence target samples.

The major distinguishing factor of this work is the Class-Relationship Aware Consistency loss and the synthetic feature augmentation (L234), which should be highlighted upon in my opinion. Also note that certain works such as [P1] optimize class confusion matrices that naturally results in feature alignment during adaptation.

[P1] Jin et al, “Minimum Class Confusion for Versatile Domain Adaptation”, ECCV 2020


2. Readability / Missing Details: 

    - For most parts, the manuscript is easy to follow, however there are some inconsistencies in the notation. For eg. the LHS in $\mathcal{L}_t(\boldsymbol(h))$ (L113) should have superscript $i$. In L184, $\mu$ should be bold (vector).

    - Could the authors clarify what is meant by “probability over all target data” in L200? Does it refer to the target data in a mini-batch, or the entire target dataset (L110)? If it is the entire target data (of size $m$) then each backpropagation through loss $\mathcal{L}_{div}$ would require 1 epoch.

    - L183 (“$\boldsymbol{U}$ is then forwarded into a three-layer convolution network $\boldsymbol{\Omega}$”): Could the authors clarify the rationale behind the use of a convolution network? Here, the ordering of the sources in the matrix $\boldsymbol{U}$ would play a major role in determining the performance of the network, due to the nature of the convolution operation. Furthermore, I couldn’t find details about this network.

    - The assumptions in Section 4 are better understood by referring to the detailed proofs in the supplementary material. I suggest making Section 4 more elaborate (for e.g. the discussion in Introduction in L30-L77 is repetitive which can be trimmed down to save space).


3. Approach / Evaluation:

- It seems that the Consistency Loss $\mathcal{L}_{crc}$ (Eq. 8) is computationally intensive, with $O(n^2 * K^2)$ operations. However, from the ablative analysis in Tables 1-3, the consistency loss results in the least gains which suggests that the loss is inefficient. Could the authors comment on this aspect? For instance, what is the increase in training time by incorporating this loss?

- An ablative study on the use of $\mathcal{C}_p$ and $\mathcal{C}_d$ is missing. Note that, using $\mathcal{C}_d$ is similar to using the cluster centroid for pseudo-labeling in DECISION [1]. Therefore, it is important to understand how the combination of $\mathcal{C}_p$ and $\mathcal{C}_d$ helps.

On a side note, it would be a good idea to present standard deviation along with the mean statistics since the average performance of the methods are close.

---------
Post Rebuttal comments: The authors have addressed my major concerns. After going through all the reviews and the author responses, I would like to vote for an acceptance.
limitations_and_societal_impact: Adequately addressed in the Supplementary Material.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, an algorithm for multi-source domain adaptation is proposed. The algorithm is developed in a source-free regime to address the concern for privacy. A loss function is designed to integrate the source-specific end-to-end classifiers using solely the target domain samples, which preserves privacy. The loss function consists of four terms to benefit from enforcing source-specific transferability, anchor-induced pseudo-labels, and class relationship-aware consistency. A PAC-learning style theoretical analysis is provided to demonstrate that using multiple sources may improve the generalizability on the target domain. Finally, experiments on five standard UDA datasets are provided to demonstrate that the method is effective.
main_review: 1. The proofs for both theorems seem correct. However, I feel the analysis does not offer much benefit to the paper. When it comes to the assumption, note:
- the loss function usually is not bounded in deep learning, e.g., using cross-entropy.-  PAC-learning analysis is not very useful for deep neural networks because they are known to have poor VC-dimension or Natarajan dimension. Indeed, this is a primary reason that deep learning cannot be well understood within the standard PAC-learning analysis based on notions such as VC-dimension.- It is not clear to me that the used terms in the loss function satisfy the requirements for Theorem 2.
Hence the provided analysis does not offer much support when deep neural networks are used. Additionally, analyses based on PAC-learning have existential nature broadly speaking and beyond deep learning, you need to have an idea of whether the conditions are met in practice. In this sense, I don't think the theoretical analysis is very valuable. I highly am skeptical to use mathematical analysis when it does not provide much benefit, just for the sake of making the work more rigorous. My assessment would be totally different of course, if it is demonstrated that the analysis is relevant to the experiments. 

2. The proposed method outperforms the multi-source UDA methods, despite being a source-free method. This is impressive but goes against my intuition. Because other multi-source UDA methods seem to upper-bounds for the proposed method because they are using more information, i.e., the source samples. I was wondering what is the reason for this observation? Is there any way to show that if the source data points for accessible, the proposed method can have even better performances? I think such an experiment can serve both as an ablative experiment and also broaden the novelty of your work even in simple multi-source UDA setting.

3. The small improvement of the proposed method over the single-source UDA algorithms questions necessity of the proposed approach, i.e., improvement over DECISION is about 0.5% on average. If this is the case, one might ask why not just use the existing source-free methods by averaging? I understand we see a marginal improvement, but this also comes with a significantly more rigorous procedure. Additionally, not all source-free methods are used for comparison. It may be possible that if we use other methods, e.g., Universal Source-Free Domain Adaptation, we get better results. So, at least all the existing source-free methods should be used to demonstrate the proposed method offers an advantage in terms of performance. I would like to include the best source-free single domain UDA algorithm in the results. Please check the literature for this purpose.

4. The primary concern that I have for this work is selecting the values for the hyper-parameters k1 and k2. Figure 3 (a) and (b) are not accurately interpretable in my eyes but I think they show performance can vary a few percent when these hyper-parameters are not tuned properly. Hence, proper value selection for these hyper-parameters seems to be essential to observe the reported performances. Now, the question is how do you tune these parameters? From the descriptions, it seems that the authors have used performance on the target domain to select the optimal values. This is however mathematically wrong because it is as if test error has been used for hyper-parameters tuning. Using the test error for hyper-parameter tuning questions whether the proposed method can stably outperform the baselines given the small differences between the performance, as low as 0.3% in some cases. Please clarify how have you tuned the hyper-parameters in your results and how should the user tune them for other new datasets. If the test error has been used, then the reported performances are not acceptable. 

5. I would like to know how does Algorithm 1 scale when we have many source domains? Is it possible to offer complexity analysis? I think theoretical analysis on direct aspects of the algorithm will be more useful.


Minor comments

- The title is confusing: in the absence of punctuation, multi-source free is confusing. Maybe it is better to change it to multisource-free or use different wordings.

- Line 38-40 are a little confusing. It is implied having a massive labeled source of data is the concern for the current MSDA methods, whereas privacy is the issue you address. Irrespective of implementing UDA in a source-free regime, we will need a labeled source domain data. 

- Figure 2 is difficult to decode, e.g., what are the colors for each domain?. I understand you have a page limit, but it can be clarified in the Appendix.

----------------------------------
Following internal discussions with the rest of reviewers, I finalized and maintained my updated rating.
limitations_and_societal_impact: I think the authors have addressed these aspects adequately.
review_title: null

Meta-review: The theoretical contribution for multi-source domain adaptation without source data is nice. Though some components in the proposed method are borrowed from existing techniques, the overall design of the proposed method is based on the insights of the theoretical analysis. As [1] is the first work (if I am not wrong), which only provides theoretical analysis, the authors should add a section to discuss the difference in terms of the theoretical findings between the proposed method and [1] in a revision. 

Overall, I recommend acceptance for this work.
Decision: Accept (Poster)