Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes creating prototypes(p) for the source data and aligning these prototypes with the target(t) data via bidirectional conditional transport measure. They use last linear layer's weights as prototypes, so it saves cost, and their method do no require source data for adaptation. For the alignment, they define bidirectional transport, that has  prototypes to targets part and targets to prototypes part in the objective function. For the former, they add prior prototype distributions that comes from Bayes' rule and approximate via EM algorithm. The latter  corresponds to entropy minimization. They obtain state of art results in several domain adaptation scenarios.
main_review: 
#### *Originality:*

- Their way of using prototypes (using last layers' weights) and formulation of CE resembles supervised contrastive learning(SupCon)[1]. However, this is just one part of their model, and further formulating the alignment in a two way conditional distributional setting and estimating class priors together is novel as of my knowledge.  


- Defining the pointwise cost between a prototype and target as doc product makes the whole t->p cost an entropy minimization term, this idea is simple, but the connection is a nice contribution.


- The related work is almost sufficiently cited in my opinion, however a couple of citations might worth adding[2,3,4]


#### *Quality:*

- The claims are supported well in a couple of different ways: 

    - The visualizations over synthetic data (Figure 1, 2 and 3) are well designed and convincing. 
    - The experiments are extensive and apparently superior 
    - Table 6 is a good ablation study that decreases the doubt about the roles of each loss term or their direct alternatives.
    - The connections to the clustering and optimality are simple but enough support.


 - Did you achieve the results on Figure 1b by training a simple case(like Figure 2) or is it just an abstract scenario in order to visualize your idea?


- How would you extend your algorithm to the cases of unseen classes? Would it work by adding nodes and prototypical weights to the output layer?


- The last layer is actually a direct representative of the source classes, is this allowed within the data-privacy practices?

#### *Clarity:*

- Overall, the paper is written clearly and the flow is good.


- There are several improvements I can suggest, for example the authors mentioned advantages of their method(other than main contributions) throughout the paper(such as imbalance, missing class within a batch, parameter costs, outlier robustness etc.) these arguments could be given in a more compact way and in one place.

#### *Significance:*

- The authors achieves really good results on a variety of tasks (single source, multi source, easier and harder datasets), with an intuitive method, their implementation can draw interest. 

- Especially given that utilizing prototypes is attempted a fairly good amount of times in unsupervised domain adaptation, and mostly with an adversarial training support and the results are not as superior as the paper at all, again as of my knowledge. So I believe both their method and results will be useful for the field. 

 

 
limitations_and_societal_impact: The authors mention hardness of eliminating bias problem in domain adaptation problems and energy cost encountered in large deep learning methods.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a prototype-based framework for UDA. The core idea is to employ a probabilistic bi-directional transport framework that aligns target domain samples with the source domain prototypes, where the prototypes are modelled using the classifier weights. A key highlight of the method is that it is applicable in the absence of source domain data, which is highly useful in practice. Benchmarking on three datasets shows the method achieves excellent results.
main_review: Strengths
========
+ The paper is well organized and written. The approach is well motivated.
+ The source-data-private setting is practical. I especially liked the discussion around the use of classifier weights as prototypes, as compared to taking feature mean.
+ The results look promising. The analysis in Section 4 (and Appendix) is comprehensive. The visualizations in Appendix Section E are interesting.

Weaknesses / Clarifications
=====================
I do not have significant criticism of the paper, however addressing the following could improve the submission:

- Regarding fixed $\mu$: Once the model is trained on the source domain, the prototypes are kept fixed. This would enforce the target domain to acquire the semantics learned by the source domain. I agree that this would lead to a more stable training, however, this restriction may not be suitable in category-shift scenarios (e.g., see [P2]). Could the authors comment on this aspect?

- Comparison with prior works: There are several source-data free approaches in the literature [P1, P2] that have been missed in the related work. In particular, [P2] also uses a prototype based metadata to perform “source-free” adaptation, however recommends using separate latent spaces for source and target domain. IMO, these should be discussed in the related work since ideas such as source-data-private setting, and compact feature clusters are already described in these works.

- L86: “missing classes”: It is unclear what is meant by the term “missing classes” here. Does it imply that the method is applicable under category-shift scenario as well (e.g. open-set domain adaptation).

- Classical DA methods work under the theoretical framework of [5] where the adapted model’s generalization bound depends on three terms (source-domain error, divergence between the domains, a constant that depends on the chosen hypothesis space). Here however, in the source-private-data setting, the source domain error becomes immaterial since the feature extractor is updated solely based on the target domain. It would be good to see some theoretical discussion on why the proposed method works (since the adapted feature extractor would not be ideal for the source domain). An interesting analysis could be analyzing the performance of source domain data on the adapted model.

References:

[P1] Kundu et al., “Universal Source-Free Domain Adaptation”, CVPR 2020

[P2] Kundu et al., “Class-Incremental Domain Adaptation”, ECCV 2020

Minor Comments
=============
- Please fix typos (e.g. Fig. 2 caption, “imabalance”). Avoid single worded lines (e.g. Table 5 caption has the word Resnet50 consuming an entire line).
- Fig. 1b is not very informative/novel. This could be removed and the space obtained could be used to plot visualizations shown in Appendix (esp. Class proportion estimation).

limitations_and_societal_impact: Adequately addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper deals with the problem of unsupervised domain adaptation (UDA) based on a prototype-oriented framework which is suggested to solve the fundamental issues on UDA (e.g., sampling variability in the source domain, instance class-mismatching in batch, source data privacy). The main contribution suggested in this paper is utilizing the linear classifier's weights as class prototypes with bi-directional transport loss for feature alignment between source and target domains. 
main_review: Originality:

 As suggested in the main contribution, the author claims the originality of using learning parameters for class prototypes and bidirectional prototype-oriented transport loss.

When it comes to constructing class prototypes with learnable parameters, I doubt the originality since the below paper also utilizes the learnable parameters to construct the class prototype for labeled data.

- Semi-supervised Domain Adaptation via Minimax Entropy, Saito et al.

It would be more convincing if the author can prove the originality and show the main difference from the above paper.

Quality:

The motivation of using learnable parameters for class prototype and technical approach using prototype-oriented transport loss seems to be reasonable.  It is supported with various unsupervised domain adaptation settings (one source, multi-source, source-data-private). Furthermore, explicit comparisons on the number of parameters and converging speed prove its effectiveness on practical domain adaptation. 

However, I have one concern related to the quality of this paper.
Comparison with proposed baselines seems to be insufficient. Specifically, the proposed method needs to be compared with other prototype-based alignment methods. 

- Contrastive Adaptation Network for Unsupervised Domain Adaptation, Kang et al.
- Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation, Yue et al.

Clarity:

It is easy to follow and well-written. The paper contains well-supported theoretical analysis and experiments.

Significance:

Suggesting the framework that requires no additional model parameters and converges fast seems to be applied in real-world domain adaptation problems.

limitations_and_societal_impact: I am curious what kind of limitations that this paper could have when compared with other prototype-based alignment papers.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: In this paper, a UDA algorithm is proposed. The core idea is based on benefiting from the class prototypes in the source domain to align two domains in a shared embedding space, rather than using the source sample features. The class prototypes are learned on the source domain using a cross-entropy loss to relax computational load. Upon learning the prototypes, a transport-based loss function is used to align the target domain features with the learned prototypes. Experiments on several UDA scenarios using standard benchmark datasets are provided to demonstrate that the method is effective.
main_review: 1. UDA is a highly explored area and new contributions should introduce novel ideas that can address existing challenges or boost state-of-the-art performances significantly. More specifically, the idea of using prototypes for UDA has been explored extensively in the literature. Transport-based optimization also has been used extensively for this purpose. Source-free UDA algorithms based on using prototypes have been developed as well, e.g.,:

- Universal domain adaptation through self-supervision, NeurIPS 2020

- Unsupervised model adaptation for continual semantic segmentation, AAAI 2021

It is true the proposed algorithm is different in the idea implementation details but the broad solution is not novel due to the precedent. Performance results also do not demonstrate a significant boost over prior works. Finally, there is no theoretical analysis to improve our understanding about using a similar approach to address UDA. Hence, I think this work is not suitable for publication at NeurIPS.

2. Recent UDA algorithms perform more extensive experiments to demonstrate the generalizability of the method and offer benefits over prior works. In this aspect, I think the experiments need significant improvement:

- Results on some important benchmarks, including, ImageCLEF, cross-domain digits, Office-Caltech, etc, are missing.

- Multi-source and source-private experiments are even less extensive in terms of both used datasets and comparison with prior works.

- Comparison has been done against mostly methods that have not been developed in the past three years. Please check the literature to include the existing state-of-the-art performance. 

- An ablative study on Eq (7) is helpful to determine the relative importance of the proposed terms. 

-In lines 49-55, it is claimed that the proposed approach is advantageous compared to adversarial learning because of robustness and faster convergence properties. Experiments are based on using test error versus learning iteration to demonstrate this claim. But I think this is not fully convincing. Comparisons should be performed based on time because the algorithms are different in terms of computations at each iteration. In other words, the execution time per iteration may be very different among these methods. A second set of experiments to demonstrate robustness is going to be very helpful.


-----------------------------------
After more elaboration, I finalized my updated rating and decided to maintain it. I think this work is best suited for future venues after incorporating all the review feedback.
limitations_and_societal_impact: The authors have explored these aspects in a short "broader impact" section.
review_title: null

Meta-review: Thanks for your submission to NeurIPS.

This paper had somewhat mixed reviews, with 3 mostly positive and 1 more negative reviewer.  During the rebuttal phase, the negative reviewer maintained his/her position that the paper was not ready for publication, so we did not fully reach consensus on the paper.  However, I appreciated the responses that the authors provided to all the reviews, particularly to the more negative reviewer.  There are some valid concerns here---particularly with regard to the experimental comparisons---but overall I am positive about this paper and am happy to recommend it for acceptance at the conference.  Please try to incorporate as much of the rebuttal material as you can in the final paper, as it will strengthen the paper, as well as making sure to address as many of the reviewer criticisms as possible.
Decision: Accept (Poster)