Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper studies the partial domain adaptation problem. The authors propose an implicit semantic response alignment method where the source domain has more categories than the target domain. The motivation is to further utilize the extra source categories to boost the adaptation performance. To achieve this, anew module class2vec is proposed to extract the implicit semantics and explore the relationship among different categories, and align the semantic response of source and target data. Experiments on three benchmark datasets demonstrate the effectiveness of the proposed methods on different backbone networks. 
main_review: Strong points
1.	The existing PDA methods design different strategies to lower the influence of source data that belongs to irrelevant classes. Despite the effectiveness of this practice, they inevitably overlook the potential” shared” knowledge among different categories, and treat the irrelevant class data as noise. Instead, the proposed implicit semantics idea investigated the “shared” knowledge across categories, which can be regarded as a concept of subclass, where different categories might share the same subclass.
I like this idea, since it is intuitive for partial domain adaptation. I think this idea might be also applied to other problems, like class-incremental continual learning, etc.
2. The presentation is clear and easy to follow. 
3. The technical part is solid, although the semantic extraction is a simple auto-encoder. Considering this is the first paper to explore this direction, I am ok with it.
4. The proposed technique is an add-on module, which can be applied to the existing partial domain adaptation methods. The authors demonstrate the improvements with three partial domain adaptation methods, where the improvements on A^2TK and BA^3US are significant.
5. Experiments well validate the idea, including the standard algorithmic performance comparison and several in-depth explorations. The topic attention weighting and cross-class interaction are helpful to readers.

Weak points
1. Section 3.1 talks about motivation, which is duplicated with some content in the introductory part. It can be more concise.
2. Figure 4 is a good indicator to show why the proposed method works. It would be better if the authors can employ the category similarity to predict which category can be boosted. Some categories are dramatically enhanced while the candles category suffers from the negative transfer. 

Typo: figure 1 caption, “purposed” -> “proposed”

limitations_and_societal_impact: No major concerns on limitations or potential negative social impact.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In this paper, the authors consider the partial unsupervised domain adaptation (PDA) problem. Different from the existing methods that assign lower weights to the source-specific classes and adapt one-to-one shared class mapping, the authors explore the relationship among different classes and expect to extract useful/similar information from other classes in a finer granularity for further boosting the positive transfer. The authors design a class2vec module to extract the implicit semantic topics from the visual features. Semantic responses of source and target data are aligned to retain the relevant information contained in multiple categories by weighting the features, instead of samples or classes. Extensive experiments on three benchmark datasets demonstrate the effectiveness of the proposed implicit semantic discovery and alignment.
main_review: Strengths:
1. The idea of implicit semantic discovery and alignment seems interesting to me. The illustrative example in the Introduction helps to understand the common character among different classes, which can be utilized for further boosting knowledge transfer. 
2. Inspired by the word2vec, the authors propose a similar method, class2vec, which presents each class into several implicit semantic topics. This module is an easy plug-in component, which can work with different PDA backbones to further improve their performance.
3. The technical details are clear and easy to follow. The authors propose the source code for easy use, which will benefit the community.
4. Experiments seem sufficient. The authors demonstrate the proposed technique on BA^3US has significant improvements compared with several SOTA baselines. The authors also demonstrate the effectiveness of implicit semantic discovery and alignment on different backbones. 
5. The paper explains the results well. For example, Figure 3(c) provides how the proposed method works at the semantic level. It shows that the weighting schema discoveries the information that responds to the same implicit topic in source and target samples on the feature level, where only partial instances and partial visual features are activated to a certain implicit semantic topic. Figures 4&5 and Table 3 show when the implicit semantic discovery and alignment works or does not work. It helps when selecting a method for real-world applications.

Weaknesses:
1. For Figure 3(c), it is necessary to add the class category of each sample. By this means, we can tell whether some instances/categories share common semantics or not.
2. There are some presentation issues to improve the paper. For example, lambda_reg seems not discussed. The second best is not highlighted in Table 2 (W->D). Imagenet in line 229 should be ImageNet.

Concerns:
In Eq. (1), the authors extract the semantics at the instance level and aggerate them together to present the class level semantics. I am not sure whether adding another constraint that the instances from the same class share the same semantic would contribute to the positive transfer. 

limitations_and_societal_impact: The limitations and societal impact are included in the main paper. 
Please refer to the main review for suggestions in detail.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a new partial domain adaptation (PDA) method. Different from previous methods that ignore the irrelevant source classes, this work argues that source data coming from extra classes also contributes to the adaptation. Motivated by this, the proposed method first identifies implicit semantic, then performs semantic alignment. The strength of this work is proved on common benchmarks.
main_review: Originality: Although the method is novel, its motivation still needs more clarification. The difference with previous contributions is clearly claimed. 

Quality: This is a complete work but the proposed method is not technically sound (see details in Limitation).

Clarity: The writing of this paper needs further improvement. Furthermore, some points in the method section are not clearly explained (see Limitation).

I raise my rating to 6 based on the author's response and other reviewer's comments.
limitations_and_societal_impact: (1)	The motivation is not solid. Although the source examples from extra classes may share similar semantic information with target examples, these source examples also introduce noises and uncertainty.

(2)	Is that necessary to introduce another network (i.e., topic attention receptor) to get the attention by using its gradients? What is the motivation of this design? Why do we need this attention?

(3)	The section “Semantic topic alignment” is confusing. For example, which normalization method is used here? What is the formal definition of A?

Minor comments: In Line 155, it is more common to use Ct \subset Cs
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper considers the problem of partial domain adaptation. To address this problem, the authors propose the Implicit Semantic Response Alignment (ISRA) method, which attempts to align the source and target domains on the semantic topic level instead of class level. Experiments on several datasets show the benefit of the proposed ISRA.
main_review: Pros:

+ This paper is well-written and easy to follow. The motivation and the overall organization are clear.

+ The proposed method is novel. As far as I know, this is the first work that tries to align the source and target domains in the semantic topic level. Also, the proposed method achieves good results on several datasets, and, it can be injected to different methods, such as SHOT, BA3US.

+ The authors provide good experiments to verify the effectiveness of the proposed method and also provide some analyses to reflect why the proposed method can work well. 

Cons:

- Although the proposed method considers the partial domain adaptation, other settings, such as closed-set and open-set, also share the same semantic topics between the source and target domains. Therefore, I think the proposed method can also be applied to other settings. I would like to see whether the proposed method can improve the results on other settings. If not, please explain the probable reason.

- In hyperparameter analysis, the authors use office-31 to validate the parameters. However, since (1) the improvement on the office-home is not consistent and (2) the number of classes is limited, the results on office-31 can not well-reflect the parameter sensitivity of the proposed. Please provide the experiments on the office-home and the avg results on all directions should be reported.

- In addition, the proposed method seems sensitive to the number of implicit topics. We can find that, even using a small value, the results are not consistent for A->W setting.

- SHOT is a source-free method, how can the proposed method learn the semantic embedding for both the source and target domains?

- Why not evaluate the proposed method on the VisDa and DomainNet? DomainNet has a lot of classes and the classes will share more semantic topics, which is more suitable for the proposed method. Because the authors argue that the poor results on the office-31 are caused by the limited number of classes. Thus, when there are more classes (such as DomainNet), the proposed method will introduce more improvements.

- There are a lot of duplicate words between the Introduction and the motivation of the Method. Please modify one of them.

- In Fig~1, it would be better to show the details of the implicit semantic response alignment, including the structure of Auto-Encoder, the alignment process. Also, for better understand the proposed method, the author can provide an algorithm part.

Overall:

In summary, this paper proposes a novel method for partial domain adaptation and achieves promising results. However, some important experiments and explanations are missed in the current version. Therefore, I will give borderline reject this time and would like to see the response of the authors.

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper focuses on the partial domain adaptation problem. The proposal is an implicit semantic response alignment method where the source domain has more categories than the target domain. The philosophy behind sounds quite interesting to me, namely, utilize the extra source categories to boost the adaptation performance. This philosophy leads to a novel algorithm design I have never seen.

The clarity and novelty are clearly above the bar of NeurIPS. While the reviewers had some concerns on the significance, especially from Reviewer SLrJ and BdsE, the authors did a particularly good job in their rebuttal. Thus, most of us have agreed to accept this paper for publication! Please include the additional experimental results in the next version.
Decision: Accept (Poster)