Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to deal with the open-set domain adaptation problem by not only considering the known classes matching but also the target-unknown distribution segregating. Both the theoretical analysis and corresponding algorithm for dealing with the problem are provided. in general, the paper is well-motivated, and the experiments are solid.
strengths_and_weaknesses: Strengths
+ The proposed method has some novel ideas that separate the target unknown classes from the shared classes in the feature space. 
+ The estimation of the open-set recognition is achieved by posterior inference, which is interesting.
+ Theoretical analysis is provided.
+ The experiments are extensive and good results are achieved on several benchmark datasets.

Weaknesses

- The paper a little bit over claim their main observation and insights. The baseline methods, such as STA also claimed the unknown classes should be separated from the known ones. Though the proposed method is very different from STA as compared and discussed by the authors, the key insights are not brand new.

- Though the difference between the proposed method and STA is compared, it is still unclear why the proposed method by separating the unknown classes from the known ones in the feature space with an adversarial learning framework is superior to STA with a classifier. More insights, explanations, and analyses would be better.

- In the experiments, the implementation of other baseline methods is questionable. For example, the DANCE method only achieves very low performance on the Office-Home dataset with the EfficientNet backbone. It is unclear why the very low implementation results are achieved, leading to unreliable comparisons.
questions: + what are the key differences in the insights between the proposed method and previous methods?
+ Why do some baseline methods only achieve very low performance? the reliability of the experimental comparisons are questionable.


###################### After rebuttal ###################

The authors have addressed my main concerns. I will keep my original score as borderline accept.
limitations: No
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed an unknown-aware domain adversarial learning framework for open-set domain adaptation (OSDA), where the target domain has a larger label set than that of the source domain. This also means that the model needs to see novel class samples during the adaptation, which brings extra difficulty for domain adversarial learning. This work tackles this problem by introducing a three-dim discriminator for aligning the source (Ds) and target known (Tk) while separating the target unknown (Tu). The authors provide both theoretical and experimental analysis to support this type of domain adversarial learning and their methods are compared with state-of-the-art OSDA works.
strengths_and_weaknesses: Strengths:

1.This work proposed an unknown-aware domain adversarial learning (UADAL) framework to handle the OSDA problem and the idea of aligning Ds and Tk while separating Tu via explicit adversarial learning is new.

2.This work provides theoretic analysis of the optimization process for the proposed UADAL.

3.The experimental results show that the proposed UADAL can achieve better performance than the state-of-the-art OSDA methods.

Weaknesses:

1.The framework is based on the assumption that (as in L192) Tu will have higher entropy than Tk, which might be too strong to hold.

2.Domain adversarial learning is known to be unstable during optimization and lacks interpretability.

questions: 1.Regarding weakness 1, how can UADAL handle situations where: (1) Entropy(Tu) ≈ Entropy(Tk), under the case that there is a clear domain shift between the source and target domains; (2) Entropy(Tu) < Entropy(Tk), where there might exist target-unknown samples that yield lower entropy than the target-known ones and cause possibly negative effect when models are optimized via Eq. (11) and Eq. (12).

2.It is better to provide not just HOS but also OS, OS* and UNK for the Office-31 and Office-Home benchmarks (i.e., Table 1), as some recent and previous methods did not include HOS as the metric when reporting results.

3.Comparisons with some recent OSDA works are missing, including JPOT (IJCAI’20) on Office-31 and Office-Home and LGU (WACV’22) on VisDA.

References:

JPOT: Xu, et al. “Joint Partial Optimal Transport for Open Set Domain Adaptation,” IJCAI, 2020.

LGU: Baktashmotlagh, et al. “Learning to Generate the Unknowns as a Remedy to the Open-Set Domain Shift,” WACV, 2022.


###################### After rebuttal ###################

I would like to thank the authors' detailed comments which well addressed my concerns. I will keep my score unchanged.


limitations: The limitations stated in the supplementary materials, i.e., suppL281 – 289, are the limitations for the general DA problem, but not specifically for their own methods. It is the current paradigm of DA that cannot tackle streamlined data gathering and needs sufficient source domain categories, not just for this paper. More concrete limitations of the proposed methods in this paper are recommended included.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses open-set domain adaptation (ODA) problem, where target domain contains some unknown categories not seen in the source domain. Starting from the widely used adversarial training, the authors propose to introduce the known class recognition into it. Furthermore, the authors propose to deploy posterior inference to estimate the probability of belonging to unknown class. Thorough quantitative and analysis experiments prove the efficacy of the proposed method.
strengths_and_weaknesses: ## Strengths
1. Motivation of the paper is clear, that is, the unknown separation should be explicitly considered into the current adversarial training (more specifically STA [1]), which is the first contribution of the paper.
2. The probability of belonging to unknown is estimated by posterior inference, which is conducted on the whole target data. This avoid introducing extra hyperparameter to reject unknown categories as most existing ODA methods did. This is the second contribution.
3. The paper proposes to use $|\mathcal{C}_s| + 1$ -way classifier, which could directly lead to a decision boundary for unknown category. The deployed entropy minimization on it is expected to further improve the performance.
4. The experiments including both quantitative ones and other analysis, such as thresholding analysis, open-set recognition ability compared to other mechanism in Fig.6 and 7.


### Weakness

The method seems to be a little heavy, as the posterior inference demands access to all target data, and also the existing of two classifier heads ($E$ and $C$). Details in Questions below.


### Reference
[1]. Liu, Hong, et al. "Separate to adapt: Open set domain adaptation via progressive separation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

questions: 1. Did the authors try using only one classifier head $C$?

As the first $\mathcal{C}_s$ dimension of $C$ can be directly utilized in the same way as $E$ did. Having only one unified classifier may improve the performance since two classifier heads may have disagreement.

2. Ablation study of entropy minimization in Eq. 19.

My feeling about this entropy minimization is that it may improve performance a lot. It will be interesting to see the effect of this term.
limitations: Maybe one limitation is that the posterior inference takes much time if with large dataset.
review_title: null

Meta-review: metareview: This paper proposes a novel method called UADAL (Unknown-Aware Domain Adversarial Learning) for Open-Set Domain Adaptation (OSDA). Specifically, the proposed method performs source and target-known distribution alignment while simultaneously separating source and target-unknown distributions in the feature alignment procedure. In the OSDA, the idea of source and target-known distribution alignment while simultaneously separating the source and target-unknown distributions through explicit adversarial learning is novel. Furthermore, a theoretical analysis of the optimization process of the proposed UADAL is conducted. All three reviewers had similar positive comments on this paper and were satisfied with the authors' feedback. Thus the AC recommends it for acceptance. 
confidence: Certain
Decision: Accept