Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces the problem of domain adaptation under open set label shift (OSLS), where the class-conditional distributions p(x|y) are domain-invariant, and p(y) can change.  Domain Adaptation under label shift and positive-unlabeled (PU) learning can be all considered as special cases of OSLS.   This paper also provides the identifiablity of OSLS, including necessary condition (weak positivity) and sufficient conditions (strong positvity). Under the strong positivity condition, the OSLS problem can be broken into k PU problems.  The PU learning algorithms cannot scale to datasets with large number of classes because of error accumulation. This paper then proposes PULSE framework to solve this problem by exploiting the joint structure of the problem with source-class re-sampling. Experiments across 7 semi-synthetic benchmarks shows that the proposed PULSE consistently outperform OSDA baselines.
strengths_and_weaknesses: Strengths
- Open Set Label Shift (OSLS) an interesting problem, assuming p(x|y) is the same and p(y) is changing.
- Theories on identifiablity of the problem and convergence analysis
- New algorithm PULSE to solve the problem
- Good experimental results

Weakness
- All experimental results are semi-synthesized and are on relative small datasets, such as CIFAR-10, CIFAR-100. It is good to have results on some popularly used datasets for domain adaptation such as DomainNet.
- The writing is not easy to follow, the references (e.g., Algorithm 2&3) switch from main paper and supplementary materials without clear instruction. There are also some minor grammar errors including in the formula (line 153)
questions: - All experimental results are semi-synthesized and are on relative small datasets, such as CIFAR-10, CIFAR-100. It is good to have results on some popularly used datasets for domain adaptation such as DomainNet.
- Make the writing better

 
limitations: - The benchmarks used in this paper are all semi-synthesized, and thus lack proof from real-world application
- The paper mentions that "In future work, we hope to bridge the gap between the necessary and sufficient identifiability conditions."
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper scopes the common Open Set Domain Adaptation problem to specifically consider Open Set Label Shift (OSLS) problems, where $p(x|y)$ is constant, but the class proportions may change between source and target, and there may be a new unseen class added in testing. In this setting, the goal is to identify instances of the unseen class, while also performing adequately on the previously seen classes. The paper proposes a new framework, PULSE, which combines classical Positive and Unlabeled Learning techniques and label reweighting techniques in order to tackle this new problem. The method is empirically shown to perform well in a variety of OSLS problems, and theoretical analysis is conducted to create sufficient conditions for identifiability of the OSLS problem. 
strengths_and_weaknesses: * **Originality**
    To the best of my knowledge, this is the first time I have seen the scoping of the open set domain adaptation problem. By focusing purely on label shift, the authors are able to create some novel theoretical results as well as a highly performant framework for tackling the OSLS problem. In particular, their framework combines two standard techniques to great effect; class reweighting to handle the label shift and PU techniques to handle the open set nature of the problem. 
* **Quality**
    The claims are well-substantiated both theoretically and empirically, and the results are impressive. The authors perform a detailed evaluation on existing open source domain adaptation methods as well as a slight ablation study by performing standard PU techniques without any label reweighting. 
* **Clarity**
    The paper is mostly well organized and written. A worthwhile addition in the related work section would be to make it more clear exactly who your setting differs from each of the main categories that are defined there. Further, it may be worth a few sentences relating why identifiability is useful and related to the remainder of the paper; Section 4 felt significantly out of place compared to the rest of the paper.
* Some minor nitpicks
     * line 48 - matrix submatrix appears to be a typo. 
     * line 257 should likely be a heading
* **Significance**
    By scoping the OSDA problem to focus on label shift, the authors were able to show relatively significant gains compared to standard OSDA methods - this kind of scoping appears quite fruitful for other researchers to build off of. The theoretical contributions in the paper are strong evidence that their framework, PULSE, is able to perform quite well on a variety of scenarios which leaves it as a valuable benchmark for future work. 
questions: * How robust is PULSE expected to be in general OSDA problems, not just label shift?
limitations:    It would be worthwhile to have a further discussion on limitations of PULSE with regards to the amount of labeled data of the target domain - for example, with no labels it would be difficult to approximate what the label shift is. It could also be useful to discuss the types of models or datasets that PULSE is expected to work on (e.g. does it perform worse as the number of classes increase, the computational efficiency with regards to larger models, etc.); and that it inherits limitations from its particular stages (e.g. any limitations of importance weighted label shift or CVIR/BBE).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The goal of this paper is to solve the open set domain adaptation under the label shift setting. The authors proposed a PU learning-based framework to first estimate the label shift and then classify the novel class. Moreover, the authors also gave sufficient and necessary conditions to the open set label shift in order to make the target label marginal identify. The experimental results showed that the PULSE framework could achieve great performance improvement. 
strengths_and_weaknesses: 1) The idea of combing PU problems with the OSLS is interesting, the authors used an ingenious way to merge the PU problems into the OSLS. The reduction from the k-PU to a single PU is attractive (equation 3).

2) The definitions and theorems are straightforward and easy to be understood, and the mathematics proof seems solid. 

3) The author used a two-stage method to separately estimate the label shift and to identify the novel class, which seems effective in solving both the domain adaptation and novel class identification. The results of the PULSE seems good.

4) I have no doubts in the originality to the best of my knowledge, the quality of this paper is good, some more details need to be discussed for clarity as I stated later, and the contribution of this paper has some significance to the OSLS regime.

questions: 1) The authors need to better organize the paper and improve the mathematical notations, e.g. in line 48 "the matrix submatrix" is not so clear.


2) For the extended BBE algorithm 2, which is the importance weight w? ($\hat{c}_j$ I assume), the author should be more clear.

3) Since the key point in this paper is to estimate the target label marginal distribution using source data, this process relies heavily on Theorem 1 to guarantee the target domain estimation on the source domain. As we can find from Theorem 1, the threshold would be very important, so the author should discuss more about the threshold e.g. is that a hyperparameter? Or is there any trade-off between varying the threshold with the performance on target data? I also wonder that is that directly using the source classifier for target in Algorithm 2 line 6 is guaranteed by this Theorem?

4) Since the PULSE framework is built on the strong positive condition to guarantee the target inevitability, I found no regularization for the data in this paper, so the author also needs to explain why their algorithm can achieve this strong positive condition.

5) In algorithm 2 , the author should expand the loss function in line 6 and line 10 to make it more clear. And The author should explain more on the connection between the $\alpha$ with the novel target and why they are equal and can be derived this way (why don't use the estimation from algorithm 2 but use algorithm 4 instead in algorithm 3, however, I found the resample based estimation should be derived from algorithm 2 as described in line 805).

6) Both in algorithm 2 and algorithm 4, the z is ambitious in the paper, are they discrete or continuous variables, and do we need to compute all of the values from 0 to 1 for estimation? Will that increase the computation burden? Also, for the c in algorithm 2 line 6, do they have the connections with the threshold in Theorem 1 since their notation is all c? 

7) How about the results from the general domain adaptation datasets like USPS, OFFICEHOME, DOMAINNET etc. The experiment setup is not convincing, the random choice for source and target in small datasets (like cifar) may make the source and target only have the limited domain discrepancy, which in return limit the domain adaptation problems back into the novel class detection problems under PU setup. Since the domain discrepancy, the author proposed in the paper is mostly between the positive samples and the unlabeled samples, and unlabeled samples are consists of the combination of both positive and negative samples as described in CVIR, I have little concern on if this problem would degrade into open-set classification (since the domain discrepancy is limited) instead of the open-set domain adaptation as we can find in the experiments.

8) In the paper line 188 section 5, why only satisfying the strong positive condition the OSLS can be reduced into K PU? It seems most of the OSLS problem can all be reduced into K PU problems that way in equation 2.
limitations: 1) This paper needs to be well organized.
2) Some details need to be completed as described in the questions.
3) The experiments in this paper is not convincing.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces domain adaptation under Open Set Label Shift (OSLS). Specifically, it assumes the target has one more novel class that was not previously seen in the source domain while allowing label shifts between source and target domains. This work provides theoretical findings of OSLS, specifically, the necessary condition "weak positivity" and two sufficient conditions "strong positivity" and "separability". The author further proposes a framework to solve OSLS, named PULSE which combines techniques from both areas of (1) positive and unlabeled learning and (2) DA under label shift. The effectiveness of the proposed methods is demonstrated in language, image, and medical datasets. 
strengths_and_weaknesses: originality
---
The paper focuses on a special case of the open set domain adaptation where the target domain has one novel class that is not previously unseen in the source domain. The big concept of open set DA is not novel while the special case is well studied yet, thus can be considered novel. The paper delivers an identifiability analysis of the OSLS which is novel. 

quality
---
The paper provides sold theoretical analysis as well as extensive experiments on 5 datasets across multiple application domains. 


Minor issue, typos:

1. Line 46: … the matrix submatrix… 
2. Line 67: double periods after "(Sec.7)" 
3. Line 154: \frac{1}{p_t(y=k+1)} is missing in the closed form of p_t(x|y=k+1)
4. The paper seems not define what the metric "novel prevalence estimation" is. 

clarity
---
The paper has many contents and is pretty dense. The majority of the paper is well-written and clear. However, I find the method section a bit obfuscated. One reason is that there is no headings or subsection title to remind the reader what is the focus of each paragraph. Another reason is that the author seems to jump to the details too quickly. The logic of the current text is kind of linear. I would like to suggest the author organize section 6 and each of its subsections to have an overview-and-details structure. 

significance
---
The paper does a good job on the specific problem it aims to solve. The analysis is solid and the empirical results are illustrative. My main concern is that the paper's significance is constrained by the practicality of the proposed problem. The problem setting is a bit artificial to me. I am not sure how practical it is that we assume the target domain has exactly one novel class unseen. The experiment is kind of synthetic/semi-synthetic since the author chooses source and novel classes randomly. 
questions: 1. Does OSLS apply to medical applications such as COVID diagnosis?

The author motivates the OSLS problem by giving a motivating example in the medical domain: the disease proportion changes seasonally and sometime may come with a new disease, like COVID-19. I would like the author to analyze whether the OSLS is suitable for the disease diagnosis problem. For example, let us consider a DA problem where the source domain has two classes, normal people and flu patients while the target domain will have one more class, COIVD patients. How are your identifiability conditions satisfied in this application? Does the PLUSE framework still apply?

2. Does all the 5 tasks in the experiments satisfy "strong positivity"? Could the author include benchmarks not satisfying this condition?

The author claims that "we observe that techniques proposed under strong positivity were empirically stable and outperform other baselines developed under separability. This is intuitive for many benchmark datasets where for each class there exists a subdomain that only belongs to that class." First I want to ask, do all 5 benchmarks used in your experiments all satisfy that there exists a subdomain that only belongs to that class? Second, I would like to point out that samples belonging to different classes are disjoint is the property of object recognition tasks however, there are many other types of tasks that do not satisfy such disjoint property. For example, in age prediction (classify different age groups). It is unlikely that there exist samples in the age group [20,30)'s portrait will never appear in the age group [30,40) or [10,20) since the boundary between different classes is not perfectly clear here. Thus I think it is necessary to include other benchmarks that do not satisfy "strong possibility" for the sake of a more complete evaluation of the proposed method. 

limitations: This work does not raise potential negative societal impact.
review_title: null

Meta-review: metareview: The paper addresses an interesting domain adataption question and proposes an novel and elegant
solution supported with relevant theory. Although some issues have been raised, all reviewers agree
that the paper worth be published, and we expect the authors to take into account the comments
of the reviewers (eg discussing limitations of PULSE, checking positivity conditions...)
confidence: Less certain
Decision: Accept