Domain Adaptation under Open Set Label Shift
Authors: Saurabh Garg
Sivaraman Balakrishnan
Zachary Chase Lipton
Venue: NeurIPS 2022 Accept (2022)
Keywords: Domain Adaptation
Label shift
PU learning
deep learning
open set domain adaptation
deep learning

Abstract: We introduce the problem of domain adaptation under Open Set Label Shift (OSLS), where the label distribution can change arbitrarily and a new class may arrive during deployment, but the class-conditional distributions $p(x|y)$ are domain-invariant. OSLS subsumes domain adaptation under label shift and Positive-Unlabeled (PU) learning. The learner's goals here are two-fold: (a) estimate the target label distribution, including the novel class; and (b) learn a target classifier. First, we establish the necessary and sufficient for identifying these quantities. Second, motivated by advances in label shift and PU learning, we propose practical methods for both tasks that leverage black-box predictors. Unlike typical Open Set Domain Adaptation (OSDA) problems, which tend to be ill-posed and amenable only to heuristics, OSLS offers a well-posed problem amenable to more principled machinery. Experiments across numerous semi-synthetic benchmarks on vision, language, and medical datasets demonstrate that our methods consistently outperform OSDA baselines, achieving $10$--$25\%$ improvements in target domain accuracy. Finally, we analyze the proposed methods, establishing finite-sample convergence to the true label marginal and convergence to optimal classifier for linear models in a Gaussian setup. Code is available at https://github.com/acmi-lab/Open-Set-Label-Shift. 

Domain Adaptation under Open Set Label Shift
Saurabh Garg, Sivaraman Balakrishnan, Zachary C. Lipton
Machine Learning Department,
Department of Statistics and Data Science, Carnegie Mellon University
sgarg2
sbalakri zlipton Candrew edu
Abstract
We introduce the problem domain adaptation under Open Set Label Shift (OSLS) where the label distribution can change arbitrarily and new class may arrive dur- ing deployment, but the class-conditional distributions p(z(y) (1) are comain-invariann OSLS subsumes domain adaptation under label shift and Positive- Unlabeled (PU) learning. The learner's goals here are two-fold: (a) estimate the target label distri- bution, including the novel class; and (b) learn target classifier. First, establish necessary and sufficient conditions for identifying these quantities. Second, moti- vated by advances in label shift and PU learning, propose practical methods for both tasks that leverage black-box predictors. Unlike typical open set domain adaptation problems, which tend to be ill-posed and amenable only to heuristics, OSLS offers a well-posed problem amenable to more principled machinery. Ex- periments across numerous semi-synthetic benchmarks on vision, language, and medical datasets demonstrate that our methods consistently outperform open set domain adaptation baselines, achieving 10 (0-25% improvements target domain accuracy. Finally, we analyze the proposed methods, establishing finite- sample convergence the true label marginal and convergence optimal classifier for linear models in Gaussian setup
1 Introduction
Suppose that we wished deploy a machine learning system recognize diagnoses based on their clinical manifestations. If the distribution of data were static over time, then we could rely on the standard machinery of statistical prediction. However, disease prevalences are constantly changing. violating the assumption of independent and identically distributed (iid) data. In such scenarios, we might reasonably apply the label shift assumption, where prevalences can change but clinical manifestations cannot. When only the relative proportion of previously seen diseases can change, principled methods can detect and correcting for label shift on the fly [56] 45 27). But what if new discase, like COVID-19, were to arrive suddenly?
Traditional label shift adaptation techniques break when faced with a previously unseen class, distinct literature on Open Set Domain Adaptation (OSDA) seeks to handle such cases [51 5 14] 70
59 251). Given access labeled source data and unlabeled target data, the goal in OSDA is to adapt classifiers in general settings where previous classes can shift in prevalence (and even appearance), and novel classes separated out from those previously seen can appear. Most work OSDA is driven by the creation of and progress on benchmark datasets (e.g., DomainNet, OfficeHome). Existing OSDA methods are heuristic in nature, addressing settings where the right answers seem intuitive but are not identified mathematically. However, absent assumptions on: (i) the nature of distribution shift among source classes and (ii) the relation between source classes and novel class, standard impossibility results for domain adaptation condemn us to guesswork [8).
43
73
58
Code is available at htttp::/ggiti http://hw.....ooeeeeaaaaaaaaeaaa Set--pee--------ae--------------
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Target Classification
Novel Prevalence Estimation
Source Domain
Target Domain
0.25
04
Distribution
0.
Absglute Entr
02
Absent
0.2
0.15
Dog
Cat
Deer
?
Shiff
Dog
Cat
Deer
Ape
0.30
Invariant p(xly)
0.00
Best OSDA
kPU
Labeled data p,(x,y)
Unlabeled data p,(x)
Domain
PULSE (Ours)
Domain Adaptation under OSLS. An instantiation of OSDA that applies label shift assumption but allows for new class to show up in target domain. Right: Aggregated results across seven semi-synthetic benchmark datasets. For both target classification and novel class prevalence estimation, PULSE significantly outperforms other methods (lower error better). For brevity, we only include result for the best OSDA method. For detailed comparison, refer Sec.
Figure I: Left:
In this work, we introduce domain adaptation under Open Set Label Shift (OSLS), a coherent instantiation of OSDA that applies the label shift assumption but allows for new class show up in the target distribution. Formally, the label distribution may shift between source and target (a))) # pt()). but the class-conditional distributions among previously seen classes may not (i.e.. Vy E (1,2.... kj, (..p((xlv) = Pt (zly)). Moreover, a new class y = k + 1 may arrive in the target period. Notably, OSLS subsumes label shift [56]
(when (when (y k 1) 0) and learning from 1). As with label shift and PU learning, our goals are two-fold. Here, we must (i) estimate the target label distribution pt (y) (including the novel class prevalence); (ii) train (k- 1)y target-domain classifier.
67
Positive and Unlabeled (PU) data (20] 42 24
First, we characterize when the parameters of interest identified (Sec. Namely, define (necessary) weak positivity condition, which states that there exists a subset of each label's support that has zero probability mass under the novel class and that the submatrix of p(zlyy consisting only of rows in that subset is full rank. Moreover, we prove that weak positivity alone is not sufficient. We introduce two sufficient conditions: strong positivity and separability, either of which (independently) ensures identifiability.
Focusing cases with strong positivity, show that OSLS reduces l PU learning problems (Sec. 5) However, we demonstrate that straightforward applications of this idea fail because (i) bias accumulates across the k mixture proportion estimates leading to grossly underestimating the novel class's prevalence; and (ii) naive combinations of the k: PU classifiers are biased and inaccurate.
Thus motivated, propose the PULSE framework, which combines methods from Positive and Unlabeled learning and Label Shift Estimation, yielding two-stage techniques for both label marginal estimation and classification (Sec. 6). Our methods build recent ad in label shift [45] 4
and PU learning 38 35 29 that leverage appropriately chosen black-box predictors to avoid the curse of dimensionality. PULSE first estimates the label shift among previously seen classes, and then re-samlles the source data to formulate a single PU learning problem between (reweighted) source and target data to estimate fraction of novel class and to learn the target classifier. In particular, our procedure builds on the BBE and CVIR techniques proposed in Garg et al. (29). PULSE is simple to implement and compatible with arbitrary hypothesis classes (including deep networks).
We conduct extensive semi-synthetic experiments adapting seven benchmark datasets spanning vision (CIFARIO, CIFARI00, Entity30), natural language (Newsgroups-20), biology (Tabula Muris), and medicine (DermNet, BreakHis) (Sec. 77). Across numerous data modalities, draws of the label distributions, and model architectures, PULSE consistently outperforms generic OSDA methods, improving by 10-25% in accuracy on target domain. Moreover, PULSE outperforms methods that naively solve & PU problems on both label distribution estimation and classification.
Finally, we analyze our framework (Sec. g). First, we extend Garg et al. [2 's analysis of BBE to derive unite-sample error bounds for our estimates of the label marginal. Next, we develop new analyses of the CVIR objective (2] that PUL.E relies the classification stage. Focusing on Gaussian setup and linear models optimized by gradient descent, we prove that CVIR converges to

---

a true positive versus negative classifier in population. Addressing the overparameterized setting where parameters exceed dataset size, we conduct an empirical study that helps to elucidate why, on separable data, CVIR outperforms other consistent objectives, including nnPU [88 and uPU [21].
Related Work
(Closed Set) Domain Adaptation (DA) Under DA, the goal is adapt a predictor from a source distribution with labeled data to target distribution from which we observe only unlabeled
examples. DA classically explored under two distribution shift scenarios [67] (i) Covariate
shift 77
74
19
18
32 where p(y remains invariant among source and target; and (ii) Label
77 where p(xl)) shared across source and target. In these settings most theoretical analysis requires that the target distribution's 's support is subset of the source support g However, recent empirically work in DA 448 68| 69
shift
56
45
27
80 79 26 64 focuses on settings motivated by benchmark datasets (c.g., WILDS [57 39 Office-31 31 [5S OfficeHome [71]. DomainNet (52))
where such overlap assumptions are violated. Instead, they rely on some intuitive notion of semantic equivalence across domains. These problems are not well-specified and in practice, despite careful hyperparameter tuning. these methods often do not improve over standard empirical risk minimization on source data alone for practical, and importantly, previously unseen datasets (57).
62 extends DA to settings where along with distribution shift among previously seen classes, we may observe novel class in the target data. This setting is also known as universal domain adaptation 73 [5]] Rather than making precise assumptions about the nature of shift between source and target as in OSLS, the OSDA literature is primarily governed by semi-synthetic problems on benchmark DA datasets (c.g. DomainNet, Office- 31 and OfficeHome). Numerous OSDA methods have been proposed (5 14 70
Open Set Domain Adaptation (OSDA) OSDA [IT
43 73
58 59 25 [T].
At a high level, most OSDA methods perform two steps: (i) align source and target representation for previously seen classes; and (ii) train a discrimination to reject novel class from previously seen classes. The second step typically uses novelty detection heuristics to identify novel samples.
PU Learning Positive and Unlabeled (PU) learning is the base case of OSLS. Here, we observe labeled data a single source class and unlabeled target data contains data from both the novel class and the source class. In PU learning, our goals are: (i) Mixture Proportion Estimation (MPE), i.e., determining the fraction of previously seen class in target: and (ii) PU classification, i.e., learning to discriminate between the novel and the positive (source) class. Several classical methods were proposed for both MPE 224 23 63 36
and classification 24 22 21]. However, classical MPE methods do not scale to high- ligh-dimensional settings (53). More recent methods alleviate these issues by operating in classifier output space 29 35). For classification, traditional methods fail when deployed with models classes with high capacity due to their capacity of fitting random labels (75). Recent methods 22 38 [6]. avoid over-fitting by employing regularization or self-training techniques.
Other related work A separate line of work looks at the problem of Out-Of- Df-isrr-iiiiittonnnn (OOD) detection 34
50 76. Here, the goal is to identify novel examples, i.e., samples that lie out of the support of training distribution. The main different between OOD detection and OSDA is that OOD detection we do not have access to unlabeled data containing a novel class. Recently, Cao et al. 13 proposed open- wwrld semi-supervised learning, where the task is to not only identify novel classes in target but also to separate out different novel classes in an unsupervised manner.
Our work takes step back from the hopelessly general OSDA setup, introducing OSLS, well- posed OSDA setting where the sought-after parameters can be identified.
3 Open Set Label Shift
Notation For a vector v e IRd, we use uy to denote its jth entry, and for an event E, we let 11 [[]] denote the binary indicator of the event. By AA,, we denote the cardinality of set A.
Let X' e Rd be the input space and y 21, k ++1} 1} be the output space for multiclass classifica- tion. Let P. and P, be the source and target distributions and let ps and p. denote the corresponding probability density (or mass) functions. By E. and E, we denote expectations over the source and target distributions. We assume that we are given loss function E: A x y - TR, such that {(2, y) is the loss incurred by predicting 2 when the true label is y. Unless specified otherwise, we assume that E
3

---

is the cross entropy loss. As in standard unsupervised domain adaptation, we are given independently and identically distributed (jid) samples from labeled source data {[z), V1). (2g, 3/2...... (In, 3n)}
Pn and iid samples from unlabeled target data 1 .................. #n+mm P
Before formally introducing OSL.S, we describe label shift and PU learning settings. Under label shift, we observe data from k classes in both source and target where the conditional distribution remain invariant (i.e., Ps Pa yy) = Pr (x for all classes y E [1, k)) but the target label marginal may change (i.e., (v) * Pa (v)). Additionally, for all classes in source have a non-zero support, i.e., for all y é [1. R.. py(y) > c, where e > O. Under PU learning, we possess labeled source data from a positive class and unlabeled target data from mixture of positive and negative class with a goal of learning positive- www.eeerruuuaetteeeeeee
classifier on target. We now introduce the OSLS setting:
Definition 1 (Open set label shift). Define Yr+ = y and Ys MM + 1}. Under OSLS, the label distribution among source classes Y. may change but the class conditional p(æly) for those clusses remain invariant between source and target, and the target domain may contain a novel class, i.e.
Palcly 5) po(xly= =
Vj
and
paly = 1)
(1)
Additionally, we have non-zero support for all k (previously-seen) labels in the source distribution, for all E y., Po(y)= e for some c > 0.
Note that the label shift and PU learning problems can be obtained as special cases of OSLS. When no novel class is observed in target (i.e., when pv(y = k + 1) = 0), we recover the label shift problem, and when we observe only one class in source (i.e., when k = 1). the OSLS problem reduces to PU learning. Under OSLS, our goal naturally breaks down into two tasks: (i) estimate the target label marginal (r(y) for each class y E Y: (ii) train a classifier f X A to approximate pe (y)zz).
4 Identifiablity of OSLS
We now introduce conditions for OSLS, under which the solution is identifiable. Throughout the section, we will assume access to population distribution for labeled source data and unlabeled target data, i.e., p. (z, y) and (xx) is given. To keep the discussion simple, we assume finite input domain X' which can then be relaxed to continuous inputs. We relegate proofs to App. B
We first make a connection between target label marginal pv()) estimation and learning the tar- get classifier Pt (ylz) showing that recovering (v(y) is enough to identify Pr pe(y/zr). In population, given access (y). the class conditional pr(z = k + 1) can be obtained in closed form as (p=)+)),) (r (r)=))) (flv==t)/pp( 1). We can then apply Bayes rule to obtain (v(()r). Henceforth, we will focus our discussion on identifiability of p. (y) which implies identifiability of pt p.(y)ar). In following proposition, we present weak positivity, necessary condition for pv(y) to be identifiable.
Proposition 1 (Necessary conditions). Assume (b))) > for all y E Yt. Then po(y) is identified only if pe (aly = + 1) and pa px(p))) for all y E Vs satisfy weak positivity, i.e,, there must exists a subdomain X = X such that:
(i) (Xmplyly = le k+ + O; and (ii) the matrix (ps(xly)) (ox,,)) is full column-rank.
Intuitively, Proposition [ states that if the target marginal doesn't ' lie on the vertex of the simplex A then their must exist subdomain Xwp where the support of novel class is zero and within Xwp- Pe(y) for source classes is identifiable. While it may seem that existence of subdomain X is enough, we show that for the OSLS problem, existence doesn't 't imply uniqueness. In App. B we construct an example, where the weak positivity condition is not sufficient. that example, we show that there can exist two subdomains Xwp and X satisfying weak positivity, both of which lead to separate solutions for Pe(y). Next, we extend weak positivity to two stronger conditions, either of which (alone) implies identifiability.
Proposition (Sufficient conditions). The target marginal pe(y) is identified for all y e Yi 1}. Po(0ly= = k + 1) and ps (xly) satisfy either:
(i) Strong positivity, i.e., there exists Xyp = X such that pe (xply = k 1) and the matrix [p.(aly)) me wew. is full-rank and diagonal; or

---

(ii) Separability, i.e., there exists Xee C 2, such that pel Pepps = h + 1) = 0, (x = and the matrix (catri(ppxx1)))) (e)( Xe,, full column- rank.
Strong positivity generalizes the irreducibility condition [TQ], which sufficient for identifiability under PU learning, to k PU learning problems. Note that while the two conditions in Proposition 2 overlap, they cover independent set of OSLS problems. Informally, strong positivity extends weak positivity by making additional assumption that the matrix formed by p(zly)) inputs in Xwp is diagonal and the separability assumption extends the weak positivity condition to the full input domain of source classes instead of just Xwp. Both of these conditions identify support region of X' which purely belongs to source classes where we can either individually estimate the proportion of each source classes (i.e., under strong positivity) or jointly estimate the proportion (i.e., under separability).
To extend our identifiability conditions for continuous distributions, the linear independence condi- tions on the matrix [ps (=:lt)) (uslyl/ex........ 1e....e., has the undesirable property of being sensitive to changes on sets of measure zero. We may introduce stronger notions of linear independence as in Lemma 1 of Garg et al. (27). We discuss this in App. B.2
Reduction of OSLS to PU Problems
Under the strong positivity condition, the OSLS problem can be broken down into A PU problems as follows: By treating a given source class y, E V/ as positive and grouping all other classes together as negative observe that the unlabeled target data is then a mixture of data from the positive and negative classes. This yields a PU learning problem and the corresponding mixture proportion is the fraction pyy j) (proportion of class yj) among the target data. By iterating this process for all source classes, we can solve for the entire target label marginal Pe(y). Thus, OSLS reduces to instances PU learning problem. Formally, note that Pz(x) can be written as:
pcly -))y= (r(x)) pv (xly = 5) + pr -)) (Enerro) 1 - (aly - =i)
(2)
individually for all j E Ve. By repeating this reduction for all classes, we obtain separate PU learning problems. Hence, natural choice is to leverage this structure and solve k: PU problems to solve the original OSLS problem. In particular, for each class j E Ve. we can first estimate its prevalence fy j) in the unlabeled target. Then the target marginal for the novel class is given by p:(y = + = 1- Σ (1 = i). Similarly, for classification, we can train & PU learning classifiers f.. where fi is trained to classify source class versus others in target. An example is classified as belonging to the class y = k + 1, if it rejected by all classifiers fi as other in target. We explain this procedure more formally in App. A.I
This reduction has been mentioned in past work [60 72). However, to the best of our knowledge, no previous work has empirically investigated both classification and target label marginal estimation jointly. Sanderson and Scott (60 focuses only on target marginal estimation for tabular datasets and Xu et al. [72] assumes that the target marginal is known and only trains & separate PU classifiers.
In our work, we perform the first large scale experiments evaluate efficacy of the reduction of the OSLS problem to k-PU problems. With plugin state-of- sfute-of-the-and art PU learning algorithms, we observe that this naive reduction doesn't it seale datasets with large number of classes because of error accumulation in each of the k MPEs and k one- oee-versus-other PU classifiers. To mitigate the error accumulation problem, we propose the PULSE framework in the next section.
The PULSE Framework for OSLS
We begin with presenting our framework for OSLS problem under strong positivity condition. First, we explain the structure OSLS that we leverage in PULSE framework and then elaborate design decisions we make to exploit the identified structure.
Overview of PULSE framework Rather than simply dividing each OSLS instance into PU problems, we exploit the joint structure of the problem to obtain single PU learning problem. To begin, we note that if only we could apply a label shift correction to source, i.e., re-sample source classes according to their relative proportion in the target data, then could subsequently consider
5

---

Algorithm/ 1 Positive and Unlabeled learning post Label Shift Estimation (PULSE) framework
input- Labeled source data (X$ yy) and unlabeled target samples X
1: Randomly split data into training (Xf . Xf and hold out partition (x X3.
2: Train source classifier fa on labeled source data (Xf,
Pely = j)
3: Estimate label shift ( = j) Eksy, P. k)
using Algorithm 2 and hence importance ratios
((j) among source classes j e V
4: Re-sample training source data according label distribution f get (X, yi) and (X y2). [ train a discriminator fa and estimate novel class fraction (r = k + 1).
5: Using Algorithm
a()) [f](e]];
6: Assign [f((=)]; = (fa(z)) Ekey, @(k) (10)][fa66 [f((a] for all j E V. and [f((z)]k+1 -1 fa(z). And
for all j E Vv. assign (y = j) = (1 Pr = k + 1)) Fb = j). output Target marginal estimate P. E A and target classifier fe(-) E A*
the unlabeled target data as mixture of (i) the (reweighted) source distribution; and (ii) the novel class distribution (i.e., Pe (aly k + 1)). Formally, we have
Pe(x)) Σ puly (xly = j) Σ pr - Po Pa JEyi
+ pa - k 1)p 1)
Pa -
eey,
= =L k + ()), + = k + +
1),
(3)
where pz() is the label-shift-correeeed source distribution, i.e., p1(()) = Liey, w())ps j). p:ly-t))/Pp p/ = j) for all j E V.. Intuitively, p4(() pe (v p. ky where vn(j) = ( is re-normalized label distribution in target among source classes and w(j)'s are the importance weights. Hence, after applying a label shift correction to the source distribution pl()),, we have reduced the OSLS problem to single PU learning problem, where px((x) plays the part of the positive distribution and pt (xly = k + 1) acts as negative distribution with mixture coefficients 1- --rr k 1) and (y k + 1) respectively. We now discuss our methods (i) to estimate the importance ratios w( wyy); and (ii) to tackle the PU learning instance obtained from OSLS.
Target marginal estimation among source classes While traditional methods for estimating label shift breakdown in high dimensional settings (78), recent methods exploit black-box classifiers to avoid the curse of dimensionality [45] 4 D. However, these recent techniques require overlapping label distributions, and a direct application would require demarcation of samples from p', (x) sub-population in target, creating cyclic dependency. Instead, to estimate the relative proportion of previously seen classes in target, we leverage the is PU reduction described in Sec. 5 with two erucial distinctions. First, we normalize the obtained estimates of fraction previously seen classes to obtain the relative proportions in p'.(v). ) In particular, we do not leverage the estimates of previously seen class proportions in target directly estimate the proportion of novel class which avoids issues due to error accumulation. Second, we exploit a k-way source classifier fa trained on labeled source data instead of training -oee PU classifiers. We tailor the recently proposed Best Bin Estimation (BBE) technique from Garg et al. [29]. We describe the modified BBE procedure in App. (Algorithmm 2) After estimating the relative fraction of source classes in target (i.e., f((())= = (v (1/))//Kyyy-- E/yy, (== for all j E Y.,,, we re-sample the source data according to (()y) to mimic samples from distribution (x).
Label shift correction:
PU Learning: Separating the novel class from previously seen classes After obtaining a PU learning problem instance, we resort to PU learning techniques (i) estimate the fraction of novel class p. = k+ 1+1); and (ii) learn a binary classifier fa(ı) to discriminate between label shift corrected source (x) and novel class p. (aly = k + 1). With traditional methods for PU learning involving domain discrimination, over perrramaameteeeeed models can memorize the positive instances in unlabeled, assigning them confidently to the negative class, which can severely hurt generalization on PN data [38 29.. Rather, we employ Conditional Value Ignoring Risk (CVIR) loss proposed in Garg et al. 29 which was shown to outperform alternative approaches. First, we estimate the proportion of novel class Pr(yy) = k + 1) with BBE. Next, given an estimate Pr = k + 1), CVIR objective discards the highest loss (1 - -P(yy k+ + 1)) fraction of examples on each training epoch, removing the incentive to overfit to the examples from p' (z). Consequently, we employ the iterative procedure that alternates

---

between estimating the prevalence of novel class Pr(y = k + 1) (with BBE) and minimizing the CVIR loss with estimated fraction of novel class. We detail this procedure in App. C (Algorithm 3
Combining PU learning and label shift correction Finally, to obtain a ( -wayy classifier f((x)
on target we combine discriminator fa and source classifier f. with importance--rewiigteed label shift correction. In particular, for all j E V.. [fa(x)]; = (fa(z)) L,cy, (<)
"(])[[f(()],, and [fr =
1 - ff(x)) Overall, our approach outlined in Algorithm I proceeds as follows: First, we estimate the label shift among previously seen classes. Then we employ importance re-weighting of source data to formulate a single PU learning problem to estimate the fraction of novel class (y = k + 1) and to learn a discriminator fa for the novel class. Combining discriminator and label shift corrected source classifier we get (A: + 1) -way target classifier. We analyse crucial steps in PULSE in Sec. g
Our ideas for PULSE framework can be extended to separability condition since continues to hold. However, in our initial experiments, we observe that techniques proposed under strong positivity were empirically stable and outperform methods developed under separability. This is intuitive for many benchmark datasets where it is natural to assume that for each class there exists a subdomain that only belongs to that class. We describe this in more detail in App. C.T
7 Experiments
Baselines We compare PULSE with several popular methods from OSDA literature. While these methods are not specifically proposed for OSLS, they are introduced for the more general OSDA problem. In particular, we make comparions with DANCE [59], UAN 773. CMU [25]. STA 46.. Backprop-ODX (or BODA) (58 We use the open source implementation available at https://gi thubb. com/ thuml For alternative baselines, we experiment with source classifier directly deployed on the target data which may contain novel class and label shift among source classes (referred to as source-only). We also train a domain discriminator classifier for source versus target (referred to as domain disc.). This is adaptation of PU learning baseline 24] which assumes no label shift among source classes. Finally, per the reduction presented in Sec. 5 we train k PU classifiers (referred to as k-PU). We include detailed description of each method in App. F.I.
Datasets We conduct experiments with seven benchmark classification datasets across vision, natural language, biology and medicine. For each dataset, we simulate an OSLS problem as described in next paragraph. For vision, we use CIFAR1O, CIFAR100 40) and Entity30 [61]. For language, we exper- iment with Newsgroups-20 (http: ////nee..ee..m com/ j jasiaaanneessiiieeeea /0Newsgrouss dataset. Additionally, inspired by applications of OSLS in biology and medicine, we experiment with Tabula Muris [17] (Gene Ontology prediction). Dermnet (skin disease prediction htttps://dernnee
org/). and BreakHis [66] (tumor cell classification). These datasets span language, image and table modalities. We provide interpretation of OSLS problem for each dataset along with other details in App. F.2
OSLS Setup To simulate an OSLS problem, we experiment with different fraction of novel class prevalence, source label distribution, and target label distribution. We randomly choose classes that constitute the novel target class. After randomly choosing source and novel classes, we first split the training data from each source class randomly into two partitions. This creates a random label distribution for shared classes among source and target. We then club novel classes to assign them a new class (i.e. k + 1). Finally, we throw away labels for the target data to obtain an unsupervised DA problem. We repeat the same process on iid hold out data to obtain validation data with no target labels.
Training and Evaluation We Resnet18 (33) for CIFARIO, CIFAR100 100, and Entity30. For newsgroups, we use a convolutional architecture. For Tabular Muris and MNIST, we use a fully connected MLP. For Dermnet and BreakHis, we use Resnet-50. For all methods, we use the same backbone for discriminator and source classifier. For kPU, we use a separate final layer for each class with the same backbone. We use default hyperparameterss for all methods. For OSDA methods, we use default method specific hyperparameters introduced in their works. Since OSDA methods do not estimate the prevalence of novel class explicitly, we use the fraction of examples predicted in class k + 1 as a surrogate. We train models till the performance on validation source data (labeled) ceases to increase. Unlike OSDA methods, note that we do not use early stopping based on performance on held-out labeled target data. To evaluate classification performance, we report target accuracy on all classes, seen classes and the novel class. For novel class prevalence estimation, we report absolute difference between the true and estimated marginal. We open-source our code and by simply
7

---

Table 1: Comparison of PULSE with other methods. Across all datasets, PULSE outperforms alternatives for both target classification and novel class prevalence estimation. Ace (All) is target accuracy, Acc (Seen) target accuracy on examples from previously seen classes, and Acc (Novel) is recall for novel examples. MPE (Novel) is absolute error for novel prevalence estimation. Results reported by averaging across 3 seeds. Detailed results for each dataset with all methods in App.
CIFAR-10
C-FAR-100
Acc (All)
Acc (Seen)
Acc (Novel)
MPE
Acc (All)
Acc (Seen)
Acc (Novel)
MPE (Novel)
Method
(Novel)
Source-Only
67.1
87.0
46.6
66.4
UAN A
15.4
19.7
25.2
0.214
18.1
40.6
14.8
0.133
BODA 58
63.1
66.2
42.0
0.162
36.1
17.7
81.6
0.41
DANCE
59
70.4
85.5
14.5
0.174
47.3
66.4
1.2
0.28
STA 46
57.9
69.6
14.9
0.124
42.6
48.5
34.8
0.14
CMU 26
62.1
77.9
41.2
0.183
35.4
46.0
15.5
0.161
Domain Disc.
24
47.4
87.0
30.6
0.331
45.8
66.5
39.1
0.046
k-PU
83.6
79.4
98.9
0.036
36.3
22.6
99.1
0.298
PULSE (Ours)
86.1
91.8
88.4
0.008
63.4
67.2
63.5
0.078
Entity30
Newsgroups20
Tabula Muris
BreakHis
DermNet
Acc
MPE
Acc
MPE
Acc
MPE
Acc
MPE
Acc
MPE
Method
(All)
(Novel)
(AlI)
(Novel)
(All)
(Novel)
(All)
(Novel)
(All)
(Novel)
Source-Only
32.0
39.3
33.8
70.0
41.4
BODA 158
42.2
0.189
43.4
0.16
76.5
0.079
71.5
0.077
43.8
0.207
Domain Disc.
43.2
0.135
50.9
0.176
73.0
0.071
56.5
0.091
40.6
0.083
k-PU
50.7
0.394
52.1
0.373
85.9
0.307
75.6
0.059
46.0
0.313
PULSE (Ours)
58.0
0.054
62.2
0.061
87.8
0.058
79.1
0.054
48.9
0.043
changing single config file, new OSLS setups can be generated and experimented with. We provide precise details about hyperparameters, OSLS setup for each dataset and code in App. F3
Results Across different datasets, we observe that PULSE consistently outperforms other methods for the target classification and novel prevalence estimation (Table J. For detection of novel classes (Ace (Novel) column), kPU achieves superior performance as compared to alternative approaches because its bias to default to (k+ 1)th class. This evident by the sharp decrease in performance on previously seen classes. For each dataset, plot evolution of performance with training in App. F.4 We observe more stability in performance of PULSE as compared to other methods.
We observe that with default byperparameters, popular OSDA methods significantly under perform as compared to PULSE. We hypothesize that the primary reasons underlying the poor performance of OSDA methods are (i) the heuristics employed to detect novel classes; and (ii) loss functions incorporated to improve alignment between examples from common classes source and target. To detect novel classes, a standard heuristic employed in popular OSDA methods involves thresholding uncertainty estimates (c.g.. prediction entropy, softmax confidence 73 25 59] at a predefined threshold k. However, fixed may not for different datasets and different fractions of the novel class. In App. FS we ablate by (i) removing loss function terms incorporated with an aim to improve source target alignment; and (ii) vary threshold x and show improvements in performance of these methods. In contrast, our two-stage method PULSE, first estimates the fraction of novel class which then guides the classification of novel class versus previously seen classes avoiding the need guess k.
Ablations Different datasets, our setup span different fraction of novel class prevalence ranging from 0.22 (in CIFAR1O) 10) to 0.64 (in Tabula Muris). For each dataset, we perform more ablations on the novel class proportion App. F6 For kPU and PULSSE in the main paper, we include results with BBE and CVIR 22]. In App. F.8. perform experiments with alternative PU learning approaches
8

---

and highlight the superiority of BBE and CVIR over other methods. Moreover, since we have access unlabeled target data, we experiment with SimCLR [T3] pre-training on the mixture of unlabeled source and target dataset. We include setup details and results in App. F.7 While pre-trained backbone architecture improves performance for all methods, PUL.SE continues to dominate other methods.
8 Analysis of PULSE Framework
In this section, we analyse key steps of our PULSE procedure for target label marginal estimation (Step 3, 5 A and learning the domain discriminator classifier (Step 5, Algorithm( [T. Due to space constraints, we present informal results here and relegate formal statements and proofs to App. D
Theoretical analysis for target marginal estimation Building on BBE results from Garg et al. 22 we present finite sample results for target label marginal estimation. When the data satisfies strong positivity, we observe that source classifiers often exhibit a threshold Gy on softmax output of each class y Yd, above which the top bin (i.e.. [Cy. I]) contains mostly examples from that class y. We give empirical evidence to this claim in App. [.I] Then, we show that the existence of (nearly) pure top bin for each class in Sa is sufficient for Step 3 in Algorithm to produce (nearly) consistent estimates:
Theorem (Informal). Assume that for each class y E Vs. there exists a threshold cy such that for the classifier Ss. if [fs(1)] > Cy for any x then the true label for that sample is y. Then, we have l
pel1 < O (VR (4g/6//6)// v (og(4k/0)/mm
The proof technique simply builds on the proof of Theorem in Garg et al. 229]. By assuming that recover close to ground truth label marginal for source classes, analysis to Step 5 of Algorithm to show convergence of estimate (1 = k + 1) to true prevalence Pr(y le + 1). We discuss this further in App. D.3
also extend the above
Theoretical analysis of CVIR in population While the CVIR loss was proposed in Garg et al. [29], no analysis was provided for convergence of the iterative gradient descent procedure. In our work, we show that in population on a separable Gaussian dataset, CVIR will recover the optimal classifier.
We consider a binary classification problem where we have access to positive distribution (i.e., Pp), unlabeled distribution (i.e., P., == ap, + (1 - (1-a)p,)) p)), and mixture coefficient a. Making a parallel connection to Step 5 of PULSE, positive distribution Pp here refers to the label shift corrected source distribution 2/, and pa. refers to p: pr goal is to recover the classifier that discriminates Pp versus Ph. (parallel 1', versus pcl-ly = k + 1)).
k ( = k- + 1)+ (1 (y = + 1))p Our
First we introduce some notation. For classifier and loss function (i.e., logistic loss), define VIR. (f) = inf(r E R : Pa-p.. (f(x,- -1; & T) > 1 - a}. Intuitively, (fR.(f) identifies a threshold to capture bottom I- a fraction of the loss f(x,-1) for points sampled from Pu- Additionally, define CVIR loss as C(f. w) a.p, [((z, 11 Ep.. [w(x)((z, -1; f)] for classifier I and some weights wxxx) E {0, 13. Formally, given a classifier fi at an iterate I, CVIR procedure proceeds as follows:
wr(x) = 1[(x,, f() & (IR.(f))). fi+1 f w))
(4)
(5)
We assume that are drawn from two half multivariate Gaussian with mean zero and identity covariance, i.e., x ~ Pp < x = Yp + 2| p > 0, and x ~ Pa <> x = 7700ppt + 2 0% < 0, where ~ (O, Ia). Here To is the margin and Oopt Rd is the true separator. Here, we have access to distribution Pp. Pu app + (1 a)pn. and the true proportion a.
Theorem 2 (Informal). In the data setup detailed above, linear classifier J =) a (0F trained with CVIR procedure as in 4 will converge to optimal positive versus negative classifier.
The proof uses a key idea that for any classifier not separating positive and negative data perfectly, the gradient in (5) is non-zero. Hence, convergence of the CVIR procedure (implied by smoothness of CVIR loss) implies converge to an optimal classifier. For separable datasets general, we can extend the above analysis with some modifications to the CVIR procedure. We discuss this in App. D.4

---

Empirical investigation in overparameteriied models As noted in our ablation experiments and Garg et al. 29 domain discriminator trained with CVIR outperforms classifiers trained with other consistent objectives (nnPU 38 and uPU (21)) While the above analysis highlights consistency of CVIR procedure population, it doesn't capture the observed empirical efficacy of CVIR over alternative methods in overparameterized models. In the Gaussian setup described above, we train overparameterized linear models to compare CVIR with other methods. We discuss precise experiments and results in App. E but highlight the key takeaway here. First, we observe that when a classifier is trained to distinguish positive and unlabeled data, early learning happens [47] [ 28)) Le., during the initial phase of learning classifier learns to classify positives in unlabeled correctly as positives. Next, we show that post early learning rejection of large fraction of positives from unlabeled training in equation crucially helps CVIR.
Conclusion
In this work, we introduce OSLS a well-posed instantiation of OSDA that subsumes label shift and PU learning into a framework for learning adaptive classifiers. We presented identifiability conditions for OSLS and proposed PULSE, simple and effective approach to tackle the OSLS problem. Moreover, our extensive experiments demonstrate efficacy of PULSE over popular OSDA alternatives when the OSLS assumptions are met. We would like to highlight the brittle nature benchmark driven progress in OSDA and hope that work can help to stimulate more solid foundations and enable systematic progress in this area. Finally, we hope that our open source code and benchmarks will foster further progress OSLS.
9.1 Limitations and Future Work
Here, we discuss limitations of the PULSE framework. First, to estimate the relative label shift among source classes in target, we leverage k-PU reductions with several modifications. While we reduce the issues due to overestimation bias by re-normalizing the label marginal among source classes in target, in future, we may hope to replace this heuristic step to directly estimate the joint target marginal.
Second, since our methods use CVIR and BBE sub-routines, failure of these methods can lead to failure of PULSE. For example, efficacy of BBE relies on the existence of an almost pure top bin in the classifier output space. While this property seems to be satisfied across different datasets spanning different modalities and applications, failure to identify an almost pure top bin can degrade the performance of BBE and hence, our PULSE framework.
In future work, we also hope to bridge the gap between the necessary and sufficient identifiability conditions. While we empirically investigate reasons for CVIR's efficacy in overparameterized models, we aim to extend our theory to overparameterized settings in future. In our work, we strictly operate under the OSLS settings, where we performed semi-synthetic experiments on vision, language and tabular datasets. In future, it will be interesting to experiment with our PULSE procedure in relaxed settings where p( p(zly) may shift in some natural-seeming ways from source to target.
Acknowledgments and Disclosure of Funding
We thank Jennifer Hsia for initial discussion on the OSLS problem. We also thank Euxhen Hasanaj for suggesting Biology datasets. SG acknowledges Amazon Graduate Fellowship for their support. SB acknowledges funding from the NSF grants DMS-171300S DMS-2113684 and CIF-1763734, as well as Amazon AI and Google Research Scholar Award. ZL acknowledges Amazon AI, Salesforce Research, Facebook, UPMC, Abridge, the PwC Center, the Block Center, the Center for Machine Learning and Health, and the CMU Software Engineering Institute (SEI) via Department of Defense contract FA8702-15 15- 15-D-0002 for their generous support of ACMI Lab's 's research on machine learning under distribution shift.
References
[1] Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Adapting to label shift with bias- corrected calibration. In International Conference Machine Learning (ICML), 2021.
10

---

[2] Md Zahangir Alom, Chris Yakopcic, Mst Nasrin, Tarek M Taha, Vijayan K Asari, et al. Breast cancer classification from histopathological images with inception recurrent residual convolutional neural network. Journal of digital imaging, 2019.
[3] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In Interna- tional Conference on Machine Learning (ICML), 2019.
[4] Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang. and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. In International Conference Learning Representations (ICIR), 2019.
[5] Mahsa Baktashmotlagh, Masoud Faraki, Tom Drummond, and Mathieu Salzmann. Learning factorized representations for open-set domain adaptation. In International Conference on Learning Representations (ICLR). 2019.
[6] Jessa Bekker and Jesse Davis. Estimating the class prior in positive and unlabeled data through decision tree induction. In Assosication for the Advancement of Artificial Intelligence (AAAI), 2018.
[7] Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: a survey. Machine Learning, 2020.
[8] Shai Adaptation. In International Conference Artificial Intelligence and Statistics (AISTATS), 2010.
Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility Theorems for Domain
[9] Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1893-1902, 2015.
[10] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi-supervised novelty detection. Journal of Machine Learning Research (JMLR), 11:2973-3009, 2010.
[11] Silvia Bueci, Mohammad Reza Loghmani, and Tatiana Tommasi. On the effectiveness of image rotation for open set domain adaptation. In European Conference on Computer Vision. Springer, 2020.
[12] Kaidi Cao, Maria Brbic, and Jure Leskovec. Concept learners for few-shot learning. In International Conference on Learning Representations (ICLR), 2021.
[13] Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In Interna- tional Conference on Learning Representations (ICLR), 2022.
[14] Zhangjie Cao, Kaichao You, Mingsheng Long, Jianmin Wang, and Qiang Yang. Learning to transfer examples for partial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2985-2994, 2019.
[15] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[16] Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self-pu: Self boosted and calibrated positive-unlabeled training. In International Conference on Machine Learning, pages 1510-1519. PMLR, 2020.
[17] Tabula Muris Consortium et al. A single cell transcriptomic atlas characterizes aging tissues in the mouse. Nature, 583(7817), 2020.
[18] Corinna Cortes and Mehryar Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519, 2014.
[19] Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning Bounds for Importance Weighting. In Advances in Neural Information Processing Systems (NIPS), 2010.
11

---

[20] Francesco De Comité, François Denis, Rémi Gilleron, and Fabien Letouzey. Positive and unlabeled examples help learning. In International Conference on Algorithmic Learning Theory (AL.T). Springer, 1999.
[21] Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama. Convex formulation for learning from positive and unlabeled data. In International conference on machine learning, pages 1386- 1394, 2015.
[22] Marthinus C Du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. Advances in neural information processing systems, 27:703-711, 2014.
[23] Marthinus Christoffel Du Plessis and Masashi Sugiyama. Class prior estimation from positive and unlabeled data. IEICE TRANSACTIONS on Information and Systems, 97(5):1358-13 1362, 2014.
[24] Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In International Conference Knowledge Discovery and Data Mining (KDD), pages 213-220, 2008.
[25] Bo Fu, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Learning to detect open classes for universal domain adaptation. In European Conference on Computer Vision. Springer, 2020.
[26] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. networks. The journal of machine learning research, 2016.
training of neural
[27] Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. unified view of label shift estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[28] Saurabh Garg, Sivaraman Balakrishnan, Zico Kolter, and Zachary Lipton. RATT: Leveraging unlabeled data to guarantee generalization. In International Conference on Machine Learning (ICML), 2021.
[29] Saurabh Garg, Yifan Wu, Alex Smola, Sivaraman Balakrishnan, and Zachary Lipton. Mixture proportion estimation and PU learning: A modern approach. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[30] Saurabh Garg, Sivaraman Balakrishnan, Zachary Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict ence on Learning Representations (ICLR), 2022.
performance. In International Confer-
[31] Yonatan Geifman and Ran EI- Yaniv. Selective classification for deep neural networks. arXiv preprint 17Xvv11005008000 2017.
[32] Arthur Gretton, Alexander J Smola, Jiayuan Huang, Marcel Schmittfull, Karsten M Borgwardt, and Bernhard Schölkopf. Covariate Shift by Kernel Mean Matching. Journal of Machine Learning Research (JMLR), 2009.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Computer Vision and Patlern Recognition (CVPR), 2016
[34] Dan Hendrycks and Kevin Gimpel. A Bascline for Detecting Misclassified and Out-Of- Distribution Examples in Neural Networks, In International Conference on Learning Represen- tations (ICLR), 2017.
[35] Dmitry Ivanov. DEDPUL: arXiv preprint arXiv: 1902.06965 2019.
positive-umlabeled learning.
[36] Shantanu Jain, Martha White, Michael W Trosset, and Predrag Radivojae. Nonparametric semi- supervised learning of class proportions. arXiv preprint 1rXXvv/601.0/90 2016.
[37] Heinrich Jiang, Been Kim, Melody Y Guan, and Maya R Gupta. To trust or not to trust a classifier. In Advances in Neural Information Processing Systems (NeurIPS), pages 5546-5557, 2018
12

---

[38] Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. learning with non- negative risk estimator, In Advances in neural information processing systems, pages 1675-1685, 2017.
[39] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga. Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness. Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML). 2021.
[40] Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images. Technical report, Citeseer, 2009.
[41] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[42] Fabien Letouzey, François Denis, and Rémi Gilleron. Learning from positive and unlabeled examples. In International Conference on Algorithmic Learning Theory, pages 71-85. Springer, 2000.
[43] Qing Lian, Wen Li, Lin Chen, and Lixin Duan. Known-class aware self eesemblsemlle for open set domain adaptation. arXiv preprint 1rXiv:1955.01068 2019.
[44] Haofu Liao. A deep learning approach to universal skin disease classification. University of Rochester Department of Computer Science, CSC, 2016.
[45] Zachary C Lipton, Yu-Xiang Wang, and Alex Smola. Detecting and Correcting for Label Shift with Black Box Predictors. In International Conference on Machine Learning (ICML), 2018.
[46] Hong Liu, Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Qiang Yang. Separate to adapt: Open set domain adaptation via progressive separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2927-2936, 2019.
[47] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Grande Granda. Early- learning regularization prevents memorization of noisy labels. arXiv preprint arXiv:2007.00151, 2020.
[48] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105. PMLR, 2015.
[49] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning. PMLR, 2017.
[50] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's s uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[51] Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pages 754-763, 2017.
[52] Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVE international conference on computer vision, pages 1406-1415, 2019.
[53] Harish Ramaswamy, Clayton Scott, and Ambuj Tewari. Mixture proportion estimation via kernel embeddings of distributions. In International conference on machine learning, pages 2062-2060, 2016.
[54] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 1152)) 2015.
13

---

[55] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pages 213-226. Springer, 2010.
[56]
Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the Outputs of Classifier to New Priori Probabilities: A Simple Procedure. Neural Computation, 2002.
[57] Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang. Extending the wilds benchmark for unsupervised adaptation. In NeurIPS Workshop on Distribution Shifts, 2021.
[58] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In Proceedings the European Conference on Computer Vision (ECCV), pages 153-168, 2018.
[59] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[60] Tyler Sanderson and Clayton Scott. Class proportion estimation with application to multiclass anomaly rejection. In Artificial Intelligence and Statistics (AISTATS), pages 550-858, 2014.
[61] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopu- lation shift. In International Conference on Learning Representations (ICLR), 2021.
[62] Walter J. Scheirer, Anderson Rezende Rocha, Archana Sapkota, and Terrance E. Boult. Toward open set recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2013.
[63] Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In Artificial Intelligence and Statistics, pages 838-846, 2015.
[64] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in Neural Information Process- ing Systems, 33, 2020.
[65] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent separable data. The Journal of Machine Learning Research, 2018.
[66] Fabio A Spanhol, Luiz S Oliveira, Caroline Petitjean, and Laurent Heutte. dataset for breast cancer bistopathological image classification. leee transactions on biomedical engineering, 63 (7):1455-1462, 2015.
[67] Amos Storkey, When Training and Test Sets Are Different: Characterizing Learning Transfer. Dataset Shift in Machine Learming, 2009.
[68] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation.
In European conference on computer vision. Springer, 2016.
[69] Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adaptation. In Domain Adaptation in Computer Vision Applications. Springer, 2017.
[70] Shuhan Tan, Jiening Jiao, and Wei-Shi Zheng. Weakly supervised open-set domain adaptation by dual-domain collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2394-5403. 2019.
[71] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018-5027, 2017.
[72] Yixing Xu, Chang Xu, Chao Xu, and Dacheng Tao. Multi-positive and unlabeled learning. In IJCAI, pages 3182-3188, 2017.
14

---

[73] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Universal domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2720-2729, 2019.
[74] Bianca Zadrozny. Learning and Evaluating Classifiers Under Sample Selection Bias. In International Conference on Machine Learning (ICML), 2004.
[75] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference Learning Representations (ICLR), 2017.
[76] Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In European Conference on Computer Vision, pages 102-117. Springer, 2020.
[77] Jingzhao Zhang, Aditya Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. In International Conference on Learning Representations (ICLR), 2021.
[78] Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain Adaptation Under Target and Conditional Shift. In International Conference Machine Learning (ICML), 2013.
[79] Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu. Collaborative and adversarial network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
[80] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning. PMLR, 2019.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We believe that this work, which proposes a novel instantiation of open set domain adaptation problem does not present a significant societal concern. While this could potentially guide practitioners to improve classification and mixture proportion estimation in applications where data from novel classes can arrive during test time, we do not believe that it will fundamentally impact how machine learning is used in a way that could conceivably be socially salient.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Sec. Sec. [
4 and
(b) Did you include complete proofs of all theoretical results? [Yes] See Sec. B and Sec.
D
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material clude all the experimental details App. We also open source our code at hhttpss //github. com/ acmi wawllaa/pppaa -et---eee-----i-
URL.)? [Yes] We in-
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Yes, see App.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Yes, we run all experiments with three different seeds and include results in with standard deviation in App. [F]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Yes, see App.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're
using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "Zu4Nh3_JjY5", "reviewer": "Reviewer_dUdt", "summary": "This paper introduces domain adaptation under Open Set Label Shift (OSLS). Specifically, it assumes the target has one more novel class that was not previously seen in the source domain while allowing label shifts between source and target domains. This work provides theoretical findings of OSLS, specifically, the necessary condition \"weak positivity\" and two sufficient conditions \"strong positivity\" and \"separability\". The author further proposes a framework to solve OSLS, named PULSE which combines techniques from both areas of (1) positive and unlabeled learning and (2) DA under label shift. The effectiveness of the proposed methods is demonstrated in language, image, and medical datasets. ", "questions": "1. Does OSLS apply to medical applications such as COVID diagnosis?\n\nThe author motivates the OSLS problem by giving a motivating example in the medical domain: the disease proportion changes seasonally and sometime may come with a new disease, like COVID-19. I would like the author to analyze whether the OSLS is suitable for the disease diagnosis problem. For example, let us consider a DA problem where the source domain has two classes, normal people and flu patients while the target domain will have one more class, COIVD patients. How are your identifiability conditions satisfied in this application? Does the PLUSE framework still apply?\n\n2. Does all the 5 tasks in the experiments satisfy \"strong positivity\"? Could the author include benchmarks not satisfying this condition?\n\nThe author claims that \"we observe that techniques proposed under strong positivity were empirically stable and outperform other baselines developed under separability. This is intuitive for many benchmark datasets where for each class there exists a subdomain that only belongs to that class.\" First I want to ask, do all 5 benchmarks used in your experiments all satisfy that there exists a subdomain that only belongs to that class? Second, I would like to point out that samples belonging to different classes are disjoint is the property of object recognition tasks however, there are many other types of tasks that do not satisfy such disjoint property. For example, in age prediction (classify different age groups). It is unlikely that there exist samples in the age group [20,30)'s portrait will never appear in the age group [30,40) or [10,20) since the boundary between different classes is not perfectly clear here. Thus I think it is necessary to include other benchmarks that do not satisfy \"strong possibility\" for the sake of a more complete evaluation of the proposed method. \n", "limitations": "This work does not raise potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "originality\n---\nThe paper focuses on a special case of the open set domain adaptation where the target domain has one novel class that is not previously unseen in the source domain. The big concept of open set DA is not novel while the special case is well studied yet, thus can be considered novel. The paper delivers an identifiability analysis of the OSLS which is novel. \n\nquality\n---\nThe paper provides sold theoretical analysis as well as extensive experiments on 5 datasets across multiple application domains. \n\n\nMinor issue, typos:\n\n1. Line 46: … the matrix submatrix… \n2. Line 67: double periods after \"(Sec.7)\" \n3. Line 154: \\frac{1}{p_t(y=k+1)} is missing in the closed form of p_t(x|y=k+1)\n4. The paper seems not define what the metric \"novel prevalence estimation\" is. \n\nclarity\n---\nThe paper has many contents and is pretty dense. The majority of the paper is well-written and clear. However, I find the method section a bit obfuscated. One reason is that there is no headings or subsection title to remind the reader what is the focus of each paragraph. Another reason is that the author seems to jump to the details too quickly. The logic of the current text is kind of linear. I would like to suggest the author organize section 6 and each of its subsections to have an overview-and-details structure. \n\nsignificance\n---\nThe paper does a good job on the specific problem it aims to solve. The analysis is solid and the empirical results are illustrative. My main concern is that the paper's significance is constrained by the practicality of the proposed problem. The problem setting is a bit artificial to me. I am not sure how practical it is that we assume the target domain has exactly one novel class unseen. The experiment is kind of synthetic/semi-synthetic since the author chooses source and novel classes randomly. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YmMQwFyBxX8", "reviewer": "Reviewer_iwb5", "summary": "This paper introduces the problem of domain adaptation under open set label shift (OSLS), where the class-conditional distributions p(x|y) are domain-invariant, and p(y) can change.  Domain Adaptation under label shift and positive-unlabeled (PU) learning can be all considered as special cases of OSLS.   This paper also provides the identifiablity of OSLS, including necessary condition (weak positivity) and sufficient conditions (strong positvity). Under the strong positivity condition, the OSLS problem can be broken into k PU problems.  The PU learning algorithms cannot scale to datasets with large number of classes because of error accumulation. This paper then proposes PULSE framework to solve this problem by exploiting the joint structure of the problem with source-class re-sampling. Experiments across 7 semi-synthetic benchmarks shows that the proposed PULSE consistently outperform OSDA baselines.", "questions": "- All experimental results are semi-synthesized and are on relative small datasets, such as CIFAR-10, CIFAR-100. It is good to have results on some popularly used datasets for domain adaptation such as DomainNet.\n- Make the writing better\n\n ", "limitations": "- The benchmarks used in this paper are all semi-synthesized, and thus lack proof from real-world application\n- The paper mentions that \"In future work, we hope to bridge the gap between the necessary and sufficient identifiability conditions.\"\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Open Set Label Shift (OSLS) an interesting problem, assuming p(x|y) is the same and p(y) is changing.\n- Theories on identifiablity of the problem and convergence analysis\n- New algorithm PULSE to solve the problem\n- Good experimental results\n\nWeakness\n- All experimental results are semi-synthesized and are on relative small datasets, such as CIFAR-10, CIFAR-100. It is good to have results on some popularly used datasets for domain adaptation such as DomainNet.\n- The writing is not easy to follow, the references (e.g., Algorithm 2&3) switch from main paper and supplementary materials without clear instruction. There are also some minor grammar errors including in the formula (line 153)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VLds9CgKWS", "reviewer": "Reviewer_Pd4h", "summary": "This paper scopes the common Open Set Domain Adaptation problem to specifically consider Open Set Label Shift (OSLS) problems, where $p(x|y)$ is constant, but the class proportions may change between source and target, and there may be a new unseen class added in testing. In this setting, the goal is to identify instances of the unseen class, while also performing adequately on the previously seen classes. The paper proposes a new framework, PULSE, which combines classical Positive and Unlabeled Learning techniques and label reweighting techniques in order to tackle this new problem. The method is empirically shown to perform well in a variety of OSLS problems, and theoretical analysis is conducted to create sufficient conditions for identifiability of the OSLS problem. ", "questions": "* How robust is PULSE expected to be in general OSDA problems, not just label shift?", "limitations": "   It would be worthwhile to have a further discussion on limitations of PULSE with regards to the amount of labeled data of the target domain - for example, with no labels it would be difficult to approximate what the label shift is. It could also be useful to discuss the types of models or datasets that PULSE is expected to work on (e.g. does it perform worse as the number of classes increase, the computational efficiency with regards to larger models, etc.); and that it inherits limitations from its particular stages (e.g. any limitations of importance weighted label shift or CVIR/BBE). ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* **Originality**\n    To the best of my knowledge, this is the first time I have seen the scoping of the open set domain adaptation problem. By focusing purely on label shift, the authors are able to create some novel theoretical results as well as a highly performant framework for tackling the OSLS problem. In particular, their framework combines two standard techniques to great effect; class reweighting to handle the label shift and PU techniques to handle the open set nature of the problem. \n* **Quality**\n    The claims are well-substantiated both theoretically and empirically, and the results are impressive. The authors perform a detailed evaluation on existing open source domain adaptation methods as well as a slight ablation study by performing standard PU techniques without any label reweighting. \n* **Clarity**\n    The paper is mostly well organized and written. A worthwhile addition in the related work section would be to make it more clear exactly who your setting differs from each of the main categories that are defined there. Further, it may be worth a few sentences relating why identifiability is useful and related to the remainder of the paper; Section 4 felt significantly out of place compared to the rest of the paper.\n* Some minor nitpicks\n     * line 48 - matrix submatrix appears to be a typo. \n     * line 257 should likely be a heading\n* **Significance**\n    By scoping the OSDA problem to focus on label shift, the authors were able to show relatively significant gains compared to standard OSDA methods - this kind of scoping appears quite fruitful for other researchers to build off of. The theoretical contributions in the paper are strong evidence that their framework, PULSE, is able to perform quite well on a variety of scenarios which leaves it as a valuable benchmark for future work. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8EymETiYcKF", "reviewer": "Reviewer_d8bB", "summary": "The goal of this paper is to solve the open set domain adaptation under the label shift setting. The authors proposed a PU learning-based framework to first estimate the label shift and then classify the novel class. Moreover, the authors also gave sufficient and necessary conditions to the open set label shift in order to make the target label marginal identify. The experimental results showed that the PULSE framework could achieve great performance improvement. ", "questions": "1) The authors need to better organize the paper and improve the mathematical notations, e.g. in line 48 \"the matrix submatrix\" is not so clear.\n\n\n2) For the extended BBE algorithm 2, which is the importance weight w? ($\\hat{c}_j$ I assume), the author should be more clear.\n\n3) Since the key point in this paper is to estimate the target label marginal distribution using source data, this process relies heavily on Theorem 1 to guarantee the target domain estimation on the source domain. As we can find from Theorem 1, the threshold would be very important, so the author should discuss more about the threshold e.g. is that a hyperparameter? Or is there any trade-off between varying the threshold with the performance on target data? I also wonder that is that directly using the source classifier for target in Algorithm 2 line 6 is guaranteed by this Theorem?\n\n4) Since the PULSE framework is built on the strong positive condition to guarantee the target inevitability, I found no regularization for the data in this paper, so the author also needs to explain why their algorithm can achieve this strong positive condition.\n\n5) In algorithm 2 , the author should expand the loss function in line 6 and line 10 to make it more clear. And The author should explain more on the connection between the $\\alpha$ with the novel target and why they are equal and can be derived this way (why don't use the estimation from algorithm 2 but use algorithm 4 instead in algorithm 3, however, I found the resample based estimation should be derived from algorithm 2 as described in line 805).\n\n6) Both in algorithm 2 and algorithm 4, the z is ambitious in the paper, are they discrete or continuous variables, and do we need to compute all of the values from 0 to 1 for estimation? Will that increase the computation burden? Also, for the c in algorithm 2 line 6, do they have the connections with the threshold in Theorem 1 since their notation is all c? \n\n7) How about the results from the general domain adaptation datasets like USPS, OFFICEHOME, DOMAINNET etc. The experiment setup is not convincing, the random choice for source and target in small datasets (like cifar) may make the source and target only have the limited domain discrepancy, which in return limit the domain adaptation problems back into the novel class detection problems under PU setup. Since the domain discrepancy, the author proposed in the paper is mostly between the positive samples and the unlabeled samples, and unlabeled samples are consists of the combination of both positive and negative samples as described in CVIR, I have little concern on if this problem would degrade into open-set classification (since the domain discrepancy is limited) instead of the open-set domain adaptation as we can find in the experiments.\n\n8) In the paper line 188 section 5, why only satisfying the strong positive condition the OSLS can be reduced into K PU? It seems most of the OSLS problem can all be reduced into K PU problems that way in equation 2.", "limitations": "1) This paper needs to be well organized.\n2) Some details need to be completed as described in the questions.\n3) The experiments in this paper is not convincing.\n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1) The idea of combing PU problems with the OSLS is interesting, the authors used an ingenious way to merge the PU problems into the OSLS. The reduction from the k-PU to a single PU is attractive (equation 3).\n\n2) The definitions and theorems are straightforward and easy to be understood, and the mathematics proof seems solid. \n\n3) The author used a two-stage method to separately estimate the label shift and to identify the novel class, which seems effective in solving both the domain adaptation and novel class identification. The results of the PULSE seems good.\n\n4) I have no doubts in the originality to the best of my knowledge, the quality of this paper is good, some more details need to be discussed for clarity as I stated later, and the contribution of this paper has some significance to the OSLS regime.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept