Unsupervised Domain Adaptation by Learning Using Privileged Information
Authors: Adam Breitholtz
Anton Matsson
Fredrik D. Johansson
Venue: Accepted by TMLR ()

Abstract: Successful unsupervised domain adaptation is guaranteed only under strong assumptions such as covariate shift and overlap between input domains. The latter is often violated in high-dimensional applications like image classification which, despite this limitation, continues to serve as inspiration and benchmark for algorithm development. In this work, we show that training-time access to side information in the form of auxiliary variables can help relax restrictions on input variables and increase the sample efficiency of learning at the cost of collecting a richer variable set. As this information is assumed available only during training, not in deployment, we call this problem unsupervised domain adaptation by learning using privileged information (DALUPI). To solve this problem, we propose a simple two-stage learning algorithm, inspired by our analysis of the expected error in the target domain, and a practical end-to-end variant for image classification. We propose three evaluation tasks based on classification of entities in photos and anomalies in medical images with different types of available privileged information (binary attributes and single or multiple regions of interest). We demonstrate across these tasks that using privileged information in learning can reduce errors in domain transfer compared to baselines, be robust to spurious correlations in the source domain, and increase sample efficiency.

Published in Transactions on Machine Learning Research (09/2024)
Unsupervised Domain Adaptation by Learning Using Privileged Information
Adam Breitholtz*
adambre@chalmett
Deportment of Computer Science
Chalmers Universily of Technology
Anton Matsson'
antmats @ahmmmts@chammersssse
Department of Computer Science
Chalmers University of Technology
Fredrik D. Johansson
frddilljjhannson@@hhhnness..
Department of Computer Science
Chalmers University of Technology
Reviewed on OpenReview: https: openreview. net/ met/ forum? id- H3MPHOKw
Abstract
Successful unsupervised domain adaptation is guaranteed only under strong assumptions such as covariate shift and overlap between input domains. The latter is often violated in high-dimenssonal applications like image classification which, despite this limitation, timues to serve inspiration and benchmark for algorithm development. In this work, show that training-time access to side information in the form of auxiliary variables can help relax restrictions on input variables and increase the sample efficiency of learning al the cost collecting a richer variable set. As this information assumed available only during training, not in deployment, we call this problem unsupervised domain adaptation by learning using privileged information (DALUPI). To solve this problem, we propose a simple two-stage learning algorithm, inspired by our analysis of the expected error in the target domain, and practical end-to- end-to-end variant for image classification. We propose three evaluation tasks based on classification of entities in photos and anomalics in medical im- ages with different types of available privileged information (binary attributes and single or multiple regions of interest). We demonstrate across these tasks that using privileged infor- mation learning can reduce errors in domain transfer compared to baselines, be robust to spurious correlations in the source domain, and increase sample efficiency.
1 Introduction
Deployment of machine learning (ML) systems relics on generalization from training samples new instances in a target domain. When these new instances differ in distribution from the source of training data, performance tends to degrade and guarantecs often weak. For example, supervised ML model trained to identify medical conditions in X-ray images from one hospital may work poorly in another hospital if the two sites have different equipment or examination protocols (Zech et al., 2018). In the unsuperoised domain adaptation (UDA) problem (Ben-David et al., 2006), no labeled examples are available from the target domain and strong assumptions are needed for success. In this work, we ask: How can access to auriliary variables during training help solve the UDA problem and weaken the assumptions necessary guarantee domain transfer?
In standard UDA, common assumption is that the object of the learning task is identical in source and target domains but that input distributions differ (Shimodaira, 2000). This "covariale shift" assumption is
"Equal contribution.

---

Published in Transactions on Machine Learning Research (09/2024)
plausible in our X-ray example above: Doctors are likely to give the same diagnosis based on X-rays of the same patient from similar but different equipment. However, guarantees for consistent domain adaptation also require either distributional overlap between imputs from source and target domains or known parametric forms of the labeling function (Ben-David & Urner, 2012; Wu et al., 2019; Johansson et al., 2019). Without these, adaptation cannot be verified or guaranteed by statistical means.
Input domain overlap is implausible for the high-dimensional tasks that have become standard benchmarks in the UDA community, inchuding image classification (Long et al., 2013; Ganin et al., 2016) and sentence labeling (Orihashi et al., 2020). If hospitals have different X-ray equipment, the probability of observing images from source and target domains is zero (Zech et al., 2018). Even when covariate shift and overlap are satisfied, large domain differences can have a dramatic effect on sample complexity (Breitholtz & Johansson, 2022). Despite promising developments (Shen et al., 2022), realistic guarantees for practical domain transfer remain elusive.
In supervised ML without domain shift, incorporating auxiliary variables in the training of models has been proposed to improve out-of-sample generalization. For example, learning using privileged informa- tion (Vapnik & Vashist, 2009; Lopez-Paz al., 2016), variables available during training but unavailable in deployment, has been proven to require fewer examples compared to learning without these variables (Karls- son et al., 2021). In X-ray classification, privileged information ion (PT) can come from graphical annotations or clinical notes made by radiologists that are unavailable when the system is used. While PI has begun to see use in domain adaptation, see e.g., Sarafianos et al. (2017) or Vu et al. (2019), and a theoretical analysis exists for linear classifiers (Xie et al., 2020), the literature has yet to fully characterize the benefits of this practice.
We introduce unsupervised domain adoplation by learning using privileged information (DALUPI), in which auxiliary variables, related to the outcome of interest, are leveraged during training to improve test-time adaptation when the variables are unavailable. We summarize our contributions below:
We formalize the DALUPI problem and give conditions under which it is possible to solve it con- sistently, i.c., to learn a model using privileged information that predicts optimally in the target domain. Importantly, these conditions do not rely on distributional overlap between source and target domains in the input variable (Section 2.1), making consistent learning without privileged information (PI) generally infeasible.
We propose practical learning algorithms for image classification in the DALUPI setting (Section 3), designed to handle problems with three different types of PI, see Figure 1 for examples. As common UDA benchmarks lack auxiliary variables related to the learning problem, we propose three new ovaluation tasks spanning the three types of PI using data sets with real-world images and auxiliary variables.
On these tasks, we compare our methods to supervised learning baselines and well-known methods for unsupervised domain adaptation (Section 4). We find that our proposed models perform favorably to the alternatives for all types of Pl, particularly when input overlap is violated and when training sets are small.
2 Privileged Information in Domain Adaptation
In unsupervised domain adaptation (UDA), the goal is to learn a hypothesis h to predict outcomes (or labels) Y E y for problem instances represented by input covariates X E X, drawn from a largel domain with density T(X,Y), During training, we have access to labeled samples (x. only from a source domain S(X, Y) and unlabeled samples á from T(X). As a running example, we think of S and T as radiology departments two different hospitals, of X as the X-ray image of a patient, and of Y as the diagnosis made by radiologist after analyzing the image.
2

---

Published in Transactions on Machine Learning Research (09/2024)
Input samples X-5,X-a
Privileged information W-5, W-T
Label Y-S
Test sample X-T
Digit classification with single region of interest Pl
3 MNIST digit
Images from CIFAR- 10's first five and last five classes
Bounding box indicating the location of the MNIST digit
Image from CIFAR-10's
last five classes
Celebrity photo classification with binary attributes as Pl
makeup mustache
M
Images of persons with and without hats
Binary attributes
Reported gender
Image of person with hat
Entity classification with multiple region of interests as Pl
dog person
Indoor and outdoor images
Bounding boxes indicating the location of the entities
Entities
Outdoor image
Figure 1: Examples of domain adaptation tasks with different types of privileged information (PI). During training, input samples X and PI W are drawn from both source and target domains. Labels Y are only available from the source domain. At test time, target sample X is observed. We consider three types of PI: binary attribute vectors, single region of interest, and multiple regions of interest.
We aim learn a hypothesis h H from a hypothesis set H that minimizes the expected target-domain prediction error (risk) Ri,, with respect to loss function L y + R4, i.e. to solve
min Rot (h), (T(h) = ET (L(()) bER
(1)
where we use the subscript convention Eptx) [f - SoEx p(z)f(x))))) to denote an expectation of some function f over a density p on the domain X. A consistent solut ion to the UDA problem returns a minimizer of Equation without ever observing labeled samples from T. However,
and T are allowed to differ arbitrarily, finding such a solution cannot be guaranteed (Ben-David Urner, 2012). To make the problem feasible, we assume that covariate shift (Shimodaira, 2000) holds that the labeling function the same in both domains, but the covariate distributions differ.
Assumption 1 (Covariate shift). For domains S,T X y, covariate shift holds with respect to X if
Fx E X T(X = # S(X and Vo E X T(Y x) S(Y x))
In example, covariate shift means that radiologists at either hospital would diagnose two patients with the same X-ray in the same way, but that the radiologists may encounter different distributions of patients and images. To guarantee consistent learning without further assumptions, these distributions cannot be too different- the source input domain S(x) must sufficiently overlap the target input domain T(x).
Assumption 2 (Domain overlap). A domain S overlaps another domain T with respect to on X if
Vs X T(X x)> - S(X x)) >0
3

---

Published in Transactions on Machine Learning Research (09/2024)
Table I: A summary of the different settings we consider in this work, what data is assumed to be available during training and if guarantoes for identification are known for the sctting under the assumptions of Proposition 1. The parentheses around source samples for DALUPI indicate that we need not necessarily observe these for the setting. Note that at test time only from T is observed. *Under the more generous assumption of overlapping support in the input space X, guarantees exist for all these settings.
Setting
Observed S
Observed T
Guarantee for R
y W y
SL-T
SL-S UDA
LUPI DALUPI
Covariate shift and domain overlap with respect to X guarantee that the target risk Ry can be identified by the sampling distribution described above, and thus, that a solution to Equation may be lound. Hence, they have become standard assumptions, used by most informative guarantees (Zhao et al., 2019).
Overlap is often violated in high-dimensional problems such image classification, partly due to irrelevant information that has a spurious association with the label Y (Beery et al., 2018; D' D'Amour et al., 2021). In X-ray classification, it may be possible to perfectly distinguish hospitals (domains) based on protocol or equipment differences manifesting in the images (Zech et al., 2018). There are no guarantees for optimal UDA in this case. Some guarantees based on distributional distances do not rely on overlap (Ben-David et al., 2006; Long et al., 2013), but do not guarantee optimal learning either (Johansson et al., 2019).
Still, an image X may contain information W which is both sufficient for prediction and supported in both domains. For X-rays, this could be region of pixels indicating medical condition, ignoring parts that merely indicate differences in protocol (Zech et al., 2018). The learner does not know how to find this information a priori, but it can be supplied during training as added supervision. A radiologist could indicate regions of interest W using bounding boxes during training (Trvin et al., 2019), but would not be available to annotate images at test time. As such, W is privileged information (Vapnik & Vashist, 2009).
2.1 Unsupervised Domain Adaptation With Privileged Information
Learning using privileged information, variables that are available only during training but not at test time, has been shown to improve sample efficiency in diverse settings (Vapnik & Izmailov, 2015; Pechyony & Vapnik, 2010; Jung & Johansson, 2022). Next, we show that privileged information can also improve UDA by providing identifiability of the target risk allowing it to be computed from the sampling distribution even when overlap is not satisfied in X.
We define domain adaptation by learning using privileged information (DALUPT) as follows. During train- ing, learners observe samples of covariates X, labels Y and privileged information W E W from S in dataset. Ds {(x;, Wi, ):) as well samples of covariates and privileged information from T, DJ { (=)) At test time, trained models only observe covariates I ~ T(X) and our learnóng goal remains to minimize the target risk, see Equation 1. We justify access to privileged information from T. but not labels, by pointing out that it is often easier lo annotale observations with privileged information W than with labels Y. For example, a non-expert may be able to reliably recognize the outline of an animal in an image, indicating the pixels W corresponding to it, but not identify its species (Y); see Figure 2, where it would likely be easier to identify the location of the cat in the image than to identify its breed.
To identify Ry (Equation 1) without overlap in X, we make the assumption that W is sufficient to predict Y in the following sense.
Assumption (Sufficiency of privileged information). Privileged information W sufficient for the out- come Y given covariates X if y 1. X W in both S and T.
4

---

Published in Transactions on Machine Learning Research (09/2024)
T(X)
Wi
I(W)
s(W)
S(X)
X
W
Figure An illustration of domain overlap being more plausible when we consider appropriate forms of privileged information W, such as region of interest of an image. Source and target domains S, T are here indoor and outdoor images X and the task is to identify the animal Y in the image.
Assumption 3 is satistied when X provides no more information about Y in the presence of W. If we consider W to be subset of image pixels corresponding to an area of interest, the other pixels in X may be unnecessary to predict Y. This is illustrated in Figure 2 where the privileged information w, is the region of interest indicated by the bounding box 4. Here, overlap is more probable in W than in X, as the extracted pixels mostly show cats. Moreover, when W retains more information, sufficiency becomes more plausible but domain overlap in W is reduced. The sufficiency assumption is used to replace T(v x) with Tyy w) in Proposition 1. If sufficiency is violated but it is plausible that the degree insufficincc comparable across domains, we can still obtain a bound on the target risk which may be estimated from observed quantitics. We give such a result in Appendix F.
We expect that some Pl can be selected to be sufficient for given task. However, if this sufficiency cannot be ensured, the overall performance may decrease, assuming covariate shift with respect to W is not violated. Even 80, we still anticipate the generalization error to remain of a comparable magnitude. If covariate shift is violated in W, further performance declines are expected, as the problem becomes more complex and we are not guaranteed to identify the optimal hypothesis (Johansson et al., 2019).
Assumptions 1-2 holding with respect to privileged information W instead of X, along with Assumption 3, allow to identify the target risk even for models h E H that do not use W input:
Proposition 1. Let Assumptions 1 and 2 be satisfied with respect to W (not necessarily with respect to X) and let Assumption 3 hold as stated. Then, the target risk Ry is identified for hypotheses h X + y
RT( ET(X) [ET(IIX) [FSM)))
[,W]IX]]
[[x()) /. T(w (a) / S(y ( w)L(((x), y) dyddddx
and for the squared loss, minimizer of Ry the function
|| x]]= { T(w x) S(y) I w)y dydw
hT(x) ( W W
Proof sketch. RT (h) = Jay T(x, ())((()). vdddy.. We can then marginalize over W to get T(x, y) = (w W) x] T(x) S Sw.S(w)> T(w z))(yy w)dw, where the first equality follows by sufficiency and the second by covariate shift and overlap in W. T(x), T(w x) and S(y | w) are observable through training samples. That he is a minimizer follows from the first-order condition. See Appendix C. 7
Proposition shows that there are conditions where privileged information allows for identification of target- optimal hypotheses where identification is not possible without it, i.e, when overlap is violated in X. W
5

---

Published in Transactions on Machine Learning Research (09/2024)
guides the learner toward the information in X that is relevant for the label Y. When W is deterministic in X, overlap in X implies overlap in W, but not vice versa. In the same case, under Assumption 3, if covariate shift holds for X, it holds also for W. Hence, if sufficiency can be justified, the requirements on X are weaker than in standard UDA, at the cost of collecting W. Surprisingly, Proposition does not require that X is observed the source domain as the result does not depend on S(x).
Figure/ 1 gives examples of problems with the DALUPI structure which we consider in this work. For comparison, list related learning paradigms in Table 1. Supervised learning (SL-S) refers to learning from labeled samples from S without privileged information. SL-T refers to supervised learning with (infeasible) access to labeled samples from T. UDA refers to the setting at the start of Section 2 and LUPI to learning using privileged information without data from T (Vapnik & Vashist, 2009). We compare DALUPI to these alternative settings in our experiments in Section 4.
2.2 A Two-stage Algorithm and Its Risk
In light of Proposition 1, a natural learning strategy is to model privileged information as a function of the input, T(W I)), and the outcome as a function of privileged information, ((w) s Ess [Y | w], and combining these. In the case where W is a deterministic function of X, T(W | xr) is a map f X + W, which may be estimated as a regression f and combined with the outcome regression to form h = ((((x)) We may find such functions f: 9 by separately minimizing the empirical risks
RO R4 ( 1 E Tw (s(s),) and (g)(
1
= m 1=1
Ly (o(o)))), u:)
(2)
has a desired form. Note Hypothesis classes F,g may be chosen so that = = (h = go f: ff( E Fx that Lw and Ly may in general be different loss functions.
We can bound the generalization error of estimators h 5 o when W Rdw and the loss is the squared loss. We do this by placing an assumption of Lipschitz smoothness on the space of prediction functions: Vig E G, w. w' E W: ([a(w) g(w')ll2 S M| w212. To arrive at a bound, we first define the p-weighted empirical risk of the outcome model g in the source domain, (g) m ((u::L_W(g((uuu ys) where T(w)
is the density ratio of T and S, p(w)= = S(w)) When the density ratio is unknown, we may use density estimation (Sugiyama et al., 2012) or probabilistic classifiers to estimate it. We arrive at the following result, proven for univariate Y but generalizable to multivariate outcomes.
Proposition 2. Suppose that W and Y are determánistic in X and W, respectively, and that Assumptions 1- 3 hold with respect to W. Let G comprise M M-Lipschila mappings y: W + y with W C Rdw and let the loss be the squared Euclidean distance, assumed to be uniformly bounded over W. Let p(w)= TWw)//(ww and d and d' be the dimuddd-dmesssnss of 9 and F, respectively. Assume that there are m labeled samples from S and n unlabeled samples from T. Then, for any h = go E EGxx x F, with probability at least 1 1,
(h)
d log 7 + log 2/4
d' log + + log du
R (g) + A +0+
3/5
+
5
2
m
Proof sketch. Decomposing the risk of h o o, we get
RT(h) = (x(()(((X)) r)2]
< 22 (((W) - r)² + ( -
S 2Br((((W) - -r)² + (X))(())) 9(W))
- (((W) - 2
- W]
= -2 + 22 (f) = 22 22 (f)
The first incquality follows the relaxed triangle inequality, the second from the Lipschitz property, and the third equality from Overlap and Covariate shift. Treating each component of ve as independent, using
6

---

Published in Transactions on Machine Learning Research (09/2024)
standard PAC learning results, and application of Theorem 3 from Cortes et al. (2010) allows us to reweight the risk with the density ratio by also adding additional term which contains the Rényi divergence. Then with a union bound argument, we get the stated result. See Appendix D for a more detailed proof. ]
When F and 9 contain the ground-truth mappings between X and W and between W and Y, in the infinite- sample limit, minimizers of Equation 2 minimize Ry as well. Our approach is not limited to classical PAC analysis but could, under suitable assumptions, be carried out under another framework, e-g- using PAC- Bayes analysis to obtain a bound that contains different sample complexity terms, However, such a bound would then hold in expectation over a posterior distribution on H instead of uniformly over H. We sketch a proof of such a bound in Appendix E.
Furthermore, if sufficiency violated but it is plausible that the degree of insufficiency comparable across domains, we can still obtain a bound on the target risk which may be estimated from observed quantities. We give such a result in Appendix F.
3 Image Classification With Privileged Information
We use image classification, where X an image and Y is discrete label, as proof of concept for DALUPI. To show the versatility of our approach, we consider three different instantiations of privileged information W: binary attribute vector, a single region of interest, multiple regions of interest. The two-stage estimator, see Figure 3a, is used in the first two cases. With multiple regions of interest as privileged information, we use an end-to-end model based on Paster-R-CNP (Ren et al., 2016), see Figure 3b. We detail each setting below and illustrate them in Figure 1.
3.1 Binary Attributes as Pl
First, we consider the case where each image x; is accompanied by privileged information in the form of binary vector E 10, indicating the presence of d attributes in the image. In this sctting, we can directly apply two-stage estimator (Equation 2). For the first estimator f, we use convolutional neural network (CNN) trained on observations from T (and possibly S) to output vector of attributes úú from the input Z₁. For the second estimator 9. we use a multi-layer perceptron classifier, trained on source domain tonss that predicts the image label O: given the vector of attributes wi. We use the categorical cross-entropy loss to train both f and J. The resulting classifier, h(x)) = g f((())) is subsequently evaluated on target domain images.
3.2 Single Region of Interest as Pl
Next, we consider privileged information as a subset of pixels Wi taken from the image I; and associated with an object or feature that determines the label yı E (1,...,, K}- In our experiments, this PT is provided as a single bounding box with coordinates t, E R² enclosing the region of interest Wy Here, we use two CNNs, d and 9. and deterministic function to approximate the two-stage estimator (Equation 2). The network d is trained to output bounding box coordinates f, as a function of the input I,, and the pixels ú; within the bounding box are extracted from x; and resized to pre-specified dimensions through ş. The composition of these two functions, f(x;) = o(x,, d(xx)), returns úh, The second network Å trained predict yi given the pixels w; contained in bounding box l based on observations from S. We use the mean squared error loss for d and the categorical cross-entropy loss for ÿ. Finally, h(x) }(f(x)) is evaluated on larget domain images where the output of î is used for prediction with ÿ. See Appendix A.1 for further details.
3.3 Multiple Regions of Interest as PI
Finally, we consider a setting where privileged information indicates multiple regions of interest in an image. We use this PI in multi-label classification problems where the image I; is associated with one or more categories k from set (1...., K), encoded in multi-category label y (0, 1 (e.g., indicating findings
7

---

Published in Transactions on Machine Learning Research (09/2024)
Test
X-J
Train E2E
X,W.Y-S
Test
X-T
X.W-J
Lw (W, (W.W)
x, W--
RPN (P.P) as
f:x
W
f:X - W
Line (f.T)
Train stage!
Leat (Pu)
W
W
W
deet (Pu. T)
Train stage II
W
Ø: y
9:
LP(PY)
9
(a) Two-stage estimator.
(b) End-to-end estimator.
Figure 3: A schematic representation ion of the train and test flow for DALUPT using (a) the two-stage estimator presented in Section 2.2 and (b) end-to-end architecture based on Faster R-CNN (Ren et al., 2016). In the (wo-stage procedure, the networks f and Ò learned through empirical risk minimization of Lw and Ly, respectively. At test time, f and are combined into h = o(f(X)). The end-to-end estimator uses region proposal network (RPN) to produce regions of interest in the input image X. The RPN, which serves the network f, is followed by detection network ğ that predicts the class of any object within region proposal. Training is guided by regression losses LPRN (T', T) and Lest (Tv, T),, well as by classification losses PPNN (P, P) and Lddt (Pv). Here, T and 7 denote mround-trutie and predicted bounding box coordinates, respectively, and Tu are the predicted coordinates for a region proposal with ground-truth label U. Further, is the RPN's predicted probability that a region proposal contains an object, is a binary label assigned to the proposal based on its overlap with ground-truth bounding boxes, and P, the probability of the ground-truth tauth class within the proposal, as predicted by the detection network.
of one or more diseases). The partial label (k) 1 indicates the presence of features or objects in the image from category k. In our entity classification experiment, an object j of class & [K] in the image, say "Bird", will be annotated by bounding box tij R surrounding the pixels of the bird, and an object label Miy k. In X-ray classification, lij can indicate abnormality j in the X-ray image, and Wij E (I...., K} the label of the finding (e.g., "Pneumonia").
To make full use of privileged information, Irain a deep neural network i(x;) ( (x)), where î produces a set of bounding box coordinates by and extracts the pixels diy associated with each Eyy, and where ğ predicts a label îu for each vū.- To this end, we adapt the Faster R-CNN architocture (Ren et al., 2016) which uses a region proposal network (RPN) to generate regions that are fed to a detection network for classification and refined bounding box regression. A CNN backbone combination with the RPN region of interest pooling serves the subnetwork f. producing estimates úù; of the privileged information for an image z,. For the detection network, which corresponds to the subnetwork 9, use Fast-RCNN (Girshick, 2015).
Privileged information adds supervision through regression losses L JRPN (Ê, and Leet (f., t) for region propos- Lreg als Z and class-specific bounding box coordinates f,, We use the smooth L11 loss defined by Girshick (2015) for
8

---

Published in Transactions on Machine Learning Research (09/2024)
both L and L Training is further guided by classification losses L (p.p) = -p (1-p) log P) and ( (Px) = fog where á is the RPN's predicted probability that a region proposal contains an object, p is a binary label assigned to the proposal based on its overlap with ground-truth bounding boxes, and Pu is the probability of the ground-truth class u within the proposal, as predicted by the detection network.
In Appendix A.2, we provide details of the learning objective and architecture and describe small modifica- tions to the training procedure of Faster R-CNN to accommodate the DALUPI setting Unlike the two-stage estimator, we train Faster R-CNN (both f and 9) end-to-end, minimizing both losses at once. In entity classification experiments (see Table 3 and Figure 5), we also train this model in a LUPI setting, where no information from the larget domain used, but privileged information from the source domain is used.
4 Experiments
We evaluate the empirical benefits of learning using privileged information, compared to the other data availability settings in Table 1, across four UDA image classification tasks where PI is available in the forms described in Section 3. Widely used datasets for UDA evaluation like OfficeHome (Venkateswara et al., 2017) and large-scale benchmark suites like DomainBed (Gulrajani & Lopez-Paz, 2021), VisDA (Peng et al., 2017) and WILDS (Koh et al., 2021) do not include privileged information and cannot be used for evaluation here. Thus, we first compare our method to baselines on the recent CelebA task (Xie et al., 2020) which includes Pl in the form of binary attributes (Section 4.1). Additionally, we propose three new tasks based on well-known image classification dala sets with regions of interest as PI (Section 4.2 4.4). In Section 4.1 and 4.2, we use the two-stage estimator with the subnetwork f based on the ResNet-18 architecture (He et al., 2016a). In Section 4.3 and 4.4, we use our variant of Faster R-CNN with ResNet-50 backbone.
Our goal is to collect evidence that DALUPI improves adaptation bias and sample efficiency compared to methods that do not make use of PI. We choose baselines to illustrate these two disparate settings. First, we compare DALUPI to supervised learning baselines, SL-S and SL-T, trained on labeled examples from the source and target domain, respectively. SL-S is a simple but strong baseline: On benchmark suites like DomainBed and WILDS, there is still no UDA method that consistently outperforms SL-S (ERM) without transfer learning (Gulrajani & Lopez-Paz, 2021; Koh et al., 2021). SL-T serves oracle comparison since it uses labels from the target domain which are normally unavailable in UDA. Second, we compare DALUPI to two UDA methods- metthos-ddomain domain adversarial neural networks (DANN) (Ganin et al., 2016) and the margin disparity diserepancy (MDD) (Zhang et al., 2019) have thcoretical guarantees but do not make use of PI. These baselines are all based on the ResNet architecture. In Section 4.1, we compare DALUPI also to Tn-N-Out (Xie et al., 2020), which was designed make use of auxiliary (privileged) attributes for training domain adaptation models. We do not include this model in other experiments it was not designed to use regions of interest as privileged information. The exact architectures of all models and baselines are described in Appendix A, along with details on experimental spt and hyperparameters.
For each task and task-specific setting (label skew, amount of privileged information, etc.), we train 10 models from each relevant class using hyperparameters randomly selected from given ranges (sce Appendix A). For DANN and MDD, the trade-off parameter, which regularizes domain discrepancy in representation space, increases from to 0.1 during training: for MDD, the margin paramcter is set to 3. All models are evaluated on held-out validation set from the source domain and the best-performing model in each class is then evaluated on held-out test sets from both domains. For SL-T, we use held-out validation set from the target domain. We repeat this procedure over 5 or 10 seeds, controlling the data splits and the random number generation. We report accuracy and area under the ROC curve (AUC) with 95 % confidence intervals computed by bootstrapping over the seeds.
4.1 Celebrity Photo Classification With Binary Attributes as Pl
In the case where privileged information is available as binary attributes, we follow Xie et al. (2020) who introduced a binary classification task based on the CelebA dataset (Liu et al., 2015), where the goal is to predict whether the person in image has been identified as male or female (Y) in one of the binary
9

---

Published in Transactions on Machine Learning Research (09/2024)
1.0
Table 2: Celebrity photo classification. DALUPI per-
0.8
forms comparably to the In-N-Out models in Xie et al. (2020). Note: In-N-Out results are reported as the av- erage trials with 90 % confidence intervals.
Accuracy
0.6
SI-T
SL-S
Target accuracy
0.4
DALUPI
SL-T
86.6 (86.3, 86.9)
DANN
0.2
MDD
SL-S
78.4 17 80.0)
DANN
78.2 (76.2, 80.3)
0.25
0.50
0.75
1.00
MDD
78.3 (77.5, 79.1)
Skew E
Tn-N-Out (w/o pretraining)
78.5 (77.2, 79.9)
In-N-Out (w. pretraining)
79.4 (78.7, 80.1)
In-N-Out (rep. self-training)
80.4 (79.7, 81.1)
Figure 4: Digit classification. Target domain accu- racy function of association between back- ground and label in S. As the skew increases, the target-domain performance of the non-privileged models deteriorates.
DALUPI (W 7)
76.4 (73.8, 78.6)
DALUPI (W ~ S,
80.3 (77.9, 82.7)
attributes that accompanics the data set's photos of celobritics (X). Like Xie et al. (2020), we use 7 of the 40 other attributes (Bald, Bangs, Mustache, Smiling, 5_o_Clock Clock Shadow, Oval_ Face, and Heavy_Makeupl vector of privileged information W {0, The target and source domains wearing (T) and not wearing (8) hat. The respective datasets contain 3,000 and 2,000 images. An extra 30,000 unlabeled source samples are available to train estimators (DALUPI and In-N-Out) that can utilize privileged information from both source and target. More details can be found in (Xie et al., 2020) and in Appendix A.3.
defined by people
Table shows the target accuracy for each model. We observe that when DALUPI is provided with PI from both source and target, it performs comparably to the best-performing In-N-Out model proposed by Xie et al. (2020), while outperforming other foasible basclines average. Confidence intervals overlap for all feasible models. Notably, the best-performing In-N-Out models require four more rounds of training to achieve their results (baseline, auxiliary input, auxiliary output pre-training, tuning and self-troe tefinininng) (Xie et al., 2020). Both DALUPT and In-N-Out benefit from access to privileged information from both the source and target domain (pre/self-trainnng for In-N-Out).
Finally, is worth noting that neither covariate shift, nor sufficiency are likely to hold with respect to W in this task. Specifically, photos with none of the 7 attributes active, 0, have different label rates and majority label in S and T (the rates of labels are Ys 0.64 and Y, 0.46, respectively) and therefore P(Y[W) is not constant, i.e. covariate shift is violated. In addition, the best model we have found trained on W alone achieves only 65 % accuracy, compared to the results in Table sufficiency is unlikely to hold. Thus, DALUPI robust to violations of these assumptions.
4.2 Digit Classification With Single Region of Interest as PI
We construct a synthetic image dataset, based on the assumptions of Proposition are problems where DALUPI is guaranteed successful transfer but standard UDA is not. Starting from CIFAR-10 (Krizhevsky, 2009) images upscaled to 128 x 128, we insert random 28 x 28 digit image from the MNIST dataset (Lecun, 1998), with a label in the range 0-4, into random location of cach CIFAR-10 image, forming the input image X (see Figure 1 (lop) for examples). The label Y E {O...., 4] is determined by the MNIST digit. We store the bounding box around the inserted digit image and use the pixels contained within it as privileged information W during training. The domains are constructed using CIFAR-10' first five and last five classes as source and target backgrounds, respectively. Both source and target datasets contain 15,298 images each. To increase the difficulty of the task, make the digit be the mean color of the
verify that there
10

---

Published in Transactions on Machine Learning Research (09/2024)
0.8
SL-T
SL-8
Table 3:
Entity classification. UDA models have access to all unlabeled target samples, LUPI to all Pl (source), and DALUPI to all PI (source and target).
DALUPI
0.7
DANN
AUC
MDD
0.6
Source AUC
Targel AUC
ST.T
60.1 (58.7, 61.5)
69.0 (68.1, 69.9)
SL-S
69.5 (68.6, 70.4)
63.0 (61.6, 64.2)
0.0
0.5
1.0
0.5 mpr (7)
1.0
DANN
68.1 (67.5, 68.7)
62.5 (61.9, 63.1)
"rr(S)
MDD
62.4 (61.1, 63.9)
57.7 (56.3, 59.2)
nP
TeP (S)
LUPI
69.3 (68.5, 70.1)
65.9 (65.0, 66.8)
Figure 5: Entity classification. Target domain AUC.
DALUPI
71.4 (70.3, 72.4)
68.2 (66.3, 70.1)
The performance of SL-S and SL-T is extended across the x-axes for visual purposes. DANN and MDD use an increasing fraction of larget samples & but PI.
dataset and make the digit background transparent that the border of the image is less distinct. This may slightly violate Assumption 2 with respect to the region of interest W since the backgrounds differ between domains.
To understand how successful transfer depends on domain overlap and access to sufficient privileged infor- mation, we include a skew parameter E [2 1], where the correlation between digits and backgrounds. For source image i with digit label Y; E {0,.... (0,....4) select random CIFAR-10 image with class B, E (o,..., 4) with probability P(B, = b Y, y) {e, if y; (1 e)/(c(c 1), heerwis))). For target images, digits and backgrounds are matched uniformly at random. The choice e = e yields uniform distribution and e = is equivalent to the background carrying as much signal as the privileged information. We hypothesize that c = is the worst possible case where confusion the model is likely, which would lead to poor adaptation under domain shift.
is the number of digit classes, which determines
In Figure 4, we observe the conjectured behavior. As the skew and the association between background and label increases, the performance of SL-S decreases rapidly the target domain. At c 1, performs better than random guessing, likely because the model has learned to associate spurious features in the background with the label of the digit. We also observe that DANN and MDD deteriorate in performance with increased correlation between the label and the background. In contrast, DALUPI is unaffected by the skew as the subset of pixels extracted by f only carries some of the background with it, while containing sufficient information to make good predictions. Interestingly, DALUPI also seems to be as good or slightly better than the oracle SL-T in this setting. This may be due to improved sample efficiency from using PI.
4.3 Entitity Classification With Multiple Regions of Interest as Pl
Next, we consider multi-label classification of the presence of four types of entities (persons, cats, dogs, and birds) indicated by a binary vector Y e (0, 174 for images X from the MS-COCO dataset (Lin et al., 2014). PI is used to localize regions of interest W related to the entities, provided bounding box annotations. We define source and target domains S and T as indoor and outdoor images, respectively. Indoor images are extracted by filtering out images from the MS-COCO super categories "indoor" and "appliance" that also contain at least one of the four main label classes. Outdoor images are extracted using the super categories "vehicle" and "outdoor". In total, there are 5,231 images in the source and 5,719 images in the target domain; the distribution of labels is provided Appendix A.5.
Sufficiency is likely to hold in this task because the pixels contained in bounding box should be sufficient for an annotator to classify the entity according to the four categories above, irrespective of the pixels outside
11

---

Published in Transactions on Machine Learning Research (09/2024)
Table 4: X-ray task. Test AUC for the three pathologies in the target domain for all considered models. Boldface indicates the best-performing feasible model; SL-T uses target labels.
ATL
CM
PE
SL-T
57 (56, 58)
59 (55, 63)
79 (78, 80)
SL-S
55 (55, 56)
61 (58, 64)
73 (70, 75)
Figure 6: Left: Example from the X-ray target test set with label CM. The red rect- angle indicates the bounding box predicted by DALUPI. Right: saliency map for CM for ST-S.
DANN
53 (51, 55)
55 (53, 58)
55 (51, 61)
MDD
49 (48, 50)
51 (51, 52)
51 (48, 54)
DALUPT
55 (55, 56)
72 (71, 73)
74 (72, 76)
of the box. Similarly, covariate shift is likely to hold since the label attributed to the pixels in a bounding box should be the same, whether the entity is indoor or outdoor.
We study the effect of adding privileged information by first training the end-to-end model in a LUPI setting, using all (ar v) samples from the source domain and increasing the fraction of inputs for which PI is available, (S), from from the target domain, RPI (T), from 0 to while keeping Npr((S) 1. We train SL-S and SL-T using all available data and increase the fraction of unlabeled target samples used by DANN and MDD from 0.2 to 1 while using all data from the source domain.
1. We then train the model in a DALUPI setting. increasing the fraction of (ă, w) samples
Table 3 shows the models' source and target domain AUC, averaged over the four entity classes, when the UDA models have access to all unlabeled Larget samples, LUPI to all PI from the source domain, and DALUPT to all PT from both domains. Clearly, DALUPT yields a substantial gain adaptation. As we see in Figure 5, the performance of LUPI increases as (S) increases. When additional (ă, w) samples from the target domain are added, DALUPI outperforms SL-S and approaches the performance of SL-T. We note that DANN and MDD do not benefit as much from added unlabeled target samples as DALUPT does. Their weak performance could be explained by difficulties in adversarial training. The gap between LUPI and SL-S for (S) = 0 is anticipated; we do not expect the detection network to work well without bounding box supervision.
4.4 X-ray Classification With Multiple Regions of Interest as Pl
As a real-world application, we study detection of pathologies in chest X-ray images. We use the ChestX- ray8 dataset (Wang et al., 2017) as source domain and the CheXpert dataset (Irvin et al., 2019) as target domain. As PI, we use the regions of pixels associated with cach found pathology, as annotated by domain experts using bounding boxes. For the CheXpert dataset, only pixel-level segmentations are available, and create bounding boxes that tightly enclose the segmentations. It is not obvious that the pixels within such a bounding box are sufficient for classifying the pathology. For this reason, suspect that some of the assumptions of Proposition 1 may be violated. However, as we find below, DALUPI improves empirical performance compared to baselines for small training sets, thereby demonstrating increased sample efficiency.
We consider the three pathologies that exist in both datasets and for which there are annotated findings: atelectasis (ATL: collapsed lung), cardiomegaly (CM: enlarged heart), and pleural effusion (PE: water around the lung). There 457 and 118 annotated images in the source and target domain, respectively. We train DALUPI, DANN and MDD using all these images. SL-S is trained with the 457 source images and SL-T with the 118 target images well as 339 labeled but non-annotated larget images. Neither SL-S, SL-T, DANN, nor MDD support using privileged information. The distributions of labels and bounding box annotations are given in Appendix A.6.
This study was granted IRB approval.
12

---

Published in Transactions on Machine Learning Research (09/2024)
In Table 4, we present the per-class AUCs in the target domain. DALUPI outperforms all baseline models, including the target oracle, in detecting CM. For ATL, and PE, it performs similarly to or better than the other feasible models. That SL-T is better at predicting PE is not surprising because this pathology is most prevalent in the target domain. In Figure 6, we show a singlo-fidding image from the target test set with ground-truth label CM. The predicted bounding box of DALUPI with the highest probability is added to the image. DALUPI identifies the region of interest (the heart) and makes a correct classification. The rightmost panel shows the saliency map for the ground truth class for SL-S. We see that the gradients are mostly constant, indicating that the model is uncertain. In Appendix B, we show AUC for CM for the models trained with additional examples without bounding box annotations. We find that SL-S reaches the performance of DALUPI when large amount of labeled examples are provided. This indicates that identifiability is not the main obstacle for adaptation and that PI improves sample efficiency.
5 Related Work
Learning using privileged information was first introduced by Vapnik & Vashist (2009) for support vector machines (SVMs), and was later extended to empirical risk minimization (Pechyony & Vapnik, 2010). Meth- ods using PI, which is sometimes called hidden information or side information, has since been applied in many diverse settings such as healthcare (Shaikh et al., 2020), finance (Silva et al., 2010), clustering (Fey- ercisl & Aickelin, 2012) and image recognition (Vu et al., 2019; Hoffman et al., 2016). Related concepts include knowledge distillation (Hinton et al., 2015; Lopez-Paz et al., 2016), where a teacher model trained additional variables adds supervision to a student model, and weak supervision (Robinson et al., 2020) where so-called weak labels are used to learn embeddings, subsequently used for the task of interest. Further- more, in the realm of NLP, there is the related concept of learning using feature feedback, where additional annotations that are related to the associated task label are provided (Katakkar al., 2022; Kaushik et al., 2021). These works are mostly of an empirical nature, and theorctical work on the subject either considers linear models/SVMs (Poulis & Dasgupta, 2017) or a teacher/stuudent-lpp setup where additional supervi- sion is given when the model predicts incorrectly (Dasgupta et al., 2018). The use of PI for deep image classification has been investigated by Chen et al. (2017) and Han et al. (2023) but these works only cover regular supervised learning where source and target domains coincide. Further, Sharmanska et al. (2014) used regions of interest. in images as privileged information to improve the accuracy of image classifiers, but did not consider domain shift either.
Domain adaptation using PI has been considered before with SVMs (Li et al., 2022; Sarafianos et al., 2017), but not with more complex classifiers such as neural networks. Vu et al. (2019) used scene depth as PI in semantic segmentation using deep neural networks. However, they only used Pl from the source domain and they did not provide any theoretical analysis. Xie et al. (2020) provide some theoretical results for a similar setup to ours. However, these are specifically for linear classifiers while our approach holds for any type of classifier. Motiian (2019) investigated PI and domain adaptation using the information bottleneck method for visual recoguition. However, their setting differs from ours in that each observation comprises source-domain and target-domain features, a label and PI. Another related approach that of subsidiary tasks (Kundu et al., 2022; Ye et al., 2022). However, in these settings the additional tasks performed are used to build reprosentation that helps with the main task through domain alignment. Our approach instead seeks to use information which directly relates to the main task.
6 Discussion
We have presented DALUPI: unsupervised domain adaptation by learning using privileged information (PI). The framework provides provable guarantees for adaptation under relaxed assumptions on the input features, at the cost of collecting a larger variable set, such as attribule or bounding box ammotations, during training, Our analysis inspired practical algorithms for image classification which we evaluated using three kinds of privileged information. In our experiments, we demonstrated tasks where our approach is successful while existing adaptation mothods fail. We observed empirically also that methods using privileged information are more sample-efficient than comparable non-privileged learners, in line with the literature. In fact, DALUPT
13

---

Published in Transactions on Machine Learning Research (09/2024)
models occasionally even outperform oracle models trained using target labels due to their sample efficiency. Thus, we recommend considering these methods in small-sample settings.
The main contribution of the paper is the proposed learning paradigm for domain adaptation with privileged information. Since common benchmark datasets in UDA lack privileged information related to the learning problem, we created three new tasks for evaluating our framework, see Section 4.2-4.4, which itself is a notable contribution. We hope that this work inspires the community to develop additional datasets for UDA using privileged information.
To avoid assuming that domain overlap is satisfied with respect to input covariates, require that the label is conditionally independent of the input features given the PI that the PI is "sufficient". This is a limitation whenever sufficiency is difficult to verify. However, in our motivating example of image classification, a domain expert could choose Pl so that sufficiency reasonably justified. Moreover, in experiments on ColobA, we see empirical gains from our approach even when sufficiency is known to be violated. Another limitation that we stiill rely on overlap in the privileged information, W, which may also be violated in some circumstances. It is more likely that overlap holds for W when, for example, it is a subset of X, as argued in Figure 2. Designing experiments to test how sensitive DALUPI is to violations of these assumptions is an interesting direction for future work.
The use of regions of interest as privileged information brings up an interesting point concerning the rela- tionship between the label and the privileged information. In object detection tasks, it is natural to treat the bounding box coordinates as label information. In this work, however, the learning tasks were multi-class and multi-label image classification, not object detection. Producing a perfect box W was not the goal of the learning task, and the bounding boxes were therefore neither critical for the task nor for the labels. Instead, the bounding boxes were privileged information and our experiments in Section 4.2 4.4 sought to quantify the value of this added information, compared to not having it. Therefore, we compared our method to image classification baselines. It is not obvious a priori that learning from object locations improves the adaptation image classifiers.
If there is a lack of PI available to the models one might mitigate this by either 1) using the limited amount of PI that is available to learn and assume that it is good enough to achieve reasonable overall performance; or 2) using the learned f to create "weak" PI labels for the inputs that are missing PI, similar to the work of e.g. Robinson et al. (2020). However, one should note that the latter approach might bias the model in unintended ways and, as such, should be undertaken with some caution.
In future work, our framework could be applied to a more diverse set of tasks, with different modalities of inputs and privileged information investigate if the findings here can be replicated and extended. Moreover, such work could consider different types and degrees of shifts further corroborate the stability and resistance to noise which observe here. More broadly, using PI may be viewed "building in" domain knowledge in the structure of the adaptation problem and we see this promising direction for further research.
Acknowledgments
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation
The computations and data handling were enabled by resources provided by the National Academic Infras- tructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725.
References
Sara Beery, Grant Van Horn, and Pictro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456 473, 2018.
Shai Ben-David and Ruth Urner. On the hardness of domain adaptation and the utility of unlabeled target samples. In International Conference Algorithmic Learning Theory, pp. 139 339-153. Springer, 2012.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Percira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
Adam Breitholtz and Fredrik Daniel Johansson. Practicality of generalization guarantees for unsupervised domain adaptation with neural networks. Transactions on Machine Learning Research, 2022.
Yunpeng Chen, Xiaojie Jin, Jiashi Feng, and Shuicheng Yan. Training group orthogonal neural networks with privileged information. In Proceedings of the Twenty-Siath International Joint Conference on Artificial Intelligence, IJCAI-17, pp. 1532-1538, 2017.
Corinna Cortes, Yishay Mansour, and Mchryar Mohri. Learning bounds for importance weighting. In Advances in Newral Information Processing Systems, volume 23, 2010.
Sanjoy Dasgupta, Akansha Dey, Nicholas Roberts, and Sivan Sabato. Learning from discriminative feature feedback. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018). 2018.
Antoine de Mathelin, François Deheeger, Guillaume Richard, Mathilde Mougeot, and Nicolas Vayalis. ADAPT: Awesome Domain Adaptation Python Toolbox. arXiv preprint arXit:2107.03049, 2021.
Alexander D' D'Amour, Peng Ding, Avi Feller, Lihua Lei, and Jasjeet Sekhon. Overlap in observational studies with high-dimensional covariates. Journal of Econometrics, (21()):644-65 2021.
Jan Feyereisl and Uwe Aickelin. Privileged information for data clustering. Information Sciences, 194:4 23, 2012.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario Marchand, and Victor Lempitsky. Domain-AAvenssaril Training of Neural Networks. arXiv:1505.078. 07818 [cs, stat], May 2016. arXiv: 1505.07818.
Pascal Germain, Amaury Habrard, François Laviolette, and Emilie Morvant. PAC-Bayes and Domain Adap- tation. Neurocomputing, 979:379-391 February 2020. arXiv: 1707.05712.
Ross Girshick. Fast R-CNN. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Tshaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Dongyoon Han, Junsuk Choe, Seonghyeok Chun, John Joon Young Chung, Minsuk Chang, Sangdoo Yun, Jean Y. Song, and Scong Joon Oh. Neglected free lunch learning image classifiers using annotation byproduets. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20200 20212, October 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXie preprint arXiv:1503. 02531, 2015.
Judy Hoffman, Saurabh Gupta, and Trevor Darrell. Learning with Side Information through Modality Hallucination. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 826-834 834. TEFE, 2016.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Mark- lund, Behzad Haghgoo, Robyn Ball, Katic Shpanskaya, et al. Chexpert: large chest radiograph datasct with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intel- ligence, volume 33, pp. 590-597, 2019.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain-invariant representations In The 22nd International Conference on Artāficial Intelligence and Stalistics, pp- 527- 536. PMLR, 2019.
Bastian Jung and Fredrik Daniel Johansson. Efficient learning of nonlinear prediction models with time-scries privileged information. In Advances in Neural Information Processing Systems, 2022.
Rickard Karlsson, Martin Willbo, Zeshan Hussain, Rahul G. Krishman, David A. Sontag, and Fredrik D. Johansson. Using time-series privileged information for provably efficient learning of prediction models. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics 2022, 2021.
Anurag Katakkar, Clay H. Yoo, Weiqin Wang, Zachary Tipton, and Divyansh Kaushik. Practical benefits of feature feedback under distribution shift. In Proceedings of the Fifth BlackborNLP Workshop Analyzing and Interpreting Neural Networks for NLP, pp. 346-355, 2022.
Kaushik, A. Setlur, E. H. Hovy, and Z. C. Lipton. Explaining the efficacy of counterfactually augmented data. In International Conference on Learning Representations, 2021.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in- the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637 5664. PMLR, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Jogendra Nath Kundu, Suvaansh Bhambri, Akshay Kulkarni, Hiran Sarkar, Varun Jampani, and R. Venkatesh Babu. Concurrent subsidiary supervision for unsupervised source-free domain adaptation. In Proceedings of the 17th European Conference on Computer Vision (ECCV), pp. 177-194, 2022.
Yann Lecun. Gradient-Based Learning Applied to Document Recognition. proceedliggs of the IEEE, 86(11):
47, 1998.
Yanmeng Li, Huaijiang Sun, and Wenzhu Yan. Domain adaptive twin support vector machine learning using privileged information. Neurocomputing 469:13 27, 2022.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and Larry Zitnick. Microsoft COCO: Common Objects in Context. In ECCV. European Conference on Computer Vision, September 2014.
Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2117 2125, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3730 3738, 2015.
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S. Yu. Transfer feature learning with joint distribution adaptation. In Proceedings of the 2013 IEEE International Conference on Compuler Vision, ICCV '13, pp. 2200 2207, USA, 2013. IEEE Computer Society.
David Lopez-Paz, Leon Botton, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privi- leged information. In International Conference on Learning Representations (ICLR 2016), 2016.
Mchryar Mohri, Afshin Rostamizadch, and Amect Talwalkar. Foundations of Machine Learning. MIT Press, second edition, 2018.
Sacid Motiian.
Domain Adoptation and Privileged Information for Visual Recognition. PhD thesis, West Virginia University, 2019. Graduate Theses, Dissertations, and Problem Reports. 6271.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Shota Orihashi, Mana Ihori, Tomohiro Tanaka, and Ryo Masumura. Unsupervised Domain Adaptation for Dialogue Sequence Labeling Based on Hierarchical Adversarial Training. In Interspeech 2020, pp. 1575 1579. ISCA, October 2020.
Dmitry Pechyony and Vladimir Vapnik. On the thcory of learnining with privileged information. In Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc. 2010.
Xingchao Peng, Ben Usman, Neela Kanshik, Judy Hoftman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint or vxXim17710000922 2017.
Stefanos Poulis and Sanjoy Dasgupta. Learning with feature feedback: from theory to practice. In Interna- tional Conference on Artificial Intelligence and Statistics (AISTATS), 2017.
S Ren, K He, R Girshick, and J Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137-1149, 2016.
Joshua Robinson, Stefanie Jegelka, and Suvrit. Sra. Strength from Weakness: Fast Learning Using Weak Supervision. In Proceedings of the 37th International Conference on Machine Learning, pp. 8127 8136.
PMLR, November 2020.
Nikolaos Sarafianos, Michalis Vrigkas, and Ioannis A. Kakadiaris. Adaptive SVM+: Learning with privileged information for domain adaptation. In Proceedings of the TEEE International Conference on Computer Vision (ICCV) Workshops, Oct 2017.
Tawseof Ayoub Shaikh, Rashid Ali, and M. M. Sufyan Beg. Transfer learning privileged information fuels CAD diagnosis of breast cancer. Machine Vision and Applications, 31(1):9, February 2020.
Viktoriia Sharmanska, Novi Quadrianto, and Christoph H. Lampert. Learning to Transfer Privileged Infor- mation. 1rxix:1410.038 fes, stat], October 2014. arXiv: 1410.0389.
Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z. Haochen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 19847 19878. PMLR, 17 23 Jul 2022.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the lig-likelihood function. Journal of statistical planning and inference, 90(2):227 2224,444444 2000.
Catarina Silva, Armando Vieira, Antonio Gaspar-Cure Cunha, and Joao Carvalho das Neves. Financial distress model prediction using SVM+. In Proceedings of the International Joint Conference on Neural Networks, pp. 7, July 2010.
M. Sugiyama, T. Suzuki, and T. Kanamori. Density Ratio Estimation in Machine Learning. Cambridge books online. Cambridge University Press, 2012.
Marian Tietz, Thomas J. Fan, Daniel Nouri, Benjamin Bossan, and skorch Developers. skorch: scikit-learn compatible neural network library that wraps PyTorch, July 2017.
Vladimir Vapnik and Rauf Izmailov. Learning Using Privileged Information: Similarity Control and Knowl- edge Transfer. Journal of Machine Learning Research, 16:2023 20499, 2015.
Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Newral Networks, 22(5):544-557, July 2009.
Hemanth Venkateswara, Jose Euscbio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognition, pp. 5018-5027, 2017.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthicu Cord, and Patrick Pérez Pérez. Dada: Depth- aware domain adaptation in semantic segmentation. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 7363 7372, 2019.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx- ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and local- ization of common thorax diseases. In Proceedings of the IEEE conference on compuler vision and patlern recognition, pp. 2097-2106, 2017.
Yifan Wu, Ezra Winston, Divyansh Kanshik, and Zachary Lipton. Domain adaptation with asymmetrically- relaxed distribution alignment. In International conference on machine learning, pp. 1872-6881. PMLR, 2019.
Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. In-n-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness. arXiv preprint arXiv:2012.04 04550, 2020.
Yalan Ye, Ziqi Liu, Yangwuyong Zhang, Jingjing Li, and Hengtao Shen. Alleviating style scnsitivity then adapting: Source-free domain adaptation for medical image segmentation. In Proceedings of the 30th A ACM International Conference on Multimedia, MM '22 pp. 1935 1944, New York, NY, USA, 2022. Association for Computing Machinery.
John R Zech, Marcus Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, :5 10002683, 2018
Yuchon Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International conference on machine learning, pp. 7404-7413. PMLR, 2019.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representa- tions for domain adaplation. In International Conference on Machine Learning, pp. 7523-77377777 7532. PMLR, 2019.
A Experimental Details
In this section, give further details of the experiments. All code is written in Python and we mainly use PyTorch in combination with skorch (Tietz et al., 2017) for our implementations of the net- works. For Faster R-CNN, we adapt the implementation provided by torchvision through the func- tion fasterrcin_reeneee
fpn. For DANN and MDD, we use the ADAPT TensorFlow implementation (de Mathelin et al., 2021) with a ResNet-50- based encoder. We initially set the trade-off parameter X, which controls the amount of domain adaption regularization, to and then increase it 0.1 in 10,000 gradient steps according to the formula X B(2 e P) 1)/C, where p increases lincarly from 0 to 1, 3 is parameter specified for each experiment, and C = 2/(1 + +e-1)-1 - 1. For MDD, we fix the margin parameter to 3. The source and target baselines are based on the ResNet-50 architecture when P1 is provided as multiple regions of interest; otherwise, the ResNet-18 architecture is used. The architecture of DALUPI in each experiment is specified in the respective subsection below.
We use the Adam optimizer all experiments. Learning rate decay treated as a hyperparameter. For ADAPT models (DANN and MDD), the learning rate is either constant or decayed according to po/(1 + ap)3/4, where po is the initial learning rate, increases linearly from 0 to 1, and is parameter specified in each experiment (sce below). For non-ADAPT models, the learning rate is either constant or decayed by factor 0.1 every nth epoch, where n is another hyperparameter.
For all models except LUPI and DALUPI, the classifier network following the encoder is a simple MLP with two possible settings: Either it single linear layer from inputs to outputs or thrco-layer network with ReLU activations between the layers. This choice treated as a hyperparameter our experiments. The nonlinear case has the following structure where is the number of input features:
18

---

Published in Transactions on Machine Learning Research (09/2024)
fully connected layer with neurons
ReL.U activation layer
fully connected layer with n//2 neurons
ReLU activation layer
fully connected layer with n/44 neurons.
All models were trained using NVIDIA Tesla A40 GPUs and the development and evaluation of this study required approximately 30,000 hours of GPU training. The code is available GitHub: https://gibs con/Heaalthy-AA/dalup
A.1 DALUPI With Two-stage Classifier
Here, we describe in more detail how we construct our two-stage classifier for image classífication when privileged information provided as single region of interest as in the digit classification task (Section 4.2). When privileged information provided as binary attributes, we can directly learn the two-stage estimator according Equation In this task, was found that using the cross entropy loss and using continuous outputs from provided superior performance compared to other losses. In the digit classification task, each image z; has a single label yi E {0,.... 4} determined by the MNIST digit. Privileged information is given by a single bounding box with coordinates t; E R enclosing subset of pixels w, corresponding to the digit. The training procedure is summarized in Algorithm 1 and further described below.
We first learn d which is a function that takes target image data, ã,, and bounding box coordinates, 4, and learns to output bounding box coordinates, l,, which should contain the privileged information we- Note that we do not exactly follow the setup in Equation 2 since we do not need to actually predict the pixel values within the bounding box. If we find good enough estimator of t, we should minimize the loss of in Equation 2. To obtain the privileged information we apply deterministic function á which crops and scales an image using the associated bounding box, t,. We can now write the composition of these two functions as f(x;) o(x; d(x;)) which outputs the privileged information. The function á is hard-coded and therefore not learned.
In the second step, we learn 9 to predict the label from the privileged information wi, which a cropped version of z; where the cropping is defined by the bounding box around the digit. This cropping and resizing is performed by o. When we evaluate the performance of this classifier combine the two models into one, h(x)) ((((,, d(x:)). We use the mean squared error loss for learning d and categorical cross- entropy (CCE) loss for 9.
Algorithm 1 Training of the two-stage model.
1: procedure
(æz, Wi, 14, 14)
Empirically minimize 1 - [d(ā,) t.l and obtain d. Empirically minimize T "D Di= CCE(g(w))) v1) and obtain 9. 4: Compose d, and & into h((x) 000(1, d(x))).
5: end procedure
A.2 DALUPI With Faster R-CNN
For multi-label classification, we adapt Faster R-CNN (Ren et al., 2016) outlined in Figure 7 and described below. Faster R-CNN uses a region proposal network (RPN) to generate region proposals which are fed to a detection network for classification and bounding box regression. This way of solving the task in subscquent steps has similarities with our two-stage algorithm although Faster R-CNN can be trained end-to-end. We make small modifications to the training procedure of the original model in the end of this section.
19

---

Published in Transactions on Machine Learning Research (09/2024)
Object proposals
Peature Maps
Res
Rol Pooling
RPN
det
Reference boxes
The RoT pooling layer and the classification and regression layers are part of the Fast R-CNN detection network (Girshick, 2015).
Figure 7: Faster R-CNN (Ren et al., 2016) architecture.
Seneters-centeredd The RPN generates region proposals relative to a fixed number of reference boxes anchors centered at the locations of a sliding window moving over convolutional feature maps. Each anchor assigned a binary label p e {0, 1} based on its overlap with ground-truth bounding boxes; positive anchors are also associated with a ground-truth box with location t. The RPN loss for single anchor is
RPN (pp,,,, L (f. + RPN (f, (),t),
where f represents the refined location of the anchor and p is the estimated probability that the anchor contains an object. The binary cross-entropy loss and a smooth Lı loss are used for the classification loss RPN and the regression loss RPN respectively. For a mini-batch of images, the total RPN loss is computed based on a subset of all anchors, sampled to have a ratio of up to 1:1 between positive and negative ditto.
A filtered set of region proposals are projected onto the convolutional feature maps. For each proposal, the detection wetwork- Paat R-CNN (Girshick, 2015) outputs probability p(k) and predicted bounding box location i(k) for each class k. Let (ø(0), ................................ .(())) p(K)), where Σx = 1, K is the number of classes and O represents catch-all background class. For single proposal with ground-truth coordinates t and multi-class label u E (O...., K}, the detection loss is
I det (p, Eur t) det (p, u) + I,21 Ldee (Fu, f),
(4)
where T. (p, u) log and T.det a smooth In loss. To obtain probability vector for the entire image, we maximize, for each class k, over the probabilities of all proposals.
During training, Faster R-CNN requires that all input images tation (bounding box) w and its corresponding label u. To increase sample-efficiency, we enable training the model using non-annotated but labeled samples (I, v) from the source domain and annotated but unlabeled samples (ø, .) from the target domain. In the RPN, no labels are needed, and we simply ignore anchors from non-annotated images when sampling anchors for the loss computation. For the computation of Equation -1 all ground-truth annotations from the
with at least ground-truth amno-
4, we handle the two cases scparately. We assign the label (arget domain and multiply L.dt by the indicator I,20- For non-annotated samples (a, from the source domain, there
box-specific coordinates t or labels but only the labels y for the entire image. In this case, 4 is undefined and we instoad compute the binary cross-entropy loss between the per-image label and the probability vector for the entire image.
We train the RPN and the detection network jointly as described in Ren et al. (2016). To extract feature maps, we use a Feature Pyramid Network (Lin et al., 2017) on top of ResNet-50 architecture He et al. (2016b). We use the modified model in the experiments Section 4.3 and 4.4. In Section 4.3, we also train this model in LUPI setting, where information from the target domain is used.
20

---

Published in Transactions on Machine Learning Research (09/2024)
A.3 Celebrity Photo Classification With Binary Attribute Vector
In our experiment based on CelebA (Liu et al., 2015), the inpul a is an RGB image which has been resized to 64x64 pixels, the target y is binary label for gender of the subject of the image, and the privileged information w are 7 binary-valued attributes. The attributes used in this experiment are: Bald, Bangs, Mustache, Smiling, dataset with 2,000 labeled source examples and 3,000 unlabeled target examples. We use 1,000 samples each for the source validation set, source test set, and target test set, respectively. The target oracle, SL-T, is trained using labels provided for the 3,000 target examples, with 20 % of these examples set aside for validation. The same unlabeled validation set is used to validate the first DALUPI network, f. When using privileged information from the source domain to train f, we use 30,000 extra samples (2,, w) with PI.
Beccclokkkhaaale Oval_Face and Deavy_Makeup. We use subset of the CelebA
For DALUPI, use the two-stage istior with the network î based on ResNet-18 followed by non-linear MLP. The network is an MLP with two hidden layers of with 256 neurons cach. We train the models for 100 epochs. the validation accuracy (or validation AUC for f) does not improve for 10 subsequent epochs, we stop the training earlier. For DALUPI, the early stopping patience the problem as multi-class classification with two classes and use the categorical cross entropy loss for SI.-S, SL-T, DANN, and MDD.
for each network. We treat
A.3.1 Hyperparameters
We randomly choose hyperparameters from the following predefined sets of values:
SL-S and SL-T:
batch size: (16,32,
32, 64)
initial learning rate: (1.0 x 10-4, 1.0 x 10 10-3)
step size for learning rate decay: (15, 30, 100)
weight decay: (1.0 x 10 1.0 x 10-3)
dropout (encoder): (0, 0.1, 0.2, 0.5)
nonlinear classifier: (True, False).
DALUPT:
batch size: (16, 32, 64)
initial learning rate: (1.0 x 1 1.0 x 10-4, 1.0 x 10-3)
step size for learning rate decay: (15, 30, 100)
weight decay: (1.0 x 10-4, 1.0 x 10-3).
DANN:
batch size: (16,32, 32, 64)
initial learning rate: (1.0 x 10-4, 1.0 x 10 -3
parameter for learning rate decay: (0, 1.0)
weight decay: (1.0 x 10-4 1.0 x 10-3)
0.2, dropout (encoder): (0, 0.1, 0.5)
width of discriminator network: (64, 128, 256)
depth of discriminator network: (2,3)
nonlinear classifier: (True, False)
parameter & for adaption regularization decay: (0.1, 1.0, 10.0).
MDD:
batch size: (16, 06,322.00 64)
initial learning rate: (1.0 10-1, 1.0 x 10-3)
21

---

Published in Transactions on Machine Learning Research (09/2024)
parameter for learning rate decay: (0, 1.0)
weight docay: (1.0 x 10-4, 1.0 x 10-3)
dropout (encoder): (0, 0.1, 0.2,
nonlinear classifier: (True, False)
maximum norm value for classifier weights: (0.5, 1.0, 2.0)
parameter 3 for adaption regularization decay: (0.1, 1.0, 10.0).
A.4 Digit Classification With Single Bounding Box as PI
In the digit classification task, separate 20 % of the available source and target data into a test set. We likewise 20 20% of the training data for validation purposes. For DALUPI
ResNet-18 for the function f. We replace the default fully connected layer with fully connected layer with 4 neurons to predict the coordinates of the bounding box. The predicted bounding box is resized to a 28 x 28 square no matter the initial size. We use simple convolutional neural network for the function 9 with the following structure:
convolutional layer with 16 output channels, kernel size of stride of 1, and padding of 2
max pooling layer with kernel size 2, followed by a ReLU activation
convolutional layer with 32 output channels, kernel size of stride of 1, and padding of 2
max pooling layer with kernel size 2, followed by ReLU activation
dropout layer with = 0.4
fully connected layer with 50 out features, followed by ReLU activation
dropout layer with 0.2
fully connected layer with out features.
The model training is stopped when the best validation accuracy (or validation loss for f) does not improve over 10 epochs or when the model has been trained for 100 epochs, whichever occurs first. All models are trained from scratch, without pretrained weights. We use the categorical cross entropy loss for SL-S, SL-T, DANN, and MDD.
A.4.1 Hyperparameters
We randomly choose hyperparameters from the following predefined sets of values:
SL.- and SI-T:
batch size: (2,,32,64)
initial learning rate: (1.0 x 10-4, 1.0 x 10-3) step size n for learning rate decay: (15, 30, 100) weight decay: (1.0 10-1, 1.0 x 10-3)
dropout (encoder): (0, 0.1, 0.2, 0.5)
nonlinear classifier: (True, False).
DALUPI:
batch size: (16, 32, 32,64)
initial learning rate: (1.0 1 1.0 x 10-3)
step size n for learning rate decay: (15, 30, 100)
weight decay: (1.0x 10-4 1.0 10-3).
22

---

Published in Transactions on Machine Learning Research (09/2024)
Table 5: Marginal label distribution in source and target domains for the entity classification task based on the MS-COCO dataset. The background class contains images where none of the four entities are present.
Domain
Person
Dog
Cat
Bird
Background
Source 2,963
569
1,008
213
1,000
Target
3,631
1,121
423
712
1,000
DANN:
batch size: (16,32,64)
initial learning rate: (1.0 x 10-4, 1.0 x 10-3)
parameter for learning rate decay: (0, 1.0)
weight decay: (1.0 10-4 x x 10-3)
dropout (encoder): (0, 0.1, 0.2,
width of discriminator network: (64, 128, 256)
depth of discriminator network: (2.3)
nonlinear classifier: (True, False)
parameter 3 for adaption regularization decay: (0.1, 1.0, 10.0).
MDD:
batch size: (16,35 32,64)
initial learning rate: (1.0 X 10-4, 1.0 x 10 10-3)
parameter a for learning rate decay: (0, 1.0)
weight docay: (1.0 x 10-4, 1.0 x 10-3)
dropout (encoder): (0, 0.1, 0.2, 0.5)
nonlinear classifier: (True, False)
maximum norm value for classifier weights: (0.5, 1.0, 2.0)
parameter 3 for adaption regularization decay: (0.1, 1.0, 10.0).
A.5 Entitity Classification With Multiple Regions of Interest Pl
In the entity classification experiment, we train all models for most 50 epochs. If the validation AUC does not improve for 10 subsequent epochs, stop the training earlier. No pretrained weights are used in this experiment since we find that the task is too easy to solve with pretrained weights. For DALUPT and LUPI, we use the end-to-end solution based on Faster R-CNN (see Section A.2). We use the default anchor sizes for each of the feature maps (32, 64, 128, 256, 512), and for each anchor size we use the default aspect ratios (0.5, 1.0, 2.0). We use the binary cross entropy loss for SL-S, SL-T, DANN, and MDD.
We use the 2017 version of the MS-COCO datasct (Lin et al., 2014). As decribod in Section 4.3, we extract indoor images by sorting out images from the super categories "indoor" and "appliance" that also contain at least one of the entity classes. Outdoor images are extracted in the same way using the super categories "vehicle" and "outdoor". Images that match both domains (for example an indoor image with toy car) are removed, as are any gray-scale images. We also include 1,000 negative examples, i.c., images with none of the entities present, in both domains. In total, there are 5,231 images in the source domain and 5,719 images in the target domain. From these pools, we randomly sample 3,000, 1,000, and 1,000 images for training, validation, and testing, respectively. In Table 5 we describe the label distribution in both domains. All images are resized to 320 320.
A.5.1 Hyperparameters
We randomly choose hyperparameters from the following prodefined sets of values. For information about the specific parameters in LUPI and DALUPI, we refer the paper by Ren et al. (2016). Here, Rol and NMS refer to region of interest and non-maximum suppression, respectively.
23

---

Published in Transactions on Machine Learning Research (09/2024)
SL-S and SL-T:
batch size: (16,32,64) 64)
initial learning rate: (1.0 x 10-1, 1.0 x 10-3)
step size for learning rate decay: (15, 30, 100)
weight decay: (1.0 10-4 1.0 10-3)
dropout (encoder): (0, 0.1, 0.2, 0.5)
nonlinear classifier: (True, False).
DANN:
batch size: (16, 64)
initial learning rate: (1.0 x 10-4, 1.0 x 10 10-3)
parameter a for learning rate decay: (0, 1.0)
weight decay: (1.0 x 10 1.0 x 10-3)
dropout (encoder): (0, 00,0.1,02200.5
width of discriminator network: (64, 128, 256)
depth of discriminator network: (2,3)
nonlinear classifier: (True, False)
parameter 3 for adaption regularization decay: (0.1, 1.0, 10.0).
MDD:
batch size: (1,,32,00 64)
initial learning rate: (1.0 10-4, 1.0 x 10-3)
parameter for learning rate decay: (0, 1.0)
weight decay: (1.0 x 10 1.0 x 10-3)
dropout (encoder): (0, 0.2, 0.5)
nonlinear classifier: (True, False)
maximum norm value for classifier weights: (0.5, 1.0, 2.0)
parameter 3 for adaption regularization decay: (0.1, 1.0, 10.0).
LUPI and DALUPI:
batch size: (16,32, 64)
learning rate: (1.0 10-4, 1.0 10-3)
step size for learning rate decay: (15, 30, 100)
weight decay: (1.0 x 10-1, 1.0 x 10-3)
IoU foreground threshold (RPN): (0.6, 0.7, 0.8, 0.9)
ToU background threshold (RPN): (0.2, 0.3, 0.4)
batchsize per image (RPN): (32, 64, 128, 256)
fraction of positive samples (RPN): (0.4, 0.5, 0.6, 0.7)
NMS threshold (RPN): (0.6, 0.7, 0.8)
Rol pooling output size (Fast R-CNN): (5, 7, 9)
ToU foreground threshold (Fast R-CNN): (0.5, 0.6)
ToU background threshold (Fast R-CNN): (0.4, 0.5)
batchsize per image (Fast R-CNN): (16, 32, 64, 128)
fraction of positive samples (Fast R-CNN): (0.2, 0.25, 0.3)
NMS threshold (Fast R-CNN): (0.4, 0.5, 0.6)
detections per image (Fast R-CNN): (25, 50, 75, 100).
24

---

Published in Transactions on Machine Learning Research (09/2024)
Table Marginal distribution of labels of images and bounding boxes the source and target do- main, respectively, for the chest X-ray classification experiment. PE-Effusion; NF=No Finding.
ATL=Atcloctasis; CM=Cardionegaly;
Data
ATL
CM
PE
NF
I ~ S
11,559
2,776
13,317
60,361
W ~ S
180
146
153
3 14,278
20,466
74,195
16,996
T
75
66
64
A.6 X-ray Classification With Multiple Regions of Interest as Pl
In the X-ray classification experiment, we train all models for at most 50 epochs, using pre-trained weights in the ResNet architecture of each model. the validation AUC does not improve for 10 subsequent epochs, we stop the Lraining earlier. We then fine-tune all models, except DANN and MDD, for up to 20 additional epochs. The number of encoder layers that are fine-tuned a hyperparameter for which we consider different values. We start the training with weights pretrained on ImageNet. For DALUPI, we use the end-to-end solution based on Faster R-CNN (see Section A.2). We use the default anchor sizes for each of the feature maps (32, 64, 128, 256, 512), and for each anchor size we use the default aspect ratios (0.5, 1.0, 2.0). We use the binary cross entropy loss for SL-S, SL-T, DANN, and MDD.
In total, there are 83,519 (457) and 120,435 (118) images (annotated images) in the source and target domain, respectively. The distributions of labels and bounding box annotations are provided Table Here, "NF" refers to images with no confirmed findings. In the annotated images, there are 180/146/11 and 75/66/64 examples of ATL/CM/PE in each domain respectively. Validation and test sets are pled from non-annotated images and contain 10,000 samples each. All annotated images are reserved for training. We merge the default training and validation datasots before splitting the data and resize all images to 320 x 320. For the source dataset (ChestX-ray8), the bounding boxes can be found together with the dataset. The target segmentations can be found here: https: http:://staffrrdammm at azurewebsites net. 1att--a000---d----0-------------
A.6.1 Hyperparameters
We choose hyperparameteers randomly from the following predefined sets of values. For information about the specific parameters in DALUPI, we refer to the paper by Ren et al. (2016). Rol and NMS refer to region of interest and non-maximum suppression, respectively.
SL-S and SL-T:
batch size: (16, 32, 64)
learning rate: (1.0 10-1, 1.0 10-3)
weight decay: (1.0 -4,4, 1.0 10-3)
dropout (encoder): (0, 0.1, 0.2, 0.5)
nonlinear classifier: (True, False)
number of layers to fine-tune: (3, 4, 5)
learning rate (finc-tuning): (1.0 10-5 1.0 10-4).
DANN:
batch size: (16, 32, 64)
initial learning rate: (1.0 10-4, 1.0 x 10-3)
parameter a for learning rate decay: (0, 1.0)
weight decay: (1.0x 10-4 1.0 10-3)
25

---

Published in Transactions on Machine Learning Research (09/2024)
number of trainable layers (encoder): (1, 2, 3, 4, 5)
dropout (encoder): (0, 0.1, 0.2, 0.5)
width discriminator network: (64, 128, 256)
depth of discriminator network: (2,3)
nonlinear classifier: (True, False)
parameter 3 for adaption regularization decay: (0.1, 1.0, 10.0).
MDD:
batch size: (1,,2,00 64)
initial learning rate: (1.0 10-1, 1.0 x 10-3)
parameter a for learning rate decay: (0, 1.0)
weight decay: (1.0 x 10-4, 1.0 x 10-3)
number of trainable layers (encoder): (1, 2, 3, 5)
dropout (encoder): (0, 0.2,0
nonlinear classifier: (True, False)
maximum norm value for classifier weights: (0.5, 1.0, 2.0)
parameter (3 for adaption regularization decay: (0.1, 1.0, 10.0).
DALUPI:
batch size: (16,32, 64)
learning rate: (1.0 10=4)
weight decay: (1.0 -4,4, 1.0 10-3)
IoU foroground threshold (RPN): (0.6, 0.7, 0.8, 0.9)
IoU background threshold (RPN): (0.2, 0.3, 0.4)
batchsize per image (RPN): (32, 64, 128, 256)
fraction of positive samples (RPN): (0.4, 0.5, 0.6, 0.7)
NMS threshold (RPN): (0.6, 0.7, 0.8)
Rol pooling output size (Fast R-CNN): (5, 7, 9)
IoU foreground threshold (Fast R- R-CNN): (0.5, 0.6)
IoU background threshold (Fast R-CNN): (0.4, 0.5)
batchsize per image (Fast R-CNN): (16, 32, 64, 128)
fraction of positive samples (Fast R-CNN): (0.2, 0.25, 0.3)
NMS threshold (Fast R-CNN): (0.4, 0.5, 0.6)
detections per image (Fast R-CNN): (25, 50, 75, 100)
learning rate (fine-tuning:: (1.0 10-5, 1.0 10-4)
number of layers fine-tune: (3, 4, 5).
B
Additional Results
In Figure 8a and 8b, we show some example images from the digit classification task with associated saliency maps from the source-only model for different values of the skew parameter e. We can see that for a lower value of epsilon the SL-S model activations seem concentrated on the area with the digit, while when the correlation with the background is large the model activations are more spread out.
In Figure 9, show the average AUC when additional training data of up to 30,000 samples added in the chest. X-ray experiment. We see that, once given access to a much larger amount of labeled samples, SL-S and DALUPI perform comparably in the target domain.
In Figure 10, we show AUC for the pathology CM when additional training data without bounding box annotations are added. We see that SL-S catches up to the performance of DALUPI when large amount of labeled examples are provided. These results indicate that identifiability is not the primary obstacle for adaptation, and that PI improves sample efficiency.
26

---

Published in Transactions on Machine Learning Research (09/2024)
(a) SL-S, c = 0.2
(b) SL-S, c 1.0
Figure Example images (top) and saliency maps (bottom) from SL-S when trained with source skew 0.2 (a) and (b).
1.0
1.0
0.8
0.8
AUC
0.6
AUC
0.6
SI-T
SL-T
SL-S
SL-S
0.4
0.4
DALUPI
DALUPI
DANN
DANN
0.2
MDD
0.2
MDD
10000 20000
30000
0
10000
20000
30000
Extra (I, 9) samples
Extra vI samples
(a) Source test AUC.
(b) Target test AUC.
Figure 9: Classification of chest X-ray images. Model performance on source (a) and targel (b) domains. The AUC is averaged over the three pathologies: AIL, CM and PE. The 95 % confidence intervals are computed using bootstrapping the results over five seeds.
C Proof of Proposition 1
Proposition. Let Assumptions 1 and be satisfied w.r.t. W (not necessarily w.r.t. X) and let Assumption 3 hold stated. Then, the target risk Ry is identified for hypotheses h X V,
(h) [
T(x) [ T(w x)) S(y L))(h(>), y)dydwdr.
and, for L the squared loss, a minimizer of Ry is hz)(x) = Ju T(w x) w|dw
Proof. By definition, (h) = Ja.y T(x, v) ( dydd.. We marginalize over W to get T(x,y) T(x) (x)) [T W,x) æ]]
=T [ W) x]
= =T(x)) [
T(w [))(yy w)dw
(=:(1)>>
=T(x) w=:(W)>>
T(w x))(yy w)dw
where the second equality follows by sufficiency and the third by covariate shift and overlap in W. T(x), T(w w) and S(y w) observable through training samples. That hy is minimizer follows from the first-order
27

---

Published in Transactions on Machine Learning Research (09/2024)
1.0
0.8
AUC (CM)
0.6
SL-T
SL-S
0.4
DALUPI
DANN
0.2
MDD
5000
10000
Extra (z, samples
Figure 10: Test AUC for CM in T. DALUPI outperforms the other models when no extra (z, y) samples are provided. Adding examples without bounding box annotations improves the performance of SL-S and SL-T, eventually causing the latter to surpass DALUPI.
condition of setting the derivative of the risk with respect to h to 0. This strategy yields the well-known result that
h arg min E Y²) Ey x]
By definition and the previous result, have that
y T(x,u) dy T(x)
E X x]
T(w x)S(1 | wyydway
S::((>>
T(w I))Es{Y x]dw
and we have the result.
D Proof of Proposition 2
Proposition 2. Assume that G comprises M-Lipschila mappings from the privileged informalion space W C Rdw to V. Further, assume that both the ground truth privileged information W and label Y are deterministic in X and W respectively. Let be the domain density ratio of W and let Assumptions 1-3 (Covariate shift, Overlap and Sufficiency) hold w.r.t. W. Further, let the loss L be uniformly bounded by some conslant B and let and d' be the pseudo-dimensions of G and F respectively. Assume that there are n observations from the source (labeled) domain and m from the target (unlabeled) domain. Then, with L the squared Euclidean distance, for any h h hof E E x F, w.p. least 1 1-
(1
PY.P (g) + M
2nc dlog d
+ log
+ 2/4 ST(
2d' log
log dw
ddw BM²
V
n
2n
28

---

Published in Transactions on Machine Learning Research (09/2024)
Proof. Decomposing the risk of h o Φ get
RT(h) = ET(((((()) - Y)
s 22 - r)
s 2 -Y)² + M²Ilf(()) - 9(W)l1
-
- + (I)(((X) - -WII
22)(2) 2M12 (f) 2 25 (g) +2M R (f)
The first inequality follows from the relaxed triangle inequality, the second inequality from the Lipschitz property and the third equality from Overlap and Covariate shift. We will bound these quantities separately starting with (T).
We assume that the pseudo-dimension of 9, d is bounded. Further, we assume that the second moment T(w) of the density ratios, equal to the Rényi divergence d2(T]] EwendT(w) S(w) are bounded and that the density ratios are non-zero for all E G. Let D1 {w,, 3/ Im be dataset drawn iidd from the source domain. Then by application of Theorem 3 from Cortes al. (2010) obtain with probability 1 5 over the choice of D1.
(T) = ( & (O) + 25/4 (2(115) 3/8 dlog 2me log 4
Now for (IT) we treal each component of w E W as a regression problem independent from all the others. So we can therefore write the risk as the sum of the individual component risks
R (f) Ddw R (T,
Let the pseudo-dimension of F denoted d, D2 = {x;, w; J be a dataset drawn i.id from the target domain. Then, using theorem 11.8 from Mohri et al. (2018) we have that for any 5 0, with probability least 1 5 over the choice of D2, the following inequality holds for all hypotheses f E F for each component risk
2d log d' n
log 1
RT,0 S NW RF+++
+
2n
We then simply make all the bounds hold simultaneously by applying the union bound and having it so that each bound must hold with probability dw
á which results in
2d log
en o
log dxz
B ( - RWW R() S wow NT + DAN B
à
2n
2dl log en of
log twy 8
= 12 dwB
2n
Combination of these two results then yield the proposition statement.
Consistency follows as is a deterministic function of W and W a deterministic fundtion of X and both H and F are well-spocifiod. Thus both empirical risks and sample complexity terms will converge to the limit of infinite samples.
The parts of the bound shown above can be described falling into three main categories: Empirical risk(s), domain shift and sample complexity components. A central term that figures both in the weighted empirical
29

---

Published in Transactions on Machine Learning Research (09/2024)
risk and the Rényi divergence is the density ratio T(w) Therefore, the size of the bound is governed least in part based on the proximity in W-space the source and target domains are. This is similar to other importance weighting bounds, however, since the experiment designer may choose the form of PI this be more well-behaved than the density ratio in the input space.
E Proof Sketch for PAC-Bayes Bound
We will here detail proof sketch for PAC-Bayes version of the bound we propose in the main text. For the purposes this bound we will consider the quantity En where &b a posterior distribution over classifiers ~ v. As we are basing the bound on the two-step methodology where we train two different classifiers on separate datasots we assume that we can obtain the posteriors over the component functions separately and independently i.e. h f g 1 = Wgs where f~ thy and g Dg- Let the assumptions from proposition 2 hold here. Similar to the previous section decompose the risk into two parts
Enwp Ry = ( - -Y)
= E 12R 2M
RY (f)l = 2E Ry (g) +2M B
Nom RW (
We note that since we now have expectations over the composite function h on expressions which depend only one of the components we can, for example, write the following:
E R$ (g) = Ey Rg(g)
This holds as we assume that f and g are not dependent on each other. Therefore, we can just marginalize out the part which is not in use. From this point we can use some the available bounds from the literature to estimate the resulting part e.g. Corollary from Breitholtz & Johansson (2022). Application of this result yields the following bound on the first term
< 1 E R5
KL (L(W5 + In(
E R)(9)) gr
(g) + 820 7g-OA
2y(1 7)m
Thereafter we can use another bound from the literature to estimate the second term, e.g. Theorem from Germain al. (2020). Using this obtain the following:
(E (W (1) (L(bg]laT) + In( (~O)
E R ())
T- --
na
Then bound can be constructed by combining these two results using union bound argument.
E Rr(h)
2
E
R
+ 800
(L( + In(3)
7g-
2y(1 7)m
2M²a
KT(Wy|(TT) + In(3)
+ 1-6-a
E (1)+ S:
na
F A Bound on the Target Risk Without Suffiency
The sufficiency assumption is used to replace T(y x) with T(y w) in the proof of Proposition 1. If sufficiency is violated but it is plausible that the degree of insufficiency is comparable across domains, we can still obtain a bound on the target risk which may be estimated from observed quantities. One way to formalize such an assumption that there is some y 2 1, for which
sup Tyy w. xT/T w) - Y sup S(y w. w,))((y w) zES(r/))
(5)
30

---

Published in Transactions on Machine Learning Research (09/2024)
This may be viewed relaxation of suffiency. If Assumption 3 holds, both left-hand and right-hand sides of the inequality are 1. Under Equation 5, with S,, v) equal to the right-hand side the inequality,
( T(x) / T(w x) / A, u) ())(h(r), yddydwdx. (T( - 1
However, the added assumption is not verifiable statistically.
31


Reviews:
[{"review_id": "KOH5ro3CqE", "reviewer": "Reviewer_jGW2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\n- The paper provides a theoretical framework for unsupervised domain adaptation by learning using privileged information (DALUPI). This framework relaxes the strong assumptions typically required in UDA, particularly the need for input domain overlap. This is a significant advancement in the field, as it allows for adaptation in scenarios where traditional UDA methods might fail.\n\n- The authors conduct several experiments for image classification using three different types of privileged information, including on a real-world medical imaging application.\n\n- The experimental results show that DALUPI models often outperform baseline methods, especially in scenarios with limited training data. In some cases, DALUPI even outperforms oracle models trained on target labels, highlighting the method's strong sample efficiency. This is a crucial advantage in many real-world scenarios where labeled data is scarce or expensive to obtain.\n\n## Weaknesses:\n- The theoretical guarantees of DALUPI rely on the assumption that the privileged information is ``sufficient'' for predicting labels. While this assumption is justified for some of the experimental tasks, it may not hold in many real-world scenarios. The paper could benefit from a more thorough discussion of when this assumption is likely to hold and how violations of it might impact performance.\n\n- While the paper compares DALUPI to some baseline UDA methods (DANN and MDD), it doesn't include comparisons to more recent state-of-the-art UDA techniques. This makes it harder to fully assess the relative advantages of DALUPI in the broader context of current UDA research.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Unsupervised Domain Adaptation by Learning Using Privileged Information (DALUPI), a novel approach to tackle the challenge of transferring knowledge from a labeled source domain to an unlabeled target domain. Traditional unsupervised domain adaptation (UDA) methods often rely on strong assumptions about domain overlap that are frequently violated in high-dimensional problems like image classification. DALUPI addresses this limitation by leveraging privileged information (PI)---additional data available only during training---to relax restrictions on input variables and improve sample efficiency. The authors provide theoretical guarantees for consistent learning without requiring distributional overlap between input domains, which is typically needed for traditional UDA methods.\n\nTo demonstrate the practical utility of DALUPI, the paper proposes learning algorithms for image classification using three types of privileged information: binary attribute vectors, single regions of interest, and multiple regions of interest. The authors evaluate their approach on four tasks, including celebrity photo classification, digit recognition, entity detection in natural images, and X-ray classification. Across these experiments, DALUPI demonstrates improved performance compared to baseline methods, especially when input domain overlap is violated and training sets are small. The results show that DALUPI can successfully adapt in scenarios where traditional UDA methods fail and exhibits increased sample efficiency compared to non-privileged learners.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The authors could design experiments to test how sensitive DALUPI is to violations of the sufficiency assumption. This could involve artificially degrading the quality of the privileged information to see how it affects performance. Such experiments would help understand the robustness of DALUPI in scenarios where the privileged information is not perfectly sufficient for predicting labels.\n\n- Another thing to do would be to test DALUPI on more challenging cross-domain adaptation tasks, such as adapting between drastically different image styles (e.g., photos to sketches, or Yelp reviews to tweets). This would showcase the method's ability to handle more extreme domain shifts.'\n\n- There is a whole line of work in NLP that calls this learning with feature feedback that you haven't discussed in related work. Several recent papers have shown how learning with feature feedback can help models have better performance vis-a-vis out of domain. Please include those in related work. It might also be beneficial t9o show your evaluate on those NLP tasks. [1, 2]\n\n[1] Kaushik, D., Setlur, A., Hovy, E. H., & Lipton, Z. C. Explaining the Efficacy of Counterfactually Augmented Data. In International Conference on Learning Representations. 2021.\n[2] Katakkar, A., Yoo, C. H., Wang, W., Lipton, Z. C., & Kaushik, D. Practical Benefits of Feature Feedback Under Distribution Shift. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (pp. 346-355). 2022.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cmKDhuzdck", "reviewer": "Reviewer_PbCi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Solid theoretical arguments supporting the proposed approach\n- Broad set of experiments supporting the proposal.\n- Well-organized and well-written paper.\n\nWeaknesses:\n- The PI needs to be sufficient for achieving the end task, which is quite a strong assumption. The proposed two-stage algorithm is formulated that way, the second stage receiving only the W (privileged information), not the data X itself. Maybe the X itself can be included in the W, but that’s not discussed.\n- The proposal is quite limited and narrow in terms of applicability, it makes sense for providing bounding boxes, but there are not many other obvious settings allowing it. The binary attributed for the celebrity photos seems more like a toy problem, and it is not clear at all that the binary attributes are sufficient for classification. The bounding box PI is relevant, but appears quite niche.\n- Moreover, the fact that the PI should be available from the target domain puts an extra burden for achieving some results, and obviously provides an advantage over the task achieved. For instance, providing bounding boxes over the objects of interest requires a level of intervention that is close to completing the classification task itself (even though it is argued otherwise in the text).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents the notion of unsupervised domain adaptation with privileged information (PI). This means to achieve adaptation of machine learning models trained for a given task over similar but not identical domains. We can think of this as having measurements taken by some medical imaging devices, which are analyzed by different healthcare experts for a given of identifying a structure in the image, the possible structures being the same over all cases. The task stays the same, but the measurements and expert interpretation may vary. Over this domain adaptation, the paper is proposing an approach to make use of privileged information, that is, side information elements of the data that can be used to improve training of the models. The paper is developing the idea, with some theoretical developments, a technical proposal, and experiments demonstrating the proper working of the approach.\n\nThe main contribution of the paper is to develop the use of PI from the target domain, not just the source domain, as it has been proposed previously (i.e., LUPI). The developments and experiments support the proposal that using PI for target domain is helpful in the context of domain adaptation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The limitations on the type and availability of PI for target domains should be further discussed. In particular, when the PI is not sufficient for the task, but can be helpful to improve performance as additional information the input data (X), to what extent can this be helpful?\n\nAlso, a point that is not clear to me is how to deal with missing PI from a target domain. Maybe I missed it from the paper, but it seems that in the case of having only part of the target domain data with PI, the rest not having PI, we can still proceed and use the first stage of the model to estimate the pseudo-PI of the sample. Is this correct? I think that this can be better developed, as this is an important aspect. For instance, in Fig. 4, when we show results for n_PI(T) varying between 0 and 1, I guess that the results are reported over all the dataset T from target domain, estimating PI from samples missing it.\n\nIn general, there are few details on the specific algorithms used in the main part of the paper, some more details are provided in the appendix, but even there, there are still some dots to connect by the reader. I would suggest to the authors to provide a complete, more explicit explanations on how they implemented their model in practice for the experiments conducted, both in the main paper and the appendix (for the extra details).", "broader_impact_concerns": "The contribution is rather technical and doesn’t raise any specific ethical issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ETSoGaaNSg", "reviewer": "Reviewer_hqvf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n- The overall proposed idea of exploiting additional privileged information for DA seems rather natural, reasonable and inspiring.\n- The overall clarity is mostly ok (especially the main message of the paper and its explanations).\n- The paper provides theoretical guarantees of the provided approach's performance. This is a positive contrast aspect distinguishing the work from many modern domain adaptation papers which are purely heuristical.\n- Illustrative evaluation on 3 problems with different types of privileges information is conducted (binary attributes, single region, multiple regions). The method, in general, outperforms the baselines or performs comparably.\n\n**Weaknesses**\n- While the theoretical result (risk bound, proposition 2) looks interesting, it is not very well explained. The authors should provide a deeper discussion of what each component means and how it affects the bound. I especially talk about the density ratio $p(w)=T(w)/S(w)$ which seems to be the most unclear component here. At the same time, it is one of the most important as it explains how far $T$ is from $S$. The phrase “we may use density estimation” confuses.\n- Some assumptions of the paper which are needed to derive the theoretical results look too unrealistic. For example, I think that “Suppose that $W$ and $Y$ are deterministic in $X$ and $W$, respectively” is a very unrealistic assumption. Assumption 3 is also not very practical. Given that, the derived theoretical results are not truly inspiring as they are mostly straightforward combinations of existing statistical learning theory results and decompositions.\n- I would say that the paper is not sufficiently mathematically rigorous. For example, in Proposition 1 there are sums over x,w,y while it should contain integrals over the respective data distributions (or expectations). This is somewhat strange (taking into account that sometimes the authors use the notation with expectations themself). In equation (2) there are the same loss functions $L$ for learning privileged information and prediction labels which seems to be not correct (in general, they are expected to be different, right?).\n- Synthetic experiment in section 4.1 is a little bit hard to parse. It would be nice to have a table or smth like it briefly summarizing how and where the covariate shift happens in the considered case (it is explained in the text but I do not completely understand).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the unsupervised domain adaptation (DA) problem with the privileged information. This means that the learner has access to data triplets $(x,w,y)\\sim S$ from the source data distributions and has to build a predictor $h(x)$ that will be applied to some different but related target distribution $x\\sim T$ (actually, $(x,w,y)\\sim T$ but $w,y$ are not observed during testing). The predictor is expected to perform well on $T$.\n\nThe authors of the paper claim that standard theoretically-justified approaches to DA which use only $(x,y)$ usually require non-realistic assumptions, e.g., on $S, T$, which do not generally hold in practice. The main message of the paper is that the usage of privileged information $w$ (which is, of course, not always available), can help to alleviate this issue. The authors provide a two-stage algorithm (train prediction for priviledged information $w=f(x)$ and predictor for the label $y=g(w)$ on the train data from $S$; use $h(x)=g(f(x))$ to predict on $T$) to perform DA using the privileged information and analyze their approach from the empirical risk perspective. From the practical point of view, the authors do several computational experiments (outperform some DA baselines in several tasks:  celeba, digits, entity and X-ray classification) and also demonstrate a particular (though, synthetic) experiment which evidently demonstrates the case when using the privileged information yields superior performance compared to standard approach which does not use it.\n\nSince I am not a deep expert in the field of unsupervised domain adaptation, it is possible that in my review I missed some important aspect as I am not very familiar with deep aspects of existing research/baselines in the subfield.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Since I am not a deep expert in the field of domain adaptation, I can not for sure say that the baselines considered in the paper are state-of-the-art and setups are truly ok. Therefore, my review is mostly about logical and presentational aspects of the paper. \n\nI would like the authors to Improve the presentation/explanation of their theoretical results as well as improve the mathematical rigor of their claims and proofs, see my comments in the \"weaknesses\" section above.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper meets the bar both in terms of Claims and Evidence and in terms of Audience. I would encourage the authors to address Reviewer hqvf's comments on notation and mathematical rigor to the extent that they can for the camera-ready version.
Decision: Accept (poster)