A Unified View of Double-Weighting for Marginal Distribution Shift
Authors: José I. Segovia-Martín
Santiago Mazuelas
Anqi Liu
Venue: Accepted by TMLR ()

Abstract: Supervised classification traditionally assumes that training and testing samples are drawn from the same underlying distribution. However, practical scenarios are often affected by distribution shifts, such as covariate and label shifts. Most existing techniques for correcting distribution shifts are based on a reweighted approach that weights training samples, assigning lower relevance to the samples that are unlikely at testing. However, these methods may achieve poor performance when the weights obtained take large values at certain training samples. In addition, in multi-source cases, existing methods do not exploit complementary information among sources, and equally combine sources for all instances. In this paper, we establish a unified learning framework for distribution shift adaptation. We present a double-weighting approach to deal with distribution shifts, considering weight functions associated with both training and testing samples. For the multi-source case, the presented methods assign source-dependent weights for training and testing samples, where weights are obtained jointly using information from all sources. We also present generalization bounds for the proposed methods that show a significant increase in the effective sample size compared with existing approaches. Empirically, the proposed methods achieve enhanced classification performance in both synthetic and empirical experiments.

Published in Transactions on Machine Learning Research (02/2025)
A Unified View of Double-Weighting for Marginal Distribution Shift
José I. Segovia-Martin
jsegonia@bcamath.oyy
Basque Center for Applied Mathematics (BCAM)
Bilbao, Spain
Santiago Mazuclas
smacuelas@@baamathhoor
Basque Center for Applied Mathematics (BCAM) IKERBASQUE- Basque Foundation for Science Bilbao, Spain
Anqi Liu
aliu jlu. edu
CS department, Whiting School of Engineering Johns Hopkins University Baltimore, Maryland, USA
Reviewed on OpenReview: https: // openeevee..e....e.. net/ forum? id= aPyJ5l
Abstract
Supervised classification traditionally assumes that training and testing samples are drawn from the same underlying distribution. However, practical scenarios are often affected by distribution shifts, such covariate and label shifts. Most existing techniques for correct- ing distribution shifts are based on a reweighted approach that weights training samples, assigning lower relevance to the samples that are unlikely at testing. However, these meth- ods may achieve poor performance when the weights obtained take large values at certain training samples. In addition, in multi-source cases, existing methods do not exploit com- plementary information among sources, and equally combine sources for all instances. In this paper, we establish a unified learning framework for distribution shift adaptation. We present weible-weighting approach to deal with distribution shifts, considering weight functions associated with both training and testing samples. For the multi-source case, the presented methods assign source-dependent weights for training and testing samples, where weights are obtained jointly using information from all sources. We also present general- ization bounds for the proposed met hods that show a significant increase the effective sample size compared with existing approaches. Empirically, the proposed methods achieve enhanced classification performance in both synthetic and empirical experiments.
1 Introduction
Supervised classification traditionally assumes that training and testing samples are independently and identically distributed (i.id.) drawn from the same underlying distribution. However, practical scenarios are often affected by distribution shifts, such as covariate shift and label shift. In covariate shift, the marginal distribution over the instances (covariates æ) differs while the label conditional distribution remains the same. In label shift, the marginal distribution over the labels (classes y) differs while the instance conditional distribution remains the same. Additionally, in multi-source scenarios, the training data is obtained from multiple sources, each of which has different probability distributions.
Distribution shifts are common in many practical applications, including electronic health record data ana- lysis Singh al_ (022). For example, model may be trained learn pationt's discase severity using historical patients' data, but there may exist shifts between training and testing populations due to the chal- lenges in obtaining data. from patients within the same population (Humbert- et al. 2022) Moreover,

---

Published in Transactions on Machine Learning Research (02/2025)
may also need to learn from multiple datasets collected from different hospitals due to health record fragmentation.
Multiple techniques have been developed for correcting different types of distribution shifts
(Quinonero-Candeen et al,, 2008; Sugivama & Kawanabe, 2012 Zhang et all 2013 Lipton et al. 2018
Sun al. 2011: Schweikert. et al. 2008 Zhang all (0155). Most existing distribution shift correction techniques are based on a reweighted approach. Reweighted techniques weight training samples assigning lower relevance to the training samples that unlikely testing. However, these methods require certain assumptions about the supports of the distributions and may achieve poor performance when the weights obtained take large values at certain training samples (Cortes & Mohri, 2014: Martino et al., 2018) (see Fig. D. Additionally, existing methods for multi-source distribution shift adaptation inherit the problems of single-source reweighting methods. Most methods define classification rules as a linear combination of the classifiers learned independently on each source (Zhang et al., 2015 Shni et al., 2021: Wang al. 2023) (see Fig. Thoorctical work (Mansour al., 2008) has shown that it is more effective have sample-dependent coelficients to allow different combinations of classification rules for each instance. Further details the prior work are provided in Appendix
Weights for training samples
Weight for testing samples
Main conventional single-source approaches
Proposed single-source approach
Figure 1: Different approaches for single-source distribution shift (the training and testing samples follow Gaussian distributions with probability mass concentrated in the blue and black circles, resp.). Existing methods oblain weights that take large values al certain training samples and do not consider weights at testing (i.e., weights at testing are constant). The proposed approach reduces the large values training weights by utilizing weights at testing.
Recently, the weigle-weighting approach has been proposed to correct for single-source covariate shift SSagovia-Martin et al., 2028), addressing the limitations of covariate shift reweighted methods by assigning weights for both training and testing instances. The Nouble-weighting approach does not require assumptions for supports and can avoid large values by adjusting the weights given the relation of weights (see Fig. [ However, it remains unelear how assigning weights to both training and testing data points can help more general marginal distribution shift, like label shift and multi-source distribution shift adaptation. In the latter case, weight estimation and weight employment in both training and testing require the consideration of multiple training sources.
This paper establishes a unified learning framework for distribution shift adaptation using a dougle- approach, considering weight functions that depend on the covariates and the labels. The usage of the double- weighting approach enables us to overcome the limitations of existing reweighted methods. Our framework be reduced to dealing with covariate shift, label shift, and multi-source cases. In particular, in the
2

---

Published in Transactions on Machine Learning Research (02/2025)
Weights for training samples of the source
Weight for testing samples learned from source
Weights for training samples of the source
Weight for testing samples learned from source 2
Main conventional multi-source approaches
Proposed multi-source approach
Figure Different approaches for multi-source distribution shift (the two training sources and testing samples follow Gaussian distributions with probability mass concentrated in the blue, orange and black circles, resp.). Existing methods obtain weights that take large values at certain training samples and classify each testing instance using linear combinations of feature mappings. The proposed approach reduces the large values of training weights by also utilizing weights at testing and considers classification rules that involve sample-dependent combinations of feature mappings.
covariate shift and label shift cases, approach alleviates the problem of extreme weights by also assigning weights for testing samples (see Fig. D In the multi-source settings, our proposed method leverages the rich complementary information among sources (see Fig. 2 inducing classification rules that involve sample- dependent weighted combinations of feature mappings.
The main contributions in the paper are as follows.
1. We establish unified double-weighting learning framework to deal with general distribution shifts. Our framework can be reduced to dealing with covariate shift, label shift, and multi-source cases. In general, the weightng framework alleviates problems of reweighted techniques, avoiding extreme weights considering weights associated with the training and testing samples.
2. In the multi-source distribution shift cases, the proposed woighe-wegghting method assigns source- dependent weights for training and testing samples to better utilize the information from multiple sources jointly. addition, our multi-source methodology obtains classification rules that involve sample- dependent weighted combinations of feature mappings.
3. We develop generalization bounds for the proposed methods. Our analysis shows that double-weighting techniques significantly increase the effective sample size for different types of distribution shifts adapta- tion compared with reweighted approaches.
4. We demonstrate that the proposed techniques achieve significant performance improvement in multiple distribution shift scenarios with experiments both synthetic and real-world datasets. Our results show that in cases of significant shifts between training and testing distributions, most existing techniques result in a negative transfer of information among sources, while the proposed approach achieves improved performance.
Notation. Calligrapicc upper case letters represent sets; bold lower and upper case letters represent vectors and matrices, respectively; for vector v, n denotes its transpose, 2) denotes its i-th component, |v] denotes its component-wise absolute value; 1 denotes vector with all components equal to 1; ||,1h, Illlo
3

---

Published in Transactions on Machine Learning Research (02/2025)
and || 11 denote the 1-norm, the infinity, and the Hilbert space norm of its argument, respectively; and z represent vector component-wise inequalities; (x;; m. Σ) denotes the pdf of Gaussian r.v. with mean m and covariance matrix E; and Ep(.} denotes the expectation of its argument w.r.t distribution
2 Preliminaries
This section describes the problem setup and the framework for minimax risk classifiers (MRCs), a supervised classification learning framework that allows us to correct distribution shifts using the doublo-weighting methodology.
2.1 Problem setup
Let X' be the set instances and y be the set of labels represented by the set (1,2....,,)9) [2]]. We denote by A(X's x )) the set of probability distributions over X' and V, and by T(X,)) the set all classification rules from instances X' to labcls y. For h E T(X, we denote by h(y)æ) the probability of assigning label y Y to instance æ E X. With slight abuse of nolation, we denote by h(xx) the label assignment provided by the classification rule h for the instance z. We use the notation Pte for the underlying distribution at test, (I1, 31), (22, (Zn, Un) for the training samples, and In+1, 2n+ Zn+t for the testing instances. For the multi-source case, we use the notation S for the number of training sources, [S] = {1, .,,,,....., S} for the (Fs Us,n,) for the training samples that belong to the source
set of sources, and (Is,1, Us,1),,,))))))))) 1), (1.,,,
s E [S].
The f-risk of classification rule is its expected classification loss with respect to the true underlying distribution at test Pae; i.e.,
Re Ep. (e, (x,
(1)
Supervised classification techniques use the training samples to find classification rule h that has small f-risk Re) In this paper, we consider 0-1-loss and log-loss:
fa (h, (a,, v)) P(h(x)) + y} =1 10=1-hhppp h(ylæ)
(2)
&gg (h, (zz v)) log h(ylzr)
(3)
although the results presented can be analogously used with general losses, as is done in Mazuelas et al. 2022).
In the following, we assumee that, for single-source distribution shift, samples from Per and & testing instances from Pte are available at learning. For the multi-source distribution shift, we assume that samples from Ps for 8 E [$] and t testing instances from Pte are available at learning.
Single-source marginal distribution shift (covariate and label shift). The training samples follow a distribution Pie (2, y) such that the marginal distributions of instances (resp. labels) differ, i.e., Per (2x # (t) (t)), but the label conditional (resp. instance conditional) coincide, i.c., Ptr (y(a:) Pto (ylzr) (resp. (xy (=lg))
(resp. (v) #
Multi-source marginal distribution shift (multi-source covariate and label shift). The training samples from each of the S training sources follow distribution (z v) such that the marginal distributions of instances (resp. labels) differ, i.c., (z( # (z) if # s' E [S], (z) * Pre(z) (resp. pz(v) # (y) if * & E [S), (y) + (y)( for E [$]; but the label (resp. instance conditional) conditional coincide, iee, (x(2)) = Pte (y)z)) (resp. (=(l)))= = (ae(x)y) for s S [S].
2.2 Minimax risk classifiers
Similarly to other approaches based on robust risk minimization (RRM) also known as distributionally ro- bust learning Parnia & Tse. 2016: Fathony et all 2016), MRC methods Mazuulas et all 2202 2023) do not require that the training and testing samples follow the same distribution. Notice that these methods are referred to as minimax risk techniques because they minimize the worst-case expected loss over distri- butions in an uncertainty set, but such approach is different classical minimax analysis of classification methods (Tyybakov, 2009). The uncertainty sets U are formed by probability distributions that match
4

---

Published in Transactions on Machine Learning Research (02/2025)
data-based constraints for the expectations of function 4 : x Rm referred to feature mapping [Mazuelas et al. 2022: 2023). These feature mappings are defined using one-hot encodings of the elements of y as ((x, v) = By & (z), where By is the y-th element of the canonical basis of IR 31, & denotes the Kronecker product, and V X + R is map that represents instances as real vectors of dimension d. Given the uncertainty set U, classification rule h" an 4-MRC for 2
hM arg min max (hh p)
(4)
LET(X.)) pel
where ((h, p) denotes the expected loss of classification rule w.r.t. distribution p. and we denote by R(U) the minimax risk against u. The uncertainty set is defined as
= (p ((Xx))) [E, -| 3 A and p(æ) Pre (2), Va X}
(5)
where À is the confidence vector that assesses the inaccuracies expectations estimates, and T denotes the mean vector expectation estimates.
The mean vector in b is an estimate of the expectation of the feature mapping Ep ((r,v) with respect to the underlying distribution. In cases without distribution shift, the n training samples (x1, 31), (22, .2...... (In, Vn) are drawn from the underlying distribution at test Pte that the expectation Ep.. ((z, y) can be estimated using averages of training samples
((z;, Mi)
(6)
As shown the following, the expectation estimate modified in situations affected by distribution shifts account for the difference between the training and the testing distribution.
2.3 Novelty with respect to weightig methods for single-source covariate shift
The results in the paper provide unified approach for the adaptation to general types of distribution shifts, including label shift and multi-source shifts. the other hand, the methods in (Segovia-Martin et all 2023) address only the case of single-source covariate shift. The methods presented in the following address broader class of distributional shifts that not only include covariate shift but also general shifts, label shifts, and multi-source shifts. The paper presents new algorithms for label shift, multi-source covariate shift, and multi-source label shift. In addition, the experimenial results in Section 6 below demonstrale that the proposed methods obtain better performance than existing approaches, particularly in scenarios involving distribution support mismatch, aligning with the thcoretical performance guarantces and generalization bounds presented in this paper.
Scenarios with multiple sources impose unique challenges and require different algorithms and classification rules. The analysis of how the weights for different sources interact with each other and how they contribute to the sample complexity is completely different from the single-source case addressed in (Segovia-Marlin el. al. 2023). Appendix [B further discusses the contribution of the techniques presented with respect to those in (Segovia-Martin et all, 2023).. In particular, the appendix shows how the new methods presented for multi-source adaptation can leverage the complementary information from different sources.
3 Unified Youble-Weighting Framework for Single-Source Distribution Shifts
This section describes the unified learning framework for double-weighting for single-source distribution shift adaptation. We also present generalization bounds for the proposed framework comparison with the reweighted framework.
3.1
Double-weighting pairwise distributions
The proposed framework considers weights B(x, y) for the training distribution Per and weights axx, y) for the testing distribution Pte (see Fig, []] For any function f, we exploit the fact that
E (F. v) Bp (llss,,).r (xF
(7)
5

---

Published in Transactions on Machine Learning Research (02/2025)
can be achieved if we have
Pte y) v) (tr( (1,,))
(8)
that can be attained by multiple choices of the weights a(x, y) and ((x, y). For the reweighted approaches, the choice of weights is
alr,v) 11, 3(x, v) Pre y)
(9)
(r, v)
if (zr v) > 0 + (te v) > 0 (Sugivama & Kawanabe, 2012). However, the equality in r can be obtained in multiple ways, for instance the usage of
a y) min (1, Ptr Ptl(2,3) (ro
0) B(x, y) min
Pte (te y) Per (I,, C)
(10)
allows us to satisfy 7 for any C > O. Notice also that weights as those in (10) above can be utilized for general train and test distributions without requiring assumptions for their supports.
As shown in the following subsection, the weight functions ((x, y) assign relevance to the training samples, while the weight functions a(x, y) determine the confidence in the predictions for the testing instances. Weights as in (10) can alleviate the limitations of reweighted approaches in distribution shift adaptation. By applying [8) if the ratio (te ( y) is large, using a small ax y) enables having (re v)a y) = (r, (x, y) with moderate values of ((x, Using weights as in (10, we predict with high confidence (a(x, y) s 1) by taking large C. Given that ((x,8) y) $ C, we can consider small values of ((x, y) by sacrificing prediction confidence (decreasing C).
The usage of weights a(x, v) and B(x, y) tailored to general single-source distribution shift scenarios offers significant improvement over methods based the reweighted approach. The proposed approach handle general single-source distribution shifts. In addition, this approach results in enhanced generalization in comparison with the reweighted approach, as shown in Theorem 3.2
3.2 MRC learning using general double- weighttn
This subsection describes the proposed learning methodology for the unified view of double-weighting using weights y) and ((x,y).
Uncertainty sets. To address the single-source distribution shift, we construct an uncertainty set U defined in terms of constraints of the expectation of a weighted feature (2 y). Then, the uncertainty set U can be defined as
u = p x ))) (E, ( - -r| 3 X and p(x) = Pta( Ve EX}
(11)
where the feature mapping (x, y) is defined as (1, v) = a(x, y) (2r y). The expectation of such feature mapping can be estimated using sample averages of training samples weighted by B(x,y) as
3 (1 vi). for $ v) = ((x,))((1,)))
(12)
n
Using weights a(x, y) and B(x, y) that satisfy S we can achieve that the estimate above is an unbiased estimator of (p..(z.g)) (a Pa(z.g)- In addition, the variance of such estimator can be reduced by using weights a(x y) that avoid large weights v) The constraints in the proposed uncortainty set 4 in [11 characterize weighted feature expectation matching on (a y) in the training domain with weights 3(x,
Convex optimization. MRCs corresponding with the uncertainty set can be learned by solving the convex optimization problem
min -T X ue| + (w) (M,
(13)
where &e is function defined in terms of the loss. For 0-1-loss, we have
E Pa Pa v) T μ
Poi (H, a) = 1 max
(14)
CCV
C
6

---

Published in Transactions on Machine Learning Research (02/2025)
and, for log-loss, we have
Plog (,, = log
Σ yEy
(oa
(15)
Theorem 3.1. Let T, A R such that the uncertainty set U in (IT) is not the empty set. If 4 is a solution of (33 for 0-1-loss, the classification rule
Eyee (a(x,)))
(y(y))) (Ia(
max ecy
(16)
|Cl
is a 0-1-MRC for U. If 4 is a solution of
13 for log-loss, the classifiention rule
(I) (y(x)) Evey exp (4 (I,4') u²}
(17)
is a log-MRC for U. In addition, the minimax risk I(U) is given by
R(U) TTp² + x A0|pe Pp (4" x).
(18)
Proof. See Appendix
0
Remarks. The convex optimization problem in 133 can be addressed using conventional techniques such stochastic (sub)gradient. method. Specifically, the optimization problem in (33 is an unconstrained convex optimization problem for which stochastic subgradients can be readily obtained using the testing instances In+1, 2n,2, Tn-t- Note that such a subgradient can be efficiently computed even in cases with a sizable number of classes using the greedy approach shown in (Fathony et al., 2016).
implements L1-type regularization, with the reg- ularization parameter represented by the vector X. This regularization term in 13 enables to penalize differently each component of the parameter μt, ensuring that feature components with poorly estimated expectations (i.c., components i with large ( are strongly penalized.
Rogularization. The convex optimization problem
Classification rule. The form of the classification rules allows the adjustment. of the confidence of the predictions based on the weight function axx, y). For very small values of a(x, y) for all y E y and a specific testing instance x, the classifier h" uniformly assigns labels in the y for both losses, i.c., h" hy(aa)) 1/ | for all y E y.
3.3 Generalization bounds
This subsection describes the generalization bounds of the proposed single-source methods. Such bounds are given in terms of the smallest minimax risk, R that corresponds with the uncortainty set given by the exact expectations, and is defined by
R min (x. (x (x) + (Ep.. ()) P
(19)
The MRC corresponding to that smallest minimax risk R could only be obtained by an exact estimation of the expectations of the feature mapping &, that in turn would require an infinite amount of training samples. The theorem below provides risk bounds for the proposed MRCs in terms of smallest minimax risks R showing how the proposed methods can lead to significant decrease in the estimation error compared to existing methods.
Theorem 3.2. Let be a non-empty uncertainty set given by II and h² be an M-MRC for U. If weights alx, v) and 3(x, y) are given by 110 with C =B// B/VD for 2 1, f² and p are solutions and 19 respectively, and
Ptc y)
B sup vEX (Ir
(20)

---

Published in Transactions on Machine Learning Research (02/2025)
Then, with probability at least 1 - 8 have that
R(h")) - R (1 | 1101) + -1 - -1"1lilTT - -Ep.. (x, y)
/2B2
2m
S R + x (|1 1-
1 1111)) + -1'll1
log
(21)
Dn
where M a constant satisfying (*(,,))) M for all x E X, y yeV.
Proof. See Appendix
Note that the difference between the risk R(h*) and the smallest minimax risk R decreases with the estimation error ||T- p., &. (1, Note that the term -"=0~"11 I| 1-
can be bounded by the constant term
2|| | because the optimization problem (T3) carries out an L1 penalization while that in (9] does not
have the L1 penalization term and does not depend on the training samples.
Corollary 3.3. If weights a(x, y), B(x, v) are given by 10 with C = B/VDD for D 2 1, and B as in 20 the estimation error is bounded as
2 (r, (1 r (ly w S MV
2m
2B
2m
log
S M MM
log
(22)
n
Dn
with probability 1 1-6 8, where M is constant satisfying that (0(x, S M for all X, y y
Proof. The proof is straightforward using Hoeffding's inequality. Using reweighted approach, we have that
sup Pie v)
[1 1ol
B :=
(23)
xEX.NEY Per (tr(2.
If (I,, v)/Ptr (z, y) take large values, using weights
h as in reweighted leads to large values, while if we consider weights as in (IU). we can have B(x, v) C < (te(x, ()/Ptr(r, (x, This is achieved at the cost of
using values of a(x,v) v) = Cp w)/Pto 4) < 1. Using the weights in (10) we have that
Pie (v.
sup [1 1b VD DEX (2.
(24)
This way, the methods proposed can achieve an effective sample size D times larger using the weible- given by IOI with C = B/VD. This is achioved at the cost of using classification rules with confidence in a subregion of X x V, since the region where axx, y) is significantly large shrinks when C' decreases (i.c., when D increases). Considering the ratio
minrex. a(x, v)
(25)
max a(z
we have that using weights as in (10) the ratio smaller than one, while if we consider weights given by reweighted in D that ratio is one since all testing samples have the same confidence. Ratios in (25) signific- antly smaller than one correspond to situations where some points are predicted with very low confidence. This reduction in prediction confidence for points (r, y) with (u( < (2, reflects the scarcity of training samples in such regions of X' x y.
3.4 Doight-we for different distribution shifts
This subsection describes how to apply the unified double-weighting framework presented in Section X to different types of distribution shifts. Specifically, this section we describe the learning methodology using weible-weighting under single-source covariate and label shift.
8

---

Published in Transactions on Machine Learning Research (02/2025)
3.4.1 Single-source covariate shift
In cases where Pte (v(aa) (y)l)) the reference weights in (TO) simplify to
a(x) min (1, Por(()) pte
( ((x) min Pte(a) (t)(x)
(
(26)
for any C > 0, that the weights depend only on the covariates
The double-weighting (DW) approach has been recently proposed to deal with extreme weight values in single-source covariate shift. adaptation by utilizing weights for both training and testing samples Segovia-Marte et al.. 2023
3.4.2 Single-source label shift In cases where Pte (zly) (ur(r))), the weights in (10) simplify to
aly) min (1, Ptr (z)
Ptr(() c). B(y) = min PPo
Pte Pte Pir(3)
C)
(27)
for any C 0, so that the weights depend only on the labels y.
Weights in (27 alleviate the limitations of existing reweighted label shift approaches. Considering both weights (v) and aty) we can assign low relevance to the training samples with labels that are unlikely at testing, and also assign low confidence prediction to the labels that are underrepresented in the training data.
Learning MRCs. We can derive the learning framework for label shift adaptation using the learning methodology presented in Section 3.2 by considering that the weight functions a(y) and 3(y) only depend the labels, simplifying the general distribution shift framework to address label shift scenarios.
The classification rules associated to the MRCs corresponding with the uncertainty set U in with weight functions ayy and B(y) are given by
Evec (a()') o' 8' u²)) - )
h²((²z)= aly))(z,0))*
max ccy
(28)
C
for 0-1-loss and
expla(y)0(z, u)
- exp (a(y')0(z,
hat hy(r))) Eyey
(29)
for log-loss, where μ² is a solution of considering weigh functions that only depend on the labels.
Predictive per label confidence. The values of a(y) adjust the confidence with which each label is classified. Low values of a(y) reflect lower confidence in the classification of certain labels, meaning a poor learning of those labels during training due to their underrepresentation in the training data. The components of the parameter μ associated to the labels with low value a(y) will be less reliable, since the feature components are poorly estimated. Low values of aly) imply that the classifier would not assign label y rogardless of the instance we want classify.
In this section, described the unifidd learning framework for weighting for single-source distribu- tion shift adaptation. In the next section, will extend the unified learning framework for multi-source distribution shift adaptation.
4 Unified Double- Weiell-neiggging Framework for Multi-Source Distribution Shifts
This section describes the unified learning framework of double-weighting for multi-source distribution shift adaptation. We also present generalization bounds for the proposed framework comparison with the reweighted approach.
9

---

Published in Transactions on Machine Learning Research (02/2025)
4.1 Double-weighting pairwise distributions
The proposed framework extends that presented in Section B.I for the case where we have multiple training sources. We match multiple training distributions P1: ......, Ps with the testing distribution Pte (sce Fig. 2), exploiting the fact that, for any function S, we have that.
y) = &a (x,
(30)
for [S], can be attained by multiple choices of the sets of weights {a,(z, y) sets) and {,, sesss In this paper, we propose the usage of weights tailored to general scenarios of multi-source distribution shift as follows
Po(x, v) - Per (2,
Pa 12 -e((
c): (r,
Pee Pae y)
C)
a,(x, = min
Be(r, - min
(31)
E Po (x, y)
for any C> O. Such weights satisfy B0 and also address the limitations of reweighted approaches because if the ratio (t. Ex= Pa y) take large values, using small (x,z)) can enable having (0)) with moderate values of (z))). In particular, for single-source distribution shift, the double seiihte-eighting approach presented in (10) can alleviate the limitations of reweighted approaches by sacrificing confidence in samples such that (r, > Cp.(1, v). For the multi-source case, the same improvements can be achieved by only sacrificing confidence in the points such that pte(z, y) > C (a, that is much smaller set because Σ (s(x, v) > (=( for all s' E [S]- Using the same C as in Section B.1 we alleviate the existing trade-off between expectation estimates and confidence in the classification rules because we do not need to have significant confidence in each source E [S]. Instead, we need that Σ a, (x, y) 1, that is achieved since the weights in (TT take into account the rest of the sources. This allows us to consider smaller sets of
weights (=)())))=(s)) without losing confidence in the classifiers. Specifically, weights in (31) can achieve higher confidence predictions in comparison with weights as in [0] for the testing instances such that ps (I,
take large values for multiple E [S].
The usage of weights (a, v) and {B,( y) tailored to multi-source scenarios offers a significant improvement over methods based on the reweighted approach and the double weighting for single-source distribution shift. The proposed approach can handle general multi-source distribution shifts. In addition, this approach results in enhanced generalization, as shown in Theorem 4.2
4.2 MRC learning using general multi-source bouble-weighting
This subsection describes the proposed learning methodology for the unified view of woighte-eighttng using sets of weights {a,(x, y) and (2, y) To simplify the exposition, we assume that all training sources have an equal number of training samples. In Appendix []] we extend the paper's results for cases with a general number of training samples in each source.
Uncertainty sets To address the multi-source distribution shift, we construct an uncertainty set U defined in terms of the intersection of S sets of constraints. Each set of constraints bounds the expectation of the weighted fcature La, (x.y) v) for each training source & E [S]. Then, the uncertainty set U can be defined as
u = [ EE Pa. (4, v) T=l 1 X, for 1,2,.... S and p(x) Vt Vo EX}
(32)
where the expectation of feature mapping P.. (z. v) = az v) v), for E [S], is estimated using sample averages of training samples from source s weighted by (a(x,)) y) as
T,
(Fa,, for (x. = (x
(33)
The proposed uncertainty set 21, defined in 32 is the intersection of S sets of constraints, each of which char- actorizing weighted feature expectation matching on t (x. y) in each source domain with weights a, y). This set contained in each uncertainty set U, defined using [II for each source [S] and is generally significantly smaller.
10

---

Published in Transactions on Machine Learning Research (02/2025)
Convex optimization. MRCs corresponding with the uncertainty set 32 can be learned by solving the convex optimization problem
min (p. (
2 pgl + RE (o) ((1))=11) v, (=))==))
(34)
where is function defined in terms of the loss. For 0-1-loss, we have
Σ. Eyec @ (x. v)T p4
Poi (f1,)s=11 {a.)==)) = 1 max
(35)
c
|C|
and, for log-loss, we have
Plog (HH (aolo=))) log Σ yEy
exp
D
(z,8))
(36)
Theorem 4.1. Let Ts E Rm, A, R th for s E [S] be such that the uncertainty set U in
32 not the
emply sel. If (1ggs= is a solution of 34 for 0- o-l-soss, the classification rule
Σ. Eyecc Da, (2, 1') 1M
(y(r)) = Da (x, () M.
max ecy
(37)
CC
is a 0-1-MRC for U. If { is solutions of
34 for log-loss, the classification rule
(y(z) exp Pa, (z, 1) [+
(38)
Eger exp E Da, (x, T
is a log-MRC for U. In addition, the minimax risk R(U) is given by
8
S x pil E pa())sp (11g) (as)sm))
Σ
T Σ
(39)
Proof. See Appendix
The theorem above is generalization of Theorem 3. for the multi-source case. The classification rules in 37 for 0-1-loss, and BN for log-loss involve sample specific weighted combinations of the feature mappings given by (a, y) [>ss This way, we allow different contributions from each source to classify each testing instance, as detailed in the following subsections.
Source-dependent weights for training and testing. We obtain the benefits of the double-weighting approach in the previous section by considering both weights Ba y) and (z for each pair of training and testing distribution, p=( and V)) We can assign low relevance to training samples that are unlikely at testing and also assign low confidence to the testing instances that are unlikely at training, In addition, since the uncertainty set U in (2) is contained in each uncertainty set Us defined using (11 for each source s e [S] we have that
min max ((h, < min min max ((h, p)
(40)
h pell
sE[S] h pEll
and R(U)= < min R(U.). Therefore, the minimax risk of the proposed multi-source method is smaller than the minimax risk the single-source method. This fact also differentiates the proposed approach from
"summing over the sources," " which involves obtaining an MRC associated with the uncortainty set U, for
each source E [S] and then combining the classifiers. Directly applying Section 3.1 and Section 3.2 each source will not lead to the minimax risk R((().
11

---

Published in Transactions on Machine Learning Research (02/2025)
Sample-specific weights for classification rules. Considering the set of weights {az(z, se(s) the prediction of each testing instance is obtained by combining the relevant feature mappings, leading to classi- fication rules that involve sample-specific combinations of feature mappings given by {a,(z, betsT Since the set (a, sets) depends on x and y, we allow different contributions from each source to classify each lesting instance. This way, the proposed learning methodology is capable of learning complex classification rules to classify each instance with high confidence. Even if some (,(2, y) are small for a particular source and pair (x, y), we may be able to still achieve high confidence using the information from other sources. In particular, we can leverage other (2, y) to classify that instance as long as we have Σ. as y) =
4.3 Generalization bounds
This subsection describes the generalization bounds of the proposed methods. Such bounds are given in terms of the smallest minimax risk, R that corresponds with the uncertainty set given by the exact expectations, and is defined by
R min (Pe (1. J
Σ D (I. ( us + (a ((H,)S-:: x., fasls=1). s=1
(41)
The MRC corresponding to that smallest minimax risk 1e could only be obtained by an exact estimation of the expectations of the feature mapping a for s E [S], that in turn would require an infinite amount of training samples from each source s E [S]. The theorem below provides risk bounds for the proposed MRCs in terms of smallest minimax risks R showing how the proposed methods can lead significant decrease in the estimation error compared to existing methods.
Theorem 4.2. Let a non-emply uncertainty set given by 132 and ha be an L-MRC for U. If weights {as (z, sets and (3, , y) seiss are given by [1 with C C for D and
Pro(x,3)
sup S Ei=1 (x(2, E.X
(42)
Then, with probabilily al. least 1 8 have that
R(h*) < R Σ A (1139) [p(1) + 11 -11ll2
max #E(S)
T,
Epn. Pa, bl
2B2
2m
< R
x (11601 1M.) + M - 11 1/a
log
(43)
Dn
8
1=1
where ~ = -Ip 5 14" of training samples available in each domain s E [S], and M is a constant satisfying [1 v) I M for all x E X, y y.
U&} are solutions to 39 and respectively, n. is the number
Proof. See Appendix
322 the difference between the risk R(h") and the smallest min- imax risk R decreases as the maximum of the estimation errors over the training sources s E [S], max ||T, -r.--a Ep. a. (z, o> docrcases. The set of weights (a,(2,)) y and ( allow us to reduce the estimation error in ||T,- En, &,, (x, yl as detailed in the following section. We also detail how weights for multi-source in (31) further reduce the maximum value of the weights [13,(z, g)ll for each source E [S] in comparison with weights for single-source in (T)
Note that, similarly to Theorem
4.4 Leveraging complementary information from multiple sources
In this subsoction, we explain how the usage of complementary information among different sources leads [22
improved effective sample size. The bound in Theorem 42 depends on the estimate of Ep.. Pa, (z. y) and, shown in Corollary 3% the error in the estimate depends the values of the weights (,,
12

---

Published in Transactions on Machine Learning Research (02/2025)
In Section 33 we show that the proposed single-souree double-weighring can achieve effective sample size D times larger than reweighted approach considering weights given in (TO). This was achieved at the cost. of using classification rules with significant confidence in a subregion of X' that shrinks when D increases.
In multi-source cases we can improve the trade-off presented in Section 8.3 between the error in the estimation of Ep. 4a, (x, y) and the confidence of the classification rules. In particular, using weights as in with C B/VD and B = suPeex,,ey Pte(z. v)/ Ei=1 Pi(x, v), we have that
B VD VD NEX Ei ((
1
pto(z,
(1,
sup
(44)
In the following, will compare the reference solution for multi-source doihte-weighting in (31) and the reference solution for single-source deuble-w in (Q). If we consider weights 8, v) in (1) with [z, 2 B./VD and B, defined in (20). we have that for any source 8
()
Pee B,
B = sup xEX, Ei (i(
sup TEX (x(
(45)
and often B < B, as long as the supports of the training sources significantly overlap (p.(æ,)) > 0 for multiple E [S]). When we decrease [1, v)llo, i.c., when we consider larger D, the region where the classifier has significant confidence also decreases. Similarly as in Section 33 the ratio
minseX, Ex (z(x,))
(46)
max Es a (x,
is C(/BB) using weights for multi-source in BI,, while using weights for single-source in (TO) such ratio is C(LLS) 1/B,), that smaller than C(I/B) since S B, for any [S]. This means that using weights in 3I the region of X' x y where the confidence of the classifier is significantly large is bigger than if we use weights in [IO] Using the set of weights {a,(2,,)] in (3I where each (z, y) depend on all the sources E [S], we can improve the trade-off between estimation error and confidence of the classifiers.
Using weights as in [11). we do not need significant confidence o2 (x. y) Rs 1 for each source s E [S] but rather that there is enough confidence among all sources, i.e., that Σ (x. 1.
4.5 Doughe-weighting for different multi-source distribution shifts
This subsection describes how to apply the unified multi-source double-weighting framework presented in Section 4 different types of distribution shifts. Specifically, we describe the learning methodology using double-weighting under multi-source covariate and label shift.
4.5.1
Multi-source covariate shift
In a scenario with multi-source covariate shift we have that (te()) = (u(x)) for E [S]. Hence, the reference weights in 31 simplify to
(x( Pa(l) c) Bx(x) min (te Σ (i(x) pr(x)) CC)
(x())) = min
(47)
E 1= pa(x)' (te(x))
for any C' > 0, so that the weights depend only on the covariates r.
The usage of weights {a.(x)} and {z)()) tailored to multi-source covariate shift scenarios consti- tutes a substantial improvement compared methods based on reweighted approach. Tt also constitutes a substantial improvement with respect the couble-weighting of a single-source covariate shift, since using weights in (7) we alleviate the existing trade-off since do not need to have significant confidence in each source E [S]. Instead, we need that DS a. (2 y) 1, that is achieved since the weights in (+ take into account the rest of the sources. This allows us consider smaller sets of weights (x))<{[s) without losing confidence in the classifiers.
13

---

Published in Transactions on Machine Learning Research (02/2025)
We can derive the learning for multi-source covariate shift adaptation using the learning methodology in Section 4.2 by considering that the sets of weight functions (ax(x)) and (f)(z)) only depend on the covariates. This simplifies the general distribution shift framework to address multi-source covariate shift scenarios.
The classification rules associated to the MRCs corresponding with the uncertainty set in functions (x)(t)) and (B.(2))eess) are given by
32 with weight
E
)
max (y(()) "z
(48)
CC)
|C|
for 0-1-MRC and
exp E s=1 (g(x)
h" (y1=)
(49)
Evey exp E (1) 5/
for log-loss, where (x()) is a solution of considering sets of weight functions that only depend on the covariates.
Predictive per source and per instance confidence. The classification rules learned for multi-source covariate shift are given in terms of the set of weights {a, {=){)))){[s)- The weights as(x) adjust the confidence with which testing instance x is classified using the contribution from the source Low values of (2( imply that if we rely only on the information from source s, the classifier would uniformly assign labels in the set of labels y. By considering weights (x(x) for each source, we can classify instance x with high confidence even when some of the (x) take small values, as long as Σs (2(2) (z)
4.5.2 Multi-source label shift
In scenario with multi-source label shift we have that Pre(xly) Ps (zly) for E [S]. Hence, the reference weights in 31 simplify to
Pa(U) c.) (v( min (v)()) (v) Pa (y) c)
(v = min (v((v), (y)
(50)
for any C > O, where 5,(1) = (2)) Eem Pv()), so that the weights depend only on the labels y.
The usage of weights {v) and {3,( tailored to multi-source label shift scenarios constitutes a substantial improvement compared to methods based on the reweighted approach. As for multi-source covariate shift, also constitutes substantial improvement with respect to the weighting of single- source label shift., since using weights in (50) alleviate the existing trade-off between estimation errors and confidence of the classification rules, detailed in Section 4.4
As for multi-source covariate shift, we can simplify the general distribution shift framework in Section to address multi-source label shift scenarios by considering that the sets of weight functions {ax(v))ee[s) and (8.(v) only depend on the labels.
The classification rules associated to the MRCs corresponding with the uncertainty set 4 in 32 with weight functions (v)))))e and (3((s))e[s)) given by
5
S ΣS E ee as (1)) v') "T)TH
a (v)
max CC)
(51)
Cl
for 0-1-MRC and
exp s=1 as ( 3) #
(52)
Eyey exp E (x
for loss, where [4])ee(s] is solution of 34 considering sets of weight functions that only depend on the labels.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Predictive per source and per label confidence. The classification rules learned for multi-souree label shift are given in terms of the set of weights {as, (E)(s)) The weights (,(v) adjust the confidence with which label is classified using the contribution from the source Low values of a, (y) imply that if we rely only on the information from source 8, the classifier would never assign label y regardless of the instance we want to classify. By considering weights (s(y)) for each source, we can learn how to classify with label y with high confidence even when some of the (1) take small values, as long as > (z( R 1.
Addressing other types of shifts. The unified framework proposed above can be used to adapt to multiple types of distribution shifts, such as conditional shifts or even a general distribution shift. Even thought the double weighting framework is general enough to deal with any shift, the estimation of weights for conditional shift usually requires specific methods to estimate the conditional relations between instances and labels, that, in practice, would require multiple assumptions regarding the distributions. This additional difficulty is common in the literature. For instance, the methods in (Schweikert et al., 2008; Sun et al., 2011; Chattopadlvay et. al. 2012) assume that labeled examples of the test distribution are known, while those in Zhang et al. 2013 2015) assume a specific relationship between the conditional distributions.
In this soction, we described the unified learning framework for doublo-weighting for multi-source distribution shift adaptation. In the next section, we will present the algorithms and the implementation of the proposed double-weighting for specific distributions shifts.
5 Practical Algorithm and Implementation
This section describes the implementation of the proposed techniques for double-weighting label shift (DW-LS), double-weighting multi-source (MS) covariate shift (DW-MSCS), and double- weightin MS la- bel shift (DW-MSLS) detailed in Algorithm Algorithm and Algorithm g respectively.
5.1 Practical algorithm for double-weighting label shift
A Algorithm M shows the implementation of the proposed techniques for DW-LS that further described the following. In this section, we assumee that n training samples (21,91), (22,82),... y2)...., (Zns Fn) and testing instances 2n+11 2n+2, ... Inte are available learning.
Algorithm The proposed algorithm for label shift adaptation: DW-LS
Input: Training samples (1), (x2, (22, ..., (In, Mn)
Testing instances Æn-11 Zn+2; 2n,t
Hyperparameters and
Output: Weights 3 and & Classifier parameter μ²,, and Minimax risk R(U)
Pie (te(y)/u + solution of (53
B, à f- reference solution
3: T +- "E= ( Vi)
4: n² + solution of (55
R(U) t optimum value of 55
Computing weights. We obtain weights a and B considering the reference solution 27 are obtained using the ratio w(y) == methods (Lipton et al. 2018 Azizzadenesheli et al. 2019 Garg et al., 2020 In partieular the method
The weights
for y E y, that can be estimated using multiple
15

---

Published in Transactions on Machine Learning Research (02/2025)
in Zhang et al. 2013 estimates such ratio by solving the optimization problem
mill
1
K2 (x
UxIY 1
Σ
ey
wy (vt)
1 i=1
i=1
H
b.b. l. 0 S Ey:
S B, for =1,2,...,
ey w
(53)
n
where Kr X + H is feature map corresponding with reproducing kernel Hilbert space (RKHS) H. with kernel (2 I) = (x), (z))) Ky y - Hy is feature map corresponding with a RKHS Hy with kernel (y, ū) (v), Kg (T)),,, and Uxpy is estimation of the operator that maps Hy into Hz. representing the conditional embedding of p(zly)- The optimization problem (8) minimizes the discrepancy between two empirical means of distributions Ptr and Psc. considering embeddings of conditional distributions, studied in (Fukumizu et all, 2004 Song et al. 2009: (010)). The minimization problem in (5% can be written as a standard quadratic problem that can be solved by applying standard techniques. In particular, the parameters related 2013).
53
are determined as proposed Song el al., 2009
Zhang al.
Using the estimated ratios (t)) (v), the weights and B computed using
27 as
B(y) min (v) max (U)
Pra(t()) V'EY (rr( a(y) min (vr(w) (t)
Doo(y') V'EN Per(y")
max VD
(54)
VD
for E V
Learning MRCs. After computing the sets of weights, we solve the optimization problem in 13 by approximating the expectation term in (33 using the lesting instances In 1, 2n+2. En4t and n training samples (11, 92). .... (Zn, vn), given (7),
Σ se (M,, In+ss 0r) +
B(ye)
T p min - T
pll
(1,, xy, ((;))
(55)
n +
that is an unconstrained convex optimization problem and can be addressed in practice using conventional optimization methods such as stochastic subgradient methods.
We determine hyperparameters D and À following the same approach as in Sogovia-Martin et al. 2023
detailed in Appendix D
Complexity and implementation without testing instances. The computational complexity of the proposed methods for label shift adaptation is similar to that of existing methods. In particular, the step of obtaining the parameters of the classifiers solving the convex oplimization problem in (55) has similar complexity as that for conventional methods. The main difference is that existing reweighted methods estimate the expectation term in (B) utilizing the training samples available at learning. our approach, if testing instances are available at learning, we utilize both training and testing instances to estimate the expectation term in 13] making use of the equality in
5.2 Practical algorithm for double- multi-source covariate shift
This section describes the implementation of the proposed DW-MSCS, detailed in Algorithm 2 In this section, we assume that n, training samples (x,,1, Vs.1), .... (Is.n, Vo.n.) from cach source s [S] and iesting instances I,41, 2242 are available at. learning.
Computing weights. We present an extension of the kernel mean matching (KMM), multi-source KMM
(MS-KMM), to determine weights {8, RR Or E RC Jejs)] for multiple sources by solving the optimization
16

---

Published in Transactions on Machine Learning Research (02/2025)
Algorithm The proposed algorithm for multi-source covariate shift adaplation: DW-MSCS
Input: Training samples (F1,,1, Va,1), (Fe.)). Vs,n, Ne,n,)/sE(8 [ss
Testing instances In+1: 1 22............ Intt Intt
Hyperparameters {{+}a=1 and D
Output: Sets of weights {f=) and (di))EEss
Classifier parameter {14)==1, Minimax risk R(4))
(8s)ae(s], ((s)se(s]] solution of (50)
2: Ts < E 3.
Vs,1) for 8 [S) [ & solution of (577
R(U) t optimum value of 55
problem
8
5 (2
Σ as
min
K
(In+i)
(9,
a=1
1=1
==1
8.t.
0 0 (1)
< B./vD D, for E [ng], [|S
0 S 0
(4)
1, for [1] [S]
R2
860
(1) cc for E [S]
ng
S
1 ) for i [t]
a
1
(56)
where K2 : X - Hr is a feature map corresponding with a RKHS Hr with kernel (x, 7) = (Ka
The optimization problem (56) minimizes the sum the discrepancy between two em- pirical means of distributions Ps and Pte, for s E [S], subject to multiple constraints. The hyperparameter D balances the confidence of the classification rules. The minimization problem in G66 can be written standard quadratic problem, detailed in Appendix EL that can be solved by applying standard tech- niques. The appendix also shows significant decrease in estimation error, similar to that of Theorem for multi-source covariate shift.
Learning MRCs. After computing the sets of weights, we learn the parameters of the classifier in for 0-1-loss (resp. 49 for log-loss) solving BEl. We use the set of mean vectors {Ta) defined in R3
and the of confidence vector {>e]= We solve the optimization problem in 34 by approximating the expectation using the & testing instances Int1, Fn+2: ... ,n, as
5 T.M X5 pl
min 15
Σ se ((1,)
Fati, {ax))
(57)
s=1
that an unconstrained convex optimization problem and can be addressed practice using conventional op- timization methods such stochastic subgradient methods. We determine both hyperparameters {Aa) and D following the same approach in (Segovia-Martin et al. 2023), detailed in Appendix Fl
Complexity. The computational complexity of the step for MS-KMM that obtains weights is similar to existing methods Sun el al. (2011),, that compute KMM S times. The main difference in the proposed methods is that 56) has tS additional variables and t(S + 1) additional constraints corresponding to the weights {a, Solving the optimization in (57) obtaîn the classifier parameters has similar complexity as that for conventional methods, since we consider Sm variables and conventional methods compute times convex optimization problem with variables.
17

---

Published in Transactions on Machine Learning Research (02/2025)
5.3 Practical algorithm for double-weighting multi-source label shift
This section describes the implementation of the proposed DW-MSLS, detailed in Algorithm g section, we assume that training samples (T 1e,1), (Za,2, (2..2.Ve.2):...............).).. Vs.2)...., (Fa,n, Va,n,) from cach source E [S] and & testing instances In+1, Fn+ En+z are available at learning.
In this
Algorithm The proposed algorithm for multi-source label shift adaptation: DW-MSCS
Input: Training samples (e,1,,, (Dsmn Vs,n Desess]
Testing instances Un=1.20442 2n42..... Sn+t
Hyperparameters {Aa}==1 and
Output: Sets weights {$.),e[9] and (diteetss
Classifier parameter [ and Minimax risk R(U))
(to(v)/ss()) f- solution of (5%) for s [S]
B., d, f- reference solution in
3: Ts < Em (v.) (v.,) Vs.1) for 8 E [S]
( & solution of (599
R(U) + optimum value of (59
Computing weights. We obtain the set of weights (astse(s) and {8, considering the reference solution (M)) The weights are obtained estimating the ratios pre(y)/p>()) for each source [S], and the ratios are estimated solving (33 for cach source E [S]. The set of weights {asheels) and [[s)ae(s) are computed as
Pro(3') Σx' (v) VD
Pro(y)
1 max
(y()) min
V'EY Σs Por
(u)
(x(y)
(v( min
max VD
(58)
s/ 1>==px<( (v)
v'Ey
Σx= por
for E y and s [S].
Learning MRCs. After computing the sets of weights, we learn the parameters of the classifier in 55 for 0-1-loss (resp. 52 for log-loss. We solve the optimization problem in [ by approximating the expectation
using the testing instances fn+1,fat24 2nt2. Fntt as
s Σ
s Σ
min (M,)ee(s)
TTM. M.
p
Σ se ((us) Intis (ashsets))
(59)
s=1
that is an unconstrained convex optimization problem and can be addressed in practice using conventional optimization methods such stochastic subgradient methods. We determine hyperparameters D and {A] following the same approach in for Algorithm 2 detailed in Appendix
Complexity and implementation without testing instances. The computational complexity of the proposed methods for multi-source label shift adaptation is similar to existing methods. In particular, the computational complexity of the step for KMM in (53) that obtains weights is similar to existing methods Zhang et al. (2015), that compute all the weights a single optimization problem.
Solving the optimization in 51 to obtain the classifier parameters has similar complexity as that for con- ventional methods, since we consider Sm variables and conventional methods compute S times convex optimization problem with m variables. The main difference is that existing reweighted methods estimate the expectation term in [4] utilizing the training samples available at learning. In our approach, if t testing instances are available at learning, we utilize the testing instances to estimate the expectation term in (d..
The proposed mothodology can be utilized with general types of feature mappings, including those defined using neural networks. For instance, the feature mapping ((x, y) can be given by the output of the penultimate layer g(:,0) of neural network, i.e., ( v) ey g(x, (), where 0 correspond to the weights
Remark.
18

---

Published in Transactions on Machine Learning Research (02/2025)
of the network. By feeding the data through the network, the outputs of the penultimate layer provide high-level representation of the input data, capturing complex patterns and relationships learned during the training of the network. This way, we can deal with shifts in the semantic representation space. We have used pretrained RosNets to map images into feature vectors in the new numerical results, as detailed in the general responses. The results presented Section 122 show how the proposed methodology can accommodate deep learning frameworks, obtaining good performances dealing with multi-source covariate shift.
6 Experiments
This section shows experimental results for the proposed approaches in comparison with the state-of-the-art methods. The source code for the methods and the experimental setup presented are publicly available in https://github.
The results are complemented by those in Appendix F that provide further implementation details.
6.1 Experiments for single-source label shift adaptation
This subsection shows experimental results for the proposed approach for label shift adaptation.
Baseline methods. As bascline, we employ logistic regression (No Adapt.), which ignore label shift. We also consider four label shift adaptation methods: Reweighted where we perform importance weighting using the true marginal ratios; DW where we perform double- weighting using the true marginal ratios to compute the reference solution in (77; target shift (TarS) (Zhang et all 2013) that obtain weights solving a KMM problem; black box shift estimation (BBSE) (Lipton et al. 2018) that computes marginal ratios using an invertible confusion matrix; regularized learning under label shift (RLLS) (Axizaaeenesheli et all, 2019) that, adds a regularization term the weight computation problem; and Maximum Likelihood Label Shift (MLLS) (Garg et al. (020) that assumes access to classifier that outputs the true source distribution conditional probabilities (v(lz).
Experimental details. The methods TarS (Zhang et al. 2013), BBSE (Lipton et all 20018). RLLS [Arizadencsheli et all, 2019), and MLLS (Garg et al. 2020) have been implemented as detailed in their
references. For the proposed method, the corresponding hyperparameters are obtained following the ap- proach described in (Segovia-Martin et al. 2023), as detailed in Appendix
6.1.1 Experiment with synthetic data
In the experiments using synthetic data, we show how the proposed approach can achieve label shift adapt- ation in a scenario where the existing reweighted approaches are challenged. For such results, the training and testing data are drawn from Gaussian distributions
p(aly- = 1) = N x: [0.5, 0f .0.11) - N
N (x:1-0.5, (x:))=((xx00000
"....,)).0001) .0.1))
(60)
and labels are y 1 if (2) 0, y otherwise. We fix Pve 1) = Piely = 2) = 0.50 and we consider training distribution of the form (v = 1) 6, Pvr(y = 2) 1- We use values 8 E {0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5} to simulate different relations between the marginals of training and testing labels. We assume that the true marginal probabilities (v) and Pur(y) are known, and compute weights as B(y) = (ve(w)/Pr(() for the reweighted approach, and as in 27 for the proposed double-weighting approach. In addition, for each type of label shift (value of 5), we carriod out 200 random repetitions with 100 training samples and 100 testing samples.
Results. Figure 8l shows box-plots corresponding to the classification error of existing and proposed ap- proaches. The results in the figure show how methods that do not take into account label shifts obtain poor performances. In addition, the figure show that reweighted methods obtain poor performances in the cases where takes small values (p)(y) is small) because the ratios ((y) (v) become large, as discussed in Scction 3.I The proposed methods can adapt to label shifts in situations where the shift between train- ing and testing distribution challenges reweighted approaches. In particular, the double-weightee approach allows us to consider smaller weights Bly) in those situations by reducing the corresponding weight avy).
19

---

Published in Transactions on Machine Learning Research (02/2025)
0.5
No Adlsp..
Reweighted
Classification error
0.4
nw
0.10
0.20
0.30
0.40
0.50
Parameter
Figure 3: Classification error for the synthetic experiment using different approaches. The boxplot shows the different classification error with different proportions of the labels ratios for the training sets.
6.1.2 Experiments with real datasets
In the second set of experiments, we assess the performance of the proposed methods in comparison with cxisting techniques using real datasets publicly available in the UCI repository Dua & Graff (2017). We generale label shifts in the datasets following Lipton el. al. 2018). In the "tweak-one" shift, the training distribution uniform over the set of possible labels, (v) = 1/13%, while in the testing distribution, we assign probability (v)
to half of the classes (rounded up). We set å = 0.05 for 10 repetitions and 8 0.10 for another 10 repetitions. In the "knock-out" shift, the testing distribution is uniform over the set of possible labels, while in the training distribution, we remove proportion â of the samples from the selected classes. We set 6 = 0.9 and select half of the classes (rounded up) for all 20 repetitions. In the "Dirichlet" shift, the training distribution is uniform over the set of possible labels, while in the vesting distribution follow a Dirichlet distribution with parameter 7 = 0.1. Note that we conducted experiments using Dirichlet shift exclusively on multiclass datasets. For the experiments using "redwinc", we considered only three of the labels (2, 3, 4), discarding those with a small number of samples (0, 1, 5).
Results. Table D shows the averaged classification error across various datasets and scenarios of single- source label shift, along with their respective standard deviations. The second column of Table [ describes the different types of label shifts described above.
Overall, the experimental results show how the proposed methods provide improved adaptation to single- source label shift in binary and multiclass classification sctting in situations where the distribution of training is uniform while that of testing is not and vice versa. The label shift adaptation improvement for the proposed DW methods compared to reweighted methods is also observed when the training and testing distributions are assumed to be known. Note that for "knock-out" shift, the ratios (t))//)e become large since (y) is small for certain labels y E V. In these cases, reweighted methods achieve poor performance compared with the double-weighting approach that can reduce the large values (3y) by reducing the corresponding value (y). The results agree with the theoretical results in Corollary B.3 that show how the proposed methodology results increased effective sample size in comparison with the reweighted approach.
6.2 Experiments for multi-source covariate shift adaptation
This subsection shows experimental results for the proposed approach for multi-source covariate shift adapt- ation in comparison with the state-of-the-art methods.
Baseline methods. Three single-source baselines are employed: logistic regression (LR), which ignore multi-source covariate shift, KMM (Huang et all, 2006 Gretton et all, 2008) and double-woighting covariate shift (DW-GCS) (Segovia- Martin et al., 2023), where we compute a classification rule from each of the training sources and evaluate the classification rule that achieves the smallest classification error on the testing
20

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: Classification errors in scenarios show that the proposed methods can more adequately adapt to single-source label shift. The bold values represent the lowest classification error in each scenario.
No Adapt.
Exact prob.
Eslimated prob.
Datasets
Reweighted
DW
TarS
BBSE
RLLS
MLLS
DW-LS
Adult
tweak-one
.33 + 001
.08 .02
.07 + .02
.07 + .02
.29 + .01
.24 + 24
.07 + .02
.07 + .02
knock-out
.38 +: .02
.77 + .03
.66 + .02
.30 + .04
.42 + .05
.43 = .06
.39 + .05
.28 + .03
Diabetes
tweak-one
.45 + .22
.08 1 .03
.11 +: .03
.09 1 03
.33 1 .05
.37 1 .15
.08 1 .03
.08 1 .03
knock-out
.40 + .01
.32 003
.29 + .02
.33 - .03
.45 + .66
.46 = .05
.43 - .06
.29 + .02
Mammo
tweak-one
.37 + .01
.08 + .02
.08 + .02
.12 + .22
.11 + .02
.18 + 25
.08 + .03
.11 + .01
knock-out
.31 + .01
.11 002
.0 + .01
.21 + 02
.4 +
.4 = .11
.42 # .11
.23 + .09
Usenet2
+ .02 tweak-one
.33 .01
.30
+
1
.20 1 .01
1 .02
knock-out
.38 + .03
.41 + .05
.37 + .02
.39 - 08
.39 + 00
.39 = 00
.45 = .6
.38 + .02
Credit
tweak-one
.48 + .02
.14 + 001
.11 .01
.19 + .01
.32 + .05
.29 + .17
.10 + .01
.16 + .03
knock-out
.7 + .03
23 + .06
21 + .05
.24 + .06
.25 + .08
.27 + .09
.30 + .06
.22 + .05
20news
tweak-one
.64 + .03
.59 +: .66
.4 .4 .05
.58 + 00
.59 + .07
.57 + .0
.70 + .12
.58 + .09
knock-out
.64 +: .02
.63
.61 + .03
.64 + .04
1 02
+ .03
1 .05
1 .03
dirichlet
.66 + .04
.65 + .05
.61 + .22
.65 = 004
.65 + 00
.65 = .05
.72 = .22
.64 + .02
Redwine
tweak-one
.66 + .13
.41 + .22
24
.48 + 22
.55 + 26
.59 + 25
.57 + 25
.52 + .5
knock-out
.60 + 003
.51 + .04
.49 .03
.55 + .07
.58 + .07
.58 + .07
.59 + .08
.55 + .09
dirichlet
+ .07
.56 +
.51
= .12
+
.2
= .12
= .0
=
domain. We also compare with the performance obtained by the 2-stage weighting for MS domain adaptation (2SW-MDA) method that extends the reweighted approach for multi-source covariate shift (Snn et. al. 2011).
and the MS distributionally robust learning (MS-DRL, method that considers the same combination of classifiers learned independently on cach of the sources (Wang et al. 2023).
Experimental details. The methods KMM (Huang et al. 2000), DW-GCS Segovia-Marrtn et al, 2023). 2SW-MDA (Sun et all, 2011),, and MS-DRL (Wang et al,, 2023) have been implemented as detailed in their references. In addition, we implement a practical version of the theoretical work proposed (Mansour ald 2008), called combination weighted KMM (CW KMM), where the initial set of classifiers (he) (he]se[s) are learned using KMM. Then, combine the classifiers to obtain new classification rule of the form h()lz)) = >.= (v(x)), with (x) (xx) px p.(xr) for [S], where the training probabilities (r(r) are estimated using kernel density estimation (KDE) (Sugivama & Kawanabe 2012). For the existing methods, we consider the default hyperparameter values provided by the authors. For the proposed method, the corresponding hyperparamoters are obtained following the approach described in (Scgovia-Martin et al. 2023). as detailed in Appendix D
6.2.1 Experiment with synthetic data
In the experiments using synthetic data, we show how the proposed approach can achieve multi-source covariate shift adaptation in a scenario where the existing approaches are challenged. For such results, the training and testing data are drawn from Gaussian distributions
(= = N (xs (x=1-003,0]] 0.531) p2(x) = N x: [0.5. 0f .0.51) Pie = NN [x: ,0.3,)1)
(61)
and labels are
sign(2 - -sign(z(2))
if (1) > 0.
(62)
if (1) )
21

---

Published in Transactions on Machine Learning Research (02/2025)
Notice that this synthetic data can be easily classified switching between classifiers 2g002 and eodeending on the sign of x()), However the synthetic data quite difficult to classify without knowing the change of sign corresponding to z²1), We carried out 100 random repetitions with 200 samples from each source and 200 testing samples and considered linear feature mapping. In addition, implemented an LR referred to as LR ideal that classifies using hya120 when z 2 0, and using hg000 when xx1) 0, i.e. has access the optimal assignment of the testing instances to their respective classifier.
benchmark, we
0.8
0.7
Classification error
0.6
-000++
0.6
0.1
LR
LR ideal
KAA
DW DW-GGS
29W: MDA
Dit
cw KMA
DW-MSC$
Approaches
Figure 4: Classification error for the synthetic experiment using different approaches. The boxplot shows how considering classification rules that involve iustance-dependee weighted combinations of feature mappings is needed (DW-MSCS). Note that LR ideal is lower bound as it is the ideal case when we know the optimal assignment of testing instances to their respective classifier.
Results. Figure 4 shows box-plots corresponding to the classification error of existing and proposed ap- proaches. Existing approaches obtain initial classifiers learned from each source and then combine them regardless of the testing instance x we want to classify, i.e., consider final classification rules of the form h = h1 + 72h2, with 71:72 2 0, 71 + 72 = The figure shows how those approaches obtaín poor perform- ances since the assignment of the testing instances to their respective classifier is not properly conducted. The proposed methods can adapt to multi-source covariate shift considering classification rules that involve instance-dependent weighted combinations of feature mappings, i.e., the final classification rules are given in terms of combinations of 7/1 (I) and 72(x), with (x), (x) 2 0, (xx)) + 72(2) = 1 for all x E X. This way, the classifiers learned assign labels to each testing instance x using the sources that are more likely to classify that instance.
6.2.2 Experiments with real datasets
In the second set of experiments, assess the performance of the proposed methods in comparison with existing techniques using real datasets. We consider Spam Detection, 20 Newsgroups, and Sentiment, classific- ation datasets. We also consider two common domain adaptation image datasets, Office-31 and NomainNet. For the experiments using "Spam detection" dataset, we consider a binary classification problem, where we select the labeled email set as the testing set, and the three annotated email sets as the training sources similar to done in (Duan et al., 2012). For the experiments using "20 Newsgroups" dataset, we consider 6 binary problems and multiclass problems. Each of the training sources is generated by selecting the samples from two subcategories selected randomly for each binary task, and the testing set by selecting the samples from all the subcategories. For the experiments using "Sentiment" dataset, problems. Each of the training sources is generated by selecting the samples from cach of the domains, and the lesting set by selecting the samples from all domains. For the experiments using "DomainNet" dataset, we considered multiclass problems. Each of the training sources generated by selecting the samples from each of the domains, removing one domain (the one corresponding to the experiment's name), and the testing set is obtained by combining the samples from all the domains. For the experiments using "0ffico-31" dataset, we consider 4 multiclass problems, selecting labels depending of the type of the object. Each of the
consider 5 binary
22

---

Published in Transactions on Machine Learning Research (02/2025)
two training sources generated by selecting the samples from domains 'amazon" and 'webcam', and the testing set by combining the samples from all the three domains.
Table 2: Classification errors in 31 scenarios show that the proposed methods can more adequately adapt to multi-source covariate shift. The bold values represent the lowest classification crror in each scenario.
Datasets
LR
KMM
DW-GCS
2SW-MDA
MS-DRL
CW KMM
DW-MSCS
Spam detection
500 features
.55 + .07
.45 .07
.39 .04
.3 .07
.50 .05
.47 .05
.40 .04
1000 features
45 .07
444.00
.1 .03
.5 .07
.49 .09
.46 = .05
.38 .03
1500 features
.46
40
46
.46
.45
.36 .4
2000 features
.77 .08
.39 400
.6
.19 .07
.46 .05
.35 .04
2500 features
+
.77 .05
.38 +
.7
.19 =
.47
.03
3000 features
43
.4 .07
.38 + .4
45
.00 .08
.47 47 00
334 .03
20 Newsgroups
comp vs rec
.6 .04
.5
.36 .0 .08
4
.45 .05
.111
.32
comp sci
.0
45+
.39 +0
.3
.45 .04
.43 431.11
.34 .05
comp vs talk
36
33 .05
.27 27
44
36 = .05
22 = .03
rec sci
.65
.4 .04
39 .04
46
.47 .04
4
36 04
rec va talk
.05
.3 # .03
.35 1.04
46
.39 = .05
35
299 .04
sei talk
.0
.42
.05
.51
.39 .05
.09
33 .03
comp vs rec vs sci
.05
.57 .06
51 066
.94.04
.57 = .04
54 .05
.45 .05
comp rec vs talk
.05
.47 .05
.3 .05
499
.51 = .05
.45
33 .03
comp vs sci vs talk
50
.51 + .05
44 .44
+25
.77 = .05
.49 + .04
.41 .04
rec vs talk
.3
= .05
.48
.6
= .04
49
Sentiment
All Domains books
.48 + .66
.43 1 .04
.32 3 .01
.4 .02
.49 = .04
.46 - .03
.23 = .01
.47 1
.32 1..100
1 .02
.50 =
.48
.22
dvd
.88 1: .05
.4 .04
.32 10
.48 1.11
.50 = .02
.48 .02
224 .01
electronics
.45 .04
49
.49 .03
.47 .03
25
kitchen
.6 .05
.45 .04
.33 + .03
.48 .02
.49 .02
.47 + .03
.5 .22
DomainNet
clipart
.11 .05
.05
.46 + .03
+ .06
.51 = .03
.33
.03
infograph
..4
.43 .05
453
44
.55 .05
.5 004
.30 = .03
painting
38
.44 .66
402
43
.55 .05
35 .05
.32 = .02
quickdraw
.2
.45 .04
.64
.4
.55 .04
34 004
.31 .08
real
40 .05
.45 .05
.02
.3
.54 .04
35 .0
.31 + .03
sketch
40
.45 + .05
.47 .04
444.00
.55 .04
.04
.03
Office-31
electronics
.12 .04
.19 10.5 .05
.23 .03
.09 + .03
.39 = .03
.14 .03
.11 .02
stalionery
.14 .03
16...5
.18 1 .04
.38 .07
.10 .03
.10 .03
organization
.13 .08
.17 .04
.21 21 .05
.39 = .05
.13 .03
.12 + .03
mixed
.09 + .03
.14 005
.20
.08
.03
.31 .04
.12
.04
.10 .02
Results. Table the averaged classification error across various datascts and scenarios of multi-source
covariate shift, along with their respective slandard deviations. The first column of Table 2 describes the different classification tasks. The experimental results show that the proposed methods provide improved adaptation to multi-source covariate shift in binary and multiclass classification settings. Since the testing sot contains samples from an unseen domain for the training sources, in the experiments the support the testing distribution is larger than the support of the training distribution. Therefore, methods like 2SW-MDA, MS-DRL, and CW KMM, that obtain weights using reweighted techniques, result in poor performance even when compared to LR methods. This is clearly observed in the experiments using the Sentiment dataset. In such dataset, method 2SW-MDA results in negative transfer in all cases except for the "All domains", where all four domains are used as training sources. In the other cases, one domain is removed (the domain corresponding to the experiment's name). In these experiments, it is clear that considering double- weightng and classification rules that involve desendet weighted combinations of feature mappings substantially improves the model performance.
23

---

Published in Transactions on Machine Learning Research (02/2025)
6.3 Experiments for multi- label shift adaptation
This subsection shows experimental results for the proposed approach for multi-source label shift adaptation in comparison with the state-of-Che-arl, approaches.
Table 3: Classification errois in 20 scenarios using "sentiment analysis" dataset show that the proposed methods more adequately adapt to multi-source label shift. The bold values represent the lowest classi- fication error in each scenario.
Datasets
Dirichlet
LR
KMM
DW-LS
LWC KMM
DW-MSLS
books
7 0.01
.43 1 .03
43 4 022
444 .02
.41 .02
.36 1 .06
= 0.1
.2 .02
.3 .0
.41 +1 6.6
.41 .03
.88 + .08
? = 10
.22
.42 .02
.3
.41 .03
.8 + .03
.42
4
+2
.40 .02
.6 002
?
.23
.42
.39 .02
.6 3..202
dvd
? 0.01
.3
.43
433
.43
0.1
43
.3 .02
40 .05
.42 .03
37
7 = 1
42
.331
.44.00
.11 .02
.7
10
42..02
42 .02
.31
.40 .02
26 1 1.8
7 = 100
.43 -0
.43 434.00 .02
30 .02
+ .02
26 + .02
electronics
? 0.01
.3 .22
43
.45
.03
40
0.1 7 = 10
.43 .03
43 .02
.0 4 .05
.40 + .02
.40 4.0
.43 .02
.33 .22
.35 .03
.03
32
.43
4
30
.40 .02
.8
? = 100
.42 03
42 .02
30 .02
.40 = .02
26 .03
kitchen
7 = 0.01
.43 022
.43 18+.00 .02
.45 ::4
.41 .02
.4+
0.1 Y = 1
.33
.42
43 .05
422 = .02
.45
.2
42 02
36 .06
.11 .02
31 006
10
42.002
42 02
31 31 .03
.40 .02
26
100
.42 1 .03
42
1 .02
.30 .03
.40 1 .02
.66 .00
Baseline methods. We employ three singlo-source basclines: LR, which ignore multi-source label shift, KMM (Zhang et all, (013) and DW-LS. For the baseline methods, we compute a classification rule from each of the training sources and evaluate the classification rule that achieves the smallest classification error the testing domain. We also compare with the performance obtained by reweighted method for multi- source label shift label-dependent weighted combination KMM (LWC KMM) (Zhang et al. 2015)), where the initial set of classifiers [ha)ee[s] are learned using KMM. Then, combine the classifiers to obtain new classification rule of the form híyl)z) = - h, (ylæx), with 7 (v) = (v)) E P:(y) for a E [S], where the training probabilities (5()) are estiimatdd as the ratio of samples with label y from source 8, for y E V, [S]. The methods KMM (Zhang et all (001)), and LWC KMM have been implemented as detailed in their references.
Experimental details. We assess the performance of the proposed methods in comparison with existing techniques using "sentiment analysis" datasets. We consider 4 binary problems, where each of the training sources is generated by selecting the samples from each of the domains, and the testing set by selecting the samples from all the domains. We generate label shifts each training source using Dirichlet distributions with concentration parameter 7 for all classes, while the testing distribution is uniform over the of labols. We use values 7 E (0.01, 0.1, 10, 100) to simulate different label shifts in each training source, that smaller " corresponds to bigger shifts, similar to done in (Azizzadenesheli et al.. 2019). For each type of multi-source label shift (value of 7) we carry out 20 random repetitions with 1000 training samples in each source and 150 testing samples.
Results. Table 3 shows the averaged classification error using "sentiment analysis" datasets across different types of multi-source label shifts, along with their respective standard deviations. The first column of the Table 3 describes the training source that we removed. The experimental results show how reweighted approaches struggle to effectively adapt to the label shift. between the different sources, that using
24

---

Published in Transactions on Machine Learning Research (02/2025)
single-souree KMM achieve similar results that for LWC KMM. In contrast, the proposed methods show significantly improved adaptation to multi-source label shift, leveraging the information between the different. training sources.
7 Conclusion
In this paper, presented unified framework to address general distribution shifts, including label and multi-source shifts. The learning framework for distribution shift adaptation is based on double-weighting approach, considering weight functions at training and testing that depend on the covariates and the labels. By assigning weights to both training and lesting samples, our methods alleviate the problem of existing methods when the weights functions at training take large values. In the multi-source settings, the proposed methods leverage the rich complementary information among sources, considering classification rules that involve sample-dependent weighted combinations of feature mappings. In addition, we present generalization bounds and empirical results for the proposed methods that show an improved adaptation to single-source label and multi-source shifts. The proposed unifying view of double proposed can enable techniques capable to adapt to more general range of scenarios affected by distribution shift.
Acknowledgments
Funding in direct support of this work has been provided by projects PID2022-137063NBBI and the BCAM Severo Ochoa accreditation CEX2021-001122 10011001111000 1033 funded by the Spanish Ministry of Science and Innovation, the Amazon Research Award, the Discovery Award of the Johns Hopkins University, and a seed grant from the JHU Institute of Assured Autonomy.
SNS2002-1352003 and CEX2021-001142- funded by
European Union "NoxtGencratooEUU/PPTT
References
Yasemin Altun and Alex Smola. Unifying divergence minimization and statistical inference via convex duality. In Proceedings of the 19th Annual Conference on Compulational Learning Theory, pp. 139 153, 2006.
Kamyar Axizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. arXiv preprint wxXiv1190.09973. 2019.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151-175. 2010.
John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learning bounds for domain adaptation. In Proceedings of the 21st Annual Conference on Neural Information Processing Systems, pp. 129 136, 2007.
Rita Chattopadhyay, Qian Sun, Wei Fan, Ian Davidson, Scthuraman Panchanathan, and Jicping Ye. Multi- source domain adaptation and its application to early detection of fatigue. ACM Transactions on Know- ledge Discovery from Data (TKDD), 6:1-26, 2012.
Xiangli Chen, Mathew Monfort, Anqi Liu, and Brian D. Ziebart. Robust covariate shift regression. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, pp. 1270 1279, 2016.
Corinna Cortes and Mehryar Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519:103 126, 2014.
Corinna Cortes, Mehryar Mohri, Michael Riley, and Afshin Rostamizadeh. Sample selection bias correction theory. In International Conference on Algorithmic Learning Theory, pp. 38-53, 2008.
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems, 47(4):547 555,, 2009.
25

---

Published in Transactions on Machine Learning Research (02/2025)
Dheeru Dua and Casey Graff. UCI Machine Learning Repository, 2017. URL http://arcchie..css ics. uci. edu/ml
Lixin Duan, Dong Xu, and Ivor Wai-Hung Tsang. Domain adaptation from multiple sources: A domain- dependent regularization approach. IEFF Transactions on Neural Networks and Learning Systems, 23: 504 518, 2012.
Miroslav Dudík, Robert E. Schapire, and Steven J. Phillips. Correcting sample selection bias in maximum en- tropy density estimation. In Proceedings of the 19th Annual Conference on Neural Information Processing Systems, pp. 323 330, 2005.
Farzan Farnia and David Tse. A minimax approach to supervised learning. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems, pp. 4240 4248, 2016.
Rizal Fathony, Anqi Liu, Kaiser Asif, and Brian D. Ziebart. Adversarial multiclass classífication: A risk
minimization perspective. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems, pp. 559 567, 2016.
Kenji Fukumizu, Francis R Bach, and Michael I Jordan. Dimensionality reduction for supervised learning with reproducing kernel hilbort spaces. Journal of Machine Learning Research, 5:73-99, 2004.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and ZZacaayy Lipton. A unified view of label shift cstimation. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems, pp. 3290-3300, 2020.
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. In Dalaset shift in machine learning, pp. 131 160. MIT Press, 2008.
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. Advances in Newral Information Processing Systems, 19, 2006.
Marie Humbert-Droz, Pritam Mukherjee, Olivier Gevaert, al. Strategies to address the lack of labeled data for supervised machine learning training with electronic health records: case study for the extraction of symptoms from clinical notes. JMIR Medical Informatics, 10, 2022.
Yi Lin, Yoonkyung Lee, and Grace Wahba. Support vector machines for classification nonslandard situations. Machine Learning, 46(1):191 202, 2002.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Delecting and correcting for label shift with black box predictors. In Proceedings of the 35th International Conference on Machine Learning, pp. 3122-3130, 2018.
Anqi Liu and Brian D. Ziebart. Robust classification under sample selection bias. In Proceedings of the 28th Annual Conference on Newral Information Processing Systems, pp. 37 45, 2014.
Anqi Liu and Brian D. Ziebart. Robust covariate shift prediction with general losses and feature views. arv preprint XrXio:1712. 10043, 2017.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation with multiple sources. In Proceedings of the 22nd Annual Conference Neural Information Processing Systems, pp. 1041-1048, 2008.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Multiple source adaptation and the renyi divergence. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, pp. 367 374, 2009.
Luca Martino, Victor Elvira, J Miguez, Artés-Rodríguez, and PM Djurić. A comparison of clipping strategies for importance sampling. In 2018 IEEE Statistical Signal Processing Workshop (SSP), pp. 558- 562, 2018.
26

---

Published in Transactions on Machine Learning Research (02/2025)
Santiago Mazuclas, Yuan Shen, and Aritz Perez. Generalized maximum entropy for supervised classification. IEEE Transactions on Information. Theory, 68 (8())22500-22))) 250000 2022.
Santiago Mazuelas, Mauricio Romero, and Peter Grünwald. Minimax risk classifiers with 0-1 loss. Journal of Machine Learning Research, 24:1-48, 2023.
Masahiro Nomura and Yuta Saito. Efficient hyperparameter optimization under multi-source covariate shift. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 1376-1385, 2021.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. Proceedings of the IEEE international conference on computer vision, pp. 1406 1415, 2019.
Joaquin Quinonero-Candee in machine learning. MIT Press, 2008
Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift
Gabriele Schweikert, Gunnar Rätsch, Christian Widmer, and Bernhard Schölkopf. An empirical analysis of domain adaptation algorithms for genomic sequence analysis. In Proceedings of the 22nd Annual Confer- ence on Neural Information. Processing Systems, pp. 1433-1440 2008
José I. Marti,, Santiago Mazuclas, and Anqi Liu. wouble- for covariate shift adaptation. In Proceedings of the 40th International Conference on Machine Learning, pp. 30439 30457, 2023.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihoood function. Journal of statistical planning and inference, 90(2):227 244, 2000.
Changjian Shui, Zijian Li, Jiaqi Li, Christian Gagné, Charles X Ling, and Boyu Wang. Aggregating from multiple target-shifted sources. In Proceedings of the 38th International Conference on Machine Learning, pp. 9638-96488 2021.
Harvincet Singh, Vishwali Mhasawade, and Rumi Chunara. Gencralizability challenges of mortality risk prediction models: A retrospective analysis multi-center database. PLOS Digital Health, 1, 2022.
Le Song, Jonathan Huang, Alex Smola, and Kenji Fukumizu. Hilbert space embeddings of conditional distributions with applications to dynamical systems. In Proceedings of the 26th International Conference on Machine Learning, pp. 961-968, 2009.
Le Song, Byron Boots, Sajid M. Siddiqi, Geoffrey Gordon, and Alex Smola. Hilbert space embeddings of hidden markov models. In Proceedings of the 27th International Conference on Machine Learning, pp. 991-998, 2010.
Masashi Sugiyama and Motoaki Kawanabe. Machine learning in non-stationary encironments: Introduction to covariate shift adaptation. MIT press, Cambridge, 2012.
Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8:985 1005, 2007.
Qian Sun, Rita Chattopadhyay, Sethuraman Panchanathan, and Jieping Ye. A two-stage weighting frame- work for multi-source domain adaptation. In Proceedings of the 24th International Conference on Neural Information Processing Systems, pp. 505 513, 2011.
Shi-Liang Sun and Hong-Lei Shi. Bayesian multi-source domain adaptation. In Proceeding of the 2013 International Conference on Machine Learning and Cybernetics, pp. 24 28, 2013.
Shiliang Sun, Honglei Shi, and Yuanbin Wu. A survey of multi-source domain adaptation. Information Fusion, 24:84-99 92, 2015.
Alexandre B. Tsybakov. Introduction to Nongoramelric Estimation. Springer New York, NY, 2009.
27

---

Published in Transactions on Machine Learning Research (02/2025)
Wenting Tu and Shiliang Sun. Cross-domain representation-Leanie learnnng framework with combination of class- separate and domain-merge objectives. In Proceedings of the 1st. International Workshop Cross Domain Knowledge Discovery in Web and Social Network Mining, pp. 18 25, 2012.
Zhenyu Wang, Peter Bühlmann, and Zijian Guo. Distributionally robust machine learning with multi-source data. Xiv preprint arXi::2309. 022021 2023.
Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3964-3973, 2018.
Jun Yang, Rong Yan, and Alexander G. Hauptmann. Cross-domain video concept detection using adaptive svms. In Proceedings of the 15th ACM International Conference on Multimedia, pp. 188 197, 2007.
Yao-Liang Yu and Csaba Szepesvári. Analysis of kernel mean matching under covariate shift. In Proceedings of the 29th International Conference on Machine Learning, pp. 1147 1154, 2012.
Bianca Zadrozny. Learning and evaluating classifiers under sample selection bias. In Proceedings of the 21sl International Conference on Machine Learning, pp. 114, 2004.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In Proceedings of the 30th International Conference on Machine Learning, pp. 819 827, 2013.
Kun Zhang, Mingming Gong, and Bernhard Schölkopf. Multi-source domain adaptation: A causal view. In Proceedings of the 29th AAAI Conference on Artificial Intelligncce, pp. 3150-3157, 2015.
A
Prior Work
In this section, we describe the main existing approaches in the litcrature that deal with singlo-source and multi-source covariate and label shift.
A.1 Covariate shift
Reweighted approach. Most of the Lechniques for covariate shift adaptation are based on the reweighted approach (Sueivama & Kawanabe, 2012 Shimodaira 2000: Zadrozuy, 2004 Cortes et all 2008 Dudík et al 2005: Lin et all (2022). These methods weight loss functions at training by means of the function B(x) (x) Using these weights, reweighted methods can account for the fact that some training instances are unlikely at testing, and assign low relevance to such instances at training. Reweighted methods assume the support of Per contains that of Pte (i.e., (x) 0 + (z) > 0). Even if this condition is satisfied, such methods may achieve poor performances if the ratio (te(z)/Pu<()) takes large values at certain training samples. In these cases, the learning process is dominated by few training samples (Cortes & Mohri, 2014)..
Chen et al 2016) are derived from a distributionally robust learning framework. These methods weight functions at testing using the weight. function (<x) = Par (r)))) Using these weights, such methods obtain classifica- tion rules that produce less confident predictions for testing instances are unlikely at training. Reweighted methods assume the support of Pte contains that of Ptr (i.e., (ar( > 0 + Pto(x) > 0). Even if such condition is satisfied, robust methods may achieve poor performances if the ratio Pu (z)/ take large values certain testing samples, leading to overconfident classification rules.
Robust approach. Robust methods for covariate shift adaptation (Liu & Ziebart, 2014 2017:
A.2 Label shift
The main existing techniques for label shift adaptation are based reweighted approach for covariate shift: extended to label shift. These methods exploit the fact that, for any function f, have that
Ep.n((x y) = Byy (x. v), for B(y) = (v)
(63)
Per
28

---

Published in Transactions on Machine Learning Research (02/2025)
if Pie > 0= (v) > 0 for all y E y.
Using [88 reweighted approaches weight loss functions at training using weight function ((y) defined in (63)
Using weights {(y), reweighted methods assign lower relevance to the training samples whose labels are unlikely to be observed at testing. The following describes how the different approaches estimate the weight function B(y).
Black-box predictor approach. Some existing techniques for label shift adaptation (Lipton et al. 2018)) estimate both Pte and (v) in order to compute weights in (03) Ptr(y) can be easily estimated, and Pte (3) is estimated using a confusion matrix. However, the inverse of the confusion matrix may be arbitrarily close to singular matrix for a reduced number of training samples. The regularized approach (Azizzadenesheli et al.. 2019) alleviates the problems regarding the confusion matrix by introducing a regularization term of the form ||3 - 1|l. The approach proposed in (Garg et al. 2020) applies a calibration heuristic to the classifier. In particular, they present a unified view of both (Lipton al,, (201)) and (Azizzadenesheli et al., 2019), encompassing the methods proposed in Lipton et. al. 20188) for a particular choice of the calibration method.
Conditional KMM approach. Certain label shift techniques (Zhang et al. 2013) compute the weight function Bly) generalizing the conventional KMM method (Hnang et all, 2006 Gretton et all, (2008) for label shift adaptation. In order to obtain the weight function B(y), the method in (Zhang et al., 2013) considers kernel mean embedding of marginal and conditional distributions (Song et al. 2009).
Black-box predictor and conditional KMM correct the label shift performing a reweighted approach using the weights B(y) estimated. These methods inherit the problems of standard reweighted methods for covariate shift adaptation. If the weights B(y) take large values at certain samples with specific labels, the learning process is dominated by samples with those labels.
A.3 Multi-source covariate shift
Two-stage weighting approach. The main existing techniques for multi-source covariate shift adaptation
Sun et all 2011; Nomura & Saitd (202)) extend the existing reweighted approach for covariate shift to the multi-source case. Two-stage weighting methods weight training samples using the function 3,(x) = Pto(x)/p.az) for each source s [S] to learn an initial set of classification rules. Then, they rescale functions (x) by a vector 7 that multiplies all the weights for each source E [S] by 7s- Using the rescaled weights 3. (x), such methods learn a new classification rule over all the training sources. In (Nomura & Saito, 2021), the vector 7 is computed in order to reduce the variance of the estimators, while in (Sun et al., 2011)),, it is computed in order to minimize potential mismatches between conditional probability distributions. Two-stage weighting methods inherit the problems of single-source reweighted method. These problems are related to the assumption that the support of Ptr contains that of Pte (i.c., (tr > 0 = Ptr > 0). Even if this condition is satisfied, reweighted mothods may exhibit poor performance if the ratio pte (x))/Prrrr(( takes large values at certain training samples. In these cases, the learning process is dominated by a few training samples (Cortes et al., 2008 Cortes & Mohri, 2014 Sugiyama & Kawanabe, (012)). In addition, two-stage weighting approaches only share information between sources to rescale all the weights for each source by the constant vector 7
A Approaches based on classification rules combinations. Certain multi-source domain adaptation approaches (Yang et al., 2007: Schweikert et all, 2008: Tu & Sun, 2012 Sun Shi 2013 Xu et al. 2018
Peng et al_ 2019 Wang et al. 2023) are based on combining an initial set of pre-learned classifiers (h. b from each of the sources s E [S] in a post hoc manner. For example, {Xu et all 2011:: Peng et al. 201 g)
obtain the pre-learned classifiers minimizing the discrepancy between first moments of the training and testing distributions. These approaches consider the same combination of information from sources for each testing sample to derive a new classification rule h = Σ Ysh, with Σ ? = 1. However, the pre- learned classifiers associated with each training source do not take inlo account the other training sources. In addition, the theoretical work (Blitzer et al., 2007; Mansour et all 200s 200g) provide a general theory regarding how to use the samples from the different sources in order to minimize the expected loss, showing that a linear combination rule may perform poorly and using instance-dependent combinations of information from sources can result. in improved performance.
29

---

Published in Transactions on Machine Learning Research (02/2025)
A.4 Multi-source label shift
Conditional KMM approach. The techniques presented in (Zhagg el al. 2015) exlend the KMM ap- proach for label shift correction proposed in (Zhang et al., 2013). This approach considers a set of clas- sifiers learned on each source. These classification rulcs are combined considoring the theoretical work of (Mansour et all 200s). However, similar to the reweighted methods, KMM methods also inherit the high variance problems of single-source reweighted methods when the weights obtained take large values at certain training samples. Moreover, these methods only share information among sources to rescale all the weights for each source by a constant that represents the overall relevance of such source.
B Contributions for multi-source distribution shifts
The main contributions of the proposed methodology for multi-source distribution shifts are as follows.
The methods in the paper result in improved classification rules that involve sample-dependent combina- tions of the most relevant feature mappings. Applying directly the weible-weighting approach proposed in (Segovia-Marlnn et. all 2023) would result in obtaining a set of S classification rules (h,(v)))) that we would need combine in order to obtain a classification for the testing instances. However, there is not direct way to combine the initial set of classification rules. On the other hand, the proposed approach obtains a single classification rule defined in terms of combinations of the most relevant feature mappings. This combination is given by the set of weights {a, xe]s and depends on the testing instance we want to classify. In addition, our methodology differs from the state-of-the-ar since most of the existing methods obtain a set of classification rules and combine them in a post-hoc manner.
The proposed uncertainty set U is formed by the sets of constraints, each of which characterizing weighted feature expectation matching on La, (z.y) y) in each source domain with weights a, This set is contained in each of the S uncertainty sets {U, 15 and often significantly smaller. Since U C Uus,
min max ((h, p) S min max ((, p),
(64)
h pell
h pEl4,
and R(U)= < R(UL.) for [S]. This means that the minimax risk of the proposed multi-source method is from directly applying Martiv et al., 2023) to each source, since it will not lead to our minimax risk. The classification rule obtained from our method contains instance-dependent combinations of the different feature mappings {2a, y) where the set of weights {a, y) may be interpreted as the ratio of information learned from each of the sources lo classify the instance x. Besides a smaller minimax risk, the resulting classification rule also has the advantage that we only need one a, (z. 3) from the set {a, (2, y)]= to be sufficiently large in order to have classification rules with high confidence. By allowing the sources to share information between them, we are able to alleviate the trade-off in (Segovia-Marre
smaller than the minimax risk of the single-source method. This also differentiates
et al. (023). where weights a, (25, v), for 8 E [S], E X, need to be sulficiently large in order to have classificat ton rules with high confidence. Then, we can consider more flexible values of 8.(x. y), for s E [S]. x E X, leading to better expectation estimates and better bounds than just applying (Segovia-Marthe et al. 2023 where weights are independent for cach of the sources.
The techniques presented achieve an improved trade-off between the effective sample size and the confidence of the classification rules. Using the set of weights {A, (2., y) w))s we improve the effective sample size, obtaining better generalization bounds in comparison with existing reweighted techniques and single-source double- weighting. Using the set of weights fa,(x, y)]ee(s]. we obtain classification rules that achieve significant confidence in a larger region in comparison with the double-weighting method in (Segovia-Martin et ald, 2023). By choosing carefully the reference weights {a.(2, b)s and (3,(x, y) we can improve the existing trado-off in the double-weighting methodology. In particular, we defined weights a<x,y) y) in (I taking into account that we do not need significant confidence a, (x( v) 1 for each source s [S] but rather that there is enough confidence among all sources Σ ax y) ~ 1. This way, the region where the confidence of the classification rule is large is bigger than using the weights in (Segovia-Martin et all, (023)..
The reference weights (x(x) proposed in [ take into account all the sources, and they are smaller than those in (Segovia-Martin et al. 2023), leading to reduced error due to finite sample sizes.
30

---

Published in Transactions on Machine Learning Research (02/2025)
The generalization bounds presented in Section 13 for multi-source distribution shift are non-trivial and different from the ones in Section [3] for single-source distribution shift. Specifically, our generalization risk under the true probability distribution R(h²) is bounded by the minimax risk R()):
R (b4) S R(U) +
Σ (IT Pp. La, (z, U)1 - Aa)" |K5
(65)
On the right-hand-side, we have R(U) & R(U,) Vs E [S], and, since, with probability -
2 (x) log
2
I -Ta-Bpm Epr. Po, (z, 1) S M
(66)
n
where M is a constant satislying that [}((,,))) S M for all x E X, E v, smaller weights (x))) lead better expectation estimates. Therefore, we obtain much better generalization bounds compared applying (Segovia-Martin et al., (002)) multiple times. Alternatively, if we use (Scgovia-Martin et al, 2023)
to solve S oplimization problems independently and then take the sun. the resulting generalization bounds are strictly worse. Moreover, as shown in the paper (Mansour et al. (008),, considering rules of the form hyylx) E.- h,(y)z) can lead to poor performance.
The methods introduced effectively leverage information from multiple sources. The methodology presen- ted in Section 4 is specific for multi-source distribution shift. This is not straightforward extension of (Segovia-Martín et. al., (2023), since the proposed methods elficiently leverage information from multiple sources by considering a set of weights {a, y)]se[s] and (z.(z, s)]se(s] that depend on all the train- ing sources, allowing us to obtain improved generalization bounds and allowing us adapt more general distribution shift scenarios where state-oftthe-art methods are challenged. Finally, the weights considered take into account the number of examples of each source, and may penalize more those where the expectation is worse estimated due to lower number of training samples available, as show in Section D
C Proofs for Section 3 and Section 4
In the following, prove Theorems I and while the proofs of Theorems and 3.2 are analogous by
setting S = 1. The proofs of Theorems 1.1 and 22
below are done for the case of finite X. The proofs for
infinite X' can be carried out analogously using Fenchel duality instead of Lagrange duality, similarly to as is done in Altun & Smola (2006): Mazuelas et al. (2023).
Proof of Theorem 4.1 Firstly, for each h E T(X,)), we have that
maxpeu p(h, max
1'p-44p (p)
s.t.
Σney p(z, v) = (t), Vz E X
(67)
T, X, 1 T I T As Vs [S]
where 1, p, and Pa, denote the vectors and matrix with rows l(h, (z, x)), p(a, and as (xx respectively, for x X. y E V, and
= {
0 0
(p)
oc otherwise.
Optimization problem 67 has Lagrange dual
s Σ (T. (0,) T
S
min tt
Ms,1 A + Σ (To + 22) us.2
mm1
s
Bp((t))((T) +
Σ
o a. (Ha.1 M2,2) -v) v)
==1
s.t.
Ms.1 2 0, Vs E {S]
(68)
31

---

Published in Transactions on Machine Learning Research (02/2025)
where v is the vector in I | with component corresponding with (z, for x E 2, y y given by v(x),
and I is the conjugate function of ((p) = -I"P (P)) given by
(w() sup w p + 1'p = { pzo
0 if
00 otherwise.
Therefore, the Lagrange dual above becomes
min (es ahets)))) s.b.
-E (T. - 2x) T
20) Ma,1 + Ex= (T, + Xa) T
Ma, + (p(c())((x)
(1 (M ) (1,2) > 0, Vs [
(69)
Σ Pa (2, y) (Mo,1 - m.,2) - v(x))= S -f(h,
(2, (z, Yo ER E
It is easy to see that the solution of such optimization problem {M,,1 Pe.zheess) satisfies that 11(1 (1) (8)
(6,) M,,1s,2 = 0
for E [S] and any such that A( > 0. If there exist a set {is)s- with \( > 0 and ss,1 Hs,2 4 0 for
E [S], there would exist an > 0 such that p > e, Ms. > E. Then Ps,1, Pa.2 v* (x) with
(1,))
pli p(1.) E, plit)= ((a))
Pe.1 for js + is
(6)) (t.)
P..2 6, plie==
((ae) px, for js + is
for s E [S] would satisfy the constrain in
69 and
Σ (T. 2.) T
14,11 + Σ
12,22 + Ep) (x) (Ts + Aa) T
5
5
Σ
Ts (Ps.)) (4,2) +
XJ (0a,1 + (2,2)) + +E (n) (z)
s=l
(p - -Map)) +
-2 A (Ps.1 + +b,22) + E (x)- Σ x
s=1
s
(Pe.1 (3,22 +
xo (Ms.1 + +1,22) + + Bppa(t) (z)
s=1
Σ (T. As) 1 +
Σ
(T, + Aa)² T u + (puc(t)u*()) (x)
contradicting the fact that {pa.11 14 Ha,2hee(s), 2 (x) forms a solution of 69
Then (P + (2,2) X3 2 2l and taking u4 = M Ma.2 the Lagrange dual above is equivalent
to
Es TJ + Es X |/ + (p)()))((())
min (1.he()).
E Pa. (2. p. s < (z,, Vr X,y y
that has the same value as maxpeu ((h,p) p) since the constraints in 67 are affine and I is non-empty. Therefore,
s
S + Σ XT 14, + (p).(t)((x))
min max ((hp)) = min heT(X.y) pel
T p
s=1
#=1
Σ
S (I, (I,)) ur
TH, v(x) 5 ((h Vo e Ex V.
32

---

Published in Transactions on Machine Learning Research (02/2025)
For 0-1-loss we have that
9
8 Σ
Σ Da (2, 3)) ue v(x)= - h(yl())z)) +
Da (z, T p. v(x) + 1( Y (70)
==1
==1
and, since Eyer h(yl = implics that Eyec h(yl2)) S 1 for all C S y, we have
2 uu (z, 8) ub Mz, ((x) + +1) 11, AC /VEeE X Eyec Ex (x, (M
VCC
==1
+ *(<x) 1+
AC 2
|C1
+ *(x)) 2 soil (( a, (as)s=)); Vx E E
(71)
Therefore, for each set {1s hels,, we have that any classification rule satisfying
h(ylz) aa (z, y)t pe p (a=))=1) + Vo E Kyy
(72)
is solution of
E (x) hx)())
min
(o) ((1))) v. {
Σ t (xr u1 "r,y)TT==-- pz)r)) + 1 hhulo), Vo E K.
are minimizing the function (p))())((()) and we have that v(x) 2 Pon 1o {as (=s))=1) for all x E X (70, the best possible v(x) would be at v(z) = Poi (1,,) n, {as)=m) if it satisfies the restrictions the optimization problem, i.e. if the
Specifically, since
we
classification rule h(ylx) satisfy n
To prove that Evey h(ylz)) 1 we will show that if
Es
(,, (x,1))
C' E arg
CEY
|C]
and for ((Hg) v. {as)s=1) is defined as (35), then
Σ Eyec Pa (x, 14--1 -1
(y]x)) Σ
+ (x, v)
T m
50
C
s=1
if y C' and
E Eyee Pa, (x,
(vl=)) Σ
TM + (x, (x,1)² 3
2
C
C
1=1
if y C², that
Σ
h(ylz) = Σ h(ylz)
DEY
NEE'
T Da, (x. (x.1) (e.))Tir 11, E
Eyeor Da, 1/
10'
NEC'
T (x. y) "1;1 - -Ic' E Ey Po, (It, " 2
Σ VEC
+ +1=1 1.
|C|
33

---

Published in Transactions on Machine Learning Research (02/2025)
To prove that if y $ C", then Ee Pa, (I, p2 - (Po1 (1He) (0,) - 1) < 0, we use
E Eyec Da (x,1) -1
Ex E.rerruu L (I, k 1 -1
2
|01
1c'
Ex Eyee La. (2,,))"14
M -1
Σ Lo. (x. y) 1
+
C* +1
|0' +
Then,
C +1
Σ 4 Pa (I, (r,9)² 4²= 1
ΣΣ e=ly'E
La (z, (x
ΣΣ>
Σ
Σ
Laa (I, (2,31)²
1e'
y'EC'EO
Σx=1 Eyeer
Pa, (1, 1 ug 12 -1 = yoi
(c 1. To prove that if y C', then Σ Pa, (T u1, (f001 (fH))s {a,) -1) 20,
C41
0, we use
Σ Ege E Da (2, (2,,8)"T M4 -1
Ex Execr (p)
La, (2, 3/ " m
C2'
1C' 11-1
E Eyecr (x, 1 -1
Es
La,
u >
E.e1 En'ec Pa Pr, (2, 80 11z
1C'
C²1
-1
C7 -1
Then,
5
8
CC' |e'-1 -1
Σ L (x. (x.))² u 2
Σ
Σ
Pan (x;
Σ Σ
Daa De (x,8'
|e'
D'EC'
Y'EC'
Es Eyec
D (x, TT -1 = yoi (11g) Jr { 1.
The case of log-loss is analogous to the case for 0-1-loss above taking into account that
Σ 4 (z, (2,8)² ue - v(x) S log(b(y/z)), V OxE
s
Σ nEY
a (1, u(0) S1, Va Ex
exp
Eexp exp (0, M./ m30) 1)) Vor e
>vxx) > llog
DEY
2 v(x)
Plog (H 191))))) ) Vr E X.
The lemma below is used in the proof of Theorem Lemma C.1. Let U be the uncertainty set given by classification rule. If
32 for E I8m X E Rmm for E [S], and h be
s TH + +E
S
9
R h2, = min
Ep((S))
max yey
1
1
Σ
az (z. 1) (2) T - h(ylz)
Σ
AT, (73)
==1
==1
#=1
g
S
Rlog h), min
Σ s=1
T M4g Epd(()) max yEy
TM. (x. ((, v)= H. -logh(//z))
x
(74)
34

---

Published in Transactions on Machine Learning Research (02/2025)
then, for any p U
fo (h, p) - Roi (4, h) Blog (h, p) S Rlog (11, h)
(75)
776)
Proof of Lemma C.1. The case U = 0 is trivial. For the case where 4 * 0, we will first calculate the Lagrange dual of the optimization problem mingeu Eşg for general function q : X' x y - RR Then we will consider the fact that for any p E U and E T(),)),
min (hh ((h,p) S max E(h, P) pEu
PElr
and
max t0z (h, P) min Ep (h(v/æ) 1} Dell
Bell
max petl
foog (h, P)
min Ep log h(ylz))
peur
for 0-1-loss and log-loss respectively. First, we have that minget Epg is equal to
min q q'p + ('p+++(0)
Σ p(r, 9) (r) for all æ E X
s.l.
yEY
X, I 1 PI Ass Vs V [S]
(77)
where p, q, { Joz denote the vectors and set of matrices with rows p(x, q(a, v) and as u) (x,,))
respectively, for a E X, y E y, and
{
0
fp:0
(D)
0o otherwise.
Optimization problem 77 has Lagrange dual
S Mx.1 Σ
[ + (Ms,1 - Ms.22 + v
E (T. A₂)
(T. + T
Ms.2 + (p)()))(()- fr
max (s Iu (())
(Ma1) ) (M,223mm ł 0
s.t.
where denotes the vector in X with component corresponding with (x, for E X, y y given by vz)), and fr is the conjugate function of f(p) q T p + (p) that becomes
0 if <q
(w)
oo otherwise.
Therefore, the previous Lagrange dual becomes
s Σ (T.
S
max
Ao) u,,1
Σ
+ Xo) u + (p)(())(()))
12 (m))
s.t.
(4 (u 11,22 18 2
o (Ma,1 Ms22) + = for E [S]
which is equivalent to
8
Σ (T, Az) M Σ (Ts + Xx) M
max (P
==1
s=1
+ E (p)
min VEy
g((x,)) Σ (z, mo,2)
s.t.
(
2 0.
18
35

---

Published in Transactions on Machine Learning Research (02/2025)
Taking 14, M4.1 2, for 1,2,..., S, the Lagrange dual problem is equivalent to
8
max (1, J
T Ms + E (x) min Σ
((2, v)
Σ az v) p
M4
yey
that has the same value as its primal minpeu Epg since the constraints defining U are affine and U & 0. Then, we have that
max fo (h, = min Ep (h(Ulz) - 1} pell
PEM
8 Σ
8 +
min
Σ ==1
T + E panla)
max NEy
1+
as (1),,))((z,))) - -b(ulz)
(u.)
8=1
8=1
max boo (h, () =
min pell
Ep log hw/lz)
pEll
min
Σ sm1
TH u. + Ep) max vEy
Σ as (2, 1)x,)) log h(vl(z) A AT11
(p.
Proof of Theorem 4.2 Let 4t be the uncertainty set given
by the exact mean
vectors
To0, = Ep.. Pa, (x, y), for s = 1,2..... S, ie.,
4 = (pE x 2) Lo (2, = Too,s, for E [S], and p(x) = (to Ve E E
(78)
It is clear that we have (2) E 2 then using Lemma C.1 for 0-1-loss and the definition of h(ylz) in 37 we have that
R(h") (
min
Σ
T My + (x) max yEy {1+
1+ Σ asla () n. h(oll))
(u,)
s=1
s=l
<
Too, e4 + E (x)
max yEY
1 +
(z (2 v) us TH. b(ylz)
(79)
E Σ Eyec $ (z. yT 11:
T M + Ex)
max yEV
1 max CCy
(80)
C
8=1
===1 Eyec Da, (I,
Mo 3 Epie(z)
T
1 max (C)
(81)
CC
Egec Eg
$ (2, 11.
<
Ts 14
00
+ E Ep)((e)
1+ 1+mas max
CCY
CC]
==1
g + Σ (T. To0,0) 1 - xJ 1
1=1
$=1
s=1
=R
XI (p4 |I:1) + Σ (Too,e -Ta) u + > (T. Too,o)
#=1
#=1
s=1
<R00
X [1 |
1+:) + Σ T. Toc,s TT 1 m.ll
(82)
where, for inequality
we have used the fact that
Eyec Σ L (x, (x,y)T 2
h" (y|c)) >
a. (xx
max ccy
C
36

---

Published in Transactions on Machine Learning Research (02/2025)
For log-loss, using Lemma C41 and the definition of h(ylz) in
BS
have that
R(h") RRog ( , h1)
min ( 1a
o E (x)
max yEy
(a 1) vx)
log (()l)))))))))))))
s=1
T o 8 + E (x)
max yEy
as (2) w)² T pe log h(yl2))
(83)
o/ 2 + (x) max BEy
log Σ exp Σ $ aa (o, v) paa Hz} {
(84)
yEV
2 + E (x) log
Σ
exp
aa (x, ub
(85)
NEy
To 1
T
(1))) log
E
Σ exp
T (I, y) T 148
8
+
14
+ Σ
(T Too,o) M
m
#=1
a=1
s=1
=Rº
Σ
x (11+0)
- -|x51) + Σ (oo, TTTT Σ to, 11,
8
<R
x (lug) 1x:1 + Σ Ts Toc.s T 11, - -1-1.
(86)
Using Hoeffding's inequality, we have that with probability at least 8
max [S[S]
Σ Be (x, ((,, - Ep,. (2 () 9)
S max dE[S]
|| v2
log
n
B2
2m
max M sE[S]
log
Dn
8
B²
2m
M V
2
log
(87)
Dri
Then, using 87 and Hölder incquality in 86 we have that
B
2m
R(h") < R
(1 (15) + M Mulli V2
2
log
(88)
Dn
8
with probability at least 1 8.
D Main Results of the Section 4 when each domain has different number of training samples
This section presents the main results of the Section 4 for the case with general number of training samples in each source.
Reference Solutions. We consider reference solutions of the form
as(x, y) min (s.(2,,w) v), (2 y) C.) Ba v) = min (0s(2,,) y) Pro v)
Ps (1., () Cs
(89)
(to v)
for any C 0, where
(x = Σ$ Pa (z, y) Tgg
(90)
ng
37

---

Published in Transactions on Machine Learning Research (02/2025)
Generalization bounds. Using the reference weight functions defined in 899 and 190
get the following
generalization of Theorem 4.2
Theorem D.1. Let U be non-emply uncertainty set given by 12 and h2 be t-MRC for U. If weights
(c,(z,))sspp and {3, (z, sets) given by 19 with C. = B./VD for D 2 1 and
Pto y)
B, = V sup
(91)
Es (o
Then, with probability at least.
we have that
S Σ
Pte 3)
2 2m
R(h²) & R
N
(1
1121) + M M
11111
sup sEX.gEy
log
Ed'=1
ne Po (x
8=1
where
1
10
and M is constant satisfying [1 10 S M for all a E X, y y
Proof. The proof is analogous to the proof of Theorem considering (1, and (3,(r,)))
defined in
and 3 we have that the ratios Bs//ns, do not depend on the source 8, so we can avoid that training sources with a smaller number of training samples penalise more in reducing the estimation error.
Using the weights given by 80
E Performance guarantees of MS-KMM
The MS-KMM approach in 56 is an empirical version of the population problem given the exact expectation
as
min (o) m2
Ep.. a. (x) Ep, (x)
s.t.
0 $ 3,(x) s B./VD, for [S]
< (x) S for 8 E [S]
E [x) = 3.,,())&. for s E [S]
Σ (x) -1 (1-2) vD )1-v)) for x X
(92)
Since X0
is feasible solution of 92 the value at the optimum is zero. Then, the solutions of (2))))sss)) {á. {z))ss)))) provide consistent estimators expectations because
U2
E 4x) v) 3 (x)
(93)
is satisfied for E [S] if the kernel hy is characteristic or if (p)()l (xx, v) belongs to Has analogously as shown in (Yu & Szepesvári. 2012).
The following theorem presents bounds for the discrepancy between empirical means withing the feature space for solutions of g2 when we have limited amount of samples from each source and testing distribution.
38

---

Published in Transactions on Machine Learning Research (02/2025)
Theorem E.1. For [S], if (x) and (x) are solutions of
02
with probability at least 1 8 we have
that.
By 1
1
Σ j=1
( (oonnt)Ka(anett (x((n)z)
11
He
2
2 2
1
Pto
$ 1+ V
sup
+
(94)
D
2s'=1 (sr DE
where the constant satisfies (2, S x² for all x E X.
Learning jointly achieves better estimation errors than learning independently. The difference between the empirical means in feature space of solutions of (2 depend on the maximum of the values of the weight functions (2(z) and (z) associated with each source s E [8]. In (Segovia Martín et al., 2023)
(x))lo S B./VD, with B, = supaex Pto(#)/p.(T), resulting in bounds of the order O(v B_/ Dn. In methods that obtain weights based on reweighted techniques (Sun et al., 2011 Wang et al. (x))l B, with B, = supzex Ptn(x)/px(t) resulting in bounds of the order O(vB./v B./n, + 1/11).
+ 1/L
2023)
Therefore, since the B. defined in (91) is smaller than in for DW-GCS (Segovia- Martín et ald 2023). and KMM (Huang et al. 2006: Gretton ct al 2008), the proposed MS-KMM significantly decreases the estima- tion error.
Proof of Theorem E.1 The proof is similar the proof of Theorem 4.1 in Segovia- Martín et al 2023) We consider n, + independent random variables taking values in the Hilbert space H₂ as follows
Ka (xa,))
for = 1, 2, .
fars
(95)
tā((nnt) Ka (Ints) if 0, and 1, ,t.
and we will first bound Ei= fert Eimr fo,ilme. We have that,
1 Ba WVD
for i 1,2,..., 77,
lffasllHe I
(96)
fK
if = 0, and i == 1,2,...,
Taking vs = x²
B² Drs
+ ))
and using the bounded differences inequality, we have that, for 2 voa
P
fart +
fo.t
> 1
= P
fo.
fars
fo.
> 1 E
fo
for
Hx
1=1
1=1
1=1
1=1
H2
1=1
E
E Sa,i + Eim Soli
H.
<
exp
(97)
20.
Finally, using Hölder's inequality and by independence, we have that
E
fars
fo.s
<
E
fart
fo
+
E
1l S Ve.
1=1
Therefore,
39

---

Published in Transactions on Machine Learning Research (02/2025)
2
x2
B
+
U/
Dn,
exp
exp
2t's
2k²
B3
Dr,
so that,
1 14 (rs, ( (ITss))
Σ
as (F Kr
(2 (Frttt)
72
==1
==1
14
2 2 2oog
1
(te
0
D
nEX
sup
Σ =='=l (z)
+
(98)
with probability at least 1 1-d 8.
Quadratic version of MS-KMM.
The convex optimization in 56 is quadratic problem since the
squared norm in He can be written
Σ
Kg (Ints)
1 The
Σ
( Kg (2o,))
Ha
(1) (i) kx (Intis Og
+
1
30) ki (1s,i, Fz.j)
2
(4) g0g hiv ( Fars)
n iJ=1
nst 8=1
hi (n+t1)
k (In+ Intte
hi (Za, De,1)
ki (x Darme)
O2
TT
3
+
t
t
==1
hi (Fntes Inte)
ki ka(aan, to.1)
Ka (a Damn.)
his (Ta.1
kn (144))
2 3T
a+
11.
t
Ke ( Fa41)
ka (K Intt)
Bi/mi
-a1/t
= /T/m1
////..,,////// -25/1 / K,
B///ns /ns -t//t
where K2 is the kernel matrix.
Therefore, the optimization problem 50 is equivalent to the quadratic optimization problem
B1/n1
aa/t
min (B,0),)
CH/ni,-aT//e B5/ns, 23/1 - K.
///nns //
s.t.
0
a, 1 1, for [8]
1//nn,-a - 1/'1/1 S,, for s S [S]
(99)
S
Σ
0 - < 1 VD
for i E [t]
==1
40

---

Published in Transactions on Machine Learning Research (02/2025)
F Implementation details and additional experimental details
This appendix details the datasets and settings used for the experiments in Section G
Hyperparameters. In principle, cross-validation can be used to determine both hyperparameters {As)se[s] (A for single-source) and D. However, important to note that standard cross-validateo is not applicable We hence avoid cross-validation and determine both parameters as done in (Segovia-Martinn et al. 2023 We select the value of D to achieve the lowest
when dealing with covariate shift (Sugivama al. 2007).
minimax risk R(L(), defined as the optimal value of (34
1 for single-source), over a certain range D 2 1.
The second set of hyperparameters {},)sm1 are determined solving
min 1 1e p.A.
stt T, A2
plyl (onti:)) 1 AT. X.
1=1 yEY
N 2
Σ p()) 1/8 for i =-1,.... I,...,b
(100)
VEY
for E [S] (S = 1 for single-source), that ensures the uncertainty set considered is non-empty.
Additional synthetic experimental details. For the synthetic experiments presented in Section 6 we utilize linear feature mapping (xx) = [1,x"] and implement the method referred to as no adapt. using 0-1-
loss as shown in (aazuelas et al. 2023). and reweighted and DW method using true marginal probabilities
and 0-1-loss.
For the synthetic experiments presented in Section 6.2.1 we utilize linear feature mapping (xxx and implement the existing DW-GCS method estimating weights using 0-1-loss and weighting kernel mean matching (DW-KMM) as shown in (Segovia-Martin et al., 2023).. 2SW-MDA method estimating weights using KMM as shown in (Huang et al. 2000), and the proposed DW-MSCS method using (-1-loss and estimating weights using 556
Additional real datasets experimental details. For the experiments in Section 1.2 we have considered binary and multiclass classification datasets: "Ad "Adult", "Diabetes", "Mammographic", "Usenet2", " "Credit", available at the UCI repository (Dua & Graff, 2017)) "20 Newsgroups", available at http://qwone / "Redwine", available at (Cortez el al., 200g). For the experiments using "20 Newsgroups" dataset, we utilize the 300 features with highest Pearson's correlation. We considered linear fcature mappings (xxx = [1, and 0-1-loss.
jasonn /ttpo////eeewgruuuss//ossae
Table details the characteristics of the datasets used in the experiments Section The table also shows the number of training and testing samples selected for each of the experiments.
For the experiments in Section 6.2.2 have considered three multi-source classification datasets: "20 Newsgroups", available at http:
http:///////aaaao////////rrrrrpp "Sentiment. Analysis", available at and "Spam detection", available at http:// These datasets have been previously used in single-source Mansourr et al. 2008; Ben-David et al., 2010; Sun et al., 2011
https://www.cst http::/www.....oooo/cccccaa/aaaa
www www.eemmlpdd22006.co wtwpooaallgggeeeeeeeoooo html
and multi-source covariate shift papers
Duan et al. 2012 Sun al. 2015 Wang et al., 2023). We have also considered two image classification
datasets: "D "DomainNet",, available at https://aii bu. //tps////i.bb.eeddaMM/s/// (Peng et all 2011), and "Office-31", avail- data/ datasetaaaeeeeeeeeeeeea má. For the
able at https://githut: com/ wttp://iiooggggg//aaatooooooeaaa experiments using "Spam detection" dataset, utilize multiple number of features with highest Pearson's 's correlation, and randomly sample 200 training samples from each souree and 200 testing samples in cach repetition. For the experiments using "Sentiment" dataset, the features were defined as the set of unigrams that appear five times or more in all domains. We used a binary feature vector encoding the presence of those unigrams, as done (Mansour et all, 2003). We randomly sample 1,000 training samples from each source and 150 testing samples in each repetition. For the experiments using "DomainNet" dataset, we reduce the number of classes to 4, since the original dataset has 256 different classes. The goal is to predict.
41

---

Published in Transactions on Machine Learning Research (02/2025)
Table 4: Datasets used the experiments of Section
Samples Selected training
Dataset
Type of Shift
Covariates
Samples
testing
tweak-one
500
500
Adult
14
48842
knock-out
330
500
tweak-one
100
100
Diabetes
768
knock-out
94
192
tweak-one
100
100
Mammographic
1500
knock-out
151
276
tweak-one
250
250
Usenet2
99
1500
knock-out
179
360
tweak-one
100
100
Credit
15
690
knock-out
112
198
tweak-one
300
300
20 Newsgroups
knock-out
500
9016
6017
330
400
dirichlet
300
300
tweak-one
100
100
Redwine
knock-out
690
101
189
dirichlet
100
100
Table Summary of sources using "20Newsgroups" dataset.
Sources
comp
rec rec.autos
talk
Source
comp.graphics
sci.crypl elcctrocoos sci.med sci.space
dalk.politics gums
Source 2
comp,os.. mas www.wwidddwww.e.
molorcycessss
talk. falk.politics.o politics.mideast mideast
Source comp.sys.ibr www.sy..siipp....m...mm hardware haw.ww.arrdawareeeeeeee...
baseballl crr.sport.hocker
talk, wwl.ppoolsisiiiiiooo mise talk.religion... misc
Source
nomp.sys.mac..
hww.d.awa..r..rre........ hardware
if the image an airplane, bus, ambulance or police car. We first use ResNet-18 to obtain features of dimension 512, and randomly sample 100 training samples from each source and 200 tosting samples in each repetition. For the experiments using "Office-31" dataset, we first use ResNet-50 to obtain features of dimension 2048. We have considered different classification problems, selecting five labels depending of the type of the object. For electronics, the goal is to predict if the image is laptop, monitor, mouse, keyboard or speaker. For stationery, the goal is to predict if the image a pen, paper notebook, ruler, scissors or eraser. For organization, the goal to predict if the image a file cabinet, ring binder, letter tray, clipboard or trash can. For mixed, the goal is predict the image is a backpack, mug, monitor, trash can or scissors. We have utilize the 100 features with highest Pcarson's correlation, and randomly sample half of the samples from each domain as training samples, and half of the samples from each domain, ensuring that the same number of samples from each domain, as the testing set in each repetition.
Table details the procedure followed for the creation of the different sources in the "20Newsgroups" dataset. For this experiments, we first generated four sources as shown Table Then, we generated three training sources subsampling 100 samples from one of the sources and samples from other two sources. For the testing set, we subsampled 40 samples from the three sources from which sampled 100 samples training and 30 more samples from the fourth source.
ddetails the characteristics of the datasets used in the experiments. The table also shows the parameter a used in the computation of the kernel matrix K for the KMM, DW-GCS, 2SW-MDA, CW KMM and DW-MSCS methods, which is determined using the common heuristic based on nearest neighbors with K = 50. For these experiments, we have considered 0-1-loss for the methods based on double-weighting.
For the experiments in Section 6.3 we also considered "Sentiment analysis" dataset, described in Table defining the features for multi-source covariate shift. We considered linear feature mappings of the form
42

---

Published in Transactions on Machine Learning Research (02/2025)
Table 6: Datasets used the experiments of Section 6.2.
Ratio majority class
Dataset
Sources
Covariates
Samples
Spam detection
500 features
3
500
200/200/200
200
0.5000
12.6828
1000 features
3
1000
200/200/200
200
0,5000
15.3386
1500 features
3
1500
200/200/22
200
0,5000
16.9269
2000 features
2000
200/200/200
200
0.5000
17.7283
2500 features
3
2500
200/200/200
200
0.5000
19.1219
3000 features
3
3000
200/200/ 200
200
0.5000
21.5444
20 Newsgroups
comp vs rec
3
1,000
=1,177/11,17/11,172
3,143
0.4937
18. 18.7058
comp sci
3
1,000
x1.172/L.172/11772
3,128
0,4952
18.650
comp vs talk sci
3
1,000
#1,066/L,066/1L006
2,856
0.5435
20.4118
3
1,000
11.1188/1.88/11188
3,161
0.5015
19.1100
rec vs talk
3
1,000
211,022/1,02211.082
2,889
0.5497
21 21.2683
sci talk
3
1,000
=11000//1.00//1.000
2,874
0.5483
20.8485
comp vs rec vs sci
3
1,000
11,767711,6671,777
4,716
0.3368
37.1962
comp vs
talk
3
1,000
=1,661/1,661/LL 61,661
4,444
0.3579
37.6344
comp vs sei vs talk
3
1,000
(1,659/L,,59/1.. 1,659
4,429
0.3565
37.5618
rec vs sci vs talk
1,000 3
=1,675/L,665/L,667
4,462
0.3555
38.0282
Sentiment
All domains books
3,034
2,000/2,000/2,,00//
8,000
0.5000
10.3441
3
3,034
20000/20000220000
8,000
0,5000
10.3441
dvd
3
3,034
2,000/2 20000/20000220000 2,00
8,000
0.5000
10.3441
electronics
3
3,034
2,000/2,000 2 2,000
8,000
0.5000
10.3441
kitchen
3
3,034
2,000/2,000 2,000
8,000
0.5000
10.3441
DomainNet
clipart
5
512
385/216/ /558/ /698/ 0008/1190 /1000
600
0.2931
21 21.8061
infograph
5
512
2226/22 221 6 258 /6966////1/ /9900
600
0.3028
21.5041
painting
5
512
2226/ 2226/385/22 /3/333555//22 /2////2///0 /68/ 1900
600
0.2891
21.5743
quickdraw
5
2266/385/1166 66666/9//8/ 1900
600
0.2984
21.5364
real
5
512
2226/385) 388/211666611 /258/ 1900
600
0.2994
21.4893
sketch
5
2226/385/211 /21/ /258 258/696
/698
600
0.3171
24.0266
Office-31
electronics
2
100
249/80
141
0.2181
8.6251
stationery
2
100
232/59
90
0.2234
8.7266
organization
2
100
214/65
102
0.2240
9.1813
mixed
2
100
225 /73
111
0.2451
8.5133
-(x)=[[,z)) and implemented existing reweighted methods using log-loss, and DW-MSLS using 0-1-loss. Additional experiments regarding the hyperparameters. In the additional experiments have studied the effectiveness of the proposed selection method for hyperparameters D and X.
The hyperparameter À controls the confidence with which the true underlying distribution is contained in the uncertainty set U. Table sshows the mean error and the standard deviation of the different experiments performed see the impact the hyperparameter in label shift scenarios. The proposed value of X X, is obtained by solving the optimization problem in (L0.. The rest of the values of the hyperparameter À considered of the form X A with Ån E 1,22,,,,,,22,--0 method for selecting À results in performances near those obtained with the best values of A. As we see from the table, properly tuning the hyperparancter À is crucial, with small values of being preferred in order to achieve small classification errors.
2-1,1). Table how the proposed
The hyperparameter /) controls the trade-off related to the set weights (3,(z, 8) y)> {as (z, y))sEss As presented in Section 4.4 [1, S B/VD, where B suPaex (re y) ES Ps (2 v). and
43

---

Published in Transactions on Machine Learning Research (02/2025)
Table 7: Classification error in 6 scenarios using DW-LS methods varying the value of the hyperparameter
X. The bold values represent the lowest classification error in each scenario.
Dataset
Ap
Ag
Ap 2
Ap 2
An =
An 10=2
Ag =
Redwine
knock-out
.52
.09
.52 .09
.09
.54 + .10
.55 + .09
.57 .07
.64 .07
tweak-one
.49 24
.47 24
.4 + 224
.46 +6 331
.53
+ .27
.59 21
.65 + 20
dirichlet
.53 +11
.52 - .11
.50 + 110
+2..10
.62 + .09
.62 08
.66 + .03
20news
knock-out
.64 .04
62 = 02
64 003
.69 04
77 .02
.73 .22
.74 .02
tweak-one
.61 + .08
.58 + .04
59 + .05
.59 + 10
.61 + .06
.68 + .07
.70 .07
dirichlet
.65 + .20
.65 + .02
.63 .02
.65 + .44
.72 .03
.72 + .0
.74 .02
min E.=1 max.:)/ea E os(2,9) = 1/VD. Considering small values of D results poor expeclation estimate, since the expectation estimate is bounded by (2, y) S B/vD, discussed in Section 2 On the other hand, considering small values of D results in classification rules with significant confidence in a large region. If D 1, then the weight functions (2)x)) = for all 8 E S. This way, the doible-weighting approach is simplified to the standard reweighted approach, and we equally combine the training sources to obtain the classification rules. Figure b and Figure show the different values of the sum of the weights alpha, Σg (z),, when we consider different values of the hyperparameter D for the synthetic experiments corresponding to Section 6. of the paper. In the figures
how the sum E.=1 (z) decreases when we increase the value of the hyperparameter D. In addition, Figure 7 how the value of the weights (3.(z) he[[]] is also reduced when we increase the value of the hyperparameter D. As we can see in the figure, we avoid large weights 3, by reducing the corresponding value a,
D=1.6
D=1.6
60
0.0
0.5
0.7
20
066
1.5
0.5
Figure Visualization of Σ (x) for hyperparameter D 1.6. Left: Plot of the testing instances on 2 together with the colorbar representing the sum values the values of E (x(x). Right: Histogram of the values of ESe (x) for the testing instances.
44

---

Published in Transactions on Machine Learning Research (02/2025)
D=25
D=25
0.9
-005 0.5
0.1
0.2
0.3
0.5
0.6
0.7
0.9
20)
Figure 6: Visnalization of E. (x) for hyperparameter D = 25. Left: Plot of the testing instances on IR? together with the colorbar representing the sum values the values of Σ (z)).. Right: Histogram of the values of E (x) for the testing instances.
16
D=1
10
D=16
102
104
10
101
10
109
D=25
16
102
10
10f
Figure 7: Histogram of (8)(2) for hyperparameter D correspond to those of the reweighted methods.
Note that for D 1 the weights
45


Reviews:
[{"review_id": "mDBb79I7nT", "reviewer": "Reviewer_jqcK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. This paper focuseson  an important problem.\n2. This paper proposes a reweighted based method in both training and test samples, which is novel.\n3. This paper makes a lot of theory analysis.\n4. This paper proposes a novel method for DA based on the theory.\n5. Better results is obtained than baselines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on domain adaptation, an important problem in machine learning. This paper proposes a weighted-based method for domain adaptation in multiple scenarios, including single-source domain, multi-source domain, and label shift scenarios. The main idea is to reweight both training and test samples. Besides, it also considers whole-domain importance in multi-domain scenarios. This paper makes a lot of theory analysis and contributes many meaningful results. Based on these theories, a novel method is proposed. Extensive experiments are conducted in both synthetic and real dataset, and the results show the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My biggest concerns are about the experiments. 1) The datasets in the experiments are not the common datasets in domain adaptation. So I suggest conducting the experiments on the most common datasets, such as small-scale datasets, Office-31, middle-scale datasets, Office-Home, and large-scale dataset DomainNet. I think selecting one or two datasets is enough to show the effectiveness of the proposed method. 2) The comparison baseline is outdated, and more recent methods published in the past two years are suggested to compare.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bExDIgp4mV", "reviewer": "Reviewer_SA1f", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "However, to me the technical contribution of this manuscript seems to be rather trivial. For example, Eq. 10 poses one of the major differences between the previous double-weighting [1], which considering the difference between joint distributions of training and testing domains when computing sample weights. However, this modification does not bring significant differences to the following derivation compared to those in [1]. In fact, most derivations from page 5-8 are nearly identical to [1] (e.g., Eq. 11-18 correspond to Eq. 9-16 in [1]). When coming to the newly discussed label shift scenario, it merely replaces the variance shift $\\mathrm{p_{tr}}(x), \\mathrm{p_{te}}(x)$ of [1] with $\\mathrm{p_{tr}}(y), \\mathrm{p_{te}}(y)$.\n\nMulti-source double-weighting approach is the other major contributions claimed in this paper, yet similar observations can be found, where the introduction of multiple sources brings limited differences compared to the existing work. Due to the double-blind reviewing protocol, I can’t tell whether this is an extended journal version of [1]. If not, the current version shows limited difference compared to [1] and should therefore be revised majorly.\n\n[1] José I. Segovia-Martín, Santiago Mazuelas, Anqi Liu. Double-Weighting for Covariate Shift Adaptation. ICML 2023", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a unified variant of double-weighting approach to tackle distribution shifts between training and testing domains. The proposed paradigm, which is mostly built and inherited from the previous double-weighting method [1], extends the double-weighting method from specifically considering the covariates shift ($\\mathrm{p_{tr}}(x), \\mathrm{p_{te}}(x)$) to a more general case involving the shift of joint distributions ($\\mathrm{p_{tr}}(x, y), \\mathrm{p_{te}}(x, y)$). Furthermore, the unified double-weighting is applied to a multi-source scenario, achieving noticeable improvement compared to existing methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Strength and Weaknesses for details.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sqhU6maoZn", "reviewer": "Reviewer_B8SF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper has solid theoretical support.\n2. The structure and organization of the paper are well done.\n3. The paper considers a relatively comprehensive range of scenarios.\n\n**Weaknesses:**\n\n1. Can this method address the case of conditional shift?\n2. The paper seems to be an extension of the ICML 2023 paper; could you further emphasize the contributions and differences? It would be helpful to provide a detailed comparison in the appendix.\n3. Domain adaptation methods involve more than just weighting. Could you compare this approach with some recent SOTA methods?\n4. Is it possible to adapt this method to deep learning models?\n5. Could you conduct experimental comparisons of the proposed method using image datasets, such as DomainNet?\n6. Could you include more experimental analyses, such as the impact of hyperparameters, or provide visualization-based analysis?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a weighted general framework for the problem of distribution shift adaptation by applying a dual-weighting technique, which assigns weights to both training and testing samples. This idea is interesting and insightful.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The recommendations from the reviewers were Accept, Leaning Accept, and Leaning Reject. The reviewers are convinced that the paper shows the proposed method has great values through the theory and the experiments. The problem and the extension are highly relevant. It is good work and the acceptance criteria seem to be met.

If the acceptance is approved, I request the authors to prepare the camera ready version based on the latest version incorporating the discussions with the reviewers. Furthermore, here are a few of minor issues, which the authors could check:
- I could not find the clear definition of $\mu^\infty$ in Theorem 4.2. There seems to be a typo $\mu^\infty$ for $\mu^*$.
- I suppose that Theorem 4.2 is not for fully presenting the rate of convergence of the classifier because $\mu^*$ used in the bound is a random quantity and depends on $n$. There could be a remark on that.
- I'm not sure if it's clear which loss was used for the proposed method in the experiments.
- I don't think the standard 01-loss refers to the loss defined in Eq. (2), unless the range of the function is $\\{0, 1\\}$. They are very different and I find it confusing.
- The name "minimax risk classifier" might be also misleading. People might think it's referring to a classifier achieving the rate of the minimax risk in the sense of, e.g., Tsybakov (2009).
Decision: Accept (poster)