Knowledge-Guided Wasserstein Distributionally Robust Optimization
Authors: Zitao Wang
Ziyuan Wang
Molei Liu
Nian Si
Venue: ICML 2025 poster (2025)
Keywords: wasserstein distributionally robust optimization
knowledge-guided learning
difference-of-convex optimization
shrinkage-based transfer learning
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: We propose a novel knowledge-guided Wasserstein distributionally robust optimization framework for regression and classification, proving its equivalence to shrinkage estimation based on collinear similarity with prior knowledge.

Abstract: Wasserstein Distributionally Robust Optimization (WDRO) is a principled framework for robust estimation under distributional uncertainty. However, its standard formulation can be overly conservative, particularly in small-sample regimes. We propose a novel knowledge-guided WDRO (KG-WDRO) framework for transfer learning, which adaptively incorporates multiple sources of external knowledge to improve generalization accuracy. Our method constructs smaller Wasserstein ambiguity sets by controlling the transportation along directions informed by the source knowledge. This strategy can alleviate perturbations on the predictive projection of the covariates and protect against information loss. Theoretically, we establish the equivalence between our WDRO formulation and the knowledge-guided shrinkage estimation based on collinear similarity, ensuring tractability and geometrizing the feasible set. This also reveals a novel and general interpretation for recent shrinkage-based transfer learning approaches from the perspective of distributional robustness. In addition, our framework can adjust for scaling differences in the regression models between the source and target and accommodates general types of regularization such as lasso and ridge. Extensive simulations demonstrate the superior performance and adaptivity of KG-WDRO in enhancing small-sample transfer learning.

Knowledge-Guided Wasserstein Distributionally Robust Optimization
Zitao Wang' Ziyuan Wang?
Molei Liu
*3 Nian Si
Abstract
1. Introduction
Traditional machine learning methods or empirical risk min- imization often suffer from overfitting and a lack of general- ization power, particularly in high-dimensional and small- sample-size settings. In recent years, distributionally ro- bust optimization (DRO) has emerged as powerful frame- work for mitigating the effects of model misspecification and enhancing robustness in machine learning generaliza- tions. Among various DRO formulations, Wasserstein-DRO (WDRO) gained more attention due to its tractability and generalizability. Specifically, in WDRO, one optimizes over worst-case distributions within ambiguity set defined by Wasserstein ball centered at an empirical measure.
Wasserstein Distributionally Robust Optimization (WDRO) is principled framework for robust esti- mation under distributional uncertainty. However, its standard formulation can be overly conserva- tive, particularly in small-sample regimes. We propose a novel knowledge-guided WDRO (KG- WDRO) framework for transfer learning, which adaptively incorporates multiple sources of ex- ternal knowledge to improve generalization accu- racy. Our method constructs smaller Wasserstein ambiguity sets by controlling the transportation along directions informed by the source knowl- edge. This strategy can alleviate perturbations on the predictive projection of the covariates and protect against information loss. Theoretically, we establish the equivalence between our WDRO formulation and the knowledge-guides shrinkage estimation based on collinear similarity, ensur- ing tractability and geometrizing the feasible set. This also reveals a novel and general interpreta- tion for recent shrinkage-based transfer learning approaches from the perspective of distributional robustness. In addition, for scaling differences in the regression models between the source and target and accommodates general types of regularization such as lasso and ridge. Extensive simulations demonstrate the su- perior performance and adaptivity of KG-WDRO in enhancing small-sample transfer learning.
However, one persistent challenge with WDRO is its ten- dency to be overly conservative, which can lead to subopti- mal performance in practice as found in (Liu et al., 2024). In many real-world scenarios, prior knowledge can be lever- aged to improve model performance and robustness. This transfer learning approach falls under the category of Do- main Adaptation, which adapts models trained on source domain to perform well on a related target domain with limited labeled data.
A key application in clinical trials, where the binary out- come Y {0,1} indicates treatment success or failure, and the high-dimensional covariate X encodes a patient's physical and health conditions along with treatment details, Data scarcity is common common-esspccilly for underrepresented populations. To address this, we leverage a classifier trained on a majority group (parameterized by 0) as a reference to estimate a classifier for the minority group (parameterized by 3). This knowledge- guided transfer learning reduces uncertainty by anchoring the search for / in the direction of 6. In such context, transfer learning has proven to be ver- satile approach for improving performance on target task. Despite its successes, the integration of prior knowledge into WDRO frameworks has remained an open question.
framework can adjust
Equal contribution 'Department of Statistics, Columbia Uni- versity, New York, USA. DDeaarment of Industrial Engincering and Management Sciences, Northwestemn University, Evanston, USA. 'Department Biostatistics, Peking University Health Sci- ence Center; Beijing International Center for Mathematical Re- search, Peking University, Beijing, China. 'Department of Indus- trial Engineering and Decision Analytics, Hong Kong University of Science and Technology, Hong Kong, China.. Correspondence to: Nian Si niansi hkk, Molei Liu <molciliu@ bjmu.edu.cn.
In this work, we introduce Knowledge-Guine -uidded Wasserstein Distributionally Robust Optimization (KG-WDRO), a novel framework that adapts the Wasserstein ambiguity set us- ing external knowledge (parameters). We assume access to prior predictors of pre-trained models, which can guide the predictive model in the target dataset. By constrain- ing the transport cost along directions informed by prior
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Knowledge-Guided WDRO
knowledge, our approach addresses the conservativeness of vanilla WDRO while preserving robustness. Intuitively, this strategy allows the model to focus its uncertainty regions where prior knowledge is less reliable, effectively robustify showledge-guidees generalization.
WDRO and shrinkage-ban estimation methods, offering a novel perspective that unifies and interprets broad range of knowledge transfer learning approaches through the lens of distributional robustness. Table 1 provides an overview of them, highlighting their key capabilities and advantages and comparing them with our framework. Technicalities: Leveraging Toland's Duality (Theorem F.I), we reformu- late the innermost maximization in WDRO's strong duality (Proposition 2.1) into univariate optimization problem (Toland's Duality). This reformulation enhances tractability while accommodating more general cost functions. Em- pirical Validation: Through extensive experiments, we demonstrate the effectiveness of KG-WDRO in improving small- sample transfer learning.
1.1. Related Works
1.1.1. WASSERSTEIN DRO
Wasserstein DRO has recently garnered significant attention due to its tractability (Mohajerin Esfahani & Kuhn, 2018; Blanchet & Murthy, 2019; Gao & Kleywegt, 2023) and gen- eralizability (Blanchet et al., 2019a; Gao et al., 2022). No- tably, Blanchet et al. (2019a) and Gao al. (2022) demon- strate that Wasserstein DRO with mean square loss is equiva- lent to the square root lasso (Belloni et al., 2011). Similarly, Shafieezadeh-Abarne Abadeh et al. (2015; 2019); Blanchet et al. (2019a); Gao et al. (2022) establish that Wasserstein DRO with logistic loss and hinge loss corresponds to their regular- ized counterparts. Morcover, the statistical properties of the WDRO estimator have also been investigated in (Blanchet al., 2021; 2022; Gao, 2023). However, leveraging external knowledge in Wasserstein DRO has been open problem.
Below is an overview of our main results for the linear regression case.
Example 1. Suppose 0 is an accessible prior predictor for a linear model parameterized with 3. We show that the shrinkage-baus based leamnng regression problem. which estimates a target predictor |3 by solving
inf lyy - -x,3122 + vol13 =01lpp B
can be interpreted as Wasserstein distributionally robust optimization (WDRO) problem of the form (WDRO), where the loss function is least squares, ((X, 3) (Y x)², and the ambiguity set (Px:: (2,5o) is defined as ball around the empirical measure. The cost function 02 augments the standard transport cost by the constraint x 00 u²0 so that
1.1.2. TRANSFER LEARNING
Improving prediction accuracy for target populations by in- tegrating diverse source datasets has driven methodological advances in transfer learning. Contemporary approaches aim to address challenges including distributional hetero- geneity and limited labeled target data. A common assump- tion is that the target outcome model aligns partially with source models, enabling knowledge transfer. For example, recent frameworks employ selective parameter reduction to identify transferable sources and sparse or ridge shrinkage to leverage their knowledge (Bastani, 2020; Li et al., 2021; Tian & Feng, 2023). Subsequent works tackle covariate distribution mismatches and semi-supervised scenarios, en- hancing robustness when labeled target data is scarce (Cai et al., 2024; He et al., 2024; Zhou et al., 2024). Further innovations include geometric or profile- based adaptations, where the target model is represented as a weighted com- bination of source coefficients (Gu et al., 2024; Lin et al., 2024).
02 ((x, v), (u, w))
=||x ul? + x lv rl + 00 -(x u)
This establishes a distributionally robust optimization (DRO) perspective broad class of transfer-learning methods as will be discussed in Section 3.
1.3. Notations & Organizations
We summarize the mathematical notations used in this work. The positive integers N, M, and d denote, respectively, the target sample size, the number of sources, and the dimension of the support of the covariate X. The integers and e [1, ooj are reserved for pairs of Hölder conjugates satisfying p- + + 1 for p, q E (1, oc). as well as the pair 1 and 0o. For a distribution P supported on the Euclidean space Rd we use PN to denote the empirical measure of IP with sample size N. In modeling the target-covariate relationship, the distribution is often factorized as P = PYIX x P.X For a vector V E Rd, ||o||p denotes the p-norm, where E [1, oo]. and Vt denote the transpose of v. For any two vectors u, e E Rd, the notation cos (u, v) denote the cosine of the angle between u and v, calculated by cos (u, u) [)][u[|2|[||)) =
1.2. Our Contribution
Our contributions are fourfold. Framework: We introduce KG-WDRO, a principled and flexible framework that inte- grates prior knowledge into WDRO for linear regression and binary classification. This framework mitigates the conservativeness of standard WDRO, enables automated covariate scaling adjustments, and prevents negative trans- fer. Theory: We establish the equivalence between KG-
2

---

Knowledge-Guided WDRO
Table
Overview recent transfer learning techniques. Each column represents key capability: Ridge-type Lasso-type Regular- ization type used: Scale Adjustment Robustness against feature- wise scaling; Continuous outcome / Binary outcome Supports regression classification; Partial Transfer Selections of prior knowledge; Multi-Source ensemble Profiles on multiple prior knowledges.
RIDGE
LASSO
SCALE ADJUSTMENT
CONTINUOUS
BINARY OUTCOME
PARTIAL TRANSFER
MULTI-SOURGS ENSEMBLE
METHODS
-TYPE
TYPE
OUTCOME
KG-WDRO
BASTANI (2020)
LT AL. (2021)
TIAN & FENG (2023)
GU ET AL. (2024)
LIN ET AL. (2024)
v. All vectors are assumed to be column vectors, Other specialized notations are defined in context as needed.
think of the value c(u, v) as the cost of transferring one unit mass from u E R to v E Rd, then E, [,,[,(U, V)] gives the average cost of transferring under the plan x. The optimal transport cost (P, Q) gives measure of discrep- ancy between probability distributions on IR..
The remainder of the paper is organized as follows. Section 2 provides review of the WDRO framework, including the strong duality result. In Section 3, introduce our KG-WDRO framework and demonstrate its equivalence to shrinkage-based estimations in both linear regression and binary classification. Section 4 presents comprehensive results from our numerical simulations. All proofs and detailed descriptions of the numerical simulation setups are provided in the appendix.
If c(u, v) defines metric on Rd, then for any p (o) the optimal transport cost,
inf Ex [c(U, Io( 1/P (TET((
DV/P(P Q)
defines a metric between probability distributions and metrizes weak convergence under moment assumptions. is called the p-Wasserstein distance. We direct the interested readers (Villani, 2009, Chapter 6) for more details. is worth mentioning that none of our judiciously chosen cost functions qualify as metrics on the support of the data.
2. Preliminaries
We first begin with short overview of the distributionally robust framework on statistical learning.
2.1. Optimal Transport Cost
2.2. Distributionally Robust Optimization
Let IP and Q denote two probability distributions supported on TRd, and we use P x Rd)) to label the set of all prob- ability measures on the product space Rd x Rdd. We say that an element P(Rd x Id)) has first marginal and second marginal if
In standard statistical learning framework, one generally assumes that the target-covariate pair (X,Y) E Rd x R e Rd+1 follows a data-generating distribution = x on the support Rd-1, One then seeks to find 'best' parameter 3 that relates Y to X through a parameterized model by solving the stochastic optimization,
x (AA x R P(A), T(R x B) Q(B).
for all Borel measurable sets B, E Rd. The class of all such measures is collected as II[P, Q), and is called the set of transport plans, which is always non-empty. Choose a non-negative, lower semi-continuous function e Rd x IRd + (0, oo] such that c(u, v) = whenever v. then the Kantorovich's formulation of optimal trans- port defined as
inf Ep (((XY:))) 3
(SO)
The loss function l(x, 3: 3) provides quantification of the goodness-of-fit in the parameter 33 given the realized observation (x, y). Since only samples [(x,, vi))
are observed, we can typically only solve the empirical objective.
D.(P, Q) inf Ex V)).
N
TET((P.))
inf E. (XX,Y inf N
(1..)))). (ERM)
is well-known that (Villani, 2009, Theorem 4.1) there exists an optimal coupling n" that solves the Kantorovich's problem inf inf Ez (e(U, V)). Intuitively, we may
Therefore the distribution P that underlies the data- generating mechanism is uncertain to the decision-maker.
3

---

Knowledge-Guided WDRO
This motivated the distributionally robust optimization (DRO) framework, which entails solving the following min- imax stochastic program:
3. Knowledge-Guided Wasserstein DRO
In this section, we propose new cost functions for the Wasserstein DRO framework that leverage prior knowledge for transfer learning. For linear regression and binary classi- fication, these cost functions act regularizers, encouraging collinearity with prior knowledge.
inf sup E,(e(X,Y;,)),
(DRO)
# PaPam
where the ambiguity set Pamb represents class of probabil- ity measures supported on Rd+1 that are candidates to the true data-gene -eneerting distributions. In Wasserstein- the ambiguity set is constructed by forming 'Ó-ball' around the canonical empirical measure IN associated to the decision- maker-defined transport cost c, i.e. we let the ambiguity set Pamb be chosen as:
3.1. Knowledge-Guided Transport Cost
It is shown in (Blanchet et al., 2019a, Theorem 1) that using the squared g-norm on the covariates the cost function
22 (2 y). (u, b)) = |- ml co ly -y (I)
e/.
== {P E P(R = 8).
(WDRO)
equates Wasserstein distributionally robust linear regres- sion with p-norm regularization on the root mean squared error (RMSE). The cost function 02 perturbs only the ob- served covariates {2r ) while keeping the observed tar- gets { fixed. Keeping the observed target Y as fixed often leads to more mathematically tractable reformulation, another intuition is that we trust the mechanism by which the target Y is generated once X is known.
This ambiguity set captures probability measures that are close to the observed empirical measure in the transport cost De, which may be taken as class of candidates of measures perturbed from PN. The solution ²DRO (DRO) that solves the worst case expected loss should perform well over the entire set of perturbations in the ambiguity set. This is in contrast to SERM that solves (ERM) only performs well on the training samples. This adds a robustness layer to the WDRO problem (WDRO). For a comprehensive overview of different constructions of ambiguity sets, we direct the interested reader to (Kuhn et al., 2024, Section 2).
In the presence of prior knowledge 0 that may aid in infer- ring B, we aim control the extent of perturbation along the direction of B.
Specifically, we constrain the size of the prediction discrep- ancy 0-I I - 0Ju = O'A, where A == x - u. To achieve this goal, we henceforth augment the cost function C2 with an additional penalty term that accounts for the size of the perturbation in the direction of O:
2.3. Strong Duality of Wasserstein DRO
The Wasserstein DRO problem involves an inner maximiza- tion over an anfinit-ddimensinne set, which appears com- putationally intractable. However, the distribution P., is discrete, strong duality of the Wasserstein DRO reformu- lates it as a simple univariate optimization.
02.x ((x, y), (u, u))
=|| + 0c .1 - v| + AI(IOA)).
(2)
Proposition 2.1 (Strong Duality, (Blanchet et al., 2019a, Proposition 1)). Let e IRd+1 x Rd++1 [0, oo] be lower semi-continuous cost function satisfying el c((x,, y), (u, u)) 0 whenever (I, v) (u, v). Then the distributionally robust regression problem
where À > 0 and h(x) : - U {0} is non-negative, monotone increasing function of |æ]] such that h(O) 0. Recall that in the cost function c2).), the targets y remain fixed. Intuitively, the new cost function (2) encourages the Wasserstein ambiguity set to include distributions whose marginals in X generate predictions that align with the data based on the prior predictor O. The parameter À controls the level of confidence in the prior knowledge. We call this kind of cost functions mhowledge-guided Since C2 upper bounds the cost function C2, we have B3 (P c2,x2) S B& (P 22.x,) S BB ; (2) whenever X2 > \1-
inf sup SERd (FB)
Ep [[(X, 81],,,
is equivalent to,
N (2,, (, ue 31
inf inf BER 920
78 +
1=1
where (2 Wi: 3) given by.
The corresponding optimal transport problem given by:
sup (luu,,; rc((u, w). (2:, v:))) (0,0)ER4+1
inf PX) xen(O
Ex
For more general results, see (Blanchet & Murthy, 2019, Theorem 1) and (Gao et al., 2022, Section 2). The ex- changeability of sup and inf in Wasserstein-DDe is also established by (Blanchet et al., 2019a, Lemma 1).
can also be expressed as:
inf =En(QN
Ex (X), + XE, (h((0' AI)] .x)

---

Knowledge-Guided WDRO
This formulation regularizes the original optimal trans- port problem by penalizing large values of the expectation Ba [a[((((=A))]-
any additional distributional assumptions on (X, Y), we obtain the following finite-dimensional representation.
Theorem 3.3 (Linear Regression with Strong- -Tans-rrransfr Consider the least-squared loss f(X, Y; (Y 5 x²² then for any q E [1, we have
For any user-defined function h that measures the discrep- ancy in generalization with respect to the prior knowledge 0, we refer to it as weak-transserring of knowledge if X < +00, of knowledge if X +oc. In the case of strong- transferrn,,,,,, to ensure the finiteness of the optimal transport problem, the minimizing transport plan I must satisfy the orthogonality condition 0 0, -- almost surely. Consequently, the value of 0* X remains unchanged after perturbing PÑ within (P, result, this should promote $DRO - as - co. Inded, we have the following proposition on the bound of the mini- max objective with strong transferring.
inf sup 3ER' (F
Ep [(Y - 10V--3'X'² 3'x]²]
and
inf BERR,DEG
(VMSEN (3) + vō1ß .blb))
where p is such that p + -
N: 22,00 ) As a
From the above result, we observe that the knowledge- guided WDRO problem for linear regression is equivalent to regularizing the RMSE with a p-norm distance the linear span O The regularization parameter is entirely determined by the size (or radius) of the Wasserstein ambiguity set. Im- portantly, the penalty term focuses on the collinearity with the prior knowledge rather than their algebraic difference or angular proximity.
Proposition 3.1. Let ((X, 3) (Y 3 x)2 denote the least square loss, then
inf sup BER* (x:22
Ep BY Br x)
inf Ery [(Y (af) x) OER
Consider the case when there is only a single prior knowl- edge f1. the penalty term does not constrain the solution iDRO be close to oj. but rather to k 0, for some x e R to be optimized. Consequently, this knowledge transfer au- tomatically robustify solution against scaling of covariates, Furthermore, it can prevent negative transfer by adapting its sign to be positively correlated with f which is the solution to population objective (SO). When + oc, the penalty term becomes dominant, forcing 33 to lie in for any p 2 1. This reduces the WDRO problem to a simple constrained regression problem,
Thus, the minimax optimizer under knowledge guidance from A achieves out-of-sample performance that is at least as good the in-sample performance of the naive domain adapter &v&. 0. A similar statement applies to the binary classification settings as discussed in Theorem 3.6.
Remark 3.2.
The above framework extends to incorporate multi-sites prior knowledge, meaning that instead of a single prior knowledge coefficient 01, consider a set of coef- ficients {01, . 0xz) }. Let θ = span(0,, ....... bxr represent the linear span of these prior knowledge coeffi- cients. In the case of durong-transfenen
inf MSE (3),
we must ensure that rank(@) < d; otherwise, the set of orthogonality con- ditions {0T, A O; m [M]] would imply that the pertur- bation A is identically zero (A 0). This would render the ambiguity set redundant and reduce the WDRO prob- lem (WDRO) to the ERM problem (ERM). This result is confirmed by the statements of Theorems 3.3 and 3.6.
BEe
reflecting the complete reliance on the prior knowledge and prevents excessive shrinkage towards the null estimator. Remark 3.4. We now discuss two special cases of the penalty term, p = 2 (ridge-type regularization) and p 1 (lasso-tppe regularization). For simplicity, we consider the case of single prior knowledge vector 6.
3.2. Linear Regression
Ridge-type. The penalty term can be explicitly calculated
We begin by examining the WDRO problem (WDRO) for linear regression within the strong-transferiing domain. Fol- lowing this, we present a specific case within the weak- transferring domain. Let == span(O....., (0.....,0MM ,0M) represent the linear span of the prior knowledge.
as
BTO 1001
min || k8112 KEB
3
0
=
-||-1
||
12
where 8-0 the component of B3 orthogonal to This penalty term shrinks distance to the line the direction of O. Furthermore, note that
3.2. 1. TTRONG-TRANSFERGE
Define the cost function 02 ((x,)) (), , (u, v)) || ul u + 00. ly v| + oo 10 Tul +... + 00 -00 ml), and for set of observed samples {(I, wi)hEI we use
||ه-0| 1l2 ||31|2 (B, 0) |1812 cos-/
which represents a trade-off between the magnitude of B and its angular proximity to the prior knowledge A. This
MSE (8) = N -. - BTx.).. Without making
5

---

Knowledge-Guided WDRO
Pesor Knowtelge 0
Price Raowlddge
Prior Kuzwledge À = O.
1.0
1/00 x K
Contoni Val of [ 1a
BA80, =
Coctout P908 Velcoee of 1
12.17
Cocttout Valloo
(0.0)
10, 0f
8/88, xs i
100 a k.
Figure 1. The two-dimensional contour plots of the regularization term in Theorem 3.3 and Theorem 3.5 with À ranging from +00 to 2 0.1. The prior knowledge parameter is taken as 0 = (2, 1)". . The area between the black contours constitute feasibility set of the regularization tern when written in its equivalent constraint form. The feasibility set shrinks in the direction of 0, to circle of radius K when + from above.
trade-off is illustrated in the leftmost figure of Fig. 1, draw- ing the feasibility set of the regularization as constraint. This regularization is closely related but different to the one proposed in (Gu et al., 2024), where they penalize large values of computational relaxation of sin (3,0) 0).
With 1/2 Ta
00T and 113112 BT
|
Write PA = 00 00 /(10 + A). we note that as À + oc, we have Px-1 - Po 00 /1 recovering the projec- tion matrix onto the prior knowledge 6. Consequently, || - |||-
sparse, the penalty term min, | x01|, promotes sparse representa- tion learning. Consider simple example where the dimen- sion is d = 3 and 0 = (1, 0, 0)T In this case, we have:
Lasso-type. When the prior knowledge
We observe that the action
3T0 1011 + 1
Pa-
min ||3 - -0011 min (13) - <| + |82| + 1sal)
|321 + |as =: ||3-1|11,
is exactly the ridge regression of 3 onto 0 with a regular- ization parameter A- Thus, the finiteness of X, which can reflect caution in the prior knowledge 0, induces a shrinkage effect on the component of explainable by 0 in the dot product geometry. Since V Ia P. we have || > ||,3-0 ||2 for any finite À > 0, this implies the inclusion of feasibility set
where 3-1 = (82, R. This formulation enforces sparsity only on the last two components of 3, reflecting the sparsity pattern of 0.
3.2.2. WEAK TRANSFERRING
For the special case of 1 p 2, we define the weak- transferring cost function C2.x ((, y), (u, v)) = || - u1 + 10"xx-0'u>> + oc . lyy o| with 0 < X < +00. Here, we select h(x) = x² as the user-defined function on controlling the size of perturbation in 0. For simplicity, we consider single prior knowledge vector 0 in this setup. This result can be straightforwardly extended multi-source setup with different values of X's.
11 2-1 <K) 10 S KK.
as plotted in Fig. for an illustration on R.. The contour {B E R [1.31| = K} forms an ellipse centered around the origin O. The ellipse has a major axis of half length
|10| 3 + A-1
K
aligned with the direction of 0, and a mi- X-1
nor axis with half-length K aligned with the direction of g-, As À + 0, representing no-confidence in 0, the half-length of the major axis converges to K, resulting in a perfect circle as in ridge regression.
Theorem 3.5 (Linear Regression with Weak Transferring). Consider the least-squared loss E(X,Y: = (Y -" x, then for p= q = 2 we have
The two-dimensional hyper-parameters (5, 1-1 enable the use of data-driven methods, such as grid-search cross- validation, for transferring domain, the inclusion of A-1 allows the data to self-determine the informativeness of the source samples.
inf BER P:B. (P
sup
Ep [Y - 33
tuning. Unlike the strong-
inf SERd
(VMSEN(3) (8) Vo 11311,,.))
6

---

Knowledge-Guided WDRO
3.3. Binary Classifications
the statements of Theorems 3.3 and 3.6 can be easily gener-
alized. Define the space of positive-definite symmetrie trices as S and the cost function: c2 ((z, v), (u, (u, ==
In this section, we focus on the context of binary clas- sification, where the goal is to predict the discrete label Y E {-1, based on the covariates X E R.. Unlike the previous section, we use the g-norm, rather than its square, to account for distributional ambiguity in the covari- ate distribution. Define the strong-transfene transferring cost function C1.00 ((2, y), (u, v)) == || - u/lg + 00.10 o + 00. 10[= 0iul +... + 00 1 F7 u/. We consider two loss func- tions here. The logistic loss function is given by
|| uI3 + 00 ly ol + 00 >mm= 1 6 u|-
Corollary 3.7 (Theorem 3.3). For the least-squares loss (( = (Y - x) and any A E s
inf SER* (P
sup
EP [(V - 8)
= inf BER ( (VMSEN(3) + Vōll3 - )
( log (1+e-ver x))xx)
This formulation enables the use of metric learning meth- ods to determine A directly from the data, as detailed in (Blanchet al., 2019b). For example, if the two- dimensional prior 0 = [0,,0]] is known primarily in- fluence the first component of the truth B = [01 c, 82). we can select A = diag(dı, d2) with di < d2. This imposes a weaker penalty on perturbations in the first direction, result- ing in a weighted penalty term: min, ((B, - 1/d))/d + (32- - - which prioritizes aligning 31 with 01, while B3 is determined more flexibly based on the data. We call this sub-coefficient-veetor transferring, or the ability to partially transfer prior knowledge. A similar corollary applies to Theorem 3.6, as stated in Corollary D.I.
which is the negative log-likeliten log-likelhhood of the model that postu- lates
P(Y 1|X = = 1r log P(Y = 1| = x)
The hinge loss is given by
Y3 (1- (1-))
which is typically used for training classifiers that look for 'maximum- mmaximum-margins' in class boundaries, most notably sup- port vector machines.
Suppose Y E {-1,1} is binary and without any distri- butional assumptions on X, we have the following result which recovers regularized logistic regressions and support vector machines.
Finally, we again draw the reader's 's attention to Table 1, which compares several transfer learning methods discussed in Section 1.1.2. Notably, our proposed KG-WDRO frame- work brings together a broad range of desirable capabilities within a single, unified approach to transfer learning.
Theorem 3.6 (Binary Classification with Strong Transfer- ring). Suppose the loss function E(X,Y ;3)) () is either the logistic loss log (1+ e-+e-Va8 v x) or the hinge loss (1
4. Numerical Results
Y x)* then for any E [1, oo] we have
In this section, we present numerical simulations validate the effectiveness of the proposed KG- KG-WDRO method. We compare learners across different settings, including high- dimensional sparse models, correlated covariates, and multi- source prior knowledge, for either linear regression or binary classification tasks. Performance is evaluated using out-of- sample classification error for binary classifiers and out-of- sample R² for linear regressors.
inf sup (PB,,ENSCI,,
Ep [:(
SERMP:B,
inf BER DEG N
Aax Vs: 9) + 011B3 +
where p is such that p + -1=
For the single-source experiments, target-source coefficient pairs (3, 0) are generated from a multivariate normal distri- bution:
3.4. Lub-Coefficient-Vee Vector Transferring
In this subsection, we generalize the statements of Theorems 3.3 and 3.6 for p = 2 to arbitrary norms induced by positive- definite quadratic forms. Let A E ddxd be a positive- definite symmetric matrix. The norm || = Ax induces a metric on Rdd defined as da u) ||æ ullA. known as the Mahalanobis distance. Since A is positive def- inite, it admits decomposition A = I with I' invertible, and the norm ||æ||| = |[Tæ|2 measures length in the geom- etry distorted by I. By (Blanchet et al., 2019b, Lemma 1), the dual norm of || ||a ||- -I-IlA 1. Using Proposition F.6,
NN (0). 0² (o² (102 0² )²
poo))
(3, ~~ N
(3)
where o is the correlation between B and 8, and the ex- pected length of 0 is approximately a - 0.5. We scale 3 as 8 + sB with s E (0, 1] to study the stabilizing effects of strong prior knowledge in small-sample settings. The Limension-to-sampll ratio d/N is varied by fixing and in- creasing N. Performance is averaged over 100 simulations.
7

---

Knowledge-Guided WDRO
15
1.0
KG-WORC
KG-WDRD Weak
Trans-GLM
KG-WDRD Strong
Cut-rrSample Accuracyy
WDRO
Sample R.Squarnl
TransRidge
0.3
0.8
WDRO
0.7
0.7
Simulation 1: Logistic Regression High-Dimentttnnll Spa
Simulation 2: Linear Regression
higr-Dimensional Correlated 100
0.5
50 1004(]]
0.5
20, s=1
70 s=1
0.4
57
Correlation
Correlation
0.3
0.9
Out- Samale 1.Squared
Out of Sample n. 1.Sqqared
0.9
0.0
Simu Multi-Siles
Simulaatio Mulli
0.5
b, 1. -0.5, 0.2
0.6
b. 50 10040]
1001[c]
KG-WDRO
70
KG-WDRO
70
6.5
Trans: Lasso
0.5
Trans Lassa
WDRO
5.70
0.75
6.80
0.85
0.90
0.95
0.70
5/5
0.80
0.55
0.90
0.25
Correlation of Parameters
Correlation of Porameters
Figure 2. Out-of-sample performance plot of the proposed KG- KG-WDRO method for high-dimensional regression tasks, compared against benchmark methods. The plot shows performance variations p. representing the correlation between true and prior coefficient pairs, increases. Results are displayed for four specific settings across three experimental groups.
Each dataset consists of three parts: data - (train, val, test). The (train, val) pair shares the same size, and hyperparameters are selected based on validation performance. The source data contains 800 samples, with source truth 0 estimated accordingly. Out-of-sample perfor- mance is measured the test set of 5000 data points.
perform similarly (see Table 3 for detailed results). Both transfer learning methods, ßKG and STransG, significantly outperform the target-only learner, BWDRO-
4.2. Simulation 2: Linear Regression with & Transferring
In this simulation, we compare two learners on high- dimensional linear regression with correlated covariates against our proposed learners, 3KGweak (Theorem 3.5) and PKGssrong (Theorem 3.3), both using p 2. There is sparsity in the regression coefficients. The competing learners are the target-only vanilla WDRO learner @WDRO (Blanchet et al., 2019a, Theorem I) and the Trans-Ridge algorithm adapted from (Li et al., 2021, Algorithm 1), de- noted as Irransh- The covariates are fixed dimension 100, with pairwise correlation of 0.3. The experiment is conducted across six settings, varying the sample size N E {50, 70, 90). signal strength {0.8, 1}, and truth- prior correlation (0.3, 0.5, 0.7, 0.8, 0.9, 0.95].
4.1. Simulation 1: Logistic with é -Strong Transferring
In the first experiment, we compare two learners for bi- nary classification tasks with high-dimensional sparse efficients against our proposed KG- KG-WDRO learner, 8KG+ derived using Theorem 3.6 with = 1. The competing learners are the target-only vanilla WDRO learner ŚWDRO (Blanchet et al., 2019a, Theorem 2) and @iransG, obtained via the A-Trans-GI.M algorithm (Tian Feng, 2023, Al- gorithm 1). The target-source pair (3, 0) is generated using (3) with a dimension of 50 and augmented with 100 ros for sparsity, resulting in total dimension of 150. We test six settings, varying the sample size N E (20, 50, 80). signal strength s E (0.5, 1}. and truth-prior correlation E (0.3, 0.5, 0.7, 0.8, 0.9, 0.95).
As shown in the upper-right plot of Figure 2, the perfor- mance @TransR lags significantly behind both 3KGatrong and fKGweak until the correlation p becomes sufficiently high. Across all settings, 8KGstrong and 3KGweak consis- tently outperform ATransh when p is moderate low, as documented in Table 4. Furthermore, all three transfer learn- ing methods demonstrate superior performance compared
The comparison between fKG and BTransG is highly com- petitive, with 8KG consistently outperforming @TraneG by up to 2% in accuracy when the sample size is small (N- 20) across all values of p, as shown in the upper-left plot of Figure 2. In larger sample size scenarios, both learners
8

---

Knowledge-Guided WDRO
to the target-only learner, SWDRO-
goal is to predict county-level election outcomes in eight target states (Table. 2) using data from the remaining states as external source knowledge. The features include county- level demographics such as population size and ethnicity proportions, and the base model is logistic regression. The cleaned dataset consists of 3,111 counties and 761 standard- ized predictors across 49 states. We use data from 2, 100 counties as the source predict outcomes in the eight tar- get states (approximately 100 counties each). KG-WDRO outperforms Trans-GLM in 5 out of 8 states and reduces the overall log-loss by 7.6%. Both transfer learning methods significantly outperform the standard WDRO estimator.
4.3. Simulation 3: Transfer Learning with Multiple Sites
In the final set of experiments, we validate our methods multi-source transfer learning setting with high-dimensional sparse linear regression. The significant components of the three source coefficients are generated using (3) with correlation and dimension 50, denoted as {01, 02 0z). We construct a linear combination, 03 = a0) + b02 + cla, and generate = p0g + E. where E ~ (0, (1 - varr)r((s)) ensuring Corr(3,0s) = p. then scaled to match the magnitude of 0g. and all vectors are augmented with 100 zeros, yielding total dimension of 150. Our proposed method, 9KG (Theorem 3.3, 1). is compared against the oracle Trans-Lasso algorithm (Li et al., 2021, Algorithm 1) (3TransL) and the vanilla WDRO learner $WDRO. The experiment spans six settings: [a, c [1, -0.5, 0.2] and [1, I, 1], with @ = 0.9 and 0.6, respectively. Sample sizes vary in N (50, 60, 703. The truth-prior correlation ranges in p (0.7, 0.75, 0.8, 0.85, 0.9, 0.95).
5. Conclusion
We propose the knowledge-guided Wasserstein distribution- ally robust optimization (KG-WDRS framework, which utilizes prior knowledge of predictors to mitigate the over- conservativeness of conventional DRO methods. We estab- lish tractable reformulations and demonstrate their superior performance compared to other methods. For future work, we aim to provide statistical guarantees of our proposed estimators. Furthermore, based on these statistical proper- ties, we plan to develop principled approach for selecting hyperparameters such as d and A.
When [a, b, c| [1, - 0.5, 0.2). the contributions of the 0's to the generation of fs are unequal. In this case, it is not surprising that SKG outperforms TrransL, as shown in the bottom-left plot of Figure 2. When fg is an equal-weighted average of the 0's ((a, c] = [1, 1, 1)), the performance of $KG and 3TrausL becomes similar. However, fKG still demonstrates superior performance in larger sample sizes and higher correlations, as documented in Table 5.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There might be potential societal consequences of our work, none of which feel need to be specifically highlighted here.
Table 2. Log-loss values for WDRO, KG-WDRO, and Trans-GLM across the eight target states and overall in the U.S. election dataset. The best-performing method in each state and overall is highlighted in distinct colors.
References
Bastani, H. Predicting with proxies: Transfer learning in high dimension. Management Science, 67(5):2964-290) 2020.
STATE
WDRO
KG-WDRO
TRANS-GLM
ARIZONA
22.43
7.54
8.52
GEORGIA
46.60
27.22
24.89
ILLINOIS
20.02
8.78
15.49
MICHIGAN
43.43
25.23
24.79
Belloni, A., Chernozhukov, V., and Wang, L. Square- root lasso: Pivotal recovery of sparse signals via conic programming. Biometrika, 98(4):791-806, 2011. doi:
MINNESOTA
38.67
23.42
27.63
MISSISSIPPI
34.31
14.99
16.75
N. CAROLINA
41.48
19.02
18.56
10. 10.10993/bimmetasr04...
VIRGINIA
64.84
21.20
22.84
OVERALL
311.77
147.39
159.49
Blanchet, J. and Murthy, K. Quantifying distributional model risk via optimal transport. Mathematics of Opera- tions Research, 44(2):565-600, 2019.
4.4. A Real Data Analysis
To demonstrate the practical applicability of our KG-WDRO framework, evaluate it the Tra (Tian & Feng, 2023) dataset, which compiles 2020 U.S. presidential elec- tion results at the county level (see their references for data sources). Each county is labeled as '1' if the Democratic candidate won, and '0' otherwise. We compare KG-WDRO with Trans-GL.M on a binary classification task, where the
Blanchet, J., Kang, Y., and Murthy, K. Robust wasserstein profile inference and applications to machine learning. Journal of Applied Probability, 56(3):830-83) 2019a.
Blanchet, J., Kang, Y., Murthy, K., and Zhang, Data- driven optimal transport cost selection for distributionally robust optimization. In 2019 Winter Simulation Con-

---

Knowledge-Guided WDRO
ference (WSC), pp. 3740-3751, 2019b. doi: 10.1109) BSS44000.22019.900
Luenberger, D. G. and Ye. Y. Linear and Nonlinear Pro- gramming. International Series in Operations Research & Management Science. Springer New York, NY, 2008.
Blanchet, J., Murthy, k., and Nguyen, V. A. Statistical anal- ysis of wasserstein distributionally robust estimators. In Tutorials in Operations Research: Emerging optimization methods and modeling techniques with applications. pp- 227-254. INFORMS, 2021.
Mohajerin Esfahani, and Kuhn, D. Data-driven distribu- tionally robust optimization using the wasserstein met- ric: Performance guarantees and tractable reformulations. Mathematical Programming, 171(1) 2018.
Blanchet, J., Murthy, K., and Si, N. Confidence regions in wasserstein distributionally robust estimation. Biometrika, 109(2):295-315, June 2022.
Rockafellar, R. T. Convex Analysis. Princeton University Press, Princeton, NJJ, 1970. ISBN 978 7781400873173.
Shaficezadeb-Abadchh, S., Esfahani, P. M., and Kuhn, D. Distributionally robust logistic regression. In Advances Neural Information Processing Systems, volume 28, 2015.
Cai, T., Li, M., and Liu, M. Semi-supervised triply ro- bust inductive transfer learning. Journal of the American Statistical Association, pp. 1-14, 2024.
Gao, Finite-sample guarantees for wasserstein distri- butionally robust optimization: Breaking the curse of dimensionality. Operations Research, 71(6):291-23066 2023.
Shaficezadeh-Abadee S., Kuhn, D., and Esfahani, P. M. Regularization via mass transportation. Journal of Ma- chine Learning Research, 20(103):1-68, 2019.
Tian, Y. and Feng, Y. Transfer learning under high- dimensional generalized linear models. Journal of the American Statistical Association, 1186544):684-2669, 2023.
Gao, R. and Kleywegt, Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research, 48(2):603 655, 2023.
Gao, R., Chen, X., and Kleywegt, A. J. Wasserstein distribu- tionally robust optimization and variation regularization. Operations Research, 72(3):1 177-1191, 2022.
Toland, J. F. Duality in nonconvex optimization. Journal of Mathematical Analysis and Applications, 66(2):399 1978.
Gu, T., Han, Y., and Duan, R. Robust angle- based transfer learning in high dimensions. Journal of the Royal Statis- tical Society Series B: Statistical Methodology, 12 2024. ISSN 1369-7412.
Toland, J. F. A duality principle for non-convex optimisa- tion and the calculus of variations. Archive for Rational Mechanics and Analysis, 71:41-61 61, 1979.
Villani, C. Optimal Transport: Old and New, volume
He, Z., Sun, Y., and Li, R. Transfusion: Covariate-shiff robust transfer learning for high-dimensional regression. In International Conference on Artificial Intelligence and Statistics, pp. 703-711. P....R 2024.
338 of Grundlehren der mathematischen Wissenschaften. Springer Berlin, Heidelberg, 1 edition, 2009.
Zhou, D., Li, M., Cai, T., and Liu, M. Model-assisted and knowledge-guide transfer regression for the underrep- resented population. arXiv preprint arXiv: 2410.06484, 2024.
Kuhn, D., Shafiee, S., and Wiesemann, W. Distributionally robust optimization, arXiv preprint arXiv: 241 2411.02549, 2024.
Li, S., Cai, T. T., and Li, H. Transfer learning for high- dimensional linear regression: Prediction, estimation and minimax optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):149-17 11 2021. ISSN 1369-7412.
Lin, Z., Zhao, J., Wang, F., and Wang, H. Profiled trans- fer learning for high dimensional linear model. arXiv preprint ar Www::2406.00770
2024.
Liu, J., Wang, T., Cui, P., and Namkoong, H. Rethinking distribution shifts: Empirical analysis and inductive mod- eling for tabular data. arXiv preprint arXiv: 2307.05284, 2024.
10

---

Knowledge-Guided WDRO
A. Additional Details in Numerical Results
This section provides details to supplement Section 4. outline the data-generating distributions for all three sets of experiments, the hyperparameter grids, and the learnrr used to identify prior knowledge. We present the exact numerical results for all three sets of experiments. Recall that the notation 3, which works rescaling the magnitude of | such that 3 s3. The notation d is the dimension of the covariates, and N the sample size. Finally, the symbol represents the correlation between the true B and the prior 0.
(0, 1] represents the signal strength of the true parameter
A.1. Simulation Results
A.1.1. SIMULATION 1: LOGISTIC REGRESSION
SETTING
= 0.3
= 0.5
0.7
0.8
P 0.9
p 0.95
WDRO
KG-WDRO 0.587
0.647
0.714
0.748
0.794
0.817
0.565
N
20
TRANS- GLS
0.585
0,641
0.702
0.735
0.778
0.800
=
KG-WDRO
0.586
0.647
0.713
0.751
0.797
0.823
0.619
N
50
TRANS-GLL GL-
0.586
0.645
0.710
0.752
0.792
0.823
=
KG-WDRO
0.583
0.646
0.713
0.751
0.798
0.823
0.654
N 80
TRANS-GIM
0.584
0.646
0.714
0.755
0.800
0.826
0.5
KG-WDRO 0.581
0.634
0.690
0.721
0.762
0.787
0.549
N 20
TRANS-GLM 0.579
0.626
0.674
0.708
0.748
0.760
0.5
KG-WDRO 0.580
0.635
0.689
0.728
0.768
0.794
0.588
N 50
TRANS-GLM,
0.579
0.633
0.693
0.723
0.769
0.789
0.5
KG-WDRO
0.581
0.637
0.700
0.732
0.775
0.790
0.617
N 80
TRANS TRANS-GLM GLM
0.581
0.638
0.702
0.737
0.779
0.799
Table 3. Out-of-sample classification accuracies for Simulation 4.1, comparing KG-WDRO, Trans-GLan and WDRO across six settings with varying values of p.
A.1.2. SIMULATION 2: LINEAR REGRESSION
SETTING
P 0.3
P = 0.5
P 0.7
0.8
P 0.9
P 0.95
WDRO
1 50
KG-WDRO (WEAK)
0.585
0.645
0.740
0.801
0.870
0.912
0.108
KG-WDRO (STRONG)
0.583
0.646
0.741
0.800
0.871
0.910
PRDGE-RIDGE
0.391
0.548
0.706
0,786
0.870
0.915
KG-WDRO (WEAK)
0.707
0.745
0.803
0.843
0.894
0.924
0.513
N = 70
KG-WDRO (STRONG)
0.704
0.743
0.803
0.842
0,892
0,923
TRANS-RIDGE
0.599
0.692
0.788
0.838
0.893
0.925
$ = N 90
KG-WDRG (WEAK)
0.806
0.827
0.859
0,881
0.911
0,932
0.758
KG-WDRO (STRONG)
0.804
0.825
0.857
0.880
0.910
0.930
TRANS-RIDGE
0.762
0.802
0.849
0.877
0.910
0.932
0.8
KG-WDRO (WEAK)
0.563
0.621
0.716
0.777
0.850
0.894
0.030
N 50
KG-WDRO (STRONG)
0.561
0.622
0.716
0.777
0.849
0.892
TRANS-RIDGE
0.213
0.405
0,600
0.700
0.803
0.858
0.8
KG-WDRO (WEAK)
0.673
0.713
0.774
0.818
0.872
0.905
0.361
70
KG-WDRO (STRONG)
0.670
0.710
0.774
0.816
0.869
0.903
TRRAN-RIDGE
0,470
0.585
0.704
0.768
0.837
0.875
8 0.8
KG-WDRO (WEAK)
0.768
0.791
0.826
0.851
0.886
0.911
0.703
N = 90
CD-W (STRONG)
0.765
0.788
0.825
0.851
0,885
0.909
TRANS-RIDGE
0.671
0.724
0.785
0.821
0.863
0.890
Table Out-of-sample R² for Simulation 4.2, comparing KG-WDRO (Strong), KG-WDRO (Weak), Trans-Ridge, and WDRO across six settings with varying values of

---

Knowledge-Guided WDRO
A. 1.3. SIMULATION 3: MULTI-SITES
Here, recall that the notation & denote the correlation of generating the three prior knowledge under the scheme (3).
SETTING
p 0.7
0.75
p 0.8
/
0.85
p 0.9
p 0.95
WDRO
[1. -.5, 0.2]
KG-WDRO
0.560
0.640
0.713
0.783
0.850
0.916
-0,584
Q 0.9, N 50
TRANS-LAS LASSO
0.578
0.625
0.673
0.723
0.767
0.815
[1. -0.5, 0.2]
KG-WDRO
0.674
0.728
0.776
0.825
0.875
0.926
0.027
Q 0.9, 60
LASSO
0.666
0.697
0.732
0.770
0.808
0.850
[1, 01.50.5,.2 0.2]
KG-WDRO
0.793
0.820
0.848
0.878
0.907
0.939
0.375
2 N 70
LASSO
0.756
0.779
0.805
0.832
0.857
0.882
1,
KG-WDRO
0.565
0.642
0.715
0.785
0.852
0.916
-2.837
0 0.6, N 50
TRANS-LASSO
0.628
0.680
0.735
0.790
0.838
0.889
1,
KG-WDRO
0.673
0.729
0.778
0.829
0.877
0.928
-0.015
0 0.6, N 60
TRANS-LASSO
0.708
0.744
0.786
0.826
0.863
0.902
1,
KG-WDRO
0.797
0.825
0.852
0.880
0.911
0.942
0.354
0 0.6, N 70
TRASO-LASSO
0.794
0.820
0.844
0.868
0.894
0.919
Table 5. Out-of-sample R² for Simulation 4.3, comparing KG-WDRO, Trans-Lasso, and WDRO across six settings with varying values af p.
A.2. Simulation Setup
Let Ber(p) denote bernoulli distribution with probability parameter p. U[a b] denote uniform distribution supported on [a, b). and ((H, o?) denote a univariate normal distribution with mean j and variance o².
A.2.1. SIMULATION 1: LOGISTIC REGRESSION
In this simulation, the coefficients are generated in high-dimensional sparse setting. The dimension of the nonzero components is set to 50, which is then augmented with 100 components introduce sparsity. The nonzero components of the true coefficient-prior pair (8, 0) are generated using the multivariate normal scheme in (3), with component variance ²² 0.4 and {0.3, 0.5, 0.7, 0.8, 0.9, 0.95}. The target labels are generated as Y Ytarget ~ Ber (1/(1 + exp (-6" X)), and the source labels are generated as Ysource ~ Ber (1/(1 + exp (-0" X)), where X ~ -1[122 2|150 150 The sample size N for (aargett Yiarget) is varied across {20,50 50, 80}, while the sample size for the source data (X (Xsource, Ysource) is fixed at 800. Each dataset is paired with a validation set of the same size for hyperparameter selection.
Let grid; denote a hyperparameter grid ranging from 0.0001 to with 10 log-spaced values, and let grid2 denote a hyperparameter grid ranging from 0.0001 to 2 with 20 log-spaced values. The ÁWDRO estimator is learned by selecting the best-performing hyperparameter
grid, using validation data. For the A-Trans-GLa GLMMn learner (Tian & Feng, 2023, Algorithm 1), the transferring step is optimized using grid1, and the debiasing step is optimized using grid2. For the KG-WDRO learner 8KG proposed in Theorem 3.6 with = 1, the prior 0 is first learned from the source data using the vanilla WDRO method on grid,, followed by learning 8KG on grid, with the learned AWDRO as input.
The simulations are conducted on the parameter grid N (20,50) 50, 80) x p (0.3, with each configuration repeated 100 times. The average results are reported.
0.5, 0.7 0,7,0,8,,,8,0888 0.9, 0.95) x (0.5, 1),
A.2.2. SIMULATION 2: LINEAR REGRESSION
In this simulation, the coefficients are generated in high-dimensional correlated setting. The dimension of the coefficients is set to 100 and the components of the true coefficient-priic pair (B, 0) are generated using the multivariate normal scheme in (3), with component variance x² = 0.1 and E {0.3, 0.5, 0.7, 0.8, 0.9, 0.95}. The target labels are generated as Yearget N X, vO.5), and the source labels are generated as Ysource ~ AO(O X, v0.5), where X WN 2) with
ji=j,
Σij
for all i,j 1,2, ..., 100.
0.3
if $
12

---

Knowledge-Guided WDRO
The sample size N for (Xtagget> Yearget) is varied across (50, 70, 90), while the sample size for the source data (Xsorcee; Ssource) is fixed at 800. Each dataset is paired with a validation set of the same size for hyperparameter se- lection.
Let grid, denote a hyperparameter grid ranging from 0.0001 to 1 with 10 log-spaced values, and let grid, denote a hyperparameter grid ranging from 0.0001 to 1.5 with 20 log-spaced values. The ÁWDRO estimator is learned by selecting the best-perforning hyperparameter on grid, using validation data. For the Trans-Ridge learner adapted from (Li et al., 2021, Algorithm 1), the transferring step is optimized using grid,, and the debiasing step is optimized using gridg. For the KG-WDRO learner 5KGatrong proposed in Theorem 3.3 with p = 2, and the 3KGweak learner proposed in Theorem 3.5, the prior 0 is first learned from the source data using the vanilla WDRO method on grid1, followed by learning fKGGstrong on grid2 with the learned OWDRO as input. The X-1 grid for BKe is 0.0001 to 8 with 20 log- values.
The simulations are conducted on the parameter grid N E {50, 70, 90} x p E {0.3, 0.5, 0.7, 0.8, 0.9,0.95) x s S {0.8, with each configuration repeated 100 times. The average results are reported.
A.2.3. SIMULATION 3: MULTIPLE SITES
In this simulation, the coefficients are generated in nigh-dimensional sparse setting. The dimension of the nonzero components is set to 50, which is then augmented with 100 zero components to introduce sparsity. The number of external source is 3, we generate the their coefficients 0,, 02 03 using the scheme (3). We construct a linear combination, 0s = a&1 + b02 + c0s, and generate the target coefficient 3 = pds + E, where & ~ N(0, (1 (ar(ss)), ensuring Corr(3, 0s) = p. The target coefficient ,3 is then scaled to match the magnitude of 0s-
The target labels are generated as Yiarget ~ -N(B"X X, vo.5), and the source labels are generated as Ysource, N( X, vO.5) for m E [3], where X ~ ~N(0,, with
[1
if j, for all i,j
Ea
0.1
if H
The sample size for the target data ranges in (50, 60, 70}.
Let grid; denote a hyperparameter grid ranging from 0.0001 to with 15 log-spaced values, and let gridy denote a hyperparameter grid ranging from 0.0001 to 3 with 20 log-spaced values. The ÅWDRO estimator is learned by selecting the best-performing hyperparameter on grid, using validation data. For the oracle Trans-Lasso learner (Li al., 2021, Algorithm 1), the transferring step is optimized using grid,, and the debiasing step is optimized using grid2 using all three source data. For the KG-WDRC learner fKG proposed in Theorem 3.3 with 1. the priors 01, 02, 03 are first learned from the three source data using the vanilla WDRO method on grid1, followed by learning fKG grid2 with the learned 0,, WDRO, O2 WDRO 03, WDRO as input.
The simulations are conducted on the parameter grid N e (50, 60, 70} x p E {0.3, 0.5, 0.7, 0.8, 0.9, 0.95] x [a,b,c] e {1, 0.5, 0.2), (1, 1,),, with each configuration repeated 100 times. The average results are reported.
B. Proof of Results in Regression.
Proof of Proposition 3.1. This result follows from the observation that, under the constraint imposed by 02 for any p# E (. Ni (2) and any a E R, the marginal distributions of (Y,a0T) X) must agree under Py and P## That is,
(N a0'x) E A B) = 1 ((Y, a00xxxxx" E A x B)
for all Borel measurable sets A, B C R. Consequently. for any F# E BB (P'::
N: 22 we have
EI [(Y - =00xxxx) Ep# [(Y a0 x)
then choosing 3 = at, we have
sup Ep [(V 1003-3x'" x1x = sup inf Ep [(Y - 38t x) SER (Fw)
inf
(P BERd
S sup Ep [(Y a0 x) P:Bs(PN)
EFN [[Y a0 x)²],
13

---

Knowledge-Guided WDRO
where the first equality invoked the Minimax Theorem of WDRO (Blanchet al., 2019a, Lemma 1). Taking infimum over completes the proof.
Lemma B.1. Let Ja Re R be defined A TR (3*))² 2r(3)3rA depending on some I&d and let r(3) be
non-negative real-valued function in 3. Then the convex conjugate ( Fe R is given by
(3A"
2r (r)
if E span B,
13(47)
+00
otherwise.
Therefore the biconjugate (A): Rd + of Sz(A) has represeniation:
(a + 2r))
(2) sup oeR
(((FA))
Proof. The convex conjugate (A*) is defined as
f(() = sup
(A - ( +2(3))))++((3(()
AER"
where A", E IRd and r(3) E IR are taken as fixed values. Orthogonalize A aB + in the direction of 3 with a E IR and w E Rd such that fTw 0. Then, we have A A = aA BT + A w,,, and the convex conjugate becomes
('A)) sup (a(A))) 3) +
a 21113112 2ar 2ar (3)11113)
a.
S.t w
Fixing w. the objective is a negative quadratic function in a, hence the objective in a bounded from above by a finite value. Now, if A is not orthogonal to w, the term sup,, A T is unbounded, and the convex conjugate f* (*()) = +00. A orthogonal w, then the convex conjugate attains finite value. Note that A T 0 A span 3. Hence condition on { aB: a R}, we have
(AA") sup (a(A*))) a² 2لاا + 2r(3)a|18113
(A 2r(3)111113
1
where the optimal solution a' + 2r(
A
2| and the coefficient a given by the projection scalar a
||311
The biconjuagte
(A) = sup (A'A* f*(A)) A-
therefore bounded from below if and only if A E span 3. Let A ag for some R, then substituting we get the representation.
(a + 2+)3111111)
(*)(A) = sup
A (T
a
(a + 2r(3))2
sup
<(AT 3)
4
can be readily verified that (*)(A) í(A) as promised by the Fenchel-Morent Theorem (Theorem E.4).
Lemma B.2. Let go (A) IRd + R. be defined as A E Rdd given by
10 for some 0 Rd Then the convex conjugate gā (A*) )) is
if = at and |a] 1
96 (Ar)
+ 00
otherwise.
14

---

Knowledge-Guided WDRO
Therefore the convex conjugate of the funcctinn gl () = Em 90.. (A) for some is given by
f = Em On and 10mml S y for each m,
9" 0~(~~) = +00
otherwise.
Proof. The convex conjugate is defined
g = sup (A [A*A A 10'A)),
A |0||
again, orthogonalize A = ab + w, where a
and f'w = 0. Now by the change of variable u == A'A, the convex
conjugate is now
_0(4)) = sup
T + - - -lu)) |
s.t @'w = O.
Thus the convex conjugate (-*) +o if A & span 0. If A of for some R, then
gā g6 (c00 = sup |101 0: 1=110112 [ea]
= sup (au - |u])
if |o| 1,
too otherwise,
where the last equality holds by noting that sup, au - |u| = |. (a) is the convex conjugate of the absolute value function
(Proposition E.2). This proofs the convex conjugate of (A*). Now g(A) = >E 90.. (4) =Y](A), the convex conjugate of g(A) is
('(A) (ge, +... g0 (A²)
inf (gh, (At) .... + gos. (Air) s.t Ai+ =
where the second line follows from the infimal convolution property of sum of convex conjugates (Theorem E.5). We know that g" is finite if and only if oo... (Ar,) for all m E [M], that is A = am for some O'm E [-1, 1] for all E [M]. M
Hence q" (A") 0 if and only if A Em=1 Om,.0. and O. E [-1,1] for all E [M]. Finally
calculate the
convex conjugate (- ("gg)((>))))) = "9" (A)
by the scaling law of convex conjugates (Proposition E.3) given
> 0. This concludes the proof.
We now give the proof to Theorem 3.3.
Proof of Theorem 3.3. Let r(3) = - x. Then first consider the cost function
(2((2,)) v), (u, v)) = |a mll + xo by 2| d(0)a: du + +... d(0 O).
where we replaced the transferring strength from +oo valued distance function d(x) IR R that is monotone function in (z]. with d(0) = O. We will then let d(x) except at z 0. Then the supremum function
(2,33:
sup (v.4)
{e(u, 8) - (o((u, v), (2,8)))
is finite if and only if y. Then, we have
(u, nz (3) yc ((u, v). (xry))
(v - 3Tu)²² > ||x -u11, 7d (0 0J 7d(0)/x 0h5
15

---

Knowledge-Guided WDRO
Denote by A x; get
(u, ( (x.
=r(3)²+ - -a00 (..(())))..--((((
2r(3) sup (fa(A)- -(A)} A
Consider the objective in A
sup [(3 2r - -... - yd(0)
A
where we let (a) = (8' 2r(3)8"A A and ((4) == ~ + 7d(O]A) +... + yd(0%/) This is a convex + concave optimization, we express the convex part of fa(A) as supremum of infinitely many affine functions. Then by Lemma B.I, we have fa( = (A) = SUPEER (a(3TA) (a + +rr(3))
then we may write
4
sup (fz(A) - 9(4)}
(a(3T)) (a + 2r(3))²
sup A
sup oER
4
(a)(3 -((A)) (a+ 2r(3))
sup
sup A
ga(33 (a+ 2r(3))
sup
(Toland's Duality)
( (A), with gi (A) = 7|, and go(A) = >m d(O), m A). Then we can compute the convex conjugate of g using the infimal convolution property (Theorem E.5). Then
where 9" is the convex conjugate of g. Let ( = g1 (A) + go
g (1))
inf A1+
(0i(At) g6 (b2)).
We know that (i() = 4y by Lemma B.2, we have,
|il where p-1 + 9- (Proposition E.2). Now suppose d(x)) = A/ze| for some À > 0,
0
if -M Em= fm and |aml $ > for each m,
( =
+00
otherwise.
Then the convex conjugate g"(A*) is
9" (A() = inf yi (A- AA), A
M
s.t A
Σ
am fm and aamml S 7. for each m,
which is equivalently,
1 inf A+ Σ an Om
(A')
4y
s.t 1aml S 7 for each m.
Letting À - oc, we recover the cost function 02,x0, and when X - oc, each Omm is now free in TR. Then we have 1 (A"A") 4y
infaee || 1---0122 O1ß with θ = span (01, ........ Axs the validity of this tactic follows from (Luenberger & Ye,
16

---

Knowledge-Guided WDRO
1
2008, Theorem 1, Section 13.1). Then have g" (a:3)
inf ||, .1IE Suppose o 0, then dividing by a, we
47
get
g (a = a²
inf
||3 - 016
4y aee
1 - inf || = 0, the representation 9 (a.3) 47
a² inf |13 -lp is valid for all
If 0, then - (0))
47
a R. Therefore following the proof of (Blanchet et al., 2019a, Theorem 1),
= 3)
+ sup
1
a²
inf DEB
|| - 0|6 - (a - (7)))
4
7
inff ||B D|P
1 1) a² -2--arl (0r(3))a
sup
4 a
r(3)2
if inf v 113 0112 S 7
1
||3 in
01?
t00
otherwise.
Then the minimization objective can be simplified as
inf SER"
min
76
+
(T): vi
720
n
1
N
r:(3)
inf
inf
76
8 12
7 1=1
7 inf / ||3 U13
7
inf
inf
76 + MSE (3)
8 >2inf
7 inf || - 013
inf B
(SS((3) + vo ||33 - 011p)
0
where the last equality follows because 76 + MSE (3)
7 inf 8 02 is convex function in that tends to -o0
approaching the boundaries inf, ||B - 0| and +o,, so the optimization over y can be solved using first-order condition. Then by Proposition 2.1, strong duality holds and,
inf
sup P.D. (5.
Ep [(Y
B xT inf (VMSE(B) + vō113 +V80)+++++1++0000 V0113-011, 011,)
This reduces the dimenti-ai optimization fintee-dimensional problem, where we interchanged inf, and the quadratic function, since the quadratic function is monotone increasing on the positive reals.
The next proof is Theorem 3.5 with the weak transferring cost function 02.1 (( v), (u, b)) = || - ul12 + A(0 OTu)² + ly - v| with some À > 0. The statements generalizes to multi-sites by first considering orthogonalizing the prior knowledge (01....,xx) OMr).
Proof of Theorem 3.5. Following the proof of Theorem 3.3, we solve the optimization problem
sup
- T/ - >>(0"
AERd
where we recall that r is the dual-variable in statement of Proposition 2.1, À > is the transferring strength, 0 E Rd is the prior knowledge, and r(8) = (y - x² the residual in 8.
17

---

Knowledge-Guided WDRO
Then let O be orthogonal matrix, whose first column is 0/||0|12- then use Á 0-1A. The objective function now becomes
- > >111
where the last term follows because 0 = (110122.0) 0. O......),, 0), and A denotes the first component of A. Now define
D = diag
1ll
and consider the change of variable A = DĀ, then the last two terms become
A
A 2 + A110 |
11111
Σ
A -
1
1lll
Therefore, the objective becomes
sup (-0))))000-00))00-----00000-0000 ~111
sup 111101-1111111 2r 1| '1a1/1
711
sup ((118ll, - 1 2r(8)11B1lv, (
r(3))118113
which has finite optimal value
whenever || v with 1 denoting the positive-definite symmetric matrix,
7 |1
V la
00T,
1001
that is independent of the choice of O. The first equality follows because we applied Cauchy-Schward inequality and since A e RRd is free, there some Å that achieves equality. The rest of the proof follows exactly along the proof of Theorem 3.3 by completing the optimization over the dual problem using Proposition 2.1.
C. Proof of Results in Classification.
Lemma C.1. Consider the convex function hz(x): Id + IR by E IRd + log (1 + exp (-) for some > 0 and x' E R. Then for every y > O, the constraint optimization problem Hx(x)) defined as,
sup ho(x) - -y|12' - =x1la-
FER"
s.t
O3 2) = 0,
has optimal objective value,
ho(a') if inffeeul18 k011p S % +00
Hg(x')
otherwise,
where p. q (1) with p- + -1
This lemma is a simple extension of (Shaficezadeh--baadhh et al., 2015, Lemma I), Following their proof, it is shown that
Proof.
ha(x) = h17 (a) sup
x (a)),,
05a51
where
aolog (a) + a) log a-a) if E [0, 1], otherwise,
h (a)
+00
18

---

Knowledge-Guided WDRO
the convex conjugate of the function log (1 + e**) (Proposition E.2). Then it is shown that the objective Ha must has representation
sup inf sup((a;3 + 1)'=' x- (0) - x²))), 0SoSi Sy
s.t f (r x²) = 0.
Fixing a and g, then the inner maximization in
sup((a3 + q)'xx x''''''²');
0' x') - 0,
has solution Ta subject to a3 +q 1 for some E R derived using the first-order condition of the Lagrangian duality or +00 otherwise. Then condition on {a. + pull E RR,, the objective has representation
Ha(x')
sup inf (la,3)' he (a)) s.t q = p a3
05a S4
sup inf x' -
05agi
Consider the constraint 1 - aBl1p $ y over μ.. Suppose a > 0, then dividing by -a, we get the equivalent constraint - 0
|a|| -
|1,} a
P
& y over μ. Defining the change of variable x = then since the Lagrange multiplier μ E IR is
free, we have x free, and the constraint becomes inf, |a|l| k0 p S y over K R. If a 0, then inf I| - 0l
0 = 0. ..inff ||B3 rxllpp. So the equivalent constraint inf, |a|]]|8 k0Ipp Ip y is valid for all [0, 1]. Then condition {a + g pQlp E RR, the objective becomes,
Ha(x') = sup (la.3)'x' (00)))'-hh'(0 -h((a)) s.t sup |a| inf ||
k@l1, - n
uSas1
05a
(la3) - (a)) s.t inf ||3 kdIp S 7
sup 0Sa5]
Recognizing that
((a3)'x' -h'')))))))))))))))
sup a(33x') [i*(()) h: (3'='' h osas1
sup 0sasl
we get
Hz(x')= = (hg(x") if inf |13--00) 7
+00 otherwise.
The above Lemma C.1 is easily generalized to incorporate multiple orthogonality constraints {0T, (a' - x) = 0:m E [M] using the exact same Lagrangian formulation. Again, recall span (0,,..., O. Thus the optimal objective value under multiple constraints becomes
Hs(x') = (hx(')) if inf ||3 01pp S 7,
+00 otherwise.
We now give the proof to Theorem 3.6.
Proof of Theorem 3.6 Logistic Loss. Using Proposition 2.1, apply the strong duality, and consider the inner opti- mization problem
= inf 20 { 77 + ΣN suPaer (log (1+ +e e-visTu) >/10, -ulle) vdll)
log (1+
sup (5
Ep
n
s.t AT (x; u) = for all m E [M] and E [N].
19

---

Knowledge-Guided WDRO
For each i [N], apply Lemma C.1 the maximization problem,
supuERd (log + ====B=)) - ~1122 - ulla)
Ha(
s.t 0 (2z u) = 0, for all m E [M.]
which has solution
log 1 (1 if inf ||3 - f1p S 7.
+00
otherwise.
Therefore, the maximization problem supe:De, ( Ee [oog (1 (1+e-rs'x is bounded from above if and only if y 2 info || -l|pp Under this condition, this reduces the inner optimization problem.
EP log (1 + (1+
70 Σ log 1 + =uoffr))
sup o0 (.)
inf
2int [18-
(1+e + dinl || - U11.-
log
7
0
This concludes the proof.
We now give the proof the maximum margin classifier using the hinge loss.
Proof of Theorem 3.6 for Hinge Loss. As in the case to the proof of Theorem 3.3, we first consider the relaxed cost function
M 8m u/.
c1 (xx (u, v)) = || ull4 + oc. 1--u vl + A
Σ
where we relaxed the transferring strength from +oo to some finite value À >> strong duality, we can solve the worst case hinge loss by solving the dual problem
O.
We will then let A + t00. Again, by
sup (1 u²++ 711u xilla 7d
Σ 10 (x: wi) }
inf
70
+
Let A
xr, then have
sup (1- u)² >/||u- -111 -> Σ 10 (xc x)1
M
sup
(1 - ( + x))+ - ~11 - 7>
0T Al
M
= sup sup A 05e51
a(1 - T x)) >11 7. Σ
10T |
m=1
sup sup 05a51
-oyj3 A ~1| -
0
A + a(1 y13xx))
A
m==1
Where in the second equality we used x suPososi O.. Fixing a, consider the inner minimization A,
sup
aays A >lllll - - Σ 0 A1
9" 00(-ay8))
20

---

Knowledge-Guided WDRO
I1 gi(A1) and >d 2m 10 A2] =: 92(22), then by the infimal convolution property of convex conjugates (Theorem E.5), we know that
where (²(" is the convex conjugate of g( >الاا| 7 Em=1 O0 Al.
M
Set >|1
g'(4") A1+A3=A- (gī (45))
inf
From Lemma B.2, we know that if g (A"") is finite, then 95 subject to A Emmi amom and |aml S xy for all m E [M]. Now it is well known that (Proposition E.2).
( = (7ll (At) - - [[AAT)>>>>
where (x) denotes the convex indicator on the set C., Therefore, letting X + ce, the constraints {|aml S Aa/m E [M]] redundant, and we have
0
if infaee - - 011 S 7.
9" (A7))
+00 otherwise,
where let θ == span (01,-... 0r Therefore, 99" (-oyi) is finite if and only if inf oy- =lIp S Now y +1, remove y, and this leaves the condition that inf, |a, - d0Ipp S 7. which is equivalent to inf ||3 UlIp S for all E (0, 1]. including
O. Taking supremum over [0, 1]. the final condition is info || 0lpp 7 Therefore, assuming the dual problem is bounded from above, it reduces as
sup sup 05a51 A
-ayyB' >|l 7d
Σ
0 Al + aa(1 y 2)²xx)
m=l
sup usa51
(I(into [| a(1 y)²²x))
=(1 - y3*x)*
given
inf || - 1 l S 7
Finally, the dual form of the distributionally robust optimization problem is #
N
M
inf inf
76
sup
(1 - v: u²+ 7//uu- xillo yy
Σ
10 (x: w)]
720
i=1
mmm
1 Σ (1- yi:3zz.) xz1+
inf
inf
70
72info
1| 1130011>
" 1=1
inf
31 xx)+ + 8||3
011p
8,0
This completes the proof.
D. Proof of Results in Mahalanobis Norm Regularization
Proof of Corollary 3.7. This is direct consequence of the convex conjugate of || 1 given in Proposition E.6.
Define the cost function o ((x, y). (u, v)) = |æ ulla + 30 ly
ΣM 10 x 0 u/.
Corollary D.1 (Theorem 3.6). Suppose the loss function 8(X, 3) is either the logistic loss log (1+E- or the
hinge loss (1 - x3*X)* then for any A E saxd we have
inf BERd
sup P.B. rett
Ep [[(X,;;
N
inf BER JEO
Σ N
1
(x, ye B) + 611B - 0lIA
Proof. For the logistic loss case, this is direct consequence of the dual norm of ||œ]]| for the hinge loss case this is a direct consequence of the convex conjugate of || Both given by Proposition E.6.
21

---

Knowledge-Guided WDRO
E. Useful Results Convex Conjugation
this section we review some results on concept of convex conjugates that repeatedly come up in the proofs. For more details on convex conjugations, the interested readers can consult (Rockafellar, 1970, Section 12 16).
Definition E.1 (Convex Conjugate). Let f IR + be a real-valued function on the Fuclidean space, then the convex conjugate of f is the function f IR + that evaluates x= E IR"" by
(x")
sup xEdom (f)
(x"xx- (-x)).
This is also called the Legendre transformation of f, and the Legendre-Fenche transformation for f defined on arbitrary real topological vector spaces, Here we collect some examples of convex conjugates that appeared in the appendix. These are well-known.
1 + =1
Proposition E.2. Let p.q 2 such that
p
1. The convex conjugate of the absolute value function f(x) |a:| on IR is given by (11" = -s (x*), the convex indicator function the set {|=*| < 1|x² E RJ..
2. The convex conjugate of the q-norm |alll on TR& is given by || [1]()*)) I1 11, 21,") the convex indicator function on the set {||=" II 2 1 1xx E R...
3. The convex conjugate of 1 =1lzlla on Id is given by 11 (x*)
1 -
2
4. The convex conjugate log (1 + e) on given by
x log (x*) + (1 x²) log (1 x=x) if (0,1)
ifx" = 0,
+00
otherwise.
Another easy consequence from the definition of convex conjugation is the below scaling laws.
Proposition E.3 (Scaling Laws). Let f (x*) ) be the convex conjugate of f(x) on Rd. Then we have,
1. the convex conjugate of (ax) whenever + given by f f*1x"
2. the conver conjugate of a (x) whenever a given by (f*(x") /u).
Let I' (RR) denote the class of proper convex lower-semi continuous functions on Fd. the next statement says that this conjugation induces an one-to-one symmetric correspondence on the class (R+). It is a cornerstone modern convex analysis and used in the proof of Theorem 3.3 and Lemma C.I.
Theorem E.4 (Fenchel-Moreau)) Let f be a proper convex, lower semi-continuous function on Rd then
1. the convex conjugation f f a bijection I (RR);
2. ʃ (ET(RC)
- 1 -.
Proof. For a proof please consult (Rockafellar, 1970, Section 12).
The next statement concerns the commutativity of convex conjugation and function summation. Its usefulness profound, and applied the proof of Theorem 3.3 and Theorem 3.6.
22

---

Knowledge-Guided WDRO
Theorem E.5 (Infimal Convolution Property of Convex Conjugation). Let s JM be proper convex functions on R. then
= +.. f..
and
fM) inf (M. af + + 2iM x²).
Proof. For a proof please consult (Rockafellar, 1970, Theorem 16.4).
Proposition E.6. Let A E Stx,, then the dual norm of ||=||| is | The Cauchy-Schware inequality x²u S ||lla|/||A holds, and equality is attainable. The convex conjugate of | is given by A| ll (I(")), and the convex conjugate of | is given by ||=" 1//1/44.
Proof. The dual norm of || Il, the Cauchy-Schwart inequality and attainability of equality follows from (Blanchet et al., 2019b, Lemma 1). Now to compute the convex conjugate of || 13, we want to evaluate
sup (x" x (x=1)))
aER
By the Cauchy-Schward inequality we have x'I < ||> [Lr||A| |20 llA 1, and we have
x= x - 13 z | || - -1==1
Hence
(x Tx=TZ= x | - sup(t|>="1l>> (2) = 1 112--
sup
=E
120
By attainability of equality in the Cauchy- inequality, the supremum are equal, and we have
sup (x x - [|~11² = x -1
ZERd
This proof's the convex conjugate of || R.. Now consider the convex conjugate of ||*||A, then we need to evaluate
sup
(x x ||<||x),
xER
again, by Cauchy-Schward and the attainability of equality, we have
sup
(x r - |
sup [|~~|||||~*||
wER.
xER*
sup([|=||a
[| a1a-1 1))
zER
0
if | S 1,
too
otherwise.
This completes the proof.
F. Toland's Duality
The duality theory of Toland's (Toland, 1978; 1979) concerns the minimization of nonconvex functions, in particular, applies the minimization of the difference convex functions (DC problems). The duality holds under minimal conditions, and one tries see if the DC problem can be transformed into something more manageable.
Theorem F.1 (Toland's Duality). Let f and be functions Rd, if f E I (RRd), then have
inf (f(z) g(x)} FERd
inf z' eRd
{g"(2*)) ff (x)))
Toland's ' duality is implicitly used in the proof to Theorem 3.3 and Lemma C.I which also sketches a proof to the above duality theorem.
23


Reviews:
[{"review_id": "bMM7v1lIsF", "reviewer": "Reviewer_RPYq", "summary": "The paper introduces a transfer-learning variant of Wasserstein Distributionally Robust Optimization. Given some external knowledge, which the authors represent by a vector $\\theta$, they construct an ambiguity region based on a Wasserstein distance with $\\theta$-dependent cost function. This makes the ambiguity region less pessimistic. They show that the resulting optimization problems are equivalent to several known forms of regularization.\n\n## Update after rebuttal\nI appreciate the authors response. My review remains unchanged.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the encouraging and detailed feedback. Here we list our responses to the weaknesses and questions suggested by the reviewer.\n\n**W1: Selection of $\\lambda$.** \n\nThe additional parameter $\\lambda$ is introduced to model the decision maker’s confidence in using the prior knowledge $\\theta$ as a proxy for the parameter of interest $\\beta$. Similar to how the budget constraint $\\delta$ is specified, $\\lambda$ can be selected using data-driven methods, such as grid-search cross-validation over the pair $(\\delta, \\lambda)$, as demonstrated in Simulation 2 of Section 4.2 (Line 372, Right).\n\n**Q1: Unhelpful $\\theta$ and Convergence.**\n\nLet $\\beta^*$ denote the solution to the stochastic optimization (SO) problem in the target domain (Line 125, Right). When the correlation between $\\beta^*$ and $\\theta$ is small, we can employ the weak-transferring mechanism developed in Section 3.3.2 (Line 250, Right). In such cases, the data-driven selection of $\\lambda$ is expected to yield small values, effectively reducing the KG-WDRO problem to a nearly standard DRO formulation, thereby not hampering the learning. This intuition on the positive relationship between the informativeness of the prior knowledge and the size of $\\lambda$—is demonstrated in the figure (https://figshare.com/s/05b069e330136338fa4e) through the ablation study on $\\lambda$ in Section **W3** of our rebuttal to Reviewer LRuU.\n\nFor sufficiently large datasets, following the approach in (Blanchet et al, 2022) (Line 476, Left), we can select the Wasserstein ball radius on the order of $n^{-1}$, i.e., $\\delta_n = C/n$ for some constant $C > 0$. This ensures that the KG-WDRO solution, $\\beta_{\\text{KG-DRO}}$, converges to the optimum $\\beta^*$ at an optimal rate of $O(n^{-1/2})$. Although this result has not been formally proven, we aim to establish it in future work. Notably, this $O(n^{-1/2})$ convergence rate holds for all $\\lambda \\in [0,\\infty]$, including $\\lambda=\\infty$.\n\n**Q2: Optimization and Transfer Learning when $\\lambda = \\infty$.**\n\nWhen $\\lambda = \\infty$, our formulation is equivalent to adding equality constraints on the support of the perturbation. Specifically, if we define the perturbation as $\\Delta = x - x'$ where $x’$ is the perturbed value of $x$, then the constraint $\\Delta^\\top \\theta = 0$ must hold. This does not restrict the learning problem to a constraint optimization but rather constrains the support of the perturbation. Moreover, we do not think that transfer learning contradicts optimization problems. While our goal is to perform transfer learning—leveraging prior knowledge for the target domain—our computational method is based on solving optimization problems. To draw an analogy, in deep learning, the objective might be classification, yet the method relies on minimizing loss functions through gradient descent or similar optimization techniques.", "claims_and_evidence": "The claims are supported by proofs and the approach is well-motivated.", "methods_and_evaluation_criteria": "The proposed methods are adequate.", "theoretical_claims": "I did not check the proofs.", "experimental_designs_or_analyses": "The experimental setup seems reasonable.", "relation_to_broader_scientific_literature": "WDRO is a popular optimization framework that has been related to several forms of regularization. This work introduces a WDRO-variant for transfer learning and relates it to several regularization measures.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "Strengths:\n* The paper is very well written and pleasant to read\n* The results are very elegant. The proposed criteria seem intractable, but are rewritten to more tractable-looking regularized problems.\n\nWeaknesses:\n* The proposed setup introduces an additional parameter $\\lambda$ without any guidance on how to choose this value.", "other_comments_or_suggestions": "-", "questions_for_authors": "Suppose the prior knowledge $\\theta$ is *unhelpful* for the learning problem at hand, how much does this hamper the learning? Will the method, given sufficient data, still converge to the optimum? Does this depend on whether $\\lambda=\\infty$ or not?\n\nWhen setting $\\lambda=\\infty$, we are basically just adding some constraints to the optimization problem. Can this really be considered transfer learning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GF9KREq8sf", "reviewer": "Reviewer_UVsb", "summary": "The authors believe that traditional Wasserstein Distributionally Robust Optimization (WDRO) has a conservative tendency, which can lead to suboptimal performance. They argue that in real-world scenarios, prior knowledge can be leveraged to enhance model performance and robustness. Therefore, they propose that integrating prior knowledge into the Wasserstein Distributionally Robust Optimization framework remains an open question. In this work, the authors introduce a novel framework called Knowledge-Guided Wasserstein Distributionally Robust Optimization (KG-WDRO), which utilizes external knowledge (parameters) to adjust the Wasserstein ambiguity set by constraining the transportation costs along directions indicated by the prior knowledge. The authors summarize that this strategy allows the model to concentrate uncertainty in areas where the prior knowledge is less reliable, effectively enhancing the robustness of knowledge-guided generalization.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the encouraging and detailed feedback. \n\n**W1: Explanation of Using Prior Knowledge.**\n\nOur transfer learning approach falls under *Domain Adaptation*, which adapts models trained on a source domain to perform well on a related target domain with limited labeled data.\n \nA key application is in clinical trials, where the binary outcome $Y\\in\\\\{0,1\\\\}$ indicates treatment success or failure, and the high-dimensional covariate $X$ encodes a patient’s physical and health conditions along with treatment details. Data scarcity is common—especially for underrepresented populations. To address this, we leverage a classifier trained on a majority group (parameterized by $θ$) as a reference to estimate a classifier for the minority group (parameterized by $β$). This knowledge-guided transfer learning reduces uncertainty by anchoring the search for $β$ in the direction of $θ$. We will include this example in the new draft to clarify our setting.\n\n**W2: Model Complexity**\n\nWe would like to emphasize that in our settings , due to the inherently small sample size or lack of labeled data in the target domain, it is necessary to limit the complexity of our machine learning models. For example, in clinical trials, there are usually only ~100 observations. Therefore, neural-network type methods may overfit. \n\nThe tractable reformulation we obtained for generalized linear models and SVMs provides valuable insights into how shrinkage-based or penalized estimation for transfer learning objectives should look like. This distinguishes our approach from many previous works, as discussed in Table 1. Furthermore, as we will elaborate below, the techniques underlying KG-WDRO can be directly extended to more complex machine learning models, including neural networks.\n\n**W3: Typo**\n\nThanks for catching this. We will correct the $\\inf$ to $\\min$ in the statement on (Line 182, Left).\n\n**W4: Connection to Regularization.**\n\nYes, we acknowledge that the proposed method can be interpreted as a form of regularization, and in fact, this tractable reformulation is a key contribution of our paper. It allows us to efficiently solve the infinite-dimensional minimax problem as a convex program.  \n\nThis suggests that the performance improvement is directly attributable to the integration of prior knowledge into the cost function, which translates into a regularization/penalty term that encourages collinearity with the prior. Therefore, both the prior knowledge and our method play important roles. Regarding the impact of the quality of prior knowledge, the upper two subfigures in Figure 1 demonstrate that our method improves regression mean squared error  and classification accuracy even when the correlation between the prior knowledge and the true parameter is as low as 0.3, with hyperparameters selected via cross-validation.\n\n**W5: High-dimensional Models.**\n\nOur method remains applicable in high-dimensional settings. We can apply this principle to neural networks as follows: suppose we pre-trained a multilayer perceptron (MLP) on the source. We can fix all the hidden layers and treat the output layer’s parameters as prior knowledge, denoted by $θ$. Then, on the source domain, we fine-tune the MLP’s output layer using the KG-WDRO objective, effectively re-learning the output layer on the target domain but using the previous output layer as knowledge guidance. Mathematically, suppose the pretrained MLP is represented by $f(X) = θ^\\top h(X)+b$, where $h:\\mathbb{R}^d\\to\\mathbb{R}^k$ is the nonlinear hidden layers, then the KG-WDRO framework for the MLP on the target data is to solve the optimization problem $ \\inf_{β,c,κ}\\Vert \\mathbf{y}-β^\\top h(\\mathbf{X})-c\\Vert_2+\\sqrt{\\delta} \\Vert[θ,b]-κ[β,c]\\Vert_p,$ by taking the hidden layers $h(\\cdot)$ as fixed.\n\nFollowing this, we conduct an additional experiment (https://figshare.com/s/29e12260c87f084eeb54) between KG-MLP versus naive MLP in a setup similar to Figure 2 of Main Text, which will be added to the new version of the paper. Specifically, we consider an 11-dimensional setting. The response $y$ is the sum of 5 nonlinear ($\\sin, e^{-x^2}, \\log(|x|+1), \\tanh$ and an interaction) and 5 linear basis. The linear coefficients for source and target are generated using the same method in Figure 2 for different correlations. We use a 2-layer MLP with 11 and 10 nodes in each layer. The pretrained MLP was trained on 5,000 data points, then KG-MLP was fine-tuned on 130 target points, while the naive MLP used the same data. We see that KG-MLP outperforms naive MLP consistently, especially when correlation is high.\n\n**Lack of Real Data**\n\nWe refer the reviewer to **W3** in the rebuttal to Reviewer 85zy to see the addition of the real data analysis.\n\n**$\\Delta$ on Line 802**\n\nHere $\\Delta$ is just a free variable of the $d$-dimensional space $\\mathbb{R}^d$ to represent the perturbation $x-x'$ in the ambiguity set, we will refurbish the appendix for better exposition.", "claims_and_evidence": "1. Introducing prior knowledge is a very good question, but the authors' explanation of it in the early part of the paper is quite vague.", "methods_and_evaluation_criteria": "Is this method still viable in high-dimensional knowledge scenarios, such as neural networks? Lack of more practical scenarios.", "theoretical_claims": "The theoretical framework is built on strong assumptions and has notable limitations. Additionally, the professors embrace significant hypotheses in their analyses, such as the definition of delta in Line 702. Why is delta defined in this particular manner?", "experimental_designs_or_analyses": "The numerical results presented need to incorporate more real-world scenarios for better applicability and relevance.", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Weakness:\n\n1.Introducing prior knowledge is a very good question, but the authors' explanation of it in the early part of the paper is quite vague.\n\n2. The authors use linear regression and binary classification problems to validate the proposed method. Is this appropriate? Linear regression and binary classification are not very complex problems, and there are already many methods to solve them. To validate KG-WDRO, the authors should provide problems that match its complexity.\n\n3. There is an error in the formula on line 182; \"inf\" and \"min\" represent different concepts.\n\n4. From Theorem 3.2, it appears to resemble a regularization method, making it difficult to determine from a theoretical perspective whether the improvement in performance is due to the prior knowledge or the method proposed by the authors. What is the impact of the amount of prior knowledge on performance improvement?\n\n5. Is this method still viable in high-dimensional knowledge scenarios, such as neural networks?", "other_comments_or_suggestions": "n/a", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yTP5UDZP1X", "reviewer": "Reviewer_LRuU", "summary": "This work introduces a framework for transfer learning called Knowledge-Guided Wasserstein Distributionally Robust Optimization. In face of the overly conservative property of WDRO, the proposed framework adapts the Wasserstein ambiguity set using external knowledge (augment the transport cost function with an penalty term involving with prediction discrepancy).  They establish the equivalence between KG-WDRO and shrinkage-based estimation methods, and demonstrate the effectiveness of KG-WDRO in improving small-sample transfer learning through numerical simulations.\n\n## update after rebuttal\n\nAfter reading other reviews and the rebuttals, I decide to maintain my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the encouraging and detailed feedback. \n\n**W1: Inconsistent Writings**\n\nWe acknowledge that the connection between WDRO and transfer learning may not be immediately clear. We will refine our writing to ensure a smoother and more intuitive transition between these two concepts.\n\nThe main contribution of our paper is to establish that a broad class of shrinkage-based transfer learning objectives can be equivalently formulated as a WDRO problem. This provides a distributionally robust perspective on transfer learning. The title includes the phrase “knowledge-guided”, which refers to a type of transfer learning known as *domain adaptation*-adapting models trained on a source domain to perform well on a related target domain. In our framework, the prior knowledge is represented by the model parameters learned from the source domain, and the optimization in the target domain is guided by this prior knowledge. Applying the DRO principle allows us to rigorously derive a penalized estimation framework, while also ensuring robustness to distributional shifts. \n\n**W2: Notations**\n\nWe thank the reviewer for pointing out these oversights. We will add the relevant definitions to the main text. The notation $δ\\geq 0$ denotes the radius of the Wasserstein ball centered at the empirical measure. The free variable $κ\\in\\mathbb{R}$ in the optimization is interpreted as the projection coefficient of $β$ onto $θ$. In $π(A\\times\\mathbb{R}^d)$, $π$ denotes a probability on the product space $\\mathbb{R}^d \\times \\mathbb{R}^d$, and $A$ is a Borel measurable subset of $\\mathbb{R}^d$.\n\n**W3: Hyperparameter Tuning**\n\nWe use cross validation (CV) to tune the hyperparameters $δ$ and $λ$. In the future, our goal is to develop an automated hyperparameter selection method, akin to the methods introduced in (Blanchet et al, 2019a)(Line 462, Left) and (Blanchet et al, 2022)(Line 476, Left), building on the theoretical framework of WDRO.\n\nFor ablation study, we conducted experiments where we set $λ=0$ (no transfer), reducing KG-WDRO to a standard WDRO formulation. As shown in the upper two subfigures of Figure 1, our method consistently improves regression error and classification accuracy, when the correlation between the prior $θ$ and the true $β$ is as low as 0.3. \n\nIn this revision, we include a new ablation study in high-dimensional regression setting, which will be added to main text. We fix the values of $δ$ to either $δ^*$, where $δ^*$ is tuned via CV on a standard WDRO estimator, or fixed to $δ=3$. Using fixed values, we fit KG-WDRO estimators across a grid of $λ$. The resulting out-of-sample (OOS) performances are plotted in the figure (https://figshare.com/s/05b069e330136338fa4e). When the correlation between the prior $θ$ and the true $β^*$ is high, setting $λ\\to∞$ yields the best OOS performance. As the correlation decreases, smaller values of $λ$ lead to better results. This finding highlights the need to include $λ$ to control the extent of bias, and aligns with the intuition that stronger correlations warrant larger values of $λ$. Red dots in the plot represent the OOS performance obtained via CV. The $λ$-coordinates of these red dots follow the trend of the curves. The red dots lie above the curves, indicating improved performance when $δ$ is tuned.\n\n**Q1: Smaller Ambiguity Set**\n\nYes. For the case when $λ=∞$, we can upper bound the minimax objective (Line 239, Left) by $\\inf_{α\\in\\mathbb{R}} \\mathbb{E}_{\\mathbb{P}_N}(Y-(\\alpha θ)^\\top X)^2$, which is the in-sample risk of the prior $θ$ adapted on target data. In standard WDRO, this upper bound is trivially given by $\\mathbb{E}(Y^2)$. We can show that the ambiguity set becomes strictly smaller when $λ= ∞$ compared to when $λ=0$. We will provide these detailed quantitative discussions on the ambiguity set in the new draft. \n\n**Q2: Mitigating Conservativeness**\n\nThe goal of KG-WDRO is to ensure robustness against statistical noise while avoiding over conservativeness. One could simply use the empirical risk minimization solution, $β_{ERM}$, from Problem (ERM) (Line 133) to prevent over-conservatism but in a small-sample regime, computing $β_{ERM}$ is often infeasible without introducing bias, such as that induced by standard WDRO. Even in cases where $β_{ERM}$ is computable, its out-of-sample performance is typically poor due to the high uncertainty associated with limited data.  \n\nLinear interpolation between the worst case and the random case may also fail. It would inherit both high uncertainty (from the random distribution) and high conservatism (from worst-case distribution), leading to a suboptimal trade-off between bias and variance. Overall, there is no free lunch if no additional information is provided. In contrast, our framework leverages prior knowledge in a structured manner to mitigate conservativeness while maintaining robustness, thereby offering a principled alternative to simple interpolation-based approaches.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I checked part of them.", "experimental_designs_or_analyses": "I checked the experimental part.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "I think the references are relatively sufficient.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- Integrate prior knowledge into WDRO for linear regression and binary classification. Provide equivalence between KG-WDRO and shrinkage-based estimation method. Interprets a broad range of knowledge transfer learning approaches through the lens of distributional robustness.  \n- The perspective of the research is supported by theoretical advancements and empirical evidence.\n- The experimental section presents extensive results, qualitative and quantitative.\n\n**Weaknesses:** \n\n* Lack of consistency in writing. It is hard to follow the connection between transfer learning and WDRO, the authors mention WDRO in the title, then suddenly mention transfer learning in the beginning of the abstract, without further discuss how the proposed framework benefits transfer learning. \n* Lack of introduction of the notation before using them.\n  * In Example 1.  What is the meaning of $\\delta, \\kappa$ ?\n  * In section 2.1. What does the notation $\\pi (A \\times \\mathbb{R}^d )$​ mean? \n* Lack of justification and ablation experiment for the selection of hyperparameters $\\delta$ and $\\lambda$ .", "other_comments_or_suggestions": "No", "questions_for_authors": "- In line 20, the authors mention \"Our method constructs smaller Wasserstein ambiguity sets\", the reader may want to acquire a quantitative analysis about it, can it be further discussed? \n- The authors mention ''This framework mitigates the conservativeness of standard WDRO''. According to my understanding, the conservativeness of vanilla WDRO is due to worst-case optimization. And it seems to be relatively easier to bypass the worst case if one wants to alleviate the conservativeness of WDRO, do the authors demonstrate the challenge of directly using some vanilla methods (for example, linear interpolation between worst case and random case)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z1XhRrAgzs", "reviewer": "Reviewer_85zy", "summary": "This paper investigates distributionally robust optimization (DRO), focusing on Wasserstein distance-based DRO (W-DRO) while introducing a novel knowledge-guided cost function to further enhance the robustness and performance of DRO frameworks. The authors provide an extensive and thorough review of DRO, elaborating on its properties, and specifically analyzing the mathematical underpinnings of W-DRO. Building on this theoretical foundation, the authors propose a Knowledge-Guided Transport Cost (KGTC), which incorporates knowledge coefficients that reflect prior knowledge about the data or task to guide the transport cost within the Wasserstein ball. By embedding these knowledge-guided adjustments, the model aims to achieve better generalization and robustness to distributional shifts. The proposed method is validated theoretically and empirically on linear regression and binary classification tasks, demonstrating promising improvements over existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the encouraging and detailed feedback. Here we list our responses to the weaknesses and questions suggested by the reviewer.\n\n**W1: Lack of Convergence and Guarantees**\n\t\nWe acknowledge the absence of an explicit discussion on convergence rates and statistical guarantees in this paper. However, the primary focus of our work is to establish the theoretical equivalence between shrinkage-based transfer learning and the Wasserstein Distributionally Robust Optimization (WDRO) framework and the statistical properties are beyond the scope. This perspective provides a unified approach to analyzing transfer learning problems.\n\nMany prior studies on specific cases of our proposed method have already explored the convergence of the optimal estimator as the radius shrinks; e.g. [1]. We plan to leverage the WDRO framework to establish formal statistical guarantees in future work. \n\n**W2 I: Intuition on Cost Function**\n\t\nThe standard WDRO cost function takes the form $c(x,x') = \\Vert x-x'\\Vert_q^2$, allowing perturbations of the covariate in all directions. However, if we believe that prior knowledge $\\theta$ serves as a trustworthy proxy for $β^*$ (the true optimum of Problem (SO), Line 125, Right), then it is natural to minimize perturbations in the predictive direction of $\\theta$. Specifically, we constrain the perturbed point $x'$ so that the discrepancy between the predictions of $x$ and $x'$ under $θ$ is small , i.e., $θ^\\top x' ≈θ^\\top x$. Since $x$ is a point in the empirical measure, this ensures that the transported measure $P_N$ is mapped to distributions that preserve predictions under $θ$. Consequently, taking the worst-case over the ambiguity set yields estimators that perform at least as good as using $θ$ naively.\n\n**W2 II: Multi-Source Knowledge**\n\nIn applications such as electronic health records, multiple clinical trials are conducted in different hospitals. These source domains typically represent majority but distinct populations. When the target domain involves a mixed-ethnicity population, it is reasonable to expect that the target estimator should be some combination of estimators derived from the source domains. This is naturally captured by our penalty term, for the case of two source data, this takes the form  $\\inf_{κ_1,κ_2}\\Vertβ-κ_1θ_1-κ_2θ_2\\Vert_p,$ allowing the model to automatically search for the best combination of source knowledge. We refer to this as a “multi-source ensemble” of prior knowledge, where the learning process profiles and distills useful information from multiple sources.\n\n**W3: Practical Applications**\n\nThanks for your advice.  To illustrate the applicability of our KG-WDRO framework in the real world, we apply it to the TransGLM dataset [2] on 2020 U.S election results at the county level. \n\nCounties are labeled as '1' if the Democrat candidate won, and 0 otherwise. We assess KG-WDRO vs. TransGLM by classifying county-level outcomes in eight target states, using data from the remaining states as source knowledge. The cleaned dataset includes 3111 counties and 761 standardized predictors across 49 states.\n\nUsing 2100 counties as source, we predict results in eight target states (~100 counties each). KG-WDRO outperforms TransGLM in 5 of 8 states, reducing overall logloss by 7.6%, see table (https://figshare.com/s/e00c2d14f2c15ac02ed9). Both transfer learning methods significantly outperform the standard WDRO estimator. We will add this experiment to our main text.\n\n**W4: Computational Complexity**\n\nTheorem 3.2 transforms the infinite-dimensional problem (Line 240) into a tractable convex program (Line 242). Given the number $M$ of external sources is finite, the convex program takes the form $\\inf_{β,κ}\\Vert \\mathbf{y}-\\mathbf{X}β\\Vert_2+\\sqrt{δ}\\Vertβ-κ_1θ_1-...-κ_Mθ_M\\Vert_p,$ where $κ\\in\\mathbb{R}^M=[κ_1,...,κ_M]^\\top$. Setting $κ=[0,...,0]^\\top$, this reduces penalty term to a $p$-norm regularization, like Lasso or Ridge. When $κ$ is free, there are $M$ parameters. Consequently, the total number of parameters increases from $d$ (dimension of $\\beta$) to $d+M$. As long as the number of knowledge sources remains finite—which is a reasonable assumption given the cost of experiments—the parameter size of the program remains of order $O(d)$ in high-dimensionality.  \n\nIts computational complexity remains roughly the same as traditional regularization techniques. In our experiments, the KG-WDRO optimization problem is efficiently solved using CVXPY with Mosek. Additionally, we conduct an empirical study to demonstrate that their runtimes remain similar (https://figshare.com/s/9af171f8a0dcc3eb32da).\n\n[1] Blanchet, J., Murthy, K., & Si, N. (2022). Confidence regions in Wasserstein distributionally robust estimation. Biometrika, 109(2), 295-315.\n\n[2] Tian, Y., & Feng, Y. (2023). Transfer learning under high-dimensional generalized linear models. Journal of the American Statistical Association, 118(544), 2684-2697.", "claims_and_evidence": "This is a rigorous paper, most of the claims are backed with theoretical support. Only more intuition would be better.", "methods_and_evaluation_criteria": "No.", "theoretical_claims": "I didn't the details of the proofs.", "experimental_designs_or_analyses": "I have check the Experimental designs, which are quite reasonable.", "relation_to_broader_scientific_literature": "This paper enhances the W-DRO with a novel regularization, which enhances the performance under extreme situations.", "essential_references_not_discussed": "No additional references needed.", "other_strengths_and_weaknesses": "Strengths\n- Theoretical Rigor and Solid Analysis:\n  - The paper presents strong theoretical foundations with rigorous and logical arguments supporting the development of the Knowledge-Guided Transport Cost (KGTC).\n  - The analytical treatment of both standard DRO and Wasserstein DRO is comprehensive, offering valuable insights into the properties and limitations of existing approaches.\n  - The extension toward knowledge-guided variants is well-motivated from a theoretical perspective, making a notable contribution to the DRO literature.\n- Comprehensive Coverage of Use Cases:\n  - The authors consider multiple scenarios, including both strong and weak transfer settings, showcasing the generality and adaptability of their proposed framework.\n  - The method is applied to both regression (linear regression) and classification (binary classification)tasks, which highlights its versatility and potential for broader applications.\n- Promising Empirical Results:\n  - The numerical experiments support the theoretical claims, demonstrating significant performance gains over standard DRO formulations.\n  - The improvements observed in diverse tasks highlight the practical potential of incorporating knowledge-guided components into DRO.\n \nWeaknesses and Areas for Improvement\n- Lack of Discussion on Convergence and Statistical Guarantees:\n  - One notable omission is the absence of a discussion regarding convergence rates and statistical guaranteesof the proposed method.\n  - While the method is shown to be effective in terms of robustness and performance, readers would benefit from understanding whether there are provable bounds on convergence speed or finite-sample guaranteesthat back up the empirical observations.\n- Intuition Behind Knowledge-Guided Cost Function Needs Clarification:\n  - Although the knowledge-guided cost function is mathematically well-defined, there is a lack of intuitive explanation regarding why and how incorporating multiple knowledge coefficients leads to better transport cost estimation and performance gains.coefficients would help clarify the practical utility of the approach.\n- Practical Applicability and Generalization to Real-World Scenarios:\n  - While the theoretical and empirical studies are strong, the real-world applicability of the knowledge-guided cost function is underexplored.\n  - It would be beneficial to provide concrete real-world scenarios or case studies where domain knowledge is readily available and can be leveraged for improved robustness (e.g., healthcare, finance, or autonomous systems).\n- Computational Complexity and Overhead Not Addressed:\n  - A major concern is the potential computational overhead introduced by incorporating the knowledge-guided cost function.\n  - Intuitively, the use of knowledge coefficients and the corresponding adjustments in transport cost computation could lead to higher space and time complexity, especially in large-scale problems or high-dimensional settings.\n  - The paper does not discuss the computational implications, nor does it provide empirical measurements(e.g., runtime, memory usage) to reassure readers that the method remains computationally feasible.\n  - Adding an analysis or even a simple comparison of computational efficiency between standard W-DRO and KGTC-enhanced W-DRO would significantly strengthen the practical relevance of the paper.", "other_comments_or_suggestions": "Please see the weaknesses.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a novel extension of Wasserstein-based DRO approaches that incorporates external knowledge into the ambiguity set via a modified cost function. By adjusting the Wasserstein distance, the framework aims to reduce the inherent conservatism of standard approaches. The authors demonstrate that the resulting optimization is equivalent to known regularization techniques and support their claims through theoretical analysis and numerical results. 

Most of the reviewers' assessments for the paper are positive, and the overall contribution is conceptually solid, with clear potential for influencing future work in robust learning methods. Even though the paper has some limitations such as the lack of performance guarantees, the usage of prior knowledge to constraint the ambiguity set seems both novel and useful. I would suggest the authors to update the manuscript following the comments provided by the reviewers.
Decision: Accept (poster)