Towards Backwards-Compatible Data with Confounded Domain Adaptation
Authors: Calvin McCarter
Venue: Accepted by TMLR ()

Abstract: Most current domain adaptation methods address either covariate shift or label shift, but are not applicable where they occur simultaneously and are confounded with each other. Domain adaptation approaches which do account for such confounding are designed to adapt covariates to optimally predict a particular label whose shift is confounded with covariate shift. In this paper, we instead seek to achieve general-purpose data backwards compatibility. This would allow the adapted covariates to be used for a variety of downstream problems, including on pre-existing prediction models and on data analytics tasks. To do this we consider a modification of generalized label shift (GLS), which we call confounded shift. We present a novel framework for this problem, based on minimizing the expected divergence between the source and target conditional distributions, conditioning on possible confounders. Within this framework, we provide concrete implementations using the Gaussian reverse Kullback-Leibler divergence and the maximum mean discrepancy. Finally, we demonstrate our approach on synthetic and real datasets.

Published in Transactions on Machine Learning Research (11/2024)
Towards Backwards-Compatibe Data with Confounded Do- main Adaptation
Calvin McCarter mecarter. calvin@gmail.com
Reviewed OpenReview: httpp://opaaaeeeiee..ee///rrrrrs
Abstract
Most current domain adaptation methods address either covariate shift or label shift, but not applicable where they occur simultancously and are confounded with cach other. Domain adaplation approaches which do account for such confounding are designed to adapt covariates to optimally predict a particular label whose shift is confounded with covariate shift. In this paper, we instead seek to achieve general-purpose data backwards compatibillty, This would allow the adapted covariates to be used for a variety downstream problems, including on pre-existing prediction models and on data analyties tasks. To do this we consider a modification of generalized label shift (GLS), which call confounded shift. We present novel framework for this problem, based minimizing the expected divergence between the source and target conditional distributions, conditioning possible confounders. Within this framework, we provide concrete implementations using the Gaussian reverse Kullback-Leible divergence and the maximum mean discrepancy. Finally, we demonstrate our approach on synthetic and real datasets.
1 Introduction
The heterogeneity of scientific data is major obstacle to training useful Al models for scientific research Bronstein & Naef 2024). The recent success of AlphaFold (Jumper et al. 2021) for protein structure prediction is in large part due to the availability of a large-scale standardized dataset in the form of the Protein Data Bank (Burley et al. 2019). In contrast, for many other biological problems, data obtained from a variety of settings which vary due to both scientifieally-reeeaall differences and also due to differences in experimental conditions which give rise to "batch effects" (Leek et al. (2010). And unlike protein structure data, for which ground- truth quantities relative quantities (Bronstein & Naef 2024. These reported quantities that are relative to some detection threshold or noise level are particularly subject to batch effects caused by technical differences in experimental setups, assays, and even computational techniques for processing raw sensor data (Cai et al. 2018
absolute distances, many other scientific data modalities report
Within various scientific fields, such as genomics (Johnson et al. 2007 Sprang et al. 2022) proteomics Gregori et al. 2012 Pelletier al. 2024). and neuroscience (Yamashita et al.) 2019 Torbati et al.) 2021
there exist separate preexisting literatures on domain adaptation methods for combining multiple datasets affected by technical artifacts. We cannot hope to discuss all of them in detail, but generally speaking. these methods seek to align datasets so that they have similar feature distributions (Johnson et al. 2007 Shaham et al. 2017) or similar intra-datasct sample-sample relationships (Haghverdi et al.) 20188.. This objective is dangerous when the to-be-combined datasets not only difler for technical reasons, but also due to variables that are to be scientifically investigated or predicted (Hicks et al.) 2018 Zindler et al. 2020 Antonsson & Melsted 2024). In this case, the feature distributions of the two datascts should not be aligned to be equal, because the difference in feature distributions confounded with a difference in distributions for some scientifiallly-reeeaanl variable.
1

---

Published in Transactions on Machine Learning Research (11/2024)
In this work, we propose a feature-space domain adaptation method (Pan et al. 2010 Kouw et al. 2016 aiming to provide scientists with general-purpose backwards-compatible data. This backwards compatible patibility constraint is especially useful in situations where features are used by a downstream model that cannot be updated for organizational or regulatory reasons. In other words, we will try to estimate what the features would have looked like, had they been obtained using the samne technical process as the reference dalaset. This use-casc rules out previous domain adaptation methods (Huang al. 2006; Zhang et al.) [013 Tachet des Combes et al., 2020) that reweight samples match distributions, instead of transforming their features.
This also rules out the substantial literature (Wang & Mahadevan, 2009 Liu & Tuzel 2016; Tzeng et al.) 2017
Chen et al. 2022) that aims to learn new (often lower-dimensiona] domain-invarian representation.
Our proposed method for feature-space confounded domain adaptation, which we dub ConDo, comes with another unusual twist. We make the assumption that the user observes the confounding variables, sidestepping the problem of latent confounders, but only at training time, and not inference time. This means that the feature-space transformation function must map from one feature-space representation to another, without also taking in the confounding variable as input. This imposition notably rules out ComBat. (Johnson el al.) 2007) and its many variants (Zhang et al. 2020 Torbati et al. 2021 Radua et al. 2020), which are commonly used for batch correction in genomics, neuroscience, and elscwhere. We motivate and illustrate this kcy
assumption in Section 2.1
To begin to address this challenge we assume a modification of generalized label shift (GLS) (Tachet des Combes el al.) 2020) which we call confounded shift. Confounded shift does not assume that the confounding variable(s) are identically distributed in the source and target domains, or that the covariates are identically distributed in the source and target domains. Rather, it assumes that there exists adaptation from target covariates to source covariates such that the source's conditional distribution of covariates given confounders is equal to that of the adapted-target' conditional distribution. However, we do not assume that the adapted-tarretts covariates and source's covariates have the same distribution.
In the rest of the paper, we describe and empirically evaluate the ConDo framework for adapting the target to the source, based on minimizing the expected divergence between adapted-targe and source conditional distributions, i.e. conditioning on the confounding variables. We show how to compute the expectation with respect to prior distribution over the confounders, and for the prior we propose estimator of the product of the source and larget confounder distributions. We propose using the Gaussian reverse Kullback-Leibler Leibler divergence (KLD) and the maximum mean discrepancy (MMD) as divergence functions. Furthermore, using this framework we provide concrete implementations based on the assumption that the vsrrret batch effect is "simple". In particular, we restrict the adaptation to be affine, or even location-scale (i.e. with a rotation representable by a diagonal matrix). This assumption especially intended to adapt structured data, such as biometric sensor outputs, genomic sequencing data, and financial market data, where domain shifts are typically simple, yet where the input-output mapping is often nonlinear. We are not, in this paper, attempting to enable image classification model for photographs to be adapted to hand drawnings, though we hope our framework can be extended to such nonlincar transformations.
2 Preliminaries
We begin by providing concrete motivating example. We then introduce our notation, describe standard approaches to lincar domain adaptation, and provide background on generalized label shift.
2.1 A motivating example
Suppose you have patient health outcome data and EEG data from the first version of an EEG device, depicted as "V1 training data" in Figure I Using this datasct, you have alrcady conducted various statistical analyses and trained health outcome prediction models, including for seizure and depression. But then the EEG machine gets updated to V2, and we obtain a small amount of EEG data collected from the V2 machine, along with patient seizure status, depicted as "V2 training data". The V2 data distribution appears shifted relative to that from the V1 machine. At this point, it seems appropriate to perform covariate shift domain adaptation, to learn an explicit feature-space transformation that adapts the V1 and V2 data to look alike.
2

---

Published in Transactions on Machine Learning Research (11/2024)
Source
Target
V1 training data
V2 training data
V2 test data
R
X
Z
X
Y
EEG data
Setzure
Depression
covariate shift
Adapt V2 to V1
(V1 va V2) label shift (mostly healthy
Predict seizure
Predict depression
X
Z
individuals)
Diagram depicting our molivating scenario. There is confounded shift between source (V1) and target (V2) domains. The different shades of blue of the EEG data portray the "covariate shift" between VI features and V2 features. The seizure confounding variable differs distribution between source and larget; we portray this "label shift" in seizure status proportions (x vs +) between V1 and V2. With VI training data, we had previously learned prediction models for seizure and depression given V1 features. With V2 training data, we learn mapping from V2 features to V1 features via ConDo. At V2 test time, neither scizure nor depression status are provided, but we combine the ConDo V2-to-V1 mapping with the previously-learned prediction models. The EEG data correspond to the features variable X, seizure stalus corresponds to the confounding variable Z, and depression status corresponds to the downstream prediction variable
Figure 1:
Yet additionally, while the V1 dataset comes from a large number of low-risk and high-risk patients, the V2 dataset thus far is mostly comprised of seizure-free individuals. Ignoring the aforementioned covariate shift problem, this latter problem in isolation would fall into the label shift domain adaptation problem setting, Our hypothetical scenario combines these two problems: it has both covariate shift and label shift which are confounded with each other.
Our goal to learn a feature-space transformation correcting for the technical effects between source (V1 machine) and target (V2 machine) domains, while being aware the actual neurological differences between the two datasets. We would like learn a single feature-space transformation (and transformed dataset) that can be used not only for single prediction task, but for multiple downstream tasks and statistical analyses. For example, we might want to combine the V1 and (adapted) V2 data and then assess for statistical correlation between EEG features and patient age.
The key restriction will be that, at inference time, we will not observe these confounding biological variables. To see why the feature transformation may take in only the FFG data as input, consider the fact. that we may want predict scizure and depression risk on incoming V2 test samples. Even though we could use our confounder (seizure) in the training data to learn transformation, on test samples do not yet know health outcomes: indeed, it is what we want to predict! However, will have access to seizure status while learning the feature transformation, which we will assume fully accounts for the confounding between V1 and V2, as will be formalized in Section 3
The aforementioned example also illustrates why standard domain adaptation approaches may not be applicable. new EEG embedding space invariant to the V1-vs-V2 domain shift would not be a "general- purpose" solution for variety of downstream prediction and statistical inference tasks. In contrast, we would like to preserve much information as possible, and not only the principal components of variation, only
3

---

Published in Transactions on Machine Learning Research (11/2024)
the subspaces that are relevant for seizure prediction. Transfer learning Caruana 1997 Daume III & Marcu 2006) and model fine-tuning (Girshick et al.) 2014) may also be inapplicable, since the preexisting seizure and depression diagnostic models might be the responsibility of other organizations, or may be non-accessible or non-updatable for regulatory reasons.
2.2 Notation
X and Z respectively denote the feature (covariate) space and the confounder space. X and Z denote random variables which take values in X and Z, respectively. A joint distribution over covariate space and confounder space Z is called domain D. In setting, there is source domain Ds and target domain Dr; we assume Ns and Nr samples from the source and target domain, respectively. D DD denote the marginal distributions of covariates under the source and target domains, respectively; DS. Dÿ denote the corresponding marginal distributions of confounders.
We assume that feature variables are real-valued vectors, so denote samples as æg MR, ØT R.Mr, We make no such assumption for the confounders, denoting each observation as 2, but we assume the existence of user-specified confounder-space kernel function k =(ma)). We will seek optimize the mapping 9 : Xy Xs from target feature-space to source feature-space. We will also sometimes consider target variable (output label) Y E V, for which we have classification hypothesis h X' + y that has been trained on data from the source domain, corresponding to the standard unsupervised domain adaptation (UDA) setting.
For arbitrary distributions and Q, we assume we have been given a distance or divergence function denotod by d(P, Q). By [(, Σ) we denote the Gaussian distribution with mean 4 and covariance Σ. By |-| we denote the absolute value; by det(.) denote the matrix determinant. By A we denote the matrix transpose.
2.3 Affine domain adaptation based Gaussian Optimal Transport
Domain adaptation has a closed-form affine solution in the special case of two multivariate Gaussian distributions. The optimal transport (OT) map under the type-2 Wasserstein metric for x ~ ~V Es) to a different Gaussian distribution N(ur, Er) has been shown (Dowson Landau 1982 Knott & Smith 1984 to be the following:
pT
A(x 1s)) Ar + (UT Apss), where
A = Σ 1/2 Σ Σ'
1/2
Σ = AT.
This mapping has been applied to domain adaptation Flammrr el al. 2019 and other applications (Mallasto & Feragen 2017 Muzellec & Cuturi 2018 Shafieezadeh Abadeh et al. 2018 Peyré et al.) 2009) in machine learning. For univariate Gaussians N (iis, d')) and N(pr, o4),, the above transformation simplifics to
OT pr
QT ØT x + (pT og
x pS)
ps).
(2)
0g
0g
2.4 Affine domain adaptation minimizing the maximum mean discrepancy
An alternative approach can be derived from representing the distance between target and source distributions as the distance between mean embeddings. This leads to minimizing the (squared) maximum mean discrepancy (MMD), where the MMD is defined by a feature map á mapping features æ E X' to a reproducing kernel Hilbert space H. We denote the feature-space kernel corresponding to & as (x(x(")), ((+))) = (0(x("))) (()())))) Bocause the feature-space vectors are assumed to be real, MMD-bbasod adaptation methods typically use the radial basis function (RBF) kernel, which leads the MMD being zero if and only if the distributions are identical.
4

---

Published in Transactions on Machine Learning Research (11/2024)
If the transformation is affine from source to target, the loss can be written follows:
MMD²(Dr,, Ds) =I (01)) w((1)) ~D, (x (m)))
2 ((r)) (m (1))
(x(x (m1) Ax
Ax (nz)
+ b)
+ E ml1r) Dg kx (Aw (ma)
(Ax(ma)) + b, Aw (ma) + b).
(3)
Prior work has sometimes instead assumed a location transformation (Zhang et al.
2013
nonlinear
transformation (Liu et al. 2019ah. Notably, while previous MMD-bbsed domain adaptation methods have
matched feature distributions (Zhang et al. 2013 Liu et al. 2019a Singh et al. 2020 Yan et al. 2017); joint distributions of features and label [Long et al. 2013). or the conditional distribution of label given features (Long et al.) 2013), they have generally not considered matching the conditional distribution of
features given labels. One exception this IWCDAN (Tachet des Combes et al. 2020). which however aligns datasets via sample importance weighting rather than leature-space Iranslormation.
2.5 Background on Covariate Shift, Label Shift, and Generalized Label Shift
Domain adaptation methods typically assume either covariate shift or label shift. With covariate shift, the marginal distribution over covariates differs between source and target domains. However, for any particular covariate, the conditional distribution of the label given the covariate is identical between source and target. With label shift, the marginal distribution over labels differs between source and target domains. However, for any particular label, the conditional distribution of the covariates given the label is identical between source and target domain.
More recently, generalized label shift was introduced allow covariate distributions differ between source and target domains (Tachet des Combes et ol. 2020). Generalized label shift (GLS) instead assumes that, given transformation function X == g(X) applicd to inputs from both source and target domains, the conditional distributions of X given Y = y are identical for all y. This is a weak assumption, and it applies to our problem setting as well. However, it is designed for the scenario where we simply need to preserve
information only for predicting Y given X ~ DD..
3 Confounded Domain Adaptation
Consider our motivating scenario in which our ultimate goal to reuse a classification hypothesis h: X y in a new deployment setting. We treat the deployment setting as the larget domain. And instead of learning an end-to-end predictor for the deployment domain, we learn an adaptation g from to the source domain for which have large number of labeled examples. Then, to perform predictions the deployment (target) domain, we first adapt them to the source domain, and then apply the prediction model trained on the source domain. In other words, we do not need to retrain h, and instead apply h to incoming unlabeled target samples. Similarly, other prediction tasks and statistical analyses can be applied after combining hhapted-target feature data and source feature data.
In many real-world structured data applications, new data sources are designed with "badwwards-compptibliitt in mind, with the goal that updated sensor and assays provide at least much information as the earlier versions. We therefore assume the existence of "true" noise-free mapping g from the deployment (target) domain to the large labeled dataset (source) domain. The algorithms developed under our framework could instead be applied when treating the deployment setting as the target domain and the labeled dataset as the source domain. However, this easier setting would allow retraining h on adapted data, and is thus not the focus of this paper.
3.1
Our Assumption: Confounded Shift
In our case, given X D we instead want to recover what it would have been had we observed the same object from the data generating process corresponding to the source domain X DY. In other words, the mapping g(X) should not only preserve information in X useful for predicting Y, but idcally all information in X D2X that is contained in X ~ XXD.
5

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: Domain adaptation settings
Name
Shift
Assumed Invariant
Covariate Shift
D
* D
Ve E X, [ = x) Dr(Y =x)
Labol Shift
D + D
Vy E DE,,DD())) =y) =)= y)
Generalized Label Shift
D
# D.
Vy E ( = - =
Confounded Shift
D/ D
-Dr(o(X))V =y)
Graphical representation We may formulate setting with latent variables X corre- sponding to features before the target-to-source mapping. We begin by defining an in- dicator variable D which specifies whether a sample is taken from the target or the source domain. We may then depict our assumption with the following graphical model
D
Z where latent features X always follow the target domain distribution PD, (X 3|2 = =), while observed features follow
X
8(x- D=T 66(x-9(8) D = S
2) =) = = =
(4)
where is the Dirac delta. By inspection of the graphical model, our setting a combination of prior probability shift and covariate observation shift as defined (Kull Flach 2014). Note that latent features X are generated from confounders Z, which motivates using generative model for domain adaptation.
Relation to Generalized Label Shift Suppose GLS intermediate representation g(X) were extended to be function of both X and the sourco-vs-target indicator variable D. Then, given this extended representation DJ, we restrict x((X, D)) as follows,
0((,DD) x(X) X D-S
D=T
(
that samples from the target distribution are adapted by g(-), while those from the source distribution pass through unchanged. For comparability, we let Y= Z, i.e. we consider the scenario where our downstream prediction task is to predict the confounding variable which is unavailable at test time. With this extended representation, as well as the restriction on ÿ, confounded shift and GLS coincide. The previous assumptions as well as our confounded shift assumption are summarized in Table [ Note that while confounded shift. is stronger than GLS, both allow DY + DX;; and just as GLS allows (2(x) + DX(X), we analogously allow (g(x) +D..
Application to downstream prediction tasks For scenarios where Y # Z (where downstream task is to predict a variable other than the confounder), we do not require dependence correlation between the confounding variable Z and downstream prediction variable Y. Rather, we assume that the conditional distribution of Y given features has not changed from source to larget, i.e. pD.(( (Ylg What is necessary is that we correctly map features from target to source; then any downstream prediction model trained on source data can be reused via the composition o g, as long as the downstream conditional distribution (estimated by h) has not changed.
3.2 Main Idea
Our primary aim to infer a transformation that is broadly applicable, so given observed source domain features X will seek to reconstruct with minimal error. Our secondary aim is to minimize

---

Published in Transactions on Machine Learning Research (11/2024)
downstream prediction tasks, which for simplicity is assumed to be binary classification. Formally, the hypothesis is fixed binary classification function h X' - {0, 1). We seek to choose which minimizes the accuracy loss induced (by unknown shift ") on hypothesis h under distribution Dj:
PD ( * h(x)).
(6)
As in typical domain adaptation settings, we expect that Nr < Ns, with an abundance of prediction labels on our source datasct (i.c. from the V1 scnsor).
Our proposal, which dub ConDo, is to minimize the expected distance (or divergence) d between the conditional distributions of source and target given confounders, under some specified prior distribution over the confounders:
min E ENDD ()) fo
(7)
Four ingredients are neoded to turn this into concrete algorithm: a featuro-space transformation fo X (Section 3.3 a prior confounder distribution Dz (Section 3.4), a sampler from the conditional distributions (x Z 2) (Section 3.5 and cistance/divergeen function d (Section 3.6
3.3
Feature-space transformation function
Our goal is to find the optimal linear transformation g(æ) Ax + of the target to source. In certain scenarios, particularly scientific analyses, it is important for explainability that each ith adapted feature [Ax(") + b); be derived only from the original feature [xx(*] So we will examine both full affine transformations and also transformations where A is restricted to be diagonal A = diag(a). The latter is sometimes referred to as a location-scale adaptation (Zhang et al. 2013); this further requires that the source and target features have the same dimension.
3.4 The product prior over confounding variable(s)
Our approach is motivated by the desire to minimize the distance between the conditional distributions Ds(a)/ and D = only where we can estimate them both with high accuracy. These conditional distribution estimators may be poor extrapolators, with noisy estimates in low-density regions of DJ and D This suggests choosing to perform minimization over confounder values that are likely under both source D and target D/ distributions, which motivates using the product of the two distributions.
We estimate the product of D and D/ as follows. For efficient sampling, we use a non-parametric estimator of the product prior, with non-negative support over the union of confounder values in the source and target datasets, so that it can be represented as probabilistic weights attached to each sample. We note that empirical distributions may have non-intersccting support, such as Z is a continuous variable. This motivates smoothing the estimators of D& and DJ before taking their product; this avoids taking the product of two Dirac delta functions, which undefined.
Given a kernel ky over the confounder space, we compute,
N
Nr
DZ
W 8(2 ((m)) + Σ
WT 5(2 Z(nn),, where
(m)
(8)
(n)
E
Ns
kz (Z))) (1) {(m))
INT 2e (Z)1) ((a Nr Nr 1/z (Z( (1) 2(1ح
w
x
Ns No E
No
kz (4 b(
and
j= Σ
1=
D Kz z(a))
-= 2 Z (n)
(n)
W
x
(9)
No Ne No
kz ((1
where ws and WT are normalized so Σ, w + Σ, w (m) continuous confounders and Dirac delta kernel for categorical confounders.
1. In practice we will use an RBF kernel for
7

---

Published in Transactions on Machine Learning Research (11/2024)
3.5 Sampling from the conditional distributions
When the confounding variable is discrete, and cach value occurs in multiple examples in our dataset, then we merely sample from these datapoints with replacement. Otherwise (i.e. the confounding variable is contimuuus), we instead need to sample from conditional generative models for 2 = 2) and Ds(2 Z = 2). While diffusion modeling (Sohl-Dickstein et al.) 2015) possible, we found that multiple imputation with chained equations (MICE) (Van Buuren et al. 1999), as implemented in MICE-Forest (Wilson et al. 2022 with GiM (Ke el al. 2017), provided better conditional generation results, replicating the experimental results in (Jolicoeur-Men) Martinrau et al.) 2024). To do this, we concatenate the original dataset and second copy with all features masked and all confounder(s) unmasked. We then request Kx multiple imputations, which are then used as conditionally-genete generated features per each observed Z = 2. This process is performed separately for source and target.
3.6 Conditional distribution distance/divergence function
Below, propose using the reverseKKLD and the MMD in our loss function. Both yield simple, efficient. algorithms, including closed-form solution for the reverse KLD with location-scale adaptation. Note that, as we discuss in Future Work, other divergences are possible within our framework. In particular, optimal transport (OT))based distances are likely to offer higher accuracy at greater computational expense. However, we limit ourselves to these two divergences for their low computational cost, and to focus on the overall proposed framework rather than the computational challenges and opportunities that arise from combining ConDo with OT.
3.6.1 Reverse Kullback-Leibler divergence under Gaussianity
It can be straightforwardly shown that the linear map Eq. derived from OT leads to adapted data being distributed according to the target distribution. That is, ue + A(x uQ) ~ N(p,, Ep). Therefore, the Gaussian KLD from the target distribution to the adapted source data distribution is minimized to O, and similarly for the KLD from the adapted source data distribution to the target distribution. This motivates using the Gaussian KLD as a loss function, with either the forward KLD d(P,.) Q) == d (Ple( or reverse KLD d(),Q) == (Q)(Q) Note that we do not model the features as being Gaussian distributed, but rather that the conditional distribution of cach sample's fcatures given confounders having Gaussian noise.
Whether forward or reverse KLD, minimizing Fq- 7 requires estimating the conditional means and conditional covariances, according to both the source and target domain estimators, evaluated at each 2 ~ Dz. (If the transformation is location-scale rather than full affine, KLD minimization requires only the conditional variances for each feature.) Given samples in the prior distribution, each with weight given
(n) (n) by Wn, 1 < S N, let the source and target estimated conditional means be given by ug Mr and the conditional covariances be given by (m) (n)) respectively.
For the forward-KL.D, this leads to the following objective:
N
min log (Idot(A)) + A.b
Σ wn * tr (r((AA) n=1
( A (T''=||)
+ - +b- (m) )
(A +
b
p4 (n)
((Aus" (n) AT AΣ 5 A
(10)
While the forward KLD from target to adapted-source appears to be the natural choice, we instead propose to use the reverse KLD. Due to computational tractability and well-conditioned behavior, the reverse KLD
has found wide use in variational inference (Blei 2017). knowledge distillation (Agarwal et al. 2023).
and reinforcement learning (Kappen el al. 2012 Levine 2018). We will see that it also confers benefits in
8

---

Published in Transactions on Machine Learning Research (11/2024)
domain adaptation. The reverse-KLD leads to the following:
min - log (Idet(A))) + Σ wn
tr (n) -1
E A A
A.b
Aps (n)
+ b
(m) pr
Σ/
(n) (Aps (n) +b -b-a))))
(11)
This is more efficient to optimize, requiring a single matrix inversion per sample, rather than once per sample after each optimization update to A. This also minimizes the negative log-abs-determinant of A, which functions as a log-barrier away from 0, maintaining the same sign of the determinant, across oplimization iterations. This is useful, because the linear mapping between two Gaussians is not unique. The goverse-KLD. combined with an initial itcrate (c.g. the identity matrix) with positive determinant, chooses the mapping which preserves rather than reverses the orientation. In contrast, the forward-KLD objective is liable to produce iterates with oscillating signs of det(A).
That the (- log dct(A))) term arises naturally out of the reverse-KLD is of potential independent interest. Preventing collapse inlo trivial solutions is known problem with MMD-based domain adaptation (Singh et al. 2020 Wu et al. 2021). The WweesseKKLD objective may inspire a new regularization penalty for this problem. The log-det heuristic was previously proposed (Fazel et al. 2003) as a smooth concave surrogate for matrix rank minimization, while here prevents rank collapse.
Furthermore, in the case of location-scale adaptation, the reverse- KLD be obtained via fast exact closed-form solution. Further details are given in Appendix
Using the Gaussian KLD requires an estimate of the mean and covariance of the features given each value 2 of the confounder. We obtaín this by generating Kx samples of æ ~ Ds(-I = 2) and x ~ ( = 2) per each 2, then computing the empirical means and covariances. We optimize the ConDo-KLD objective with the PyTorch-mininize (Feinman 2021 implementation of the trust-region Newton conjugate gradient.
method (Lin & Jorge 1999).
3.6.2 The conditional maximum mean discrepancy
We extend MMD-based domain adaptation to match conditional distributions by sampling from the prior confounder distribution. For a particular 2 E Z sampled from the prior, suppose we have a way of sampling from (r = 2) and Ds(æ) Z 2). Then, we have
((Dr(1Z = =), Ds(1IZ = =)) -DDD Ds(.12 = 2))
(12)
E ((r)) ml Dr (177 hix ( (m1) x((1)))
2B (- (2 (x (12) + b)
E Rg(a)) pllo))" ~ D. ( kx AA (m2) + b, (12) + b).
(13)
For the feature-space kernel kx, we use the RBF kernel by default. We minimize this objective with stochastic oplimization. For each minibatch, we sample Kz values from the confounder prior; for each confounder value we sample Kxx feature-vectors from each of (r = 2) and (s Z = =2).
Additional implementation details Further implementation details and computational complexity analysis are provided in Appendix B Our software, with a Seikit-learn (Pedregosa et af. 2011 compatible API and with experimental scripts, available at https://github. wtwttttu.coooeeeeee com/ calvinmccatter addaaa
adapter
4 Experiments
We compare ConDo to baseline methods on three synthetic, two hybrid, and three real data settings. When available, evaluate against the latent ground-truth features by measuring the root-mean-squared error
9

---

Published in Transactions on Machine Learning Research (11/2024)
(rMSE) between these and the adapted features. We also compare the different methods unsupervised domain adaptation (UDA), measuring the performance of source-trained classifiers on adapted target test data, to evaluate our ability to adapt data for use by non-retrainable downstream models. For these experiments, we employ TabPEN (Holmmann et al. 2022), a statc-of-the-art pretrained Transformer- based tabular classifier.
We optimize the MMD and ConDo-MMD objectives with AdamW (Toshchilov & Hutter, 2017 : with weight decay- = 10-4, 81 = 0.9, and (2 = 0.999. We set Kx = 20 (for ConDo-KLD and ConDo-MMD) and Kz 8 (for ConDo-MMD); for MMD, each step also uses 8 evaluations of the MMD loss with 20 samples. We run for 5 epochs (with an early stopping patience of 3 epochs) with learning rate = 10-3, unless otherwise indicated in the Appendix; for both MMD and ConDo-MMD, an epoch is defined as min(Ns, Nr8 steps.
4.1 Synthetic 1d features with 1d continuous confounder
We first examine confounded domain adaptation in the context of single-dimeusional feature confounded by a 1d continuous confounder. We analyze the performance of vanilla and ConDo adaptations, when the effect of the continuous confounder is linear homoscedastic (left column), lincar heteroscedastic (middle column), and nonlinear heteroscedastic (right column). In each case there is confounded shift, because confounder is distributed as Uniform[O, 8] in the target domain, and as Uniform[4, 8] in the source domain. This setting is illustrated in the top row of subplots in Figure 2
For each of the the three settings, we apply the baseline linear domain adaptation methods, Gaussian OT and MMD, and our ConDo versions of these methods, ConDo Gaussian KLD and ConDo MMD. We then evaluate the different methods via not-mmea--squrre-erroo (rMSE) on a heldout dataset of 100 samples, comparing the known ground-truth to the estimates from each adaptation method. Note that in this and all following settings in the rest of the paper, neither the baseline nor the ConDo methods have access to the confounding variable values of the heldout test samples. The results are shown on each of the remaining rows of subplots in Figure 2
We repeat the above experimental setup, but with modifications to verify whether approach can be accurate even when its assumptions no longer apply. We run experiments with-and- label shift (i.c. different distributions over the confounder between source and larget), with-and-without feature shift (i.e. with and without batch effect), and withhot additional iid W 1) noise, for a total of 8 settings. Heldout test errors are shown Figure B. ConDo strongly outperforms vanilla adaptation whenever there is target shift, and is non-inferior otherwise. In Figure ST we also show that ConDo offers improved estimation of the feature-space mapping parameters whenever there is target shift, and is non-inferior hewwisee..
4.2 Synthetic 1d features with multi-dimeenional continuous confounders
We extend the previous experiment to consider the scalability of ConDo to multidimensional confounders. For the noise-free, label-shift, feature-shift setting, keeping all other experimental settings and hyperparameters identical, we vary the number of confounders from to 32. We first augment the number of confounders by appending additional irrelevant "confounders"; sampled from (O 1), to our inputs to the ConDo method. The results, shown in Figure 1 A), indicate that ConDo is very robust to the inclusion of a moderate number of non-confounders.
We next augment the number of confounders by generating a noisy additive docomposition of our original confounder. We first uniformly sample the "true" confounder as before, and generate the feature from it before. We then generate random multidimensionall confounder summing to the "true" confounder of the desired dimensionality (Dickinson] 2010), and provide this to the ConDo methods. The results, provided in Figure 4BB,, show that ConDo is still effective in spite of this subtle relationship between the feature and the multi-dimennsonal confounders.
We furthermore use this latter setting to examine the behavior of ConDo when confounders are partially observed. Results, shown in Figure S4 indicate that typically ConDo fails to take advantage of partially- observed confounders while remaining non-inferior to baselines.
10

---

Published in Transactions on Machine Learning Research (11/2024)
100
IMSE: 30.6 6.628 (25.313)
IMSE: 31.317 (24.003)
IMSE: 16.612 (14.564)
target
50
source: batch-effected
source: true (unobserved)
25
100
iMSE: 9.951 (17.741)
IMSE: 8 8.871 (11.790)
IMSE: 3.507 (3.353)
75
target
source: batch-effected
50
source: true (unobserved)
source:
100
iMSE: 6.976 (6.520)
IMSE: 8.834 (7.602)
IMSE: 6.726 (6.123)
75
target
source: batcheffeeted
50
source: true (unobserved)
25
source: MMD
(MSE: 0.593 (1.258)
IMSE: 209 (1.205)
IMSE: 1.900 (2.077)
75
target
source: batch-effected
50
source: (unobserved)
25
ConDo Gaussian KLD
100
IMSE: 0.542 (1.057)
IMSE: 0.792 (1.018)
IMSE: 0.643 (0.583)
75
target
source: batch-effected
source: true (unobserved)
25
source: ConDo MMD
Figure 2: ConDo methods are superior to Gaussian OT when confounded label shift and feature shift are present. The columns, in order, correspond to a confounder with a linear homoscedastic effect, a confounder with a linear heteroscedastic effect, and a confounder with a nonlinear heteroscedasticc effect. The first row depicts the problem setup, while the remaining rows depict the performance of Gaussian OT and our ConDo mothods. Red points overlapping with green points indicative of high accuracy. In cach subplot, we provide the rMSE on training source data (depicted), and in parentheses, the rMSE heldout source data (not depicted) generated with confounder sampled from target prior Dz. The printed rMSEs are averaged over 10 independent random simulation runs, while the plots depict the results from the final simulation run.
11

---

Published in Transactions on Machine Learning Research (11/2024)
NoisoFree TargelShift FeatureShift
Naisy TargetShift FealureShift
15
MMEE
10
Noise Free TargetShifl NoFesture Shift
Noisy TergetShifl NoFesture
15
IMSE
10
NoiseFrae NoTargetShitt FeatureShiff
Naisy NoTaaggetShft FoaturaShiff
mise
L
NoiseFree No NargeettSiitt NoFeatureShift
Noisy No TargetShift NoFeatureShift
MSE
Gaussian
MMD
ConDo Gaussian KLD
ConDo MMD
Figure 3: Test errors for experiment with synthetic 1d features with 1d continuous confounder. For cach adaptation method, we compute the rMSE of true target feature values vs inferred target feature values after adaptation, then average over 10 simulations.
12

---

Published in Transactions on Machine Learning Research (11/2024)
(A)
10
source - Gaussian OT
rMSE
source MMD
10
source ConDo Gaussian KLD
source ConDo MMD
16
32
1
32
16
32
# confounders
# confounders
# confounders
(B)
10
10
source Gaussian
source MMD
10
source ConDo Gaussian KLD
source ConDo MMD
X
0
8 16
32
1
2
8 16
32
16
32
# confounders
confounders
confounders
Figure Results for transforming 1d data with multiple continuous confounders, with extra irrelevant (0,, 1) confounders, shown in (A), and with noisy additive decomposition, shown in (B). The rMSFs are averaged over 10 random simulations columns, in order, correspond to a confounder with linear homoscodastic effect, a confounder with lincar heteroscedastic effect, and confounder with a nonlinear heteroscedastic effect.
shown for heldout test data (100 samples per simulation). The
4.3 Synthetic 1d and 2d features with 1d categorical confounder
Here, we generate 1d features based on the value of binary confounder. The source distribution is a mixture of Gaussians (525AV(5, 12) + 0.75. 22), while the target 0.75A((5, 12) 0.25A 2²), and the confounder variable indicates the true mixture component. We also use this setting analyze the performance of ConDo for variety of sample sizes. For each sample size under consideration, run 10 random simulations, and report the rMSE compared to the actual pre-feature shift values.
Results are shown in Figure 5 In Figure 5(A) we see that the ConDo methods outperform the baselines, and that they are robust to small sample sizes. As the number of samples increases, ConDo MMD converges to the correct transformation. We see Figure BBB) that ConDo Gaussian KLD is not as fast as the closed form Gaussian OT solution, but still scales nicely with sample size; meanwhile MMD and ConDo MMD have comparable runtimes.
The raw data and results plotted in Figure S5 in the Appendix. In Figure S6 also show that ConDo leads to lower-error estimates of the true feature-space mapping. We extend the previous experiment to 2d features, showing reduced rMSE and improved classification accuracy in Figure S7 and improved feature shift parameter estimation Figure Sg
4.4 ANSUR II anthropometric survey data
We next evaluated approaches on the ANSUR (Gordon et al. 2014) dataset, comprising 93 anthropometric measurements spanning the gamut from ankle cireumference wrist height of 6068 military personnel. We then synthetically created source and target datasets with known ground-truth feature-space transformations
13

---

Published in Transactions on Machine Learning Research (11/2024)
(A)
(B)
10-
4
10
10-1
source MMD
time
source
source ConDo MMD
10
20
50
100
200
500
10
20
50
100
200
500
(samples
Figure 5: Results for transforming 1d data with a 1d categorical confounder. (A) Plot of rMSE vs sample size for each of the domain adaptation methods. Each rMSE was averaged over 10 simulations, with the vertical lines indicating 1 standard deviation over the simulations. (B) Plot of runtime vs sample size for each of the domain adaptation methods.
(A)
(B)
Gender
Height > 5ft6in
Gaussion OT
MMD
Source
ConDo Gaussian KLD
Target (pat-reature shift)
ConDo MMD
Target (post feature shift)
Density
Gaussian OT
MMD
ConDo Gaussian KLD
0.4
1.0
2
3
5
Accuracy
Accuracy
Relative rMSE
(C)
(D)
Gender
Height 5ft6in
Gaussian OT
MMD
Source
Gaussian
Target (pre-feature shift)
ConDo MMD
Target (post-feature shift)
Density
Gaussian OT
2-
MMD
ConDo Gaussian KLD
ConDo
0.4 Accuracy
1.0
1.0
Accuracy
Relative rMSE
Figure 6: Results on ANSUR II for affine perturbation (A-B) and location-scale perturbation (C-D). Model accuracies depict the average over 10 simulations, with error bars depicting the standard deviation. Kernel density plots in (B,D) depict the density of pro-adaptation rMSE divided by post-adaptation rMSE, averaged over simulations, measured against ground truth data. We see that ConDo is less likely to produce worsened data (i.e. relative rMSE > 1).
14

---

Published in Transactions on Machine Learning Research (11/2024)
follows. We generated the source (and the target) dataset as random subsample of 500 individuals with a 75%-25% (and a 25%-75%) male-female split, with gender as the confounding variable. Then, we linearly perturbed the target dataset, as either a random affine or random location-scale transform. For random affine, we use random positive-determinant matrix A = =ddia((d) wwhere U, V are each Haar-distributed orthogonal matrices and d, ~ Uni 2]; for random location-scale, Ai ~ Unif(0.5, 2), b, ~ Unif[0.5, 2]. We then trained TabPFN (Hollmann et al., 2022) models for MaleOrFemale and Height 5ft6in (the median height) on source data. We applied domain adaptation methods, with ConDo methods having access to naleale- as the confounding variable. Prediction models are then applied on adapted target-to-source features. In Figure G we show accuracies of the prediction models and rMSEs for the target dataset features, from 10 independent random simulations. We see that ConDo improves upon vanilla adaptation, with ConDo Gaussian KLD providing the best. performance across all metrics.
In the Appendix Section C.4, we include results in additional settings. Similar performance shown for when TabPFN prediction models are instcad trained on adapted source-to-target features in Figure SS9 We do not observe systematic difference due to ConDo in feature-space mapping parameter estimation, shown in Figure S10 Furthermore, results for only shift in confounder (gender) distribution in Figure SS12] for only feature shift in Figure [SI1] and for neither shift in Figure $13 are also provided, showing improvement, noninferiority, and noninferiority, respectively, for our approach.
4.5 Image color adaptation
We here apply domain adaptation to the problem of image color adaptation, treating cach image as a datasct with 3 features (for the RGB components). We start by adapting back and forth between two ocean pictures taken during the daytime and sunset (the Python Optimal Transport library (Flamary et al. 2021 Gaussian OT example), depicted in the top row of Figure A()). In this scenario, there is no confounding, since the images contains water and sky in equal proportions. Thus, conditioning on each pixel label (a categorical confounder, taking on a value of either "water" or "sky"), expected to be unnecessary. Below the top row, we show the results of affine adaptation for each of the different methods. On the left column of Figure 7(A), we show the result of applying the learned source-to-targel transform. On the right column of Figure [.A), compute the inverse of the aforementioned source-to-target transform, and treat it as auget-to-souree transform. We see that both Gaussian OT and Condo Gaussian KLD learn adaptations that work properly on both source-to-target and target-to-soure.. MMD produces images with appropriate color balances, but with colors applied to inappropriate parts of the image. ConDo MMD produces a good source-to-target image, but its inverse mapping is bad.
Next, attempted color adaptation between the ocean daytime photo and another sunset photo including beach, water, and sky, shown in Figure BB) Here, there is confounded shift, so ConDo utilizes pixels labeled as "sky", "water", or "sand". Now the source-to-targe adapted images have poor (grayish) color balances for the bascline methods, while they have good color balances for the ConDo mothods. For the inverse target-to-source mapping, both MMD and ConDo MMD struggle.
More results and details are given in the Appendix Section C.5
4.6 California housing price prediction
Here, we apply ConDo to an unsupervised domain adaptation setting derived from the California housing datasct (Pace & Barry 1997). We split the data into source and target domains based on the first fcature, Median Income, defining the source domain as being the housing districts with income less than or equal to the median. The geographical results of this source-target split are shown in Figure g S(A), with examples plotted according to their LatLon features. We see that higher-income districts are geographically clustered, and thus use Lat coordinates as the two confounding variables. As our classification task, we predict whether the mean house value in each district exceeds the median over the entire dataset, 1.797 (in $100k). As features, we use all remaining features aside from Median Income.
We train a TabPFN classifier on the source domain, and evaluate on target domain, repeated over 10 random simulations, each with 500 source training samples and 500 target test samples in each simulation.
15

---

Published in Transactions on Machine Learning Research (11/2024)
(A)
(B)
Original Images
Original Images
Day
Sunset
Day
Beach
Gaussian OT
Gaussian
Day Sunset
Day Sunset
Day Beach
Day
Beach
MMD
MMD
Day
Sunset
Day Sunset
Day + Beach
Day
Beach
ConDo segmentation
ConDo segmentation
Day pixel labels
Sunset pixel labels
Day pixel labels
Beach pixel labels
ConDo Gaussian KLD
ConDo Gaussian KLD
Day Sunset
Day Sunset
Day
Beach
Day Beach
ConDo MMD
ConDo MMD
Day Sunset
Day Sunset
Day
Beach
Day
Beach
Figure 7: Image color adaptation results without (A) and with (B) confounded shift. We see that ConDo is non-inferior in (A). In (B), we see that non-ConDo methods produce gray-ish Day - Beach images. Meanwhile, ConDo methods produces light blue sky and yellow clouds in Day + Beach images. ConDo Gaussian KLD is the only method produce normal- lormal-looking images for all four tasks.
We perform location-scale domain adaptation to make the target features resemble the source features before applying the classifier. As shown in Figure B-C), vanilla Gaussian OT and MMD methods make the performance even worse than no adaptation. We also see that the ConDo methods better than the vanilla methods, and that ConDo Gaussian KLD improves upon no adaptation. In the Appendix Section [.6] we show similar results for training classifier on adapted poource-to-ttraee tartarget data, with ConDo outperforming the baselines.
16

---

Published in Transactions on Machine Learning Research (11/2024)
(A)
(B)
(C)
larget
40
No Adaptation
Latituee
38
Geussian OT
MMD
34
ConDo Gaussian KLD
-120
-115
0.0
0.1
0.2
0.3
0.4
0.5
0.6
1.0
0.2
0.4
0.6
Longitude
Accuracy
Figure Results on CA Housing. (A) Source-target split based on median income. Accuracy (B) and Fl-score (C) on target test data, over 10 random simulations, Error bars depict the standard deviation over simulations.
4.7 SNAREseq single-cell multi-omics dataset
We apply domain adaptation to a SNAREseq single-cell dual-omics dataset (Demetci et al. 2022). In the dataset, 1047 cells are co-assayed with RNA-seq for gene expression and ATAC-seq for chromatin accessibility. Each sequenced cell associated with one of four cell types; unique cell identities are also given, so matched sample pairs across the two modalities are known. We treat ATAC-seq assay features as source domain and RNA-seq assay features target domain. For ConDo, we perform adaptation controlling for cell type, the confounding variable.
We first simulate unsupervised domain adaptation in which we perform cell type classification with TabPFN (Hollmann el al. 2022 In each simulation, we have a large source domain training set 500 source cells. We also simulate having small set of C cells for which we have both source and target domain data available use for adaptation; we simulate unseen cells using larget domain data (ATAC-seq features). Because the feature dimensionality differs for the two assays, we only perform MMD and ConDo MMD adaptation with affine transforms. Results for 10 independent random simulations, depicted in Figure g show that ConDo MMD outperforms MMD. Similar advantage for ConDo is shown when classifiers were instead trained on adapted source-to-target features in Figure $16 in Appendix Section
{5, 10, 20, 50, 100). We then evaluate the classifier on the remaining
C.7
0.65
0.45
0.60
Fl-scare
0.40
MMD
ConDa MMD
0.35
0.50
10
20
50
100
10
20
100
Number of pairs
Number of pairs
Figure 9: SNAREseq cell lype classifier performance on test set. Classifier performance (acouracy and Fl-score) is shown function of the number of paired samples available to the domain adaptation methods. The error bars depict the standard error computed over 10 random simulations.
Next, we evaluate whether domain adaptation methods produce data showing desirable clustering patterns. We concatenate the adapted-source and target datasots, and compute the silhouctte score (the mean of the silhouette coefficients over all samples). This silhouette score calculation requires both features and label for each sample; compute silhouette separately for labels defined the assay, the cell type, and the
17

---

Published in Transactions on Machine Learning Research (11/2024)
cell identity. A silhouette score approaching +1 indicates that samples have small distance to other points in the same group (i.e. with the same label), compared to points in the next nearest group (i.e. with the smallest average distance). A score of O indicates overlapping groups, while a score of -1 indicates discordance between the features and labeling. We desire large scores for cell identity labels and cell type labels, and a score close to 0 for assay labels. ConDo improves upon the baseline for all three settings, as shown in Table 2
Table 2: Silhouette scores after combining the adapted ATAC-seq and the RNA-seq data into single dataset with 2 1047 samples. Silhouette scores are computed for different. labelings, given in the column headings. "Cell identity" denotes the pairs of samples from the same cell for two assays; "cell type" denotes samples grouped by cell lype; "assay" denotes samples being labeled as either ATAC-seq or RNA-seq.
Method
Cell identity (1) Cell type (t) Assay (+0+)
MMD ConDo MMD
-0.4556
0.4574
0.0379
-0.4425
0.4940
0.0218
4.8 Gene expression microarray batch effect correction
We analyze performance on the bladderbatch gene expression dataset commonly used to benchmark batch correction methods (Dyrskjøt et al., 20044 Leek 2016). We use all 22,283 gene expressions (i.e. features) from this bladder tissue Affymetrix microarray dataset; bladderbatch was preprocessed so that each feature is approximately Gaussian. We altempt a location-scale transform, as is typical with gene expression batch effect correction. We choose the second largest batch (batch 2, with cancer samples out of 18 total) as the source domain, and the largest batch (batch 5, with 5 cancer samples out of 19 total) as the target domain. The confounder is 1d categorical (cancer or non-cancer).
For each method, compute the silhonette scores of the adapted datasets, with respect to the batch variable (and, in parentheses, the test result variable). We desire the silhouette score to be large for the cancer status, and to be close to 0 for the batch label. Because the cancer fractions are roughly the same (4/18 vs 5/19) for batches 2 and 5, we do not expect to need to account for confounding. Results are shown in Table 3 We see that all domain adaptation methods improve upon no adaptation. For cancer status, MMD has the largest silhouette score as desired. For the batch variable, all adaptations produced slightly negative scores. While this is less concerning than the larger magnitude positive value (0.0884) for no adaptation, may suggest overfitting the silhouette score is closest to zero for ConDo MMD.
We repeat the experiment after inducing confounding by removing samples. For 10 random simulations, we remove half (7) of the non-cancer samples in batch 2, so that batch 2 is 4/11 non-cancerous, while batch 5 remains 5/19 non-cancerous. Results are shown in Table 4 We see that ConDo methods outperform their baseline counterparts, and that ConDo Gaussian KLD in particular produces gene expression data in which variation concords with cancer status rather than batch.
Table Silhouette scores on bladderbatch dataset, on entire dataset without induced confounding. Silhouette scores computed for different labelings, given in the column headings.
Method
Cancer Status (1) Batch (-0+)
No Adaptation Gaussian OT
0.2798
0.0884
0.3008
-0.0279
MMD
0.3123
-0.0214
ConDo Gaussian KLD
0.3121
-0.0195
ConDo MMD
0.3107
-0.0176
18

---

Published in Transactions on Machine Learning Research (11/2024)
Table 4: Silhouette scores on bladderbatch dataset, after inducing confounding in the data. Silhouette scores are computed for different labelings, given in the column headings. We also include the standard errors computed over 10 random simulations.
Method
Cancer Status (t)
Batch (+0+)
No Adaptation Gaussian OT
0.2984 = 0.0009
0.0982 + 0.0010
0.2934 + 0.0006
-0.0329 + 0.0001
MMD
0.2966 = 0.0010
-0.0372 + 0.0001
ConDo Gaussian KLD
0.3212 + 0.0007
-0.0276 + 0.0002
ConDo MMD
0.3196 = 0.0008
-0.0284 + 0.0002
5
Related Work
As far as we are aware, previous work on domain adaptation does not describe or address our exact problem. There is a large body of research in domain adaptation which maps both source and target distributions to a new latent representation where they match (Baktashmotlagh et al. 2013 Yan et al. 2017 Ganin et al.) 2016 Gong et al. 2016). These however cannot achieve data backwardds-omppaiibiii
because they create new latent domain. Other domain adaptation methods are also inapplicable to our setting since they match distributions via reweighting samples (Corles & Mohri 2011
Tachet des Combes el al. 2020 or dropping
features (Kouw et al. 2016).
Prior rescarch exists for performing domain adaptation when both features and label are shiftod, inchuding the generalized label shift (GLS) generalized larget shift. (GeTarS) (Zhang el al. 2013 Rakotomamonjy et al. 2020 Tachet des Combes et al. 2020). However, these methods assume the specific prediction setting where the label is the confounder, and optimize composite objectives that combine distribution matching and prediction accuracy. In our case, the confounder may not be the label of our prediction model of interest, and indeed we may not even be mapping covariates for the purpose of any downstream prediction task. Furthermore, by conditioning on confounders, our framework can handle multivariate confounders or even complex objects which are accessed only via kernels. Landciro et al. introduced the term confounding shift to describe form of GLS/GeTarr but it does not match our confounded shift assumption, since the confounding variables are unobserved. Their method, which comprises confounder dctection and adversarial confounder-robuss classification, is substantially different from our approach.
The most relevant domain adaptation methods for our context perform asymmetric feature transformation, in which source features are adapted to target fcatures, and are thus compatible with genoral-purpose backwards compatibility. EasyAdapl (Daumé III| 2007) and bassydapt++4 (Dauné III el al. 2010) are notably successful approaches performing supervised learning with data from multiple domains, but they do not provide transformed features for data analysis. Furthermore, Adsyt relies on feature concatenation; expect to not have confounders available at inference time, which means that we cannot utilize them in the concatenated features at training time either.
Previous work which explicitly matches conditional distributions Long et al. 2013 instead uses the conditional distribution of the label given the features, rather than our approach of matching the features conditioned on the labels. It also constructs a new latent space, rather than mapping from source to target for backwards compatibility.
Optimal transport has also been proposed framework for domain adaptation (Courty et al. 2014). and within this framework, unbalanced optimal transport has been proposed to address distribution shift (Chizat et al. 2018). This relaxes the OT constraint for conscrvation of mass to a penalty on deviations from mass conservation, yielding improvements in label shift scenarios (Yang & Uhler 2018). However, this approach requires challenging adversarial training (Yang & Uhler 2018), and fails to utilize potentially available side information from confounding variables. Our work is more similar in spirit with optimal transport with subset correspondence (OT-SI) (Liu al. (2019b), which implicitly conditions on categorical confounder (the sample's 's subset) to learn an optimal transport map. Our framework explicitly conditions on confounders
19

---

Published in Transactions on Machine Learning Research (11/2024)
and is thus more general, allowing contimuous, multivariate, and (using kernels) even general objects confounding variables.
6
Discussion
6.1
Limitations
The first main limitation of our framework is that we assume access to all confounders at training time. While our experiments suggest that our approach works well given a superset of the true confounding variables, our experiments suggest that ConDo fails to benefit from receiving a subset of the true confounders. Controlling for partially-latent or fully-latent confounding remains as future work. A second limitation is that we assume a deterministic mapping (and in our conerete implementations, a linear mapping) between feature spaces. It would be nontrivial to exlend our approach to non-deterministic mappings with distributional regression conditional generative modeling. A third limitation is that, even if we are able to learn the ground-truth feature adaptation, we may only use the adapted features for downstream prediction tasks where the conditional distribution of the target variable given features is the same for source and target.
Furthermore, despite our limiting assumptions, both our proposed divergences, the reverse KLD and the MMD, suffer from non-identiffabiiity. Multiple transformations may match the source distribution to the target distribution, and both our objective functions are indifferent among such transforms. We currently rely on gradient flow from the initial parameters to make sensible choice, but this offers no guarantees.
6.2 Future work
Optimal transport (OT) offers principled criteria, minimal transport cost, to choose among transformations which provide equal fit. to the data. This suggests replacing the reverse KLD and MMD with an OT-based alternative, such as the Wasserstein distance. Yet, while minimal transport cost is excellent "prior", is not the only defensible choice. For example, Lp regularization, empirical Bayes weight sharing such as used by ComBat (Johnson et al. (007), and constraints (c.g. non-negativity or zero-off-diagonal for lincar mappings) may instead be preferred, and may be fruitfully combined with KLD and/or MMD.
Due to the affine restriction on the transformation, our proposed approach more appropriate for adaptation settings where source and target correspond to different versions of experimental assays and similar settings where the required adaptation is affine (or even location-scale.. Tt would be useful to examine whether our framework extends gracefully to nonlinear adaptations parameterized by neural networks.
Finally, thus far our analysis of ConDo has been purely empirical. Theoretical analysis would surely be appropriate, particularly before relying on ConDo for statistical inference tasks.
Along with our provided software, we have included code for our experiments, in the hope that these may form a benchmark that accelerates future progress.
7 Conclusion
We have shown that minimizing expected divergences / distances after conditioning on confounders is a promising avenue for domain adaptation in the presence of confounded shift. Our proposed use of the reverse KLD are (lo our knowledge) new in the field of domain adaptation, and may be more broadly useful. Focusing settings where the effect of the confounder is possibly complex, yet where the source-target domains can be linearly adapted, we demonstrated the usefulness of algorithms based on our framework. These experiments show that conditioning on confounders via our ConDo framework improves the quality of learned adaptations for a variety domains and tasks.
20

---

Published in Transactions on Machine Learning Research (11/2024)
References
Agarwal, Rishabh, Vieillard, Nino, Stanezyk, Piotr, Ramos, Sabela, Geist, Matthieu, & Bachem, Olivier. 2023. Gkd: Generalized knowledge distillation for auto-regressive sequence models. arXio preprint 0rXiv:2306.1366 8
Antonsson, Sindri E, & Melsted, Páll. 2024. Batch correction methods used in single cell RNA-sequencing analyses are often poorly calibrated. bioRrio, 2024-03.
Baktashmotlaghh Mahsa, Harandi, Mehrtash T, Lovell, Brian C, & Salamann, Mathieu. 2013. Unsupervised domain adaptation by domain invariant projection. Pages 769-776 of: Proceedings of the IEEE International
Conference on Computer Vision. 19
Blei, David M, Kucukelbir, Alp, & McAuliffe, Jon D. 2017. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518), 859-877. g
Bronstein, Gordon, Naef, Luca. 2024. The Road to Biology 2.0 Will Pass Through Black-Box Data. Towards Data Science.
Burley, Stephen K, Berman, Helen M, Chen, Li, Bhikadiya, Charmi, Bi, Chunxiao, Dutta, Shuchismita, Feng, Zukang, Di Costanzo, Luigi, Ghosh, Sutapa, Christie, Cole, et 2019. Protein Data Bank: the single global archive for 3D macromolecular structure data. Nucleic acids research, 47(11), D520 D528.
Cai, Hao, Li, Xiangyu, Li, Jing, Liang, Qirui, Zheng, Weicheng, Guan, Qingzhou, Guo, Zheng, & Wang, Xianlong. 2018. Identifying differentially expressed genes from cross-site integrated data based on relative expression orderings. International Journal of Biological Sciences, 18() 892. [
Caruana, Rich. 1997. Multitask learning. Machine learning, 28, 41 11-75.
Chen, Dexiong, Fan, Bowen, Oliver, Carlos, & Borgwardt, Karsten. 2022. Unsupervised manifold alignment with joint multidimensional scaling. arXie preprint or arXiv:2207.0000 2
Chizat, Lenaic, Peyré, Gabriel, Schmitzer, Bernhard, & Vialard, Fravgois-Xavier. 2018. Unbalanced optimal
transport: Dynamic and Kantorovich formulations. Journal of Fumctional Analysis, 274(11), 3090 3123. 19
Cortes, Corinna, & Mohri, Mehryar. 2011. Domain adaptation regression. Pages 308- 323 of: International Conference on Algorithmic Learning Theory. Springer. 19
Courty, Nicolas, Flamary, Rémi,, & Tuia, Devis. 2014. Domain adaptation with regularized optimal transport. Pages 274 289 of: Machine Leurning and Knowledge Discovery Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part I 14. Springer. 19
Daumé III, Hal. 2007. Frustratingly Easy Domain Adaptation. Pages 256-263 of: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics.
19
Daume III, Hal, & Marcu, Daniel. 2006. Domain adaptation for statistical classifiers. Journal of artificial Intelligence research, 26, 101-126. 1
Dauné III, Hal, Kumar, Abhishek, & Saha, Avishek. 2010. Frustratingly easy semi-supervised domain adaptation. Pages 33-59 of: Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing.
19
Demetci, Pinar, Santorella, Rebecca, Sandstede, Björn, Noble, William Stafford, & Singh, Ritambhara. 2022. SCOT: single-cell multi-omics alignment with optimal transport. Journal of Computational Biology, 29(1), 3-18.
17 29
Dickinson, Mark. 2010. Generate random numbers summing to predefined value. StackOverflow.
10
Dowson, DC, & Landau, BV. 1982. The Fréchet distance between multivariate normal distributions. Journal
of multivariate analysis, 12(3), 4
21

---

Published in Transactions on Machine Learning Research (11/2024)
Dyrskjøt, Lars, Kruhøffer, Mogens, Thykjaer Thomas, Marcussen, Niels, Jensen, Jens L, Moller, Klaus, & Orntoft, Torben F. 2004. Gene expression in the urinary bladder: common carcinoma in situ gene expression signature exists disregarding histopathological classification. Cancer research, 64(11), 4040 4048.
Fazel, Maryam, Hindi, Haitham, & Boyd, Stephen P. 2003. Log-det heuristic for matrix rank minimization with applications to Hankel and Euclidean distance matrices. Pages 2156 2162 of: Proceedings of the 2003 American Control Conference, 2008., vol. 3. IEFF. g
Feinman, Reuben. 2021. Pylorch-minnmmze:: library for numerical oplimization with aulograd. 9
Flamary, Rémi, Lounici, Karim, & Ferrari, André. 2019. Concentration bounds for linear monge mapping estimation and optimal transport domain adaptation. arXiv preprint arXi:: 10155. 4
Flamary, Rémi, Courty, Nicolas, Gramfort, Alexandre, Alaya, Mokhtar Z., Boisbunon, Aurelie, Chambon, Stanislas, Chapel, Laetitia, Corenflos, Adrien, Fatras, Kilian, Fournier, Nemo, Gautheron, Léo, Gayraud, Nathalie T.H., Janati, Hicham, Rakotomamonjy, Alain, Redko, levgen, Rolet, Antoine, Schutz, Antony, Seguy, Vivien, Sutherland, Danica J., Tavenard, Romain, Tong, Alexander, & Vayer, Titouan. 2021. POT:
Python Optimal Transport. Journal of Machine Learning Research, 22(78), 1 8. 15 27
28
Ganin, Yaroslav, Ustinova, Evgeniya, Ajakan, Hana, Germain, Pascal, Larochelle, Hugo, Laviolette, François, Marchand, Mario, & Lempitsky, Victor. 2016. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1), 2096 2030. 19
Girshick, Ross, Donahue, Jeff, Darrell, Trevor, & Malik, Jitendra. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. Pages 580-587 of: Proceedings of the IEEE conference on computer vision and pattern recognition.
Gong, Mingming, Zhang, Kun, Liu, Tongliang, Tao, Dacheng, Glymour, Clark, & Schölkopf, Beruhard. 2016. Domain adaptation with conditional transferable components. Pages 2839-2848 of: International conference on machine learning. PMLR. 19
Gordon, Claire C, Blackwell, Cynthia L, Bradtmiller, Bruce, Parham, Joseph L, Barrientos, Patricia, Paquette, Stephen P, Corner, Brian D) Carson, Jeremy M, Venezia, Joseph C, Rockwell, Belva M, et al. 2014. 2012 anthropometric survey of us army personnel: Methods and summary statistics. Army Natick Soldier Research Development and Engineering Center MA, Tech. Rep. 13
Gregori, Josep, Villarreal, Laura, Méndez, Olga, Sánchez, Alex, Baselga, José, & Villamueva, Josep. 2012.
Batch effects correction improves the sensitivity of significance lests in spectral counting- based comparative discovery proteomics. Journal of Proteomics, 75(13), 3938-3951.
Haghverdi, Laleh, Lun, Aaron TL, Morgan, Michael D, & Marioni, John C. 2018. Batch effects in single-cell RNN-sequencing data are corrected by matching mutual nearest neighbors. Nature biotechnology, 36(5), 421-427.
Hicks, Stephanie C, Townes, F William, Teng, Mingxiang, & Trizarry, Rafael A. 2018. Missing data and technical variability in single-cell RNN-ssqqeeccig experiments. Biostatistics, 19(4), 562 578. [
Hollmann, Noah, Müller, Samuel, Eggensperger, Katharina, & Hutter, Frank. 2022. Tabpfu: A transformer that solves small tabular classification problems in a second. arXiv preprint arXix:2207.00 01848.
10
Huang, Jiayuan, Gretton, Arthur, Borgwardt, Karsten, Schölkopf, Bernhard, & Smola, Alex. 2006. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems, 19.
2
Johnson, W Evan, Li, Cheng, & Rabinovic, Ariel. 2007. Adjusting batch effects in microarray expression
data using empirical Bayes methods. Biostatistics, 8(1), 118 127.
2 20
Jolicoeur-aartnnaau, Alexia, Fatras, Kilian, & Kachman, Tal. 2024. Generating and imputing tabular data via diffusion and flow-based mandient-boosted trees. Pages 1288-1296 of: International Conference on Artificial Intelligence and Statistics. PMT.R.
22

---

Published in Transactions on Machine Learning Research (11/2024)
Jumper, John, Evans, Richard, Pritzel, Alexander, Green, Tim, Figurnov, Michael, Ronneberger, Olaf, Tunyasuvunakool, Kathryn, Bates, Russ, Zidek, Augustin, Potapenko, Anna, et al. 2021. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873),
Kappen, Hilbert J, Gómez, Viceng, & Opper, Manfred. 2012. Optimal control a graphical model inference problem. Machine learning, 87(2), 159-182, g
Ke, Guolin, Meng, Qi, Finley, Thomas, Wang, Taifeng, Chen, Wei, Ma, Weidong, Ye, Qiwei, & Liu, Tie-Yan, 2017. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30.
26
Knott, Martin, Smith, Cyril 1984. On the optimal mapping of distributions. Journal of Optimization Theory and Applications, 43(1), 39 49,
Kouw, Wouter M, Van Der Maaten, Laurens JP, Krijthe, Jesse H, & Loog, Marco. 2016. Feature-level domain
adaptation. The Journal of Machine Learning Reseorch, 17(1), 5943-5974. 4.
2 19
Kull, Meelis, & Flach, Peter. 2014. Patterns dataset shift. In: Pirst International Workshop on Learning over Multiple Conteats (LMCE) at ECML-PKDD.
Landeiro, Virgile, Tran, Tuan, & Culotta, Aron. 2019. Discovering and controlling for latent confounds in text classification using adversarial domain adaptation. Pages 208-305 of: Proceedings of the 2019 SIAM International Conference on Dala Mining. SIAM.
19
Lock, Jeffrey T, Scharpf, Robert B, Bravo, Héctor Corrada, Simcha, David, Langmead, Benjamin, Johnson, W Evan, Geman, Donald, Baggerly, Keith, & Trizarry, Rafael A. 2010. Tackling the widespread and critical impact of batch effects in high-throughput data. Nature Reviews Genetics, 11(10), 733-739. 1
Leek, JT. 2016. bladderbatch: Bladder gene expression data illustrating batch effects. R package version, 100). 18
Levine, Sergey. 2018. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arxi:: 009.00 S
Lin, Chih-Jen, & Jorge, J. 1999. Moré. Newton's method for large-scale bound constrained problems. SIAM Journal on Optimization, 9(1100-1127), 10-1137. g
Liu, Jie, Huang, Yuanhao, Singh, Ritambhara, Vert, Jean-Philippe, & Noble, William Stafford. 2019a. Jointly embedding multiple single-cell omics measurements. In: Algorithms in bioinformaties:.... International Workshop, WABI..,, proceedings. WABI (Workshop), vol. 143. NIH Public Access.
Liu, Ming- Yu, & Tuzel, Oncel. 2016. Coupled generative adversarial networks. Advances in neural information
processing systems, 29. 2
Liu, Ruishan, Balsubramani, Akshay, & Zou, James. 2019b. Learning transport cost from subset correspon- dence. In: International Conference on Learning Representations. 19
Long, Mingsheng, Wang, Jianmin, Ding, Guiguang, Sun, Jiaguang, & Yu, Philip S. 2013. Transfer feature learning with joint distribution adaptation. Pages 2200 2207 of: Proceedings of the IEEE international conference on computer vision.
19
Loshchilov, Ilya, & Hutter, Frank. 2017. arXim:1711.05101
Decoupled weight decay regularization. arXiv preprint
10 28
Mallasto, Anton, & Feragen, Aasa. 2017. Learning from uncertain curves: The 2- Wassersteei metric for Gaussian processes. Advances in Neural Information Processing Systems, 30. 4
Muzellec, Boris, & Cuturi, Marco. 2018. Generalizing point embeddings using the wasserstein space of elliptical distributions. Advances in Newral Information Processing Systems, 31.
23

---

Published in Transactions on Machine Learning Research (11/2024)
Pace, R Kelley, & Barry, Ronald. 1997. Sparse spatial autoregressions. Statistics 8 Probobility Letters, 33(3), 291-297. 15
Pan, Sinno Jialin, Tsang, Ivor W, Kwok, James T, & Yang, Qiang. 2010. Domain adaptation via transfer component analysis. IEFE transactions on neural networks, 22(2), 199 219-210. 2
Pedregosa, Fabian, Varoquaux, Gaël, Gramfort, Alexandre, Michel, Vincent, Thirion, Bertrand, Grisel, Olivier, Blondel, Mathieu, Prettenhofer, Peter, Weiss, Ron, Dubourg, Vincent, et al. 2011. Scikit-learm: Machine learning in Python. the Journal of machine Learning research, 12, 2825-2830.
Pelletier, Simon J. Leclereq, Mickaël, Roux- Dalx-Dalvai, Florence, de Geus, Matthijs B, Leslic, Shannon, Wang, Weiwei, Lam, TuKiet. T, Nairn, Angus C, Arnold, Steven E, Carlyle, Becky C, et al. 2024. BERNN: Enhancing classification of Liquid Chromatography Mass Spectrometry data with batch effect removal neural networks. Nature Communications, 15()), 3777. [
Peyré, Gabriel, Cuturi, Marco, et al. 2019. Computational optimal transport: With applications to data science. Foundations and Trendst in Machine Learning, 11(6-)) 355 607.
Radua, Joaquim, Victa, Eduard, Shinohara, Russell, Kochunov, Peter, Quidé, Yann, Green, Mclissa J, Weickert, Cynthia S, Weickert, Thomas, Bruggemann, Jason, Kircher, Tilo, et al. 2020. Increased power by harmonizing structural MRI site differences with the ComBat batch adjustment method in ENIGMA.
Neuroimage, 218, 116956. 2
Rakotomamonjy, Alain, Flamary, Rémi, Gasso, Gilles, Alaya, Mokhtar Z, Berar, Maxime, & Courty, Nicolas. 2020. Match and reweight strategy for gencralized target shift. arXiv preprint Xix:2200 08161. 19
Shafieezadeh Abadeh, Soroosh, Nguyen, Viet Anh, Kuhn, Daniel, & Mohajerin Esfahani, Peyman M. 2018. Wasserstein distributionally robust Kalman filtering. Advances in Neural Information Processing Systems, 31.1
Shaham, Uri, Stanton, Kelly P, Zhao, Jun, Li, Huamin, Raddassi, Khadir, Montgomery, Ruth, & Kluger, Yuval. 2017. Removal of batch effects using distribution setebbitioo-mmthhingg residual networks. Bioinformatics, 336),, 2539-2530 2546. I
Singh, Ritambhara, Demetei, Pinar, Bonora, Giancarlo, Ramani, Vijay, Lee, Choli, Fang, He, Duan, Zhijun, Deng, Xinxian, Shendure, Jay, Disteche, Christine, et al. 2020. Unsupervised manifold alignment for single-cell multi-omics data. Pages 1-10 of: Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics.
99
Sohl-Dickstein, Jascha, Weiss, Eric, Maheswaranathan, Niru, & Ganguli, Surya. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. Pages 2256 2265 of: International conference on machine learning. PMLR. g
Sprang, Maximilian, Andrade-Navarro, Miguel A, & Fontaine, Jean-Fred. 2022. Batch effect detection and correction in R.NA-sq data using machine-learning-bor automated assessment of quality. BMC bioinformatics, 23(upppl 6), 279.
Tachet des Combes, Remi, Zhao, Han, Wang, Yu-Xiang, & Gordon, Geoffrey J. 2020. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Information Processing Systems, 33, 19276 19289.
5 19
Torbati, Mahbanch Eshaghzadeh, Minhas, Davnoct S, Ahmad, Ghasan, O'Connor, Erin E, Muschelli, John, Laymon, Charles M, Yang, Zixi, Cohen, Ann D, Aizenstein, Howard J, Klink, William E, et al. 2021. A multi-scanner neuroimaging data harmonization using RAVEL and ComBat. Neuroimage, 245, 118703.
Tzeng, Eric, Hoffman, Judy, Sacnko, Kate, & Darrell, Trevor. 2017. Adversarial discriminative domain adaptation. Pages 7167-7176 of: Proceedings of the IEEE conference on computer vision and pattern
recognition. 2
24

---

Published in Transactions on Machine Learning Research (11/2024)
Van Buuren, Stef, Boshuizen, Hendrick C, & Knook, Dick L. 1999. Multiple imputation of missing blood pressure covariates in survival analysis. Stalistics in medicine, 18(6), 681-694 8
Wang, Chang, & Mahadevan, Sridhar. 2009. Manifold alignment without correspondence. Page of: IJCAI, vol. 2. 2
Wilson, Samuel Von, Cebere, Bogdan, Myalt, James, & Wilson, Samuel. 2022 (Dec.). AnolherSam Wil- son/miceforest: Release for Zenodo DOL.
26
Wu, Xiaofu, Zhang, Suofei, Zhou, Quan, Yang, Zhen, Zhao, Chumming, & Latecki, Longin Jan. 2021. Entropy Minimization Versus Diversity Maximization for Domain Adaptation. IEEE Transactions on Newral Nelworks and Learning Systems.
Yamashita, Ayumu, Yahata, Noriaki, Itahashi, Takashi, Lisi, Giuseppe, Yamada, Takashi, Ichikawa, Naho, Takamura, Masahiro, Yoshihara, Yujiro, Kunimatsu, Akira, Okada, Naohiro, et al. 2019. Harmonization of resting-state functional MRI data across multiple imaging sites via the separation of site differences into sampling bias and measurement bias. PLoS biology, 17(4), e3000042.
Yan, Hongliang, Ding, Yukang, Li, Peihua, Wang, Qilong, Xu, Yong, & Zuo, Wangmeng. 2017. Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaplation. Pages 2272 2281
of: Proceedings of the IEEE conference on computer vision and pattern recognition. 5 19
Yang, Karren D, & Uhler, Caroline. 2018. Scalable unbalanced optimal transport using generative adversarial networks. arXiv preprint Xrv: 11447. 19
Zhang, Kun, Schölkopf, Bernhard, Muandet, Krikamol, & Wang. Zhikun. 2013. Domain adaptation under target and conditional shift. Pages 819 827 of: International Conference on Machine Learning. PMLR.
Zhang, Yuqing, Parmigiani, Giovanni, & Johnson, W Evan. 2020. ComBat-seq: batch effect adjustment for RNA-seq count data. NAR genomics and bioinformatics, 2(3), 1qaa078. 2
Zindler, Tristan, Frieling, Helge, Neyazi, Alexandra, Bleich, Stefan, & Friedel, Eva. 2020. Simulating ComBat:
how batch correction can lead to the systematic introduction of false positive results in DNA methylation microarray studies. BMC bioinformatics, 21, 1-.15 !
25

---

Published in Transactions on Machine Learning Research (11/2024)
A Exact solution for 1d reverse KL divergence
For each nth sample (1 S S N) drawn from the prior distribution over the confounding variable, we have obtained an estimate of its mean and variance for the source domain (Hs.. 0 and target domain (KT,, . Then the KL objective is the following:
N
(mps, +
- MTT))
arg min
log(m)
(14)
20
204.
nE1
arg min 7I,
log(m) (20T.)) +mmmmm2 m² + (mus. b pT,,)²
(15)
Setting the partial derivative wrt b 0, we have:
N
N
6"
PT,)
m
Us ) FT -mps.
(16)
n=1
R=1
Substituting this into our objective, we have
N (2g +m²o3, + (mps.. + PTT - māg pT,) 2
arg miin
-
(17)
121
Setting the derivative wrt m to O, we obtain the following quadratic equation:
0 = -=
1 [ )o.)) + 1 Cus. Tis))) m² +
-(Fr-Pr PT, Mus. - Tis) m + Σ of
(18)
We then apply the quadratic formula, choosing the positive solution.
B Implementation details and computational complexity analysis
Recall that we have Ng and Nr source and target samples, respectively; let use Ng and Ny to denote the unique values of the source and larget confounders. Let N = max(Ns, Nr) and Ñ = max(Ns, Nr). We use M = (MM,
as the feature dimension, and let the complexity of evaluating kzz (2(2(m)), e(na)) be OCC). Recall that Kx is the number of multiple imputations per Z value. Also recall that K₂ the number of sampled values used per ConDo-MMD minibatch.
The complexity of computing the product prior weights is O(CN²). We achieve this speedup by, rather than assigning weights to cach sample, instead assigning weights to each unique value of the confounder and multiplying the weights from Eq. 9 with the number of observations of each unique value.
When confounding variables are discrete, conditional sampling has negligible runtime cost. When confounders are continuous, the runtime is dominated by MICE-Forest (Wilson el al. 2022) training Light (Ke et al, 2017) models for each feature given all the other features. The runtime of each LightGBM model is O(NM), we run the MICE imputation algorithm for fixed number (2) of iterations, and generate Kx imputations, the total runtime complexity for this step M(N
The space complexity of conditional sampling is potentially burdensome. Compared to input datasets of size Ng x Ms and Ny x Mr, we generate datasets of size KxN x Ms and Xx x Ms, for source and target, respectively. In our provided software, we implement an option to subsample a fraction F < of N samples at each epoch before performing conditional sampling, reducing datasets to size FKx x Mg and FKxNs x Ms.
For ConDo-KLD, we compute means and (inverse) covariances for each confounder value. Computing these statistics from the conditionally sampled data, for continuous confounders. has complexity O M2
26

---

Published in Transactions on Machine Learning Research (11/2024)
Computing these from raw data, for discrete confounders, has complexity O(NM² + M Computing the ConDo-KL.D objective at each Newton conjugate gradient iteration has complexity O(N
Evaluating ConDo-MMD on single minibatch has complexity M)) this is substantially less than that of full-batch optimization, which would be O(NN2M).
C Additional Material for the Experiments
C.1 Synthetic 1d feature with 1d continuous confounder
In the source domain, the confounder - (n,form|(48) 8]: in the target domain the confounder
(m) Uniform[0, 8). For the homoscedastic scenario, X ~ ~N (4 (1) For the heteroscedastic sccnario, X(a) N 4Z(") + 1, 1. (Z(m) + 1)²). For the nonlincar scenario, X()) N( ((|/ReLU(Z(n) - 5)]² +
I, ((ReLU(Z(m) 5)]² + 1)²). Then, for the source domain, have x 2x 5; for the target domain, we have x = (n)
In these experiments, we used the default hyperparameters, with the exception using learning rate 0.01 for 100 epochs, for both MMD and ConDo-MMD.
In Figure ST show the error in estimation of the feature alignment parameters (A,b) ConDo outperforms the baselines when target shift is present, and is not systematically worse when shift only feature shift is present.
We show train errors in Figure 52 with-and-withhut label shift, with-and-without feature shift, and with- ant-without additional noise, for a total of 8 settings.
C.2 Synthetic 1d feature with multi-dimennsioal continuous confounders
In all these experiments, we used the default hyperparameters, with the exception of using learning rate 0.01 for 100 epochs, for both MMD and ConDo-MMI.
In Figure S3
we show the rMSE on the training set for multi-dimension contiguous confounder experiments.
In Figure S4 we show the effect of running ConDo with partially-observed confounding variables. We see that, on the one hand, ConDo is typically non-inferior to the baselines when confounders are partially observed. On the other hand, ConDo performance quickly degrades as the number observed confounders reduced, so that when half of the confounding variables are observed, ConDo not better than the baselines.
C.3 Synthetic 1d and 2d features with 1d categorical confounder
In all these experiments, we used the default hyperparameters, with the exception of using learning rate 0.01 for 100 epochs, for both MMD and ConDo-MMD.
C.3.1 Synthetic 1d feature with 1d categorical confounder
Figure 55 provides additional results for our experiments with 1d feature confounded by Id categorical confounding variable. We see that both ConDo methods match the true distribution better than the baseline methods. In particular, as sample size increases, ConDo MMD) converges towards the true target distribution. We confirm this Figure S6 showing that ConDo leads to lower-error estimates of the true mapping, and that ConDo MMD in particular has decreasing error as function of sample size.
C.3.2 Synthetic 2d features with 1d categorical confounder
The procedure for generating data in this scenario is adapted from the Python Optimal Transport library Fllamary et al. 2021 The batch-effected source data are generated first, with N = 200 split between two
27

---

Published in Transactions on Machine Learning Research (11/2024)
circles centered at (0.0) and (0, the points are distributed with angle distributed iid around the circle from [0, 2x], and with radius sampled iid from (0, 1).
Next, we analyze the performance of our approach on 2d features requiring an affine transformation. We also use this setting to assess the downstream performance of classifiers which are fed the adapted source-to-target features. The synthetic 2d features, before the batch effect, form a slanted "8" shape, shown in blue/ /green Figure ST A linear classifier separating the upper and lower loops depicted in cyan, for both the source and target domains.
Our results are shown in Figure S7 On the left column, we compare methods in the case where there is no confounded shift. (This setting is from Python Optimal Transport (Flamary et al. 2021).) In the middle column, we have induced a confounded shift: One-fourth of the source domain samples come from the upper loop of the "g", while half of the target domain samples come from the upper loop. This allows us assess the affects of confounded shift on downstream prediction of the confounder (up-vs-down), as well a non-confounder (leit-v--right). In the right column, we have induced confounded shift before, while making the true target transform more challenging, by using randomly generaate true affine transform. The affine transform matrix is set to be random yet positive-determinaat via
Cos Φ1
- sin Φ1
dj cos Φz
- sin Φ2 cos Φ
A
sin Φ1
cos Φ1
d2
sin 02
Φ 02 ~ Unif[ n, T].
d1. d2 Uniff0.5, 1.5].
As shown in Figure S7 all methods perform similarly where there is no confounded shift (left column), but the vanilla domain adaptation approaches fail in the presence of confounding (center and right columns).
In Figure SS we show performance on estimating the feature mapping parameters. ConDo outperforms the corresponding baselines.
C.4 ANSUR II anthropometric survey data
In all ANSUR II experiments, we used the hyperparameters given in the main text. We used the default. parameters for TabPEN. In Figure 59 we show results for the same experimental setup as Figure G except that TabPFN prediction models are instead trained on adapted source- target features, then applied on target features. We see the same pattern, with ConDo methods outperforming their baseline counterparts, especially for the affine transformation scenario.
show feature-space mapping parameter estimation performance. In this setting, MMD, ConDo Gaussian KLD, and ConDo MMD perform equally well. Meanwhile, Gaussian OT performs poorly on the affine transform scenario, while outcompeting the others on the location-scale transform scenario.
In Figure S10
In Figure S12 we show results for larget feature data simulated with no feature shift (A = la b - 0), but with male-female split of 75%-25% in source and 25%-75% in target. As before, we show domain adaptation methods attempting to learn affine and location-scale feature transformations. In Figure [S11] show results for target feature data simulated with no shift in the confounder distribution (probability of sampling male and female each set to 0.5 for both source and target), but with domain adaptation methods attempting to learn affine and location-scale feature transformations. In Figure [S13] we show results for target feature data simulated with no change in the shift in the confounder distribution and with no feature shift, but with domain adaptation methods attempting to learn affine and location-scale feature transformations.
C.5 Image color adaptation
The day photo is 669 x 1000, leading to a dataset of size 669,000 x 3 (for 3 RGB channels). The sunset photo is 750 x 1000, while the beach photo is 723 964.
Both MMD and ConDo MMD are run for 100 epochs with the AdamW optimizer (Loshchilov Hutter 2017 with learning rate of 10-2 and weight decay of 10-4. We use batch size of 8; each "sample" in the batch
28

---

Published in Transactions on Machine Learning Research (11/2024)
consists of the MMD loss over paired 20-pixel datasets. Each epoch is defined as single pass-throught over randomly generated 0.01 min(669 1000, 750 x 1000) unique paired 20-pixel datasets (for the day-sunset task, and analogously for the other task). This randomly-generated dataset of datasets is reused across epochs.
In Figure $14 we see that the ConDo methods do better on Sunset - Beach, producing a brighter image, particularly for the clouds and the horizon. We that both MMD and ConDo MMD fail to produce good inverse mappings.
C.6 California housing price prediction
The California Housing dataset has 20,640 samples (before subsampling as described in the main text), and the classifier receives input features (with median income removed). In all experiments, we used the hyperparameters given in the main text. We used the default parameters for the TabPFN classifier.
In Figure S15 we repeat the previous experiment, except that TabPFN prediction models are instead trained on adapted source-to-target features on the train-set; then the model provided target test-set features directly. We see the same pattern as before, with ConDo methods outperforming their basclinc counterparts.
C.7 SNAREseq single-cell multi-omics dataset
For our experiment with SNAREseq data (Demetci et al. 2022), the ATAC-seq dataset has 19 features, and the RNA-seq dataset has 10 features. The data was preprocessed (Demetci et al.) 2022), inchuding unit normalization as in (Demetci et al. 2022). In all SNAREseq experiments, we used the hyperparameters given in the main text.
In Figure ST6 we show results for the setting where classifiers were instead trained adapted source-to-targel features. In other words, using the paired samples, learned transformation from source domain to target domain. We then applied this transformation to the 500 source domain training samples, and fitted the classifier on these transformed features. Finally, we applied the classifier to (untranslormedl target domain features, and then evaluated predictions.
C.8 Gene expression microarray batch effect correction
In all bladderbatch experiments, we used the hyperparameters given in the main text, with the exception of using learning rate 0.01 for 100 epochs, for both MMD and ConDo-MMD.
29

---

Published in Transactions on Machine Learning Research (11/2024)
NoiseFree- NoiseFrreeTargettShtt Stitttrrhhtt
Homoeceedasiic Lineer
Heteroscedosiic Linear
Homorcedasiic Linear
Heter oscedosciiiiiic Linear
Nontneerr
ineor
30
80
20
10
0.4
0.4
14-A1
14-41
0.2
0.2
NoiseFree- TwaarrreeTrageShii-oooeeeuussss
Homascadastic
Hiteroscedastic Linear
Homoscadastic
Histrosccdastic
Linear
Linear
30
20
200
10
10
1.5
1.0
14-A)
NolseFrree-No Fortutth
arrgethhuttFFFssurShhit
FeatureShift
Hamoscedastic
Hoteroscedastic Linear
Homoscedastic
Heteroscedastic
Nonlinaar
Linear
Nortinear
0.15
0.10
0.10
0.05
0.05
0.00
0.00
NoiseFree-No TargetShitt--Me Stw.iitttaeeeeeSttheeeee
Noiey-NoTargetShin Stteetthhit--FFattueShhht
Homoscedastic
Heteroscedastic Linear
Homorcedastic
Heteroscedastic
Linear
Nonlinear
Linear
Linear
Non
IA.A1
Saussian OT ConDoo MMD
0.2
Parameter estimation for transform data with a continuous confounder. We depict the error in estimating and A, respectively, with the Le vector norm and induced Le matrix norn (spectral norm). The error bars in the barplots depict standard deviations the 10 random simulations.
Figure S1:
30

---

Published in Transactions on Machine Learning Research (11/2024)
NoiseFree TargetShift FeatureShift
Noisy TargetShift FeatureShift
10.0
7.5
CULLA
MSSE
5.0
Homo- scedasto
Helero- scedaslic
Nonlinear
NoiseFree
TargetShift NoFeatureShift
Noisy TargelShift NoFeaaturoShift
WILLLL
AMSE
NoiseFree NoTargetShift FeatureShift
Noisy No TargettShift FeatureShift
L
IMSE
NoiseFree No TarreettSiit NoFeatureShift
Noisy NoTargetShii NoFeatureShift
MMSE
Gaussian DT
MMD
ConDo Gaussian KLD
ConDo MMD
Figure S2: Train errors for experiment with synthetic 1d features with 1d continuons confounder. For each adaptation method, we compute the rMSE true target feature values vs inferred target feature values after adaptation, averaged over 10 simulations.
31

---

Published in Transactions on Machine Learning Research (11/2024)
(A)
10
10
source Gaussian OT
<MSE
source MMD
source ConDo Gaussian KLD
source ConDo MMD
16
32
16
16
32
# confounders
confounders
confounders
(B)
10
Gaussian
rMSE
source MMD
source ConDo Gaussian KLD
source ConDo MMD
1
16 32
1
32
# confounders
confounders
confounders
Figure S3: Results on training data for transforming 1d data with multiple continuous confounders, with extra irrelevant ((,0 1) confounders, shown in (A), and with noisy additive docomposition, shown in (B). The rMSEs are averaged over 10 random sinulations are shown for training data (200 samples per simulation). The columns, in order, correspond to confounder with a linear homoscedastic effect, confounder with linear heteroscedastic effect, and confounder with a nonlinear heteroscedastic effect.
32

---

Published in Transactions on Machine Learning Research (11/2024)
(A) 2 confounders
10
source Gaussion OT
10
source HMD
ConDo Gaussian ConDo
16
18
contounders
contoundors
(B) confounders
10
10
Gaussian
source MMD
10
Con Guussian KLD
ConDo NMD
16
16
32
32
confounders
observed
confounders
(A) 8 confounders
10
1492
source ConDo Gaussian KLD source ConDo NMD
sbsorved confounders
oeserved contounders
observed confoundors
(B) 16 confounders
source Gaussinn OT MMD
10
observed confourders
observed confounders
observed confoundors
B) 32 confounders
10
source Gaussian OT
30
source MMD
ConDo Gaussian KLD
obsorved confounders
observed confounders
observed confounders
Figure S4: Results for transforming ld data with multiple continuous confounders with noisy additive decomposition, while varying both the number of actual confounders and the number provided to ConDo methods. In each row, we show results for a given number of actual confounders; within each plot, we show ConDo performance as a function of how many were provided to ConDo. The rMSEs are averaged over 10 raudom simulations are shown for heldout test. data (100 samples per simulation). The columns, order, correspond to confounder with linear homoscedastic effect, confounder with linear heteroscedastic effect, and confounder with nonlincar heterosccdastic effect.
33

---

Published in Transactions on Machine Learning Research (11/2024)
(A)
(B)
(C)
20
15
target
(unoboerved)
reature
source batch effected
source Gaussian OT
source ConDo Gauss KLD
ConDo MMO
hotdog
not, hot_ho
hotdog
not, hotdog
hotdog
not, hotdoo
confounder
contounder
confounder
(D)
(E)
(F)
target
true (unobserved)
festure
feature
10
festure
source Gaussian OT
5
Gaussian
source ConDo NMD
hotdog
not hotdog
hotdog
not, hotdog
hotdog
not hot_hotdog
confounder
confounder
confounder
Figure $5: Additional results for 1d feature confounded by 1d category. Results are shown (A) N 10, (B) N 20, (C) N 50, and (D) N 100, (E) N 200, and (F) N 500 samples in each of the source and target training sets. We depict the original, latent, shifted, and adapted data, for each value of the categorical confounder.
N=10
N=20
N=50
N=100
N=200
N=500
Gaussian OT
MMD
ConDo Gaussian KLD ConDo
5
5
5
5
||b BI|=
|
B||2
bb-b|p
|b
BI2
|b B1
| BII
Gaussian OT
MMD
ConDo Gaussian KLD ConDo MMD
0.2 IA-A
0.2
0.2
0.2
0.2
0.2
IA-A]2
IA-A|J:
||A-Al|a
||A-A|I
|A-A|:
Figure S6: Parameter estimation performance for transform of 1d data with categorical confounder. We depict the error in estimating and A, respectively, with the L2 vector norm and induced Lg matrix norm (spectral norm). The error bars in the barplots depict standard deviations over the 10 random simulations, Columns correspond to varying sample sizes.
34

---

Published in Transactions on Machine Learning Research (11/2024)
Confounding
Confounded
Confounded Random Wransoorm
rMSE: 3.85 (3.84)
riSE: 3.56 (3.85)
rMSE: 2.91 (2.91)
(0.14) ACC: 0.84 (0.84)
IMSE: (1.07)
Acc: 0.92 (0.71)
Acc: 0.60 (0.58)
target
source: batch-effected
source: true (unabserved)
Gaussian OT
IMSE: 109 (1.08) 0.75 (0.73)
iMSE: 0.89 (0.91)
IMSE: 0.89 (0. 0.66 (0.60)
(0.67)
target
source: batch-effected
4
(unabserved)
source: MMD
0
iMSS: 0.21 (0.20)
IMSE: 0.22 (0.23)
IMSE: 0.10 (0.11)
8 Acc: 0.85 (0.83)
Acc: 0.92 (0.75)
Acc: 0.92 (0.75)
target
batch-effected
source: true (unotserved)
source: ConDo Gaussian KLD
(0.82)
(0.74)
target
source: batch-effected
source: true (unabserved)
source: ConDo MMD
Figure S7: Results of affine transform of 2d data with categorical confounder. We print the rMSE and accuracy on the training data (and on heldout test data parentheses). These values are the result of averaging over 10 random simulations, while the plot is generated from the final simulation.
No Confounding
Confounded
Confounded Random Transform
Gaussian OT
ConDo Gaussian KLD
ConDo MMD
bb-bjp
Gaussian
MMD
Gaussian
ConDo MMD
2
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
[A-AI:
[A-Ap
[A-A]
Figure S8: Parameter ostimation for affinc transform of 2d data with a categorical confounder. We depict the error in estimating b and A, respectively, with the L2 vector norm and induced L2 matrix norm (spectral norm). The error bars the barplots depict standard deviations over the 10 random simulations.
35

---

Published in Transactions on Machine Learning Research (11/2024)
(A)
Gender
Height 5ft6in
Gaussian OT
ConDo Gaussian KLD
ConDo MMD
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Accuracy
Accuracy
(B)
Gender
Height 5ft6in
Gaussian OT
ConDo MMD
0.0
0.2
0.6
0.8
1.0
0.0
0.4
0.8
Accuracy
Accuracy
Figure S9: Results on ANSUR II, for the setting where TabPEN prediction models are trained on adapted caurce-to-targel features and applied to target features, for affine transformation (A) and location-scale transformation (B).
(A)
(B)
Gaussian OT
Gaussian OT
MMD
MMD
ConDo Gaussian KLD
ConDo Gaussian KLD
ConDo MMD
ConDo MMD
2
10
Ilb-511,
IA IA-AlI
lb lb-51,
IA-AII,
Figure S10: Feature mapping parameter estimation performance on ANSUR II, for affine (A) and location- scale (B) transforms. We depict the error in estimating and A, respectively, with the L2 vector norm and induced Le matrix norm (spectral norm). The error bars in the barplots depict standard deviations over the 10 random simulations.
36

---

Published in Transactions on Machine Learning Research (11/2024)
(A)
(B)
Gender
Height Stt6in
Gaussian OT
NMD
Source
Condo Gaussian KLD
Target (ere-fatuure shift
Condo MMD
Target (posureesturee
Denaty
MMD
ConDoo Gaussian KLD
ConDo MND
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
Accuracy
(C)
(D)
Gender
Height 5ft6in
Gaussian OT
MMD
Source
ConDoo Geunsier KLD
Target (pre-feature shitt
(post-feature
Donatty
ConDo Gaussian KLD
ConDo MND
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
1.0
Figure S11: Results on ANSUR II, where the data is generated with only feature shift, for affine transformation (A-B) and location-scale transformation (C-D).
37

---

Published in Transactions on Machine Learning Research (11/2024)
(A)
(B)
Height 51t6in
Gauseian DT
HMD
ConDo Goursian KLD
Target (pre-feature
ConDo MMD
Target (post-fecture shift
Dondky
Caussion OT
15
MND
ConDio KLD
0.50 Accuracy
0.75
0.25
Accuracy
IMSE
Gender
Height 5ft6in
200
MMD
Gauasien
ConDDo Gaussian KLD
Target (pre
130
ConDo MXD
Target (esst-festure shitt
Dendit
Gaussian
100
MND
ConDo Gaussian KLD
50
0.50 Accuraty
0.00
0.25
0.50
0.75
Accuracy
<MSE
Figure $12: Results on ANSUR II, where the data generated with no feature shift but shift the confounding gender distribution, for affine transformation (A-B) and location-scale transformation (C-D). Relative rMSE is not available, because pre-adaptation rMSEs are all zero, raw rMSE is depicted instead.
(A)
(B)
Gender
Height Sft6in
50
Geussian OT
MMD
Target (pre-feature
Source
40
ConDo Gaussian KLD
ConDo MMD
Target (pcst-festure shift
Denaly
30
Gaussion
20-
MND
ConDo Gaussian KLD
10
Accuracy
Accuracy
IMSE
DD)
Gender
Height Sfttin
Geussian OT
MMD
Source
ConDo Gaussian KLD
Target
200
ConDo MMD
Target (post-feature
Denaity
150
Gaussion
MND
100
ConDoo Gaussian KLD
ConDo MMD
0.2
0.00
0.02
Accuracy
Accuracy
IMSE
Figure S13: Results on ANSUR 11, where no shift at all takes place, for affine transformation (A-B) and location-scale transformation (C-D).
38

---

Published in Transactions on Machine Learning Research (11/2024)
Original Images
Sunset
Beach
Gaussian OT
Sunset
Beach
Sunset
Beach
MMD
Sunset - Beach
Sunset
Beach
ConDo segmentation
Sunset pixel labels
Beach pixel labels
ConDo Gaussian KLD
Sunset
Beach
Sunset Beach
ConDo MMD
Sunset - Beach
Sunset - Beach
Figure S14: Image color adaptation results with confounded shift between Sunset and Beach image.
39

---

Published in Transactions on Machine Learning Research (11/2024)
(A)
B
Adaptation
Gaussian
MMD
ConDo Gaussian KLD
ConDo MMD
0.0
0.3
0.6
Accuracy
F1 score
Figure S15: Results on the California Housing dataset, for the setting where TabPFN prediction models are trained on adapted source-to-target features and applied to target features. We show accuracy (A) and Fi-score (B) on target test data, over 10 random simulations, with error bars depicting the standard deviation over sinulations.
0.6
0.6
F1-score
MMD
0.5
0.4
Number of pairs
Number of pairs
Figure S16: SNAREseq cell type classifier performance on test set. Classifier performance (accuracy and Fl-score) shown as a function of the number paired samples available the domain adaptation methods. The error bars depict the standard error computed over 10 random simulations.
40


Reviews:
[{"review_id": "zSf889oQWA", "reviewer": "Reviewer_Umaf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "After reviewing this paper for the second time in two years, I still find it very hard to parse. I will skip the technical details of this work in my review as they didn’t change much and they were already solid and well-justified at the time of the first submission. I will mainly concentrate on the positioning with respect to the related works and the experimental results that have changed since the last submission. (I believe that evaluating the scikit-learn-friendly implementation mentioned by the authors is out of scope of this review).\nRegarding the two criteria mentioned above, I note the following:\n\n1. Confounded shift vs. GTS: the only difference here, as stated by the authors, is that GTS will correct the shift between the conditional distributions given the target variable, while the authors would like to have a more general setup where the target variable may change at inference time and is not observed. My question then is as follows: is it a meaningful setup given that the target variable (confounder) at test time may be arbitrary and no assumption is made about it? How identifiable is such a setting? I would expect that generally speaking there is no solid statistical reason to believe one can find a mapping that will be invariant for some unobserved target variables if no assumptions are made about the latter. \n\n2. The experiments (new and old) are not helpful in better illustrating the need for such a setup as almost all of them (please correct me if I’m wrong) are artificial in how the confounded shift is defined. Is there any natural application of this idea where the shift is real-world and the confounders change from V2 training to V2 inference? The only experiment I found useful in this sense is the one related to ANSUR II data. This setup echoes the one presented in Figure 1 with a confounder that changes between V2 training (male vs. female) and test (height). In this sense, the obtained results seem intuitive as the two confounders are pretty correlated. I guess some hypothesis of this kind is needed to make this setup well-posed as well.\n\n3. The assigned editor mentioned in their last review that it would be great to compare the learning mapping to the true ground truth mapping behind the shift. Is it possible for the authors to add this experiment?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I have reviewed this paper when it was first submitted several years ago so I will skip directly to the comments related to discussing what has changed since then.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. A suggestion from my side would be to indicate for all experiments what is V1 data, what is V2 training data, and what is V2 test data accompanied by the confounders between the training and test sets. \n\n2. For all tested synthetic cases, can we compare the found affine mapping with the true one as requested by the editor in the previous round of reviews?\n\n3. Give an example of a commonly used domain adaptation benchmark where the confounding shift doesn't need to be defined artificially.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YUmZi9n0FA", "reviewer": "Reviewer_LMYA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper addresses an important and practical problem in domain adaptation---handling scenarios where both covariate shift and label shift occur simultaneously and are confounded.\n- The proposed ConDo framework is general and flexible, allowing for different divergence measures and types of confounders (continuous, categorical, etc.).\n- The theoretical formulation is sound, building on and extending concepts from optimal transport and generalized label shift.\nExtensive empirical evaluation on synthetic and real datasets demonstrates the effectiveness of the approach, especially in challenging confounded shift scenarios.\n- The authors provide a thoughtful analysis of when and why their method outperforms baselines, as well as discussing limitations. The paper is well-written and structured, with clear explanations of concepts and methods.\n\n## Weaknesses\n- Unless I missed something, the paper assumes access to all confounders at training time, which may be unrealistic in some real-world scenarios. More discussion on partial observability of confounders would be valuable.\n- The current implementation is limited to linear/affine transformations between domains. While this is justified for certain applications, extending to nonlinear transformations would broaden the applicability.\n- While the empirical results are promising, including those in the supplementaries, the paper lacks theoretical guarantees or error bounds for the proposed methods. There's nothing wrong with a fully empirical TMLR paper but having these would certainly strengthen the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel framework called ConDo (Confounded Domain adaptation) for performing domain adaptation in the presence of confounded shift between source and target domains. The key idea is to minimize the expected divergence between conditional distributions of source and target features, conditioning on confounding variables. The authors propose concrete implementations using Gaussian reverse KL divergence and maximum mean discrepancy (MMD) as divergence measures. They evaluate ConDo on synthetic and real datasets, showing improvements over baseline domain adaptation methods, especially in scenarios with both covariate shift and label shift.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Explore scenarios with partial observability of confounders, or methods to infer potential confounders when not all are known. You could see this happen more in text domains given the high dimensional data.\n- Develop theoretical guarantees or error bounds for the proposed methods, even if under simplifying assumptions. This would strengthen the theoretical contributions.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V5XUDjZkJo", "reviewer": "Reviewer_bEdS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper's presentation is clear. It also provides detailed explanations from a theoretical perspective. \n2. The proposed method mostly involves modifying the loss. The training procedure is clean and easy to implement.\n3. The problem the paper focuses on is novel, i.e., Confounded Domain Adaptation.\n\nWeaknesses:\n1. Figure 1 is misleading. Detailed illustrations are expected to well understand the figure.\n2. The current implementation assumes a deterministic and linear mapping between feature spaces, which may not be suitable for all types of data or adaptation tasks. Extending this to non-deterministic or nonlinear mappings would enhance the framework's versatility.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new framework for domain adaptation that addresses simultaneous covariate and label shifts, confounded with each other, by modifying generalized label shift (GLS). The method ensures backward compatibility, allowing adapted data to be used in existing models without re-training. Implemented using Gaussian reverse Kullback-Leibler divergence and maximum mean discrepancy, the framework is validated on synthetic and real datasets, demonstrating its effectiveness and significant improvement over traditional methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the comments of weaknesses.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has seen a major update since its previous submission and is in much better shape . All reviewers appreciated the proposed method and the response done by the authors. For all those reasons and the fact that the paper is solid and well justified, I recommend to accept the paper. But the since it remains sometimes hard to follow, some minor revisions are still needed before final acceptance. 

More details below.

- Figure 1 is much better and the caption has been improved. But it should also reflect the notations of the paper in addition to concrete example of variables. The author should add to the figure what is X, Z and Y (that is not used in the paper but should be introduced) in the Figure so that the confounder (Z) and what is the final prediction task (prediction of Y from X) are clearly stated. As a matter of fact the author use Z for Y in Table 1 which is admittedly what traditional  DA method do but it makes the whole paper harder to follow because the proposed method use a confounder different from the label (since label is not available on target). This must be clarified in the final version because there is confusion for the reader between confounder and the actual label that is predicted on target using the estimated mapping. For the same reason the relation and assumption between the confounder an Y to have good performance also need to be discussed.

- For all numerical experiments with real data and related to the above point, the way the dataset is created (and split between source and target) is described in detail but not which feature is actually used as confounder. This is a problem for reproducibility and should be stated more clearly in each of the application subsections.
Decision: Accept (poster)