SKADA-Bench: Benchmarking Unsupervised Domain Adaptation Methods with Realistic Validation On Diverse Modalities
Authors: Yanis Lalou
Theo Gnassounou
Antoine Collas
Antoine de Mathelin
Oleksii Kachaiev
Ambroise Odonnat
Thomas Moreau
Alexandre Gramfort
Rémi Flamary
Venue: Accepted by TMLR ()

Abstract: Unsupervised Domain Adaptation (DA) consists of adapting a model trained on a labeled source domain to perform well on an unlabeled target domain with some data distribution shift. While many methods have been proposed in the literature, fair and realistic evaluation remains an open question, particularly due to methodological difficulties in selecting hyperparameters in the unsupervised setting. With SKADA-bench, we propose a framework to evaluate DA methods on diverse modalities, beyond computer vision task that have been largely explored in the literature. We present a complete and fair evaluation of existing shallow algorithms, including reweighting, mapping, and subspace alignment. Realistic hyperparameter selection is performed with nested cross-validation and various unsupervised model selection scores, on both simulated datasets with controlled shifts and real-world datasets across diverse modalities, such as images, text, biomedical, and tabular data. Our benchmark highlights the importance of realistic validation and provides practical guidance for real-life applications, with key insights into the choice and impact of model selection approaches. SKADA-bench is open-source, reproducible, and can be easily extended with novel DA methods, datasets, and model selection criteria without requiring re-evaluating competitors. The code is available at https://github.com/scikit-adaptation/skada-bench.

Published in Transactions on Machine Learning Research (07/2025)
SKADA-Bench: Benchmarking Unsupervised Domain Adap- tation Methods with Realistic Validation On Diverse Modali- ties
Yanis Lalou"
yanis..alouuppllycchhiiu...cc edu
Ecole Polytechnique, IP Paris, CMAP, UMR 7641, 91120 Palaiscau, France
Théo Gnassounou"
theo the..gnaasoumou@@inrii.
Université Paris-Saclayy Inria, CEA, 91120 Palaiseau, France
Antoine Collas"
cntobne..ollass@inrii...
Université Paris-Saclay, Inria, CEA, 91120 Palaiseau, France
Antoine de Mathelin"
antoine. dww.eddemaaaammaamo com
Centre Borelli, ENS Paris-Saclay, Gif-sur- Yuelte, 91190, France
Oleksii Kachaiev
olelsiikaccaiee@@@malcccoo
MaLGa Center Dipartimento di Matematica Università degli Studi di Genova, Italy
Ambroise Odonnat
ambroise..odonnatt.Booo
Inria, Unin. Rennes 2, CNRS, IRISA, Paris, Prance
Thomas Moreau
thomas. frr
Universilé Paris-Saclay, Inria, CEA, 91120 Palaiseau, France
Alexandre Gramfort
dlezandre.grammoorr@@hhrrrr.
Université Poris-Sackss Inria, CEA, 91120 Palaiseau, France
Rémi Flamary
Fcole Polytechnique, IP Paris, CMAP, UMR 7641, 91120 Palaiseau, France
Reviewed on OpenReview: https: // openreview. net/ forum? id- k9P63DV33
Abstract
Unsupervised Domain Adaptation (DA) consists of adapting model trained on labeled souree domain to perform well on unlabeled target domain with some data distribution shift. While many methods have been proposed in the literature, fair and realistic evaluat ion remains an open question, particularly due to methodological difficulties in selecting hyperparameters the unsupervised setting. With -KADA-Bench,, we propose a framework to evaluate DA methods on diverse modalities, beyond computer vision task that have been largely explored
Equal contribution

---

Published in Transactions on Machine Learning Research (07/2025)
in the literature. We present a complete and fair evaluation of existing shallow algorithms, including reweighting, mapping, and subspace alignment. Realistic hyperparameter selection is performed with nested cross-validation and various unsupervised model selection scores, on both simulated datasets with controlled shifts and real-world datascts across diverse modalities, such as images, lext, biomedical, and tabular data. Our benchmark highlights the importance of realistic validation and provides practical guidance for real-life applications, with key insights into the choice and impact of model selection approaches. SKADA-Bench is open-source, reproducible, and can be easily extended with novel DA methods, datasets, and model selection criteria without requiring re-evaluating competitors. The code is available at https:// / githuu. cwm/ addptation/skade bench.
1 Introduction
Given some training -or source- data, supervised learning consists in estimating a function that makes good predictions on larget data. However, performance often drops when the source distribution used for training differs from the target distribution used for testing. This shift can be due, for instance, to the collection process or non-stationarity in the data, and is ubiquitous in real-life settings. It has been observed in various application fields, including tabular data (Gardner et al., 2023), clinical data (Harutyunyan et al., 2019), computer vision (Ganin et al., 2016b).
Domain adaptation. Unsupervised Domain Adaptation (DA) addresses this problem by adapting a model trained on a labeled source dataset -or domain- so that it performs well on an unlabeled target domain, assuming some distribution shifts between the two (Ben-David et al., 2006; Quinonero- Caanoner--Candlla Candela. et al., 2008; Redko et al., 2022). As illustrated in Figure 1, source and target distributions can exhibit various types of shifts -Moreno-Torre, et al., 2012): changes in feature distributions (covariate shift), class proportions (target shift), conditional distributions (conditional shift), or in distributions in particular subspaces (subspace shift).. Depending on the type of shift, existing DA methods attempt to align the source distribution closer to the larget using reweighting (Sugiyama & Müller, 2005; Shimodaira, 2000), mapping (Sun el al., 2017; Courty et al., 2017b), or dimension reduction (Pan et al., 2011; Fernando et al., 2013) methods. More recently, it has been proposed to mitigate shifts in a feature space learned by deep learning (Ganin et al., 2016b; Sun & Saenko, 2016; Long et al., 2015a; Damodaran et al., 2018b), primarily focusing on computer vision applications. Regardless of the core amgorithm used to address the domain shift, hyperparameters must be tuned for optimal performance. Indeed, critical challenge in applying DA methods to rcal-world cases is selecting the appropriate method and tuning its hyperparameters, especially given the unknown shift type and the absence of labels in the target domain.
Model selection in DA settings. Without distribution shifts, classical model selection strategics including hyperparameter optimization- rely on evaluating the generalization error with an independent. labeled validation set. However, in DA, validating the hyperparameters in a supervised manner on the target domains is impossible due to the lack of labels. While it is possible to validate the hyperparameters on the source domain, it generally leads to suboptimal model selection because of the distribution shift. In the literature, this problem is often raised but not always addressed. Some papers choose not to validate the parameters (Pan et al., 2011), while others validate on the source domain (Sun et al., 2017) or propose custom cross- validatal methods (Sugiyama et al., 2007b). Few papers focus specifically on DA model selection criteria, which we will call scorers in this paper. These scorers are used to select the methods' hyperparameters, and mainly consists of reweighting methods on source (Sugiyama et al., 2007a; You al., 2019), prediction entropy (Morerio et al., 2017; Saito et al., 2021) or circular validation (Bruzzone & Marconeini, 2010a). One of the goals of our benchmark is to evaluate these approaches in diverse and realistic scenarios.
Benchmarks of DA. As machine learning continues to flourish, new methods constantly emerge, making it essential to develop benchmarks that facilitate fair comparisons (Hutson, 2018; Pineau et al., 2019; Matison et al., 2020; Moreau et al., 2022). In DA and related fields, several benchmarks have been proposed. Numerous papers focus on Out-o--distributiin (OOD) datascts for different modalities: computer vision, text, graphs (Koh et al., 2021; Sagawa et al., 2022), time-series (Gagnon- Audet al., 2023), Al-aided drug discovery (Ji et al., 2023) or tabular dataset (Gardner et al., 2023). Due to the type of data considered, existing
2

---

Published in Transactions on Machine Learning Research (07/2025)
Covariate shift (x) + (x)
Target shift P3(v) P3(v)
Conditional shift ('(X)s) # P"(X v)
Subspace shift P3 (²((())) = P3 (o(c))
Source data
Target data
Source data
Target data
Figure 1: Illustration of different type of distribution shift between source and target domains: covariate shift, target shift, conditional shift, and subspace shift (Mathematial details are available in Section 2.1). Points represent data samples, with colors indicating different classes. These synthetic datasets are used evaluate model performance under controlled shift scenarios in the experiement part.
benchmarks are mainly focused on Deep DA methods (Musgrave et al., 2021; Wang, 2018; Jiang et al., 2022; Fawaz et al., 2023), offering an incomplete evaluation of DA literature. Moreover, only few benchmarks propose a comparison of Deep unsupervised DA methods with realistic parameters selection, on computer vision (Hu et. al., 2023; Musgrave et al., 2021) and time series (Fawaz et al., 2023) data. Those benchmarks have shown the importance of validating with unsupervised scores and reveal that Deep DA methods achieve much lower performance in realistic scenarios.
Contributions. In the following, we propose SKADA-Bench, an ambitious and fully reproducible benchmark with the following features: 1. A set of 4 simulated and 8 real-life datasets with different modalities (computer vision, NLP, tabular, biomedical) totaling 51 realistic shift scenarios, 2. A wide range of 20 Shallow DA methods designed to handle different types of shifts, 3. An evaluation of 7 deep DA methods on 4 real-world datasets from the computer vision and biomedical modalities, 4. realistic model selection procedure using different unsupervised scorers with nested cross-validation for hyperparameter selection, 5. An open-source implementation and publicly available datasets, easy to extend for new DA methods and datasets without. the need re-run the whole experiment.
In addition, we provide detailed analysis of the results and derive guidelines for practitioners to select the best methods depending on the lype of shifts, and the best scorer to perform unsupervised model selection. In particular, the effects of model selection and the scorer's ss choice on the final performances are highlighted, showing a clear gap between the unsupervised realistic scorers versus using target labels for supervised validation.
2 Domain adaptation and model selection without target labels
In this section, we first discuss the specificities of the unsupervised domain adaptation problem and introduce several types data shifts and their corresponding DA methods. Next, we discuss the different validation strategics used in the literature and the need for realistic scorers compare DA methods.
2.1 Data shifts and DA strategies
Domain Adaptation problem and theory. The thcoretical framework of DA well established (Ben- David al., 2006; Hhinonero-Cander et al., 2008; Redko al., 2022). The main results highlight that the performance discrepancy of an estimator between the source and target domains is linked the divergence between both distributions. This has motivated the majority of DA methods to search for a universal (or domain invariant) predictor by minimizing the divergence between the two domains through the adaptation of the distributions. This is done in practice by modeling and estimating the shift between the source and target distributions and then compensating for this shift before training a predictor.
Data Shifts and DA methods. A wide variety of shifts between the source and target distributions are possible. They are usually expressed as a relation between the joint distributions P*(x, y) (*ly)) = Pd (x) in the source domain and pt (xx y) = Pt (3) now discuss the main types of shifts and the strategies proposed in the literature to mitigate them. Figure
(x in the target domain. We
3

---

Published in Transactions on Machine Learning Research (07/2025)
illustrates these shifts.
In Covariate shift the conditionals probabilities are equal (i.e., P*([(z) = ('(yl=)), but the feature marginals change (i.e., (x) & (x)). Target shift is similar, but the label marginals change P²(()) # P5(2) while the conditionals are preserved. For classification problems, it corresponds to a change in the proportion of the classes between the two domains. Both of those shifts can be compensated by reweighting methods that assign different weights to the samples of the source domain to make it closer to the target domain (Sugiyama & Müller, 2005; Shimodaira, 2000).
In Conditional shift, conditional probabilities differ between domain (d.e., (*(x))) # (*(xl)) P*(y]=) * ('llx)). This shift is typically harder to compensate for, necessitating explicit modeling to address it effectively. For instance, several approaches model the shift as a mapping m between the source and target domain such that P (v((v((x) = P' (ullr) (Sun et al., 2017; Courty el al., 2017b). The estimated mapping is then applied to the source data before training a predictor.
Subspace shift assumes that while probabilitics are different between the domains (P²(x) + (x) and (* + ('(x))),, there exists a subspace Z and function o X + Z such that P)(ó(=)) = P3 (x(x)) and (*)))((z) = Pt (ylø(z)). Note that this means the shift occurs in the orthogonal complement of Z. This implies that a classifier trained on Z will perform well across both domains. Subspace methods are specifically designed towards identifying the subspace 2 and the function é, as developed in Pan et al. (2011);
Fernando et al. (2013). Note that, as discussed in the introduction, a natural extension of this idea is to learn an invariant feature space using Deep learning (Ganin et al., 2016b; Sun & Saenko, 2016).
2.2 DA model selection strategies
As seen above, DA methods are typically designed to correct a specific type of shift. However, in real-world scenarios, the nature of the shift is often unknown. This presents challenge in selecting the appropriate method and tuning its parameters when facing new problem. In this section, we discuss the validation strategies proposed in the literature to compare DA methods, focusing realistic scorers that do not use target labels.
Realistic DA scorers. In the literature, few papers propose realistic DA scorers to validate the parameters of the methods, i.e., unsupervised scorers that do not require target labels. The Importance Weighted (IW) scorer (Sugiyama et al., 2007a) computes the score as a reweighted accuracy on labeled sources data. The Deep Embedded Validation (DEV) (You et al., 2019) can be seen as an IW in the latent space with variance reduction strategy. DEV was originally proposed for Deep learning models but can be used on shallow DA methods that compute features from the data bhapping//ubspaceee (PE) scorer (Morerio et al., 2017) measures the uncertainty associated with model predictions on the larget data. Soft Neighborhood Density (SND) (Saito et al., 2021) also computes an entropy but on a normalized pairwisc similarity matrix between probabilistic predictions on target. The Circular Validation (Cire V) scorer (Bruzzone & Marconcini, 2010a) performs DA by first adapting the model from the source to the target domain and predicting the target labels. Next, adapts back from the target to the source using these estimated labels. Performance is measured as the accuracy between the recovered and true source labels.
The Prediction Entropy
The Mir Val scorer (Hu et al., 2023) also performs domain adaptation by first adapting the model from the source to the target domain and predicting the target labels. Then, it generates mixed target samples by probing intra-cluster samples to assess neighborhood density and inter-cluster samples to examine classification boundaries. The score is the accuracy between the generated targets labels and their predictions to evaluate the consistency.
DA validation in the literature. The model selection problem in DA has been widely discussed in the literature. Yet, this literature constitutes a subfield of DA and has seldom been used to validate new DA methods. Indeed, there is consensus on the best validation strategy and many papers do not properly validate their methods, lcading to over-estimated performances. Some authors do not discuss the validation procedure (Sugiyama & Müller, 2005; Shimodaira, 2000) or consider fixed hyperparamelers (Huang et al., 2006). While some methods rely on custom validation techniques (Sugiyama et al., 2007b), others use cross-validation, either on the source or the target (Sun et al., 2017; Courty et al., 2017b), or alternatively other validation strategies proposed in the literature (Courty et al., 2017a; Bruzzone & Marconcini, 2010a). A complete picture of the model selection procedures used to validate the methods considered in SKADA- -Bnnhh
4

---

Published in Transactions on Machine Learning Research (07/2025)
Source Data
Target Data
Label available
Label available
Outer loop
Final estimation with best hyperparameters
Source test Set
Targel lesi
Nested loop Hyperparameters selection
Source
Source validation
Target crain set
Target validation
Figure 2: Visualization of nested cross-validation strategy. Both source and target data are split into an outer loop and then nested loop. The nested loop tunes hyperparameters for the domain adaptation method, while the outer loop trains final classifier with the best hyperparameters and evaluates its accuracy on both source and target data. Note: Target sets have no labels during the nested loop, reflecting unsupervised Domain Adaptation.
in their original papers presented in Table in Appendix B. The goal of SKADA-Bench is therefore constitute a dedicated benchmark to compare scorers from the literature and report performances that can be expected in real use cases for the considered methods.
3 A realistic benchmark for DA
In this section, we present our benchmark framework. First, we introduce the parameter validation strategies. Then, we present the compared DA methods followed by a description of the datasots used in the benchmark.
3.1 Nested cross-validation loop and implementation
We discuss below the nested cross-validation and the implementation details of the benchmark.
Hyperparameter validation loop. We propose nested loop cross-validatin procedure, depicted Figure 2. First, the source and target data are split into multiple outer and train sets (outer loop in Figure 2). The test sets kept to compute the final accuracy for both the source and target domains. For each split in the outer loop, use nested loop select the DA methods' parameters. Here, the training sets are further divided into nested train and validation sets (nested loop in Figure 2). Note that no labels are available for the target nested train and validation sets in this loop. The target training set is used to train the DA method, while the target validation set allows to compute the unsupervised score and select the best model.
For both loops, the data split randomly times using stratified sampling with an 80%/20% train/tess split, except for Deep DA methods, where only one split is computed for the outer loop due to computation time. For one given method, evaluate all the unsupervised scorers discussed earlier, well supervised scorer that uses target labels, over all the nested splits. A fttrr averaging, the scores over the splits, the best hyperparameters are selected according to each scorer and then used to train a final classifier on the outer training sets. Although the supervised scorer cannot used in practice, is included in our results to actually evaluate the performance drop due to the absence of target labels. To limit complexity and perform a fair comparison of the methods, we timcout hours for performing the nested loop. Additionally, we chose not to use the CircV scorer for Deep DA methods, training neural networks twice is compulationally expensive.
Base estimators and neural networks. Existing shallow domain adaptation methods typically rely on either base isttmo trained on the adapted data or an iterative estimation process Lo adapt this estimator to the target data. The choice the base estimator is crucial, as significantly impacts the final performance. Before validating the hyperparameters of the DA mothods, we determined the best estimator for each datasct using a grid-scarch on the source data. We tested multiple hyperparameters for Logistic Regression, SVM with RBF kernel, and XGBoost (Chen & Guestrin, 2016), selecting the ones that maximize the average
5

---

Published in Transactions on Machine Learning Research (07/2025)
accuracy on the source test sets. Note that for some methods that specifically require an SVM estimator (i.., JDOT and DASVM), we only validate SVM as the base estimator. We validated the base estimator separately from the DA methods parameters to reduce computational complexity and avoid too complex hyperparameter grids that can compromise the reliability of DA scorers. For Deep DA methods, we similarly select an appropriate architecture and experimental setup for training on the source data. for each dataset: a two-layer convolutional neural network for MNIST/USPS, a 5esNet50 (He et al., 2016) pretrained on ImageNet. (Deng et al., 2009) for Office31 and Office Home, and a ShallowF BCI. These architectures are widely used and well-supported in the literature of computer vision (Musgrave et al., 2021) and BCT (Schirrmeister et al., 2017). During the nested loop, only the DA parameters for each method are validated.
(Schirrmeister et al., 2017) for
Best scorer selection and statistical test. For all methods, we select the best validation scorer as the one that maximizes the averaged accuracy on the target domains for all real datasets. This provides a reasonable and actionable choice of scorer for each DA method for practitioners. For all methods and datascts, we perform paired Wilcoxon signed-rank test at the 0.05 level to detect significant gain drop in performance with respect to the no DA approach, denoted by "Train Sre" in the following. The test is done using the accuracy mcasures of the DA method with the selected scorer and the Train Src for all shifts and outer splits, ensuring between 10 and 60 values depending on the dataset. Note that these statistical tests are not performed on Deep DA methods, as the number of splits is too limited for meaningful testing.
Python impllementaton.. The benchmark code available on GitHub upon publication of the paper." Our benchmark is implemented following the benchopt framework (Moreau et al., 2022), which provides standardized ways of organizing and running benchmarks for ML in Python. This framework facilitates reproducing the benchmark's results, with tools to install the dependencies, run the methods in parallel, or cache the results to prevent redundant computations. It also makes it easy to extend the benchmark with additional datasets and methods, enabling it to evolve to account for the advances in the field. In the supplementary matcrials, we provide examples demonstrating how to add DA methods or datascts to the benchmark. Using this framework, we aim lo make SKADA-Bench a reference benchmark to evaluate new DA methods in realistic scenarios with valid performance estimations.
3.2 Compared DA methods
In this section, we present the different families of domain adaptation methods that we compare in our benchmark. The shallow methods are grouped into four categorics: reweighting methods, mapping methods, subspace methods, and others. For Deep DA methods, we consider three domain invariant feature methods. We provide a brief description of each method and the corresponding references.
Reweighting methods. These methods aim reweight the source data to make closer to the target data. The weights are estimated using different methods such as kernel density estimation (Dens. RW) (Sugiyama & Müller, 2005), Gaussian estimation (Gauss. RW) (Shimodaira, 2000), discriminative estimation (Discr. RW) (Shimodaira, 2000), or nearest-neighbors (NN RW) (Loog, 2012). Other reweighting estimate weights by minimizing divergence between the souree and target distributions such as Kullback-Leibler Importance Estimation Procedure (KLIEP) (Sugiyama et al., 2007b) or Kernel Mean Matching (KMM) (Huang et al., 2006). Finally, we also include the MMDTarS method (Zhang et al., 2013) that uses a Maximum Mcan Discrepancy (MMD) to estimate the weights under the target shift hypothesis.
Mapping methods. These methods aim to find mapping between the source and target data that minimizes the distribution shift. The Corrclation Alignment method (CORAL) (Sun et al., 2017) aligns the second-order statistics of source and target distributions. The Maximum Mean Discrepancy (MMD-LS) method (Zhang et al., 2013) minimizes the MMD to estimate an affine Location mapping. Finally, the Optimal Transport (OT) mapping methods (Courty et al., 2017b) use the optimal transport plan to align with a non-lincar mapping of the source and target distributions with exact OT (MapOT), entropic regularization (EntOT), or class-based regularization (C (ClassRegOT). Finally, the Linear OT method (Flamary et al., 2020) uses linear mapping to align the source and target distributions, assuming Gaussian distributions.
"Qu code available in supplementary materials.
6

---

Published in Transactions on Machine Learning Research (07/2025)
Subspace methods. These methods aim to learn subspace where the source and target data have the same distribution. The Transfer Component Analysis (TCA) method (Pan et al., 2011) searches for a kernel embedding that minimizes the MMD divergence between the domains while preserving the variance. The Subspace Aligmment (SA) method (Fernando et al., 2013) aims to learn a subspace where the source and target have their covariance matrices aligned. The Transfer Subspace Learning (TSL. method (Si et al., 2010) aims to learn a subspace using classical supervised loss functions on the source (e.g., PCA, Fisher LDA) but regularized so that the source and target data have the same distribution once projected on the subspace. Finally, the Joint Principal Component Analysis (JPCA) method is simple baseline that concatenates source and target data before applying PCA.
Others. We also include other methods that do not fit into the previous categories. The Domain Adaptation SVM (DASVM) method (Bruzzone & Marconeini, 2010a) is a self-labeling method that iteratively updates SVM estimators by adding new target samples with predicted labels and removing source samples. The Joint Distribution Optimal Transport (JDOT) method (Courty et al., 2017a) aims to learn a target predictor that minimizes an OT loss between the joint source and target distributions. The Optimal Transport Label Propagation (OTLabelProp) method (Solomon et al., 2014) uses the optimal transport plan to propagate labels from the source to the target domain.
Deep DA methods. These methods aim to reduce the divergence between the source and target data distributions within the learned feature space while simultaneously learning a classifier on source data. The training loss consists in a traditional supervised loss on labeled source data and a second term measuring the discrepancy between the source and target distributions. The methods implemented in the Deep DA Benchmark use different discrepancies, such as covariance distance (Sun & Saenko, 2016) for DeepCORAT, adversarial loss (Ganin et al., 2016a; Zhang et al., 2019) for DANN or MDD, MMD distance (Long et al., 2015a) for DAN, optimal transport distance (Damodaran et al., 2018a) for Deep.DDOT class confusion (Jin et al., 2020) for MCC, and graph spectral aligmment (Xiao et al., 2023b) for SPA. Note that these approaches are not part of the main shallow DA benchmark but have boen added to provide an interesting comparison of DA performances between shallow and Deep methods on computer vision and biomedical data.
3.3 Compared datasets
In this section, we present the datasets used in our experiments. We first introduce the synthetic datasets that implement different known shifts. Then, we describe the real-world datasets from various modalities and tasks such Computer Vision (CV), Natural language Processing (NLP), tabular data, and biosignals.
Simulated datasets. The objective of the simulated datasets is to evaluate the performance of the DA methods under different types of shifts, Knowing that multiple DA methods have been built to handle specific shifts, evaluating them with this dataset will demonstrate whether they perform as expected and if they are properly validated.
The four simulated shifts in 2D, covariate (Cov. shift), target (Tar. shift) conditional (Cond. shift) and Subspace (Sub. shift) shift are illustrated in Figure 1. The source domain is represented by two non-lincarly separable classes generated from one large and several smaller Gaussian blobs. In the experiments, the level of noise has been adjusted from Figure to make the problem more difficult. For the subspace shift scenario, the source domain consists of one class represented by a large Gaussian blob and another class comprising Gaussian blobs positioned along the sides of the large one. The target domain is flipped along the diagonal, making the task challenging in the original space but feasible upon diagonal projection.
Real-word datasets. The real-world datasets used in our benchmark are summarized in Table 1. We select 8 datasets from different modalities and tasks: Computer Vision (CV) with 0ffice31 (Koniusz et al., 2017), Office Home (Venkateswara et al., 2017), and MNIST/USPS (Liao & Carneiro, 2015), Natural Language Processing (NLP) with 20Newsgroup (Lang, 1995) and Amazon Review (McAuley et al., 2015), Tabular Data with Mushrooms (Dai et al., 2007) and Phishing (Mohammad et al., 2012), and Biosignals with BCI Competition IV (Tangermann et al., 2012). The datasets are chosen to represent a wide range of shifts and to evaluate the performance of the methods on different types of data.
Data are preprocessed to extract relevant features, while koeping computational costs reasonable. Images are embedded using deep pre-trained models (except MNIST/USPS where the images are vectorized), and textual
7

---

Published in Transactions on Machine Learning Research (07/2025)
Table 1: Characteristics of the real-world datasets used in SKADA-Bench
Dataset
Modality
Preprocessing
adapt
classes
samples
features
Office 31
Decall + PCA
CV
470
350
100
(Koniua et al., 2017)
(Donahue et al., 2014)
Office
ResNet + PCA
(Venkaleswara et al., 2017)
CV
(He el al., 2016)
12
65
3897 + 850
100
HNIST/USPS
CV
Vect PCA
3000
10000
50
(Liao & Carneiro, 2015)
LLM + PCA
20 Kewagroup (Lang, 1995)
NLP
(Reimers & Gurevych (2019),
6
3728 d: 174
50
Xiao et al. (2023a))
Anazon Review
LLM + PCA
(MeAuley & Leskover (2013),
NLP
(Reimers Gurevych (2019),
2000
McAuley et al. (2015))
Xiso et al. (2023a))
Mushrooms
Tabular
One Hot Encoding
1062 516
117
(Dai et al., 2007)
Phishing
(Mohammad et al., 2012)
Tabular
NA Cov+'T
2
5527 = 1734
30
BCT
(Tangermann el al., 2012)
Biosignals
(Barachaut el al., 2012)
288
253
data is embedded using Large Language Models (LLM) (Reimers & Gurevych, 2019; Xiao et al., 2023a). The tabular data are one-hot encoded to transform categorical data into numerical data. The biosignals from Brain-Compuri Interface (BCI) data are embedded using the Srate-of-the-art tangent space representation proposed in Barachant et al. (2012). For images and text, we apply a PCA reduce the dimensionality of the embeddings to reasonable dimension to avoid big computational costs. For Deep DA methods, only 4 datasets are used: Office31, Office Home, MNIST/USPS and BCI. Since these methods focus on learning feature representations, the data are used in their raw form. The datasets are split into pairs of source and target domains totaling 51 adaptation tasks in the benchmark. More details about the datasets and pre-processing are available in Appendix C.
4 Benchmark results
We now present the results of the benchmark. Training and evaluation across all shallow experiments required 1,215 CPU-hours standard Slurm (Yoo et al., 2003) cluster, while the Deep DA experiments required 244 GPU-hours. We first discuss and compare the performances of the methods on the different datasets. Then, a detailed study of the unsupervised scorers is provided.
4.1 Performance of the DA methods
Results table. First, we report the realistic performances the different methods when using their seleeted scorer on the different datasets in Table 2. The cells showcasing a significant change in performance with the Wilcoxon test are highlighted with colors. Blue indicates an increase in performance, while red indicates a loss. The intensity of the color corresponds to the magnitude of the gain loss the darker the shade, the larger the positive negative change. Cells with NA values indicate that the method was not applicable to the dataset (DASVM is limited to binary classification) or that the method has reached timeout. We also report the best scorer and the average rank of the methods for all real datasets. In addition to Table 2 providing realistic performance estimations with the best realistic scorer, we also report in Table 20 (Appendix E) the results when using the non-realistic supervised scorer.
Simulated data with known shifts. DA methods tend to show a significant gain on the shift they were designed for. It is especially true for mapping methods which greatly outperforms the Train Sre approach under conditional shift (Cond. shift), almost reaching the Train Tgt performance for EntOT and ClassReg The results also highlight that the mapping methods strugglc with target shift (Tar. shift), which is a well-known limitation of this kind of approach (Redko et al., 2019). On the contrary, reweighting methods provide robust performance on target shift. Regarding covaratiate shift (Cov. shift), the improvement with reweighting methods is very limited although reweighting is specifically designed for this kind of shift. We believe that using a complex base estimator (here an SVM with an RBF kernel) enables us train an estimator that works well on both source and target, reducing the impact of importance weighting as previously highlighted
8

---

Published in Transactions on Machine Learning Research (07/2025)
Table 2: Accuracy score for all datasets compared for all the shallow methods for simulated and real-life datascts. The color indicates the amount of the improvement. A white color means the method is not statistically different from Train Sre (Train on source). Blue indicates that the score improved with the DA methods, while red indicates a decrease. The darker the color, the more significant the change.
MNIST/ MNIST/USPS USPS
20NewsGroups
AmazonReview
Selected Scorer
Coud. shift
OfficeHome
Mushrooms
Cov, shift
Tar. shift
Sub. shiff
Office31
Phishing
BCI
Rank
Train Sre
0.88
0.85
0.66
0.19
0.65
0.56
0.54
0.59
0.7
0.72
0.91
0.55
10.66
Train Tgt
0.92
0.93
0.82
0.98
0.89
0.8
0.96
1.0
0.73
1.0
0.97
0.64
1.55
Dens. RW
0.88
0.86
0.66
0.18
0.62
0.56
0.54
0.58
0.7
0.71
0.91
0.55
IW
12.20
Reweighting
Disc. RW
0.85
0.83
0.71
0.18
0.63
0.54
0.5
0.6
0.68
0.75
0.91
0.56
CireV
8.75
Gauss. RW
0.89
0.86
0.65
0.21
0.22
0.44
0.11
0.54
0.55
0.51
0.46
0.25
CireV
16.45
KLIEP
0.88
0.86
0.66
0.19
0.65
0.56
0.54
0.6
0.69
0.72
0.91
0.55
CireV
10.56
KMM
0.89
0.85
0.64
0.16
0.64
0.54
0.52
0.7
0.57
0.74
0.91
0.52
CircV
11.74
NN RW
0.89
0.86
0.67
0.15
0.65
0.55
0.54
0.59
0.66
0.71
0.91
0.54
CircV
9.15
MMDTarS
0.88
0.86
0.64
0.2
0.6
0.56
0.54
0.59
0.7
0.74
0.91
0.55
IW
10.81
CORAL
0.74
0.7
0.76
0.18
0.65
0.57
0.62
0.73
0.7
0.72
0.92
0.62
CircV
5.08
Mapping
MapOT
0.72
0.57
0.82
0.02
0.6
0.51
0.61
0.76
0.68
0.63
0.84
0.47
PE
10.21
EntOT
0.71
0.6
0.82
0.12
0.64
0.58
0.6
0.83
0.62
0.75
0.86
0.54
CircV
9.40
ClassRegOT
0.74
0.58
0.81
0.11
NA
0.53
0.62
0.97
0.68
0.82
0.89
0.52
IW
8.25
LinOT
0.73
0.73
0.76
0.18
0.66
0.57
0.64
0.82
0.7
0.76
0.91
0.61
CireV
4.06
MMD-LS
0.78
0.72
0.76
0.56
0.65
0.56
0.55
0.97
0.63
0.85
NA
0.5
MixVal
8.22
Other Subspace
JPCA
0.88
0.85
0.66
0.15
0.62
0.48
0.51
0.77
0.69
0.78
0.9
0.54
PE
8.98
SA
0.74
0.68
0.8
0.11
0.65
0.57
0.56
0.88
0.67
0.78
0.89
0.53
CircV
7.80
TCA
0.52
0.47
0.51
0.62
0.04
0.02
0.07
0.61
0.61
0.49
0.48
0.26
DEV
17.58
TSL
0.88
0.85
0.66
0.2
0.63
0.48
0.45
0.63
0.69
0.45
0.89
0.26
IW
15.09
JDOT
0.72
0.58
0.82
0.13
0.6
0.42
0.59
0.79
0.67
0.65
0.79
0.47
IW
11.42
OTLabelProp
0.72
0.59
0.8
0.07
0.66
0.56
0.62
0.86
0.67
0.64
0.86
0.5
CircV
10.01
DASVM
0.89
0.86
0.65
0.15
NA
NA
NA
0.87
NA
0.83
0.85
NA
MixVal
7.29
The results reported in Table 17 of Appendix E reveal that reweighting methods significantly outperform Train Sre when using linear base classifier.
in (Byrd & Lipton, 2019) for deep neural networks.
Real data with unknown shift. The performance of reweighting methods is often close to Train Sre baseline on real datasets. For example, NN RW reaches only a 54% accuracy on MNIST/USPS compared to 54% for Train Src. This result can be be due to the violations of the same-support assumption, which is crucial for reweighting to work effectively (Segovia-Martin et al., 2023), and which likely true for the three CV datasets. In this case, hyperparameter tuning frequently select configurations leading to near-uniform weighting, which explain the close performance to Train Src.
The performance of mapping methods is cntaaet-ddeendeent potentially due to the number of classes and presence of largel shift. Mapping methods excel on MNIST/USPS and 20NewsGroup which respectively contain 10 and 2 classes, but fail on Office31 and OfficeHome with 31 and 60 classes. For example, ClassRegOT achieves 62% on MNIST /USPS and 97% on 20NewsGroup, but only 53% on OfficeHome. Additionally, while mapping performs well on the NLP dataset 20NewsGroup, it results in negative transfer on Amazon Reviews which has target shifts.
It is notable that simple transformations that significantly outperform Train Src in ranking average across all modalities are LinOT, CORAL, JPCA, and SA, which all rely on linear transformations such as scaling, linear projection, or rotations. These methods are robust across datasets and modalities, offering effective aligneent with minimal risk of negative DA. For instance, LinOT consistently ranks among the top 5 methods and achieves substantial gains over Train Sre: +10 points on MNIST/USPS (64% vs. 54%), +33 points on 20NewsGroup (82% vs. 59%), and +6 points on BCT (61% vs. 55%).
the best in average across all modalities. Indeed, most methods
9

---

Published in Transactions on Machine Learning Research (07/2025)
Supervised p
p
SND -0.00
DEV p= -0.04
Happlrs Cdher
1.0
0.0
Rearighting
Sunscese
-0.5
Cross-vallsation
0.6
1.0
0.4
0.4
0.2
0.2
2-
0.0
0.0
1-
-2.0
PE p= 0.10
CircV pp= 0.71
MixVal p= 0.34
Cross-validation score
0.9
0.8
00.1
0.8
0.6
0.7
-0.2
0.4
0.6
0.2
0.5
-0.3
0.4
0.0
0.50
1.00
0.25
0,50
0.75
1.00
0.25
0.50
1.00
Target acouracy
Target accuracy
Target accuracy
Figure 3: Cross-val score a function of the accuracy for different supervised and unsupervised scorers. The Pearson correlation coefficient is reported for each scorer by p. Each point represents imner split with DA method (color of the points) and dataset. A good score should correlate with the target accuracy.
Computational cost. While full computational results are reported in Appendix E.7 and Figure 11, we briefly highlight that several top-performing methods- such as LinOT, CORAL, and SA-also exhibit some of the lowest training and testing times, averaging under 20 seconds per task. This reinforces their appeal in practice: they combine competitive performance with high efficiency, making them especially suitable for settings with limited computational resources many adaptation tasks.
Take-away for DA users. Reweighting methods are best suited for scenarios where the same-support assumption holds and perform particularly well when paired with regularized hypotheses like lincar models. Even when assumptions are not fully satisfied, reweighting tends to be robust to negative trausfer. Mapping methods are highly effective under moderate numbers of classes and in the absence of target shift but carry a significant risk of negative transfer if target shift is present. When the type of distribution shift is uncertain, simpler transformation-basde methods like LinOT, CoRAL, JPCA, and SA provide modest performance improvements while minimizing risks of negative DA, making them reliable and safe default options.
Selected scorer per DA method. We observe that the best scorer differs across methods, Circular Validation has been selected 10 out of 20 times as the best scorer, followed by Importance Weighting out of 20 times. Table 20 the supplementary material provides the non-realistic accuracy results with the supervised scorer. It worth noting that the supervised scorer generally outperforms the unsupervised ones. For example, En achieves +5 points with the supervised scorer versus with the CircV scorer (88% 83%) on the 20NewsGroups dataset. It is crucial to choose the model realistically to avoid producing overly optimistic results, as many data analysis papers have done (sce Table 4).
These results show the methods' sensitivity to parameter selections and the difficulty of using realistic scorers. This might also explain why DA methods are not widely used in practice: they are very difficult to tune and might docrease performances compared with no adaptation.
4.2 Study of validation scorers
We now investigate the performance of the various scorers to select hyperparameters of the DA method. First, we consider the relationship between the cross-val score and the accuracy for each inner split. In Figure 3, we plot for each scorer the cross-val score as function of the accuracy computed on the test set and report the Pearson correlation coefficient p. As expected, the supervised scorer highly correlated with the accuracy (p- = 0.98), as it has access to the target labels. We observe that SND, DEV, and PE do not provide
10

---

Published in Transactions on Machine Learning Research (07/2025)
good proxy to select hyperparameters that give the best-performing models (p 0.06). On the contrary, Vix IW and CircV are correlated with the accuracy, p 0.34, p = 0.56 and p p= 0.71 respectively. This is coherent with their selection as the best scorer in most scenarios in Table 2. Still, while those scorers are well correlated with the target accuracy, it is important to note that they have a large variance. For instance, a score close to IW or CircV corresponds to an accuracy between 0.5 and 1.0.
Furthermore, we provide in Figures 9 and 10, from Appendix E, several visualizations that illustrate the relationship between the accuracy achieved when using a supervised scorer and the accuracy obtained when using different unsupervised scorers. We also visualize in Figure 8 the drop in performance when using the best-unsupervised scorer instead of the supervised scorer. Interestingly some methods such as KMM, EntOT, and ClassRegOT can lose up to 10% accuracy when using realistic scorers, which might come from their higher number of parameters or their sensitivity to them.
Our results thus show that most scorers have poor results when evaluated on many datasets. Of the six methods under considcration, only two achieve satisfactory performance, although incurring large variance in their results. This shows that proper hyperparameter selection is still an open question, that needs attention from the research comunity to guide practitioners toward real life applications of unsupervised DA technics.
4.3 Deep DA methods
Table 3: Accuracy scores for Deep methods on selected real-life datasets using DA scorers. LinOT is reporled as the overall lop hop-performing shallow method. Green indicates that the score improved with the DA methods. The darker the color, the more significant the change.
Although most of the recent work on domain adaptation focus on Deep methods for computer vision tasks, shal- low methods are competitive in many applications such as tabular data (Grinsztajn et al., 2022) or datasets with a rela- tively small number of training examples such as BCI (Cheval- lier et al., 2024). Moreover, shallow methods can also ben- efit from recent advances in Deep learning by using Deep pre-trained feature extraction (transfer learning). However, to the best of our knowledge, the literature lacks quantitative comparison between shallow methods applied on Deep pre-trainod feature extraction and Deep DA methods. To this end, we ran a benchmark using the same pipeline as in Table 2 with three Domain Invariant Deep DA methods on the CV and BCI datasets.
MNIST/USPP /USPS
Selected Scorer
OfficeHome
Office31
Rank
BCI
Train Src
0.85
0.77
0.58
0.54
6.19
Train Tgt
0.98
0.96
0.83
0.56
2.07
DeepCORAL (Sun & Saenko, 2016)
0.93
0.77
0.59
0.54
Mix Val
3.29
DAN (Long et al., 2015b)
0.86
0.75
0.56
0.53
IW
4.76
DANN (Ganin et al., 2016a)
0.9
0.79
0.59
0.41
MixVal
4.98
DeepJDOT (Damodaran el al., 2018b)
0.9
0.82
0.62
0.54
PE
2.92
MCC (Jin et al., 2020)
0.93
0.83
0.66
0.53
Mix
2.38
MDD (Zhang et al., 2019)
0.87
0.78
0.56
0.4
MixVal
4.96
SPA (Xiao et al., 2023b)
0.91
0.78
0.56
0.41
DEV 5.39
LinOT (Flamary et al., 2020)
0.64
0.6
0.57
0.61
CireV
Results table. The results are available in Table 3 with comparison to the best performing shallow method from Table 2. One of the most notable and expected difference is on MNIST/USPS. Shallow methods struggle to achieve good performances, even on Train Tgt, as they rely on PCA for feature extraction. Deep methods, on the other hand, use CNNs, leading to large accuracy gains on train on Sre and Tgt but also on Deep DA methods. However, it is important to note that while DeepJDOT, DANN and MCC improve performance on all datasets, they remain far from the train on Tgt accuracies, partly due to the difficulty tuning their parameters (see Appendix E.3 with the supervised scorer). The superior performance of Deep DA methods on CV datasets can be attributed to the relationship between classification in the DA subspace and the disentanglement of semantic (discriminant) content from style (domain shift) (Gonzalez-Garde et al., 2018; Gabbay et al., 2021). Numcrous studies have demonstrated that semantic embeddings can be effectively recovered, supporting the assumption that a (nonlinear) subspace shift is reasonable for CV tasks. However, for the BCI dataset, where the amount of data is limited, the performances of Deep DA methods are inferior to some other shallow methods (i.e., LinOT for example). Finally, a method like DANN, which often considered as a baseline in the community, has been shown be hard to validate and requires setup that can be difficult to determine across different settings. These results emphasize, that while Deep invariant. DA
11

---

Published in Transactions on Machine Learning Research (07 (07/2025)
methods can be effective, they do not consistently yield good results across modalities, whereas shallow DA methods can achieve similar or superior performances with less effort and fewer computational resources in low data regimes.
Limitations and future work. The evaluation of deep DA methods in this benchmark is limited to a single outer split to ensure computational feasibility. While practical, this setting may introduce variance performance estimates, especially on small datasets. Moreover, although all deep methods completed within the allocated time, the 4-hour timeout may restrict the exploration of more complex architectures or longer training procedures on larger datasets.
We view these constraints as limitation of the current benchmark and an opportunity for future work. A comprehensive evaluation of deep DA methods would benefit from larger datasets (> 10 samples) with more stable training dynamics, and variacve evaluation using a single train/ hotn/validaiioo/teeee split. Extending DA-Bench to support such large-scale evaluations represents a promising direction for improving the practical assessment of deep DA approaches.
5 Conclusion
In this work, we introduced SKADA-Bench, extensive benehmark for unsupervised domain adaptation, carefully evaluating the impact of the model selection criteria and covering diverse modalities: computer vision, natural language processing, tabular data and biosignals. While being quite comprehenvise methods, our results also provide a comparison of three common deep DA baselines on computer vision and biosignals. Importantly SKADA-Bench can be easily extended with new datasets and methods to push further the state-offthe-art. Our findings reveal that few shallow DA methods consistently perform well across diverse datasets and that model selection scorers significantly infiuence their effectiveness. While deep DA methods show similar trends, they often require more extensive hyperparameter tuning and architectures tailored to each modality. Notably, they tend to perform significantly better than shallow methods some modalities, such as computer vision, while facing challenges on others such as biosignals. For each DA method, we provide the optimal model selection scorer for unsupervised hyperparameter tuning based on our experiments.
shallow
Acknowledgments
This work benefited from state aid managed by the Agence Nationale de la Recherche under the France 2030 programme, reference ANR-23-IACI-0006, ANR-22-PESN-0012 and ANR-20-CCIAA-0116 This research was also supported in part by the French National Research Agency (ANR) through the MATTER project (ANR-23- ERCC-0006-01) and the BenchArk project (ANR-24-AS2--0003) It received funding from the Fondation de l'Ecole polytechnique. Additionally, this project received funding from the European Union's 's Horizon Europe rescarch and innovation program under grant agreement 101 101120237 (ELIAS).
All the datasets used for this work were accessed and processed on the Inria and IP Paris (IDCS) compute infrastructures.
References
Bruno Aristimunha, Igor Carrara, Pierre Guetschel, Sara Sedlar, Pedro Rodrigues, Jan Sosulski, Divyesh Narayanan, Erik Bjarcholt, Barthelemy Quentin, Robin Tibor Schirmeister, Emmannel Kalunga, Ludovic Darmet, Cattan Gregoire, Ali Abdul Hussain, Ramiro Gatti, Vladislav Goncharenko, Jordy Thielen, Thomas Moreau, Yannick Roy, Vinay Jayaram, Alexandre Barachant, and Sylvain Chevallier. Mother of all bei benchmarks v0.5. foi.org// 10. 5081/220o0....../ 10034223, 2023. DOI: 10.5281/2cnodo.1003222..
Alexandre Barachant, Stéphane Bonnet, Marco Congedo, and Christian Jutten. Multiclass brain-computer interface classification by riemannian geometry. IEEE Transactions on Biomedical Engineering, 59(4): 920-928. 2012. doi: 10.1109/ D0.1109/TBBB...
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Systems, volume 19. MIT Press, 2006. URL http::/proceeenggg.... http:p////ppcccoooeeoooopp//pp/p
pdf.
Lorenzo Bruzzone and Mattia Marconeini. Domain adaptation problems: A dasvm classification technique and circular validation strategy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(5): 770 787, 2010a. doi:
Lorenzo Bruzzone and Mattia Marconeini. Domain adaptation problems: A dasvm classification technique and a circular validation strategy. IEEE Transactions on Pallern Analysis and Mochine Intelligence, 32(5): 770-787, 2010b. doi: 10.1109/TPAML2200 TP0ML.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International conference on machine learning, pp. 872 881. PMLR, 2019.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aom sigkdd international conference on knowledge discovery and data mining, pp. 185-794, 2016.
Sylvain Chevallier, Igor Carrara, Bruno Aristimunha, Pierre Guetschel, Sara Sedlar, Bruna Lopes, Sebastien Velut, Salim Khazem, and Thomas Moreau. The largest eeg-based bei reproducibility study for open science: the moabb benchmark. arXiv preprint arXiv:2404.15319 2024.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In I. Guyon, U. Von Luxburg, S. Bengio, Н. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017a. URL https: pttt::://procceiing... nww.ieseripp... httppp/ppppppppeteeepppppppprrre wtwps//pppe/ee/////// f11e/0000230011465300aaaaaaa2 Paw.eee----aappppppppp påf.
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(9):1853-186, 2017b.
doi: 10.1 209/TPAMM..01..001
Wenyuan Dai, Qiang Yang, Gui-Rong Xue, and Yong Yu. Boosting for transfer learning. In Proceedings of the 24th International Conference on Machine Learning, TCML. '07, pp. 193-200 200, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 9781595937933, doi: (0.114/1273446.1227 https://doi http:///oooorrg////// 10. 1111/1173/1/111 1273521.
URL
Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Europeon semanticschoolar
Conference on Computer Vision, 2018a. URT htttp:://ap...senntttcccooooo www.saschhhaaaa.orrrrrrreeeeeeee org/CorpusID:43311
Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Computer Vision ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV. pp. 467-483 Berlin, Heidelberg, 2018b. Springer- Verlag. ISBN 978-3-030-01224-3 doi: 10.1007 0tt/:/99988------00
078-0-030001125-0 28. URL https::////ol orgg
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEFE conference on computer vision and pottern recognátion, pp. 248-255, 255. Icco, 2009.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 647 655, Bejing, China, 22 24 Jun 2014. PMLR. URL http:/innmmm.....ess//////// wwwppprrssss/soeee https///vv/ddnnaueeeeee donahue html.
attps://proceedings. mlr,
Hassan Ismail Fawaz, Ganesh Del Grosso, Tanguy Kerdoneuff, Aurelic Boisbunon, and Illyyne Saffar. Deep un- supervised domain adaptation for time series classification: benchmark. arXiv preprint 2xXi:23312008857 2023.
13

---

Published in Transactions on Machine Learning Research (07/2025)
Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars. Unsupervised visual domain adaptation using subspace alignment. In 2013 TEFE International Conference on Compuler Vision, pp- 2960 2967, 2013. doi: 10.1109/ICCV.20133 2013.368.
Rémi Flamary, Karim Lounici, and André Ferrari. Concentration bounds for linear monge mapping estimation and optimal transport domain adaptation, 2020.
Aviv Gabbay, Niv Cohen, and Yedid Hoshen. An image is worth more than a thousand words: Towards disentanglement in the wild. Advances in Neural Information Processing Systems, 34:9216 9228, 2021.
Chan-Chistoophe Gagnon-Audet, Kartik Ahuja, Mohammad Javad Darvishi Bayazi, Pooneh Mousavi, Guillaume Dumas, and Irina Rish. WOODS: Benchmarks for distr--dis generalization in time series. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: Featured Certification.
//openreview net/forumnid-mvtteea
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural net Journal of machine learning research, 17(59):1-35 35, 2016a.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-advensarial training of neural networks. Journal of Machine Learning Research, 17(59):1-33 35, 2016b. URL attp://jmlz wttlii/llrr./.eeeeee....eeee.eee www.ooog//eee....e wwtg/appers/11/11522225 .hmml.
Joshua P Gardner, Zoran Popovi, and Ludwig Schmidt. Benchmarking distribution shift tabular data with tableshift. In Thirty-sevent Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL httpp:://peanerrie..eee///
Théo Gnassounuu, Oleksii Kachaiev, Rémi Flamary, Antoine Collas, Yanis Lalou, Autoine Mathelin, Alexandre Gramfort, Ruben Bueno, Florent Michel, Apolline Mcllot, Virginic Loison, Ambroise Odomnat, and Thomas Moreau. Skada Scikit adaptation, July 2024. URL https: / 12666838.
5281/2enodo
Abel Gorzalez-Garcia, Joost Van De Weijer, and Yoshua Bengio. Image-lo-image translation for cross-domain dlsentanglement. Advances in neural information processing systems, 31, 2018.
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical labular data? Advances in neural information processing systems, 35:507 520, 2022.
Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(1), June 2019. ISSN 2052-4463. doi: 10. 10.1088/4411597-09-00003
URL http: //xx. http://xxxxxxooo//o/ooooooooooot
.ww.torooooooo..oooooo..ooooooo. orgg wtt///..//// 100.11000/11157701-0000000
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi:
Dapeng Hu, Jian Liang, Jun Hao Liew, Chuhui Xue, Song Bai, and Xinchao Wang. Mixed samples probes for unsupervised model selection in domain adaptation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.). Advances in Neural Information Processing Systems, volume 36, pp. 37923 37941. Curran Associates, Inc., 2023. URL https:// neurips...... cctpaper_fileeeeeeeeeeeeeeppppee files/ Satte///eatt2te--ppee-----reeeee .dff.
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006. URL https:s//procceeeiinne nwurips..co. cc/ paper paper_files/pageeeee happr:/ieeepgeeooooffoooooollloe
Matthew Hutson. Artificial intelligence faces reprodueibility crisis, 2018.
14

---

Published in Transactions on Machine Learning Research (07/2025)
Yuanfeng Ji, Lu Zhang, Jiaxiang Wu, Bingzhe Wu, Lanqing Li, Long-Kai Huang, Tingyang Xu, Yu Rong. Jie Ren, Ding Xue, Houtim Lai, Wei Liu, Junzhou Huang, Shuigeng Zhou, Ping Tuo, Peilin Zhao, and Yatao Bian. Drugood: Out-of-distribution dataset curator and benchmark for ai-aided drug discovery a focus on affinity prediction problems with noise annotations. Proceedings of the AAAI Conference on Artificial Intelligence, 37(73:8023 8031, Jun. 2023. doi: 10. 10.1609/aaae v3777....77 URL https:: aaai org/ index. pha/ /ttp//////arrcceeeiee///e///
Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. Transferability in deep learning: A survey, 2022.
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation, 2020. URL httpps//arxiiv.org/bb///2/.. 03699.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, lan Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Becry, Jure Leskovee, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benehmark of in-the-wild distribution shifts. Tn Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5637 5664. PMLR, 18 24 Jul 2021. URL https: https://proceeddngs mlr. wwt.mmeer........e.. press/v139/koh21aa html.
Piotr Koniusz, Yusuf Tas, and Fatih Porikli. Domain adaptation by mixture of alignments of second-or higher-order scatter tensors. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7139 7148, 2017. doi: 10. D0.109//CVPR.217..735
Ken Lang. Newsweeder: Learning to filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning, pp. 331-339, 1995.
Zhibin Liao and Gustavo Carneiro. Competitive multi-scale convolution, 2015.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 97 105, Lille, France, 07-09 Jul 2015a. PMLR. URL httpss//proceeddiin
www.mmlrrr.sss... pttppp///////ooot/// html.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks, 2015b. URL https: //arxivv/// org wwtg/abb/1502.00000 02791.
M Loog. Nearest beighbor-basedd importance weighting. In SN (ed.), The 2012 IEEE workshop on machine learning for signal processing (MLSP) proceedings, pp. 1 1-6, United States, 2012. IEFF. ISBN 978-1-4673- 1026-0/12. The 2012 IEEE workshop on machine learning for signal processing (MLSP): Conference date: 23-09-2012 Through 26-09-2012.
Peter Mattson, Vijay Janapa Reddi, Christine Cheng, Cody Coleman, Greg Diamos, David Kanter, Paulius Micikevicius, David Patterson, Guenther Schmuelling, Hanlin Tang, Yu-Yeon Wei, and Carole-Jean Wu. Mlperf: An industry standard benchmark suite for machine learning performanco. IEEE Micro, 4062):8-16, 2020. doi: 10.1109/ M0/099MMM.000 MtT.:/MM..202292922 292.2444444.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pp. 165 172, 2013.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR '15, pp. 43 52, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450336215. doi: 10.1145 10.1145/2766462. 22655//66666222222 27677555. 10. 10e111117/26444 2767755.
URL /ttpp::/ddi...o
15

---

Published in Transactions on Machine Learning Research (07/2025)
Rami M. Mohammad, Fadi Thabtah, and Lee McCluskey. An assessment of features related to phishing websites using an automated technique. In 2012 International Conference for Internet Technology ond Secured Transactions, pp. 492 497, 2012.
Thomas Moreau, Mathurin Massias, Alexandre Gramfort, Pierre Ablin, Pierre-Antoine Bannier, Ben- jamin Charlier, Mathieu Dagréou, Tom Dupre la Tour, Ghislain Durif, Cássio Fraga Dantas, Quentin Klopfenstein, Johan Larsson, En Lai, Tanguy Lefort, Benoît Malézieux, Badr Moufad, Binh Nguyen, Alain Rakotomamonjy, Zaccharie Ramzi, Joseph Salmon, and Samuel Vaiter. Benchopt: Reproducible, efficient and collaborative optimization benchmarks. In Alice H. Oh, Alekh Agarwal, Danielle Bel- grave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL Sttpp://openrewee
Jose G. Moreno-Torres, Troy Raeder, Rocio Alaiz Radrinoori,,,e, Nitesh V. Chawla, and Francisco Herrera. A unifying view dataset shift in classification. Pottern Recognition, 45(1):521-530, January 2012. doi: 20100///1patcog.2111......
Pictro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal-entropy correlation alignment for unsupervised deep domain adaptation, 2017.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Unsupervised domain adaptation: A reality check. arXiv preprint arXix:2111.1677 2021.
Sinno Jialin Pan, Ivor W. Tsang, Jamos T. Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2))199 210, 2011. doi: 10.1109/TNN, 2010.80
Joclle Pincau, Koustuv Sinha, Genevieve Fried, Rosemary Nan Ke, and Hugo Larochelle. Iclr reproducibility challenge 2019. ReScience C, 5(2):5, 2019.
Joaquin Canden, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. MIT Press, 2008.
Tevgen Redko, Nicolas Courty, Rémi Flamary, and Devis Tuia. Optimal transport for multi-source domain adaptation under target shift. In The 22nd International Conference artificial intelligence and statistics, pp. 849-858. PMLR, 2019.
levgen Redko, Emilie Morvant, Amaury Habrard, Mare Sebban, and Younès Bennani. A survey on domain adaptation theory: learning bounds and theoretical guarantees, 2022.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), EMNLP CONNL/IICNLPP.... (1), pp. 3980-3990. Association for Computational Linguistics, 2019. ISBN 978-1-950737-90-1. URL de/db/cont / /e////mm//////eeee//ee hwtpn///emmnpp011-11111 html:
uni-trier.
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang. Extending the WILDS benchmark for unsupervised adaptation. In International Conference on Learning Representations, 2022. URL hutps:://ppnreee saaeeanreviee.ne//frrmeee nett//orumm.d==2222ottt.t
Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9164-9173, 2021. doi: ACC1199/TTCCA48822220000000000
Robin Tibor Schirmeesterr Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for ceg decoding and visualization. Human brain mapping. (11):5391-54200 -420, 2017.
16

---

Published in Transactions on Machine Learning Research (07/2025)
José Megovia-Martin, Santiago Mazuelas, and Anqi Liu. Donble-weighting for covariate shift adaptation. In International Conference on Machine Learning, pp. 10439-30455) PMT.R, 2023.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2):227 244, 2000. ISSN 0378-3758. doi: https: //dbb..rr/110.101///0377- (00000000000000000000010---44 URL http:://www..ceeendrrrcccccoo pis//03788375800001544
Si Si, Dacheng Tao, and Bo Geng. Bregman divergence-based regularization for transfer subspace learning. IEEE Transactions on Knowledge and Data Engineering, 22(7):929 942, 2010. doi: 10.1109/TKDE.2000
Justin Solomon, Raif Rustamov, Leonidas Guibas, and Adrian Butscher. Wasserstein propagation for semi-supervised learning. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 306 314, Bejing, China, 22-24 Jun 2014. PMLR. URL https: ttttps:/pproceednngge mlr. ppwss/v32/sooessee
html.
Masashi Sugiyama and Klaus- Robert Müller. Iuput-dependent estimation of generalization error under covariate shift. Statistics & Risk Modeling, 23(4):249 279, 2005. doi: 200.11552//tmdd.205.222233 URL
https://doi. org/ 10. 1524/stnd. 2005 23.9.249.
Masashi Sugiyama, Matthias Krauledat, and Klaus-Roben Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8:985-1005, 05 2007a.
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007b. URL ptteps:/pproceedingg introoceednngn.neeriipee /atrripps.c/ppppr_tee//////perre
Paper. f11e//eeabeccd0077a00000a... Pappeer..1-Ppp.......... påf.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Gang Hua and Hervé Jégou (eds.), Compuler Vision ECCV 2016 Workshops, pp. 443-454- Cham, 2016. Springer International Publishing. ISBN 978-3-319-49409-8
Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation Alignment for Unsupervised Domain Adaptation, pp. 153-171. Springer International Publishing, Cham, 2017. ISBN 978-3-319-58347-1 doi: 10.1007 978-3-319-5834711 8. URL https://dolt http::///dooooo////// 100.111000000--111---88888
Michael Tangermann, Klaus-Robert Müller, Ad Aertsen, Niels Birbaumer, Christoph Braun, Clemens Brunner, Robert Leeb, Carsten Mehring, Kai Miller, Gernot Mueller-Putz Guido Nolte, Gert Pfurtscheller, Hubert Prcissl, Gerwin Schalk, Alois Schlögl, Carmen Vidaurre, Stephan Waldert, and Benjamin Blankertz. Review of the BCI Competition IV. Frontiers in Neuroscience, 6, 2012. ISSN 1662-453X. URL https: //www.ffrete frontiersin org/articles/10 301333389///n fnins.iinn.....o/ 2002.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
Jindong Wang. Everything about transfer learning and domain adapation, 2018.
URL http://
transferlearning.xyy xyz.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023a.
Zhiqing Xiao, Haobo Wang, Ying Jin, Lei Feng, Gang Chen, Fei Huang, and Junbo Zhao. Spa: A graph spectral alignment perspective for domain adaptation, 2023b. URL https://arxis: www.riiii.or//bbbae wtwp/rrg/////rree 17594.
Andy Yoo, Morris Jette, and Mark Grondona. Shurm: Simple linux utility for resource management. Workshop on job scheduling strategies for parallel processing, pp. 44 60. Springer, 2003.
17

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan. Towards accurate model selection in deep unsupervised domain adaptation. In Kamalika Chaudhuri and Ruslan Salakhuidinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7124-7133. PMLR, 09-15 Jun 2019. URL https://proceeddin www.innsmm.......mmm.m..e.m.eamm wtt..pres//v///// you19a. html.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 819 827, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL https://prrceedingss mlr. press/v28/zhanggee html.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael I. Jordan. Bridging theory and algorithm for 05801.
domain adaptation, 2019. URL https://arxiv..
Erheng Zhong, Wei Fan, Qiang Yang, Olivier Verscheure, and Jiangtao Ren. Cross validation framework choose amongst models and datasets for transfer learning. In José Luis Balcázar, Francesco Bonchi, Aristides Gionis, and Michèle Sebag (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 547-562, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg. TSBN 978-3-642-1558 1593998-.
A Appendix
Appendix
Reproducibility. The entire code and results of SKADA-Bench are open-sourced at https://github. com/ scikit- awwettttaaatttiiiinaaaaa henn/skaaa-bente The implementation of the DA met hods and scorers is provided along with access to the simulated and real-world datasets. All the performance tables and figures can be reproduced effortlessly, and guidelines with minimal working examples are given add new DA methods and datasets.
Roadmap. In this appendix, we provide additional information regarding the validation procodure used in the literature for each DA method implemented in SKADA-Bench in Section B. We provide detailed description of the data and preprocessing used in SKADA-Bench in Section C. In Section D, we give minimal working Python examples to add a new DA method and dataset in SKADA-Bench. Finally, we provide the detailed benchmark results in Section E. In particular, the results per dataset can be found in Section E.1. We discuss in Section E.2 the impact of the choice of base estimator on the performance of DA methods for the simulated datasets. The results of each DA method with the supervised scorer on all the datasets are given Table 20 of Scction E.3, which parallels Table A thorough analysis of the effect of using realistic unsupervised scorers is also provided in Section E.6. Finally, the computational efficiency of each DA method is studied in Section E.7 and the hyperparameters used for grid search are given in Section E.8. We display the corresponding table of contents below.
Table of Contents
B Model selection in Domain Adaptation
20
C Datasets description and preprocessing
21
D Adding new methods and datasets to SKADA-Bende
21
D.1 Adding a new DA method
22
D.2 Adding a new dataset.
22
18

---

Published in Transactions on Machine Learning Research (07 (07/2025)
E Benchmark detailed results
23
E.1 Results per datasets
23
E.2 Impact of the base cstimators on the simulated datasets
34
E.3 Unrealistic validation with supervised scorer ................................
................................
E.4 Fl-score of benchmark
................................
39
E.5
Comparison of the rank of DA scorer
................................
39
E.6 Comparisons between supervised and unsupervised scorers
40
E.7
Computational efficiency the DA methods
E.8
Hyperparamcters grid search for the DA methods and neural networks training
................................ 45
19

---

Published in Transactions on Machine Learning Research (07 2025)
B Model selection in Domain Adaptation
Table 4: Validation procedure in Domain Adaptation methods. NA stands for not applicable and means that there are no hyperparameters. None means that validation procedure has been conducted or that it is not. specified in the original paper.
Method
Validation Procedure
Comment
Density Reweight
Bandwidth fixed by Silverman method
None
Reweighting
(Sugiyama & Miller, 2005)
Discriminative Reweight
NA
No hyperparameters
(Shimodaira, 2000)
Gaussian Reweight
Not specified in (Shimodaira, 2000)
None
(Shimodaira, 2000)
Likelihood CV (Sugiyama et al., 2007b)
KLIEP
Integrated CV
(Sugiyama et al., 2007b)
target Fixed data-dependent
KMM
None
(Huang et al., 2006)
hyperparameters
NN Reweight (Loog, 2012)
Number of neighbors
None
fixed to one
MMDTarS
Not specified
CV
(Zhang et al., 2013)
done source target
Coral
Mapping
NA
No hyperparameters
(Sun et al., 2017)
OT mapping
CV target/CirrcCV
Unclear in the text
(Courty et al,, 2017b)
Lin. OT mapping
NA
No hyperparametens
(Flamary et al., 2020)
MMD-LS
Not specified if done on source or target
CV
(Zhang et al., 2013)
SA (Fernando et al., 2013)
2-fold CV on source
Subssp
TCA
Target subset used to time parameters Not specified in (Si et al., 2010)
Validation on target
(Pan et al., 2011)
TSL. (Si et al., 2010)
None
JDOT (Courty et al., 2017a)
Reverse CV (Zhong et al., 2010)
Other
OT label prop
NA
No hyperparameters
(Solomon et al., 2014)
DASVM (Bruzzone & Marconcini, 2010a)
Circular Validation
(Bruzzone & Marconeini, 2010a)
In Table 4, we provide additional information on the validation procodures used in the original papers that proposed the different domain adaptation methods implemented in SKADA-Bench. The first column is the name of the method, the second column contains the procedure used to select hyperparameters and the last column provides additional details. What is striking is that many methods do not conduct specify a validation procedure to select the hyperparameters, which limits the performance of the proposed method on a novel dataset. Several others rely on cross-validaatin using target data. However, since targel labels are typically unavailable in practical scenarios, this validation approach unrealistic. Overall, many methods have been evaluated with unrcalistic or not reproducible validation procedures, making the performance of the proposed methods appear overooptimistic. A key contribution of our work is the extensive comparison of realistic, unsupervised scorers for selecting optimal hyperparameters and base estimators in DA methods.
20

---

Published in Transactions on Machine Learning Research (07 (225))
C Datasets description and preprocessing
The simulated dataset proves that DA methods can work well under the proper shift (see Table 2). However, in real-world applications, we do not have prior knowledge of the type of data shift. Hence, finding the appropriate domain-adaptation method between reweighting, mapping, and subspace methods is a challenging task. In this section, we introduce 8 real-world datasets coming from different fields. Table summarizes the 8 classification datasets used in this benchmark with the corresponding data modality, preproccssing, number of source-targel pairs (# adapt), number of classes, samples, and feature dimensions.
Computer Vision. First, three computer vision datasets are proposed: Office31 (Koniusz et al., 2017), Office Home (Venkateswara et al., 2017), and MNIST /USPS (Liao & Carneiro, 2015). We create embeddings for Office31 using the Decaff preprocessing method (Donahue et al., 2014) and for Office Home using a pre- trained ResNet50 (He et al., 2016). These embeddings, as well as vectorized MNIST/USPS, are dimensionally reduced with a Principal Component Analysis (PCA). These three datasets encompass 3, 4, and 3 domains, respectively and all pairs of adaptations are used as DA problems. MNIST/USPP contain clear and blurry images digits, Office3l differentiates between images captured by various devices, while for OfficeHome,, its by image style.
NLP. The second task is Natural Language Processing (NLP). Two datasets are studied: 20Newsgroup (Lang, 1995) and Amazon Review (McAuley et al., 2015). The 20Newsgroup dataset contains 20.000 documents categorized into 4 categories: talk, rec, comp, and sci. The learning task is to classify documents across categories. First, the documents are embedded using a Large Language Model (LLM) (Reimers & Gurevych, 2019; Xiao et al., 2023a), and then PCA is applied for dimensionality reduction.
For the Amazon Review dataset, the task is to classify comment ratings. This dataset spans four domains (Books, DVDs, Kitchen, Electronics), and the domain shift results from these varying types of objects. Similar to the 20Newsgroup dataset, comments are embedded using the same LLM and then reduced in dimensionality using a PCA.
Tabular data. We propose two tabular datasets. The first one is the Mushroom dataset (Dai et al., 2007), where the task is to classify whether mushroom is poisonous or not. The two domains are separated according to the mushroom's 's stalk shape (enlarging vs. tapering). The tabular data are one-hot-encooed to transform categorical data into numerical data. The second dataset is Phishing (Mohammad et al., 2012). The classification problem involves determining whether a webpage is a phishing or a legitimate one. The domains are separated according to the availability of the IP address. Since the data are already numerical, preprocessing is done on this dataset.
Biosignals. The last task is BCI Motor Imagery. The dataset used is BCI Competition IV (Tangermann et al., 2012), often used in the literature (Barachant et al., 2012) and availaoble in MOABB (Aristimunha et al., 2023). The task is to classify four kinds of motor imagery (right hand, left hand, feet, and tongue) from EEG data. In this dataset, nine subjects are available. The domains are separated based on session number, For each subject, session is considered as the source domain and session 2 is considered as the target domain. The data are multivariate signals. To embed the data, we first compute the covariance and then project this covariance on the Tangent Space as proposed in Barachant et al. (2012).
D Adding new methods and datasets to SKADA-Bench
Using the benchopt framework for this benchmark allows users to easily add novel domain adaptation (DA) methods and datascts. To that end, users should adhere to the benchopt (Moreau et al., 2022) conventions. We provide below the guidelines with examples in Python to add a new DA method and a new dataset to SKADA-Bench.
21

---

Published in Transactions on Machine Learning Research (07 (07/2025)
D.1 Adding new DA method
A new DA method can be easily added with the following:
Create file with class called Solver that inherits from DASolver and place it in the solvers folder.
This class should implement get_estimator () function, which returns class inheriting from sklearn. BaseEstimator and accepts sample sample_weight weight as fit parameter. In the benchmark we used the Domain Adaptation toolbox SKADA (Gnassounou et al., 2024) that provides many DA estimatos with correct interface.
We provide below example of Python implementation to add a new DA method to SKADA-Bench
# Python snippet code to add DA method
from benchmark. utils. base solver import DASolver from sklearn. base import BaseEstimator
class MyDaEstimator (BascEstimator): def init (self, paraml=10, param2-'auto "):
self paraml paraml self. param2 = param2
def fit (self, X, sample weight=None):
# sample_weights weight<O are source samples # Semple_weightae # y conlains -1 for masked largel samples
are target samples
# Your code here store stuff in self for later predict return self
def predict (self, X):
# do prediction on target domain here return ypred
def predict_probasset X):
# do probabilistic prediction on target domain here return proba
class Solver (DASolver):
name "My DA method"
# Param grid to validate default default_param___ grid = 'paraml's : 100 100).
"param2'; l'auto" "manual" ']
def cot__ssimattrre (ootoorrrsoh (
return MyDAEstimator ()
D.2 Adding new dataset
A new DA dataset can be easily added with the following:
Create file with class called Dataset that inherits from BaseDataset and place it in the datasets
folder.
22

---

Published in Transactions on Machine Learning Research (07/2025)
This class should implement get_data function, which returns dictionary with keys X, y, and sample__dee domain.
We provide below example of Python implementation to add dataset to SKADA-Bence
# Python snippet code to add a dataset
from benchopt import BaseDataset from sklearn datasets import make blobs import numpy np
class Dataset (BaseDataset): name = "cxample_datee dataset"
def get_data data self):
X source, y source make blobs (
100, centers=3,
features=2, random_statee state=0
_taargt,, y __tageet = make_bloo (
n samples=100, centers=5, features=2, random_staal state=42
#
for source samples
# comple_domain<l for target samples
sample domain = up- array ([1] len (X source) + [-2] len target))
return dict(
X-np. concatenate (( (o_soe X X_target), axis=0)
y=np. concatenate ((y ((y_source, y target))
sample domain-sample domain
By following these guidelines, users can seamlessly integrate their own datasets and DA methods into SKADA-Bench. It results in a user-friendly benchmark that enables fast, reproducible, and reliable comparisons of common and novel DA methods and datasets. We will provide users with precomputed result files and utilities, allowing them to run only the new methods or datasets. This will speed up new comparisons and avoid unnecessary computations.
E
Benchmark detailed results
E.1 Results per datasets
In Table 2 of the main paper, the reported performance for each method on given dataset an average over the number of shifts, ie., the number of source-targel pairs denoted by #adapt in Table 1. In this section, provide additional details the performance of methods for each shift in each dataset. These results are presented in separate tables for each datasct
These detailed tables where cell in green denote a gain wrt Train Sre (average outside of standard deviation of Train Src) better illustrate the challenges of domain adaptation (DA) methods. They show that not all shifts are cquivalent within a given dataset. For example, Table 12 reveals that only shifts in the Revzon dataset present a DA problem (defined Src and Train Tgt). While for the other shifts, we achieve similar performance whether train source
3% difference in accuracy between Train
23

---

Published in Transactions on Machine Learning Research (07 (07/2025)
target data. Additionally, some specific shifts present DA problem that method can successfully address. This can be seen in the dsl amz shift. in the Office31 dataset, as shown in Table 7. Finally, some DA methods perform consistently across all shifts within dataset, as demonstrated by the results for the 20Newsgroup dataset in Table 11.
Table 5: Accuracy score for MNIST/USPS dataset for each shift compared for all the methods. A white color means the method does not increase the performance compared to Train Sre (Train on the source). Green indicates that the performance improved with the DA methods. The darker the color, the more siguificant. the change.
MNIST MNIST-UUSPS +USPS
USPS MNIST
Mean
Rank
Train Src
0.66 0.02
0.43 + 0.02
0.54 + 0.02
12.00
Train Tgt
0.96 + 0.0
0.96 # 0.01
0.96 + 0.01
1.00
Dens. RW
0.66 0.02
0.42 1 0.02
0.54 + 0.02
13.25
Disc. RW
0.6 0.02
0.4 - 0.02
0.5 = 0.02
19.00
Reweighting
Gauss. RW
0.11 0.01
0.11 0.01
0.11 = 0.01
20.00
KLIEP
0.66 0.02
0.43 + 0.02
0.54 + 0.02
13.25
KMM
0.64 = 0.02
0.41 + 0.03
0.52 + 0.02
18.00
NN RW
0.66 = 0.02
0.42 = 0.02
0.54 0.02
12.00
MMDTarS
0.66 0.02
0.42 + 0.02
0.54 + 0.02
12.75
CORAL
0.74 0.01
0.51 0.01
0.62 + 0.01
5.50
Mapping
MapOT
0.69 0.02
0.54 0.02
0.61 + 0.02
4.00
EntOT
0.66 0.02
0.54 0.02
0.6 + 0.02
5.00
ClassRegOT
0.66 0.01
0.53 = 0.06
0.59 = 0.04
11.50
LinOT
0.74 = 0.02
0.53 + 0.02
0.64 + 0.02
3.25
MMD-LS
0.66 - 0.02
0.47 + 0.02
0.56 + 0.02
8.25
Other Subspace
JPCA
0.66 0.02
0.43 0.02
0.54 + 0.02
12.00
SA
0.71 0.03
0.36 + 0.11
0.54 + 0.07
12.00
TCA
0.08 0.07
0.11 0.02
0.09 = 0.05
21.00
TSL
0.66 0.02
0.43 - 0.02
0.54 + 0.02
10.50
JDOT
0.73 = 0.02
0.53 + 0.02
0.63 + 0.02
3.50
OTLabelProp
0.71 = 0.03
0.53 + 0.02
0.62 + 0.02
6.50
Table 6: Accuracy score MNIST/USPS dataset for each shift compared for all the Deep DA methods. A white color means the method does not increase the performance compared Train Sre (Train on the source). Green indicates that the performance improved with the DA methods. The darker the color, the more significant the change.
MNIST- USPS
-UPS
USPS MNIST
Mean
Rank
Train Sre
0.94
0.76
0.85
5.0
Train Tgt
0.99
0.99
0.99
1.0
DANN
0.94
0.88
0.91
4.0
DeepCORAL
0.97
0.89
0.93
2.5
Deep.DDOT
0.96
0.9
0.93
2.5
24

---

Published in Transactions on Machine Learning Research (07/2025)
Table 7: Accuracy score for Office31 dataset for each shift compared for all the methods. A white color means the method does not increase the performance compared to Train Sre (Train on the source). Green indicates that the performancs improved with the DA methods. The darker the color, the more significant. the change.
Rank
8.40
1.00
11.40
13.30
19.80
10.60
10.20
12.30
10.00
6.20
13.30
11.80
9.20
8,90
8,00
14.10
12.10
20. 00.00
9.20
6.20
8.75
0. 1 0.02
0,88 + 0.02
0.57 + 0.04
0 0.58 - 0.03
0.2 $ 0.03
0.59 - 0.03
0.59 + 0.03
0.58 = 0.02
0.57 = 0.05
0.6 - 0.03
0.54 = 0.03
0.58 + 0.02
0.55 1 0.08
0.6 0.03
0.59 0.03
0.58 + 0.02
0.07
0.03 0.0
0.50 + 0.03
U. = 0.03
Menn
0.55
0.61
0.01
0.01
0.01
0.46 + 0.01
0.03
+ 0.01
0.02
0.48 + 0.01
0.49 H 0.01
0.5 1 0.01
0.43 1 0.01
0.01
0.25
0.01
0.00
0.01
0.02
0.01
NA
wol web- -ama
0.04 +
0 49
0.
0.49
0.11
0.48
0.47
0.46
0.25
0.49
0.48
0.48
0.49
0.49
0.49
0.9 1 0.02
0.95 + 0.02
0.83 + 0.09
0.89 + 0.03
0.38 + 0.04
0.91 E 0.02
0.87 + 0.03
0.91 = 0.01
0.81 = 0.14
= 0.03
+ 0.03
1 0.02
0.04
0.02
0.02
0.01
0.0
0.9 0.02
0.77 + 0.04
0.89 18994 + 0.02
dal al-+web
0.91
0.85
0.88
0.84
0.9
0.9
0.89
0.84
0.04
0.5 1 0.02
0.78 + 0.02
0.5 F 0.02
0. 49 + 0.02
- 0.02
0.49 1 0.02
1 0,03
0.49 0.02
0.5 = 0.03
0.5 0.03
0.47 0.01
0.5 0.5 0.03
0.51 0.51 0.02
0.02
0.03
0.02
0.02
0.49 1 0.08
0.5 0.02
0.49 + F00
0.04 +
del ----
0.05
0.49
0.49
0.49
0.49
0.49
0.5
0.51 F 0.03
0,96 + 0.01
0.51 + 0.04
0.03
0.11 + 0.04
0.5 1 0.04
0.54 2 0.00
0.5 - 0.03
0.52 = 0.04
0.33 Z 0.03
0.5 Σ 0.04
0.56 1 0.02
0.59 T 0.04
0.56 + 0.05
0.51 + 0.04
0.51 + 0.02
0.5 0.03
0.03 1 0.0
0.51 d 0.04
0.63 0.02
0.63 + 0.02
AmI web
0.19
0.53 1 0.04
0.94 + 0.03
0.49 + 0.05
0.56 + 0.06
0.34 + 0.03
0.55 1 0.05
0.56 1 0.06
0.54 + 0.04
0.53 + 0.04
0.53 + 0.04
0.47 + 0.03
0.52 + 0.03
0.58 1. 0,07
0.54 + 0,05
0.55 + 0.04
0.53 + 0.04
0.53 1: 0.04
0.58 +: 0.03
amn--
0.42 E
+
0.64 +.
Train Src
Train Tgt
Dens. RW
Dise. RW
Ganss. RW
MMDTarS
ClassRegOT
MMD-LS
OTLabelProp
KLIEP
KMM
NN RW
CORAL
MapOT
EntOT
LinOT
JPCA
TCA
JDOT
TSL
SA
Mlappine
Other
25

---

Published in Transactions on Machine Learning Research (07/2025)
Table 8: Accuracy score for Office31 dataset for each shift compared for all the deep DA methods. A white color means the method does not increase the performance compared to Train Src (Train on the source). Green indicates that the performance improved with the DA methods. The darker the color, the more significant the change.
amzz Adsl
aunz >wbb
del Pammz
del del-wwbb web
web >amz
web web-idal del
Mean
Rank
Train Src
0.72
0.75
0.61
0.94
0.63
0.99
0.77
4.17
Train Tgt
0.99
1.0
0.87
0.99
0.88
0.99
0.95
1.25
DANN
0.75
0.8
0.64
0.96
0.62
0.98
0.79
3.75
DeepCORAL
0.77
0.77
0.61
0.98
0.63
0.99
0.79
3.33
DecpJDOT
0.79
0.8
0.68
0.97
0.69
1.0
0.82
2.17
26

---

Published in Transactions on Machine Learning Research (07/2025)
Table 9: Accuracy score for OfficeHomee dataset for each shift compared for all the methods. A white color means the method does not, increase the performance compared to Train Src (Train on the source). Green indicates that the performance improved with the DA methods. The darker the color, the more significant the change.
10:42
1.00
13:00
12.00
0.02
12:42
3.00
3.42
0.54
6.46
6:50
16.70
0.01
0.01
0.92
0.02
0.00
0.02
0.02
0.01
002
0.0
0.0
0.01
= 0.2
E 100
003
0.30
0.8
0.35
0.54
0.30
0.36
0.7
0:11
0.38
0.30
0.57
0.56
D74 0.12
0.
0.15
0.00
0.02
0.08
1 0:00
1 0.
1 0:01
0.02
24 0.00
PEL 0
0EE5 910
ONE
6:0
0.73
0.73
0.74
0.74
0.06
007
023
0:4
075
071
1.74
Yoo heitendd wolpprt
0.30 1 0.01
0.7 * 0:01
0.01
0.01
+ 0.01
0
0:01
0:01
0.01
: 0.01
0. 0:01
+
0.35
0.37
0.08
0.43
01
0.22
0-49
0.49
Reatenrid -err
0.00
0.02
- 0.00
0.01
0.01
_ 0.02
0.00
0:01
= 0.01
0.5 = 02
= 0:01
000 0.02
Tealent
0.63 =
=
0.75
0.60
0.00
0.0
0.0
0.5
0.8
0.6
Prodhcc -o www--retttoraaa
0.11 4 0:00
83 + 0:01
0. + 0.00
1 9:0
+ 0.01
1 101
1 0
4 102
+ 000
4 010
F 0:0
F 00
2 mm0
100
1 100
0.4
0.22
0
0:21
0:22
12
0.00
0.7
prou produer produc -eddiprrrr
0.38 1 0x2p
0.7 + 0:01
0/2
+ 0.00
000
+- 100
002
001
0011
= 001
2 000
ST.
120 000
0.50 =
0.38
0.38
0.34
0.50
0.50
0.35
0:57
0:5H
0:50
0
36
U3 = Da0
0.00
6,00
- 0000
000
0.4 --102 1002
21 1100 000
157999 02
0.9 == 030
== 0
2 00
000
0.0
2
023 7
0.72
0.35
0.48
03
03
0
0.54
000
022
cilprr wos srraaoood
0.03 0.30
0.01
0.00
0.02
900
0.0
+ 0:11
11 0.0
L 0:00
# 0
mm
6:83
0:00
6.5
641
663
020
69
Clpaae Cljaatt- mnredtee
0.8 I 1001
+ 0:01
0:00 0.01
0055+000 0.0
1 0.0
012
0:01
0 0.0
00
00
0.0 - 0.
028 E 000
0.91
0:38
=
0.01
0.00 _
07.55
032
0.5
6.55
030
06
0:46 + 0.02
+ 00
+ 0:02
0.45 + 0.02
+ 0.02
1 0
T 0:00
+ 0:01
+ 02
2 0.02
3 OZ
MA
0.00 E 0002
Allan wark
0.00 1
0011
0.75
0.45
0.45
01
055
0.5
0.40
0:45
odl Pevereddent
D.3 I. 0.00
8154 + 0:01
0.7 0.8
1 01
1 0.01
18 4 0:01
0- + 000
0 00
0.02
0.7 1
0.7 1
0.00
01
22
0.
0
0.3 = 0001
0.01
0.01
0.05
= 0.01
0.01
0001
001
0011
001
0.09
63 = 0.02
N.
0.01
0:02
03
0.03
0.
50
05
08
04
0.57
0
0:34 1. 0:02
0:00
0.31 2 0.02
0.04
0.02
0.02
0:30 0:02
0.36 03
0.28 0.02
0.30 0.02
0.30 + 0.09
0.34 4 0.09
0.34 = 0.00
NA
0.02
+
0:71
0.32
01 LabelPPp
ClawRegOT
Pran Stc. Train Fist
CORAL MMD'WW
MM
Dmox
Das:
Gaun,
KMM
RW
MapO'T
LivOTT
JPC
3DOT
27

---

Published in Transactions on Machine Learning Research (07/2025)
Table 10: Accuracy score for OfficeHome dataset. for each shift compared for all the deep DA methods. A white color means the method does not increase the performance compared to Train Sre (Train on the source). Green indicates that the performance improved with the DA methods. The darker the color, the more significant the change.
clipart- Shaart-rrat Freal trealworld world
product prochct-+reealvotte +reawworld
real realworld worrl >elipart
realworld +product
clipart-a cliprt--ppan product
product prochct-sclipart +clipart
art art-iclipart
art-sproduct
art +product
art. realloldddd
clipart- Eart
product aart
real maalwordd-m >art
Mean
Rank
Train Sre
0.42
0.62
0.75
0.53
0.6
0.62
0.52
0.31
0.73
0.67
0.41
0.76
0.58
1.83
Train Tgt
0.78
0.91
0.86
0.8
0.93
0.85
0.8
0.78
0.86
0.78
0.76
0.92
0.83
1.00
DANN
0.44
0.63
0.73
0.58
0.61
0.62
0.54
0.38
0.75
0.67
0.44
0.76
0.6
4.25
DeepCORAL
0.47
0.64
0.75
0.63
0.59
0.65
0.59
0.39
0.76
0.7
0.45
0.78
0.62
3.00
DeepJDOT
0.47
0.65
0.74
0.63
0.65
0.66
0.59
0.44
0.77
0.71
0.47
0.79
0.63
2.33
28

---

Published in Transactions on Machine Learning Research (07/2025)
Table 11: Accuracy score for 20News Groups dataset for each shift compared for all the methods. A white color means the method does not increase the performance compared to Train Src (Train on the source). Green indicates that the performance improved with the DA methods. The darker the color, the more significant the change.
Pank
17.38
1.00
17.67
15.50
17.33
16.50
12.17
16:42
14.17
11.67
7.75
6.50
2.67
6:50
3.00
11.75
3.83
15:50
8:50
5.00
10.00
0.30 1 0.04
0.0
0.58 + 0.04
+ 0.03
0.54 + 0.04
0.6 1 0.05
0.7 1. 0.06
0.59 + 0.04
0.59 + 0.04
0.73 = 0.02
0.76 + 0.02
0.83 1 0.02
0.96 1 0.04
0.82 + 0.04
0.98 # 0.01
0.74 # 0.04
0.85 + 0.02
0.56 1 0.11
0.77 E 0.01
0.86 E 0.02
0.72 = 0.06
Menn
1.0
0.71 0.03
0.03
0.02
0.02
0.03
0.75 0.03
0.03
0.71 + 0.03
0.78 0.02
0.79 + 0.02
0.87 1 001
0.0
0.97 1 0.0
0.83 + 0.07
0.96 + 0.01
0.71 F 0.03
0.87 + 0.01
0.56 1 0.01
0.8 I 0.01
0.83 1 0.04
0.79 # 0.02
talk -sci
0.99
0.71
0.78
0.48
0.71
0.71
0.05
0.0
0.05
0.04
1 0.05
0.14
0.06
0.52 + 0.06
0.78 = 0.01
0.01
0.82 11 0.02
0.99 = 00
= 0.01
0.98 + 0.01
0,84 + 0.02
0.84 + 0.03
0.35 1 0.09
0.84 + 0.01
0.96 + 0.01
0884 = 0.05
talk ---0
0.52 +
0.52
0.57
0.52
0.65
0.52
0.83
0.91
0.7 1 0.01
0.0
0.89 + 0.03
100
000
0.7 T 11001
4 0102
0.7 4774000 1000
0.7 + 000
0.76 - 0.00
0.78 - 0.02
0.87 1 001
001
0.1
+ 0.00
0.70000 000
0.00
NA
0.79 + 0.02
0.05
+ 0.01
sci-stalk
0.99
0.76
0.71
0.96
0.79
0.95
0.7
0.87
0.84
0.78
0.34 T 0.05
0.0
0,05
0.02
1 00
0.06
0.03
0.06
0.01
0.02
1 0.05
1 000
0.72 1 0.01
0.0
0.73 # 0.05
0.85 + 0.01
NA
0.68 I 0.02
0.78 + 0.01
0.1
0.53
7.0
1.00
0.57
0.7
0.50
0.54
0.02
0.68
0.78
0.98
0.99
0.36
0.56 + 0.05
0.0
- 0.00
0.0
0.56 1 0.03
- 0.00
0.57 - 0.06
0.57 + 0.05
0.79 = 0.02
0.84 = 0.00
0.82 1 0.02
033 1 0.1
0.92 + 0.0
0.98 - 0.01
0.72 = 0.06
0.84 + 0.01
0.52 1 0.08
0.84 + 0.01
0.95 1 0.01
0.77 + 0.02
rec ttlk
0.56 -
1.0
0.45
0.60
0.71
0.32 1 0.03
0.0
0.49 + 0.03
0.45 + 0.03
0.7 0.07
0.52 1 0.03
0.65 0.06
0.52 0.06
0.52 + 0.03
0.61 + 0.03
0.67 + 0.02
0.82 + 0.0
0.92 + U.14
0.72 + 0.02
0.99 + 0.0
0.71 + 0.04
0.83 + 0.03
0.59 1 0.29 25
0.67 + 0.02
78 1 0.01
+ 0. 16
1.0 -
0 0.59
OTLahelProp
Ganss. RW
ClassRegoO
Trrain Sre
Train Tit
Dens. RW
Disc. RW
KLIEP
KMM
NN RW
MMD
CORAL
MapOT
EntOT
LinOT
MMD-LS
JPPAA
JOOT
DASVM
TCA
SA
Mladpiin
Oiher
29

---

Published in Transactions on Machine Learning Research (07/202) /225)
Table 12: Accuracy score for AmazonReview dataset for each shift compared for all the methods. A white color means the method does not increase the performance compared to Train Src (Tmii on the souree). Green indicates that the performance improved with the DA mothods The darker the color, the more significant the change.
3:15
1.000
6:33
7-32
19.38
6.30
16:10
10:08
9.00
3.60
14.30
10:14
8:27
18:45
5.91
D7 0.0
0:01
0.02
0.00
0.05
0.00
0:05
0.06
0.00 1.00
0,67 0,02
0.7 0.02
0.11 0.00
= 0.01
= 0.02
06 00.100
0.00
077 1
Steu
0.75
0.2
0.68
0.53
0.60
0.57
0.2
0.00
0.57
bitelumm wewerteeeenn
0.11
I 0:
I 0
0.7 + 0.02
NA
4 01
4 02
8. - 0.02
0.7 0 0.02
# 0.00
9 0.01
# 0.03
0.09 LO0
06T G.
1.71
0:71
0:71
0:38
0.08
0.64
0
0.71
0.0
0.60
0.71
Intcctons weddeeed
0.02
0
0.
0.01
8A
03
0
0
02
0.85 0,02
0.7 = 0.02
0.48 = 0.01
0 041
0.7 = 0
0.04 - 00
0.7 2 0
0.00 =
0.7
0.72
0.7
08
0.84
0.64
Aitolbna -eiwokk
0.71 0.00
I 0:01
0.02
0.7 E 0.02
NA
0.06
1 0.03
4 0,02
$ 0.02
$ 0.11
4 0:01
0.11 + 0.02
0.64 1 0.6
0.02
4
0.72
0.71
0.00
0.00
0.80
0.68
0.71
0.08
0.71
Mirtnoicesssieee brtccten
6:01
- 0.01
= 0.01
= 0.01
- 0:01
0-03 2 8.05
0.73 NA
1.00
0.80 2 0.88
0.74 = 0,01
NA
0.7 = D.11
0.74 = 0.02
0.69 1 00
+ 0.01
0.75 alectronics
0.71
0.75
0.02
0.76
0.74
0.70
0.21 0.11
0.02
2. 001
0.85
NA
0,02
NA
0.03 + 0.13
+ 0.01
0.01
00
NA
Alectroster
0.82 =
0.72
0:71
0.09
0.
0.81
0.88
0.69
0.04
0.89
Aloc Ieatnale- Sonkee
tuna D.01
0:01
1 0.01
0.08
000g
th
0000
1 0.01
0.02
+ 011
+ 0001
0.01
1 0.0
0.72
0.90
0.08
0.09
663
0.41
0.8
666
0.85
0.09
0464
0.00
ded Ritccin
1.74 0.01
B.6 0:01
0:01
0.7 0:01
= 0:15
= 011
R:4 = 0:66
- 0.92
+ 000
6.61 = 0.10
0:00 + 0.0
0.73 = 0.1
0. 1 0.0
07 07200
0 1
680 =
0.72
0.61
0.73
0.72
6.2
0.35
1.72
dert www.eettenn
2 0.05
1 0.02
- 0.02
+ 0.06
02
1.50 000
0.02
0.02
0.01
03 - 0.00
+ 0:1
02 = 0.01
0.52 + 0.0
12
0 0.
0.
03
0.84
RG
0.5 1
041
0.53
0.11
did bol-bbooaa
2 0:01
0-01
+ 0.01
+ 0.08
0:01
0.88
T 0:01
0.72 + 0.01
+ 9:01
+ 0.02
0.01 1 0.0
0.73 1
0.71
22
72
54
72
0.6
04
0.60
0.35
0.00
7B
0.72
bookk Aritechon
U.02
0:01
0.02
0.01
0001
04
01
0.02
00
0.82
0
= 0.03
001
00
=
0.21
0.71
02
09
63
0,72
0.23
0.34
0.00
0.72
0.71
Lonte wwlectteeee centreeiie
5.05 b002
B:Y
0:03
0.11
0.01
0.0g
8.06
0.0
0.04 600
0.59 + 0.02
0.01 + 6.6
0,02
0.52 1 00
h.0
63
0.63
0.58
65
0.
0.69
0.65
Prap
Trato Tram Tet
Dener TWW
Disc: RWV
Cheoo RW
RWW TTuS
MMD-LS
KLIBP KES0
COKAL
BatOT
kino7?
JP PeA
IDOT
S.A.
FSL
30

---

Published in Transactions on Machine Learning Research (07/2025)
Table 13: Accuracy score for Mushrooms dataset for each shift compared for all the methods. A white color means the method does not increase the performance compared to Train Sre (Train on the source). Green indicates that the performance improved with the DA methods. The darker the color, the more significant the change.
enl enl-
tap Cap->enl
Mean
Rank
Train Sre
0.67 0.01
0.77 0.01
0.72 + 0.01
8.50
Train Tgt
1.0 = 0.0
1.0 + 0.0
1.0 0.0
1.00
Dens. RW
0.67 0.01
0.76 + 0.0
0.71 + 0.01
9.00
Reweighting
Disc. RW
0.73 = 0.06
0.78 + 0.01
0.75 + 0.04
4.50
Gauss. RW
0.56 + 0.0
0.46 + 0.0
0.51 0.0
17.50
KLIEP
0.66 = 0.02
0.77 = 0.01
0.72 0.01
10.75
KMM
0.7 0.02
0.78 0.01
0.74 + 0.01
6.00
NN RW
0.67 0.05
0.75 0.01
0.71 = 0.03
12.00
MMDTarS
0.7 0.02
0.77 0.01
0.74 + 0.01
6.00
CORAL
0.66 + 0.02
0.77 +: 0.01
0.72 + 0.02
11.50
Dea space Mapping
MapOT
0.65 = 0.01
0.62 = 0.02
0.63 - 0.01
14.00
EntOT
0.82 0.01
0.67 + 0.01
0.75 + 0.01
8.00
ClassRegOT
0.63 0.01
0.62 1 0.01
0.62 + 0.01
15.50
LinOT
0.72 0.01
0.81 0.01
0.76 + 0.01
4.00
MMD-LS
0.85 0.01
NA
0.85 + 0.01
4.00
JPCA
0.64 0.02
0.78 + 0.02
0.71 + 0.02
10.00
SA
0.53 = 0.02
0.86 + 0.01
0.7 + 0.02
10.00
OTLabelProp
0.68 0.01
0.61 + 0.01
0.64 + 0.01
11.50
31

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 14: Accuracy score for Phishing dataset for cach shift compared for all the methods. A white color means the method does not increase the performance compared to Train Sre (Train the source). Green indicates that the performance improved with the DA methods. The darker the color, the more significant the change.
adress tno ip adress
no ip adress >ip adress
Mean
Rank
ip
Train Sre
0.94 0.01
0.88
+ 0.01
0.91 + 0.01
7.0
Train Tgt
0.97 0.01
0.97 = 0.01
0.97 + 0.01
1.0
Dens. RW
0.94 # 0.01
0.88 + 0.01
0.91 + 0.01
6.5
Disc. RW
0.94 + 0.01
0.88 + 0.01
0.91 + 0.01
9.5
Reweighttnng
Gauss, RW
0.51 + 0.0
0.41 + 0.0
0.46 0.0
0.0
20.0
KLIEP
0.94 + 0.01
0.89 + 0.01
0.91 0.01
5.0
KMM
0.94 = 0.01
0.89 - 0.02
0.91
6.5
NN RW
0.94 0.01
0.89 + 0.01
0.91 + 0.01
6.5
MMDTarS
0.94 + 0.01
0.88 + 0.01
0.91 + 0.01
5.5
CORAL
0.93 + 0.01
0.91 + 0.01
0.92 + 0.01
5.5
Mapping
MapOT
0.83 = 0.01
0.84 + 0.08
0.84 + 0.02
15.0
EntOT
0.87 + 0.04
0.85 + 0.03
0.86 + 0.04
16.0
ClassRegOT
0.87 = 0.02
0.89 - 0.02
0.88 + 0.02
9.0
LinOT
0.91 0.01
0.9 + 0.02
0.91 + 0.02
7.0
MMD-LS
NA
0.88 1 0.01
0.88 + 0.01
11.5
Other Subspace
JPCA
0.92 + 0.01
0.89 + 0.01
0.9 + 0.01
8.0
SA
0.9 + 0.02
0.88 + 0.02
0.89 + 0.02
11.0
TSL
0.88 + 0.02
0.84 + 0.02
0.86 + 0.02
14.0
JDOT
0.8 = 0.02
0.8 - 0.01
0.8 + 0.02
18.0
OTLabelProp
0.86 0.01
0.86 + 0.01
0.86 + 0.01
16.0
DASVM
NA
0.88 + 0.01
0.88 + 0.01
15.0
32

---

Published in Transactions on Machine Learning Research (07/2025)
Table 15: Accuracy score for BCI dataset for each shift compared for all the methods. A white color means the method does not increase the performance compared to Train Sre (Train on the source). Green indicates that the performance improved with the DA methods. The darker the color, the more significant the change.
Rank
8.56
1.44
9.61
6.89
20. .4
8.17
10. 10.80
8.11
2.67
11.11
9.39
12.33
3.78
10. 11
12.78
6.33
19.50
17.00
13.11
10.00
0.55 1 0.06
0.64 - 0.05
0.55 1 0.06
0.56 + 0.06
0.25 + 0.01
0.55 1 0.06
0.33 1 0.06
1 0.06
+ 0.06
0.62 - 0.07
: 0.05
0.54 0.05
0.51 1 0.06
0.07
0.51 + 0.06
0.07
0.07
0.05
0.06
0.5 + 0.06
0.51 +
Monn
0.54
0.55
0.47
0.61
0.59
0.27
0.25
0.46
0.54 1 0.1
0.61 + 0.06
0.53 +: 0.09
0.08
0.25 + 0.01
0.55 1 0.08
0.52 1 0.07
0.51 + 0.08
1 0.08
0.56 + 0.06
0.49 + 0.05
0.52 + 0.06
0.49 1 0.06
0.53 + 0.07
0.08
0.49 + 0.09
0.06
0.29 1 0.1
0.22 1 0.03
0.05
0.5 de 0.09
0.56
0.54
0.49
0.54
0.49
0.89 = 0.08
0.8 = 0.03
0.67 = 0.06
0.08 = 0.03
0. 0.26 + 0.00
1 0.08
0 0.01
0.01
= 0.07
1.07
0.75 0.1
0.08
= 0.0T
0.7 0.05
0.09
0.66 0.12
0.68 + 0.04
0.07
0.24 + 0.09
U 0.62 - 0.05
+ 0004
0.60
0.64
0.68
0.09
0.61
0.74
0.76
0.73
0.31
0.00 0.06
$
0.62 # 0.07
0.75 + 0.03
0.06
0.06
0.25 + 0.00
0.00
0.00
0.63 - 0.00
0.62 + 0.00
+ 0.06
0.53 - 0.04
0
0.04
0.08
0.00
0.1
0.06
0.01
0.00
0.06
0.66
0.63
0.6
0.63
0.64
0 73
0.67
0.62
0.71
0.65
0.60
0.27
0.26
0.53
0.39
0,41 # 0.00
0.47 + 0.08
0.4 0.07
- 0.08
0.25 + 0.00
0.11 1 0.08
0. 0.07
+ 0.05
0.41 - 0.08
0.40 - 0.06
0.29 - 0.05
0.32 + 0.02
- 0.00
0.00
0.34 + 0.1
0.33 + 0.03
0.4 + 0.00 0.06
1 0.07
0.27 T 0.00
0.3 + 0.03
0.33 + 0.06
0.42
0.4
0.41
0.31
0.46
0.26
0.39 T 0000
0.41 + 0.08
0.39 # 0.04
0.4 - 0.00
+ 0.00
1 0.04
0,31 1 0.00
= 0.06
0.39 - 0.04
0.45 - 0.05
= 0.04
- 0.06
1 0.04
- 0.06
0.05
4 0.05
- 0.04
1 0.05
0.07
0.04
1.07
0.25
0.39
0.36
0.32
0.33
0.35
0.46
0.39
0.31
0.39
0.27
0.23
0.28
0:35
0.49 + 0.00
+ 0.00
0.5 + 0.06
0.19 += 0.00
0.25 + 0.00
0.48 1 0.07
0.44 # 0.04
0.47 # 0.09
0.5 S 0.06
0.49 = 0.08
0.39 + 0.03
0.37 + 0.04
0.4 1 0.05
0 + 0.09
0.35 + 1.10
0.47 + 0.07
0.48 17 0.07
0.21 1 0.1
0.26 1 0.03
+ 0.07
0.41 + 0.0
0.52
0.30
0.72 + 0.05
0.82 + 0.03
0.72 + 0.04
0.76 + 0.05
0.25 + 0.01
0.71 1 0.04
0.74 + 0.05
0.69 + 0.03
0.73 + 0.05
0.81 + 0.04
0.04 + 0.05
0.79 E 0,06
0.71 +. 0.07
0.79 + 0.04
0.63 E 0.16
0.72 + 0,06
0.8 + 0.03
0.24 +. 0.07
0.27 1. 0.05
0.61 + 0.05
0.64 + 0.05
+ 0.05
0.61 + 0.07
0.05
0.51 0.06
0.25 + 0.01
0.52 1 0.04
0.45 - 0.08
0.45 = 0.05
0.52 = 0.04
0.56 = 0.06
0.35 - 0.04
0.05
0.07
0.05
0.05
0.58 0.09
0.26 0.03
0.26 0.05
0.4 + 0.06
0.51
0.51
0.45
U.43
0.55
0.46
0.39
U.37
0.59 + 0.05
0.74 + 0.05
0.59 0.06
0.63 0.06
0.28 + 0.07
0.59 + 0.05
0.61 + 0.06
0.58 + 0.06
0.59 + 0.06
0.77 + 0.08
0.61 + 0.08
0.67 + 0.07
0.61 + 0.07
0.76 0.1
0.66 0.1
0.57 0.07
+ 0.09
0.24 0.04
0.63 + 0.06
0.74
0.57
OTLabelProp
Train Sre
Train Tgt
Dens. RW
Disc. RW
GausS. RW
KLEEP
KMM
NN RW
MMDT
CORAL
MapOT
EntOT
ClassRegO11
LinOT
MMD-LS
JPCAA
TCA
TSL
JDOT
SA
33

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 16: Accuracy score for BCI dataset for each shift compared for the deep DA methods. white color means the method does not increase the performance compared to Train Sre (Train on the source). Green indicates that the performance improved with the DA methods. The darker the color, the more significant the change.
Meani
Rank
Train Sre
0.59
0.29
0.67
0.43
0.38
0.29
0.67
0.76
0.67
0.53
2.61
Train Tgt
0.57
0.43
0.71
0.53
0.34
0.29
0.62
0.69
0.79
0.55
2.17
DANN
0.52
0.24
0.55
0.38
0.4
0.22
0.38
0.66
0.64
0.44
4.44
DeepCORAL
0.63
0.26
0.66
0.36
0.48
0.33
0.57
0.72
0.66
0.51
2.72
DeepJDOOT
0.53
0.34
0.64
0.43
0.43
0.43
0.57
0.62
0.6
0.51
3.17
E.2 Impact of the base estimators on the simulated datasets
As mentioned the main paper, possible to partly compensate for the shift by choosing the right base estimator. In this part, we provide the results on the Simulated dataset for three different base estimators: Logistic Regression (LR) in Table 17, SVM in Table 18, and XGBoost, in Table 19. Observing the two first. rows for covariate shift, we see that with LR (Table 17), there is significant drop in performance between training on the source v.s. training on the target (~ 10%), while using SVC (Table 18) only Icads to drop (~~ 3%). Finally, using XGBoost (Table 19) mainlains the performance. The reweighting DA methods help compensate for the shift. when using a simpler LR estimator. However when using an SVC, as shown in the main paper, the reweighting does not help to compensate for the covariate shift. we look at the other shifts, the problem is harder. The subspace methods help with subspace shift, and the mapping methods help with the conditional shift.
These Tables show the importance of choosing the right base estimator. It is clear that choosing appropriate base estimator can partially compensate for some shifts.
34

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 17: Accuracy score for simulated datasets compared for all the methods with LR. A white color means the method does not increase the performance compared to Train Sre (Train on the source). Green indicates that the performance improved with the DA methods. The darker the color, the more significant the change.
Cond. shift
Cov. shift
Tar. shift
Sub. shift
Mean
Rank
Train Sre
0.8 0.02
0.81 1 0.03
0.68 1 0.03
0.06 1 0.01
0.59 1 0.02
10.50
Train Tgt
0.91 0.02
0.92 + 0.01
0.79 0.03
0.97
0.01
0.9 + 0.02
2.00
Dens. RW
0.88 0.03
0.84 +: 0.04
0.66 + 0.03
0.07 1 0.02
0.61 1 0.03
7.50
Disc. RW
0.55 0.02
0.78 1 0.05
0.7 0.04
0.06 0.01
0.52 1 0.03
13.25
Reweighting
Gauss. RW
0.89 0.02
0.85 + 0.03
0.64 + 0.08
0.06 = 0.01
0.61 + 0.02
8.00
KLIEP
0.8 0.02
0.81 + 0.04
0.69 0.03
0.07 0.02
0.59 + 0.03
8.25
KMM
0.84 0.03
0.82 + 0.05
0.66 + 0.04
0.07 = 0.02
0.6 1 0.04
7.88
NN RW
0.81 0.02
0.82 + 0.04
0.67 =
0.01
0.59
0.03
7.75
MMDTTar
0.8 0.02
0.84 + 0.04
0.66 + 0.03
0.07 = 0.02
0.59 + 0.03
10.75
CORAL
0.73 0.05
0.68 1 0.11
0.75
0.02
0.55 1
12.25
Mapping
MapOт
0.73 0.03
0.6 + 0.04
0.79 + 0.03
0.03 + 0.01
0.54 1 0.03
13.75
EntOT
0.72 0.05
0.61 + 0.04
0.79 1
0.01
0.54 1
12.50
ClgssRegOT
0.87 0.08
0.59 + 0.04
0.79 + 0.03
0.03 + 0.01
0.57 0.04
11.50
LinOT
0.77 = 0.03
0.65 + 0.06
0.76 + 0.04
0.04 + 0.02
0.56 + 0.04
12.00
MMD-LS
0.7 = 0.1
0.64 - 0.06
0.78 + 0.04
0.38 = 0.22
0.63 + 0.1
10.75
Other Subspace
JPCA
0.02
0.81 + 0.03
0.68 + 0.03
0.06 0.01
0.59 + 0.02
11.25
SA
0.8 0.02
0.62 + 0.04
0.78 + 0.03
0.04 + 0.02
0.56 + 0.03
11.25
TCA
0.44 = 0.29
0.49 = 0.06
0.54 + 0.11
0.54 = 0.23
0.5 + 0.17
15.50
TSL
0.8 0.02
0.81 - 0.03
0.68 + 0.03
0.06 = 0.01
0.59 - 0.02
11.00
OTLpbelProp
0.73 0.03
0.59 + 0.04
0.79 + 0.08
0.03 = 0.01
0.53 + 0.08
13.50
35

---

Published in Transactions on Machine Learning Research (07/2025)
Table 18: Accuracy score for simulated datasets compared for all the methods with SVC. A white color means the method does not increase the performance compared to Train Sre (Train on the source). Green indicates that the performance improved with the DA methods. The darker the color, the more significant the change.
Cond. shift
Cov. shift
Tar. shift
Sub. shift
Mean
Rank
Train Sre
0.88 + 0.03
0.85 + 0.04
0.66 0.02
0.19 + 0.03
0.65 + 0.03
9.38
Train Tgt
0.92 + 0.02
0.93 + 0.02
0.82 = 0.03
0.98 + 0.01
0.91 + 0.02
1.25
Dens. RW
0.88 + 0.03
0.86 + 0.04
0.66 = 0.02
0.18
0.04
0.64 0.03
8.88
Disc. RW
0.85 + 0.04
0.83 + 0.04
0.72 = 0.04
0.18
0.03
0.64 0.04
10.75
Reweighting
Gauss. RW
0.89 + 0.03
0.86 + 0.04
0.65 = 0.02
0.21 0.04
0.65 0.03
7.00
KLIEP
0.88 + 0.03
0.86 + 0.04
0.66 = 0.02
0.19 0.03
0.65 0.03
8.12
KMM
0.89 = 0.03
0.87 + 0.04
0.64 0.04
0.15 0.05
0.64 = 0.04
9.50
NN RW
0.89 + 0.03
0.86 + 0.04
0.67 = 0.02
0.15 + 0.04
0.64 + 0.03
9.12
MMDTarS
0.88 + 0.03
0.86 + 0.04
0.64 = 0.03
0.2 = 0.04
0.65 + 0.03
9.12
CORAL
0.74 + 0.04
0.7 + 0.11
0.76 = 0.08
0.18
0.04
0.59 0.07
11.50
MapOT
0.72 + 0.04
0.57 + 0.04
0.82 = 0.03
0.02 0.01
0.53 0.03
14.25
Mapping
EntOT
0.71 = 0.04
0.6 + 0.04
0.82 = 0.03
0.12 0.06
0.56 0.05
12.75
ClassRegOT
0.74 + 0.09
0.58 + 0.04
0.81 0.03
0.11 0.06
0.56 + 0.06
12.75
LinOT
0.73 + 0.05
0.73 + 0.08
0.76 0.06
0.18 = 0.04
0.6 + 0.06
11.75
MMD-LS
0.65 + 0.08
0.68 + 0.11
0.79 = 0.05
0.55 = 0.31
0.67 + 0.14
10.75
Other Subspace
JPCA
0.88 + 0.03
0.85 + 0.04
0.66 = 0.02
0.15 + 0.05
0.64 0.04
11.25
SA
0.74 + 0.04
0.68 + 0.04
0.8 = 0.03
0.11 0.03
0.58 0.03
12.50
TCA
0.46 + 0.21
0.48 + 0.09
0.55 0.11
0.56
0.51 0.15
15.62
TSL
0.88 + 0.03
0.85 + 0.04
0.66 0.02
0.19
0.03
0.65 0.03
9.62
OTLabelProp
0.72 + 0.04
0.58 + 0.04
0.81 = 0.04
0.04 0.05
0.54 0.04
14.00
36

---

Published in Transactions on Machine Learning Research (07/2025)
Table 19: Accuracy score for simulated datasets compared for all the methods with XGBoost. white color means the method does not increase the performance compared to Train Sre (Train on the source). Green indicates that the performance improved with the DA methods. The darker the color, the more significant the change.
Cond. shift
Cov. shift
Tar. shift
Suh. shift
Mean
Rank
Train Src
0.89 0.02
0.84 + 0.04
0.66
0.03
0.21 = 0.03
0.65 + 0.03
9.25
Train Tgt
0.89 0.02
0.93 + 0.02
0.77 + 0.03
0.98 1 0.01
0.89 1 0.02
2.25
Dens, RW
0.88
0.03
0.84 = 0.03
0.67 + 0.03
0.22 = 0.04
0.65 + 0.03
8,25
Reweighting
Disc. RW
0.68 = 0.06
0.84 + 0.03
0.66 = 0.03
0.2 = 0.03
0.6 0.04
12.25
Gauss. RW
0.87 0.03
0.84 + 0.03
0.67 0.03
0.22
0.03
0.65 + 0.03
9.12
KLIEP
0.88 0.03
0.84 + 0.03
0.67 + 0.03
0.21 + 0.03
0.65 + 0.03
0.03
7.12
KMM
0.87 0.04
0.84 = 0.04
0.67 = 0.04
0.04
0.65 0.04
7.62
NN RW
0.88 0.03
0.84 + 0.04
0.66 = 0.08
0.2 = 0.03
0.65 1665 0.08 0.08
10.50
MMDTarS
0.88 0.03
0.86 + 0.04
0.63 0.03
0.22 0.03
0.65 0.03
7.50
CORAL
0.71 0.04
0.71 + 0.11
0.74 - 0.08
0.17 = 0.05
0.58 + 0.07
12.75
Mapping
MapOT
0.7 0.04
0.59 + 0.03
0.17 0.05
0.56 + 0.04
13.25
EntOT
0.69 0.05
0.61 1 0.04
0.8 T 0.03
0.2 1 0.02
0.57 1 0.04 0.04
12.25
ClassRegOT
0.82 0.11
0.59 1 0.03
0.03
0.16 0.04
0.59 1 0.05
12.00
LinOT
0.72 0.04
0.68 + 0.06
0.76 1 0.04
0.19 1 0.04
0.59 1 0.05
12.00
MMD-LS
0.64 0.07
0.68 + 0.08
0.78 + 0.04
0.59 0.25
0.67 + 0.11
10.25
Other Subspece
JPCA
0.88 0.03
0.84 1 0.03
0.67 T 0.03
0.14 1 0.05
0.63 1 0.03
10.50
SA
0.72 0.04
0.69 1 0.04
0.78
0.04
0.58 1 0.04
11.75
TCA
0.48 0.05
0.5 - 0.05
0.51 + 0.05
0.51 = 0.06
0.5 + 0.05
15.50
TSL.
0.89 0.02
0.84 + 0.04
0.66 + 0.03
0.21 + 0.03
0.65 + 0.08
9.25
OTLabelProp
0.72 0.05
0.59 + 0.04
0.81 0.04
0.04 0.05
0.54 + 0.04
13.00
E.3 Unrealistic validation with supervised scorer
Table 20 shows the results when we choose the supervised scorer that is when validating on target labels. It is important to highlight that this choice is impossible in real life applications due to the lack of target labels. When using the target labels, the method's parameters are better validated. This can be seen by the significant increase in the table (blue values), which are numerous in this table compared to the one with the selected realistic scorer. For example, the method MMDTarS, which is made for Target shift, compensates all the shift simulated covariate shifts when we select the model with a supervised scorer. When looking at the rank, 11 DA methods have a higher rank than Train Sre compared findings hold for Deep DA where the accuracy Table 21 overall better than when using unsupervisod scorers.
when using realistic scorer. The
37

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 20: Accuracy score for all datasets compared for all the methods for simulated and real-life datasets. In this table, each DA method validated with the supervised scorer. The color indicates the amount of the improvement. A white color means the method is not statistically different from Train Sre (Train on source). Bhue indicates that the performance improved with the DA methods, while red indicates decrease. The darker the color, the more significant the change.
MNIST MNIST/USPS
/USPS
20NewsGroups
Revzon
Cond. shift
OfficeHome
Mushrooms
Cov. shift.
Tar. shift
Sub. shift
Office31
Phishing
BCI
Rank
Train Src
0.88
0.85
0.66
0.19
0.65
0.56
0.54
0.59
0.7
0.72
0.91
0.55
10.66
Train Tgt
0.92
0.93
0.82
0.98
0.89
0.8
0.96
1.0
0.73
1.0
0.97
0.64
1.55
Dens. RW
0.89
0.87
0.67
0.2
0.65
0.56
0.54
0.59
0.7
0.76
0.91
0.55
12.20
Reweighting
Disc. RW
0.86
0.84
0.73
0.23
0.64
0.54
0.54
0.62
0.69
0.78
0.91
0.56
8.75
Gauss. RW
0.89
0.86
0.65
0.21
0.22
0.44
0.11
0.54
0.55
0.51
0.46
0.25
16.45
KLIEP
0.89
0.88
0.66
0.2
0.65
0.56
0.54
0.58
0.7
0.75
0.92
0.55
10.56
KMM
0.89
0.87
0.67
0.19
0.64
0.55
0.53
0.71
0.66
0.75
0.92
0.54
11.74
NN RW
0.89
0.86
0.67
0.15
0.65
0.55
0.55
0.59
0.66
0.72
0.91
0.54
9.15
MMDTarS
0.88
0.93
0.66
0.27
0.65
0.56
0.54
0.59
0.7
0.74
0.91
0.56
10.81
CORAL
0.74
0.84
0.82
0.19
0.66
0.57
0.62
0.75
0.7
0.72
0.92
0.62
5.08
Mapping
MapOT
0.87
0.63
0.82
0.14
0.6
0.51
0.6
0.77
0.68
0.63
0.84
0.47
10.21
EntOT
0.89
0.61
0.82
0.47
0.66
0.58
0.63
0.88
0.68
0.81
0.87
0.53
9.40
ClassRegOT
0.91
0.59
0.82
0.15
NA 0.59
0.66
0.98
0.68
0.89
0.9
0.52
8.25
LinOT
0.89
0.81
0.81
0.19
0.66
0.58
0.65
0.88
0.71
0.81
0.91
0.61
4.06
MMD-LS
0.88
0.85
0.81
0.73
0.65
0.56
0.56
0.98
0.69
0.89
NA
0.58
8.22
Other Subspace
JPCA
0.88
0.85
0.66
0.19
0.65
0.56
0.56
0.84
0.7
0.8
0.9
0.55
8.98
SA
0.74
0.81
0.8
0.13
0.66
0.57
0.56
0.93
0.7
0.91
0.89
0.59
7.80
TCA
0.4
0.46
0.5
0.58
0.04
0.02
0.11
0.49
0.61
0.46
0.49
0.27
17.58
TSL
0.88
0.85
0.66
0.86
0.62
0.48
0.45
0.7
0.69
0.57
0.9
0.26
15.09
JDOT
0.72
0.57
0.82
0.13
0.61
0.41
0.57
0.8
0.67
0.63
0.8
0.46
11.42
OTLabelProp
0.9
0.76
0.81
0.14
0.66
0.56
0.64
0.89
0.67
0.69
0.86
0.51
10.01
DASVM
0.89
0.86
0.64
0.12
NA
NA
NA
0.83
NA
0.76
0.86
NA
7.29
Table 21: Accuracy score compared for the Deep methods with the supervised scorer for a selection of real-life datasets.
MNIST/ /USPS
OfficeHome
Office31
BCI
Rank
Train Sre
0.85
0.77
0.58
0.54
6.19
Train Tgt
0.98
0.83
0.56
2.07
CORAL
0.93
0.82
0.63
0.54
3.29
DAN
0.91
0.79
0.61
0.55
4.76
DANN
0.9
0.76
0.6
0.42
4.98
DeepJDOT
0.93
0.83
0.63
0.54
2.92
MCC
0.94
0.81
0.66
0.55
2.38
MDD
0.91
0.83
0.58
0.42
4.96
SPA
0.92
0.78
0.56
0.4
5.39
38

---

Published in Transactions on Machine Learning Research (07 (07/)025)
Table 22: Fl score for all datasets compared for all the shallow methods for simulated and real-life datasets. The color indicates the amount of the improvement. A white color means the method not statistically different from Train Sre (Train on source). Blue indicates that the score improved with the DA methods, while red indicates a decrease. The darker the color, the more significant the change.
MNIST /USPS
2roups
AnazonRevieg
Selected Scorer
Cov. shift
Tar. shift
Cond. shift
Sub. shift
Office31
OfficeHome
Mushrooms
Phiihiig
BCI
Rank
Train Sre
0.88
0.87
0.62
0.17
0.64
0.56
0.52
0.56
0.65
0.72
0.91
0.53
10.95
Train Tgt
0.92
0.92
0.82
0.98
0.89
0.8
0.96
1.0
0.69
1.0
0.97
0.63
1.70
Dens. RW
0.88
0.88
0.62
0.16
0.61
0.56
0.52
0.55
0.65
0.72
0.91
0.52
IW
12.71
Disc. RW
0.85
0.86
0.7
0.16
0.63
0.53
0.48
0.56
0.63
0.76
0.91
0.54
CircV
8.39
Reweighting
Gauss. RW
0.89
0.88
0.61
0.18
0.15
0.4
0.03
0.43
0.49
0.35
0.29
0.1
CircV
16.53
KLIEP
0.88
0.88
0.62
0.17
0.64
0.55
0.52
0.56
0.65
0.72
0.91
0.52
IW
10.66
KMM
0.89
0.87
0.6
0.15
0.63
0.54
0.51
0.69
0.5
0.74
0.91
0.49
CircV
11.58
NN RW
0.89
0.88
0.63
0.14
0.64
0.55
0.52
0.56
0.64
0.71
0.91
0.5
CireV
8.22
MMDTarS
0.88
0.88
0.6
0.17
0.57
0.56
0.52
0.56
0.65
0.74
0.91
0.53
IW
11.02
CORAL
0.74
0.76
0.74
0.16
0.65
0.57
0.62
0.72
0.65
0.72
0.92
0.62
CircV 5.00
MapOT
0.72
0.65
0.82
0.02
0.59
0.5
0.61
0.76
0.59
0.63
0.84
0.47
PE 10.49
Mapping
EntOT
0.71
0.67
0.82
0.12
0.63
0.57
0.59
0.83
0.49
0.75
0.85
0.53
CircV
10.15
ClassRegOT
0.74
0.66
0.81
0.11
NA
0.53
0.62
0.97
0.67
0.82
0.89
0.52
IW
6.49
LinOT
0.73
0.78
0.75
0.16
0.65
0.57
0.64
0.81
0.65
0.76
0.91
0.61
CircV
4.20
MMD-LS
0.77
0.77
0.75
0.54
0.64
0.56
0.54
0.97
0.6
0.85
NA
0.48
MixVal
7.58
Other Subspace
JPCA
0.88
0.87
0.62
0.14
0.61
0.47
0.5
0.76
0.61
0.78
0.9
0.51
PE
9.55
SA
0.73
0.74
0.8
0.1
0.64
0.57
0.55
0.88
0.56
0.77
0.89
0.52
CircV
7.95
TCA
0.51
0.56
0.5
0.61
0.0
0.0
0.02
0.53
0.46
0.44
0.47
0.19
DEV
17.94
TSL
0.88
0.87
0.63
0.17
0.63
0.47
0.45
0.59
0.58
0.28
0.9
0.21
PE
15.46
JDOT
0.72
0.66
0.82
0.13
0.59
0.41
0.59
0.8
0.61
0.65
0.79
0.46
IW
10.74
OTLabelProp
0.72
0.67
0.8
0.07
0.65
0.54
0.62
0.86
0.58
0.64
0.86
0.49
CircV
10.80
DASVM
0.89
0.88
0.61
0.13
NA
NA
NA
0.87
NA
0.82
0.85
NA
MixVal
7.12
E.4 F1-score of benchmark
We provide in Table 22 a version of Table 2 where the performance measure reported the F1-score. is interesting to note that the dynamic of which methods work best and are the more robust very similar to the accuracy performance which illustrate the robustness of the benchmark.
E.5
Comparison of the rank of DA scorer
To provide a more detailed assessment of the scorers' performance, we present a critical difference diagram of their rankings in Figure 4. The diagram highlights that the unrealistic supervisod scorer significantly outperforms all others. Among the unsupervised scorers, CireV and IW achieve the best performance, with their rankings being very close and not statistically different according to statistical test. Next, we observe group comprising PE, DEV, and MixVal, where DEV and MixVal are also not statistically distinguishable. Finally, SND emerges as the worst-performing scorer in the benchmark.
To give a more detailed perspective, we present a visualization in Figure 5, showing the rank of each scorer for each DA method. In the right part of the figure, the supervisod scorer (in pink) is consistently the top-ranked, as expected, across all methods. Similarly, CircV (in rod) and IW (in orange) consistently outperform other scorers.
39

---

Published in Transactions on Machine Learning Research (07 (07/2025)
2.5
3.0
3.5
4.0
4.5
5.0
5.5
Supervised (2.6)
(5.4) SND
IW (3.4)
(4.5) MixVal
CircV (3.6)
(4.3) DEV
PE (4.2)
Figure 4: Critical difference diagram of average ranks for scorers, computed across shallow methods and shifts (lower ranks indicate better performance). Black lines between scorers indicate pairs that are not statistically different based on the Wilcoxon test.
DASVM Dens.
Disc. RIV
Dens, RW
DASVM
Disc.
RIV
OTLabe|Prop
Office31
OTLabelProp
CircV
JDOT
J.A
KLIEP
Mushrooms
20NewsGroups
JDO1
KUEP
Phishing
MiXAl
MMWM
MNIST/UPPS
AmazonReview
1SL
SND
Supervised
TCA
NN RUY
TCA
3A
5A
MMMDTar
JPCA
CORAL
MapaT
MMD-L5
MapOT
LinT
EntOT
LinOT
EntO
ClessRegOT
ClassRogCT
Figure 5: Illustrations as spider plots for all methods of the accuracy on each dataset (left) and the scorers rankings (right). For methods with no accuracy results (NA in Table 2) we replace the value by 0. We provide both spider plot in the same Figure to allow comparison of the scorer ranking while having the possibility to check the performance for each method.
E.6 Comparisons between supervised and unsupervised scorers
Impact the cross-validation score. We observe in Figure the cross-validation score function of the final accuracy for various DA methods type and for both supervised and unsupervised scorers. As expected, observe good correlation between accuracy and cross-validation score with the supervised scorer. An important remark is that the Circular Validation (CireV) (Bruzzone & Marconcini, 2010b) shows some correlation between accuracy and cross- validation score. indicates that this unsupervised scorer might be the most suitable choice for hyperparameter selection. Using spearman correlation shows the same conclusion (see Figure 6). This is supported by our extended experimental results in Table for which the CireV selected the best scorer the most often. A similar trend can be observed for the Importance Weighted (IW) (Sugiyama et al., 2007a) which is also confirmed in Table 2.
40

---

Published in Transactions on Machine Learning Research (07 2025)
Supervised p
IW p 0.32
SND p= -0.09
DEV p= -0.07
1.0
1.0
0.0
0.8
0.8
0.5
Cres-validation
0.6
1.0
0.4
Mapping other
0.2
0.2
2-
Reweightt
0.0
2.0
PE -0,00
CircV p 0.63
MixVal p 0,12
1.0
0.1
0.8
0.6
0.4
0.3
0.4
0.0
0.00
0.25 0.50
0.75 1.00
0.00
0.25 0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Target accuracyy
Target accuracy
Figure 6: Cross-val score as a function of the accuracy for different supervised and unsupervised scorers. The Spearman correlation coefficient reported for each scorer by p. Each point represents an inner split with a DA method (color of the points) and a dataset. good score should correlate with the target accuracy.
Supervised
CircV
Cross (estonn scone Cross vaa-valdaron Scoee Cross Cros-valliaee sc0re
1.00
1.00
0.0
0.0
Roneighona
0.75
0.75
-0.5
0.8
0.50
1.0
0.25
1.5
0.5
-0.3
0.00
-20
1.00
1.00
0.0
1.00
Mapping
0.75
0.75
-0.5
-01
0.75
0.50
0.50
-1.0
0.50
-02
0.25
-15-
0.25
0.7
0.00
2.0
-03
0.00
1.00
Subspass
0.75
-0.
0.50
0.50
-02
0.50
0.25
0.25
-1.5
0.25
0.00
0.00
Cross voossvaiidaton on score
0.8
-0.5
-0.1
Other
0.4
-03
0.4
0.2
accuracy
accuracy
accuracy
Figure 7: Cross-val score as function of the accuracy for various DA methods and different supervised and unsupervised scorers. Each point represents inner split with DA method (color of the points) and dataset. A good scorer should have score that correlates with the target accuracy.
41

---

Published in Transactions on Machine Learning Research (07/202) 2025)
0.10
AACC w.r.t. supervised
0.05
Dens: RW
Disc. RW
Gauss. RW
KUIEP
KMM
NN RW
MMDTarS
CORAL
MapOT
EntOT
GassRegoT
LinoT
MMD-L
IPCA
TCA
TSL
IDOT
OTLabeiProp
Figure 8: Change of accuracy of the DA methods with the best realistic unsupervised scorer (Table 2) w.r.l. the supervised scorer.
42

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Supervised scorer v.s. the best realistic unsupervised scorer. We plot the loss in performance of the DA methods with the best realistic unsupervised scorer compared to using the supervised scorers in Figure 8.
Supervised scorer v.8. realistic unsupervised scorers. We present a scatter plot in Figure 9 and Figure 10, the accuracy different DA methods using both supervised scorer and unsupervised scorer. this figure, points below the diagonal indicate decrease in performance when using the unsupervised scorer compared to the supervised one. The colors represent different. types of DA methods. We SND, DEV and PE scorers all lead to large performance loss compared to the supervised scorer. While IW and CircV results are much more concentrated near the diagonal, indicating small loss in performance. This concentration explains why these two scorers have been selected as the best scorers for most of the methods in Table
that the
Unsupervised scorer
IW
SND
DEV
PE
CircV
MixVal
1.00
Mapping
Dther Reweighting
0.50
Subspace
0.25
0.00
0.0
1.00,0
0.5
1.00.0
0.5
1.00.0
0.5
1.00.0
0.5
1.0000
Supervised Supervised Supervised
Supervised scorer Supervised Supervised
Figure 9: Accuracy of the DA methods using unsupervised scorers as a function of the accuracy with the supervised scorer. Colors represent the type of DA methods.
43

---

Published in Transactions on Machine Learning Research (07/2025)
IW
SND
DEV
PE
CircV
MixVal
Unsupervised vvervised Scarer Unsupervised Scorer
Reweighting
0.75
0.50
0.25
0.00
1.00
0.75
Mapping
0.50
0.00
1.00
Subspace
0.75
Unsupervised
0.50
0.25
0.00
Unsupervised Scorer
1.00
23h
Ansasrlestev
Bcl
Other
0.50
Offinal
Mosling
0.25
0.00
0.0
0.5
1.0 0.0
0.5
1.0
0.5
1.0 0.0
0.5
1.0
0.5
1.0 0.0
0.5
1.0
Supervised Scorer Supervised Scorer
Supervised Scorer Supervised Scorer Supervised Scorer Supervised Scorer
Figure 10: Accuracy of the DA methods using unsupervised scorers as function of the accuracy with the supervised scorer for the different lypes of DA methods. Points below the diagonal represent decrease in performance when using the unsupervised scorer compared to the supervised one. Colors represent the dataset which the DA method is applied.
E.7 Computational efficiency of the DA methods
Figure 11 shows the average computation time for training and testing each method. These results are based on one outer split, while we ran the benchmarks for five outer splits. Each method has a different time complexity. Interestingly, more time-consuming methods are not necessarily more performant than others. For instance, the highest-ranked methods LinOT, CORAL, and SA-alse have some of the lowest training and testing times. It's also worth noting that during the experiments, enforced 4-hour timeout. Thus, the more time-intensive methods might have been even slower without this timeout.
44

---

Published in Transactions on Machine Learning Research (07/2025)
10a
Mean computationa time (in sec)
NO DA
Reweighting
Mapping
Subspace
Other
10a
101
Train Tgt
Train Src
NN RW
Gauss. RW
KLIEP
Disc. RW
Dens, RW
MMDTarS
KMM
CORAL
LinOT
MapOT
EntOT
ClassRegOT
MMD-LS
JPCA
SA
TSL
OTLabelProp
TCA
JDOT
DASVM
Figure 11: Mean computing time train and test each method for every experiment outer split.
E.8 Hyperparameters grid search for the DA methods and neural networks training
In this section, first report the grids of hyperparameters used in our grid search for each DA method.
We also detail the configuration and hyperparameter grids for training neural networks in our Deep DA benchmark. We provide an overview of the key settings for each dataset, including batch sizes, optimizer parameters, learning rates, and the number of training epochs. Additionally, we outline the hyperparameter grids used for grid search across the Deep DA methods.
45

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 23: Hyperparameter grids used the grid search for each DA method. The hyperparameter grids were designed be minimal yet expressive, allowing cach method to perform optimally. We selected parameters based on what seemed most reasonable, according to our best knowledge.
Method
Hyperparameter Grid
KLIEP
'ev': [6),
'gamma': [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0, 'auto", 'scale").
'max mmax_iter' [1000].
'n_centers's [100].
'random state': [0],
'tol': [1e-06]
KMM
[1000.0]
'gamma': [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0, None),
max iterr: [1000].
'smooth weights": [False].
'tol': [1e-06]
NN RW
'laplace smoothing': (Tru, False
MаpOT
iter's [1000000]
'metric':
['sgeuclidean, "cosine", 'cityblock].. 'norm': ['medion"]
JPCA
'n_ components': [1, 2, 5, 10, 20, 50, 100)
SA
'n components:: [1, 2, 5, 10, 20, 50, 100)
TCA
'kerne':: ['rbf]]
'mu': [10, 100).
n_coomponents': [1. 2, 5, 10, 50, 100]
CORAL
asssume centered': [False, True],
'rog': ['auto']
MMDTar5
gamma': [0.0001, 0.001, 0.01, 0.1, 10, 10.0, 100.0, 1000.0, None].
max iterr: [1000],
'reg': [1e-06].
'tol': [1e-06]
ClassRegOT
'max inner iter': [1000].
'max__ter': [10],
'metric': ['sgeuclidean', 'cosine", 'cityblock'].
'norm': ['Ipl1'].
'tol': [1e-06],
(rre cl. reg e)%: [(.11,, [0.1]), ((0.5), (0.5].. ((L.0) [1.0]]
Dens. RW
bandwidth': [0.01, 0.1, 1.0, 10.0, 100.0, 'soott, sillverman'
Disc. RW
doomain classifier': [LR', 'SVC', 'XGB"
Gauss, RW
'reg': [aauto'
DASVM
'max iter'ss [200]
JDOT
'alpha:: [0.1, 0.3, 0.5, 0.7, 0.9),
literr max': [100],
'thr weights': [1e-07).
'tol's [1e-06]
EntOT
"max__eee [1000],
'metric': ['sqenclidean', 'cosine", 'cityblock)).
'norm": ['median'].
'reg e:: [0.1, 0.5, 1.0),
'tol': [1o-06]
LioOT
"blas': [Tre,, False],
'rog': [le-08, 1e-06, 0.1, 10]
TSL.
"base Tlase_method? ['fdda]],
'leugth scale': [2],
'max iter': [300].
'mu': [0.1, 1, 10).
'n_ n__components': [1, 2, 10, 20, 50, 100),
'reg': [0.0001],
'tol': [0.0001]
MMD-LS
"gamma': [0.01, 0.1, 1, 100).
mmx__iters [20].
'rog_k': [1e-08]
'reg. m': [1e-08].
'tol': [1e-05]
OTLahelProp
metticc:: ['sqeuelidean", 'cosine', 'cityblack',
(nn_iter -tte__aax,, max, reg)': [[[10000), [None]), ([100), (0.1, 1]]
46

---

Published in Transactions on Machine Learning Research (07 (07/2023) /025)
Table 24: Configuration of Deep learning models for each datasct. This includes recommended Batch sizes, optimizer settings, learning rates, and maximum epochs.
Dataset
Configuration
mnist_ _
Neural net: 2-layer CNN
Batch size:
Optimizer:
SGD,
momentum=0.6,
Learning rate: 0.1
Epochs: 20
Learning rate scheduler: LRSched- uler(Stepl.R, size=10 gamma=0.2)
office31
Neural net: ResNet50
Batch size: 128
Optimizer:
SGD,
momentum=0.2
neight_decay--t
Learning rate: 0.5
Epochs: 30
Learning rate
scheduler:
StepLR,
step__siee size=10, gamma=0.2
officehome
Neural net: ResNet50
Batch size:
Optimizer:
SGD,
momentum=0.6,
1eccy=__ee
Learning rate: 0.05
Epochs: 20
Learning rate
scheduler:
StepLR,
step size=10, gamma=0.2
bei
Neural net: FBCSPNet.
Batch size: 64
Optimizer: AdamW
Learning rate: 0.000625
Epochs: 200
Learning rate scheduler: ConeneAnneal- ingLR
47

---

Published in Transactions on Machine Learning Research (07/2025)
Table 25: Hyperparameter grids used in the grid search for each Deep DA method. The hyperparameter grids were designed to be minimal yet expressive, allowing cach method to perform optimally. We selected parameters based on what seemed most reasonable, according our best knowledge.
Method
Hyperparameter Grid
DANN
'reg': [0.001, 0.01, 0.1, 1.0]
DeepCORAL
'reg': (0.00001, 0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0
DeepJDOT
'reg cl': [0.0001, 0.001, 0.01
'rog dist": [0.0001, 0.001, 0.01]
DAN
'reg': [0.00001, 0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0
MCC
'reg': [0.01, 0.1, 1]
'temperature': [1, 2,
MDD
'reg': [0.001, 0.01, 0.1
'gamma': [1, 3
SPA
'rog': [0.001, 0.01, 0.1, 1]
'rog nap': [0, 1]
48


Reviews:
[{"review_id": "pSaSmx6rwk", "reviewer": "Reviewer_jw5k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The study benchmarks DA methods across multiple modalities (CV, NLP, tabular, biomedical), ensuring broad applicability.\n\n2. DA-Bench incorporates unsupervised model selection, making results more practical for real-world applications.\n\n3. The paper also provides practical takeaways on which DA methods perform best under different types of shifts.\n\nWeakness:\n\n1. Although the benchmark evaluates 20 shallow DA models, its scope for deep DA models is very limited. The study evaluates only three deep DA methods, which might not capture the full spectrum of state-of-the-art deep DA approaches. Recent works have progressed the state-of-the-art significantly and is needed to be included in the new benchmark. One such example is compared to DANN (2016), AnyDA [1] has shown to be significantly superior. Other examples of new models can be [2] and [3]. Inclusion of such new works is vital in the benchmark for guiding the community towards optimal model selection.\n\n2. The Figure 1 of the paper needs more clarity and proper explanation. It is difficult to understand the different types of shifts. The authors should add quantitative explanations and discuss more thoroughly what the plots try to convey. The coloring scheme of the graph also needs to be clarified.\n\n3. In Table 2 and Figure 3, the study evaluates several scorers, but most of them show low correlation with actual model performance and have high variance, leading to unreliable hyperparameter selection. This scorer performance variability is a major limitation of the paper because it suggests that DA methods, even when well-designed, may fail in practical deployment due to unreliable validation techniques. The study does a great job of exposing this issue but does not provide a definitive solution. Even considering the \"Take-away for DA users\" in Section 4.1, the authors should support their discussions with quantitative values and highlight the different empirical results which lead to such inferences.\n\n4. The paper lacks in providing much theoretical analysis regarding why certain DA methods perform better in different shift scenarios. Given that the study identifies major limitations in existing scorers (e.g., poor correlation with true model performance, high variance), it would have been valuable to suggest a new scoring mechanism or perhaps a novel DA scheme backed by theoretical justification to handle the limitations.\n\n[1] Chakraborty, O., Sahoo, A., Panda, R. and Das, A., 2023. AnyDA: Anytime domain adaptation. In The Eleventh International Conference on Learning Representations.\n\n[2] SDAT Rangwani, H.; Aithal, S. K.; Mishra, M.; Jain, A.; and Radhakrishnan, V. B. 2022. A closer look at smoothness in domain adversarial training. In International Conference on Machine Learning, 18378–18399. PMLR.\n\n[3] Zhu, D.; Li, Y.; Shao, Y.; Hao, J.; Wu, F.; Kuang, K.; Xiao, J.; and Wu, C. 2023a. Generalized Universal Domain Adaptation with Generative Flow Networks. arXiv preprint arXiv:2305.04466.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces DA-Bench, a benchmarking framework aimed at evaluating unsupervised domain adaptation (DA) methods across different data modalities, including computer vision, natural language processing, biomedical, and tabular data. The study addresses the challenge of realistic model selection in DA settings, where hyperparameter tuning is difficult due to the absence of labeled target data. This open-source and extensible work uses nested cross-validation and unsupervised scorers to fairly compare methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please go through the Weakness section to find the detail suggestions and changes. In brief, the suggested changes are as follows:\n\n1. The benchmark should include recent state-of-the-art approaches towards optimal model selection.\n2. Modify Figure 1 to enhance clarity and add more explanations.\n3. The inferences drawn in Section 4.1 need more quantitative analysis mapping their discussions to the empirical results obtained.\n4. The paper should include more theoretical analysis and novel ideas towards overcoming the challenges portrayed in the benchmark.", "broader_impact_concerns": "Nil.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "S3ZSwVcjki", "reviewer": "Reviewer_a9qk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper evaluates unsupervised domain adaptation methods across diverse modalities (computer vision, NLP, tabular, biomedical, assessing 20 shallow methods across different categories alongside 3 deep DA methods.\n\nThe use of nested cross-validation with unsupervised scorers (e.g., IW, CircV) to tune hyperparameters reinforce that DA methods are evaluated fairly in realistic unsupervised settings. It mirrors real-world constraints where target labels are absent, providing a practical benchmark.\n\nThe benchopt framework makes it reproducible allowing other researchers to extend it with new DA methods and datasets.\n\nWeaknesses:\nWhile the paper evaluates three deep DA methods, it does not cover more recent or advanced deep DA techniques (e.g., contrastive learning-based DA, transformer-based DA, or self-supervised DA approaches). This feels narrow compared to the 20 shallow methods across all datasets, potentially underrepresenting deep DA’s capabilities. \n\nThe timeout (4 hours) for nested loops and single outer split for deep methods might skew results, favoring faster methods or missing optimal deep DA performance. The authors could discuss this trade-off’s impact more explicitly.\n\nThe nested cross-validation strategy can be computationally expensive, requiring significant resources. Providing guidance on optimizing computation efficiency or developing faster heuristics for model selection could improve usability", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces DA-Bench, a comprehensive and reproducible benchmark for evaluating unsupervised domain adaptation methods, addressing gap in assessment across diverse data modalities. The authors evaluate 20 shallow DA methods (reweighting, mapping, subspace alignment, and others) and 3 deep DA methods. They extend beyond the vision-focused benchmarks, offering new insights into DA performance across modalities.\n\nThey implement hyperparameter tuning approach, using nested cross-validation with five unsupervised scorers (e.g., Importance Weighted, Circular Validation) to mimic real-world conditions where target labels are unavailable. The benchmark reveals that unsupervised scorers like IW and CircV correlate better and are most effective than others.\n\nThe results demonstrate that simple linear methods (e.g., CORAL, LinOT) consistently outperform across multiple datasets offering modest gains, while reweighting suits specific shifts and mapping excels with moderate class counts. Deep DA methods perform well in vision tasks, but shallow methods remain competitive in tabular and biomedical data. By open-sourcing the framework, the authors enable future extensions, making DA-Bench a valuable tool for advancing DA research and application.\n\nThe paper claims to compare UDA methods under realistic conditions, addressing critical issues in model selection, performance validation, and benchmarking disclosing the restrictions of current validation strategies, and offering actionable insights for practitioners.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While the paper evaluates three deep DA methods, it does not cover more recent or advanced deep DA techniques (e.g., contrastive learning-based DA, transformer-based DA, or self-supervised DA approaches). This feels narrow compared to the 20 shallow methods across all datasets, potentially underrepresenting deep DA’s capabilities. Expand the Evaluation by adding more state-of-the-art deep DA methods, especially Transformer-based DA models, Contrastive learning-based DA approaches and Self-supervised DA techniques. Also try to test them on across modalities to strengthen the paper’s claim of comprehensive benchmarking.\n\nThe timeout for nested loops and single outer split for deep methods shows biased results. Add a discussion to show how these affect performance. Provide computational cost analysis for different DA methods and discuss potential optimizations or alternatives for large-scale settings.\n\nThe paper acknowledges that DA methods are highly sensitive to hyperparameters, but does not provide practical solutions. Discuss possible automated hyperparameter tuning approaches and provide best practices for tuning DA methods in real-world scenarios.\n\nThe benchmark currently focuses only on classification tasks, which limits its applicability to other areas. Include an evaluation of DA methods for regression tasks, especially for applications like biomedical predictions and financial forecasting.\n\nWhile DA-Bench covers multiple data modalities, the selection is still limited. Add more diverse datasets, such as satellite imagery, time-series financial data, or speech/audio data.\n\nWhile pre-processing is mentioned, details are referred to an appendix. More in-text explanation of how pre-processing affects results could improve transparency and interpretation.", "broader_impact_concerns": "Can the authors add a broader impact statement section", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wZyF4CBpt9", "reviewer": "Reviewer_N4gg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The focus on rigorous evaluation with proper model selection respecting the unsupervised DA setting is a key strength of this work. \n- A broad range of shallow DA algorithms is evaluated across a range of domains, in combination with 6 different scorers, which is a significant effort.\n- The comparison of scorers is in itself useful, though somewhat tangential to the main focus of the work.\n- Code for the benchmarking framework will be released upon publication, which will allow others to build on the work.\n\nWeaknesses;\n- It is unclear whether the comparisons of DA algorithms are practically relevant, as there are domain-specific DA methods developed for images and text (e.g. the deep DA methods compared in the paper), and moreover many of the methods being compared are not that recent. The most recent shallow DA method is from 2020 and most recent Deep DA method is from 2018. \n- Datasets used are also slighly dated. For example, in the image domain, while Office-31 and Office-Home are still widely used for evaluation, the field is moving on to more complex datasets like VisDA, DomainNet and WILDS.\n- Recent papers proposing new model selection strategies also include comparisons of various scorers, though using deep DA methods. \n\nOverall, the paper is to be commended for its rigorous approach and systematic evaluation. However, the methods compared are rather old so the practical relevance of the work is somewhat diminished. In addition, a key message that unsupervised model selection should be used for realistic evaluation purposes has already been communicated in several previous works, as indicated in the paper. To be more relevant the paper should focus on settings/datasets where shallow DA methods are state-of-the-art, deep DA methods, or a specific domain, and perform a deeper analysis. That said, the comparison of scorers is interesting. By TMLR's criteria, the evidence provided does support the main claims of providing a benchmarking platform and evaluation of a range of methods on several datasets, and the work may be interesting to some in the community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper describes a benchmark for domain adaptation (DA) algorithms that utilizes classification datasets from 4 different domains (image, text, time-series and tabular data). It focuses on a fair evaluation of algorithms with strict model selection protocols ensuring that target domain data is not utilized. Algorithms are tuned using a range of unsupervised model selection strategies (scorers), and the performance of scorers is also compared. While the paper focuses primarily on shallow DA algorithms, evaluating 20 such algorithms, it also includes 3 deep DA algorithms in a separate evaluation on a subset of the datasets. No single one DA algorithm was found to outperform others across the range of datasets and the paper attempts to provide guidance for choosing a DA algorithm and scorer in practical situations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Suggestions:\n- For the correlation metric used for scorers (e.g. Fig 3), Spearman seems more suitable as model selection depends only on the ranking of scores.\n- In Sec 4.2 \"Of the five methods under consideration\" -> should it be six methods?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)