Addressing Parameter Choice Issues in Unsupervised Domain Adaptation by Aggregation
Authors: Marius-Constantin Dinu
Markus Holzleitner
Maximilian Beck
Hoan Duc Nguyen
Andrea Huber
Hamid Eghbal-zadeh
Bernhard A. Moser
Sergei Pereverzyev
Sepp Hochreiter
Werner Zellinger
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Domain adaptation
parameter choice
model selection
aggregation
importance weighting

Abstract: We study the problem of choosing algorithm hyper-parameters in unsupervised domain adaptation, i.e., with labeled data in a source domain and unlabeled data in a target domain, drawn from a different input distribution. We follow the strategy to compute several models using different hyper-parameters, and, to subsequently compute a linear aggregation of the models. While several heuristics exist that follow this strategy, methods are still missing that rely on thorough theories for bounding the target error. In this turn, we propose a method that extends weighted least squares to vector-valued functions, e.g., deep neural networks. We show that the target error of the proposed algorithm is asymptotically not worse than twice the error of the unknown optimal aggregation. We also perform a large scale empirical comparative study on several datasets, including text, images, electroencephalogram, body sensor signals and signals from mobile phones. Our method outperforms deep embedded validation (DEV) and importance weighted validation (IWV) on all datasets, setting a new state-of-the-art performance for solving parameter choice issues in unsupervised domain adaptation with theoretical error guarantees. We further study several competitive heuristics, all outperforming IWV and DEV on at least five datasets. However, our method outperforms each heuristic on at least five of seven datasets.

Published as a conference paper at ICLR 2023
ADDRESSING PARAMETER CHOICE ISSUES IN UNSU- PERVISED DOMAIN ADAPTATION BY AGGREGATION
Marius-Constantis Dinu
Markus Holzleitner'
Maximilian Beck'
Duc Hoan Nguyen
Andrea Huber!
Hamid Eghbal-zadeh" Bernhard A. Moser
Sergei V. Pereverzyev
Sepp Hochreiter
Werner Zellinger
EELLIS Unit Linz and LIT Lab, Institute for Machine Learning, Johannes Kepler University Linz
Dynatrace Research
Instittte of Advanced Research in Artificial Intelligence
"Software Competence Center Hagenberg
Johann Radon Institute for Computational and Applied Mathematics, Austrian Academy of Sciences
ABSTRACT
We study the problem of choosing algorithm srper-parameters in unsupervised domain adaptation, i.e. with labeled data in a source domain and unlabeled data target domain, drawn from different input distribution. We follow the strategy to compute several models using different hyper- parameters, and, to subsequently compute a linear aggregation of the models. While several heuristics exist that follow this strategy, methods are still missing that rely on thorough theories for bounding the target error. In this turn, we propose a method that extends weighted least squares to vector-valued functions, e.g., deep neural networks. We show that the target error of the proposed algorithm is asymptotically not worse than twice the error of the unknown optimal aggregation. We also perform large scale empirical comparative study on several datasets, including text, images, dlectroenceehaloggra. body sensor signals and signals from mobile phones. Our method outperforms deep embedded validation (DEV) and importance weighted validation (IWV) all datasets, setting a state-of-the the-ttt-rrt performance for solving parameter choice issues in unsupervised domain adaptation with theoretical error guarantees. We further study several competitive heuristics, all outperforming IWV and DEV on at least five datasets. However, our method outperforms each heuristic on at least five of seven datasets.
INTRODUCTION
The goal of unsupervised domain adaptation is to learn model on unlabeled data from a target input distribution using labeled data from different source distribution (Pan & Yang 2010 Ben-David et al. 2010). If this goal is achieved, medical diagnostic systems can successfully be trained on unlabeled images using labeled images with a different modality (Varsavsky et al. 2020 Zou et al.)
2020); segmentation models for natural images can be learned using only labeled data from computer simulations Peng et al. (2018); natural language models can learned from unlabeled biomedical abstracts by means of labeled data from financial journals (Blitzer et al. (0066;; industrial quality inspection systems can be learned on unlabeled data from new products using data from related
products (Jiao al. 2019 Zellinger et al. 2020).
However, missing target labels combined with distribution shift makes parameter choice a hard problem (Sugiyama et al. 2007] You al.] 2019 Saito et al. [021 Zellinger et al., [021] Musgrave ct al. 2021). Often, one ends up with a sequence models, e.g., originating from different hyper-
parameter configurations (Ben-David al. 2007 Saenko al. 2010 Ganin al. 2016 Long et al.)
I Large scale benchmark experiments available https:: hhtp:://////.bb....////p wtt.:/..oooppptff.ce
dinu @ @l. jku.at wemerzzelingerrrncc...eea www.e/liiiineerrcc.......m..

---

Published as a conference paper at ICLR 2023
2.00
1.5
1.75
tar
target
given models regression function optimal aggregation our approach
1.50
1.25
0.5
1.00
0.75
0.50
-0.5
0.25
0.00
0.0
0.0
2.0
Figure 1: Unsupervised domain adaptation problem (Shimodaira] 2000 Sugiyama et al. 2007 You
al. 2019). Left: Source distribution (solid) and target distribution (dashed). Right: A sequence of different linear models (dashed) is used to find the optimal linear aggregation of the models (solid). Model selection methods (Sugiyama et al., 2007 Kouw et al.] 2019) You et al., 2019 Zellinger et al.) 2021 cannot outperform the best single model in the sequence, confidence values as used in Zou ct al. 2018 are not available, and, approaches based on averages or tendencies of majorities of models (Saito et al.] 2017) suffer from high fraction of baggeerrrormodel in the sequence. In contrast, our approach (dotted-dashed) is nearly optimal.
addition, the model computed by our method provably approaches the optimal linear aggregation for increasing sample size. For further details on this example we refer to Section C in the Supplementary Material.
2015 Zellinger et al. 2017 Peng et al. 2019). In this work, we study the problem of constructing an optimal aggregation using all models in such a sequence. Our main motivation is that the error of such optimal aggregation is clearly smaller than the error of the best single model in the sequence.
Although methods with mathematical error guarantees have been proposed to select the best model in the sequence (Sugiyama al. 2007; Kouw et al. 2019 You et al. 2019 Zellinger et al. 102II. methods for learning aggregations of the models are either heuristics or their theory guarantees are limited by severe assumptions (cf., Wilson & Cook (2020)) Typical aggregation approaches are (a) to learn an aggregation on source data only (Nozza el al. 2016). (b) to learn an aggregation on a set
of (unknown) labeled target examples (Xia al. 2013 Dai al. 2007 M & Marcu 2006 Duan
et al.
2012). (c) to learn an aggregation on target examples (pseudo- labeled based on confidence
measures of the given models (Zhou et al., 202 [ Ahmed et al., 2022] Sun 2012) Zou al., 2018
2017). (d) to aggregate the models based on data-structure specific transtormations (Yang al. 2012 2, Ha & Youn 2021). and, (e) use specific (possibly not available) knowledge about the given models, such as information obtained at different time-steps of its gradient-based optimization
Saito et al.
process (French et al. 2018 Laine & Aila 2017 Tarvainen & Valpola 2017 Athiwaratkun et al. 2019:A1-Stouhh & Reddy] (201)) or the information that the given models are trained on different (source) distributions (Hoffman et al. 2018 Rakshit et al. 2019 Xu et al., 2018 Kang et al., 2020 Zhang et al. 2015). One problem shared among all methods mentioned above that they cannot guarantee a small error, even if the sample size grows to infinity. See Figure for a simple illustrative example.
In this work, we propose (to the best of our knowledge) the first algorithm for computing aggregations of vector-valued models for unsupervised domain adaptation with target error guarantees. We extend the importance weighted least squares algorithm (Shimodaira) 2000) and corresponding recently proposed error bounds (Gizewski et al., (022) linear aggregations of vector-valued models. The importance weights are the values of an estimated ratio between target and source density evaluated the examples. Every method for density-rate ratio estimation can be used as basis for our approach, e.g. [Sugiyama et al. (2012); y): Kanamori et al. (2012) and references therein. Our error bound proves that the target error of the computed aggregation asymptotically at most twice the target error of the optimal aggregation.
In addition, we perform extensive empirical evaluations on several datasets with academic data (Trans- formed Moons), text data (Amazon Reviews (Blitzer al. (0066), images (Mini (Peng
et al. 2019 Zellinger et al. (021)) 1), electroencephallgraahh. signals (Sleep-EDF (Eldele et al.] 2021 Goldberger et al.) (2000)), body sensor signals -UCI-HAR (Anguita et al. 2013). WISDM (K Kwapisz
et al. 2011)), and, sensor signals from mobile phones and smart watches (HHAR (Stisnn et al.
2015
2

---

Published as a conference paper at ICLR 2023
We compute aggregations of models obtained from different Slyer--paameee settings of 11 domain adaptation methods (c.g., DANN (Ganin et al.) (016) and Deep-Corat [Sn & Saenko (2016)) Our method sets state of the art for methods with theoretical error guarantees, namely importance weighted validation (IWV) (Sugiyama et al., 2007) and deep embedded validation (DEV) (Kouw) et al. 2219), on all datasets. We also study (1) classical least squares aggregation on source data only, (2) majority voting on targel predictions, (3) averaging over model confidences, and (4) learning based on pseudo-labels. All of these heuristics outperform IWV and DEV on at least five of seven datasets, which is result of independent interest. In contrast, our method outperforms each heuristic on least five of seven datasets.
Our main contributions are summarized as follows:
We propose the (to the best of our knowledge) first algorithm for ensemble learning of vector-valued models in (single-source) unsupervised domain adaptation that satisfies a non-trivial target error bound.
We prove that the target error of our algorithm is asymptotically (for increasing sample sizes) at most twice the target error of the unknown optimal aggregation.
We outperform IWV and DEV, and therefore set a new sate-oof-thearr performance for re-solving parameter choice issues under theoretical target error guarantees.
We describe four heuristic baselines which all outperform IWV and DEV on at least five of seven datasets. Our method outperforms each heuristic on at least five of seven datasets.
Our method tends to be more stable than others w.r.t. adding inaccurate models to the given sequence of models.
2 RELATED WORK
It is well known that aggregations of models in ensemble often outperform individual models (Dong
jet a 2020 Goodfellow et al. 2016). Traditional ensemble methods that have shown the advantage of aggregation are Boosting (Schapire 1990, Breiman 1998). Bootstrap Aggregating (bagging)
(Breiman 1994 1996)) and Stacking (Wolper 1992; Breiman 1996b). For example, averages of
multiple models pre-trained on data from a distribution different from the target one have recently been shown to achieve state-of-ther
performance on ImageNet (Wortsman et al. (2022) and their good generalization properties can be related to flat minima (Hochreiter & Schmidhuber 1994 1997).
However, most such methods don't 't take into account a present distribution shift.
Although some ensemble learning methods exist, which take into account a present distribution shift, in contrast to our work, they are either relying on labeled target data (Nozza et al.) 2016)]
Xiia et al. 2013) III Mareu, 2006; Dai al. 22077: Mayr al.) 2016), are restricted by fixing
the aggregation weights to be the same (Razar & Samothrakis] 2019). make assumptions on the
models in the sequence or the corresponding process for learning the models (Yang et al. [0012] Ha
& Youn 1021 French al. 2018 Laine & Aila 2017 Tarvainen & Valpola, [201 Athiwarat
ct [lal]|2019 Al-Stouhi & Reddy|
| et al. 2018 Rakshit et al. 2019 Xu et al. 2018
Kang al.
2020: /hang al. 2015), or, learn aggregation based the heuristic approach of
[pseudo-Jabbeling some target data based on confidence measures of models in the sequence (Zhou
al. 2022 Ahmed et al., 2022; Sun 2012; Zou et al., 2018 Saito et al.) 2017). Another crucial difference of all methods above is that none of these methods can guarantee a small target error in the general setting (distribution shift, vector valued models, different classes, single source domain) described above, even if the sample size grows to infinity.
Another branch of research are methods which aim at selecting the best model in the sequence. Al- though, such methods with error bounds have been proposed for the general setting above (Sugiyamal al. 2007; You et al. 2019 Zellinger al., (002), they cannot overcome a limited performance of the best model in the given sequence (cf. Figure aand Section 6 in the Supplementary Material of Zellinger et al. (202I). In contrast, our method can outperform the best model in the sequence, and our empirical evaluations show that this is indeed the case in practical examples. A recent kernel-based algorithm for univariate regression, that is similar to ours, can be found in Gizewski et al. (2022). However, in contrast to Gizewski et al. (2022), method allows much more general form of veluer models which are not necessarily obtained from regularized kernel least squares, and, can therefore be applied to practical deep learning tasks.
3

---

Published as a conference paper at ICLR 2023
Our work employs technical tools developed in Caponnetto & De Vito (2007 2005). In fact, extend Capoonetto & De Vito (2007; 2005) to deal with importance weighted least squares. Finally, it is important note Huang al. (2006), where a core Lemma of our proofs is proposed.
AGGREGATION BY IMPORTANCE WEIGHTED LEAST SQUARES
This section gives summary of the main problem of this paper and our approach. For detailed assumptions and proofs, we refer to Section of the Supplementary Material.
Notation and Setup Let X Rd4 be compact inpur space and C R be a compact label space with inner product (., bhy such that for the associated norm !!vlly S holds for all E y and some yo > Following (Ben-David et al. (2010), we consider two datasets: source dataset (x.y) ((I1, 31))))))..... .1....., (In, vn)) E (X' x ))) independently drawn according to some source distribution (probability measure) p on X' y and an unlabeled target dataset x' (a',...,, z'm) e xT with clements independently drawn according to the marginal distribution 4x of some target distribution q on x V. The marginal distribution
on X' is analogously denoted as px- We further denote by (f) = Sxxy (f(x) vlly dg(x, y) the expected target risk of vector valued function f X + y w.r.t. the least squares loss.
Problem Given a set f1...., fi X + y of models, the labeled source sample (x, and the unlabeled target sample x', the problem considered in this work is to find model f : - y with minimal target error (f).
Main Assumptions We rely (a) on the covariate shift assumption that the source conditional distri- bution p(y(#) equals the target conditional distribution q(y) ar), and, (b) the bounded density ratio assumption that there is function
+ [0, B] with B such that dqx (x) = ((x) (2x
Approach Our goal is to compute the linear aggregation f=
c:ft for c₁ ....... Cr R with
minimal squared target risk R, (Een c.fi). Our approach relies on the fact that
2
/
arg min R. ER
Σ cafs
arg min CE....
Σ caf((()) fg(z))
dgx(2)
for the regression functions given by fa(x) = fy vylyyly see e.g. Cucker & Smale 2002
Proposition 1). Unfortunately, the right hand side of Eq. (I) contains information about labels fg(2 which are not given in our setting of unsupervised domain adaptation. However, borrowing an idea from importance sampling, it possible to estimate Eq. @ More precisely, from the covariate shift assumption we get fp(r) Jy y dp(u(x) = fa(x) and we can use the bounded density ratio B to obtain
arg min Ry CE
Eafi
cff
/ CL x
arg min
((x)
- c.f((x) - fz(z)
dpx(x)
which extends importance weighted least squares (Shimodaira 2000 Kanamori et al. 2009 to linear
aggregations E'=1 cift of vector-valued functions f1...., fi. The unique minimizer of Eq. (2) can
be approximated based on available data analogously to classical least squares estimation detailed in Algorithm| [ In the following, we call Algorithm Importance Weighted Least Squares Linear Aggregation (IWA).
(r) is guaranteed by the fact that X' x y is Polish, i.c., a separable and complete metric space, c.f. Dudley 2002 Theorem 10.2.2.).
2The existence conditional probability density q(ylr)) with 7( v) = ((y==)) gx (xr)
valuue integrals defined the sense of Cchesgue-Bochheee
4

---

Published as a conference paper at ICLR 2023
(Einr a.ft)
The optimal aggregation f= defined in Eq. is clearly better than any single model selection since
Relation to Model Selection
== arg min ERR R₂
(f*)
min
Ry
Σ cifs
Eafs
min CE
Rg
Eafi
Σ cafa
min far.Jfi
Rf(
(3)
C1
However, the optimal aggregation f cannot be computed based on finite datasets and the next logical questions are about the accuracy of the approximation f in Algorithm |
Algorithm 1: Importance Weighted Least Squares Linear Aggregation (IWA).
Input
Set fi..... target sample x'.
ft :
X + 3 of models, labeled source sample (x,y) y) and unlabeled
Output
Linear aggregation f Σ/ E J. with weights ē (Ci, (..,.... , G R
Step 1 Use unlabeled samples x and x' approximate density ratio dax by some function dox
3 X > [0, B) using a classical algorithm, e.g. Sugiyama et al. 2012))
Step 2 Compute weight vector č G-19 with empirical Gram matrix G and vector g defined by
G =
1
g -=
1
B(xx)
(yk [((ww))))
i,j=1
k=
Return
Linear aggregation f Ei
TARGET ERROR BOUND FOR ALGORITHM
Let us start by introducing some further notation: (p) refers to the functions from X' to V/ associated to a measure p on X' with corresponding inner product (.,-) LP(p) (z(p) (this space basically consists of all y-valued functions whose y-norms are square integrable with respect the given measure p)) Moreover, let introduce the (positive semi-definite) Gram matrix
Bochner space of
()) ( i=1 We can assume that G is G = -(Fr. ا-ري and the vector g = ((3fp-
invertible (and thus positive definite), since otherwise some models are too similar to others and can be withdrawn from consideration (see Section D). Next, we recall that the minimizer of Eq. (2) is
c² cc . G) G-1g, y, see Lemma
However, neither nor the vector y accessible practice, because there no access to the target measure yx- Driven by the law of large numbers we try to approximate them by averages over our given data and therefore arrive the formulas for G and ğ given in Algorithm I This leads to the approximation F. Up this point, we were only considering an intuitive perspective on the problem setting, therefore, we will now formally discuss statements on the distance between the model f and the optimal linear model f = E/m of ʃi, measured in terms of target risks, and how this distance behaves with increasing sample sizes. This is what we attempt with our main result:
Theorem 1. With probability 1 - 8 it holds that
R₂( R.y(5g) s (R, (f*) Ry(fa)) + (10g)
(log] !) (n m m1)
(4)
for some coefficient C >> O not depending on m, n and and sufficientll large m and n.
Before we give an outline of the proof (see Section A let us briefly comment on the main mes- sage of Algorithm 1. Observe, that (Cucker Smale 2002 Proposition 1) R,) - (f, = ||/ fal can be interpreted as the total target error made by Algorithm sometimes called excess risk. Indeed, in the deterministic setting of labeling functions, fg equais the target labeling function and the excess risk equals the target error of Ben-David et al. 22.10). Eq. (4) compares this error for the aggregation f. computed by Algorithm to the error for the optimal aggregation f* Note that the error of the optimal aggregation f* is unavoidable in the sense that is determined by
5

---

Published as a conference paper at ICLR 2023
the decision of searching for linear aggregations of fi...., ., only. However, if the models fi..... ..... fi are sufficiently different, then this error can be expected to be small. Theorem tells us that the error of f approaches the one of f with increasing target and source sample size. The rate of convergence least linear. Finally, we emphasize that Theorem I does not take into account the error of the density-ratio estimation. We refer to the recent work Gizewski et al. (2022), who, for the first time, included such error in the analysis of importance weighted least squares.
Let us now give brief outline for the proof of Theorem 1 One key part concerns the existence of Hilbert space H with associated inner product (---244 (a reproducing kernel space of functions from X' + ))) which contains all given models f1..... ft and the regression function fg fp. The space H can constructed from any given models that are bounded and continuous functions. Furthermore, Algorithm does not need any knowledge of H. which is modeling assumption only needed for the proofs, so that we can apply many arguments developed in Caponnetto & De Vito (2007) (2055. H also not necessarily generated by prescribed kernel such as Gaussian linear kernel, and, further smoothness assumption is required, see Sections A and B in the Supplementary Material.
Moreover, in this setting one can express the excess risk follows: R. (f) (gy ||A[ - for some bounded linear operator late the entries of G' and terms of the inner product (..-) instead. Using properties related to the operators that appear in the construction H. in combination with Hoeffding- like concentration bounds in Hilbert spaces and bounds that measure, c.g., the deviation between empirical averages in source and target domain (as done in Gretton et al. (2006) Lemma 4)), we can quantify differences between the entries of G' and G (and I and I respectively) in terms of n, m and 6. This leads to Eq.
+ H. This also allows us to formu-
5 EMPIRICAL EVALUATIONS
We now empirically evaluate the perfor- mance of our approach compared to classi- cal ensemble learning baselines and state- of-the-art model selection methods. There- fore, we structure our empirical evaluation as follows. First, we outline our experimen- tal setup for unsupervised domain adapta- tion and introduce domain adaptation methods for analysis. Second, de- scribe the ensemble learning and model se- lection baselines, and third, present the datasets used for our experiments. We then
1.0
0.9
0.8
0.7
0.6
hoMdual modeeaalmedeteeo ROUTEE bo Garptt
0.5
Scaled Angoregation weights
0.4
conclude with our results and detailed discussion thereof.
5.1 EXPERIMENTAL SETUP
To assess the performance of our ensemble learning Algorithm IWA. perform nu- merous experiments with different domain adaptation algorithms different datasets. By changing the hyper-parameters of each algorithm, we obtain, results of apply- ing these algorithms, sequences of mod- els. The goal of our method is to find optimal models based on combinations of candidates from each sequence. As do- main adaptation algorithms, we consider the AdaTime benchmark suite, and run our experiments on language, image, text and time-series data. This suite comprises a col- lection of 11 domain adaptation algorithms. We follow their evaluation setup and ap- ply the following algorithms: Adversarial
Sequence Individual Models
Figure 2: Top: Mean classification accuracy (y-axis) of our method (IWA), source-only regression (SOR), deep embedded validation (DEV) and individual mod- els (green: source accuracy, orange: target accuracy) used in the aggregation for the HHAR dataset (Stisen at (015) over seeds. The individual models (x-axis) are trained with DIRT (Shu et al.) 2018) for different hyper-parameter choices. Bottom: Sealed Aggregation weights (y-axis) for individual models (x-axis) com- puted by IWA, SOR and DEV (average over 3 seeds). Instead of searching for the best model in the sequence, IWA effectively uses models in the sequence and obtains a performance not reachable by any procedure selecting only one model.

---

Published as a conference paper at ICLR 2023
Spectral Kernel Matching (AdvSKMM Liu & Xue,
2021 Deep Domain Confusion (DDC) Tzeng
cet al.] 2014), Correlation Alignment via Deep Neural Networks (Deep-Coral) (Sun et al. 2017). Cen-
tral Moment Discrepancy (CMD) (Zelinner et al. 2017), Higher-order Moment Matching (HoMM)
(Chen et al.
2020). Minimum Discrepancy Estimation for Deep Domain Adaptation (MMDA) Rah-
man al.j
2020), Deep Subdomain Adaptation (DSAN) (Zhu et al. 2021). Domain-Adversariia
Neural Networks (DANN) (Ganin et al., (016), Conditional Adversarial Domain Adaptation (CDAN)
(Long al. 2018), A DIRT-T Approach to Unsupervised Domain Adaptation (DIRT) (Shu et al.
2018) and Convolutional deep Domain Adaptation model for Time-Series data (CoDATS) (Wilson al. 2020). addition to the sequence of models, IWA requires an estimate of the density ratio between source and target domain. To compute this quantity we follow (Bickel et al. (007) and Yoou ct al. 2019] Section 4.3), and, train a classifier discriminating between source and target data. The output of this classifier is then used to approximate the density ratio denoted as 3 in Algorithm [ Overall, to compute the results in our tables we trained 16680 models over approximately a timeframe of 1500 GPU/hours using computation resources of NVIDIA P100 16GB GPUs.
For example, consider the top plot of Figure where we compare the performance of Algorithm I to deep embedded validation (DEV) (You al. 2019 a heuristics baseline source-only regression (SOR, see Section (.2) and each individual model in the sequence. The bottom plot shows the the scaled aggregation weights, i.e. how much each individual model contributes the aggregated prediction of IWA, DEV, and SOR. In this example, the given sequence of models is obtained from applying the algorithm proposed in Shu et al. (2018) with different hyper-parameter choices to the Heterogeneity Human Activity Recognition dataset (Stisen et al. 2015). See Section D.3 in the Supplementary Material for the exact
values.
5.2 BASELINES
As representatives for the most prominent methods discussed in Section I we compare method, IWA, to ensemble learning methods that use linear regression and majority voting as heuristic for model aggregation, and, model selection methods with theoretical error guarantees.
Heuristic Baselines The first baseline is majority voting on target data (TMV). It aggregates the predictions of all models by counting the overall class predictions and selects the class with the maximum prediction count as ensemble output. In addition, we implement three heuristic baselines which aggregate the vector-valued output, i.e. probabilities. of all classifiers using weights learned via linear regression. The final ensemble prediction is then made by selecting the class with the highest probability. The three heuristic regression baselines differ in the input used for the performed regression. Source-only regression (SOR) trains a regression model on classifier predictions (of the given models) and labels from the source domain only. Target majority voting regression (TMR) uses the same voting procedure explained above to generate pseudo-labels on the target domain, which are then further used to train a linear regression model. In contrast, target confidence average regression (TCR) selects the highest average class probability over all classifiers to pscudo-label the target samples, which is then used for training the linear regression model.
Baselines with Theoretical Error Guarantees We compare IWA to the model selection methods importance weighted validation (IWV) (Sugiyama et al., (007) and deep embedded validation (DEV) (You et al. 2019), which select models according their (importance weighted) target risk. Both methods assume the knowledge of an estimated density ratio between target and source domains. In our experiments we follow Bickel al. (2007), ): You al. (2019) and estimate this ratio, by using a classifier that discriminates between source and target domain (see Supplementary Material Section for more details).
5.3 DATASETS
We evaluate the previously mentioned methods according to a diverse set of datasets, including language. image and time-series data. All datasets have train, evaluation and test split, with results only presented on the held-out test sets. For additional details we refer to Appendix [[nd]] D
TransformedMoon This specific form of twinning moons is based Zellinger al. (0021) The source domain consists of two-dimensional input data points and their transformations to two opposing moon-shaped forms.
MiniDomainNet is a reduced version of DomainNet-2019 (Peng et al. 2019) consisting of six different image domains (Quickdraw, Real, Clipart, Sketch, Infograph, and Painting). In particular,
7

---

Published as a conference paper at ICLR 2023
MiniDomainnNet (Zelinger et al., 2021) reduces the number of classes of DomainNett20119 to the top-five largest representatives in the training set of each class across all six domains.
AmazonReviews is based on Blitzer et al. (2006) and consists of text reviews from four domains: books, DVDs, electronics, and kitchen appliances. Reviews are encoded in feature vectors of bag-of- words unigrams and bigrams with binary labels indicating the rankings. From the four categories obtain twelve domain adaptation tasks where each category serves once as source domain and once as target domain.
UCI-HAR The Human Activity Recognition (Anguita et al. 2013) dataset from the UC Irvine Repository contains data from three motion sensors (accelerometer, gyroscope and body-worn sensors) gathered using smartphones from 30 different subjects, It classifies their activities in several categories, namely, walking, walking upstairs, downstairs, standing, sitting, and lying down.
WISDM (Kwapisz et al. 2011) is a class-imbalanced dataset variant from collected accelcrometer sensors, including GPS data, from 29 different subjects which are performing similar activities as in the UCI-HAR dataset.
The Heterogeneity Human Activity Recognition (Stisen et al. 2015) dataset investigate sensor-, device- and sorkloaddspeciffe heterogemeities using 36 smartphones and smartwatches, consisting of different device models from four manufacturers.
HHAR
EDe The Sleep Stage Classification -ime-series setting aims to classify the electroencephalog- raphy (EEG) signals into five stages i.e., Wake (W), Non-Rapid Eye Movement stages (NI, N2, N3), and Rapid Eye Movement (REM). Analogous to Ragab et (2022); Eldele et al. (2021), we adopt Sleep-EDF-20 dataset obtained from PhysioBank (Goldberger al. (000), which contains EEG readings from 20 healthy subjects.
We rely on the AdaTime benchmark suite (Ragab et al. 2022 in most evaluations. The four time- series datasets above are originally included there. We extend Tdaeime to support the other discussed datasets as well, and extend its domain adaptation methods.
5.4 RESULTS
We separate the applied methods into two groups, namely heuristic and methods with theoretical error guarantees. All tables show accuracies of source-only (SO) and target-best (TB) models, where source-only denotes training without domain adaptation and target-best the best performing model obtained among all parameter settings. We highlight bold the performance of the best performing method with theoretical error guarantees, and in italic the best performing heuristic. See Table [ for results. Please find the full tables in the Supplementary Material Section
Outperformance of theoretically justified methods: On all datasets. our method outperforms IWV and DEV, setting a new state of the for solving parameter choice issues under theoretical guarantees.
Outperformance of heuristies: It is interesting to note that each heuristic outperforms IWV and DEV on at least five of seven datasets. Moreover, every heuristic outperforms the (average) target best model (TB) in least two cases, making it impossible for any model selection method to win in these cases. These facts highlight the quality of the predictions of our chosen heuristics. However, each heuristic is outperformed by our method on at least five of seven datasets.
Information in aggregation weights and robustness w.r.t. inaccurate models: It is interesting to observe that, in contrast to the other heuristic aggregation baselines, the aggregation weights ......y of our method tend to be larger for accurate models, see Section D.5 Another result is that our method tends to be less sensitive to a high number of inaccurate models than the baselines, see Section D.6 This serves as another reason for its high empirical performance.
CONCLUSION AND FUTURE WORK
We present constructive -bsory method for approaching parameter choice issues in the setting of unsupervised domain adaptation. Its theoretical approach relies on the extension of weighted least squares to vector-valued functions. The resulting aggregation method distinguishes itself by wide scope of admissible model classes without strong assumptions, e.g. support vector machines, decision trees and neural networks. A broad empirical comparative study on benchmark datasets for language,
8

---

Published as a conference paper at ICLR 2023
images, body sensor signals and handy signals, underpins the theory-based optimality claim. It is left for future research to further refine the theory and its estimates, c.g., by exploiting concentration bounds from Gretton et al. (2006) or advanced density ratio estimators from Sugiyama et al. (2012).
Table 1: Mean and standard deviation (after + of target classification accuracy on Amazon Reviews, Sleep-EDF, UCI-HAR, HHAR and WISDM datasets over three different random initialization of model weights and several domain adaptation tasks.
Amazin Beriew
Houristic
Thruetical arree guarutiens
Method
TMR
TCR
IWV
1K6
BoMM
0.068 (000
(.778:20000)
0.72812:0.0 O033
6.277199.300
0.761
0.700199.011)
(..00100.110
0.700/20.3122
AISSKM
6-780 6.786:2.9 805]
0.279) 20.008
6.7781899.011
25.0055
0.7001290.122
0.780 2:000:00000 000
6.220]450.122
DIRT
0.291) (0.9)))
8.T4 (0.068)
0.756: (0.000
0.7%% (0:00)
0.600 0066
0.778
027701190000
0.787 (0.00
0.78% (0.000
0.765
8.775 (1100
0009)
0.279:20:000)
6.7781=003000
0.767
0.708100:0113
0.780:200010
0.770):40.0000
-0:02)
0.780
0.783
0:2851
MNII
(.707(590033
-.7677=000131
0.755:21100
0.0871400.00
0.701
0.700120.000
0.782 0.007
CADAAS
0.085 20.0.5 01
0.295 -0000)
1728:200.00
0.200220.12)
0.2891200-000
0.700
0.3781253.000
0.798 1:00::0000 000
0.701 (4..1)4
Doup- Core
0.985 20.0.2
8.285)2.0:000
0.2821.20.2000
20.000
0.2001,23.067
0.785: 2.0 000)
6.220]20.318)
CDAN
0,767 01
76 0101
0.7971
0
0.7700+0000.4
788 (-00010)
(-277(+000013
DANN
0 -0.0.21
0
8.796-000000
0.790:200000
0.780)
0.80015:0.011
0.770 20.011
0.778149:012)
0.797 009)
0.708140.0122
0.708
0.792
(-701--00000
0.600(3-0.070)
0.700)+000177
0.750(+0.019)
0.987(200003 31
0. -.000)
00000:00000
0.200(20.000
0.089(10.000)
0.722)56.013
0.061(293.115
(7.000
[.781(40.312)
Houristi
Throntiaa arree guarations
Method
S0
TMR
TCR
PWV
DEV
190 (ours)
TB
B&MM
B1675 (366)
025)
0.2181
0.720 17700200.000
0.6781200005)
0.747 (10
0.218
AdvSKM
0.065
0. 0.208
-00 0251
0.212:20027)
0.212) 2220000000
0.02312.0.00)
0.700 20.00)
0.6021250.006
0.722 20.005
026
0.700|
DIRT
0.655 035)
-0 M01
0.745 (715:-0000 0119)
6. 0-98814-9.0 0199
6.7001--0000 0013
0.679(-00005
0.861--0000 (06)
0.749 -0.0100
0.729 (10 2077
DDC
0U6452 -00
0.777 0200
6. 7.24 22.0 0037
0.212)29.08)
0.800123).000
0.5001230.00)
6.704)440:001
CMD
0.633 00
7000
0.732 (0.000
0.223) (000000
0.7001 (0.095)
0.716 0:716[-00000
0.6000 420)
11.01
0.725 (40.000
MNDA
0
-.727)=000000
0.720 20.008
6.714)=0:005
0.704)=00000
0.600)=3:044)
0.745 021
0.716)+00002
Card
0.615
0.708 00.01.5
6. 1177: 0688
0.212) (0.00))
0.6911 100000
0.700 (6.063)
0.875 (.0,07)
(7.5510.021
0.700) (0.300
CDAN
0.62 20 1000
0.749 =0 (16
0. 0.7391t:00.0 0155
6.7291690.00 (21)
00000)
0.622--90.03)
0.748:=0.011 (1.))
0.713
DANN
0661 20 6477
0.
8.2222-0000
6. -.8772.0 0066
0.687 (10
0.6631.00.000
200)
0.724:200018)
6.210)200:00
DSAN
01633
-0 9000
0.780)54000066
0.7091540.000
(772(000000
0.289(50006)
0.700) 0.067
0855
(0033))
0.7255 (10093))
0.717(10.088
0.737
0:212[10.000
UCIBAR
Houriek
Thrervtical
Mathod
50
TMV
TYR
TCR
SOR
IWV
DEV
IWA (oun)
TB
BcMM
0.250) 10.0785
0,830 1.0 0000
0.8181 10:000
0.808E10.00 10.022
0.283110.300)
0.5081 10.000
0,500)
0.826/10000 10:000
0.854:.10.00
DIKT
0.313
2013:-.0.000 (13)
0.907 240 0006
0.500)-40.000
0.7561490.00 (06)
0,8001440 307
0.5061
0000-40.333 13:
0.000:100 015
0.928:
(344)
DDC
0.200 20.051
0.806 2..0.
0.807 240 (40.020)
0.820 0.002))
0.7681150.000 106
0.724 0666
0.754 1000
0.804:100
(10 028
0.702:20.000
CMD
D1 7786 -0.058)
0. 8..63 4-2 0137)
01809:400 04.000 40.01.0
0.5301140.00 2014
0.731 000)
0. 0.500 0664
0.81111-0.000 (000
2-222-00 025
0.8587:441 (007)
MMIDA
0.28822000 20.08)
0 0.8199 2.0 02.2
0,812 3-
0.500 240.002
0.2801.30.06 063)
0.773 +000073
0.767 110.000)
0.807 1-0 025
0.8401-00.00
CODAATS
2916 03177
0. 0 (M801
0.8391 48.0072
0-3371
0.7801 6-0 0075
440.0m
0.7141 40.077
0.8800 01:0
0.8072-40.077
0.772)
CDAN
0.8471
0.5401150.0541
0,5021
0,781
20.
0.860:10000 018
0.8531,24..00
DANN
07256: 10.026)
0.858:
1.0 .08)
0.856 10.055
0,350) (10.657
0.8001 110
0.7131 10.057
0.750 10.015)
0.8
025
0.847:10.007
DSAN
0.940 022
0.852 (#:1.020)
0, (67.0)4
0.744[20.005)
0.808) 023
0.803)
Aug
0 2.800(+0.017 011011)
0.833 40003)
+.700(+00300)
0.771(40.00)
0.703)-40.000)
0.835 020)
BBAR
Thrometiaal piarantees
Mathod
50
TMV
TMR
TCR
SOR
RWV
DEV
PWA (ram)
TB
HoMM
B/73 0.011)
7377
00.014)
22500--0 9123)
0.7561
40.011
0.7100 40/007)
0.7331 40.00)
7590 (117)
0.761 00.0003
AdVSKM
(1718)
7296 00 (951
0.740) (5:00)
0.7181
0.8700 (0.0-0)
0.750 +6/061
0.7986 (0.05))
7922
(0.01)
0.710/
DIKT
0
2..00
24006)
0.700):00006)
0.741)2000005
0.701
0.810:
0.530)
CMD
11.7 0.754:24410000
0.747
0.770
001)
0.641140.0001
0.776) 24 000
0.
MMEDA
02238-0.000
01
0.531 20.011)
0.750 0.100
0 20.006
0.710 (10000)
0.781(20.047)
0.780
00000-0.01100 0177
0.786 00.0
CODATTS
-710x-0-0000
7002 00.023)
11275 00000
0. 0/77//1-91 (56)
0.739 20 c000
0:721/66.000
0-12
000
1. 0.7651140 2000
Deep Coml
14 11000
0.7000944.000
0.754 (61))
7744 006
0. 2051
07725-0000
0.
0.758:2000
0. 0.06110 .63)
0.77412000055
0.779140.006
0::10:00000
0.700) 2400000
3.77
0.7721-40000
0.7211
0
0.0.99
200044
0..04
0.741
0,500150.001
Arg
0 042
M. 0.3771499000 079)
0.751150:008
0.700(H0)07)
0.721(40.06)
0. 0.77000.013
007764440.0055
WISDMI
Heurieti
Theoreiiol erree giarantees
Mathed
(mum)
RoMM
01153
4.14 (276)
PD3 00-4)
0-700/40.004)
0,7000-44)0001
0.7744 COOTT
AASSKM
0.77M: 633.633
0.701-40/007)
0.717(400000
0.717 135)
7777 031)
0.770 00.04-1
DIRT
0.255
0.2982200015 20015)
0.208
0.29)229-000 99.6
0.297 =40.097)
0.704)
25.08
0.707/20.061)
0.008:20.015
0.516) 2003
DOC
137744 41/9)
0.7577 +000773
0.7011440000
0.781(40.063
792)
0.770 (000
2.7255.00000 9221
0.760 (775224000022
0.7541=0.0123
0.718)20.084)
0.501120.11.
0.771:20.00
0.742)
MMDDA
CODATS
0.8822-40.00
0.7000(0000000
0,78514400044
7G4 015
240
Deep Cord
(077)
0.718 241.035
0.000:200000
0.504 (00)
0.500180.10
720/ 020
0.77 240 (6.44
CDAN
0.
0.2900 0577
0.2632.100000 0.000.0
0 166 11.9 451)
0.7881 (10
0. 7000 2.0.667
0.760[20.00)
0755 0-40)
0.770)
DANN
11714/ 1422
728 01))
3-001:3- 9087)
0.7140 (700
=-0 (422)
0.7001/44.004 (641)
6.778 01.0
0.710) (044)
DSAN
0.0000000000
0.022)
00000.00 9052
0.256):-0.025)
0.767] (40.000
78700 1:000:000000 000)
0.5031240.025)
0.770) 240.064)
0,700 (00007)
0.7077
0.700(
0.700/800077)
0.704-00000
0.771 (400

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The ELLIS Unit Linz, the LIT AI Lab, and the Institute for Machine Learning are supported by the Federal State Upper Austria. IARAI supported by Here Technologies. We thank the projects AI- DeepFlood (LIT-2019-8) Y--YU 213), Medical Cognitive Computing Center (MC3), INCONTROL-RL. (FFG-88 1064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), AIRI FG 9-N (FWF. 36284, FWF-36235), and ELISE (H2020-ICT-2019-3 ID: 951847). We further thank AU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL GmbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, TÜV Austria, Frauscher Sensonic, and the NVIDIA Corporation. The research reported in this paper has been funded by the Federal Ministry for Climate Action, Environment, Energy. Mobility, Innovation and Technology (BMK), the Federal Ministry for Digital and Economic Affairs (BMDW), and the Province of Upper Austria in the frame of the COMET-Competence Centers for Excellent Technologies Programme and the COMET Module S3AI managed by the Austrian Research Promotion Agency FFG.
MOTION
AI-SNN 1LLT=2018-6-YOU--0
REFERENCES
M. Abadi, A. Agarwal, Barham, E. Brevdo, Z. Chen, C. Citro, G. Corrado, Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas,
O. Vinyals, P. Warden, M. Wattenberg. M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-scale machine learning heterogeneous systems, 2015. Software available from tensorflow.org.
W. Ahmed, P. Morerio, and V. Murino. Cleaning noisy labels by negative ensemble learning for source-free unsupervised domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp- 1616-1625, 2022.
S. Al-Stouhi and C. K. Reddy. Adaptive boosting for transfer learning using dynamic updates. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 60-75. Springer, 2011.
D. Anguita, A. Ghio, Oneto, X. Parra, and J. L. Reyes-Orth Ortiz. A public domain dataset for human activity recognition using smartphones. European Symposiun on Artificial Neural Networks, pp. 437 437-442, 2013.
B. Athiwaratkun, M. Finzi, P. Izmailov, and A. G. Wilson. There are many consistent explanations of unlabeled data: Why you should average. International Conference on Learning Representations (2019), 2019.
S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira. Analysis of representations for domain adaptation. Advances in Neural Information Processing Systems, pp. 137-144, 2007.
S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine Learning, 19(1-2): 2010.
S. Bickel, M. Brückner, and Scheffer. Discriminative learning for differing training and test distributions. In Proceedings of the 24th international conference on Machine learning. pp. 81-88, 2007.
A. Bietti and J. Mairal. Invariance and stability of deep convolutional representations. Advances in neural information processing systems, 30, 2017.
A. Bietti and Mairal. Group invariance, stability deformations, and complexity of deep convolu- tional representations. The Journal of Machine Learning Research, 20(1):876-924, 2019.
L. Biewald. Experiment tracking with weights and biases, 2020. URL https: www wandb com/ Software available from wandb.com.
10

---

Published as a conference paper at ICLR 2023
J. Blitzer, R. McDonald, and F. Pereira. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 conference on empirical methods in natural language processing, pP- 120-128, 2006.
L. Breiman. Bagging predictors. Technical Report 421, Department of Statistics, UC Berkeley, 1994.
L. Breiman. Bagging predictors. Machine Learning, 26(2):123-140, 1996a.
L. Breiman. Stacked regressions. Machine Learning, 24(1):49-64, 1996b.
L. Breiman. Arcing classifier (with discussion and rejoinder by the author). The Annals of Statistics, 26(3):801-849 1998. doi: 10. 00.1214/ao0/102449100799
A. Caponnetto and E. De Vito. Risk bounds for regularized squares algorithm with operatorval- ued kernels. Technical report, CBCL paper CR--CSSA------OO----O MIT, 2005.
A. Caponnetto and E. De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331-366 2007.
C. Chen, Z. Fu, Z. Chen, Jin, Z. Cheng, X. Jin, and X.-S. Hua. Homm: Higher-order moment matching for unsupervised domain adaptation. Association for the Advancement of Artificial Intelligence (AAAI), 2020.
M. Chen, Z. Xu, K. Weinberger, and F. Sha. Marginalized denoising autoencoders for domain adaptation. Proceedings of the International Conference on Machine Learning, pp. 767-774, 2012.
Cucker and S. Smale. On the mathematical foundations learning. Bulletin of the American mathematical society, 39(1):1-49, 2002.
C. O. da Costa-Luis. Tqdm: fast, extensible progress meter for python and cli. Journal of Open Source Software, 4(37):1277, 2019.
W. Dai, Q. Yang. G. R. Xue, and Y. Yu. Boosting for transfer learning. In Proceedings of the 24th International Conference on Machine Learning, pp. 193-200, 2007.
X. Dong, Z. Yu, W. Cao, Y. Shi, and Q. Ma. A survey on ensemble learning, Frontiers of Computer Science, 14(2):241-258, 2020.
L. Duan, W. Tsang, and D. Xu. Domain transfer multiple kernel learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(3):465-447) 2012.
R. M. Dudley. Real analysis and probability, volume 74. Cambridge University Press, 2002.
E. Eldele, Z. Chen, C. Liu, M. Wu. C.-K. Kwoh, X. Li, and C. Guan. An attention baasdd deep learning approach for sleep stage classification with single-channel eeg. IEEE Transactions Neural Systems and Rehabilitation Engineering, 2021.
A. Fermanian, P. Marion, J. P. Vert, and G. Biau. Framing mn kernel method: A neural ode approach. Advances in Neural Information Processing Systems, 34, 2021.
G. French, M. Mackiewicz, and M. Fisher. Self-ensembling for visual domain adaptation. Interna- tional Conference on Learning Representations, 2018.
Y. Ganin, E. Ustinova, Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Domain-adversaria. training of neural networks. Journal of Machine Learning Research, 17(Jan):1-35, 2016.
E. R. Gizewski, L. Mayer, B. A. Moser, D. H. Nguyen, S. Pereverzyev Jr, S. V. Pereverzyev, N. Shepeleva, and W. Zellinger. On a regularization of unsupervised domain adaptation RKHS. Applied and Computational Harmonic Analysis, 57:201-22 -2200 2022.
A. L. Goldberger, L. N. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. Stanley. Physiobank, physiotoolkit, and physionet components of new research resource for complex physiologic signals. Circulation, (01 2000.
11

---

Published as a conference paper at ICLR 2023
1. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.
A. Gretton, K. M. Borgwardt, M. Rasch, B. Schölkopf, and A. J. Smola. A kernel method for the pro-sample-problene
Advances in Neural Information Processing Systems, pp. 513-520, 2006.
J. M. Ha and B. D. Youn. A health data map-based ensemble of deep domain adaptation under inhomogeneous operating conditions for fault diagnosis of planetary gearbox. IEEE Access, 791 18-79227, 2021.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Hochreiter and Schmidhuber. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994.
S. Hochreiter and J. Schmidhuber. Flat minima. Neural computation, 9(1):1-42, 1997.
J. Hoffman, M. Mohri, and N. Zhang. Algorithms and theory for multiple-source adaptaatoo Advances in Neural Information Processing Systems, 31, 2018.
J. Huang, A. Gretton, K. Borgwardt, B. Schölkopf, and A. Smola. Correcting sample selection bias by unlabeled data. Advances in neural information processing systems, 19, 2006.
H. Daume III and D. Marcu. Domain adaptation for statistical classifiers. Journal of artificial Intelligence research, 26:101-126, 2006.
J. Jiao, M. Zhao, J. Lin, and C. Ding. Classifier inconsistency- based domain adaptation network for partial transfer intelligent diagnosis. IEEE Transactions on Industrial Informatics, 16(9): 5965-5974, 2019.
T. Kanamori, S. Hido, and M. Sugiyama. A least-squares approach to direct importance estimation. The Journal of Machine Learning Research, 10: 10:1391-1445, 2009.
Kanamori, T. Suzuki, and M. Sugiyama. Statistical analysis of kemee-basedd least-squares density- ratio estimation. Machine Learning, 86(3):335-36,, 2012.
G. Kang, L. Jiang, Y. Wei, Y. Yang, and A. G. Hauptmann. Contrastive adaptation network for single-and multi-source domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 2020.
D. P. Kingma and J. Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980. 2014.
W. M. Kouw, J. H. Krijthe, and M. Loog. Robust importance-weigheer cross-validation under sample selection bias. IEEE International Workshop on Machine Learning for Signal Processing, pp. 1-6. IEEE, 2019.
A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
J. R. Kwapisz, G. M. Weiss, and S. A. Moore. Activity recognition using cell phone accelerometers. Sigkdd Explorations, 12(2):74-82, 2011.
Laine and T. Aila. Temporal ensembling for semi-supervised learning. International Conference on Learning Representations (ICLR), 2017.
Q. Liu and H. Xue. Adversarial spectral kernel matching for unsupervised time series domain adaptation. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 30, 2021.
M. Long, Y. Cao, J. Wang, and M. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the International Conference on Machine Learning, pp. 97-105, 2015.
M. Long, Z. Cao, J. Wang, and M. Jordan. Conditional adversarial domain adaptation. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
12

---

Published as a conference paper at ICLR 2023
C. Louizos, K. Swersky, Y. Li, M. Welling, and R. Zemel. International Conference on Learning Representations, 2016.
The variational fair auto encoder.
C. Ma and L. Wu. The barron space and the flow-induced function spaces for neural network models. Constructive Approximation, 55(1):369-406, 2022.
A. Mayr, G. Klambauer, T. Unterthiner, and S. Hochreiter. Deeptox: toxicity prediction using deep learning. Frontiers in Environmental Science, 3:80, 2016.
K. Musgrave, S. Belongiie, and S.-N. Lim. Unsupervised domain adaptation: A reality check. arXiv preprint arXiv:2/ (t.::2//1../672 2021.
D. Nozza, E. Fersini, and E. Messina. Deep learning and ensemble methods for domain adaptation. In JEEE 28th International Conference on Tools with Artificial Intelligence (ICTAI), pp. 184-189. IEEE, 2016.
S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10))1345-13355 2010.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Pret- tenhofer, R. Weiss, V. Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
X. Peng, B. Usmanand N. Kaushikand D. Wang, Hoffman, and K. Saenko. Visda: A synthetic-to- real benchmark for visual domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 2021-2023 2018
X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. Moment matching for multi-source domain adaptation. Proceedings of the IEEE International Conference on Computer Vision, pp- 1406-14155 1415, 2019.
S. V. Pereverzyev. An Introduction to Artificial Intelligence based on Reproducing Kernel Hilbert Spaces. Birkhäuser Cham, 2022.
1. Pinelis. An approach to inequalities for the distributions of infiniteddimensional martingales. In Probability in Banach Spaces, 8: Proceedings the Eighth International Conference, pp. 128-134. Springer, 1992.
M. Ragab, E. Eldele, W. L. Tan, C.-S. Foo, Z. Chen, M. Wu, C.-K. Kwoh, and X. Li. Adatime: A benchmarking suite for domain adaptation on time series data, arXiv preprint arXiv:2203. 08321,, 2022.
M. M. Rahman, C. Fookes, M. Baktashmotlagh, and S. Sridharan. On minimum discrepancy estimation for deep domain adaptation. Domain Adaptation for Visual Understanding, 2020.
S. Rakshit, B. Banerjee, G. Roig, and S. Chaudhuri. Unsupervised multi-source domain adaptation driven by deep adversarial ensemble learning. In German Conference on Pattern Recognition, pp.
485 498. Springer, 2019.
H. Razar and S. Samothrakis. Bagging adversarial neural networks for domain adaptation in non- stationary eeg. In 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1-7. IEEE, 2019.
L. Rosasco, M. Belkin, and E. De Vito. On learning with integral operators. Journal of Machine Learning Research, 11(2), 2010.
K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In European conference on computer vision, pp. 213-226. Springer, 2010.
K. Saito, Y. Ushiku, and Harada. Asymmetric tri tri-training for unsupervised domain adaptation. In International Conference Machine Learning, pp. 2988-2999 PMLR, 2017.
13

---

Published as a conference paper at ICLR 2023
K. Saito, D. Kim, P. Teterwak, S. Sclaroff, T. Darrell, and K. Saenko. Tune it the right way. Unsupervised validation domain adaptation via soft neighborhood density. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9184 9193, 2021.
R. Schapire. The strength weak learnability. Machine Learning, 5:197-227, 1990.
H. Shimodaira. Improving predictive inference under covariate shift by weighting the log log-likelihood function. Journal of Statistical Planning and Inference, 90(2):227-244, 2000.
C. Shorten and T. M. Khoshgoftaar. A survey on image data augmentation for deep learning, Journal of Big Data, 6(1):1 -8, 2019.
R. Shu, H. Bui, H. Narui, and S. Ermon. A dirt-t approach to unsupervised domain adaptation. International Conference Learning Representations (ICLR), 2018.
A. Stisen, H. Blunck, S. Bhattacharya, S. Prentow, M. B. Kjærgaard, A. Dey. T. Sonne, and M. M. Jensen. Smart devices different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems, SenSys '15, pp. 127-140, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450336314, doi: 10.1 10.1145/28099955880777
G. Strang. Linear algebra and its applications. Orlando, FL, Academic Press, Inc., 1980.
M. Sugiyama, M. Krauledat, and K. M. Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(5), 2007.
M. Sugiyama, T. Suzuki, and T. Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
B. Sun and Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Proceedings of the European Conference on Computer Vision, pp. 443-4444444 450, 2016.
B. Sun, Feng, and K. Saenko. Correlation alignment for unsupervised domain adaptation. Domain Adaptation in Computer Vision Applications, pp. 133-171,00 2017.
W. Tuand S. Sun. Dynamical ensemble learning with model-friendly classifiers for domain adaptation. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), pp. 1181-1184. IEEE, 2012.
A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results, Advances in neural information processing systems, 30, 2017.
G. Teschl. Topics in Linear and Nonlinear Functional Analysis Amer. Math. Soc., Providence, to appear, 2022a.
G. Teschl. Topics in Real Analysis Amer. Math. Soc., Providence, to appear, 2022b.
Tzeng, Hoffman, Zhang, K. Saenko, and Darrell, Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 14 1412.3474, 2014.
Varsavsky, M. Orbes-Arteaga, C. H. Sudre, M. S. Graham, P. Nachev, and M. Cardoso. Test-time unsupervised domain adaptation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 428 436. Springer, 2020.
G. Wilson and D. J. Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(5):1-44 2020.
G. Wilson, J. R. Doppa, and D. J. Cook. Multi-source deep domain adaptation with weak supervision for time-series sensor data. Special Interest Group on Knowledge Discovery and Data Mining (SIGKDD), 2020.
D. H. Wolpert. Stacked generalization. Neural Networks, 5:214 2599, 1992.
14

---

Published as a conference paper at ICLR 2023
M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Carmon, S. Korbblith, and L. Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv preprint arXiv:2203.05482 2022.
R. Xia, C. Zong, X. Hu, and E. Cambria. Feature ensemble plus sample selection: domain adaptation for sentiment classification. IEEE Intelligent Systems, 28(3):10-18, 2013.
R. Xu, Chen, W. Zuo, Yan, and Lin. Deep cocktail network: Multi-soune unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3964-3973, 2018.
B. Yang, Q. Mao, Q. L.. Xiang, I. W.-H. Tsangand K. M. A. Chai, and H. L. Chieu. Domain adaptation for coreference resolution: An adaptive ensemble approach. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 744-753, 2012.
K. You, X. Wang. M. Long, and M. Jordan. Towards accurate model selection deep unsupervised domain adaptation. In Proceedings of the International Conference on Machine Learning, pp. 7124-7133. PMLR, 2019.
W. Zellinger, T. Grubinger, E. Lughofer, T. Natschläger, and Saminger-Platz Central moment discrepancy (emd) for Nomain-invariiat representation learning. International Conference Learning Representations, 2017.
W. Zellinger. T. Grubinger. M. Zwick, E. Lughofer, H. Schöner, T. Natschläger, and S. Saminger-Phas Multi-source transfer learning of time series in cyclical manufacturing. Journal of Intelligent Manufacturing, 31(3):777-777, 2020.
W. Zellinger, N. Shepeleva, M.-C. Dinu, H. Eghhal-zadeh, H. Nguyen, B. Nessler, S. Pereverzyev, and B. A. Moser. The balancing principle for parameter choice in regularieee domain adaptation. Advances in Neural Information Processing Systems, 34, 2021.
K. Zhang, M. Gong, and B. Schölkopf. Multi-source domain adaptation: causal view. In Twenty- ninth AAAI conference on artificial intelligence, 2015.
K. Zhou, Y. Yang. Y. Qiao, and T. Xiang. Domain adaptive ensemble learning, IEEE Transactions on Image Processing. 30:8008-985 00118,01,22220 2021.
Y. Zhu, Zhuang, J. Wang, G. Ke, Chen, Bian, H. Xiong, and Q He. Deep subdomain adaptation network for image classification. IEEE Transactions on Neural Networks and Learning Systems, 1713 2021.
D. Zou, Q. Zhu, and Yan. Unsupervised domain adaptation with dual-scheme fusion network for medical image segmentation. In IJCAI, pp. 3291-3298. 2020.
Y. Zou, Z. Yu, B. K. Kumar, and J. Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pp. 289-305 305, 2018.
15

---

Published as a conference paper at ICLR 2023
A NOTATION AND PROOF OF MAIN RESULT
The aim of this section is to give full proof of our main result, Theorem in the main paper. We start by introducing and summarizing the notation and the required concepts from functional analysis and measure theory, so that we can state and prove the required lemmas.
Summary of Notation
imput space C [gd and label space y with inner product (., .)y ) is assumed be a separable Hilbert space such that for the associated norm ||llly yo holds for all y/ E y and some y/o > O. Note that this setting is more general than the one from the main text, where we assumed y C Rd2 (the simplification in the main text improves readability and respectation of space limits).
Spaces:
Source data set: (x,y) (1,1, (In, Un)) E (X x 3) independently drawn according to source distribution on X' x y and an unlabeled target dataset x' (z',.... x'm)) E xm independently drawn according marginal distribution gx of target distribution q on x y (the corresponding marginal distribution of on X' is similarly denoted px))
Datasets and Distributions:
Source Risk: R,( = SXxx lffx)) ully dp(x,8) v)-
Source Regression function fp(a)) = fy y dd(()ax), (Vector valued) integral in the sense of Lebesgue-Bochrte
Target Risk:
15(x)) v1ا dg(#r v)
Target Regression function fạ(x) = Sy dg z). (Vector valued) integral in the sense of Lebesgue- Bochner. kebesgue-Bochher.
Problem
Given: sequence ft. X of models, source sample yx and unlabeled target sample x'
Aim: find aggretation
E Cifs with minimal Ry(().
Main Assumptions
covariate shift: p(y[>) = ((vlz) and thus fp fa-
bounded density ratio: there is 3 X - [0, B] such that (2x(x) B(x) (px(xx)
Existence of the associated conditional probability measures is guaranteed by the fact that X x y is Polish separable and complete metric space), c.f. Dudley (2002) Theorem 10.2.2.).
Notation from functional analysis/operator theory Let U and V denote separable Hilbert spaces (i.e. they admit countable orthonormal bases) with associated inner products (.. (or (.,-) respectively). Let us briefly recall some notions from functional analysis that we need in order to set up our theory. There are lots of standard references on these aspects, e.g. Teschl (2022a) and Teschl (2022b):
C(U, V): space of bounded linear operators U + V with uniform norm Illl L(U):
space of bounded linear operators U + U.
For A E C(U, V), its adjoint is denoted by A E C(V, U) (and uniquely defined by the equation (Au, v) {u, A A'o) for any u E U, v V).
If A L(U) and = A": A is called self-adjoint.
If E L(U) self adjoint and (Au, u) 2 0 for any u E U, then A is called positive. Equivalently: there exists (unique) bounded and self-adjoint B = V e L(U) such that B² = A.
16

---

Published as a conference paper at ICLR 2023
Trace of an operator A L(U): Tr(A) Ex (Aes, ek²u for any orthonormal basis (ek)k=I of U (independent of choice of basis). IfTr(A) < co: A is called trace class.
C2(U:: separable Hilbert space of FillbertScchmide operators on U with scalar product
(A, B) C2(U) Tr((B'A A) and norm ||A)l VTr(A A) 2
[[A)
A V is called Hilbert--Shmidle if A trace class. Also here: ||Al| S
/Tr(A*A) A)
For (probability) measure q on X' (or Y)) and appropriate functions F: X (e.g. strongly measurable and || is integrable wrt. g) we denote the usual (U-valued) Bochner integral F as fx Fx) (z)).. We denote the associated L"-spaces by L"(X, q, U), or Lg(g) for short, if the associated spaces are clear from the context.
Assumptions on models We assume that the regression function = fq as well as the models JI.-., belong to a hypothesis space H [ CC C (px) n (gx),, where C(X. y) denotes the space of bounded continous functions X' Y. The space H should satisfy the following assumptions, which are discussed in much greater detail in Caponnetto & De Vito (2007) and
Caponnetto & De Vito (2005):
Hypothesis 1. (Caponnetto & De Vito 2007 The space H. is separable Hilbert space of functions f: + such that:
For all E there is Hilbert- operator K₂ : + H satisfying
= )<)==Kk, E
(5)
The function from X X to R
(x, t) + (Kev, K K2w) is measurable Vu, w E V;
(6)
There is k > 0 such that
Vo E X. Tr Tr( SK,
(7)
Moreover assume that the norms |[ n. k == 1,2, ..... under control, such that put a threshold " > 0 and consider ||/all I n-
Further useful observations
Then we have
K;K, (.K,=K(1,))) L2(J)) Vx
(8)
Given x E A' the operator
Tg = K2K E C2(H).
(9)
a positive Hilbert-Schmidt operator and ensures
ITTNll<(o S [Talle,(H) [1 x) l))l S
(10)
Let Tax: 7. + H. be
Tex = -/ To dxx(x),
where the integral converges in C2(H) a positive trace class operator with
T (l I (Txxllesc)) I Tr((Txx) / Tr(Tz) s
(11)
Following Proposition in Caponnetto & De Vito 2007), we have the minimizers Sa of expected risk R2 are the solution of the tollowing equation:
T Tgx g,
17

---

Published as a conference paper at ICLR 2023
where
(exg (2) E H.
9
x
with integral converging in H.
Next we define the operators
1 m
Tx
K K
m
1=1
1 n
Tx,8
B(x,) ((x,)Ka Kr. K
12
9xy.8
n
In the sequel we adopt the convention that C' denotes generic positive coefficient, which can vary from appearance to appearance and may only depend on basic parameter such as px, qx, N. B, yo and others introduced below, but not on n, m and error probability >
We will need the following statements.
Lemma 1.
With probability at least - Ô we have
|Tex - Tx $ 1|Txx Txellaa(H)) SC logg 5) m 8) +
(12)
[Tx - -Tx (( SC (ogt 5)
(13)
[Tx, - 9x < log 1
σ )
(14)
where C > 0 does not depend and 8.
The proof of Lemma iis based on Lemma of Huang et al. (006),, which we formulate in our notations as follows
Lemma 2. ((Huang al. 2006 Let & be a map from U U such that l1ő(x))lu S R for all
x E X. Then with probability at least - 8 it holds
Σ
1 Σ ((x))(Zz))
S 511+
2 RV
2
B2 1
Moreover, we will need a concentration inequality that follows from al. (2010).
Pinelis (992 see also Rosasco
Lemma (Concentration lemma). Ef
........ . Sn are zero mean independent random variables with values in a separable Hilbert space U, and for some D > 0 one has ||E: $ D, i = 1.2..... n,
then the following bound
D log
1
3
<
i=
holds true with probability at least
8.
Proof of Lemma
Let us start by proving 12 by introducing the map E: X' - L2(H) as E(x) = K2 Tax- From
10 and
[TT it follows that
[K(=))l<<)T))) & &1K, 11Txxll 2k.
18

---

Published as a conference paper at ICLR 2023
Moreover, have
1 ((r)dax(x)= = / K2K K2 Txx = 0.
Therefore, for zy.J 2, m. drawn i.idd from the marginal probability measure qx, the corresponding operators y = E(zn) can be treated as zero mean independent random variables in C2 (H), such that the condition of Concentration lemma are satisfied with D 2k. and
2x 2 log 3
[Tx Tyx Txx c2()))
K K
Txx
C2(P()
C2(H)
To obtain 113 for any f E H we define map 65 X - H as 25(x) = K. It clear that
[lor(=)) IK.K IIfll - 1f
Therefore, for the map Ø = y the condition of the above Lemma is satisfied with R = x fl Then directly from that lemma for any H we have
1 Σ
1 Tx,9fll
Σ
25(S5)
((x))o((zi)
m j=1
n
(v
2
B2
1
$ +
2 22og
+
llflll
n
m
S log 3)
n=
llfl
m
that proves 13)
Consider now the map F : X x y + H defined by
F(x, v) = B(x) v).
Recall that | |K I (Tr(K;Ke) S Vk. Then we obtain:
[F(x,))) S Kalleco. /f - //| 18(x)) s By0 BVK.
Moreover, for p(x, y) = -(yll=)xx(x)) we have
fr F (x, y)dppp v) = / Kn)(a)) J, (()Vaoll))))))
v) dplul=)dpxxx)) = 0,
such that for (I,, y1), i 2.2..... n drawn i.d from the measure p(z, y) the corresponding values Fi F (2i, Di) are zero mean independent random variables in H.
Then for the just defined F; (xx, K2, (1.) - ys) the conditions of Lemma 3 are satisfied with
D 2yo B /n. such that
B(x:) (,,(() vi)
n
H
7
Σ Bzx ) Kr. fg Σ ((x, (r Kz. 1
7
2yo B /K
2 2 11
Txx 9x.y
This bound gives
14
19

---

Published as a conference paper at ICLR 2023
A Aggregation for vector-valued functions Next construct new approximant in the form of a linear combination of approximants f1. f2....;.. ft, computed for all tried parameter values. The linear combination of the approximants is computed as
f
ckfk.
(15)
... fi belong to RKHS H. it clear that f E H. Now we want argue on how close we can get to f.. Following Proposition in Caponnetto & De Vito (2007 we have
Since Si; .2....;
R₂( - Rg = llf falle((xx) T|TTxx fa) la)l
(16)
Next we observe that the best approximation f of the target regression function fg by linear combinations corresponds to the vector c* (ci...., cF) of ideal coefficients in (15) that solves the linear system Gc 9 with the Gram matrix G ((VTTx Sux VTaxx fu) x4) k,u= and the right-hand
side vector g (a FTx fre) x Let us provide prove of this short observation in the next lemma. Note that the entries G and can equivalently also be formulated in terms of (-5-) (oux) as done in the main text. We are going to use this formulation in the next lemma in order to be compatible with the main text (switching to the inner products in terms of H would not change the argument of the proof all):
Lemma 4. The best L (2x)) aaproximaaiin f of the target regression regression.uunction fy by linear combi- nations corresponds to the vector c" = (ct,...,c7) G g.
Proof. Let us denote 16 by f(c) and rewrite this expression appropriately:
f(c) Σ ae (f.ffi)1((x))
Σ cs (fr, fa) + (fas (((x))
1-=1
Taking the derivative with respect to o; yield:
0f(c)
= Ees (f.fB1100 (fes fa) mexx)) dc;
1=1
Setting these derivatives to zero (for all i e (1..... 1}) gives the claimed equation. Noting that the Hessian is equal to 2G (and thus positive-definit) ensures that o* is global minimum of f. r
But, of course, neither Gram matrix G nor the vector ğ is accessible, because there is no access the target measure qx, switch to the empirical counterparts G and
Then the following lemma is helpful to gain some information on the error made by the empirical average:
Lemma 5. With probability å we have
Si) ( (Voonfu fxx
1
<C log's 1 m-
fall))))
m
(VTerele VTx
SC log 1 8 (n + m2)
((.)) (fa(
where C >> does not depend n, m and 8.
20

---

Published as a conference paper at ICLR 2023
Proof. Keeping in mind that fgz for H. we have
(Verrfur Vox fi) = (Ix fhr ful 2 + ([Tax Txx)fus Jas Suyn
1
Kag K2g (usTu)
+ +(TTx - -Txr)Sus Skhr
74
K2g fis Kogb + 1(Texx -ux Sk)
m
((ll))))all)))) + ((Txx - Tu)
m
Moreover, from 12 with probability 1- we have that
((Tax - Txx)Sur Sk S Cllfullx [15lll
loog' 5) m 2.
Then
(uTex Tax
m Σ
(1(3)))((())))) SC logt 2)) m
m
/=1
Now, we prove the second statement Lemma
5
We have
(Viux Taxlaa)
(Ska Tax fa) (Sn, Tax 9x + (fn, 9n.
((x,) (fk, KaMtii + (fr, Tax - 9xv.8)
1 n ((x,) frr fa + (fkr Fax 9xy,8)
B(x,) (fa( wyy + (fa, Taxfa 9x
From Lemmal [ with probability 1 8 we have
[T fq - fxy,0llz
< 1ax - Tx 11 + Txx Sa - 2x.y
S 1Txxsa - Sal + Tx Sq Tx,
+ [[xx 9xx.lll4
< C 1llfallx
log² 1 0)
m + +C' [1lllll H
log²
1 1
(n m 5) + 1Txxof - -xxy.ellx
C1
log t/ σ)
m - + +C Ifallx
log²
1/4
1
(n + m- b) + C (log² 8) 7)n=
Then
log' 2) (n + m m-3)) -)
(Ss, TaxTa - 2xy - C Cllfella
Therefore,
(VTrnch (Taefa) Σ 3(x;) (fa(ds).
so [log* <(1os== 3 (n- + +m= -²).
21

---

Published as a conference paper at ICLR 2023
Towards main generalization bound Next similar arguments in Theorem 4 from Gizewski et al. (2022) to obtain our main result, Theorem Lemma suggests to approximate G and 9 by their empirical counterparts:
1 m
G
(17)
kuu=1
0
((x)) (Mis fa (zx))))
(18)
k=1 which can effectively computed from data samples. Moreover, again from Lemma 5 we can argue that with probability - 8 it holds:
l Tlle S C (los+ 5) (n (n- + mmmb) b),
(19)
G|lE(R)) G SC ( logg 2)) m
|[
(20)
With the matrix G at hand one can easily check whether or not it is well-conditioned and G exists (otherwise one needs get rid of models with similar performance). Thus the norms a and
((3)
can be bounded independently m and n, due to the fact that all their entries can be à ((RR)
bounded as follows (we only do calculation for the entries of G):
GR
(Iala)),
K far K K3 Fu)
m
m
m
m
( K Skz Su
(Tag
m
m
j=1
f llfull I kn/
m
200
where used the reproducing property (5) to obtain the equality in the first line and (00) for the last inequality. Now assume that m so large that with probability 1 we have
|| (G-
(21)
C(RI)
Moreover we can use the following simple manipulation:
G- (G - -
-G-
Then 21 ensures that the Neumann series for (I - (G - GGG converges and we obtain the following bound:
G ((R))
((R))
0().
(22)
G-1
G
c
((R)
Now we are in the position to prove our main generalization bound for unsupervised domain adaptation:
Proof of Theorem
7
We have already discussed that the coefficients in the best approximation f" to (ci, .g,,..,, G7) = G-1 Since:
Jg are given by c*
(g g) G G -ōe G - G--
- --
22

---

Published as a conference paper at ICLR 2023
then from 19 22 with probability 1 - 5 we have
|| c² l| = S |G |G''Il((R)) (l0 ø1lx + ||G G ((
C log
8 (n- + -==).
1
(23)
Moreover:
R₂( Rg,(())
VT 9x (J (2)
Tax (f" f4)
T Tax (f 5*)
7
71
2 2 V Tax (5* -6o
Tax 15 -
1
11
T Txx -5m) 11
2 (R, (f*) Rg(la)) 2
SR, (R*) R,(())) +2
Σ c ēxl ||
Tax fu
11
112
2 (f*) Rg((,)) + 21||c² ق
max
TTxx
74
2
2 (R, R,,(,)) +2
TTx
alles
(24)
[(H()
The statement of the theorem follows now from 2(a + 12)).
23
24
(using again the inequality (a + b)?
On the dependence of the error bound on the number of models An interesting question is. how the bound in Eq. (4) depends on l. To this end, let us have a look at the second term in the last line Eq. 24 T llc analyzes a sampling operator, thus does not depend on the number of models, same goes for ytt which is just uniform bound on all our models. To analyze ||c. a| a let us have look the individual factors in the first inequality of Eq. (3). not overload notation, C' > 0 is used here for any absolute constant that is independent of (, m, and 5.
GG G- The individual entries of G - are (in absolute values) bounded by Lemma
5 The proof arguments only involve norm bounds on the associated sampling operators, the uniform bound n on all the models and the bound B on 3, thus the absolute constant C there is independent of I. By the definition of the matrix Frobenius norm, we thus get
|| - Gllo(r) S Ci logg* 1 (n +m= =²)
(25)
Igg- Similar arguments as before lead to واğ gllc(Ry S Cvi (log's 1) (n 1 + 1)
| "l(yy It natural to assume that there is some constant 2
Otherwise, G LOR)
we can, e.g., orthogonalize our models and coefficients without changing the aggregation, but with reducing the conditioning number (i.e., with reducing
G
). It is also natural
assume that m and n are large enough such that I(n - m 20 Then applying
Eq. (25 to Eq. we can deduce that || 1 (CRR) < 2c.
| |::::: This quantity can also be assumed to be known independently l, since is given by our data only.
23

---

Published as a conference paper at ICLR 2023
Combining the previous points gives us ||e* - - CI2 (log 2)) (n + +m=) which finally leads to the refined bound
R2, (fa) (R, (f*) - (fa))) + C13 (108)) 1 2) (n mmm"
for sufficiently large I, and n and error probability 8 0.
B CONSTRUCTION OF FUNCTION SPACES
Let us give short discussion the construction of our required function space mentioned in the previous Section A] the reproducing kernel space H. As mentioned already in the main text, the explicit knowledge of H. is not required. models can be regarded an element of some reproducing kernel space (RKHS) H. satisfying the assumptions This is immediate if f X + IR is a real valued continuous function and we take k(x,9) = f( f(v) as the associated reproducing kernel. In the case f X Y and Y is finite dimensional, it is not hard to see that a similar construction is possible, as this case can again be boiled down to the construction of kernel with real-valued output, see e.g. Remark 1 in Caponnetto & De Vito 2007) for details.
just need to rely on its existence. First, any of our
Overall end up with a finite sequence of spaces (H+) of functions living the same domain X' (we have | - 1 as we also take into account the regression function), and the existence of RKHS containing all given models and the regression function is real restriction. For example, in case of real valued functions, this assumption is automatically satisfied, as linear combinations of functions with the same domain which stem from finite sequence of RKHSs belong an RKHS. This follows from a classical result by N. Aronszajn and R. Godement, see e.g. [Pereveryyev 2022 Theorem 1.4.).
There is also ongoing research on constructing function spaces (and especially associated reproducing kernels) for families of neural networks that are used in applications, see e.g. Ma & Wu (2022) for ReLU networks, Fermanian al. (202I) for recurrent networks and Bietti & Mairal [2017] 2019) for convolutional neural networks. Incorporating these into our work may lead refined generalization bounds that also reflect the nature of our models. We leave the details open for future work.
C DATASETS
This section provides overview over all applied datasets from language, image, and time-series domains.
Illustrative example: For the illustrative example (Figure in the main paper) rely on the following setting, taken from Shimodaira (2000); ; SSuiyama et al. (2007); You et al. (2019): The data points are labelled with y sint mx) with random noise sampled from the normal distribution N (0, (1) Moreover px N (1, ax N (2)(1) The density ratio 3 can be computed analytically and bounded. We aggregate several linear models with our approach and compare it to the optimal linear model, whose coefficients have been evaluated using computer algebra system.
Academic Dataset We rely on the Transformed Moons dataset (Zellinger et al. 202I), allowing us visualize and address low-dimensional input data. The dataset consists of two-dimensional input data points forming two classes with a "moon-shaped" support. The shift from source to target domain is simulated by transformation in input space depicted in Figure 3 The results are shown in the following table:
Language Dataset To evaluate our method on a language task, we rely on the Amazon Re- views (Blizerr et al. (0066 dataset. This dataset consists of text reviews from four domains: books (B), DVDs (D), electronics (E), and kitchen appliances
(K). Reviews are encoded in 5000 dimensional feature vectors of bag-of-wa of-words unigrams and bigrams with binary labels: label 0 if the product is ranked by to 3 stars, and label if the product is ranked by or stars. From the four categories, we obtain twelve domain adaptation tasks, where each category serves once source domain and once
24

---

Published as a conference paper at ICLR 2023
Table 2: Mean and standard deviation (after =) of target classification accuracy on Transformed Moons dataset over three different random initialization of model weights and 11 domain adaptation methods.
Trunsformsed Moons
Heurietic
Throretical errHe piarnntees
Mathed
TMV
RcMM
0.001(-000005)
0:381: 4 0.000)
7.540C-400 940)
(.0000-00000))
0.963/ +0.011)
0:5644 (..000
0.508(1-0005 00000
0.510(-0.0000
0:548(
AM1SKM
0.005
L.00:: (007
1.928:200000
0.058 221.000)
0:5851 (5500003
0.9801499.000
1.000:20.000 0000
1.000] 250.000
DIKT
0.827/20011))
0.830:200031
0:981 241,0003
0.050) (20 (600)
0:364
255.007)
0.3001550.000
0.983:20.000
0.200) 2000
DDC
0.9041 (1155)
0.99622.00000 (N00)
10.000
0:310) (0.000)
0.981
0:594 100002)
0.591111100003
0.9R
0.998: 0000000
(1954:24100000
0.9051=:71
00300):000000)
0.5001=00:00
1.000)
MMIDA
1.606
CODAYN
7.000
10500:400100
1/0047:44/0000
0.9811 (10
0.971140000
0.97H 17-)
10000:000 010
1.000) (605)
Deep Corl
1,00000.000 0.099
2.808:.000000
1.00022000 20.020
1.00):20.000 2200000
0.063(200.00)
1.000228.000
1,000120.000 200)
1.000:000000 000
1.200 2000000
CDAN
1 0.000
2.000:2.00000 (837
1000020.0000
1.000:200.0000
1.000 (55.000)
L.0001 200)
1.000:0000000 000
1.000
BANN
0.:011 -0 00055
-0000--00 (N0)
10:00 -0 (200)
1/0000-44000000
+0.0000
0:996 (-M40-000000
0.9991) (00)
0000 (803)
1/200) 2003
DSAN
0.0000000000)
1.000:20000001
1.000:000000
-00000))
+0000)
0.5001
1.000:000000 000
1.000|240.0000
D
(100005000000
00907:300007
96967
0.989C+00
0-981(6600002
0.907(2-0000
9.917
1.0
0.6
0.4-
langet
source class
0.0
class
0.0
0.2
0.4
0.6
0.8
1.0
Figure 3: Transformed Moons dataset. Source data is depicted as blue + and orange Target data points are shown as black dots.
as target domain (e.g., see Table 15p. We follow similar data splits as previous works (Chen et al. 2012 Louizos et al., 2016 Ganin et al. 2016). In particular, we use 4000 labeled source examples and 4000 unlabeled target examples for training, and over 1000 examples for testing.
Image Dataset Our third dataset is MiniDomainNNt, which is based on the DomainNet-2019
dataset (Peng et al., 2019) consisting of six different image domains (Quickdraw: Q. Real: R. Clipart: C, Sketch: S, Infograph: I, and Painting: P). We follow Zellinger et al. (20TT) and rely on the reduced version of 20mainNet-2019, referred to as MiniDomainNet, which reduces the number of classes to the top-five largest representatives in the training set across all six domains. To further improve computation time, rely ImageNet (Krizhevsky et al., (012)) pre-trained ResNet- 18 (He et al.) 2016) backbone. Therefore, we assume that the backbone has learned lower-level filters suitable for the "Real" image category, and we only need adapt to the remaining five domains (e.g., Clipart, Sketch). This results in five domain adaptation tasks.
Time-Series Dataset We based our time-series experiments on the four datasets included in the AdaTime benchmark suite (Ragab et al. 2022), which consists of UCI-HAR, WISDM, HHAR, and Sleep-EDF. The suite includes four representative datasets spanning 20 cross-domain real-world scenarios, i.e., human activity recognition and sleep stage classification. The first dataset is the Human Activity Recognition (HAR) (Anguita et al.) (013) dataset from the UC Irvine Repository denoted as UCI-HAR, which contains data from three motion sensors (accelerometer, gyroscope and body-worn sensors) gathered using smartphones from 30 different subjects. It classifies their activities in several categories, namely, walking, walking upstairs, downstairs, standing, sitting, and lying down. The WISDM (Kwapisz et al. (011)) dataset is a class-imbalanced variant from collected accelerometer sensors, including GPS data, from 29 different subjects which are performing similar activities as in the UCI-HAR dataset. The Heterogeneiry Human Activity Recognition (HHAR) (Stisen et al.] 2015) dataset investigate sensor-, device- and workload-speciff heterogeneities using 36 smartphones and smartwatches, consisting of 13 different device models from four manufacturers. Finally, the Sleep Stage Classification time-series setting aims to classify the electreeneephalography
25

---

Published as a conference paper at ICLR 2023
(EEG) signals into five stages i.e., Wake (W), Non-Rapid Eye Movement stages (N1, N2, N3), and Rapid Eye Movement (REM). Analogous to Ragab et al. (2022); Eldele et al. (202I), we adopt the Sleep-EDF-20 dataset obtained from PhysioBank (Goldberger et al. 2000), which contains EEG readings from 20 healthy subjects. For all datasets, each subject is treated as an own domain, and adopt from source subject to target subject.
D EXPERIMENTAL SETUP
This section is meant to provide further details on the overall computational setting of our experiments. We start by giving an overview on the used computational resources for the specific datasets and the implementation tools. Next, we describe the network architectures for the individual datasets in greater detail. In the third subsection we elaborate on the construction of our models, and the fourth subsection is devoted to matrix inversion. Finally, in the last subsection, we describe the detailed empirical results and give the complete tables.
D.1 COMPUTATIONAL RESOURCES AND IMPLEMENTATIONS
Overall, to compute the results in our tables, we trained 16680 models with an approximate compu- tational budget of 1500 GPU/hours one perfrfrrromm computing station using 8 8xNVIDIA P100 16GB, 512GB RAM, 40 Cores Xeon(R) CPU E5-2698 v4 @ 2.20GHz on CentOS Linux
Transformed Moons: 11 methods 14 parameters 1 domain adaptation tasks x 3 seeds 3 density estimator classifier = 465 trained models
Amazon Reviews: 11 methods X 14 parameters 12 domain adaptation tasks x 3 seeds + 12 x 3 density estimator classifier = 5580 trained models
MiniDomainNet: 11 methods parameters x 5 domain adaptation tasks x seeds + 5 x 3 density
estimator classifier = 1335 trained models
11 methods 14 parameters x 5 domain adaptation tasks 3 seeds + 5 3 density estimator classifier = 2325 trained models
UCI-HAR:
Sleep-EDF: 11 methods x 14 parameters domain adaptation tasks x 3 seeds + 5 density
estimator classifier = 2325
HHAR: 11 methods 14 parameters 5 domain adaptation tasks x 3 seeds + 5 x 3 density estimator classifier = 2325 trained models
WISDM: 11 methods 14 parameters domain adaptation tasks 3 seeds + 5 x 3 density estimator classifier = 2325 trained models
In Total:
465 + 5580 1335 + 4 2325 16680 trained models
All methods have been implemented in Python using the Pytorch (Paszke et al. 2017 BSD license) library. For monitoring the runs we used Weights & Biases (Biewald) 2020 MIT license). We use Scikit-learn (Pedregosa al., [011]) library for evaluation measures and toy datasets, and the TQDM (da Costa-Luis! 2019) library, and Tensorboard (Abadi et al., 2015) for keeping track of the progress of our experiments. We built parts of our implementation on the codebase of Zellinger et al. (2021 MIT License) Ragab et al. (2022, MIT License).
D.2 ARCHITECTURES AND TRAINING SETUP
In this subsection, we provide details on the model architectures and the training setup for every dataset. Our base architectures are based on the AdaTime benchmark suite, which is a large-scale evaluation of domain adaptation algorithms to support 11 of-te-of-the-arr model architectures on multiple dataset types ranging from language, image to time-series data, addressed by Transformed Moons, Amazon Reviews, MiniDomainNet and the four time-series datasets (UCI-HAR, WISDM, HHAR, and Sleep-EDF) spanning in total 38 cross-domain real-world scenarios.
time-series data. We extended the benchmark suite
26

---

Published as a conference paper at ICLR 2023
Transformed Moons For the Transformed Moons dataset use two sequential blocks with fully-connected layers. DD-BatchNorm, ReLU activation functions and Dropout. The full architecture specification can be found in Table The domain classifier (density ratio estimator) uses the same architecture. We train the class prediction models for 50 epochs and the domain classifier for 80 epochs with learning rate 0.001, weight decay 0.0001 and batchsize 128 using the Adam optimizer (Kingma & Ba 2014). We share the same base architecture and training setup across every domain adaption method (e.g., DANN, HoMM, CMD). Additional
are reported in Table 9
Table Model architecture for the Transformed Moons dataset. The values for neural network layers correspond to the number of output units.
Architecture
Layers
Values
Input units
2
MLP Block
Polne-connected Layer
128
Batch Normalization 1D Layer
128
ReLU Activation
Dropout
MLP Block 2 Fully-connected Layer
128
Batch Normalization ID Layer
128
ReLU Activation
Dropout
Fully-connected Layer
128
Methods
See Table and Table for details.
Amazon Reviews For the Amazon Reviews dataset we use two sequential blocks with fully- connected layers, ID-BatchNorm, ReLU activation function and Dropout, analogous to the setup for Transformed Moons. We also use the same architecture for the domain classifier. We train the class prediction models for 50 epochs and the domain classifier for 80 epochs with learning rate 0.001, weight decay 0.0001 and batchsize 128 using the Adam optimizer (Kingma & Ba, 2014). We share the same base architecture and training setup across every domain adaption method (c.g., DANN, HoMM, CMD). Additional ppper-prrametrrs are reported in Table
Table Model architecture for the Amazon Reviews dataset. The values for neural network layers correspond to the number of output units.
Architecture
Layers
Values
Input units
5000
MLP Block
Fully-conneeted Layer
128
Batch Normalization ID Layer
128
ReLU Activation Dropout
MLP Block 2 Tolle-connected Layer
128
Batch Normalization 1D Layer
128
ReLU Activation
Dropout
Fully-connected Layer
128
Methods
See Table 7 and Table for details.
MiniDomainNet Following the pre-trained setup from Peng et al. (201)). we use a frozen ResNet- 18 backbone model which was trained on ImageNet, and operate subsequent computations the
27

---

Published as a conference paper at ICLR 2023
512 dimensional extracted features. To alleviate overfitting effects pre-computed features, perform data augmentation on the images of each batch and forward each batch through the backbone. We incorporate zero padding before resizing the images to 256x256 to avoid image distortions. Furthermore, in alignment with data augmentation techniques from Shorten & Khoshgoftaar (019)). we perform random resized cropping to 224x224 with random viewport between 70% and 100% of the original image, random horizontal flipping, color jittering of 0.25% on each RGB channel, and a 12 degree rotation.
After the ResNet-18 backbone output, we add a projection layer, and define the domain adaptation layers on which we use the domain adaptation methods to align the representations. The backbone and projection layers are defined as common architecture across the different domain adaptation methods. Additional layers are further added for the classification networks, according to the requirements of the individual domain adaptation methods (e.g., CMD, HoMM). The number of layers/neurons in the upper layers of our architecture have been tuned in order to achieve the best performance in the source-only setup. See Table for detailed description of the architecture used. We perform experiments on all 5 domain adaptation tasks as defined in Section C for each of the previously listed methods, and with 3 repetitions based on different random weights initialization. All class prediction models have been trained for 60 epochs and domain classifiers for 100 epochs with Adam optimizer, learning rate of 0.001, Br 0.9, f32 0.999, batchsize of 128 and weight decay of 0.0001. Additional hyper-parameten are reported in Table
Table 5: Model architecture for the MiniDomainNet dataset. The values for neural network layers correspond to the number of output units.
Architecture
Layers
Values
Backbone Output Layer
R8 (Adaptive Average Pooling Layer)
512
Fully-connected Layer
128
Methods
See Table 7
and Table for details.
AdaTime Unless stated otherwise, we follow the implementation and hyper-pan parameter settings as reported in Ragab et al. (2022). We extended the AdaTime suite to comprise collection of 11 domain adaptation algorithms. We learned all domain adaptations models according to the following approaches (see also Table 6 Table 7 and Table 8): Deep Domain Confusion (DDC) (Tzeng et al..
2014), Correlation Alignment via Deep Neural Networks (Deep-Coral) (Sun et al. 2017), Higher- order Moment Matching (HoMM) (Chen et al. 22020), Minimum Discrepancy Estimation for Deep Domain Adaptation (MMDA) (Rahman al. (2020), Central Moment Discrepancy (CMD) (Zellinger) ct af. 2017). Deep Subdomain Adaptation (DSAN) (Zhu et al., (021). Domain-Adversarii Neural
Networks (DANN) (Ganin et 2016)) Conditional Adversarial Domain Adaptation (CDAN) (Long)
et al. 2018), A DIRT-T Approach to Unsupervised Domain Adaptation (DIRT) (Shu et al. 2018).
Convolutional deep Domain Adaptation model for Time- -eries data (CoDATS) (Wilson et al. 2020
and Adversarial Spectral Kernel Matching (AdvSKM) (Liu & Xue 2021). The backbone architecture of all models a 1D-CNN network. consists three CNN blocks and each block has convolutional layer, followed by 1D) batch normalization layer, ReI.U activation function, 1D max pooling and Dropout, In the first block, the kernel size of the convolutional layer is set according to the dataset reported in Ragab et al. (2022). After the convolutional blocks, apply an ID adaptive pooling layer. All methods are trained for 100 epochs on all datasets. The batch size is 32, except for SDeep-EDF, where use batch size of 128. All models are trained with Adam optimizer (Kingma & are reported Table 10
Ba 2014) and weight decay of 10 Additional
D.3
MODEL SEQUENCE
Our algorithm, IWA, constructs an ensemble from a sequence of different classifiers, e.g. obtained from sequence of possible hyper-pan parameerr configurations in domain adaptation algorithms. To obtain this sequence of models, we train multiple models for every domain adaptation task across all datasets with different hyper-parameter choices. For the experiments the language, im-
28

---

Published as a conference paper at ICLR 2023
Table Model backbone for the AdaTime suite. Kernel size, stride, output channels of the convolu- tional layers are dataset dependent and are chosen according to Ragab et al. (2022).
Architecture
Layers
Conv Block
Convolutional ID Layer
Batch Normalization ID Layer
ReLU Activation
Max Pooling ID Layer
Dropout
Conv Block 2
Convolutional ID Layer
Batch Normalization ID Layer ReLU Activation Max Pooling 1D Layer
Dropout
Conv Block 3
Convolutional ID Layer
Batch Normalization ID Layer ReLU Activation Max Pooling 1D Layer
Dropout
Adaptive Pooling 1D Layer
Methods
See Table and Table 8 for details.
Table 7: Model architecture for the AdaTime dataset. Layer haper-parameters dataset dependent and are chosen according to Ragab et al. (0022).
Method Architectures (Part 1)
DANN
Class Output Head Domain Classifier Head
Fully-connected Layer
Fully-connected Layer
ReLU Activation
Fully-connected Layer
ReLU Activation
Fully-connected Layer
DeepCoral
Class Output Head
Fully-connected Layer
DDC
Class Output Head
Pully-connected Layer HoMM
Class Output Head
Fully-connected Layer
CoDATS
Class Output Head
Fully-connected Layer
ReLU Activation
Fully-connected Layer
ReLU Activation
Fully-connected Layer
DSAN
Class Output Head
Fully- colly-connected Layer
age and academic dataset, the values of the hyper-parameters. are shown in Table 9 For the time-series datasets we use the best syper-parameterd Ragab et al.] (2022), and, to obtain good sequence of values. For all settings except Mininom,,n,et, multiply each parameter by
29

---

Published as a conference paper at ICLR 2023
Table Model architecture for the AdaTime dataset. Hyper-parameters are dataset dependent and are chosen according to Ragab et al. (2022).
Method Architectures (Part 2)
AdvSKM
Class Output Head AdvSKM Embedder
Pully-comnected Layer
Fully-connected Layer
Fully-connected Layer
Batch Normalization ID Layer
Cosine Activation
Fully-connected Layer
Fully-connected Layer
Batch Normalization ID Layer
Cosine Activation
AdvSKM Embedder 2
Fully-connected Layer
Fully-connected Layer
Batch Normalization ID Layer
ReLU Activation
Fully-connected Layer
Fully-connected Layer
Batch Normalization ID Layer
ReLU Activation
MMDA
Class Output Hcad
Fully-comnected Layer
CMD
Class Output Head
Fully-connected Layer
CDAN
Class Output Head Domain Classifier Head
Fully-connected Layer
Fully-connected Layer
ReLU Activation
Cully-connected Layer ReL.U Activation
Fully-connected Layer
DIRT
Class Output Head Domain Classifier Head
Fully-connected Layer
Fully-connected Layer
ReLU Activation
Fully-connected Layer
ReLU Activation Fulle-connected Layer
E {0, 0.0001, 0.000,00111000 0.01, 0.05, 0.25, 0.5, 0.75, 2,5, 2,), In this way, we generate a
sequence of 14 hyper-parameter choices. Due to computational limitations, in MiniDomainNet use E {0, 0.0001, 0.001, 0.01, 0.1, 1,5, 10). All values are listed in Table 9 and Table 10
30

---

Published as a conference paper at ICLR 2023
Table 9: Domain adaptation hyper-parameter sequences for experiments on the datasets Trans- formed Moons, Aevzos,,views, and MiniDomainNet. We multiply each hyper-paramter with set of scaling factors obtain a sequence. Due to computational limitations, for MiniDomainNet we use only E {0, 0.0001, 0.001, 0.01, 0.1, 1,5, 10).
{0, 0.0001, 0.001, 0.01, 0.05, 0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 5, 10} to
Datasets
Method
Hyper-parameter
Transformed Moons
Amazon Reviews
MiniDomainNet
DANN
Classification loss weight
0.9603
0.9603
0.9603
Domaín loss weight
À 0.9238
x 0.9238
À 0.9238
DeepCoral
Classification loss weight Coral loss weight
0.05931
0.05931
0,05931
À x 8.452
À x 8,452
À x 8,452
Classification loss weight MMD weight
0.1593
0,1593
0.1593
0.2048
0.2048
À 0.2048
CMD
Classification loss weight CMD loss weight
0.96
0.96
0.96 À x 5.52
x 5.52
À 5.52
HoMM
Classification loss weight HigDer-onder-MMD loss weight
0.2429
0.2429
0.2429
À 0.9824
X x 0.9824
À 0.9824
CoDATS
Classification loss weight
0.5416
0.5416
0.5416
Adversarial loss weight
À 0.5582
x 0.5582
À 0.3582
DSAN
Classification loss weight
0.4133
0.4133
0.4133
Local MMD loss weight
À x 0.16
À x 0.16
A x 0.16
AdvSKM
Classification loss weight
0.4637
0.4637
0.4637
Adversarial MMD loss weight
0.1511
0.1511
À 0.1511
MMDA
Classification loss weight
0.9505
0.9505
0.9505
MMD loss weight
À 0.5476
x 0.5476
À 0.5476
Conditional loss weight
À 0.5167
x 0.5167
0.5167
Coral loss weight
0.5838
x
À U.5838
CDAN
Classification loss weight Adversarial loss weight Conditional loss weight
0.6636
0.6636
0.6636
À 0.1954
X x 0.1954
X 0.1954
À x 0.0124
A x 0.0124
À 0.0124
DIRT
Classification loss weight Adversarial loss weight Conditional loss weight
0.9752
0.9752
0.9752
À 0.3892
x 0.3892
À 0.3892
X 0.09228
À O. 0.09228
À 0.09228
Virtual adversarial loss weight
À 0.49
À X 0947
À 0. 1947
31

---

Published as a conference paper at ICLR 2023
Table 10:
for experiments on the time- We multiply each hyper-paramter with a set of scaling factors (0, 0.0000,,00000 0.01, 0.05, 0.1, 0.25, 0.5, 0.75, 1, 1,5, 10) to obtain a sequence.
Domain adaptation
series data.
Datasets
Method
Hyper-paramettr
UCI-HAR
Sleep-EDF
WISDM
IIHAR
DANN
Classification loss weight
9.74
5.613
0.9603
Domain loss weight
À 5.43
À x 0.324
À 1.857
À 0.9238
DeepCoral
Classification loss weight
8.67
9.39
8,876
0.05931
Coral loss weight
0.44
x 0.19
x 5.56
x 8.452
DDC
Classification loss weight
6.24
2.951
0.1593
MMD loss weight
6.36
x 8.923
x 7.595
0.2048
CMD
Classification loss weight CMD loss weight
0.96
0.96
0.96
0.96
5.52
x 5.52
x 5.52
x 5.52
HoMM
Classification loss weight
2.15
0.197
0.1913
0.2429
Higher-order-MMD weight
9.13
1.102
4.239
0.9824
CoDATS
Classification loss weight
6.21
9.239
7.187
0.5416
Adversarial loss weight
1.72
1.342
6.439
0.5582
DSAN
Classification loss weight
1.76
6.713
0.1
0.4133
Local MMD loss weight
1.59
6.708
À 0.1
0.16
AdvSKM
Classification loss weight
3.05
2.5
3.05
0.4637
Adversarial MMD weight
À 2.876
À 2.5
À 2.876
X 0.1511
MMDA
Classification loss weight
6.13
4.48
0.1
0.9505
MMD loss weight
X 2.37
À x 5.951
À 0.1
À 0.5176
Conditional loss weight
7.16
X 6.13
0.4753
0.5167
Coral loss weight
8.63
x 3.36
X
0.5838
CDAN
Classification loss weight Adversarial loss weight Conditional loss weight
5.19
6.803
9.54
0.6636
2.91
x 4.726
x 3.283
0.1954
À 1.73
X 1.307
Xx
À x 0.0124
DIRT
Classification loss weight Adversarial loss weight Conditional loss weight
7.0
9.183
0.9752
À 4.51
x 7.411
À x 0.1
À 0.3892
À 0.79
X x 2.564
À x 0.1.1
À X 0.09228
Virtual adversarial loss weight
À 9.31
X X 3.583
X
À 0.1947
32

---

Published as a conference paper at ICLR 2023
D.4 MATRIX INVERSION
Matrix inversion is a well-known numerical task, especially in cases of limited computing precision and ill-conditioned matrices. In our case, similar models in the given sequence can cause numerical instability due to limited compute precision. That is, occasionally a tabula rasa inversion of the matrix G in Algorithm is numerically unstable. Various standard approaches can be applied to handle this common issuc, including the exclusion of similar models and various regularization techniques. In our computational setup, we rely on the Python routine numpp.. linalg, pinv, which is based on the eigendecompostion of G (coinciding with the singular value decomposition in our case due to ) and an eigenvalue-based regularization based on a treshold value reond for small eigenvalues, see Strang (1880, pages 138-140) for details. The choice of reond depends on the scale of the Gram matrix and can therefore be chosen by source data only. Based on evaluating our method on source data only (target domain is fixed be source domain) on several choices for rcond, we obtain stable choice for rcond of 10-1 for all datasets.
D.5 CORRELATION ANALYSIS: TARGET ACCURACY OF INDIVIDUAL MODELS VS. AGGREGATION WEIGHT
Figure 2 in the main paper suggests that there is a positive correlation between the target accuracy of each individual model (orange dashed line, Figure top) and the respective aggregation weight (red bars, Figure Sacure@bbotter bottom), if the linear aggregation of the models is computed by our method IWA. In the following, we analyze whether this trend holds throughout all other experiments.
More precisely, for a given sequence S1, Jt of models, we compute the Pearson correlation coefficient between the aggregation weights C C (see Algorithm [ and the correspond-
ing target accuracies + 1ly: = fi (z;) ..... Σ i-11 1ly: = fi(x:)] for the target test data (x1, "i,.,,)..., Mt). where [[]] 1 iff P is true and [[ O otherwise. In this context, positive correlation coefficient means that the aggregation algorithm assigns a higher weight to models performing better on target samples. We calculate these coefficients for our method IWA and the other linear regression baselines SOR, TCR, and TMR for all domain adaptation methods across all datasets and cross-domain scenarios. Note that for this analysis cannot compare to the other baseline TMV, as the count-based aggregation by majority voting does not involve the computation of aggregation weights.
In Figure 4 and 6 we compare the resulting correlation coefficient distribution of our method IWA to the ones for the heuristic baselines SOR, TCR, and TMR. Figure compares the correlation coefficients for IWA on different datasets.
We find that IWA shows a stronger positive correlation than other methods, between a model's target accuracy and its aggregation weight.
D.6 SENSITIVITY ANALYSIS: EFFECT OF ADDING INACCURATE MODELS
We study the sensitivity of our method with respect adding inaccurate models in the given sequence of models. We define an innacurate model as having a target accuracy lower than 80% of the target accuracy of the model computed without domain adaptation (SO). In particular, we add + +10, +50 and +100 inaccurate models to the given sequences of models. One inaccurate model is constructed as follows: First, a model chosen uniformly random from the given sequence of models. Second, the outputs of the chosen model are corrupted by adding, to half of the elements of its vector-valued output, random Gaussian noise with zero-mean and unit variance.
In Figure 7 we show the median performance over all domain adaptation methods for each dataset. We see, that the performance of our method (IWA) is not very sensitive w.r.t. an increase in the number of inaccurate models. This is in contrast to the heuristics (e.g. TMV, TMR or TCR) and model selection method DEV, which show a high sensitivity concerning adding inaccurate models. SOR also shows stable results; it is, however, clearly outperformed by our method in five out of six datasets. We conclude, that IWA is not only overall the best performing method, but also the most stable choice concerning inaccurate models in the given sequence.
33

---

Published as a conference paper at ICLR 2023
0.75
coefiiiient
Comelation conficians
0.50
0.50
0.25
0.00
0.23
0.25
-0.50
-0.50
171
SDR TCR
TMR
MinDomminNNet
-1.00
WISDM
UOHAR
AmaonRRviend
HHSAR
Sleap EDP
PannffrmedMoon
Figure 4: Boxplots of the correlation coef- ficients of IWA and the linear regression heuristic baselines SOR, TCR, and TMR over all datasets.
Figure 5: Boxplots of the correlation coeffi- cients of IWA for each dataset.
IWA
SOR
TCR
TMR
80
Court
60-
20
Correlation cooffkient
Correlation coefficient
Comelation confficient
Correlation coefficient
Figure 6: Histogram of the correlation coefficients of IWA and the linear regression heuristic baselines SOR, TCR and TMR over all datasets (showing the same data as Fi In comparison the heuristic baselines, IWA shows stronger positive correlation between a model's target accuracy and its aggregation weight.
D.7 DETAILED EMPIRICAL RESULTS
In this section, add all result tables for the datasets described in the main paper. Table 15
Table 16
and Table 17 show all domain adaptation tasks for the Amazon Review dataset. Table 13 shows all
domain adaptation tasks for the MiniDomainNet experiments. Table (2] Table [2] Table 23 Table 24
Table 25 Table 26 datasets.
Table 27 and Table 28 show all domain adaptation task results for the time-series
Baselines As addressed in the main paper, our method, IWA, is compared to ensemble learning methods that use linear regression and majority voting as heuristic for model aggregation, and, model selection methods with theoretical error guarantees. The heuristic baselines majority voting target data (TMV), source-only regression (SOR), target majority voting regression (TMR), target confidence average regression (TCR). The model selection methods with thcoretical error guarantees are importance weighted validation (IWV) (Sugiyama et al. 2007) and deep embedded validation (DEV) (Kouw al., 2019). The tables also provide column for source-only (SO) performance and target-best (TB) performance. We highlight in bold the performance of the best performing method with theoretical error guarantees, and in italic the best performing heuristic.
34

---

Published as a conference paper at ICLR 2023
Amazon Reviews
MiniDomainNNet
0.50
0.00
Medon Accusryy Over Al Models
0.75
Medann Auuuat Ovar A0) Modaa
0.45
IWA (oun)
TAX (sun)
1999
IMV
TCR
0.30
IAV
Setting
+10
+50
100
Settink
-10
-50
+100
Namber Random Modek Addel
Number Random Modes Added
(a)
(b)
Sleep-EDF
UCI-HAR
0.92
0.75
0.85
Medon Aocurgy Over Modes
Modia Accaney Ove A1l Madea
0.65
0.70
0.60
0.60
(wed)
0.00
(san)
TWW
TWV
ICH
1CK
TMR
0.52
45
Setting
+10
+50
100
Inibal Setting
-10
-50
100
Number Rondom Modak Added
Number of Random Madbls Added
(c)
(d)
HHAR
WISDM
0.00
0.80
0.75
Meden Accurry Over Moneee
Medin Accursyy Cvear All Mousdd
0.70
0.65
0.00
(suns)
(sun)
TMV
TWV
TCR
0.00
TCR
198
0.55
Intial Setting
+20 Number of Rondom Models Added
+50
+100
Inbill Setting
-10 Number of Random Modes Added
+50
+00
(e)
Figure 7: Sensitivity methods for parameter choice issues w.r.t adding inaccurate models in the given sequence of models; separate for each dataset (a-f), averaged over all domain adaptation methods, source-target pairs, and random seeds. Horizontal axes: Number of inaccurate models added to the initial sequence of models. Vertical axes: Target accuracy. Solid lines indicate median and shaded area indicate 50% confidence intervals.
35

---

Published as a conference paper at ICLR 2023
D.7.1 DETAILED SUMMARY RESULTS
Table 11: Average target accuracies (and average standard deviations) for all 7 datasets (e.g., Sleep- EDF, MiniDomainNet, Amazon Reviews) taken over several domain adaptation tasks (e.g.. 5 on Sleep-EDF, 5 MiniDomainNNt, 12 Amazon Reviews), 11 domain adaptation methods (e.g.. DANN, HOMM, CMD) and 3 repetitions with different random initialization of model weights. The input sequences of the approaches (e.g., DEV, IWA) consist of neural networks computed by runs of the domain adaptation methods with different hyper-paramend parameters (e.g., 8 different values of X for DANN).
Heusistic
Theoetical crror georantes
Datasel
TMV
TMR
TUR
SUR
DRV
19A (eeni)
Tranelomaal
10.000
a
0:999
Amuzam Revizno
(2287(20.0113
0-282t 2:0 000)
0.286 (0.00)
0.508 2.0 0061
0.069 (1.
0.322200004)
0.261120.0191
0.788: 10.000)
00012200012)
MiniDernoneee
0.507:50 063)
0.585/00.013 01.5)
0.520
20 01.3)
0:318:2000 01.3
(725)
05531:00001
Steep-IDD EDF
B0002 2-00000 054)
733 618)
U.22K B54)
6.725 211 (833)
0.777)-191255
6 6.700 (52)
08000 6537)
0.737: 020
0.720220.000
9.
0.833 (0/0)
0.716:200770
0.286:280000
HHAR
WISDM
0.730 (000000)
087888 (0.00))
0.765
0.735)
0.7507
0.7792 007077
0.768
0-771-00015
Table 12: Mean and standard deviation (after =) of target classification accuracy on Amazon Reviews dataset over three different random initialization of model weights and 12 domain adaptation tasks.
Amazun Beviews
Hluristi
Throrctioll crrve guaratiies
Method
1WA (ours)
RoMM
0.90K (0100))
(1.000)
0.778: 0.278-000010) (00010)
0 0:799' 6 0013
0:2771 (0:000)
0-763 +0001)
0:300110000 (0:011)
0.700M (0.019)
AMSSKM
0.765 20.00.1
730
(0779:2000100
0.779)220 (00)
6.778(1900011)
0.202
0.7061890.012)
0.780 009)
0.770) (002))
DIRT
(.281) (0.00)
0.786)
(0.008)
(1256: 1..000
0.248) 10.00)
0.6001 100011...000 4006)
0.778 10.005
0.2731110.000 (..064)
0.787
(0.701000000 006
0.290) 10.0000
DDC
0.765 01.12)
8.778
-700 0009)
0.27H 240
6.778129.000
0.707
0.710120000013
0.770]:40.00.0
0.087
8.291-000000
07200-00000)
0.280
0:29911:00070
0.76]
0.710150:015)
0.79]:30.000
0.280(10.000)
0010
CODATT
0.76K 01.333
11756 210000111000 (110
0.00915-00 0.00
0.779(+00006)
0.7731-50.00 000)
7.06 000)
0.701 (40.003)
Desp Cord
0.0855 2..00.1
2- 0091
0.758:20000 (201
0.250)20.000)
6.7821290.00)
0.701
0.7001230.077
0.786 000)
6.720):20.318)
CDAN
0.767)
00 5g
756 9101
0.757: 40 0000)
0.757
6-787[+-0001)
0.775( -0111
0.77004000014
(0800000100
6:277(40-0113
DANN
-07%/2000100
(.258)2200000
0.80012:0.0211
20.00
0.7781250.3122
20.00000000)
0.708)2500122)
DEAN
0.788 (0.00))
8 F96 (0.00))
0.790 (0.00)
0.701 10.000
0.60P1 0.0700
0.779 10/003)
0.7631 (0.017)
0.785|
0.7681 (0.017)
0.067
0.780;20000)
0.772(56.014)
Table 13: Mean and standard deviation (after +) of target classification accuracy on MiniDomainNet dataset over three different random initialization of model weights and five domain adaptation tasks.
MislDemainNet
Hearietk
Theeretical erree purrateess
Mathud
TMV
(Hum)
AA-SKM
0.220-24.112
0.516
0.314):40.014:)
0.5141:00.00
0,5121
0.510:
0.522).44..00 22022)24)007
DIKT
0.219-20.1222
0.517 (NA)
0.2161 243 240.000
0.208) 440 80M
6,2071290.300 26.8)0
0. 0.4500 25.005
0.458 (20003
0.G19) 2:0 000
(.520:24).220
DDC
0.5:0 10:02)
0:312( 2.2.0000
0.3121 11.000 11.0085
0-5111 10.009
0. 00.5
0.511 10.008
0.312) 10.615)
0.516)
0155
0.521:110:000
CMD
01200 0.500-20000
0.528) 011011
0.581
2:17 020)
0.528 440.000
0.2221 20.33
0.3181 440.005
0.400)
(353) 000
0.555)/241 (24)
MMIDA
0.58)(1.0000 (012
0.3221
0.310110.012
0.528].0..00
0.551::40:00)
1136)
0.
0,2051
Docs-Conl
3003
0.528
CDAN
0.G.6.20 20.12)
0.53M12..0.007 (1.77
0.524] 20.011
0.2251 200.00
6.8171800000 2013
0.318) 210.011
04.04)
0.652)
010
0.512220.037)
DANN
D 0989 -0.033)
0.530X 4100111
0.535
0111
2. 0.50011--1 0055
0.516[4-0-00 0013
0. 0.3110 +0.00333 0115
0.3155-40.515 (013
0.541 DRR
0.5392240000 (3-00))
DSAN
0.500'21.1122)
2:00011
0.584)140.000
0.6621 00110113
+40.008
0.566)240.223)
0.640 007:
Aug.
0.588( 9121
0.7000
0.510(000002)
0.681 011:
0.5341540000 (01))
36

---

Published as a conference paper at ICLR 2023
Table 14: Mean and standard deviation (after +) of target classification accuracy on four time series datasets over three different random initialization of model weights and five domain adaptation tasks.
SDF
Heuriskk
Theooctical
Method
S0
TMV
TMR
TCR
SOR
IWV
DEV
IWA (o3)))
TB
B&MM
01677 (0)6))
8
8.218 10.021)
0-264
0.720 100000)
0.7.7 (0.05)
0.7155 (0.041)
MSSKMM
8.
ME331
8.7111200000
0-7151=0
0.710)=00000)
0.8021=000008
0.722: 025)
0.700):+0:004
059)
(69)
DOC
B.64K
1.717 0219
0.7111
0/600 (0k)
0.0M
0.8011500000 001)
0.724 0704000.010 01.
0.704 201)
CMD
0.033
05777 0573
7:00=0 MM533
0.75002410000 (000
6.7001:590000
08600:59.00 0.710=00001)
063)
0. 0.720 (720:20.000 01.0
0.755 (53)
XIMDA
B.633 -00551
6.
6.736:2.
91.3
0.725) 290018)
6.2141.59.006
0.7M4/36.063
0.6001.20.664
0.745 2:0 031)
0.716(44..222
CHDAAS
0.072 -0 03-43
8 F2N 6770)
7 4006
0.230 (400000)
0.7550 (09)
0.083
0.800) H07)
0.715(40.063
8.716200003 -0.05
6.7
0.712:20.022
0.8941
0.6781=30077)
0.718::0
0.702)
DANN
0.611 00.071
0. 0.730 01.01:
0666
0.221 (0.000)
0.2111 10.004)
0.887 (.001)
0.6011 (0.00)
0.724)100 (0.01%)
0.710/ (0.000)
DSAN
0.63K=00 20 (K00)
2 008)
0.732)
6.72816900006
0.707(=00011) 0,28012900041
0.700):5000044
Arg.
(665(200054)
6.
0 0161
(.729)251124)
0.725/290002)
0.7171:90.00)
0.5001200667
00.07(0000200)
UCI-BAR
Houristi
Therevtical crree purrustos
Method
TMV
DEV
IWA
RoMM
0-753 (0.075)
0. 8.80 4- [MP)
0.818(+0-022) 4030
0:S188 40.0507
0-760(+00000)
1066 01000
NAYSRM
0.224:20.0500
4479)2290222
0.800) 2220
2.5201 4.00 .022)
6.7001.5)))222
0.707 (+00.100)
0.711/26.117)
1:800:100000 023)
0.811:24 (200))
DIRT
01733) 18.05
0.911 01:3)
0.9071 10.0000
0.2561
0.307 107
0.508: 10.112)
90800
0.928:
+40
0.85 0.82014::0
109
0.724
100)
CMD
24.058
0.8000 01.21
0.340
051
0,731190000
0.8121 00001-000000
00.421
025
0.3581244
2007
MMIDA
08285 -0.018)
0.819 3.0 0127)
0.812)
10 10.028
0.500 (10 02062
0.2501150.063
0.2751 440.093
0.767
0.807 1.0 025
0.840220
CaDATTS
7505 -0.1377
0. 0.850/100
01832)
440 0271
0,53011440.00 200)
0,7801 .077
0 0.501 (-00057)
0.7M):20:076
0:840:100 01.0
0.8177:24
Docp- Cord
0 0.200:20.03 26.036
0 0.8100 00:0022.000000 [09
0.800
022)
0.5081 (40.000)
0.7711.90.000 (23)
0.2681 (-0.0044
0.779
0.808::22 1:00::1:00000 016
0.800):000000 222)
CIAN
02200- 055
08400 0000
0.8091
(09)
0.500 2013
0.407)4-0 (60)
0.781 (40.007)
0.087 (100
0.800:1000 0186
0.8502.44 (200
DANN
016)
3:0 050
0.500)
0.7001
0.280)
8:840
0.847)
Avg.
0.220:00.040)
0. 01
(0.00))
0.802(400.04))
0.700(100.00)
0.774(40.000
0.765[30.000)
0.855(100.02 020
0.850;20.029
HIAR
Hauritic
Throstaal srrve guaration:
Method
TMV
TCR
SOR
PWV
DEV
(ours)
RoMM
7000 (917)
017
0.700(=0
0-700CH0/0PP)
0.7501-400001
7770:-0 007)
0.
AASSKM
0.025
3.3498..0000 0271
20/32)
0.748|:n 2..544
0.600)200.44)
0.728(10.061)
0. 0762) 031
0.740 (100),10.000 240.025)
DIRT
0/716/-000 07165
6. 2M00 0- (11)
0 07790-40.070 01:066
0 4..007400.117)
0.7w1] 40-000)
(71304000055
0-7281+0.0773
0166 (00)
0.590]1+400000
DDC
7200229 9009
0.748) 240.007
0.717)=06005)
0.711/200005
0.70G/40.006
0.T48::0000 0..02
0.723)
A774/=0022
0.700
0.764 200000
0.757 20000000
0.776 0011
0.541130.001
20000)
0.734
MMDD
0.055
CoDATS
0.000000000
0.775224:0400
0.722/
0.728)
8121::00 000
0.755
Dapp Caml
8.245-
0.065
06960 10.015)
0.769: (0.027)
0.8811 4460755
0,758 (00.8
006
0.776
CDAN
14 11755 (3177
0.7744 631
0. 00 .62)
007 000)
0.7771660.000
0:61::00000 011
0. 0.774)
DANN
0.257-000557 6571
6.
1.7282.0-912 91.1
0.274 2000
0.278::4000 110
0.722/ (03)
0 0.7081 (013
0.700/40.046)
0888:-000000 0005
0.307
(000)
DSAN
0.05)
0.80002-0.910 919)
0.800:20..00
0.724)2000065
0.741 (50/0033
0.500(240.00))
0280):00000 008)
0.530
Ang.
0.202
00005))
(0.007
0.399(4-9.08))
0.722
0.706 (00007))77)
(40.00)
Harritic
Threntikad serue
Methed
50
TMV
TMR
TCR
SOR
PWV
DEV
PWA (our)
TB
9. 1.771 (002)
0GGI:=0 0G4)
0.7411+0.044
0,7281=00:011
0.774
Ad-SKM
DIRT
0.788-8000 (1155)
9997 (986)
0.7677-44
0.7561
0.707
0788::0000 018)
0.536
DDC
0.280 (.)13)
0.770
2..152)
0.587-2-9 019)
0.287(246.073
0.741[50.006
0.741120.003
0782):0000 038
0.270 (770120.03 2000
CMD
6. 0.272:2.00000 0021
(K266:200
0.2677 24000400
0.728)220.000
02.00)2000041
0.669/20.1133
0. 0.778:200000 0822
0.7421 340.071
MMIDA
7070 0447
6. 2-000-0000000 91
17777 (3300
0.75014440000
01770) (0-077
0-700(40.0077
7800
0.775 2000
CODATS
22227
0.582) 2000077
2-700)
0.786120.004
0.041:0000
0.770) 240.000
CDAN
0.700-000577
0.003
787 12: 051)
0,768
0.700
0.700 2013
7057:000
0.775
DANN
8/224--0011
4.750-10000 (..01))
6985 9957
0.290/ 10.050
0,7201 L.C.
0,300 1.0003
078 01.0)
0.760 11..0-1)
DSAN
0.756 0200
0055 (00022)
01 7669
90344
0.254] (000025)
0.759) 1 (000
A0.000
0. 0.7221(10.013)
0.770) (..00)
Ang
0.73 0000)
0. 90.56) 00.087)
708) 0336
0.7185
0.757( (00007)
0.790(+0/069)
0.700 00.0777
(10.0)(00.000)
37

---

Published as a conference paper at ICLR 2023
D.7.2 DETAILED AMAZON REVIEWS RESULTS
Table 15: Mean and standard deviation (after +) of target classification accuracy on Amazon Reviews (Part 1) over 3 repetitions with different random initialization of model weights.
HoMM
Heuristic
Thosectical gurrsneess
Task
50
TMV
TMR
TCR
SOB
IWV
DEV
IWA (ours)
TB
D
0.778
0,758 2.000))
0.285(2:0.00 000))
0.705)=0.000)
0.752 -0.021
0.78212000211
0.794[20.077)
1.789/220000
0 0.015))
0-750[+0013)
0.261+00019)
0. 0.9071-2.00 (.616)
0.797 00.055
6:749[+00025)
0.760 0-701)-
0.758/+00012
0.771 0.015)
0. 0-788 0161
0-762)+0.616)
0.760[=0.01))
0.779-0.001
0.761 0.76[-0000 (0.015)
0:967(+00005
0.770/+00008 +-7.1
+.000)
0.291 (2 602
0.70(=000022
0 0055
0.778
00.00)
0.777(=00015
0.79: 0055
0.787(+000006
E
0.772 -0612)
0.781 (2 6031
0.781
0.781(=00003)
0.779;:00004)
0.778:=0.000 0014
0.764
0.781;+00004] 0.700120.000)
0.774
+:0.000
0.7231 90000
0.701
0.7912500016 0.791800000)
0.791 (177)
0.73
0 0.789,18..00 0.782(10001
0.7800 10107)
0.700000
0.7(-0.00)
0-29719-0000 019)
0.001 (010
0.00 10.011
0022(-0014)
0 0.608(110 (-0.011)
0.705
0
0.738
0. -.000)))
0-2361+-0-00 021)
0.331 0.05)
0.011)
6.719(-0:00)
(255(++0.011
0.735(+60011
-0000)
000000
0.828]+00 029)
0.873-00
0
0 +:0
0. 0.865
0.00
0.22112: 002)
0.721
0.7.9)20004)
0.721
07040-00007
0.22
0.702:0001
0.702:4000)
0.747700003)
0.7201
0000))
0.040
0.805(00000)
0.837
(000(000000 00005)
Aug.
0.708(4000)
0.708(00.03)
0.778(+6.01)
007011
0.7001200012)
0.778(+0.009)
0.77((400011
AdvSKM
Heuristic
Thouecical poursnless
Tock
TMV
TMR
TCR
SOR
JWV
DEV
IWA
B D
0.004))))00)
0.79| 607)
0.70(=00.003)
0.789(10.003)
0,79:20.003)
0.277 (20.005)
0.778(20.005)
0.704[=0.005
0. 0.700
2..00)
(20014))))
0.7677 012)
0..02
0.754 (3.003
0.7511-0.0077
0.761|
0.700/200012)
0.709/
2002)
0.28/ 0.28/ 9251
0.764(20011)
0.782(20002)
0.78:20.016)
0.771 (3..012)
0.764120.2151
0.774(20019) 0.782(=0.017)
D
0.77M/ 2004))
9.
(1-7 (07)
0.700)
0.704(5000m)
0.7921-0.000 00.00
0.780 20.007
0.78.20.011)
0.796 0.0099
0.789
E
0.266
0.285 6061
0,784(=00006)
0.777
8.770
20.005
0.700=9.009)
0.785:=0.00
0.770
00.01
0.005)
0.70)
0.791 ( 61.8
0.778
20.014
0.770/=0.014)
0.760:-0.013
0.080
B
0.688| -0015
22022)
-.022))
0.706/200024)
0.700 2- .62
0.704 20.022)
0.606)-0.0251
0.7121=0.00 00.02)
0.707 02001
-000))
0.704(=00002)
0.797[20.000 4001
0.781 -0 0015
0.728
20.004
0.729-20.044
0.7321 -0 0..00
0.7371110011
0.858 -0001
+.80:+0001
0.870;+001
0.878
+0.01
-0.001
0,875
-0.01
0.50% +000))
0.72)
0. 0.231 029))
0:70
-00001))
0.706-000011
0.726
006)
0.741::30001
0.
2-22 018)
0.751:=0018) 0.713)
0.752 017))
0.78
0.702(00.005
0.700)=0:011
0.750
(-0,0))
0.007)
(.0077))
0.850 008
0.856
0.8.0(20
0.877(000007)
0.858|
00000=00005)
Aug.
0.700()
0.708(+0.02)
+.009
0.779+00011)
+00001
0.700(=0.011
0.78(=0.01)
0.770(0001))
DIRT
Heurletic
Theoretical error guarantees
Tuck
TMV
TMR
TCR
IWV
DEV
IWh
+0001)
08000(00000)
0.8 0000)
9.819 (08)
0813(-0.000 (0.00)
0.20(
0 0,815( (0.50-000000
0.8 0.
0. 0.01))
0.762[-0.000
0. 0.000)
( 07)
0.78%
0.258 +0.01 1)
0 0.750-00.01)
0. (.788( (0.008000)
0. 0000)
0.77
0.005)
0.815(+-2 603)
0.815
0.8 4(+00005)
0.855-00
0.703 20.000
0.752(-0.025)
0.814;=0.000)
0.81.((-0000))
D
0.812
0.00))
0.812 03
0.805 20.013)
0.000 200044
0..01:0.00
D
0.8.3 0000))
0.81418:0.00 018)
0.8001 0.001.
0.7871 20
25.14:00.009)
D
0000)
0.836 (.00))
0 00000500000))
0.600250000))
0. 0.837 ogg
0.788 (0)005)
0 0.8322 0.000)
0 D.
R- R
00018
0015)
0.65
0. 724 6-2211-2-00 2-666
0.
7022 (0.018)
0-675(-0005
0.0786 (..15)
0.716--00066
D
20,22))))23)
0.281 2 0.629
0.718:
0.704)
20.02)
K
0,858
-00333
0.006
0.855/200005
0.893 0 000
0.8744 0.574
04.00)
0.884 (0.0099
O. 0.800
O
7000 2002)
0.701 00000
0. 1272 0 0003
0.73 20.02
07130-00000)
0.2211
O. 0
0.730
-00012)
++0001
0.721
0.727)=00019)
0.729
=0 02.022)
0.7011-0.007)
00000
00002)
+:0001
4:000)
0.568 (.0.0))
0.872 000)
0.845 20.081)
00.00
0.808[=0.03]
00000)
Avg. 0.704(200033
0.786
0.766600112)
0.8020001)
07770
0.787(-0.000)
0.700--00000)
DDC
Heuristic
Theurticaa crour guarmnlees
Taok
TMV
TMR
IWV
DEV
IWA (ours)
9. 808
0.794 0.708( 0.000
0.7766=8..01
0.778/20.02)
0.789 .0000
0.786
0 0.95 (000))
0.760/-0 00014)
0. 2600 00144)
0.7602 0.13))
9. 2.000 m7)
0.250 -0.033
6.500-00000 010)
0 33663 +-0.01 4)
0.750(+6.00
0.
0.781(-00115 0.15)
0. 0.051(-00015)
0. 0.2601+0 0:7)
0. 0.778(-0.00 0.0
0.770 00.016
6.270[=0000 016)
0 +0.501)
0.76(+00014)
00006
0. (.797(+-2000 607)
0. 0.768
0.7911+00000)
0.780(-0.005 0.012)
0.287 -0.009)
0.788(=00000 066)
0 0.796 +0960:00.000)
(.7886+00000)
D
0.762[=0.004)
0.78(=0.004) 0.002)
0. 0.7871 696
0.778 -0.00
6.770/ 0041
0 07784(+0.004)
0.778
D
0 2015
0. 01922-00133
0. 0.7901 00033)
9. 777775- 015)
01700 01115
01787 00014
0 07733200111
0 07335.0.017
0.785) 800017
E 8
00015
0. 007000000533
0. 07000-000555
0. 7. 027)
0.77 00 0723)
0.000 0.011
0 77755-000225
0. 0.700
R D
-0000)
0. 1/301122000 m02
0. 0.730[000001
0.7311+00000)
0,735 -0.00 0.01
0711 (211(-00000)
6 07000-000011
0 0375:++0.00
0. +01007
R K -0611
9 8.211e2 (000
0.711 (000)
0,857 -8001-0-0011
--0011-
0. 0100:
B
20002
0. 0.289 22 (22)
0.7281 20002)
0.721 - (013)
00015
o 200011000007
0.726 000)
0.7L9 100000)
K DD 0.201100011 00001
0000
2.75511..00 07)
0.7501 0.016)
0.748/10001) 0.012
0 7750 013)
0.745/ 100012)
K B
0.800
0.800500011
0. 000011
0.806 0000)
0. [- 008)
0.84 30.003
0.00500011
0 0056 (006)
1000000
Aug.
0. (.0000)
0.78[=0001)
000001 +.700
(0.001)
00708
0.78(=0.011)
0.273(#6.01)
38

---

Published as a conference paper at ICLR 2023
Table 16: Mean and standard deviation (after =) of target classification accuracy on Amazon Reviews (Part 2) over 3 repetitions with different random initialization of model weights.
CMD
Heartstic
Theoretical error guarantees
Tack
SO
TMV
1MR
TCR
SOR
JWV
DLV
IWA tours)
TB
00000)))
0. 1.990--0 (.00))
1.778 -0.018)
0 000)
U.T.MC+CUOO
000022
0.2851 024)
6.779
0.770-00.011
0.782|=0.01 00777
0.78(200014)
0.70770.007
0.791-00000
-0.01)
0.7922460000)
00788
0.00
0.796550000)
0.8011=00 (000
0.011)
6
0
(.707)
0.802
..000)
0.501:000011
0.029) =0014
0.777(20011) 0.
0.782
20.00
0.110100.013
A:00013
0.700
0.000)
0. 0.702] 001)
0.7511-00005)
0.041
0.579(10.03)
00.0033
0.738 (.0.000)
01002-
0.875 (08)
0.822
0 0544
1:0, 000)
K
(.0,05))
0. 0.749)-2.000 0.00)
0
(774:00.05)
6.07) (76)
0 746 (6600:-00000 1-0.004)
0.74
K
0.748 (20000))
0.706)=0.022) (E0666)
0.77 0.007)
0.630 00221
119)
0.701 (7000000116)
R +
(2002))
(.804))
(.802( (-
0. 0.80% 0.00)
2.
0.000))00041)
0 (1:07():0.006)
(.878((00.004)
Avg.
0.707(5:0000)
0.70
0.70
0. 7.222-0.009 0.700)
0.71(30.05)
O 094(100.008
U. (.7
MMDA
Heartstic
Theorecical ervor guarantees
Tisk
80
TMV
TMR
TCR
SOR
IWV
DEV
IWA (ours)
TB
0.750(000005) 0.075
9. 0-997[+-0 0.750(-
009)
0.701(-0.011
283 (0.00)
6:TH0[-0.000)
797 ++0.003)
0
0.000
0.775:++800)
0. 0-729] 0.775;+000)
008)
(0.011-
0.791 (0.022)
6:750[=0.013)
0.781[+0.000
0.000)
0.766)-00016
0.702)=00017) 0.708(=
0.902(-0 6118)
-..00-00000)
6.967(=0.001)
0755:=0.01
0.9022
0.753
0.2061 607)
-.750
6.780,5000000
0.796 0065
0.015
6.780]=0.0123
798 ::0.009
0.79:2000007
9. 2.
0.7927
07781-00
0,790
0. 0.7071-0000
0.0071-00001
7000
0.007(+0.011)
0.7441+0000
0.758-00.01)
6.708:00
0.738
10005)
0.870 0.00)
0001
0.836
0.878 000000
0,730-00.003
0.730--0.005
(.00)))
0.730 0.000)
20.00
0.732(1:0.00 004)
0,75
0. 0.701590013)
0. 0.701 (-
0.734 00000)
7500 (1.007))
(.7(50001))
0 0.00))
0.856'+0.000
9.8581+-0 0.0000000000)
08855-00
09.51-9.0139
(.82(10011)
0 0.858(100000
0. (6007()
Ang.
0.707(200000)
0.705(20000)
0.785(20000))
9.787( 07)
0,700(200017)
0 0.787(10.000)
U. (0704
CoDATS
Heuristic
Theurctical erree guarantass
Task
S0
TMV
TMR
TCR
SDR
IWV
DEV
IWA (sun))
TB
0.783 (00013)
0.8/210.002
108/200.004
0. 0.33:20.003)
1.788/10.0080 (0.)
B.792 -0.001
1.855:200000
0.7851 (000)
0.788
8.758)))0044)
0.787(=11000)
0. 2..9911..000 9..0
0.77 (11
0.773 21.0011
0. 0.788| 20 006
0.800) -00000
9.80
(1 (P7)
D
0.800 (20000)
0.8071
0.775
0. 0.700 10.0025
0 0.806
-0 009)
0.007=000007
0.750
0.706(=0.005)
0. 1.507:
0.701
0.792
0.
004
0.50 +0.007
0.7891 =0000)
00000)
0.81(=0.0009 0.008)))
0.508/20.0008
0.78%
0.777
0.5111
0.817100.011
0.7621 =0022)
0000)
0.52(=00.01
0.9241220007
0.706
0.78% 10.0013
=0.0
0.828
0000019)
0.015-
0.715
0.738/ +0.00))
0.731(-0.027)
-0.155
E
0.865(=0.000)
(.8.00)
-0.012)
1.870=00.012
0.752
0. (.732)
0.7521=0.000
0.797
0000
0.751
0,730
0. (000000000 60000)
01 1770 0..0
0.735) 20.000000077
0. 0.766
764 -000
(0.0
0. 0.800(00.011) 0.007:30.011
0. 88.86 0.0.
0,553 06.3
(2000330.011
0
8500 (0.00)
Aug.
0.760(=00012
0.700(=00000)
0.794[=000) 0.708(=00000)
0.220
0. 0.779)
0.703(
0.797 0.706(=0.01]
Deep-Coral
Hearlstic
Theoretical guarmntees
Task
SO
TMV
TMR
TCR
SOR
IWV
DEV
IWA (ours)
TB
0.
000188
9. 2 02E)
0. 0.750(-001))
0. (.
0.754:-0.00 1)
0.700 -0.005)
6 T8S 02432)
0.801 (-081)
E
20013
0.071
0.772 9081
0.764--0.013
0.7591 -0,001)
0.702
0.773/10.008
0.707/ 100.005
0 0.7001 2001
9. 1.885 6. 9131
0. 0003)
0.784/20016)
0.st1= 0.005)
0.702 9.003)
0.785/10.00 0.774
0.770.150001
(000)
8. 2 M001
0. 0.750/ 0000))
0.900 006)
0.78 20.0000000
0.7731100005
0. 0.797 1:0 004)
0.786
0.701 0.03)
0 20000)0
0. 9000))
9. 9-2851s: (08)
07781 0.00)
7101 0.000
0 076402000005
0 7784
003)
0. TH00 1000000
0.78 00016)
0001
0.011
0.780
0.9251 016)
0.757 0.0077
0.780)20.019)
0 0955
013
0.780
0.709) 024)
0.706
0.058 0.006)
0.0881200118
0.71: 024)
2:0.072
00000)
0-788 00000)
0000
0.741
0.721 122 .90)
2..00
0.74 0.701200017
0,7.0
0,800
0.877 - 0077
08081200000 00,07
0.878130.00 (07)
0,854/2000000
0.758 000)
0.
0.719
6
(
0. 00022
0.751;
0.350)
0. 7522
0.055
6.001)-00000
(0000)
0.858
0,810 (0.01
1000115
9, (. 20000)
0, 20000))
0.700
0.785(10001)
0.77K
39

---

Published as a conference paper at ICLR 2023
Table 17: Mean and standard deviation (after =) of target classification accuracy on Amazon Reviews (Part 3) over 3 repetitions with different random initialization of model weights.
CDAN
Heartstic
Theorecical error guarantees
Task
S0
TMV
TMR
TCR
SOR
IWV
DLV
IWA (ours)
1B
9. 8-0 (97)
(.803(-0.00))
0.803(00000)
0,706 (000
-.000000000)
6-792(-0.000)
0.50[++00000 (..008)
[.797((60000)
9. 2-29 (09))
0.77:);=0000) -.07))
0.776/00.013
0.640-0.013)
6.961(=00012)
0.778:+-0
+..05)+00000)
0.000)
0.79525:
0.7921
0.78/
07911-00007
0.796-000
0
(P7)
0.707)
0.790220004)
0.00
0.788(10011
0.7071-0019)
0.2977:00001
0.082
0.707
0.707(20.015)
0.727
0
0.71:
0.713
01.15
0.738(20001)
0.721 0.005
-.017)
0.723
0.741(1:0.000 .0000
0,73X 1.00000
0000)
9.879 022)
4.000
0..44-00.0121
0.86L 0.11
0.801:2000
0.8791100000
200
0.728(1000)) (O0
0. 1.700 095)
00011)
0 (04)
R
0.746 (20.01)))
0.784(500001) (.061))
0. 0.300 0.02)
0.752
-0.03
0.75 085)
0.782(=0.01) 01)
0.702)
0 (2.000)
(.80(20).00)
0.801(2000077
0.867 0.00
0.848 -0.005
0.840)20.007)
0 (200000
0.80
Avg.
0.707(2:00122)
0. 7.787560000 07)
0.787:
0.787(&0008)
07.7
0.770-00.022
0.770(20012)
0 788(100.000
0.785
DANN
Houritic
Theuretical error gearmnless
Tick
80
TMV
TMR
TCR
SOR
IWV
DEV
IWA (sums)
TB
0.783 (0004)
0,794 3.0
0,294/
0.704/2240000
0.90:3 2:0 0000
0.752)-203005
0.804 00.008)
0.7521 00011
017971-00013
7966
0906
0.785-00.133
0.7801 008)
+0.000))))
(6000(+00000))
=0.0077
0.8011-00000
0.805
094
0
07700=00000 (015
0.506:-0:015)
00.01)00.005)
0.800-0.0011
0.779 0000))
0.8100.007)
0.8(=00007)
0.700:=0.000
0.500
0.7061 2014)
0,776-00.11)
0.804 004)
00.000
0.787
0.5052-40.000
0.5050=000000
0.8077-0.0000
0.5031
0.7761 (2.22)
0.73
0.811:0.000
0.81R|
018170
0.2192-00.0
0.700(00.222)
0 0.730[=0000
0. 01.05
0.7.322-0001
0.7:70=0.001
0.758--0.1191
0.757
0.654(=0.166)
0.577-+0.001
0.529:=0.00
0.85
0.858:=0.007
0.88(=0.088
0.876
0.741/11000 (:30)
0.7411
0.74/ (063)
0..500...0000 097
0.788 01.21
07700200203
0.75
0.747 0.000)
0.700 (40
(0.011
0.77761..000 013:
0.730 00:0
(1758120.019)
0.7TH[-0.012
0.700 -.002))
O (.877(900033
0.261 90.008)
08801:900007
0.8054 000
0.99 90.00)
(185(-00005)
0.866 -0.03)
0.800 00011
Aug.
0.767(=00012)
0,792(40.000)
0.792 #00.009
0.799:20.0)
0.8(29.01)
0.776
0.778(200015)
00.000000000)
0.8(=0.01)
DSAN
Heurisic
Theureiical error gearmnless
Tusk
S0
TMV
TMR
TCR
SUR
IWV
DEV
IWA (ours)
TB
B
D
9.807
0.6801-0.108)
0 0.805/-0.00 00.014)
0.8002230001
8 -
2000))
0,788/
20000))
0.788/
(00))
7986
1:00 692)
B.773
82775 20.022)
0.784[2:002)
0.797 00.0066
0, 0.796/1::0000
6022
B
K
20012
0,780/ 3
0,789
0.799N
0.796/ -1001)
0.8151:110000
6973
0.786
0316
0.7081 20,501)
0.001:00000
0.866/10000 0003
B
2000)
(0004))
0.780 (000)
-0.0002
2000
697)
0.797
07200200111
0 00:1100000 005)
0.900/1000
00770)
0.904
0.504,2000000
0.501
0.2161
2.0.010))
0.700
0.792[1:0022)
0 (0.08):00000
0.818)200
200011
0.804 200001
00.0110
0.780
0.801:20:025)
0.007)
0.501:00.115)
0,702)
0.7107
00.15
72728 2.000
0.72/2200021
0.707120.015)
0.721|
0.71)(26.02)
0.720
0.743 2:0000
0.745/ 2:0
0,241-200000
724
0.72N
0.725120.0121
0.749
0,T021
0.796(=0.007
0.712
0.709:-0.315)
0,739 012)
0.730):a0.025
60006
0.775/40.006
0.766[-0.000
(.777
0.766
0.720)=0.265)
0.770:
0.84
00.055
0.80 1:00000000)
00,00)
86.66 2..0000000000
DUS54) 20.0000
0.816(1-0004)
9656)
0.657
Ang.
0:760
00002)
000000
0.701 (=0,000)
0.8 (09)
0. 0.763)=0.267)
0 0.705:=0.000)
O. G.700 600)
40

---

Published as a conference paper at ICLR 2023
D.7.3 DETAILED MINIDOMAINNET RESULTS
Table 18: Mean and standard deviation (after +) of target classification accuracy MiniDomainNNt dataset over three different random initialization of model weights and five domain adaptation tasks.
MiniDemainNet
Heuriete
Theeretioal ETrGE guacuntess
Mathod
TMV
(eurs)
0.516
DIRT
0- 0-450 20.022
0.517 4005
0.388 2..000
0.508 440 5066
6.8071800.00 008
0. 0.
0.458 24.053
0.5.9) 110 000
0.520220.000 220.20
DDC
0.510 -0.333)
0.512(4-2 (939)
0.319[44..000
0:311 (40.009)
0.514)--0-0033
0:311 0006
0-516:4-0
0115
0.521244-200 (110)
CMD
0..09:2000 20.01)
0.628C 2..011
0.53214:0.000
0.5321 (8:0 0280
0.581 830.000
0.5221-60).0.3
0.3181 230005)
0.5831 0::01:::0000 0001
(.558):21.024)
MMMDA
18.71
01.88(1100 2011
0-391 (0.0000
9227
0.551:
0.2551-00.003
CIAN
0.5:0: (0.028)
2.5001 1.0.01.) (0127
0.523 10.011
0.525 10 10.008
0.517 1.0.000
0.5151 10.041
0.516 10.004
0.552) 1:0 010
0.5422 01512210.007 (0.007
DANN
11 01220-25 (2.1
0.53U 0110131
013301 0311
2. 0.5081 000)
6.3161 2013
6. 0.3301 033
0.315 (332)
0.641:200 006:
01:3022210 (3.4)
DSAN
0.900 (0.022)
0.554 01.015
0...2(11.00011)
0.5251
0.3151 (1311110..000 $00.018)
0.550
0.5461 007
0.5652 (15))
01207
0.000)
0.220
0.18((900022)
+0.022
0,200880.223
011
0,234/24..02
41

---

Published as a conference paper at ICLR 2023
Table 19: Mean and standard deviation (after =) of target classification accuracy on MiniDomainNNt (Part 1) over 3 repetitions with different random initialization of model weights.
HoMM
Houridiic
Throeetiral orrer guarantres
TMV
SOR
IWV
IWA (ouns)
0.30
(1F7)
(1387(=00.000
0.357| (000
0.200
0.3766 2776030.0022
0.376/20.00
0.373: =0.0005
0.97 00)
0.802: 007)
0.418:=0.0411
0.701
7232 0.03
0.723:20.90)
0.718 20.006
0.70% 20.001)
0.704: 013)
0.705:=0.314)
0000:0
0.721:-0.0033
L50 (K2))
0:30 2:00
0.300 (00)
20.008)
0.9961 .22
0.367: 010:
0.342:20.315)
20,50)
0.377:20.013)
(0..8772..0000 (122)
0,587/
0.580 (40,005)
20000)
0.577:20.000 0040
0.377:=0.004)
0.586/=0.008
0.600:000011)
Ang.
0.50% (20.122))
0. 0..22((20001))
(.024
0.523 (00,000)
0.518(30.011)
0.511(=0..13)
0.531
0.561:200015)
AdsSKM
Heuristic
Throretical crne gurranters
Tack
50
TMV
TMR
PCK
SOK
JWV
DEV
IWA (nurs)
R
(.504(+).011)
(550):00055)
0.56:=00025)
0. 0-506:=00.05)
=.004(=00004)
0.5:9/=0.022)
0.545(=d..com
0. 0.588:
0 (8G -0.04))
R
(.38(200008)
0.573 016)
0.375/:00001)
0.571(=00000)
0. 0.0951=:0.000 =:9.097)
0.294)=0.0221
0. 0.504): 021)
0.3771-6.0133
0.596: 020
P
0.723/ +00.000)
0: 0.02/04:0000 (984)
0.7231:40.000 (0000))
0.719 +00000))))
0.722
(700)-0..041
0.603)226.000
0.721:20.000 005:
0.723) -0.000
0.322) (00007)
0.3530 (065)
0.3332 (000)
0.525) (00000))
0.339[ (-2 6001
0.335) -0 0012
03331 -0.0066
0-353) 10: (05)
0-339 00.00-000000
+0.017)
5..00+000000 (60)
0.588) +1.000 (01)
0.585(+0.000)
0.000))0005))
0.589) -0.007)
0. 0.589(-000 00.00-0
0.587) +0.
(.50) (1:00)000000 +0.000)
Ang.
(0.0088))08)
0.024
0.514/20.00)
0.514/20.00)
0.004(
0.512(20.018)
0.510/20.007
0.027)
DIRT
Heuritii
Thruectical crree guaranless
TMV
TMR
TCR
IWN (ours)
0.55 0115
0.5466/000031)
+OLDI)))
(5.00-00025))
0.541(-00005)
0.577y-0.022 025)
0.597(-00087)
0,365 (411)
0.974:..00000 9037)
0:37 (00))
0.386/20.062)
0,08
0.431:2000455
0.704/ 1031
(186)
0.7USE (126)
007001::0.00)
(15)
0.67()0.0M2)
0.713 03)
0.715
1...aaaa.... 1
0.3500
1224)
0.366(2000005
0382
(ATe) (095)
0.528)
(.520)0900000)
0.5T7E (0.0005
0.52(+00/65)
0.585
0.558
Ang.
0. 0.5230800.087
(000127))
0.BKK (0.0011
0.519:-0.022
Heuritiic
Theuretica urrer guarantes
Task
S0
TMV
TMR
TCR
SOR
IWV
DEV
IWA (ouns)
TR
R C
0.503/
240.044) (440)
D. 0.5656880.0333
0.365/2000030 (LL)
D-352:
20.000)
0.5461 241003))
(0.05))
0.568 =0 025)
0.6861_11.079
1
0.374 2.0 (10)
0.30/ (00)
0.2011 (00))
0.38/20.00 4:
0.299|
(22)
0.37% 240,000)
0.377
241.009)
0.385i 008
0.416:20.014
P
0.7TEN J.0 (025)
0. 0,7111-0000 00
0.7141-9001
01.12(2100006
0.200 30 0014
0.733 240 (000
0.718 200 003)
0.715: (012)
0.7171
0.394)
+0.00))
0.58871=-000
0.327(200000
0.586/2000
0.355:=0.00
0.9371=0.000
0.576
E.0 (22)
0.570(20.00)
[.678(-00008)
0.588(2-9.00 .055
20.000)
0.682)=0.005)
0.570)=00.007
0.585)=000221
Arg.
0.312)200005)
(3310(200007)
0.574(2-0002))
0.310
0.5122000005
0.5161-00009)
CMD
Theoretical ertur guarantees
TMV
TCR
SOR
IWV
DEV
(nans)
0.3934
0..886..0.00 (08))
0.508/
0.500:000000)
0.544/2:00022 021)
(.602(400.08))
0.503:20 009:
-6617:=0.125))
0.5771 (0.0000)
0.365) 40.000 (077
0.305/ 40. (00))
+0.01)
0. 0.395 (99)
0.379: +0.001
0.379 -0 007)
0.3801 -0.0077
0 7070--0000
0. 0.2gri+0 0.. (V55
0.7 +7.10-0000155)
0. 0.253r-0.01)
0.7223++0002)
0. 0:782
0.796 +0.00:)
0.74/-0.00 001
0.751-00.055
0.583/ + (1))
0.3652 +000000 (06)
0. 0.974(i00.000
0.561 056)
0.303/ +0.0610
0.564 =0 007)
0.57%:=0.005
0.57s/ 3.0 (222
0.582 20.01
0.580):200.01) 013
0.589[4. (955
(..015200015)
0.502) 003)
20.000)
Aug
0. 0.50%% +0.02))
0. 0.500((000009)
0.551(++0000))
+0501)+0.000)
0.502
0.54 0.518(+0002))
0-10+0003))
0.533 006)
MMDA
Henrisic
Theoretical ertor guarantees
Task
C
1 0,200 12110
50
66 102200
20 20000
APCL
0.287299000 (0000)
TMR
0.577iaas...0 (1:3)
TCR
SOR
U. 0.544(80003)
IWV
0.546/ (333)
DEV
0.585::00 (00)
DWA (ours)
U. 0.28% 1000 (000
TB
1100000
D- 1EE0 12223
B.ST 11 (R00)
0-376;
(1))
0. 0- (M0)
0. SeM 0310211
3.880 1-0 (0.021)
0-375:
0. 0.11%
(110)
01746-50001 (1140)
0.714) m111
0.717 004)
0.721.
0.723
004)
0.7252
00000
100020 240
.005)
0.5315
0,3551 2:00
0.38
0.362]
0.302 (4001199
R
0.277
0.0788
(300022)
00.083/0000000
0..0ME (N14
0.387(
0.387(
0.0883:50 (00)
0,8005
Ang.
0.
+0.022))
0.5038 00.00)
013023/ 40.013)))
0.519/+00135)
0.517(400188)
0.525(+00025)
0.827 007)
0.5400 (00009))
42

---

Published as a conference paper at ICLR 2023
Table 20: Mean and standard deviation (after =) of target classification accuracy on MiniDomainNNt (Part 2) over 3 repetitions with different random initialization of model weights.
CaDATS
Heuridik
Thronetical urrer guarantres
Task
S0
TMV
TMR
TCR
SOR
IWV
DEV
1WA (ouzs)
TB
0 0.58%) +0.072)
0.585 (225)
0.585/+00002
6077) .985)
+0.05))
0.566;20.001
0.028 058)
0.577(20.120)
0.367/
0.377
0.373
0.307:=000000
0. .8
0.373;20 0413
0.363
0.373:
0.400
=x0.1224)
2..0::00007 0.00)
0.789-20.00
0.784)220.015)
1.7266=00.1..
0.68%
0.680).0.00
0.743::0.001
0.322)
0.356:200.00
0.858:2:0.007
(:24/:00017)
0,384:20
0.345/
0.361/2000022)
(1500(400000)
09.19/49.013)
0.500)-0.00)
+0.013)
0.592/
0.022; +0.0000
0.502/20.002)
0.583/20.018)
200122
0.524/30.048)
0.636/30.015
0.00/30.002)
Deep-Coral
Heuriaic
Threrstical gearantees
Tack
TMV
TMR
PCR
IWV
DEV
IWA Cours)
0. 0.565 +0.045)
O. 00.8800--0000 (V2))
0 0.589/+0.00 (1.00)))
0.567((66.))
(.650()0.005)
0.568(=0:113)
(..02))002))
(-001)0)))))
0. 0.576/ #0.012)
6.977-00.013
0.577
0.368(=0.000 =0.01)
0.59( 685)
(.868(20.005)
0 (.008/00.008)
0.857/=00012)
0.550
P
0.713 (x0
0. 7200 (F5)
0.728 (1))
0.7222 (00)
0.714[=0000))
0.722t (1220220007)
0.738:-00 013)
0. 00733(200012)
0.518 011
0. 1.3852 003)
0.3832 (000
01 30000 .014)
0:3786 11-13)
0 0,38( 11 (0022)
0.3e81 00000
8
0 0.500 +0.00%
0. 6000 1 209)
+0.000)))
0. 0.506:+0..0 (0.005)
0.501((0000))
0.598( 00005)
(000))
0.822 (0.000000000
(0058(-00005)
Aug.
0. 10.00G/000000 0V1)
20000))
0.50(300003)
(..18(-0:003)
0.504)30014)
0.530(=0.00) 01)
0.504(
CDAN
Houridic
Theoretical ereur guarantees
Tack
80
TMV
TMR
TCR
SOR
[WV
DLV
DWA tours)
TB
20000))))
0.2871200.102 (122)
(0007)
1..00011::000 (926)
000000000000
0.602(2000025)
0.074/200.155
0.098
0-3801 200.013
0,350N
(005)
0,359 2000000
0.257/
0.36x
(M15)
0.363/110.00 0.40)
0,21551:00000
0.3377
0.200(20.001)
0.7000 (222)
T1T LA0 0023
0.7700 240000000
0.700
0.721-0:209
0.7237
0.733 00-4)
0.716/160002)
(0:349
0:2000140000
0.304/
0.442/46011
0.5762 260.027
)
0,568(46.03)
(40.066
0..05/3000077
0.007(300000)
0,202(20004))
Heuristic
Thooreical
Task
50
TMV
TVR
TCR
SDR
IWV
DEV
IWA (nurs)
TR
R
C
0. 0.60VL1.0002 0.03))
0.825 1O00/20..IZ
0.557(2:00044 0144)
0.582(100022)
0.007: 005:
(060L(-0,005)
R-1
0.578K 220
0.37(2.0.012)
(.3777:2000113)
0.389/1:0
(022)
0,579 1:0.0000000 036)
11375' 2:0.1333
0.388/20 019
0,4/200125
P
+0.0010
0: 0.9020#90.000 (..)
0.704) (1))))
0.701:50.0011
0:000: +-0 006)
0.600)+-00000 2-0
1.607:+0.0..5
0.721:=0 008
0
0.81=000435
0.5771-00.00 0085
0.377) (00))
281106 800)
0,545)+00 004))
0.551[++0008)
0.36)/+0.048)
0.874::0 005
1.502/00.0055
(..80((000024)
0.50%/ n283
0.50%) (228)
0. 6077:
0.506(40.00)
0.586 +:.066
0.586/40.016)
0.617: 005)
1.607(=0.008
Aug. (0.09))
+0.000)
0.535) +00000))
00000=00006)
0.500(+0013)
0.518(+0005)
+0.057+
0.541::=0 009)
(0.00))
DSAN
Hleurittc
Theoretical errer guarantees
TC3
SCR
DEV
IWA (oums)
0.30% (2022)
0.309(
4.307(40.000)
(371(30)
(181(-0001))
0.900(+000)
6-400[
03602
0.377
1011000.0000
0.7110-02055
90.005)
0719(-00005)
0.710-00005
0.302 100
0.564 20.001
0.3541 0211
0.34/1000000
0.407)20.083)
0372)
0.577 240.000
0.504 2400004
0.5841
0.897(2. 095)
0.504( 20.0000)
0.80(200.04)
0.615:=00016)
0. 5.000 +0.02))
+0.00)
0:551(-0013)
0.5(2((00.018)
0.525(0001;
0.313(+0.002)
0.555(=0.01)
0.5:80
0.577-00015)
43

---

Published as a conference paper at ICLR 2023
D.7.4 DETAILED TIME-SERIES RESULTS
Table 21: Mean and standard deviation (after +) of target classification accuracy on Sleep-EDF (Part
1) over 3 repetitions with different random initialization of model weights.
HoMM
Heurisic
Thoursiica guiranless
Task
SO
TMV
TMR
TCR
SOR
IWV
DEV
IWA (curs)
TB
11
0.563 (0.003)
0 0.307 (G0)
0.507 856)
0.553
0338)
0. 0.72 .02))
0.488(20.09)
(..00)
0:004(200002)
10
ootee 16)
0 0.9711.10001 ()
0.70%;
9. 9.788[+ (02))
0 0.776 (4006)
0.762[+00022)
0.51 ((0001)))
16
0061
0. 7668)+9.000 (.0))
0.7003 +6.02)
0.00% 0.00% +-0.025
0.000[+00011))
0.628(+00117) +001))
0.729(2400025
0.712-40.024)
0.784 (21
0.7844/40 0213
(.67k(=0000))
0. 0.729):
0.734):00006)
0.74;++0.006 0.0000000)))
0.7652
0.741220.001
0.582 (06)
0.832) (065
0.5.29(4:0.0)) .V)
=..022)
0.800(+00025)
0.364(260007)
0.852
0.314
Aug. 0:000(000077)
0.712(+0.030)
0.710//40000) 0:000:00
(:718(+00025)
0.98][# 022)
0.720(+0080)
0.678(+0006))
(.747(=0.018)
0.755(++002))
AdSSKM
Hlearticc
Theoretical error guaruntees
TMV
TMR
1W0x (ours)
(0.00)
ogy)
A0011 (0.00)
0.733 10/001
10.000
0.7077--0
(984)
0.753
0.754:-0
0.007
0005:+000000
0.05:+0.070)
0.736
0.630 (20003)
01.000000000 (002)
0.72 (222)
0.72(20.018)
0.000 0188
0.70852000041
0.707 0.001)
0.733 (.003)
1.8500
018331 240.0000
0.819K (032)
0,70
0.78
0.824/ 03.4
0.7591
0005(+00.000
0.7122 +0.005)
0. 0.712--0.00
6. 0.226(+0.000) 0.712)-000021
0.705(+40.00)
(.600(00.005)
0.722 0 0205
0. 0.720 0.0035
DIRT
Heenrbie
Theoretical guaruntees
Task
50
TMV
TMR
TCR
SOR
IWV
DEV
IWA (ours)
TB
0-+11 11
0.582
(011
0.4002 1015)
0. 2.800) -4) 1010)
0.5009:20 (016)
0. 0.625ta. d01)
U:447(a00
0. (-007 1989
0.54) (43)
0.502 (1,00))
12 ---55
0.712)=0.08
26.015
0 0.885) 23.0015)
20.02)
0.57 019)
0. 0.505/20.061 061
8.785
0.820) 0:00000::00 02)
0.577=9.0159
16
0.5264 240.662)
0. 2377-99.025
0.757) 230.0225 (25))
0.765k20.0221
(6671(30.071)
0.618/40.124 124)
00118(=0.124)
0.746l
18
0. 2.227129.913
0.7771 (....
0.275k-200005
8.717/36.00 844)
0.74(600002) 099)
0.715|20.072)
0.702:
(0002)
14
0. (3.0020
20.005)
0.2822) (...00
0. 0.9711.20.0133
0.829 (207
0.785(400071)
0.868/ 012)
0.874[
Ang.
(0.000)
0.745 20,000)
0.745(210.000
0.709)
0.742/20.030
20000)
(.096(00.00))
0.740 021)
0.75(2000223
DDC
Hruristic
Theortical armir puaranters
Tack
tours)
0.366200.099
0.iik 21. (29)
0.606(=0.007)
0,658(+00
0.645 006)
0.603:=0005)
12
20.071
0.7511
0.75k)=0.31)
=0.00)
7.709
0.716 (0.06)
0.700)200006)
0.755
0.788/#0.015)
0.5621 2000001
2.689129.0133
0.683
0.0751200005
0.06(
0.0231 0.00)))0.22)
0.600:10
0.725::00015)
0.000 (0000))
0. 7899 2015
0.7890 (0.0000000
0.71(=0005))
0674(-00.05)
0.724[00000)
0.712()0000)
0.733 (-0.011
0.715 (155)
0. 000000)
0.8g(+9..04)
(181(+0.00))
0.802(000025)
(800(-00014)
0.784[+001]
0.788(+0006))
0.825(+0.00
00.02)++00111)
Aug.
0.646(20.00)
0. 7.20))
0.721x20.021)
(.712(=))011)
0.000(50)025)
0.004)20.083)
0.060(200134)
0.734(=0.08L)
0.737((40.134)
CMD
Heuritte
Throretical erree
Tack
S0
TMV
TMK
TCR
SUR
TWV
DEV
TWA (ouns)
FB
0-+11 11
-0.0111
0.5611-9-013
0.561t-0.013
0.55):-000077
G. 6.653/+-0.0000 0-009)))
D. 0.607 (87)
D. 0.505 +0.000)
0.583:-0.003)
-0.077
12
0.750
(00000
0. 0.8631299.00 20.2
0 0.800220.0003 (0.2)
0. 0.F8N (16)
8.70s/ (.0.0.6)
0.784/210.00
0.8299 210 (44)
0.793/ 01.4)
0.8351 10
16
0.357 (40 (010)
0. 0.7277.9..0 211
0 74777:-4 (1.1)
27000
0.804) 3.0 01.85
0.878//200 (U0))
0,005 240 018)
0.68K 01.5)
0. 0.7.99
0.7102
(17)
0. a any
0.0000 3.0 (110)
0.741:: 0015)
0.7725 ax3 0225
0.708/
a1 (03)
6.75 (100
0/744
0.800 (08)
0085)160 (0.000)
0.808;000000 ()
007077(600001
0.779/++0061)
0.673)
+4.22900 2280
0.841):-0 000)
0,001 (0.000)
Avg.
0.700 20000
0.700(200.012)
0.770)2000119
0.718]500038)
0.729:30.00
0.7377
MMDD
Heuristic
Theorctica creur guarantees
Taok
TMV
TCR
SOR
IWV
DEV
IWA (oun)
0.5331 20.002)
24000)
0,521 20.
0.
200008)
0.006 (110009))
0.612 037)
0.616/
0.749%
0.7991244.00
0.797
0.7% 025)
0.708/ (:11)
0.747
0.793(10.01)
0.544 11.9
2.7601 9...0.0.5 0155
0.705
0.T7
0.645[
0.703
006951-0000
0.70200.014
0.739
0.678 26.046
0.9
0.7756=0.313
(.7702:40000
0.68(=40.001)
0. 6.720 =1.000
0. 0.772 005)
0,707
(742)20.07))
0.84(200.000 0209)
0.94/360.020) (29)
(=8.077
0.736)::0.014
0. 0.792) -0.073)
(..00))
0.85(10.012)
0.810(=00012)
Ang.
0,000
0. 7377
0.72772400010))
0.714(20.025)
0. 0,704[2:007]
0,00(=00,003)
0.745
0,762(4-0025)
44

---

Published as a conference paper at ICLR 2023
Table 22: Mean and standard deviation (after =) of target classification accuracy on Sleep-EDF (Part 2) over 3 repetitions with different random initialization of model weights.
CaDATS
Ho
Throretiral arree guariniees
Task
50
TMV
TMR
TCR
SOR
IWV
DEV
1WA (our)
TB
0.508
0.565/20:111 119)
1.565=20110 118)
=00 107)
0.504:00.088)
0.002:-0
027)
0.082)
0:812: 26.0144
0.812
0005; 6)
0.827( 0.012)
0.77
00.28(=0.018)
0.509/-0 007
07.00-00013)
1.574/20.0000
0.792)
0.720)
0.642; (6422240.012)
6.633/40.006)
0v076(=0.001]
0.720:=0
004
00.00
(00000)
0.74 -10
0.209:20.021
0.715:44.0033
0.724/460008
0.177/00.03
0.0.2)
0.771(=0022)
0.779
0.807)-10.013))
0.800:-0.000)
0.800(10.00)
0.700(-0.000)
0.84) 000044
0.602:20.000)
2.90977
0.730 277.000000077
0.7061)200001
(.703)
0.688/300056
0.60:300031)
0.744/=0.022
0.748(20088)
Deep-Corul
gurrantees
Tock
S0
IMV
TMK
TUR
SOR
JWV
DBV
JWA
0.3:7(-0.022)
0.5856 -0.028)
0 0.585
0.6:+00009)
0-650( (65)))))))
0 0.620 063)
0.969(=0.11)
0.500:00028)
12
20.00)
0.75x=6.02)
0,75(=0.12)
(.746(=0.016)
0. ( alls)
0.758
0. 0.768/ 20:88:=0.00 036)
0.766/26.315)
0. 0.778)-0.08)
16
0.6 0135)
(13))
0.544 (077(=200077)
(U00) 015)
0.553
0.6622 20.017:
0 07288 (0:00)
18
notte 10 (00002)
0. 2.700a8 (22)
(173) (0222)
0.71 0.7:665:002
0.720 00.077
(20.0000
0. 7.73 01.03
0 07411-00015)
0 0.730 +0-017)
0. 0.3066 9 9.00
0 8999 1880000.000
0.876(+00022
0.790
0.784(00.445)
0.82) (0.02)
0 07700-00011
Ang.
0. 0.777... 10.00)
0.717(-40.024)
(.712(50001
0.7(300031)
00000200.000
0.713)
0 (3S) (0:01)
CDAN
Heuristic
Theoreiccal erree guarsnless
Tack
50
TMV
TMR
TCR
SUR
IWV
DEV
IWih (ours)
11
20.002
(0.000)
0,499
0.5077
0.525(10.00
0.342
0.505/=00 0.007)
0.500 (.500)=0.003)
12
0.008) 1011
0.8221220.00
0.844120000
0.84/220.007 007)
240000)
0.67(=20.000)
0.805/ 0.0002
0..01.20.010)
0.019 (07077
0.7.2 24
0..22
0.706:-0000
0.741
700 (00e))
0/704/
0.773)
0.700
0.8651
0323
00.00)
0.8521
10,00000000)
0.708 (400027)
0. 200.0077
7000
(.007(300.00) (37125(30.00)
0.748: 02055
0.730 (0.0000000000
Heuristic
Thouettical
Task
S0
TMV
TMR
TCR
SOR
IWV
DEV
IWA (oun<)
TB
0.5521
20.018)
0.6022 240,05)
0.6022 210008))
0.602 (20.002)
0.672 (.00))
0.904/ 20.00))
0.000/20.114)
0.622/20.04)
0.574/
12
0.643) 20.075)
0. 0. 01.)
0 20.04))6.014)
0.708(20.015) (.702)210004)
0.656 (0006))
0.806 20005)
0.784:20.02)
0.S1D/:20.005))
16
0.67( 1360
0. 0.9776-55 2-9.0.00
0.676 20..000.02 (012
-0.000)
(0065
0u51[=00055)
0661:-0.000
0.747] (.9)
18
20.077
0. 2 0.000)
0. 0.725/-00.00 60.00)
(-725(-00001)
=0.01)
0. 700 -0004)
0.708 =00054)
0.752
0.78%/=00008
0. 0..6
26.000)
0.514;-0.0002 00.0))
0.816[=000015
0.888 0.012)
0. -0,099)
0.822(40.01)
0.81[
Aug.
06C1(=00067)
0.723) -0.000) 9..0))
0.721[=002)
0.714-00013)
(.057)
0.010
0.724(-0.018)
0.73%(+0.028)
DSAN
Heertki
Theoretcall erroe guaruntees
TMV
IWA (cur)))
2)
0311(=0.000)
(0.04)
(0.000)
0.8gri-0
0.832|-0
0.000
16617(40.116)
0017(-0.118)
0.084
0.704 22.0
0.7344
0.002)
0.71[1:00077
0.764
0.708
0,822 20,04)
0.8522
200.00)
0.813/
0.16
0.828
0.033 -0.010)
0. 0.32(+ 9.00))
0.76((60003)
00750)-0.007)
0.7785 (0.03))
0.712(+0.053)
-0.00))
0.757(-0.021)
0.758(00.000
45

---

Published as a conference paper at ICLR 2023
Table 23; Mean and standard deviation (after =) of target classification accuracy on UCI-HAR (Part 1) over 3 repetitions with different random initialization of model weights.
HoMM
Hourisic
Theoretical urner guarantres
TMV
JWV
(ouns)
0.873(200012)
0.R6T (33))
4.807:400000)
(1
20.7011
0.90
100:
1.0:=0.0)
0.854] (366)
0.500:20000
DUB00
0.585
0.854 20. (066
00.000000005
0.903:20.012)
0.508/20.0)
(.889) 1:0
0.911 024)
0.301-00224)
(-91(200024)
0.585 20.002)
0.88%
0.01(20.024]
0.944)200024)
0,49001000002
105)
0.618/40.100)
0.314/ 188)
0.630:200 043)
0.632)
Ang.
(:782)2001126)
0.328 023)
0.818(20008)
20,077
0 20,002)
0.5(20.043)
0.003)
0.822)20:012)
Ad+SKM
Heuritic
Throretical errer guamnoless
Tack
50
TMV
TMR
TUK
SUK
IWV
DEV
PWA Couns
TB
12-4 16
=: 105)
0.681/+00012)
0688)-000022)
(.888;-000012)
0-299(++0 (21)
0.59:+00 103)
0.50 104)
0.008(20.0)
0.722(-0.013)
2-411 11
(..00=-00000 0181
1.00+0.00 00
1.0000.0)
1.0/00.0) D)
0.864 066;
0.800)=00.3
0.017;=0.1188)
1.0:-000 0)
1.00=0.0)
25
0.820) 0-0
0.896
0.890
0.800
0.588 3.0 4..012)
0.526/ (50.054)
0.506:20.084)
0.806/20.0)
0.806(
0. 1331
09990)
0.3009; 1-0 (30)
0.833/ (0000))
0.853) (010
0.803) -0 0224)
0990-0000 0..01
S59[++ 1513
0.505)+0007))
0.514)/000070 (779)
0.5566;+00000 0.084)
0.483) +0.064)
0.589M(H6
0.389)
0.51 024)
0.507(-0.005)
Ang.
07720
(.8(=0.023)
0.8(+00022)
0.92(100028)
0.768( (10.04)
(.007(46
0.711;=0.008)
0.8(20.01)
DIRT
Heuristc
Theorctical crror guurantees
Task
TMK
DCR
SUR
(ours)
0.057
0.778(00013)
0.75/40.00
0.796/+00012
(8677-00.005)
0.007
0-764/-0 082)
1.00:000
1.00-2001
1.0(1:000
0.854/-001177)
1.00 20,,00
1.00==100)
1.0020.0)
1.00
0U00E (84))
0.008..00000
U 20.00
(1847(
0.847(200.0)
0.847(2:0006)
0.933(20.00
0.928 2-0.0)
1USEE 2000)
0.908 2000)
(1920:00.00
U.85E 90 (006)
0.958(00.0)
U.5ta/ (10
0.50 0.50000000000 094)
0.000 (1-00
0. (-861(-0.12))
0. 0.900((00055)
0.017)/90007)
0 06667 (10 2215
010677 (0-3955))
0.841-0.000 0.048)
0.080[-0001 (1000))
Ang.
0. 0.907 (0007)
027)
(2002)))))
0.750 (400877
0.807 (30,07M)
0.90000016)
DDC
Houriticc
Theuretical erree guarantess
Tisk
SO
TMV
TMR
TCR
SOR
IWV
DEV
IWA (oun)
TB
12 10 16
(.041
0.692 028)
(.001220012)
0.681|
0.7161290033
1.010000
1.0166.00
1.001:000
1.00
0.74 213)
0.74/860214
0.802) -20 266)
1.0)
1.00=0001
25
0
8A0 1:0
0. 8000::00000 07
0.300 (0)
0. 1..00/10.00
22.22.0.013
B 0.0001
0.000
0,666
0.808
0
(1868 0122
0.971=0.099
0.508
0.001-000013
0.908:
024)
0.917(-00221)
0,562 6:0 001)
0.662(+00000)
0.110
0.085( 0.16)
0.5/60.127)
0.542) 091
0.00(=00118)
Avg.
0770520015)
(.007
0.734/20.077
0 0.8041=0 035)
0.922/200007
CMD
Theorctical error guarantees
Taok
TMV
TMR
DEV
IWA
0.000200121
0.929:20000)
20.0)
(.70222000122
0,658
0.604| =n.012)
0.604(20002)
0.720/20.0
0.701:00.06)
00:0000
1.0
0.0
0.7716 +0.012+
1.0040.00
(Cl))
0.111 +0:373)
0.819) (0192-0.977)))
1.00 1.00-0.0)
1.00+0000
23
0. 0.800(+00000
0 0.936--0.00 0.0"
(9.55-000))
0.888 -0.00.)
0.963t +0,633)
0.800((000042)
0.935K
0.905(+0.0)
13
0. (2800--0 075)
0. 0.9.0.000.00
0.95%:00.0)
0.955(=0000)
0.836 +0.675)
0.80((00.733)
0.9.8(=0.00)
0.960/200023 012)
18
0.321
0.0399 063)
0.785 (7755120..100 20.146)
0..13))0.163)
Aug.
0. 0.788(+0004)
0.6.R(+6.000) (099
0.610((-0000))
0.751(-0.000)
0.801 +0. 10.)
0812(=0.113)
-0.02()
(0003(-00025)
MMDA
Hearistic
Theoretical ertor guarantees
Task
50
TMV
TMR
TCR
SOR
IWV
DEV
IWA (ouzs)
TB
12
0 0.070800012)
0.00g
(29)
0.8900200012)
0.081: (11)
0.87 4(12022
0 (112
0888 (40.0)
U. 0.7077 (M1))
11
00+000
0006 (0)
100
0.541/00 125)
0 (08))
0. 0.807( (077)+000000 085)
1.010000
1-00 (001100
23
0.847 1:0
0.50001.0.00 0.00
0,500 20.0))
=1.0)
0.547) 0077
0.947)
0.847 =:0.02)
0.896/10.00
08822 (:0
000000 1-00 0122
0,000
1:0
0.026/10 054)
(0883)2:00117
0.582
0.903/200 024)
0321110007
0, 0. 1099
0,360
0.3(20.004)
0.301-00000)
0.533/1-0000 064)
0.007((000020)
0,08%) 240 (77))
Ang
(885)
01992(+00033))
0.750 +0.05)
(.775(+0013)
0.007(+00035)
0.807) 023)
0080/+00050)
46

---

Published as a conference paper at ICLR 2023
Table 24: Mean and standard deviation (after =)) of target classification accuracy on UCI-HAR (Part 2) over 3 repetitions with different random initialization of model weights.
CaDATS
Heuridii
Throrttiel errur guaranties
Task
SO
TMV
TMR
TCR
SOR
IWV
DEV
19A (ous)
TB
(#: 024)
0.701=00020)
0.689/20.0) 0)
(0004(=00021)
0.694:=0.011
0.706:-0.021)
0:000000
1.0
1.0]=0001
1.0/::00)
0
0.86))50.3
0.866(=0111
1.0:00.0
+00:0000)
0.541 082)
0.889]20
0.889)
0.8(200055)
0.924:20.00 ...22)
20777
0.52212:0
(2002))
-00122
(=0000)))
(.70:502217)
0.938:10.0
0.025 0007))
(.025(+00000)
0.6330
0.701(10.12) (0055)))
0.7641(10:12
06:1)
0.76))0000)
0.892(26 022)
(.802(200020)
0.802(20.017)
0,780/201108)
(10060)
0.790/20.110)
0.846:20 028)
(.585/20.018)
Deep Neep-Coral
Hearieic
Throretical grarantees
Tack
TMV
TMR
TCR
IWV
DEV
IWA (ours)
12-+ 16
(-00))
(1-8 018)
(.081()0012)
(0000())))2)
(-0011))))))
+0.00+
0.688
0.058(=000)
0.000/2000554 054)
L. .)
100/0000
1.0(=000)
0,870 (070):00000
0.875 (775/00.85
0.585 26.0000
1.0:00.0)
1.00=0.0))
25
0,8200 000011
0.822
0.822220012)
8.88t) (001))
6 30600.0.00
0.0966(1.00)
0.903 (453)
15
0 97 d 0099
0. 0.91
0. 0.91 0002))
0. (2900(2000021
U.91 (09070
0.917 4- 036)
0.9311 01:2)
0:500(+0009)
01528(+0003)
0.508(+0003)
0. 0..69(1-6 175)
0.542((06148) (-507(()))79)
0.50 (52)
0.559 +0 (0.05-)
0.50/ 221
Aug.
0.70(200053)
(18(200)))
0. 808 (0g)
0.771:
0.708(800072)
0.773(230.008
O. 0.808 2:0 200.088)
0.522 (100000000000 (05)
CDAN
Houristi
Thoorctical crror guaraulees
Tack
S0
TMV
TMR
TCR
SUK
IWV
DEV
IWA (ouns)
T1
12
0.674
0.702 (22)
20002)
0.722 2:00012)
200012200012
0.674 20.244
0.640120:355)
0.715/20.022
0.729)
11
0.501
1.012:000
1.01::00)
L:0:+000)
0,8622 20.0000)
0.782
0.902
1.00::0000
0.832(200002
0.0891 024)
0.22
0.8471 20.0002
0.708
0.924:
0001
01901120000
0.811:20
0,438)
(6.06
0.604:20.1 126)
00.077
0.630/=0.049)
0.729/:00008)
200000
0.869(20 029)
0.843(200022
0184200022)
0.802 200.94))
0.6871 167)
0.80620.022)
Heuristic
Thooectical
Task
S0
TMV
TMR
TCR
SOR
IWV
DEV
IWA (eurs)
TB
12 16
0.088)
0,701 024)
(2701(200024)
(27014200014)
0.71912-9003)
0.0012000122)
(2701
0.729
3.722
11
100:0000
1.0110000 0)
1.0/2000)
1.0(2:000)
0.729:20.34.1
0,761 1481))
1.0:20.0)
1.00)000))
23
0. 020
0.017(+00021
0.831 (- (18)
0.800 (1)22)
0 0.8681 (0.04)
0 (-808)))002)
09911::0.112)
13
0.800 127)
0.5121 (1))
0.400-0012)
0.910==0012)
00881-00221)
(0001=0.004
0.944:00.112)
0.065/
18
0.403/50012 12)
0122
0.705)=00012)
0.701[=00067)
0. 0.78915:0.127)
0.689)
0.588:+0.113
0.639
0.674)000113 120)
Aug.
0.750(+0008)
0.600 024)
0.850(+0014)
00050(-00023)
0.8/46.060)
(.763()00004)
0.78[=0000)
0.810=00.025
ISAN
Hearistc
Theoretical error guarantees
TMV
IWA (ouns)
9.715 m%))
0.681((60002))
0.306 200)
0.715 0244
0.300[000000
1.0(+00)
0.952-00.00
0.917
0.817(-0005)
0.785/ 10g)
0-785--0.103
0.931 012:
0.008/+000
0 (5850110000 mo6)
0.552 022)
0.91
00011200022
0.554
0.854/
0.854:20.006)
0.851/ 012)
0.005
0,509/100022
0,6300 0793
0.0000100000
095)
0,500/
0.328/20.0077
0.604: 032)
(0.007
0. 0.052(
0.815(+0002))
0.815(+0002))
0.85g(t0099) 98)
(.700)
H.775
0.751;-0.087)
0.858/-0.016
0.875(-00022)
47

---

Published as a conference paper at ICLR 2023
Table 25: Mean and standard deviation (after =) of target classification accuracy on HHAR (Part 1) over 3 repetitions with different random initialization of model weights.
HoMM
Heuridtic
Thronetical error guaraniess
TMR
TCR
IWA (ouzs)
(411)
(10011)
0,560 200.0-00.....
0.786: 169;
D.7.0
0.785 106
0.879:
0.014: 01)
0.528/
0.457)
0.457(200.007
0.461:20.0077
6.489 (.0.051)
0.407
0.516
0.463,
0.548 20.07)
0.701 (:0
0,818
0.818/1:0.000
0.816:20.007)
0.512/40.0:4
DUS03)
0.818:100
0.5331
(4:6)(10000
0,914(2:0.007
0.798/300.18
(.853220.00)
0,013/20.000
0.34
Ang.
e. 0.759 (.700(200007))
009)
0, (.7.0(200.000)
0.7(201103)
0.72(39.64))
0.769/20.000 009)
0. 1.793X (0.000)
Ad+SKM
Heurieti
Throretical orror
Tack
30
TMV
TMK
TUR
SUR
IWV
DEV
TWA (ours)
VM
0:601/+00005) 085
0.72 0.728 +0.000))
0.722 -0.00000000 (010)
e. 7
(6220:00.00)
+0.00
0.600+20.00
0.718/=0 009)
0-781/+0.003 01.5)
0. 0.821 0463
0.549:-000002 (022)
0.640K-60.00 10000-00.000 00.)
0i 0.53: #5.065
14004200.005
D. 0.506 440.02000 0266
0.3022 2300226..00 (226)
0.867 000000
0.838 20.008
0.463(46.077
0.473) [3.0031
0.4844 (0 08)
6 0.621(2::0.0 (62)
440.677
0.4977 067)
0.53 0.574 07744-0011 11.)
0.79C 0011)
0066 0. 0.500
0.7880
8000 (0.5)))
(0.0075
0.81 (:10 0012)
0.51 10.001
0.902(+00 +00
+0.0113)
0 1.000(00.003
0.8955 9.000)
0.5059 +0.064
0.85++00000
0.85/+00055)
0.8844 005)
1.2000 +0.00-)
Ang.
0444
0.7.3)20.0133)
0.722240.003
0..20)
0.78(40.027)
0.725(
0.762 007)
(0.00)
DIRT
Heuristic
Theoectical crror guurnatees
TMR
IWV
IWA Cours)
0.008/+0 +111
(40
0.6291(00085)
(.000=00011)
0.008:+0011)
0.779/00.011
O.TAR
0.008) (000))
0,3088
0.706
0.008(00.0)
0.942) (0000)))
0.501
0,822 090
0.5223
0.85813.0
0.54K (.009)
0.548,10.000 04499)
0.500:000255
0,
DUMUTERI
0.SAKK
0. (87)
0.888
(,913
BANN (1-6
0.086(H00000)
0.0880(44000)
0.000-000001
0880(+0000
0.678(+00074
0.986(-0
0,080
Ang.
30038(30.003
D. (270))00003)
0.708(30000)
0,70K(80004
0 (09)
O. 0.803E
DDC
Heuristic
Thouretical crror guarantees
Tisk
50
TMV
TMR
TCR
SOR
IWV
DEV
IWA (surs)
TB
0,515 083)
0,890166-0277
0.0721200000)
(1102))
8.6:200084)
0. 0.697
0.033)
0.611
0.875/1000144 014)
0,888/1.0000 007)
0.89] 02.)
0.880
0.856
0.806)
0.582 20.017
1000000
0.407 016
8.457L40.003
0.2971
6281
0. 0.438 20044)
0. 0.618
0.439/
041)
0,500 1:0000000000
0.815/400000
0:805/100014)
0.905/400000
0.507) 016)
0.
0811=00021
0.818
0.523
0:00
0.008
0.91/(200011
0.772)
0.702[1:0002]
0.905
0.885
Avg.
6.7
0.2556000000
0770(500018)
0.748)2000155
07777500000
0.700(200047)
0.748
0.708)830030
CMD
Theoretical errie guarantees
Taok
TMV
TMR
SOR
DEV
IWA
TR
0.703
0.648)110006)
0.648(100006)
0.679 (52)
D.672 (672(220.033
0.743 021)
0.003/20.0022
0.724
0.961(+ H0
0.913 (N99)
09913--0.00
0.300 -0.00000000 10.1
0 0.80 00.05)
0.S43 +0. 156)
0.9077 (-0 006)
0.525/4460005 +0.005
0.573)++0 00188
(+0001))))1))
(+0001+0011)
0::90/+00022)
0.51(
0..66)) (66)
0.5777 (0 011)
0 0.567:+0.000
(C.62(+C.00))
0.503 +0.000)
0.769:+-0.000 0163
0. 0.60)
0.8166=0000
03316:20.0)
-0.000
0.790 40. 010)
0.758% (68)
0.811:-00011)
0.8221 -0.013
0,906 064)
0 6.990 026)
0,50
(1879)200005
0.854) (10 (65)
0.939 1.0.00)
(366022))
Aug:
0. (.764(40006)
0. (176:(+
0. (777))
0.757 (000066
0.775/ +0 02.028)
0.643/+0.075)
0.708(+0.008)
0 0.507( (+0.015)
MMDA
Heuristic
Thesretical error guorantees
Task
50
TMV
TMR
TCR
SOR
IWV
DEV
IWA cours)
TB
6. T.O 042
0. 20 M211
G. (089))31003)
0.720(20001))
0.746 (000)
6. 1.700/
0.90 199
0-910-0
0.
6. D
0.7.07
0.8077
(01)
0,002/
0000)
(.U811--0000 000)
0.682)
0.8151
0.40012-0227
0.543
0,498/
0,002)
0.9622
0.543)
0.754)
0.9021
0.000)10.0115
0.539)
0.0021
0000)
0.80571000055
0.925)
000
Axg:
(24)
0.78(-00007)
0:000(00005)
0/731((00081)
0.78(+00000)
0.805((00097)
48

---

Published as a conference paper at ICLR 2023
Table 26: Mean and standard deviation (after +) of target classification accuracy on HHAR (Part 2) over repetitions with different random initialization of model weights.
CaDATS
Heurisic
Theorticaa rrvrr georantres
Task
50
TMV
TMR
TCR
SOR
IWV
DEV
IWA (ours)
TB
00,00))
00067
000))
0.:09/=000000)
0.588(#00 122)
0:000 (0c)
0.785/+0.011 0.117
6.770/
0.906)=0.00
0.9)6
0.905/=00.0)
0.688)
0.800;00.00)
0.800;+0.000)
0.939
0.047(+00000)
6:027)
0.472
6.478/20.012 012)
0.478/40.011
0.5341 002)
0.555(0.0000)
0.472]=0.006)
0.028)+0004)
0.788)
0.872
0:00912:00018
(.97(an.005)
0.789
0.789/48.001
0.90:
(H001(+
(100001))))))
0.501(+6.1;
(.81((0007))
0.810(00077)
0.975(-0.00-)
(1072((-0000))
0.70(200063)
0.772)400000)
0.772(20.000)
0.999
0.730
(.780)400066)
0.002
(.0004))
Deep-Corul
Heuristic
poorantres
Taakk
UMV
TMR
SDR
IWV
DEV
0-466
0.002
0266)
0.731:400008)
0.751:20.128)
7000::00 (981)
6,632) +0000)
0.70 (0.38)
0.7(+60.088
0.728(-0.019)
0.705(+00020)
0.862 -0018)
0:805;40.000)
0.503/2000000 20.000
0,822
104)
0.8641 #0.0177
0.864(=0001)
0.580[=0.000 009)
(0001)000077)
0.300 0.500
1.4547.40.00
0.4541 (1541-00.00)
0. 0.457) (40 (225
0. (2.0 057)
0.40120.0122
0.505/20.117)
7989 (006)
0.801:3000000 0.M01:000000)
(00)
9623 9.0 (NW5)
6. 6.7905400924)
0.700
0 0.00030016)
1.00
0.522(+0001)
4 5
0.952(-00000)
0.955) +0.025)
0 9026 9.0.000000000 (..00)))
6.6465++0 (06)
(900()))27)
0.857(+00885)
(330
0 0-90(+001)
Avg.
0.760:39.0077 0.702)500017)
01.)
1000 016)
(.754580:33)
0.708E80009)
0.764
0 007001300306
CDAN
Heuristic
Thoorctical crrur guoranlees
Tack
50
TMV
TMR
FCR
SOK
IWV
DEV
IWA (ours)
T15
0.71711100000 000)
0.68911:000000
0.481(2:0000)
0.09911: O22)
20.03)
0.76:10.016)
0,T18)1:00000
0.742/240.0
0.502
0.901211-00000 0000)
0:302/240011
0.000/
0.9422
0.44::0000 (..013))
0.503/20.00
0,5544
0.222/240000)
0.322/.440000
0.001:000091
0.39( 697)
0.561:00.008
B.ST.
0.0011000001
0
(.978(200000))
0.519(166.00
0.870))00007)
0.876(400087)
0.98(=0.0)
0,062) 000))
0, (000022 (102
0.775(400051)
0.816 (31)
(1:00000000)
Heurisic
Tharrtical
Task
S0
TMV
TMR
TCR
SQR
TWN
DEV
TWA (ouns)
TB
0,704/200005 057)
0,4R0)LL DOG)
0,6R0/2 (DG)
(0490220000))
G. 0.085 (022)
(7244200016))
0. 0.724;20.016)
6.722/100077 071)
0.711
0.808/40 0303
2.01120 602)
0.34 002)
0,508 200000
0.568/200 140.140
0,5291 2:0 (04))
0.501)2200000
0.936: 1:0 002)
0.500
0.5011:4- 0233
D. 0.885:40.0-11 1:
6 0.5572 (61)
0.618) +-0 (22)
0.5022400022)
BA35
0.300X (6.02)
0 9.9661 0.9661666007 01.)
0.066(+-0011
0.506/:40.0:6)
0.523/20..0 60.000
0.706/ 0.3
0.706/+0.03)
0. (.844)+00000))
0.063/440.00)
0.846(260 0223
0. 0.96((00.0)
0.98/+0.0)
0.96(=-0.0)
0.779 138)
0.922)+00077)
0.922((:0007)
0.98/1:0.0)
0.98(
Aug.
0.707(+00027)
0.27 (+66000) (06)
(.7.0(++0000)
+0.03)
0.722(=001)
0.709(+0.031)
0.708;+00031)
0..10+00011)
0,50+00000)
DSAN
Heuristic
Theoretical ertor guarantees
Task
TMV
TCR
IWA (mues)
(-0.077
0:471890
0L800C-0
9.00)
0.49(-0005)
0.586,
0.790Y -0,061
(.999)
0.977 200000))
20.70200001)
0.853 2000007
0,630 454)
0.971 006)
20.00)
0.9812.0.000
0,08/ 40.000
20.005)
20.000)
0,471 438)
0,588 A00,000
Avg
(--0000))
(-0.00)
11797=00000
0.909( 9. 9.097)
0.721(+0.00
(-025))
0.500/
0.825/+0.006)
0.881(+0.000)
49

---

Published as a conference paper at ICLR 2023
Table 27: Mean and standard deviation (after =) of target classification accuracy on WISDM (Part 1) over 3 repetitions with different random initialization of model weights.
HcMM
Heuristic
Thrortical errur guarantees
TMR
IWA (ouns)
0.744
0 7383
010)
0.8341
9.01
0.568/20.08
0,814/20.011)
0.603
0.579) 1:0
0.77 014)
0.77(a0.024)
0.7221 20.00
0.579 1.579/40.050
0,579
0.746:10.027
0.77(4:0014)
1:0
0.8611200 0522)
0.8644
=10.005)
0.882(1.0.00 0.017)
0,889 110.000000000
0,822
0,504/
0.713 086)
0.5471 20,000)
0.721 (4.0.02))
0.713/ 066
(.015
0.5471-0.99
(20.002
Ang.
(2768)2000022)
0.70812000225)
0. 7.7391-22 (0.025)
0.720
0. 0.770(20.036)
0. 0.763
(.74()))653)
0.725(20.019)
0. 0.793(20.002)
AdoSKM
Henrisic
Thesneicca temor guoranters
Tack
50
TMV
TMK
TCK
SOR
IWV
DEV
IWA (ours)
T15
18 - 23
0.74
0.74
0.748
0. 0.207] a23;
01 0.711(00011)
0.764(-0.019
0
20
30
0.872
029)
0.685 d: (95)
0.886
0.872)=0.000
0.822=0.0299
0 0.872;
0
0.800
35
31
(.610)1-0044
0.608 2- 029)
0.698)2-0.027
-..08)
06.762200065)
0.619)-0.0411
0:600)200031
0.608
0.714]=00022
0.818(10 131)
0.2000
0000)))
018531 (0.62))
0888( 131)
08888+00 131)
0.844 (0009)
0.200)
0.0800 +00858)
0.080(+00006)
0. 0-201 (+-6 (09)
0.500(-00003)
0.717(+0005)
0.717(+005))
0.717(-0.019)
0.730(+0.000)
Ang.
0.747
0.782 007)
(1781
0.779)
0.742)
0.747-000054
0.747(20.054)
0.707)
(0000)))00048)
DIRT
Heuristic
Thooretical ercur guarantees
Yack
TMK
DCR
SUR
IWV
DBV
IWA tours)
0.71
01755
0.511:+0.000
0.555=00.0
0.744(+00019)
20
0,872) 1:0
0.800L20.0)
(U8461 =01.0)
6.899(11 (1)
0.872 011)
0,850
0.50(20011)
200013
0. 6999 0335
0. 0.72 (006)
0.T46(220000 (206)
0,6055 169)
0.492 (300000)
0.77 069))
0.746(3500014
0.77S/ (U0
0.877/30000 0.865
0.859:4.0
0,570
0.879:50 026)
U.SAK 00000)
0, 7300 057)
0.500(H0001))
+.004(+0.06))
0.736]+00059
0.907: 108)
0.00)
(0.00
Ang.
0, 00700(2000000
0. (.7 (22)
0.797(2800027
(.707(30.00)
0,797:
0. 0.798(10.011
0,830 (30003))
Heuristic
Theuretical errur guarantees
Tisk
SO
TMV
TMR
TCR
SOR
DWV
DEV
IWA (muns)
TR
18
1 23
0.7077
0,78R12000083
0,7331
0-367 L: 003)
0.700)000)
0.71
0.71280067)
0.744:20.010 0100
0.787
20
30
0.833/ 022))
0. 9636:.0.000 .F
0.3651240.00 ())
0,8501 (01011
0 200011)
1LS18/100022
0,833 822)
0.84 10.011
0.580 (0..000)
35-
31
0000)
0.722
0.722/ (6)
8. 7.726
L:0 (M5)
0.1055 (100)00)
0.54(10003)
0.746 1:0 027)
0.860(+00011)
0.831 (26 022)
0.850) 0.86(+-0011)
=00044)))))))
0.300:00.0229
00804)20.017)
0.748[1:00000)
0.002
0.002(=00080)
0.200 029)
=00077)))
0.748 (60.005
0.745(50.000)
0.711;200001
0.768/200008)
Avg.
0/741(20047)
0.787(200000
0.707(00055)
0,741(400447
0.741(301947)
01782
00003(2000011
CMD
Theoretical ervor guarantees
Taok
TMV
TMR
TCR
SOR
IWV
DEV
IWA (surs)
0,711 038)
019)
0,644 0.678(2200019)
0.689) 2.0.007
0.022(=00000)
0.7:100.0)
1.786:
20
0.8 +0011
0.850 +0 0227)
0.55M(+-0 (22))
0. 0.697 (.05)
(188(-001611
0 (883) 0011))
0.859(-0011)
(.904(+9..)
0.91 (911-0000 (000))
35
0 (.08(+00111
0. 0.PPP7 +- (12)
0. 01712
0.722(-0001)
0u650 (0/14))
00014)
0..0(-))024)
0.722 (200)
0.77(-0001))
0. 0,742)=00 081
0. 0.798/ 4:0 0005
0,798[=-0 000)
0.809
0.637(#-0.1 .10)
0. 0.753)=0.115)
0.62((=0.12)
0. 0.758 (::68:=0000 00.055
0.813==0.13 15)
0. 0.767 00044)
(58)
0,767[2:00555
0,707 (0007)
0.572(240061
0,501 24.004
0.570]=00006
0.742::: 0:0000000000
Aug:
0. 0.785(+00021 0.78)-0)88)
0.785 0.765(+-0000)
0-967(+6.007)
0.728(-0001)
0. 71715 -0.009)
0.060(-0000)
0.773:-0.018)
0.707(+0.055
MMDA
Hearistic
Theoretical ertor guarantees
Task
50
TMV
TMR
TCR
SOR
IWV
DEV
DWA (ours)
TB
18
0. 100000
M. 1.7997
0.767(
0.750y (111
0.750 U511)
0 08::50.0
U.SMI
(18)
20
0.872 (0011+0111
9. 8.8
(999
0.579)+00022 0155)
0.51 8/5112.0000 (0.))
0.872/46:011
0110
0 18878:00 (22))
0.801 (0.04)
35
31
0.7711-000
0. 0..000100000 .00
0.7866 200.0)
2:00033)
0.583) 4:0 107:
0.571 1:0
0.571 024)
0.77 0144
OL7 -1.00
(88T0L200013
0.840) 087)
4:00000)
015)
0.879
015)
0.804/2000000
0.9000
7
0,345/
0.655(.60 222)
0.717
0.670/5400022)
00045)
Ang
0.750(+0.05)
0.09P(+66 (N82)
0.772(+0051)
0.7455/00035)
0.755[(000985
0.759(40000)
0.75(+00025)
(0LDR3)
50

---

Published as a conference paper at ICLR 2023
Table 28: Mean and standard deviation (after =) of target classification accuracy on WISDM (Part 2) over 3 repetitions with different random initialization of model weights.
CaDATS
Heuristic
Thoordical rerur guoranters
Task
S0
TMV
TMR
TCR
SOR
IWV
DEV
IWA (ours)
TB
0.009(+00055)
(-722)
0.722
8. 0-282 058)
(.667(00.885)
0.67((00077)
0.7(#6.067)
0.750:=0.019)
0.840:=00000)
[
0.886 (200)))0000)
0.776:20.022)
0.846:=00009)
0:866 0.000)
0.004;-0.019
0.610)
0.288
0:008))))0)8)))))))))
=00114)
0.643)
0.019):0.003)
0.080)
0.714]-00003
0.780
0.727
0.399)6
0.8901 =00008))
0.964)-000000
0.742)200075)
0.727)=00009)
0.727)=00000)
0.809)00:082
0.326/400055
07773)+0000)
0.555| (.005)))))00)
0.565(#-0.000
0.75(-0.071)
0.092[H00 0.517(-0.07)
0.701(20071)
0.707
(.7.7(2)00022)
0.701(20.040)
0.0021200077)
2000070
0.735/200003)
0.764
0.812(200003)
Deep
Throretical grarantees
Tack
TMV
TMR
SUK
IWV
IWA
18
23
0.67)(+0077)
0.630(+-6
019)
0.089(+))01))
0.089 +00)19)
0.666
0.656 +0.0777
0. 0.880: 069)
0.713:-0.051)
20 30
0 (.827(==000 (50)
0.692 025)
0,801 020)
0.872 0.872 =00064)
(.84(=0004)
(-827)
0.527) =: (355)
0.885: 00.0
0.904
35 31
dt 60 0771
0 0.30)20.001)
0.867 (80)))
0,302 1220)
0.810 5x0
0. 0.00K 1.0 0.0.08))
0 0.7001200000 (0)
19
0. 0364
0. 0 (53)
0.732(400053)
(035))
0. 0.722(3-0084
0.722 83)00))
0.722 8:0 (00)
0.789) 1300
(.61(-00051) 0:608(+0007))
0-51(-00051)
0. ( (77)
0.525/ +03.9))))
0.528 +0.0000
0.054) (0.00)00000000)
Ang.
0. 0.780 00890
(96)
0. 03723
0.713
0.894(2:0057)
(1:000))
0.723 04)
0.753 (053)
CDAN
Heurisicc
Theuretical error guarantees
Tack
50
TMV
TMR
TCR
SOR
IWV
DEV
IWA (ours)
1B
0,707[2:0000)
0.728/200000
0.256:100
(2:0112
0:7(1:01)
0,872
0,858;2:0002
0.84/220.0113
0.977( (4)
0.872 020))
0.521=00029)
(5775/2:00044))
0,563 (33)
0,70N,
(11)
0.651[400144
0.722 172)
0.722
0,900
049)
0-717:400310)
20005
00.005
0.713/40.11
1.725//00.058
0.78/10.093)
0.717(5:00008)
0,767/ (:0.0707
0.78(20005)
(50085)
0.7922
0,708/
(176680005))
0.75(=0.335
0.765(200033)
0.812200022)
Heuristic
Thouecical
Task
50
TMV
TMR
TCR
SOR
IWV
DEV
IWA (ours)
TB
15
23
0,089/ 107)
0.7R8 069)
0,778(
0.078(2:0000)
0.706)240.055
DU667T (0.05)
0.7071
0.778 (:78112000119)
20 30
0,790/ m88)
0. 0.548(200.0
0.848
0,85012:0 (222)
0.86712-9.025
-0.05
0.846:-0.019) 0.750
0,885) 200000)
35
(-651 107)
0:291=00 ax4)
0.371-0-014)
0.771 (-771=000 (27)
00650:20-1660
(.851 107)
0051[-0.100 10.0
0.77(=0.0277
0.710 -00007)
0.788)=00 124)
0.000 0663
0.000[+-00006
0.984 tn 958)
0.654:=0.125
0.786-00.22
0.786
0.600))0.061)
0. (0000
0.900 Lbd 029)
0.704(4:00022)
0.648(=00022)
0.501::0.104)
0. 0.608|=0.00 0.083
0.629
0.61(=0.064)
0,008| (=00029)
Aug:
0.72
0.600
009)
(.802(+0000))
+0700)
0.745(=0:002)
0.73)
0.10(-0.113)
0.778[-0.088
(-000(=00095)
DSAN
Heuristic
Theoretical error gaaruntees
TMV
IWA (ounc)
0. 0,797(89
00.001
0.7O8I
0.850
0.55M
0.836(-0.0)
0.905(+000)
-0010)
0.00r=00
00.094
0.571
0.778:
0.87(-
0,870
0,864 2:
0.854120000
(002))
0.8391
0.870 015)
0.082) 241363)
0.828
0.877)12000005)
0.717(20.05)
0,572
0.5721
0,547
0.80812:9000 0544
0.717(=0.00)
24.000))
0..84120.047
0,707(=0000)
0.750 +3.03
0. 0.760(+-6000 (87)
0. 05087))))))))
(0755)
0.757(-0725)
7,75 0.003
(0.03(+0
0.720(-0.006)
0.817(00.119)
51


Reviews:
[{"review_id": "r9NPt2P30D", "reviewer": "Reviewer_2X6X", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a method that extends weighted least squares to deep neural networks for hyperparameter selection in unsupervised domain adaptation. In theory, the authors show that the target error of the proposed algorithm is asymptotically not worse than twice the error of the unknown optimal aggregation. Numerical validate the superiority of the proposed method over the DEV and the IWV method. ", "summary_of_the_review": "This paper proposes the first method that extends weighted least squares to deep neural networks for hyperparameter selection in unsupervised domain adaptation. The method is technically sound. \n\n------Update------\n\nI have read the author's response and most of my concerns are addressed. I will keep the score for acceptance.", "strength_and_weaknesses": "Strengths:\n\n1: This paper proposes a simple but effective method that extends weighted least squares to deep neural networks for hyperparameter selection in unsupervised domain adaptation.\n\n2: Experiments on different tasks are conducted to verify the superiority of the proposed algorithm.\n\n3: This paper is well-written and easy to follow. The background knowledge is presented well. The readers can easily understand the paper.\n\n\nWeaknesses:\n\n1. The baselines are few. Do the baselines include important and all the SOTA UDA methods?\n\n2. In theory, the authors show that the target error of the proposed algorithm is asymptotically not worse than twice the error of the unknown optimal aggregation. Such a bound cannot provide guarantees of better performance of the proposed method over others.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Vlr79ko0CYN", "reviewer": "Reviewer_V758", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work proposes a theoretical approach for choosing hyper-parameters in unsupervised domain adaptation. The main strategy is to compute an aggregation of models with target error bound, which theoretically relies on the extension of importance weighted least squares to linear aggregation of vector-valued functions. The paper conducts large scale comparative experiments on language, images, time-series classification tasks.The proposed method outperforms IWV and DEV and sets a new state-of-the-art performance.", "summary_of_the_review": "The paper proposes a new aggregation models based method in unsupervised domain adaptation. Experiments on language, images, time-series classification tasks show effectiveness of the work.", "strength_and_weaknesses": "Strenth:\n1. The proposed aggregation approch is general without strong assumptions.\n2. The paper gives detailed theoretical analysis.\n\nWeaknesses:\n1. For experiments, though the paper’s benchmarks are large and diverse, but still some popular UDA benchmarks, such as VisDA-2017, DomainNet etc are missing. Besides, the paper mainly compares with conventional methods, while certain more recent works are missing.\n2. The novelty of the paper is limited besides the combination of IWV with linear aggregation. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PF8pEqQXHF", "reviewer": "Reviewer_WzwU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how to address the issue of hyper-parameter selection in unsupervised domain adaptation. Specifically, the authors propose a method that subsequently computes a linear aggregation of the models. In addition, theories for bounding the target error are added to make the method convincing. Experiments show the effectiveness of the proposed method.", "summary_of_the_review": "Overall, this paper is well-written and contributions are not very marginal. However, more improvements should be added to the paper.\n\nStrengths:\n1. The targeted problem is important and valuable for domain adaptation (without labeled data).\n2. This paper is well-written, it is enjoyable to read.\n3. Experiments on language, image, text and time-series data show the effectiveness of the proposed method.\n\nWeaknesses:\n1. My main concern is the theories for bounding the target error. The conclusion is the target error of the computed aggregation is asymptotically at most twice the target error of the optimal aggregation. However, this bound is loose and cannot give a guarantee of superior performance.\n2. Another concern is the selection of datasets. For example, the performance gains achieved by DA methods on Amazon Reviews and WISDM are marginal (The performance of So is close to TB). \n3. The authors should conduct experiments on the datasets in the original paper of baselines. For example, experiments on digit, office and visda should be considered.\n4. More insightful analyses should be provided. For example, the visualization of the importance weight of each model should be shown. It is interesting if the importance weight can tell us what is the best model in the sequence. Now the paper only tell us what the method is and if the proposed method can achieve better results without further discussions.\n5. The scope of the selection model should also be discussed. The ensemble method works well given many effective models. However, if the candidate set of the model contains a bad model, the ensemble method may not work well.\n\n\n------Update------\n\nI carefully read the author's response and most of my concerns are addressed. I'm happy to improve my rating.", "strength_and_weaknesses": "Strengths:\n1. The targeted problem is important and valuable for domain adaptation (without labeled data).\n2. This paper is well-written, it is enjoyable to read.\n3. Experiments on language, image, text and time-series data show the effectiveness of the proposed method.\n\nWeaknesses:\n1. My main concern is the theories for bounding the target error. The conclusion is the target error of the computed aggregation is asymptotically at most twice the target error of the optimal aggregation. However, this bound is loose and cannot give a guarantee of superior performance.\n2. Another concern is the selection of datasets. For example, the performance gains achieved by DA methods on Amazon Reviews and WISDM are marginal (The performance of So is close to TB). \n3. The authors should conduct experiments on the datasets in the original paper of baselines. For example, experiments on digit, office and visda should be considered.\n4. More insightful analyses should be provided. For example, the visualization of the importance weight of each model should be shown. It is interesting if the importance weight can tell us what is the best model in the sequence. Now the paper only tell us what the method is and if the proposed method can achieve better results without further discussions.\n5. The scope of the selection model should also be discussed. The ensemble method works well given many effective models. However, if the candidate set of the model contains a bad model, the ensemble method may not work well.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HQEjv20DeSI", "reviewer": "Reviewer_AZPJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper handles the model selection problem in the context of unsupervised domain adaptation. In classical domain adaptation literature, model selection is a hard problem because of the lack of labeled target data. In recent years, several linear aggregation methods have been proposed yet they lack theoretical guarantee on the target error. Under the covariate shift assumption, the authors propose a simple yet effective linear aggregation approach based on importance weighting and provide a target error guarantee with respect to the target error incurred by the best linear aggregation. The proposed adaptation algorithm outperforms not only heuristic aggregation methods but also existing single-model selection approaches with theoretical guarantees.", "summary_of_the_review": "The authors provide a transparent algorithm with a clear presentation, hence I would suggest accepting this paper. Minor issues should be easily fixed.", "strength_and_weaknesses": "### Strengths\n\n- **Simple method**: The proposed method is based on importance weighting, which is a well-understood yet pretty simple method. In spite of its popularity, importance weighting has not been well studied in the context of linear aggregation for unsupervised domain adaptation. Hence, it does not undermine the novelty or contribution of this paper.\n- **Effective empirical performance**: The empirical performances through massive experiments demonstrate the practical effectiveness of the proposed method. It is remarkable that the proposed aggregation approach outperforms the existing methods to select a single best model.\n\n### Weaknesses\n\n- **Theory does not elucidate the effect of the model size.** Although the experiments demonstrate that the aggregation of multiple model candidates outperforms a single best model, it is not evident from the claim of Theorem 1. I tried to figure out from its proof but failed because the model size $\\\\ell$ is hidden in absolute constants of concentration bounds. Is it possible to comment on what goes on with the target error when we increase $\\\\ell$?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%