A Systematic Evaluation of Domain Adaptation Algorithms On Time Series Data
Authors: Mohamed Ragab
Emadeldeen Eldele
Wee Ling Tan
Chuan-Sheng Foo
Zhenghua Chen
Min Wu
Chee Kwoh
Xiaoli Li
Venue: ICLR 2022 Submitted (2022)
Keywords: domain adaptation
time series data
benchmarking

Abstract: Unsupervised domain adaptation methods aim to generalize well on unlabeled test data that may have a different (shifted) distribution from the training data. Such methods are typically developed on image data, and their application to time series data is less explored. Existing works on time series domain adaptation suffer from inconsistencies in evaluation schemes, datasets, and base neural network architectures. Moreover, labeled target data are usually employed for model selection, which violates the fundamental assumption of unsupervised domain adaptation. To address these issues, we propose AdaTime, a standard framework to systematically and fairly evaluate different domain adaptation methods on time series data. Specifically, we standardize the base neural network architectures and benchmarking datasets, while also exploring more realistic model selection approaches that can work with no labeled data or few labeled samples. Our evaluation includes adaptations of state-of-the-art visual domain adaptation methods to time series data in addition to recent methods specifically developed for time series data. We conduct extensive experiments to evaluate 10 state-of-the-art methods on 3 representative datasets spanning 15 cross-domain scenarios. Our results suggest that with careful selection of hyper-parameters, visual domain adaptation methods are competitive with methods proposed for time series domain adaptation. In addition, we find that model selection plays a key role and different selection strategies can significantly affect performance. Our work unveils practical insights for applying domain adaptation methods on time series data and builds a solid foundation for future works in the field.  

Under review as a conference paper at ICLR 2022
A SYSTEMATIC EVALUATION OF DOMAIN ADAPTA- TION ALGORIHTS ON TIME SERIES DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised domain adaptation methods aim to generalize well on unlabeled test data that may have a different (shifted) distribution from the training data. Such methods are typically developed on image data, and their application to time series data is less explored. Existing works on time series domain adaptation suffer from inconsistencies in evaluation schemes, datasets, and backbone neu- ral network architectures. Moreover, labeled target data are usually employed for model selection, which violates the fundamental assumption of unsupervised do- main adaptation. To address these issues, propose benchmarking evaluation suite (ADATIME) to systematically and fairly evaluate different domain adaptation methods on time series data. Specifically, standardize the backbone neural net- work architectures and benchmarking datasets, while also exploring more realistic model selection approaches that can work with labeled data just few labeled samples. Our evaluation includes adapting state-of-the-an visual domain adap- tation methods time series data in addition to the recent methods specifically developed for time series data. We conduct extensive experiments to evaluate 10 of-th----eee-aee art methods on representative datasets spanning 20 cross-domain scenarios. Our results suggest that with careful selection of sual domain adaptation methods are competitive with methods proposed for time series domain adaptation. In addition, we find that typer-parameted could be se- lected based on realistic model selection approaches without relying on labeled samples from target domain. Our work unveils practical insights for applying do- main adaptation methods on time series data, and builds a solid foundation for future works in the field.
vi-
INTRODUCTION
Time series classification problem is predominant in many real-world applications including health- care and manufacturing. Yet, time series data with its inherit temporal dynamics and complex un- derlying pattern can be more challenging than static data. Recently, deep learning has gained more attention in time series classification tasks, assuming access to vast amount of labeled data for training (Fawaz et al.] 2019). However, annotating time series data can be challenging and bur- densome due its complex underlying patterns and complicated domain knowledge (Chang al.) 2020). One way to reduce labeling burden is to leverage annotated data (c.g., synthetic or public data) from relevant domain (i.e., source domain) to train the model, while testing the domain of interest (i.e., target domain). Nevertheless, deep learning tends to perform poorly when tested on unseen data that have different distribution from the training data, which well-known as the domain shift problem. Considering time series applications, source and target domains usually rep- resent data from different subjects (persons) as in human activity recognition (HAR) (Wilson al.) 2020 Chang et al. 2020) or sleep stage classification (SSC) tasks (Phan et al. 2020 A consider- able amount of literature has been developed on Unsupervised Domain Adaptation (UDA) )) for visual applications to mitigate the domain shift problem (Long al. 2015 Ganin et al. 2016 Long al.
2018 Chen et al. 2020 Rahman et al. 2020).
Recently, more attention has been paid to time series UDA (TS-UDA) (Wilson et al. 2020 Chang jet al.) 2220; Ragab et al.; 22200: [iu & Xue 202T). However, the literature of TS- UDA methods
suffers from the following limitations:

---

Under review as a conference paper at ICLR 2022
Existing TS-UDA works lack consistent evaluation schemes including benchmark datasets, preprocessing, and backbone networks. For instance, some methods that leverage recur- rent neural network backbone network (Purushotham et al. 2017) are compared against methods with convolutional based backbone networks (Wilson et al. 2020). In addition, even with similar architecture, training procedures also vary among different algo- rithms including number of epochs. weight decay, and learning rate schedulers (Tonutti et al. 2019 Purushotham et al., 20177.
Existing TS-UDA works use labeled data from the target domain for model selection, vio- lating the basic assumption of UDA (Wilson et al. 2020 Liu & Xue (02T), and providing an overoptimistic view of their performance. It is worth noting that model selection for domain adaptation in the absence of target domain labels is long standing problem.
Most of existing algorithms are mainly application specific, and few works have been pro- posed for general TS-UDA, As a result, there is a shortage of baseline methods when ap- plying domain adaptation on time series data.
All the aforementioned challenges highly affect the performance and can be mistakenly at- tributed to the proposed domain adaptation methods.
In this work, propose systematic evaluation suite (ADATIME) to tackle the aforementioned obstacles and remove all extraneous factors to ensure a fair evaluation of different UDA algorithms on time series data. Specifically, to address the inconsistent evaluation schemes and backbone net- works, we first standardize the preparation and processing of four benchmarking datasets from two classic real-world applications, including healthcare and human activity recognition. Besides, we unify the backbone network and the training procedures when comparing between different UDA methods. Second, to select the model hyper-parameters in the absence of target labels, we explore more realistic model selection strategies for TS-UDA problem that do not require target labels. Par- ticularly, we investigate the model performance when selecting models based on source dataset (i.e.. source risk (Ganin et al. 2016).. unlabeled target data (i.e., deep embedded evaluation (DEV) (You ct al. 2019), or only few-shot labeled samples from target data, Last, to address the lack of TS- UDA baselines, re-implement various state-of- state-of-the-art the-art visual UDA methods that can be adapted to time series data while comparing to the existing TS-UDA methods
Given our standard methodology, aim to systematically study the following questions: (1) With standard backbone network and evaluation schemes, how will the visual UDA methods perform on time series data; (2) Can use realistic model selection methods rellying only few target labels- and still achieve acceptable adaptation performance on time series data; (3) How can the backbone network contribute the performance. In this paper, we conduct comprehensive experiments to answer the aforementioned three questions. Some of our findings are summarized as follows:
Visual domain adaptation methods can achieve comparable even better performance than the methods proposed for time series data.
Unlike image data, selecting models based on source risk can achieve reasonable perfor- mance and outperforms the DEV risk for time series data. Additionally, find that proposed few-shot target risk can achieve comparable performance to target risk, with af- fordable few labels samples.
Changing the base architecture can be pivotal the performance. Moreover, complex architectures with time series data can have lower generalization performance on cross- domain experiments.
2 DOMAIN ADAPTATION PROBLEM
We start by defining the unsupervised domain adaptation problem. We assume the access to a labeled source domain Xs (xx,, vf Ns that represents uni-variate multivariate time series data, and unlabeled target domain X2 {x>T)) where Ns and Ny denote the number of samples for Xs and Xy respectively. The source and target domains are sampled from different marginal distributions, i.e., Ps(x) * Pr(x)) while the conditional distribution remains stable (Ps(ylar)
2

---

Under review as a conference paper at ICLR 2022
Data Preparation
Domain Alignment
Model Selection
Datasets
X
X
X
Zirain
Z
Rest
HAR
WISDM
SSC
Seurcs Data
Source Model
ded
Target Model
Few shots
Preprocessing
Source Risk
DEV Risk
FST Risk
Target Rick
Testing
Zirain
Test
solett undid of
select
relect moddel
Sliciing
Trainkest split
Source
DEV Model
FST
Domain Alignment
Fram
X
Truin
X
Discrepancy
Adversarial
Test Resalts
Figure I: Our benchmarking methodology ADATIME consists of three main steps: Data Preparation, Domain Alignment, and Model Selection. We first prepare the train and test data in both source and target domains (i.e., X X passed through the backbone network extract the corresponding features. The domain alignment algorithm being evaluated is then used to address the distribution shift between the source and target features. Last, we pass source and target data to select best hyper-parameten for the domain alignment algorithm. We used three different model selection approaches (Best viewed in color).
XTss). Then the processed source and target data are
Pr(y)))). The main goal of UDA is to reduce the distribution shift between Ps and Pr(x), assuming they share the same label space.
The mainstream of UDA algorithms is addressing the domain shift problem by finding domain invariant feature representation. Formally, given backbone model fo X - Z. which transforms the input space to the feature space, the UDA algorithm mainly optimizes the backbone network find new invariant representation for the target features such that (fo())) results, a model trained on the source domain can generalize well to the adapted target domain.
(x(f) As
3 ADATIME: A BENCHMARKING APPROACH FOR TIME SERIES DOMAIN ADAPTATION
In this work, we systematically evaluate domain adaptation algorithms on time series data, ensuring fair and realistic procedures. Figure shows the details of ADATIME, which proceeds follows. Given a dataset, we first apply our standard time series preparation schemes on the selected source and target domains. Subsequently, the source train data X train and target train data X trvin passed through the backbone network to extract source and target features, i.e., Z and Z train respectively. The selected UDA algorithm is then applied to mitigate the distribution shift between the extracted features of two domains. Last, set the consider three practical model selection approaches (i.c., without any target domain labels or only few-shot labeled samples) namely source (SRC) risk, deep embedded evaluation (DEV) risk, and few-shot target (FST) risk. The proposed evaluation pipeline can prevent any extraneous factors and enable fair comparison between different UDA methods. The code of ADATIME will be made pub- licly available for researchers to enable seamless evaluation of different domain adaptation methods on time series data.
of the UDA algorithm,
3.1 BACKBONE NETWORK
The general domain adaptation network consists of feature extractor, classifier, as well dif- ferent components to align the domains. We refer to the feature extractor and the classifier as the backbone network. The backbone network can have pivotal contribution the UDA model per- formance, as using different architectures may result in different performance. Hence, changing the backbone network when comparing different UDA methods result in misleading outcomes,
3

---

Under review as a conference paper at ICLR 2022
which hinder the fair evaluation protocol. In most visual applications, ResNet-50 is the domi- nantly adopted feature extractor, as its complex architecture can extract representative features from images. However, in time series, using such complex architectures can lead to overfitting problems (Fawaz, et al.] 2019). Therefore, we use a ID-CNN architecture in ADATIME (see Figure 6). In addition, we study the effect of different backbone architectures the performance of TS-UDA methods.
3.2 DOMAIN ADAPTATION ALGORIHTM
While numerous UDA approaches have been proposed to address the domain shift problem Zhao et al. 2020b), comprehensive review of existing UDA methods is out of our scope. Instead, only included the most solid and recent baselines for visual UDA that can be adapted to time series. Besides, we included the state-of-the-art methods proposed for time series data. Overall, the selected algorithms in ADATIME can fit into two main categories according to the domain adaptation method, namely discrepancy based methods and adversarial based methods. The former aims to minimize statistical distance between source and target features to mitigate the domain shift problem (Tzeng al., 2014; [Sun & Saenko, 2016; Chen et al., (020). The latter leverages a domain discriminator network that enforces the feature extractor to produce domain invariant features (Ganin et al.) [2016] Tzeng et al. 2017). Each method in the above categories can also be classified according to the aligned distribution. Specifically, some algorithms only align the marginal distribution of the feature space, while other algorithms jointly align the marginal and conditional distributions.
The selected UDA algorithms are as follows: Deep Domain Confusion (DDC Tzeng et al. (0144); Higher-order Moment Matching (HoMM Zhao et al. (2020a);; Correlation Aligmment via Deep Neural Networks (Deep-CORAL Sun et al. (2017)); Minimum Discrepancy Estimation (MMDA Training of Neural Networks (DANN Ganin et al. (2016)); Conditional Adversarial Domain Adaptation (CDAN (CDANLong et al. (2018); Deep Subdomain Adaptation (DSAN Zhu et al. (202T); (DIRT-T (Shu et al. 2018)). In addition, we also include two UDA methods applied in time series classification, which are CoDATS (Wilson al.) 2020) and AdvSKM (Liu & Xue (02I). noting that there are very few TS-UDA methods in the literature. We excluded the methods proposed for time series gredictionfforecastig since they are out of our scope. Table I shows summary of the selected methods and their corresponding categories.
Rahman et al.) (200)); Domain-Adversane
Table 1: Summary of Domain Adaptation algorithms used for our benchmarking framework
Algorithm
Application
Category
Distribution
Losses
Model Selection
DDC
Visual
Discrepancy
Marginal
MMD
Target Risk
Deep-Coral
Visual
Discrepancy
Marginal
CORAL
Not Mentioned
HOMM
Visual
Discrepancy
Marginal
High-order MMD
Not Mentioned
MMDA
Visual
Discrepancy
Joint
MMD,
Target Risk
CORAL,
Entropy
DSAN
Visual
Discrepancy
Joint
Local MMD
Not Mentioned
DANN
Visual
Adversarial
Marginal
Domain Classifier,
Source Risk
Gradient Reversal Layer
CDAN
Visual
Adversarial
Joint
Conditional adversarial
Importance Weighting
Domain Classifier
DIRT-T
Visual
Adversarial
Joint
Virtual adversarial
Target Risk
Entropy
Domain Classifier
CoDATS
Time Series
Adversarial
Marginal
Domain Classifier,
Target Risk
Gradient Reversal Layer
AdvSKM
Time Series
Adversarial
Marginal
Spectral Kernel
Target Risk
Adversarial MMD
4

---

Under review as a conference paper at ICLR 2022
3.3 MODEL SELECTION APPROACHES
Model selection and hyper-parameet tuning for UDA long standing non-trivial problems due to the absence of target domain labels. However, we find that papers describing 5 out of the 10 included UDA methods target domain lahels to select hyper- syper-parameters, olates the basic assumption of UDA. In addition, another three papers used fixed hyper-parametent without describing how these parameters were selected. This issue can lead to unrealistic perfor- mance and unfair evaluation. To address this issue, we evaluate multiple realistic model selection approaches that could be applied for TS-UDA without the need of any target domain labels, such as: SRC risk (Ganin et al.) 2016) and DEV risk (You et al. 2019). Besides, we design FST risk, which utilizes affordable few labeled samples from the target domain. Detailed explanation of each risk can be found as follows:
shown in Table 1, which vi-
Source Risk (SRC Risk) this approach, we select the hyper-parametent that achieve the min- imum risk the source domain test set, which is readily available. Hence, this approach can be easily applicable without additional labeling efforts, since it relies on the existing labeled source data (Ganin et al. 2016). However, the effectiveness of the source risk is mainly controlled by the sample size of source data and severity of distribution shift.
Deep Embedded Validation (DEV Risk) This approach tries to yield unbiased estimation to the target risk. Its key idea is to consider the relationship between the source and target data when calculating the risk. Particularly, DEV risk considers the source features that highly correlated to the target features via importance weighting schemes. By ranking candidate models based on their computed DEV risks, the model with the smallest DEV risk chosen as the best candidate model to be used for adaptation, However, this approach is computationally expensive and may have unstable performance with smaller source and target datasets.
Few-Shot Target Risk (FST Risk) We propose few-shot target risk relaxed target risk. While labeling of vast amount time series data can be laborious, annotating few-shot samples can still be affordable. We use these samples as validation set to select the best hyper-parameeers. This strategy can achieve similar performance to using target risk, but does require labeling a few target domain samples.
Target Risk (TGT Risk) This approach involves leaving out a subset of the target domain samples and their labels as validation set, and using them select the best performing hyper-parameter the target domain. Using this model selection technique provides an upper bound on the performance of the UDA method, it leverages labeled data from the target domain. Even though this risk is impractical in unsupervised settings, it has been used by many UDA papers for model selection (Shu
al. 2018 Saito et al. 2017
EXPERIMENTS
4.1
BENCHMARKING DATASETS
We evaluate the various UDA algorithms on four benchmark datasets from two real-world appli- cations, namely human activity recognition and sleep stage classification. The benchmark datasets span a range of different characteristics including complexity, type of sensors, samples size, class- distribution, and severity domain shift, enabling more broad evaluation. The selected datasets are detailed as follows:
UCIHAR (Anguita et al. 2013) UCIHAR is one of the most widely used datasets to evaluate per- formance on time series data. It contains three different sensors namely, accelerometer, gyroscope, and body sensors. These sensors have been used to collect data from 30 different persons. In our experiments, treat each subject separate domain. Due the large number of cross-domain combinations, we randomly selected five cross domain scenarios, as (Liu Xue 2021 Wilson al. 2020).
5

---

Under review as a conference paper at ICLR 2022
Table 2: Details of datasets, More details about selected cross-domain scenarios for each dataset cab be found in Tables and
Number of
Total samples
Dataset
Users/Domains
Channels
Classes
Timesteps/sample
Training set
Testing set
UCIHAR
32
6
128
2300
990
WISDM
36
6
128
1350
720
SSC
20
5
3000
14280
6130
HHAR
3
6
128
12716
5218
WISDM (Kwapisz et al. 2011) WISDM is another popular activity recognition dataset for the evaluation of time series domain adaptation. In this dataset, accelerometer sensors applied to collect data from 36 subjects. This data can be more challenging because the class imbalance issue among different subjects. Particularly, some subjects may contain only samples from subset of the overall classes. Further detailed about ratio of different classes among subjects can be found in the Appendix. Similar UCIHAR dataset, we consider each subject separate domain and we randomly select five cross-domain scenarios.
SSC (Goldberger ef al. 2000) Sleep stage classification (SSC) problem aims to classify the elec- treecephalooraphyy (EEG) signals into five stages i.e. Wake (W), Non-Rapid Eye Movement stages (NI, N3), and Rapid Eye Movement (REM). We adopted Sleep-EDF dataset (Goldberger et al.) 2000 which contains EEG readings from 20 healthy subjects. We selected single channel (i.e.. Fpz-Cz) following previous studies (Eldele et al. 2021a), and 10 different subjects to construct five subject-wise cross-domain scenarios.
HHAR (Stisen et al. 2015 The Heterogeneity Human Activity Recognition (HHAR) dataset has been collected from different users using sensor readings from smartphones and smartwatches. In our experiments, consider each user as domain. We constructed 10 cross-domain scenarios from randomly selected users. We used the same smartphone device for all the selected users to reduce the heterogeneity.
Table 2 summarizes the details of each dataset, e.g., the number of selected domains, the number of sensor channels, the number of classes, the length of each sample, well the total number of samples in both training and test portions.
4.2 EXPERIMENTAL SETUP
Hyper-parameters Sweep For each algorithm and dataset combination, conducted extensive random hyper-parameter search with 100 trials. The hyper-parameters are picked by a uniform sampling from range of predefined values. Details about the specified ranges can be found in Table [8 In addition, for each set of hyper-parameters, we calculated the risk values over three different random seeds, removing the bias single seed. We picked the model that achieves the minimum risk value for each model selection strategy. For few-shot target risk experiments, we used five samples per class each dataset to compute the risk.
Datasets Preprocessing The preprocessing of time series data includes data slicing, train/test splitting, normalization. To promote fair evaluation, we preserved consistency among all the afore- mentioned processing steps when comparing between different UDA algorithms. We used a sliding window of 128 for human activity recognition datasets. For SSC dataset, we kept the original sam- ple length of 3000 time steps. Next, we split both source and target domains into train/test splits with ratio of 0.7/0.3. Finally, we normalized both training and testing data based on the training statistics (Wilson et al. 2020 Liu & Xue 2021). It worth noting that WISDM dataset is severely imbalanced among different subjects (see Figure 4(b)in in the appendix), with some classes not being available for some subjects. To stabilize the training, we selected subjects that contain samples from all the classes when constructing the source and target domains.

---

Under review as a conference paper at ICLR 2022
Backbone Network To ensure fair evaluation among all the baselines, used fixed backbone network among all the compared UDA methods. We employed a 3-layer ID-CNN with an adaptive average pooling, to handle signals with different lengths, as a feature extractor. This architecture is commonly used in series applications (Wilson et al. 2020: Eldele et al. layer fully connected network as the classifier. Yet, include CoDATS in our experiments, we used a 3-layer fully connected classifier (as proposed in their work) to differentiate it from DANN.
2021b). We adopted single
Training Procedure All the training procedures have been standardized across all UDA algo- rithms. For instance, we trained each models for 40 epochs, as performance tends to decrease with longer training. We reported the model performance after completing the last epoch. Regarding the optimization of the model, (31, (32) = (0.5, 0.99). The learning rate was set to be a tunable on each dataset. We excluded any learning rate scheduling schemes from experiments to ensure that the contribution is mainly attributed to the selected UDA algorithm.
used Adam optimizer with fixed weight decay of lc-4 and
for each method
RESULTS DISCUSSION
We conducted extensive experiments for each UDA method on all the datasets. The average F1- score of the five cross-domain scenarios for each dataset-algorithm combination is reported in Table We also show the results of each risk to compare the performance to the target risk. Detailed versions of 3 be found in Tables 5 and in the appendix. We further diseuss the main findings in the following paragraphs.
3
(a) UCTHAR dataset
(b) WISDM dataset
(c) SSC dataset
(d) HHAR dataset
Results of best models according to target risk for different methods in terms of accuracy and macro average F1-score.
Figure 2:
Visual UDA methods achieve comparable performance to TS-UDA methods on time series data. Table shows the performance of the adapted visual UDA methods along with the existing TS-UDA methods. Surprisingly, with standardizing the backbone network, we find that visual UDA methods achieve competitive or even better performance than TS-UDA methods. This finding is consistent for all the model selection strategies across the benchmarking datasets. A possible ex- planation that all selected UDA algorithms are applied on the vectorized output features of the backbone network, which can be independent from the input data modality. This finding suggests that with a standard backbone network, visual UDA algorithms can be strong baselines for TS-UDA.
Methods with joint distribution alignment tend to perform consistently better. As shown in T methods that address the joint distributions (i.e., both marginal and conditional distributions concurrently) such DIRT-T, MMDA, and DSAN, achieve the state-of-the-an performance on the four datasets, outperforming TS-UDA methods. For example, the best performing method, as selected by the target risk, is DIRT-T in both UCIHAR and WISDM datasets, and DSAN in SSC and HHAR datasets. Similarly, with respect to the different risks, DIRT-T, MMDA, and DSAN are interchangeably achieving performing best across the benchmarking datasets. Hence, considering the conditional distribution when aligning the source and target can be beneficial to the performance.
Model selection has a significant effect on the performance. Model selection strategies can yield different performance for the same UDA algorithm, shown in Table 3 This reveals the significant contribution of the model selection approach to the overall performance. While target risk achieves superior performance, it is infeasible for practical scenarios with the absence of target
7

---

Under review as a conference paper at ICLR 2022
Table The average results (from 5 cross-domain scenarios) according the minimum risk value in terms of MFl-score.
Dataset
Algorithm
SRC Risk
DEV Risk
FST Risk
TGT Risk
DDC
68.83
74.14
74.25
75.67
Deep-Coral
71.99
71.43
77.23
77.71
HoMM
75,86
78.28
81.3
84.10
MMDA
80.12
80.12
79.54
81.40
DSAN
83.31
81.07
87.13
90.96
UCIHAR
DANN
79.82
80.89
83.1
84.97
CDAN
86.55
64.66
86.79
86.79
DIRT-
86.72
82.54
88.47
92.20
CoDATS
79.05
65.12
79.1
85.47
AdvSKM
71.08
74.62
74.47
74.67
Avg/risk
78.33
75.29
81.14
83.39
DDC
54.98
52.80
50.05
55.03
Deep-Coral
55.54
53.85
49.45
57.43
HoMM
57,49
61.23
46.56
62.98
MMDA
57.53
57.30
52.12
63.97
DSAN
56.51
56.51
53.41
60.08
WISDM
DANN
53.21
54.48
49.45
57.81
CDAN
52.49
53.27
52.75
57.85
DIRT-T
60.43
53.24
62.61
66.28
CoDATS
52.72
54.27
48.64
56.57
AdvSKM
53.95
57.46
49.02
60.55
Avg/risk
53,93
53.52
51.91
60.82
DDC
59.18
59.21
59.22
59.22
Deep-Core Per-Coral
59.12
58.81
58.82
59.12
HoMM
59.06
60.95
58.70
59.06
MMDA
62.08
61.49
57.98
62.79
DSAN
58.14
59.85
58.97
60.57
SSC
DANN
60.26
57.77
60.26
60.26
CDAN
54.89
56.86
56.17
59.04
DIRT-T
58.44
59.26
58,23
59.42
CoDATS
56.76
55.79
54.64
58.44
AdvSKM
59.94
59.92
59.93
60.21
Avg/risk
59.37
59.34
59.10
60.30
Deep-Coral
70.78
69.88
70.68
72.28
HoMM
71.18
72.50
68.62
73.47
MMDA
66.20
70.23
71.07
77.04
DSAN
76.18
78.95
78.18
81.14
HHAR
DANN
76.24
72.62
73.68
76.42
CDAN
77.74
77.43
77.43
78.09
DIRT-T
75.56
78.69
78.41
80.04
CoDATS
75.11
73.72
74.74
76.09
ADVSKM
66.58
66.96
69.93
69.93
Avg/risk
72.32
72.73
73.20
75.44
domain labels. Surprisingly, source risk can achieve comparable performance to the target risk on UCIHAR, HHAR and SSC datasets. In addition, with affordable labeling efforts, our proposed few-shot target risk also achieve competitive performance the target risk. However, for severely imbalanced WISDM dataset, different model selection approaches fail to follow the target risk performance.
8

---

Under review as a conference paper at ICLR 2022
IB-NN 10-
10.BoNNe.111
30am F-ours
Dac.
HCnn
Dasy
1841
URR-
Mimi
Bowu
Mim.
BAYY
COLAS
inAs
CaDAra
MOOKH
(a) UCIHAR dataset
(b) HIARR dataset
Figure 3: Comparison between ID-CNN and ID-ResNet- 18 backbones applied on the both UCI-
HAR and HHAR dataset. Results are terms of macro Fl Fl-score.
Backbone network can be pivotal to the performance. To investigate the impact of the back- bone network on the method's 's performance, we evaluate all the UDA algorithms under two different backbone networks for both small-scale and large-scale datasets, i.e., UCIHAR and HHAR. Particu- larly, we employ ID-CNN (described in Section (4.2) and ID-ResNee-118 (i.e., standard ResNet-18 with 1D) convolutional kernel) as backhone networks. The two architectures are different in terms of their complexity and the number of trainable parameters. Figure 3 shows a comparison of different UDA algorithms under two backbone architectures. Clearly, the two backbone networks can have different absolute performance for each UDA methods. Nevertheless, the relative performance be- tween different UDA methods still be consistent when fixing the backbone network. For instance in Figure 3b, the DSAN and DIRT approaches perform best for both ID-CNN and ID-ResNet- 18 backbone networks. Besides, it is still clear that visual UDA methods perform better than TS-UDA methods with the two different backbone networks.
Accuracy metric should not be used to measure performance for imbalanced data. While it is
well-known that accuracy is not representative metric for class- imbalanddd datasets, existing TS-
UDA methods are still using it to report their performance (Chang et al. 2020; Wilson et al.) [020]
Liu & Xue 2021). We aim to re-emphasize that the accuracy metric can give over-optimistic re- sults when considering the imbalanced nature in most time series data. For example, in Figure 2(b) although CDAN achieves higher accuracy than some other methods such DDC, MMDA and DSAN, it performs the worst terms of Fl-score. In contrast, with a balanced dataset (i.c., UCI- HAR), accuracy still be representative and achieve similar performance to F1-Score, shown in Figure ((a)
CONCLUSIONS AND RECOMMENDATIONS
In this work, we provided ADATIME, a systematic evaluation methodology for evaluating the ex- isting domain adaptation methods time series data. To ensure fair and realistic evaluation, we standardized the benchmarking dataset, evaluation schemes, and backbone networks among differ- ent domain adaptation methods. Moreover, explored more realistic model selection approaches that can work with without any target domain labels or only few-shot labeled samples. Based on our systematic study and unveiled findings, suggest the following recommendations:
Future research time series domain adaptation should consider visual UDA methods as strong baselines.
Considerable amount of target labels should not be used for the UDA model selection. Instead, source risk or few-shot target risk can be better candidates for more realistic model selection approach.
The backbone network should be fixed when comparing between different UDA methods.
Larger datasets should be considered when comparing baselines to obtain reliable results.
Accuracy metric should no longer be used to evaluate imbalance datasets, as can yield misleading results.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. In European Symposium Artificial Neural Networks, pp. 437-442, 2013.
Youngjae Chang, Akhil Mathur, Anton Isopoussu, Junehwa Song, and Fahim Kawsar. A systematic study of unsupervised domain adaptation for robust human-activing recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(1):1-30, 2020.
Chao Chen, Zhihang Fu, Zhihong Chen, Sheng Jin, Zhaowei Cheng, Xinyu Jin, and Xian-Sheng Hua. Homm: Higher-order moment matching for unsupervised domain adaptation. AAAI, 2020.
Emadeldeen Eldele, Zhenghua Chen, Chengyu Liu, Min Wu, Keong Kwoh, Xiaoli Li, and Cuntai Guan. An attention- deep learning approach for sleep stage classification with single- channel eeg. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021a. doi:
N0.1109//NSRE.221..00 .30762344
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting, In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, 1JCAI-21, pp. 2352-2359, 2021b.
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep learning for time series classification: review. Data Mining and Knowledge Discovery, 33(4)9917-963, 2019.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learning Research, 17(1):2096-200 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Paseal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. JMLR, 17(1):1-35, 2016.
Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph F. Mietus, George B. Moody, Chung-Kang Peng, and H. Fugene Stanley. Physiobank, physiotoolkit, and physionet components of new research resource for complex physiologic signals. Circulation, 101(23):215-20) 2000.
2000.
Jennifer R. Kwapisz, Gary M. Weiss, and Samuel A. Moore. Activity recognition using cell phone accelerometers. Sigkdd Explorations, 12(2):74-8 -2,744--22000 2011.
Qiao Liu and Hui Xue. Adversarial spectral kemeel matching for unsupervised time series domain adaptation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelli- gence, IJCAI-21, 2021.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105, PMLR, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Conditional adversarial domain adaptation. In NeurIPS, 2018.
Huy Phan, Kaare Mikkelsen, Oliver Chén, Philipp Koch, Alfred Mertins, Preben Kidmose, and
Maarten De Vos. Personalized automatic sleep staging with single-night data: pilot study with kulbback-leibler divergence regularization. Physiological Measurement, 4166)444004 2020.
Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl. Tunability: importance of hyperpa- rameters of machine learning algorithms. The Journal of Machine Learning Research, 20(1): 1934-1965, 2019.
Sanjay Purushotham, Wilka Carvalho. Tanachat Nilanon, and Yan Liu. Variational recurrent adver- sarial deep domain adaptation. In ICLR 2017: International Conference on Learning Represen- fations 2017, 2017.
10

---

Under review as a conference paper at ICLR 2022
Mohamed Ragab, Zhenghua Chen, Min Wu, Haoliang Li, Chee- Chee-Keong Kwoh, Ruqiang Yan, and Xi- aoli Li. Adversarial multiple-tar target domain adaptation for fault classification. IEEE Transactions Instrumentation and Measuremens, 70:1-11, 2020.
Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. On minimum discrepancy estimation for deep domain adaptation. Domain Adaptation for Visual Understanding. 2020.
Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri-training for unsupervised domain adaptation. In ICML'17 17 Proceedings of the 34th International Conference Machine Learning, pp. 2988-2997, 2017.
Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. In International Conference Learning Representations, 2018.
Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjærgaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen. Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In Proceedings the 13th ACM Conference on Embedded Networked Sensor Systems, pp. 127-140, 2015.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, pp. 443-450. Springer, 2016.
Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adap- tation. In Domain Adaptation in Computer Vision Applications, pp. 153-171. 1...innge, 2017.
Michele Tonutti, Emanuele Ruffaldi, Alessandro Cattaneo, and Carlo Alberto Avizzano. Robust and subject-independent driving manoeuvre anticipation through comain-adversaal recurrent neural networks. Robotics and Autonomous Syxtems, 115: 162 2019.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell, Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Garrett Wilson, Janardhan Rao Doppa, and Diane Cook. Multi-source deep domain adaptation with weak supervision for time-series sensor data. In SIGKDD, 2020.
Kaichao You, Ximei Wang. Mingsheng Long, and Michael Jordan. Towards accurate model selec- tion in deep unsupervised domain adaptation. In International Conference on Machine Learning, pp. 7133.. PMLR, 2019.
S. Zhao, Xiangyu Yue, Shanghang Zhang, Bo Li, Han Zhao B. Wu, R. Krishna, J. Gonzalez, A. Sangiovanni-Vin Vincentelli, S. Seshia, and K. Keutzer. A review of single-source deep unsu- pervised visual domain adaptation. IEEE transactions on neural networks and learning systems, PP, 2020a.
Sicheng Zhao, Xiangyu Yue, Shanghang Zhang, Bo Li, Han Zhao, Bichen WI, Ravi Krishna, Joseph E Gonzalez, Alberto L Sangiovanni- Vincentelli, Sanjit A Seshia, et al. A review of single- source deep unsupervised visual domain adaptation. IEEE Transactions Neural Networks and Learning Systems, 2020b.
Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Guolin Ke, Jingwu Chen, Jiang Bian, Hui Xiong, and Qing He. Deep subdomain adaptation network for image classification. IEEE Transactions on Neural Networks and Learning Systems, 32(4):1713-1722, 2021, doi: 1091109/N 2.200. 2988928.
11

---

Under review as a conference paper at ICLR 2022
A DETAILED RESULTS OF OUR FRAMEWORK
Here, we provide the detailed results including mean and standard deviation for each cross-domain scenario. It can be clearly seen, that datasets with small sample size suffers from high variance among the different cross-domain scenarios.
Table 4: Detailed results of scenarios of UCIHAR dataset in terms of MF1 score.
Risk
Algorithm
0-+11 12-+16
9-+18
6-+23
7-+13
AVG
DDC
60.00=t:3.32
66.77=8.46
61.41
88.55+1.42
77.29+2.11
75.67
Deep-Coral
13.27
64.58:8.72
54.38:19.69
89.66:1..54
90,46:12.96
77.71
HoMM
83.54422.99
63.45:12.07
71.25:14.42
94.97+22499
91.41
84.10
MMDA
72.91+22.78
74.64+12.88
62.62:+2.63
91.14:10.46
90.61+2.00
81.40
Target
DSAN
99.23#1.09
65.71
89.69:7.44
97.82
94.33+0.00
90.96
DANN
98.09:11.68
62.08=1.69
70.7=11.36
85.6=15.71
93.3310.00
84.97
CDAN
98.19+1.57
61.20+3.27
71.3=14.64
96.73++0.00
93.33+0.000
86.79
DIRT-T
98.13+2.64
82.05+8.61
85.90+6.63
93.76++3.10
93.35+0.00
92.20
CoDATS
86.6544.28
61.03+2.33
80.51
92.08+4.399
92.6110.51
85.47
AdvSKM
65.7442.69
60.52+1.99
53.2545.119
79.63#8.52
88.8943.12
74.67
DDC
67.95+7.51
58.58
48.33+44.06
84.34+5.22
86.33=2.19
74.25
Deep-Coral
70.86:15.85
59.3+0.77
58.5+
1.65+
85.23+3.01
77.23
HoMM
78.87+8.37
6..34+1.07
66.97++3.41
93.8442.04
87.82+4.43
81.3
MMDA
74.3619..26
66.01
54.92+4.27
95.
93.33+0.0
79.54
Few-shot
DSAN
89.4748.776
65.97=2.59
78.0247.52
96.6842.74
92.61
87.13
DANN
87.7345.29
60.33+1.9
69.69-1..3
89.88=16.4
93.33++.00
83.10
CDAN
98.19+1.57
61.2+t3.27
71.31=14.64
96.73=0.0
93.33=1000
86.79
DIRT-T
12.33+1149
71.63+6.64
86.87+0.39
86.71+14.17
93.33++.00
88.47
CoDATS
71.6415.34
65.1
64.31
92.04
81.4116.04
79.10
AdvSKM
64.45-12.59
61.99.14.07
53.13:44.59
78.0911..2
88.8943.12
74.47
DDC
72.04:3.51
59.65:4..1
45.42
86.14
81.66
74.14
Deep-Coral
17.55+11.65
62.13+7.74
47.77+22.84
12.44+13.45
78.7+18.24
71.43
HoMM
73.38-7.33
59,84+1.43
60.02+11.83
90,48-0.8
85,9442.52
78.28
MMDA
83.2243.46
62.64+10.42
58.4342.55
96.73-0.0
94.12+1.11
80.12
DEV
DSAN
75.5819.18
61.71
67.1=4.61
93.2242.49
88.8243.08
81.07
DANN
18.27
63.26+2.49
57.49+7.77
95.86+1.84
91.71+00.4
80.89
CDAN
7..51
54.66+2.91
40.9443.18
61.31
82.06+11.91
64.66
DIRT-T
88,4419.23
58,4742.98
65.89-113225
90.56418.3
93.73110.56
82.54
CoDATS
51.81
54.81+2.76
31.83+48.89
81.23+4.07
80.98+13.74
65.12
AdvSKM
65.7442.69
60.09
53.7+4.61
79.31
88.8943.12
74.62
DDC
53.28+-5.44
6..59+6.34
41.99+1.47
89.01+2.14
85.65+7.92
68.83
Deep-Coral
62.42++1.97
62.19+4.68
31.4+7.7
88.42+1.01
87.84+2.78
71.99
HoMM
62.95-120.6
59.8240.4
53.1414.71
90.04:4..45
89.5914.01
75.86
MMDA
83.2243.46
10.64
58.4342.55
96.73-0.0
94.12+1.111
80.12
Source
DSAN
84.81+13.2
62.91
77.63+0.94
81.91+23.31
92.97+0.51
83.31
DANN
70.28+22.6
65.45+4.77
71.3446.17
90.0+22.11
90.51+0.86
79.82
CDAN
88.7314.7
60.25
81.39-L6.15
96.73-0.00
92.9710.51
86.55
DIRT-T
88,44-19.23
61.36++2.1
78.88
98.64
93.01+00.66
86.72
CoDATS
71.45+12.93
61.98+2.88
72.11
96.12.0.86
78.92+12.71
79.05
AdvSKM
69.9743.68
53.73+11.04
35.46:1.02
85.73+6.11
82.29+4.67
71.08
12

---

Under review as a conference paper at ICLR 2022
Table 5: Detailed results of scenarios of WISDM dataset in terms of MF1 score.
Risk
Algorithm
35-+31
7-+18
20-+30
6-+19
18-+23
AVG
DDC
51.84
43.56:10.66
63,83-12.04
62.58:12.72
51.36:110.81
55.03
Deep Coral
56.48:18.18
44.32:11.79
66.33+.217
63.11:3.11
56.9113.24
57.43
HoMM
58.49:47.79
53.6443.48
70.9442.8
74.88=6..55
56.96:t24.07
62.98
MMDA
54.37:5.22
47.75+2.04
66.35+2.1
76.68
74.7=2.28
63.97
Target
DSAN
70.35+1.44
44.77++4.4
69.53+4.91
49.85+4.448
65.88+7.21
60.08
DANN
62.53+6.97
43.55++2.06
53.96+3.03
79.05=15.62
49.96+45.83
57.81
CDAN
60.9348.6
58.981.5.36
60.43+11.19
59.61
49.3+1105
57.85
DIRT-T
68.59:11077
73.5719.09
72.0648.6
50.1447.43
67.03:2.441
66.28
CODATS
50.93115.46
51.211:5.66
67.28
65.2443.68
49.1=
56.75
AdvSKM
55.15+9.56
52.36+2.73
72.83+3.69
58.6444.33
63.78+11.22
60.55
DDC
38.86:114.69
43.3+11.8
56.56.10.33
62.43
49.11
50.05
Deep Coral
39.37=15.11
45.62:44.25
56.8610.72
56.46
48.95:0.19
49.45
HoMM
33.41
30.5223.6
60.99
54.7447.85
51.15#7.39
46.56
MMDA
43.47:4.995
57.911.84
55.942.97
59.8=8.45
43.5248.41
52.12
Few-shot
DSAN
52.92+14.31
51.98:44.33
64.35-1.77
49.2647.009
48.5148.15
53.41
DANN
43.1149984
43.08+1.21
57.63+2.53
55.2-4493
48.23+404
49.45
CDAN
54.53+11.31
57.03+00.53
64.55-1.006
39.76-17.63
47.85:L0.46
52.75
DIRT-T
62.52-11006
68.841794
62.3115.49
51.86-16.68
67.5610.51
62.61
CODATS
40.68-24.005
37.82+2.61
61.01
56.3=8.87
47.38+1.23
48.64
AdvSKM
17.43+12.5
73.58+1.33
71.2+3.2
78.28+3.05
67.78+0.79
49.02
DDC
18.85+16.15
65.17
70.04+L1.28
57.51
42.4+8.53
52.80
DeepCoral
42.36+11.34
47.0747.25
67.1614.86
65.0613.76
47.5748.73
53.85
HoMM
66.29410.84
48.6746.31
65.342.45
63.78
62.1147.57
61.23
MMDA
60.3447.52
41.58+8.79
64.29
55.74+13.88
64.47+100.5
57.30
DEV
DSAN
57.25+6.07
52.77+2.23
63.4
53.35+5..77
55.76+11.6
56.51
DANN
52.21111009
41.16116.62
10.
59.0943.57
48.0+0..
54.48
CDAN
49.02.14.2
57.65
65.5
44.03+10.81
50.16:10.44
53.27
DIRT-T
46.75+3.54
57.89+0.15
65.49-0.62
45.16+0..0
50.9+0.4
53.24
CODATS
40.96+19.00
42.0+3.75
69.6547.6
70.59=12.51
18.15=15.11
54.27
AdvSKM
61.91
49.8445.31
69.35+1.38
54.89+4.14
51.3+10.33
57.46
DDC
51.4745.69
43.65
65,83+2.04
62.58+2.72
51.36=+10.81
54.98
Deep Coral
53.4617.12
43.65
66.08-2.05
63.16+3.114
51.36+10.81
55.54
HoMM
57.94+7.0
43.23++0.3
65.47+1.13
63.91+4.12
56.91+3.24
57.49
MMDA
61.44:6.08
49.79-6.766
67.82+1.37
60.83+0.27
47.78+5.98
57.53
Source
DSAN
57.2516.007
52.77+2.23
63.410.7
53.35.15.37
55.76::1.46
56.51
DANN
46.56-111.22
44.0441.61
68.13
54.41=1.611
52.9212.86
53.21
CDAN
44.25-16.84
57.44
63.67-10.63
47.81
49.27-L0.12
52.49
DIRT-T
72.5944.03
57.7440.06
53.47+2.17
60.91
57,46+8841
60.43
CODATS
74.0614.777
35.8
54.21
45.54+1.33
54.0+11.22
52.72
AdvSKM
16.3+
53.66+55.72
62.59+2.7
60.37+0.81
50.84+2.66
53.95
13

---

Under review as a conference paper at ICLR 2022
Table 6: Detailed results of scenarios SSC dataset in terms MF1 score.
Risk
Algorithm
16-+1
9-+14
12-55
7->18
0-+11
AVG
DDC
55.47::1.72
63.57
55.43-12.75
67.46:t
54.17.1.799
59.22
Deep-Coral
55.5011.774
1.35
55.35
67.49:1..50
53.76+1.89
59.12
HoMM
55.91
63.49#1.14
55.46+22.71
6..50=t1.00
53.37+2.47
59.06
MMDA
62.92:10.6
71.04=2.39
65.84+1.08
70.95
43.23#4.31
62.79
Target
DSAN
59.87+2.84
70.71+2..99
65.55+0.79
68.44+1.39
38.28++3557
60.57
DANN
58.68+33229
64.29+11008
64.65++1.83
69.54+3..00
44.13+.5884
60.26
CDAN
59.65-1499
14.1811.37
64.43
67.61
39.38113.28
59.04
DIRT-T
61.3114.23
66.3914.86
66.95::1.72
70.51
33.05+:2449
59.42
CoDATS
63.84
63.51
52.541:5.94
66.06:12.48
46.28:15.99
58.44
AdvSKM
17.83+11.42
64.76+3.0
55.73+1142
67.58+3.64
55.2+4.19
60.21
DDC
55.48:11.76
63.54
55.3244..4
67.5
54.28:11.68
59.22
Deep-Coral
1..5
63.55+1.33
55.42:42.66
67.53+1.54
52.1
58.82
HoMM
15.91
63.5+1.14
55.46+2..71
1..5
53.37+2.447
59.06
MMDA
65.6310.67
65.9214.44
57.9916.43
71.5+0.97
28.943.78
57.98
Few-shot
DSAN
56.39+0.67
63.85+0063
62.47+2.6
68.92+1.67
43.25++2.76
58.97
DANN
58.68+3..3
64.3+1.08
64.65++1.3
69.54+3.0
44.13+.5844
60.26
CDAN
59.87-2..67
63.55+3.116
62.13
64.12::0.48
31.1948.26
56.17
DIRT-T
56.3315.86
65.15+1.99
64.88115.58
69.83111577
34.99110.54
58.23
CoDATS
59.8440.64
53.02+44553
17.58
55.1243.55
47.6442.4
54.64
AdvSKM
57.68+0.79
64.31+2.93
55.2942.58
67.2243.9
55.16
59.93
DDC
55.53+11877
6..57+1.22
55.35
67.46-11.55
14.1441.77
59.21
Deep-Come
55.5+1.84
63.55+1.33
55.4242.66
17.5
52.1+2.85
58.81
HoMM
55.5742.0
63.66=1148
55.87+2.93
67.49
50.93+4.31
58.70
MMDA
63.44+1.49
67.14
64.93+1.21
71.89+1..44
39.88+4..66
61.49
DEV
DSAN
58.76+22.02
69.45:1400
64.92++1.65
68.69+0.99
37.43+2.9
59.85
DANN
58.78.14.76
64.61
65.47:10.95
68.88422.81
31.13:1.74
57.77
CDAN
60.95.1..1
60.54
65.0+11.4
67.02.11.13
30.79+110.69
56.86
DIRT-T
54.42+12.46
71.33+3.72
64.9914.98
69.94+0.43
35.6243.79
59.26
CoDATS
60.03+1.18
52.22+10.55
56.96:12.4
68.6442.93
41.1=5.14
55.79
AdvSKM
57.8+0.69
64.27+2.93
55.1242.52
67.31
45.11
59.92
DDC
55.48+11.76
63.57
55.1642..76
67.5
54.24
59.18
Deep-Coral
55.5+1.74
63.5+1..6
55.35+2.64
67.5
53.76+1.89
59.12
HoMM
55.51+1.779
63.5+11.14
55.46++2.71
67.5+t1.5
53.37++2.47
59.06
MMDA
59.6+0.51
68.25+4.17
65.63+0.85
71.06+00.99
45.89+1.97
62.08
Source
DSAN
63.05.1.114
63.84110.11
17.55-11.116
68.8441.25
37.46.14.76
58.14
DANN
58.68-1..3
64.3+1.08
64.65-t1.83
69.5443.0
44.13.5.84
60.26
CDAN
62.06-10.91
63.324..022
48.8+1102
63.46.L1.18
36.86-18.23
54.89
DIRT-T
59.1113.24
65.08+1.42
65.5+4.92
67.27+1.55
35.2942.92
58.44
CoDATS
56.5211.76
68.215.72
59.7246.66
63.3113.9
36.05+8.95
56.76
AdvSKM
57.78+00.72
64.29+2.97
55.15+2.552
67.33+33.22
55.16++4.4
59.94
14

---

Under review as a conference paper at ICLR 2022
Table 7: Detailed results scenarios of HHAR dataset in terms of MFI score.
Risk
Algorithm
0-+6
1-+6
2-+7
3
4-+5
AVG
DDC
51.22:t14.11
85.11
48.6115.66
77.43:12.47
1..97
69.8710
Deep-Coral
57.64.1.59
89.81
44.15::0.92
79.3
90.534:.005
72.284:
HoMM
64.85+10.6
89.12#0.61
44.4440..6
80.2
88.73=3.01
73,4740
MMDA
61.6614.66
90.85#0.51
53.3618887
88.07:5.29
91.2411.96
77.0410
Target
DSAN
16.47+10.89
92.77+1.03
61.07+1.46
98.14+0.51
97.26+00.47
81.14+0
DANN
47.02++0557
93.02+1..9
49.06+8.33
95.77+1.91
97.24+0.51
76.42++
CDAN
56.52.1..35
92.410.76
50.76:1..24
93.0949.94
97.67=0.48
78.09
DIRT-T
64.549.4
94.8411.52
59.9
83.26:2.225
97.73+0.447
80.0410
CoDATS
46.45:10.64
92.59+0.71
48.13488.6
96.89+1.97
96.38+t2.42
76.09
ADVSKM
59.39+4.55
81,43+5.52
47.75+3.98
79.05=:042
82.03+3.24
69.93+1
DDC
11.61+1..5
18.46414.46
47.77-44.89
78.4+11.11
79.65+3.13
69.26110
Deep-Coral
59.35+4.8
86.58=6.14
44.8=2.79
77.65+2.23
85.0416.2
70.68
HoMM
54.97=5.29
84.9919.19
11.65+
78.38
83.1219.49
68.62400
MMDA
60.1316.66
84.15+10.34
55.4744.41
80.31
75.26114.3
71.0740
Few-shot
DSAN
52.92+16.13
92.67+1.39
50.85+10.21
97.11+0.39
97.33+0.84
78.18+0
DANN
53.84+6.34
86.38+12.08
57.48+1.661
78.94
91.78++8.26
73.68+00
CDAN
45.52+10.9
92.99+0.7
54.1
98.1740.37
96.39+1..77
77.43-00
DIRT-T
14.88
94.05+1..
64.63:10.3
80.6:10.45
97.9.10668
78.110
CoDATS
44.72+5.11
93.61#0.7
53.33+7.71
93.52#1.67
88.51
74.74
ADVSKM
56.25+7.15
82.68+3.1
45.9145.88
76.62+5.44
83.84+2.96
69.06
DDC
62.61+1.32
73.99+19.45
43.6140.89
76.24
75.174..66
66.32++
Deep-Coral
54.8219.16
89.31
48.4441.98
77.3942811
79.4414.64
69.8810
HoMM
63.5812.24
88,49=2
47.12:4.27
79.23+1.13
84.07=1.19
72.5
MMDA
59.52+33.77
86.53+2.06
48.99-10.42
77.8+22288
78.3+7.36
70.22++0
DEV
DSAN
58.8147.19
93.42+0064
45.61+0.5
98.4440.23
98.47+0.32
78.95
DANN
46.5410.61
90.73+1.97
46.58-13.13
83.43410.12
95.83+0028
72.6240
CDAN
45.52+0.9
92.9940.7
54.147.12
98.1740.37
96.39+1.37
77.43L0
DIRT-T
52.6349.77
93.142.06
63.49+1.95
87.08+10.06
97.13+0.44
78.69+0
CoDATS
14.7
91.98+1.01
47.56+5.004
91.83+4.56
92.52+3.14
73.72
ADVSKM
45.52+0..9
92.9940.7
54.1
98.17:0.37
96.39+1.37
77.4340
DDC
62.18+1.56
79,249.5
44.53
76.65+11.3
75.5344.9
67.6240
Deep-Coral
63.1441.57
88.2743.02
44.59:00.4
78.33
79.55+2.64
70.78
HoMM
63.14+2.11
87.58+2.59
47.27+4.922
77.62+1.67
80.28+0056
71.18+0
MMDA
65.42+1.48
69.07+1.49
41.67+40.96
76.62+3.05
78.2+6..97
66.2+0
Source
DSAN
56.4248.91
9310.54
49.6847.7
84.28-L12.64
97.53+1.33
76.18
DANN
61.04-1..52
91.78-10.61
53.44-12.85
10.95
94-1..8
76.2440
CDAN
45.52+0..9
92.99-0.77
54.141.12
98.17+00.77
96.39+1.37
77.43-0
DIRT-T
17.26410.16
94.06+0.81
57.55+9.64
81.14:0.155
97.7810.78
75.56+1
CoDATS
52.3219.64
93.36+0.11
43.67400.6
91.06+11.21
95.1642.116
75.11
ADVSKM
56.08+22.45
76.28+8.667
38.5449.01
79.73+1..3
82.29+5.25
66.58++0
15

---

Under review as a conference paper at ICLR 2022
B
CLASS DISTRIBUTION OF DIFFERENT SUBJECTS
In this section, we visualize the class distribution of each selected subjects for all the datasets.
36 sarele
2:10
10
2.0
Ihert53
U7e
U3e-2333
5-2
5.0
4.0
3.5
1.0
3.0
5.2
6.0
4.0
3.5
2.0
3.0
4.0
3.0
3.5
3.0
2.0
namber carples
Jaan2233
80
number
Clusmaa
nunber campless
1hae-120 128
namber
4.0
4.0
9.0
3.0
5/7
6.0
2.0
Classes
0.0
3.0
3.0
5.0
1.0
30
(a) HAR dataset
120
U3
Userr
1000
User
100
number of wNTTOiN
60
60
60
00
20
100
Marrz2
1000
Usec28
83
Bumber sample
60
00
of
5
80
40
20
10
22.
125
300
number TTammplless
100
80
63
80
50
WISDM dataset
Figure 4: Class distribution of selected subjects among different datasets
16

---

Under review as a conference paper at ICLR 2022
C HYPER-PARAMETER IMPORTANCE
The importance of each hyper-parameter can be valuable when you have low budget for hyper- parameter tuning. As such you can the most important hyper-parameter while fixing others to specific value. In this work, We also study how different syyper-paammtert can affect the model performance. We test the learning rate against other model specific performance for different do- main adaptation algorithms. To do so, we leverage random forest model and feed the corresponding typer-parameters as input and the target metric as output (Probst et al., 2019). In our case, we av- eraged all the model selection risks and use them as metric to calculate the importance of each parameter.
Figure we study the effect of learning rate, as well the weights of domain alignment loss (differs according to each method) and the source classification loss. We calculate the importance of these parameters while running the sweeps of three different methods i.e., DDC, Deep-CORAL and AdvSKM. We find that the learning rate is the most significant parameter especially with SSC dataset, it contributes with more than 80% of the performance. We conclude that more effort should be put in finding the best learning rate that suites each dataset. In addition, we find that the source classification loss comes next in the importance, and hence, weight should be assigned to it.
learning rate
Source elf.
Domain align. loss
100
DDC
DCORAL
AdSKM
80
HAR
WISDM
SSC
WIIDDM
SSC
HAR
WISDA
SNC
Figure 5: Parameters importance for some selected UDA methods through the three datasets.
17

---

Under review as a conference paper at ICLR 2022
D DETAILED PARAMETER RANGES FOR THE HYPER-PARAMETER SEARCH
Here, we provide the detailed ranges for each parameter among all selected domain adaptation meth- ods.
Table 8: Details of hyper-paramete tuning setup.
Method
Hyperparameter
Range
Learning Rate
10 to 10a
DDC
MMD loss
10-a to 10a
Classification loss
10-1 to 10a
Deep CORAL
Coral loss Classification loss
10-2 10a
10-1 to 10a
HoMM
High-order-MMII MMD loss
10 to 10²
Classification loss
10-1 10
MMDA
MMD loss
10 10
Coral Loss
10-2 to 10
Conditional loss
10-2 10
Classification loss
10-1 to 10a
DSAN
Local MMD loss Classification loss
10 to 10
10 to 10
DANN
MMD loss
10 to 10 10 to 10
Classification loss
CDAN
Adversarial loss
10 10
Conditional loss
10 10 to 10
classification loss
10 to 10
DIRT-T
Adversarial loss
10 10º
Conditional loss
10-2 10a
virtual adversarial Discriminator steps classification loss
to-2
10-2 10
10-1 to 10a
CODATS
Adversarial loss classification loss a
10 to 10
10 to 10
AdvSKM
Adversarial MMD loss
10-2 to 10
Classification loss
10 to 10a
18

---

Under review as a conference paper at ICLR 2022
E BACKBONE NETWORK ARCHITECTURE
Here, we present the detailed architecture of ADATIME backbone network.
Feature Extractor
1D Conv (64, K, s)
BatchNorm
ReLU
1D MaxPool (2,2)
1D Conv (128,8,1)
BatchNorm
ReLU
1D MaxPool (2,2)
1D Conv (128,8,1)
BatchNorm
ReLU
1D MaxPool (2,2)
Classifier
Fully Connected
Figure 6: Backbone network of our ADATIME, where K is the kernel size and is the stride.
F PERFORMANCE STUDY OF TWO DIFFERENT BACKBONE NETWORKS
We added ID-ResNet18 [1] as an additional backbone network. We conducted the experiments on on both small- and large-scale datasets. Table below shows the results of 1D-CNN and ID-Resnet18 on both HAR and HHAR datasets. We found that our conclusions are consistent among different backbone networks and datasets. Altgouh the absolute performance can be different, the relative performance between different UDA methods is preserved.
Table 9; Evaluation Performance of two backbone network on two different datasets
Dataset
Networks
DDC
DCoral
HoMM
MMDA DSAN
DANN
CDAN
DIRT-T
CoDATS
AdvSKM
CNN
69.87
72,28
73.47
77.04
81.14
76.42
78.09
80.04
76.09
69.93
HHAR
ResNet18
65.31
75.08
70.75
CNN
72.71
73.26
83.03
86.81
87.63
83.68
89.07
HAR
ResNet18
70.59
74.53
79.
78.19
79.71
74.05
79.13
80.745
77.75
71.77
19


Reviews:
[{"review_id": "vzWd_d9xda7", "reviewer": "Reviewer_oysD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper comprehensively adapts the state-of-the-art UDA methods to time series classification tasks. \n2. The paper proposes a novel UDA framework for time series data, which may contribute to the community.\n3. The idea of this paper is novel to some extent.\n\nCons:\n1. My main concern is the motivation. The paper only talks about how the framework is designed, while not elaborating clearly on why the framework is designed like this.\n2. The selected datasets are relatively too small and simple that the backbone network can only be a 1D-CNN in order to avoid the overfitting phenomenon. The experimental results on these toy time-series datasets are unconvincing.\n3. The organization and writing of this paper should be improved. The authors should pay more attention to the motivation and the details of the framework. \n4. It would be better if the authors explain some symbols like $X_{train}^{S}$, $X_{test}^{S}$, $Z_{train}^{S}$, $Z_{test}^{S}$, ..., etc. in Figure \n5. Some related works are missing: \n\n[1] Ruichu Cai, Jiawei Chen, Zijian Li, Wei Chen, Keli Zhang, Junjian Ye, Zhuozhang Li, Xiaoyan Yang, Zhenjie Zhang:\nTime Series Domain Adaptation via Sparse Associative Structure Alignment. AAAI 2021\n[2] Xiaoyong Jin, Youngsuk Park, Danielle C. Maddix, Yuyang Wang, Xifeng Yan. Domain Adaptation for Time Series Forecasting via Attention Sharing. arXiv:2102.06828\n\nMinors:\nOn page 2, “the the following questions\" -> \"the following questions\"", "summary_of_the_paper": "The paper proposes a systematic evaluation framework named ADATIME, which systematically evaluates different unsupervised domain adaptation methods on time series data. The whole framework consists of a feature extractor, a classifier, and a domain alignment component. The paper conducts large-scale experiments adapting the state-of-the-art visual domain adaptation methods to the proposed framework on time series classification tasks. The findings based on the experimental results reveal the key points of applying UDA to time series data.", "summary_of_the_review": "The findings are interesting and may contribute to the community. However, considering the motivation and the unconvincing experiments, I vote for \"5: marginally below the acceptance threshold\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "irSArXO3YvI", "reviewer": "Reviewer_S7S5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: A systematic experimental approach  for time series domain adaptation using multiple datasets and baseline architectures.  Comparisons between visual domain adaptation methods and time series-unsupervised domain adaptation methods are given.  Guidelines for future research are given. ", "summary_of_the_paper": "This paper presents an empirical approach for unsupervised domain adaptation of time series data. The paper points out some of the drawbacks of existing approaches due to inconsistencies in evaluation schemes, datasets, model selection rules, and base neural network architectures. The paper then presents adaptations of visual domain adaptation methods for time series data. Experimental results using ten state-of-the-art methods on three benchmark datasets spanning fifteen cross-domain scenarios are presented.", "summary_of_the_review": "While the authors' efforts in systematic experimental evaluation of existing visual domain adaptation algorithms and time series-unsupervised domain adaptation algorithms is commendable, there is no theoretical understanding at even the most basic level. hence the conclusions drawn from the experiments may be specific to datasets and baseline architectures used. The conclusions do not generalize.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P3BKBEvNJf8", "reviewer": "Reviewer_uoXZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "“Deep learning has achieved a great success in time series classification tasks, assuming access to a vast amount of labeled data for training” I am not totally convinced that this is true. The improvements over simple nearest neighbor models etc is not great. \nIn the related domain of anomaly detection, a recent paper makes a forceful claim that most of the apparent success of Deep learning for time series anomaly detection is nonsense [a]\n\nThe paper contains a nice “bake off”, but the main finding “Visual UDA methods achieve comparable performance to TS-UDA methods on time series data.” is unsurprisingly.\n\nI am very curious about how well one could do with a much simpler methods.\nFor HAR the class are WALKING, WALKING_UPSTAIRS, WALKING_DOWNSTAIRS, SITTING, STANDING, LAYING. It is easy to tell LAYING from any of the dynamic classes, simply by the fact that the variance of LAYING is less than one tenth of the dynamic classes. It is easy to tell LAYING from the other classes because G (the acceleration due to gravity), sifts from one axis to another etc. \nI understand that simply maximizing accuracy is not the full point of the paper, but I am still curious of we really need deep learning here.\n\n\n“In addition, we find that model selection plays a key role and different selection strategies can significantly affect performance.” It would be surprising of that was NOT true.\n\nHowever, I do think the experiments as detailed and forceful, and the community may find them useful\n\n\n[a] Renjie Wu, Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress.\n", "summary_of_the_paper": "The paper introduces , a standardized framework to systematically and fairly evaluate different domain adaptation methods on time series data.", "summary_of_the_review": "While the novelty is low, the experiments as detailed and forceful, and the community may find them useful\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Iybx6FWi82G", "reviewer": "Reviewer_f7Xp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n(1) This paper evaluates the previous domain adaption algorithms under a fair setting. Extensive experiments are provided. Also, the experiment results present some competitive baselines to the TS-UDA area. This benchmark will be helpful to future research.\n\n(2) Based on the experiment results, this paper provides some findings or analyses.\n\n(3) The paper is well organized and with clear clarification.\n\n## Weaknesses\n(1) The main concern is the technological novelty of this paper. This paper adopts previous methods in the time series datasets, which is technologically trivial. Especially, the proposed AdaTime framework is also trivial. Can you further clarify the difference between your AdaTime framework and the standard domain adaption protocol in image classification? It is hard for me to distinguish them. That is why I think the AdaTime is not novel. More elaborations will be very helpful for my judgment.\n\n(2) Some of the findings are also fragile. \n- Performance Gap diminishes with a sufficient amount of data: this clarification is not persuasive for me. Your conclusion is obtained from the comparison among three different datasets. It would be helpful if you controlled the variables. For example, you can keep enlarging the size of one fixed dataset and record the change of results.\n- Model selection has a significant effect on performance. I think your experiment results are also affected by the limited data. The three experiment datasets are too small to provide a robust result. Larger datasets should be included, such as the HHAR dataset used in CoDATS.\n\n(3) I am not sure about the contribution of this paper when the backbone is just CNN. Firstly, as shown in Figure 3 of the main text, different backbones result in quite different results. Not only the numerical values but the relative performance is also changed. So I think the findings of your paper may also be changed under other base models. Secondly, as your mentioned, the time series area does not have a consistent backbone. Thus, the experiments are lacking in the backbone aspect. In conclusion, I think more baselines are needed if you want to obtain a general conclusion, such as the TCN.\n\n(4) As for the metric, F1-score is a well-established convention in the time series area, such as anomaly detection and recommended system. Maybe F1-score is not popular in vision, but I still think the metric finding is too trivial.", "summary_of_the_paper": "This paper explores the unsupervised domain adaptation of time series data (TS-UDA). And it focuses on the benchmark construction. By standardizing the base model, datasets, and model selection, this paper provides a good benchmark of TS-UDA. This benchmark can facilitate future research. Also, the paper proposes some findings.", "summary_of_the_review": "\nThis paper provides a useful benchmark for the TS-UDA task. Extensive experiments are included. But because of the concern of technology novelty and the unconvincing evidence of some findings, I would like to reject it.\n\n--- after discussion---\n\nThe author addressed part of my question. But I still think this paper is below the expectation. I would like to arise the rank to 5.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DGqEmLbOdrh", "reviewer": "Reviewer_X8V1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- include extensive baseline for the experiment\n- open-sourced the code\n\n\nWeakness:\n- did not clarify what is the specific challenge in time series data beyond static data, and how to remedy them as contributions?\n- does not cover other main time series applications: time series forecasting scenarios.\n- did not cover or compare with the recent work (DAF) of domain adaptation for time series by Jin et al. https://arxiv.org/abs/2102.06828 \n--  did not justify well why domain alignments are important well. In DAF, it mentioned having both domain specific and invariance features are critical.\n-- what if using different backborns, such as attention-based ones?\n- hard to understand why many criteria are proposed. what is difference between SCC and DEV risk? when to select one of them?\n- difficult to interpret the experiment results\n-- what is visual UDA in the table 3? \n-- why were average values reported?\n- the observations are not much informative \n-- one on imbalanced data is not connected to model selection criterions. \n-- one of backborn selection is kind of implying non-systemtic aspect. What is the real take-away out of this? may it recommend to use other possible backborn, e.g., attention-based one?\n- hard to understand why authors emphasize fair and realistic procedure. Are any of other methods not fair or realistic and why?\n- ", "summary_of_the_paper": "This work proposed a systematic framework of (unsupervised) domain adaptation for time series data, with various model selections and strategies.", "summary_of_the_review": "The paper is not well-written especially experiments parts, and the message of our them are not clear in the line of overall paper take-away.Therefore, the contribution seems to be not clear beyond aggregating all existing DA methods and systematically deploying them. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work aims at giving a systematic evaluation of different unsupervised domain adaptation methods on time series classification tasks under a fair setting. By providing extensive experiments on various datasets, competitive baselines, and model selection approaches, this paper has the potential to facilitate future research on this topic if the mentioned concerns are well addressed.

After rebuttal and discussion, the final scores were 3/5/5/5/5. AC considered all reviews, author responses, and the discussions, as well as reading through the paper as a neutral referee, and reject the paper based on the following concerns:
+ *Model Selection Criterion*: As stated by the authors, employing labeled target data for model selection will violate the fundamental assumption of unsupervised domain adaptation. However, the proposed Few-Shot Target Risk (FST Risk) also requires labeling a few target domain samples. If it is possible, why not directly conduct semi-supervised domain adaptation?
+ *Experiment Details*: As a benchmark paper, it is extremely important to carefully design the experiment details to attain promising results. Among these details, a suitable network backbone for time series classification (Is CNN or ResNet-18 the best choice?  Or TCN mentioned by Reviewer f7Xp), large-scale datasets with considerable domain gap, and evaluation metrics are the first consideration to attain insightful findings.
+ *Novelty or Interesting Findings*: As pointed out by reviewers, it is obvious that the technical novelty is limited but it may be okay for a benchmark paper if solid/interesting experimental results are observed. However, some of the findings are also fragile and the experiments should be carefully conducted to make them more solid.

In summary, this paper studies a promising research direction of domain adaptation, but the work cannot be accepted before addressing the reviewers' comments. The weaknesses mentioned above will have a high probability of being asked by the reviewers of the next conference. So the authors need to make sure that they substantially revise their work before submitting it to another venue.
Decision: Reject