Year: 2021

Abstract:
Unsupervised domain adaptation methods aim to generalize well on unlabeled test data that may have a different (shifted) distribution from the training data. Such methods are typically developed on image data, and their application to time series data is less explored. Existing works on time series domain adaptation suffer from inconsistencies in evaluation schemes, datasets, and backbone neural network architectures. Moreover, labeled target data are usually employed for model selection, which violates the fundamental assumption of unsupervised domain adaptation. To address these issues, we propose a benchmarking evaluation suite (ADATIME) to systematically and fairly evaluate different domain adaptation methods on time series data. Specifically, we standardize the backbone neural network architectures and benchmarking datasets, while also exploring more realistic model selection approaches that can work with no labeled data or just few labeled samples. Our evaluation includes adapting state-of-the-art visual domain adaptation methods to time series data in addition to the recent methods specifically developed for time series data. We conduct extensive experiments to evaluate 10 state-of-the-art methods on 4 representative datasets spanning 20 cross-domain scenarios. Our results suggest that with careful selection of hyper-parameters, visual domain adaptation methods are competitive with methods proposed for time series domain adaptation. In addition, we find that hyper-parameters could be selected based on realistic model selection approaches without relying on labeled samples from target domain. Our work unveils practical insights for applying domain adaptation methods on time series data, and builds a solid foundation for future works in the field.

heading: 1 INTRODUCTION
text: Time series classification problem is predominant in many real-world applications including healthcare and manufacturing. Yet, time series data with its inherit temporal dynamics and complex underlying pattern can be more challenging than static data. Recently, deep learning has gained more attention in time series classification tasks, assuming access to a vast amount of labeled data for training (Fawaz et al., 2019). However, annotating time series data can be challenging and burdensome due to its complex underlying patterns and complicated domain knowledge (Chang et al., 2020). One way to reduce labeling burden is to leverage annotated data (e.g., synthetic or public data) from a relevant domain (i.e., source domain) to train the model, while testing it on the domain of interest (i.e., target domain). Nevertheless, deep learning tends to perform poorly when tested on unseen data that have different distribution from the training data, which is well-known as the domain shift problem. Considering time series applications, source and target domains usually represent data from different subjects (persons) as in human activity recognition (HAR) (Wilson et al., 2020; Chang et al., 2020) or sleep stage classification (SSC) tasks (Phan et al., 2020). A considerable amount of literature has been developed on Unsupervised Domain Adaptation (UDA) for visual applications to mitigate the domain shift problem (Long et al., 2015; Ganin et al., 2016; Long et al., 2018; Chen et al., 2020; Rahman et al., 2020).
Recently, more attention has been paid to time series UDA (TS-UDA) (Wilson et al., 2020; Chang et al., 2020; Ragab et al., 2020; Liu & Xue, 2021). However, the literature of TS-UDA methods suffers from the following limitations:
• Existing TS-UDA works lack consistent evaluation schemes including benchmark datasets, preprocessing, and backbone networks. For instance, some methods that leverage recurrent neural network as backbone network (Purushotham et al., 2017) are compared against methods with convolutional based backbone networks (Wilson et al., 2020). In addition, even with a similar architecture, training procedures can also vary among different algorithms including number of epochs, weight decay, and learning rate schedulers (Tonutti et al., 2019; Purushotham et al., 2017).
• Existing TS-UDA works use labeled data from the target domain for model selection, violating the basic assumption of UDA (Wilson et al., 2020; Liu & Xue, 2021), and providing an overoptimistic view of their performance. It is worth noting that model selection for domain adaptation in the absence of target domain labels is a long standing problem.
• Most of existing algorithms are mainly application specific, and few works have been proposed for general TS-UDA. As a result, there is a shortage of baseline methods when applying domain adaptation on time series data.
All the aforementioned challenges can highly affect the performance and can be mistakenly attributed to the proposed domain adaptation methods.
In this work, we propose a systematic evaluation suite (ADATIME) to tackle the aforementioned obstacles and remove all extraneous factors to ensure a fair evaluation of different UDA algorithms on time series data. Specifically, to address the inconsistent evaluation schemes and backbone networks, we first standardize the preparation and processing of four benchmarking datasets from two classic real-world applications, including healthcare and human activity recognition. Besides, we unify the backbone network and the training procedures when comparing between different UDA methods. Second, to select the model hyper-parameters in the absence of target labels, we explore more realistic model selection strategies for TS-UDA problem that do not require target labels. Particularly, we investigate the model performance when selecting models based on source dataset (i.e., source risk (Ganin et al., 2016)), unlabeled target data (i.e., deep embedded evaluation (DEV) (You et al., 2019)), or only few-shot labeled samples from target data. Last, to address the lack of TSUDA baselines, we re-implement various state-of-the-art visual UDA methods that can be adapted to time series data while comparing to the existing TS-UDA methods
Given our standard methodology, we aim to systematically study the following questions: (1) With standard backbone network and evaluation schemes, how will the visual UDA methods perform on time series data; (2) Can we use realistic model selection methods —relying on only few or no target labels— and still achieve acceptable adaptation performance on time series data; (3) How can the backbone network contribute to the performance. In this paper, we conduct comprehensive experiments to answer the aforementioned three questions. Some of our findings are summarized as follows:
• Visual domain adaptation methods can achieve comparable or even better performance than the methods proposed for time series data.
• Unlike image data, selecting models based on source risk can achieve reasonable performance and outperforms the DEV risk for time series data. Additionally, we find that our proposed few-shot target risk can achieve comparable performance to target risk, with affordable few labels samples.
• Changing the base architecture can be pivotal to the performance. Moreover, complex architectures with time series data can have lower generalization performance on crossdomain experiments.

heading: 2 DOMAIN ADAPTATION PROBLEM
text: We start by defining the unsupervised domain adaptation problem. We assume the access to a labeled source domain XS = {(xsi , ysi )} NS i=1 that represents uni-variate or multivariate time series data, and unlabeled target domain XT = {(xTi )} NT i=1 where NS and NT denote the number of samples for XS and XT respectively. The source and target domains are sampled from different marginal distributions, i.e., PS(x) 6= PT (x), while the conditional distribution remains stable (PS(y|x) =
PT (y|x)). The main goal of UDA is to reduce the distribution shift between PS(x) and PT (x), assuming they share the same label space.
The mainstream of UDA algorithms is addressing the domain shift problem by finding domain invariant feature representation. Formally, given a backbone model fθ : X → Z, which transforms the input space to the feature space, the UDA algorithm mainly optimizes the backbone network to find new invariant representation for the target features such that Ps(fθ(x)) = Pt(fθ(x)). As a results, a model trained on the source domain can generalize well to the adapted target domain.

heading: 3 ADATIME: A BENCHMARKING APPROACH FOR TIME SERIES DOMAIN ADAPTATION
text: In this work, we systematically evaluate domain adaptation algorithms on time series data, ensuring fair and realistic procedures. Figure 1 shows the details of ADATIME, which proceeds as follows. Given a dataset, we first apply our standard time series preparation schemes on the selected source and target domains. Subsequently, the source train data XStrain and target train data X T train are passed through the backbone network to extract source and target features, i.e., ZStrain and Z T train respectively. The selected UDA algorithm is then applied to mitigate the distribution shift between the extracted features of two domains. Last, to set the hyper-parameters of the UDA algorithm, we consider three practical model selection approaches (i.e., without any target domain labels or only few-shot labeled samples) namely source (SRC) risk, deep embedded evaluation (DEV) risk, and few-shot target (FST) risk. The proposed evaluation pipeline can prevent any extraneous factors and enable fair comparison between different UDA methods. The code of ADATIME will be made publicly available for researchers to enable seamless evaluation of different domain adaptation methods on time series data.

heading: 3.1 BACKBONE NETWORK
text: The general domain adaptation network consists of a feature extractor, a classifier, as well as different components to align the domains. We refer to the feature extractor and the classifier as the backbone network. The backbone network can have a pivotal contribution to the UDA model performance, as using different architectures may result in different performance. Hence, changing the backbone network when comparing different UDA methods can result in misleading outcomes,
which can hinder the fair evaluation protocol. In most visual applications, ResNet-50 is the dominantly adopted feature extractor, as its complex architecture can extract representative features from images. However, in time series, using such complex architectures can lead to overfitting problems (Fawaz et al., 2019). Therefore, we use a 1D-CNN architecture in ADATIME (see Figure 6). In addition, we study the effect of different backbone architectures on the performance of TS-UDA methods.

heading: 3.2 DOMAIN ADAPTATION ALGORITHMS
text: While numerous UDA approaches have been proposed to address the domain shift problem (Zhao et al., 2020b), a comprehensive review of existing UDA methods is out of our scope. Instead, we only included the most solid and recent baselines for visual UDA that can be adapted to time series. Besides, we included the state-of-the-art methods proposed for time series data. Overall, the selected algorithms in ADATIME can fit into two main categories according to the domain adaptation method, namely discrepancy based methods and adversarial based methods. The former aims to minimize a statistical distance between source and target features to mitigate the domain shift problem (Tzeng et al., 2014; Sun & Saenko, 2016; Chen et al., 2020). The latter leverages a domain discriminator network that enforces the feature extractor to produce domain invariant features (Ganin et al., 2016; Tzeng et al., 2017). Each method in the above categories can also be classified according to the aligned distribution. Specifically, some algorithms only align the marginal distribution of the feature space, while other algorithms jointly align the marginal and conditional distributions.
The selected UDA algorithms are as follows: Deep Domain Confusion (DDC Tzeng et al. (2014)); Higher-order Moment Matching (HoMM Zhao et al. (2020a)); Correlation Alignment via Deep Neural Networks (Deep-CORAL Sun et al. (2017)); Minimum Discrepancy Estimation (MMDA Rahman et al. (2020)); Domain-Adversarial Training of Neural Networks (DANN Ganin et al. (2016)); Conditional Adversarial Domain Adaptation (CDAN Long et al. (2018)); Deep Subdomain Adaptation (DSAN Zhu et al. (2021)); (DIRT-T Shu et al. (2018)). In addition, we also include two UDA methods applied in time series classification, which are CoDATS (Wilson et al., 2020) and AdvSKM (Liu & Xue, 2021), noting that there are very few TS-UDA methods in the literature. We excluded the methods proposed for time series prediction/forecasting since they are out of our scope. Table 1 shows a summary of the selected methods and their corresponding categories.

heading: 3.3 MODEL SELECTION APPROACHES
text: Model selection and hyper-parameter tuning for UDA are long standing non-trivial problems due to the absence of target domain labels. However, we find that papers describing 5 out of the 10 included UDA methods use target domain labels to select hyper-parameters, as shown in Table 1, which violates the basic assumption of UDA. In addition, another three papers used fixed hyper-parameters, without describing how these parameters were selected. This issue can lead to unrealistic performance and unfair evaluation. To address this issue, we evaluate multiple realistic model selection approaches that could be applied for TS-UDA without the need of any target domain labels, such as: SRC risk (Ganin et al., 2016) and DEV risk (You et al., 2019). Besides, we design a FST risk, which utilizes affordable few labeled samples from the target domain. Detailed explanation of each risk can be found as follows:
Source Risk (SRC Risk) In this approach,we select the hyper-parameters that achieve the minimum risk on the source domain test set, which is readily available. Hence, this approach can be easily applicable without additional labeling efforts, since it relies on the existing labeled source data (Ganin et al., 2016). However, the effectiveness of the source risk is mainly controlled by the sample size of source data and severity of distribution shift.
Deep Embedded Validation (DEV Risk) This approach tries to yield unbiased estimation to the target risk. Its key idea is to consider the relationship between the source and target data when calculating the risk. Particularly, DEV risk considers the source features that are highly correlated to the target features via importance weighting schemes. By ranking candidate models based on their computed DEV risks, the model with the smallest DEV risk is chosen as the best candidate model to be used for adaptation. However, this approach is computationally expensive and may have unstable performance with smaller source and target datasets.
Few-Shot Target Risk (FST Risk) We propose few-shot target risk as a relaxed target risk. While labeling of vast amount of time series data can be laborious, annotating few-shot samples can still be affordable. We use these samples as a validation set to select the best hyper-parameters. This strategy can achieve similar performance to using target risk, but it does require labeling a few target domain samples.
Target Risk (TGT Risk) This approach involves leaving out a subset of the target domain samples and their labels as a validation set, and using them to select the best performing hyper-parameters on the target domain. Using this model selection technique provides an upper bound on the performance of the UDA method, as it leverages labeled data from the target domain. Even though this risk is impractical in unsupervised settings, it has been used by many UDA papers for model selection (Shu et al., 2018; Saito et al., 2017).

heading: 4 EXPERIMENTS

heading: 4.1 BENCHMARKING DATASETS
text: We evaluate the various UDA algorithms on four benchmark datasets from two real-world applications, namely human activity recognition and sleep stage classification. The benchmark datasets span a range of different characteristics including complexity, type of sensors, samples size, classdistribution, and severity of domain shift, enabling more broad evaluation. The selected datasets are detailed as follows:
UCIHAR (Anguita et al., 2013) UCIHAR is one of the most widely used datasets to evaluate performance on time series data. It contains three different sensors namely, accelerometer, gyroscope, and body sensors. These sensors have been used to collect data from 30 different persons. In our experiments, we treat each subject as a separate domain. Due to the large number of cross-domain combinations, we randomly selected five cross domain scenarios, as in (Liu & Xue, 2021; Wilson et al., 2020).
WISDM (Kwapisz et al., 2011) WISDM is another popular activity recognition dataset for the evaluation of time series domain adaptation. In this dataset, accelerometer sensors were applied to collect data from 36 subjects. This data can be more challenging because of the class imbalance issue among different subjects. Particularly, some subjects may contain only samples from a subset of the overall classes. Further detailed about ratio of different classes among subjects can be found in the Appendix. Similar to UCIHAR dataset, we consider each subject as a separate domain and we randomly select five cross-domain scenarios.
SSC (Goldberger et al., 2000) Sleep stage classification (SSC) problem aims to classify the electroencephalography (EEG) signals into five stages i.e. Wake (W), Non-Rapid Eye Movement stages (N1, N2, N3), and Rapid Eye Movement (REM). We adopted Sleep-EDF dataset (Goldberger et al., 2000), which contains EEG readings from 20 healthy subjects. We selected a single channel (i.e., Fpz-Cz) following previous studies (Eldele et al., 2021a), and 10 different subjects to construct five subject-wise cross-domain scenarios.
HHAR (Stisen et al., 2015) The Heterogeneity Human Activity Recognition (HHAR) dataset has been collected from 9 different users using sensor readings from smartphones and smartwatches. In our experiments, we consider each user as a domain. We constructed 10 cross-domain scenarios from randomly selected users. We used the same smartphone device for all the selected users to reduce the heterogeneity.
Table 2 summarizes the details of each dataset, e.g., the number of selected domains, the number of sensor channels, the number of classes, the length of each sample, as well as the total number of samples in both training and test portions.

heading: 4.2 EXPERIMENTAL SETUP
text: Hyper-parameters Sweep For each algorithm and dataset combination, we conducted extensive random hyper-parameter search with 100 trials. The hyper-parameters are picked by a uniform sampling from a range of predefined values. Details about the specified ranges can be found in Table 8. In addition, for each set of hyper-parameters, we calculated the risk values over three different random seeds, removing the bias to a single seed. We picked the model that achieves the minimum risk value for each model selection strategy. For few-shot target risk experiments, we used five samples per class in each dataset to compute the risk.
Datasets Preprocessing The preprocessing of time series data includes data slicing, train/test splitting, normalization. To promote fair evaluation, we preserved consistency among all the aforementioned processing steps when comparing between different UDA algorithms. We used a sliding window of 128 for human activity recognition datasets. For SSC dataset, we kept the original sample length of 3000 time steps. Next, we split both source and target domains into train/test splits with a ratio of 0.7/0.3. Finally, we normalized both training and testing data based on the training statistics (Wilson et al., 2020; Liu & Xue, 2021). It worth noting that WISDM dataset is severely imbalanced among different subjects (see Figure 4(b) in the appendix), with some classes not being available for some subjects. To stabilize the training, we selected subjects that contain samples from all the classes when constructing the source and target domains.
Backbone Network To ensure a fair evaluation among all the baselines, we used a fixed backbone network among all the compared UDA methods. We employed a 3-layer 1D-CNN with an adaptive average pooling, to handle signals with different lengths, as a feature extractor. This architecture is commonly used in series applications (Wilson et al., 2020; Eldele et al., 2021b). We adopted a single layer fully connected network as the classifier. Yet, to include CoDATS in our experiments, we used a 3-layer fully connected classifier (as proposed in their work) to differentiate it from DANN.
Training Procedure All the training procedures have been standardized across all UDA algorithms. For instance, we trained each models for 40 epochs, as performance tends to decrease with longer training. We reported the model performance after completing the last epoch. Regarding the optimization of the model, we used Adam optimizer with a fixed weight decay of 1e-4 and (β1, β2) = (0.5, 0.99). The learning rate was set to be a tunable hyper-parameter for each method on each dataset. We excluded any learning rate scheduling schemes from our experiments to ensure that the contribution is mainly attributed to the selected UDA algorithm.

heading: 5 RESULTS DISCUSSION
text: We conducted extensive experiments for each UDA method on all the datasets. The average F1score of the five cross-domain scenarios for each dataset-algorithm combination is reported in Table 3. We also show the results of each risk to compare the performance to the target risk. Detailed versions of Table 3 can be found in Tables 4, 5, and 7 in the appendix. We further discuss the main findings in the following paragraphs.
Visual UDA methods achieve comparable performance to TS-UDA methods on time series data. Table 3 shows the performance of the adapted visual UDA methods along with the existing TS-UDA methods. Surprisingly, with standardizing the backbone network, we find that visual UDA methods achieve competitive or even better performance than TS-UDA methods. This finding is consistent for all the model selection strategies across the benchmarking datasets. A possible explanation is that all selected UDA algorithms are applied on the vectorized output features of the backbone network, which can be independent from the input data modality. This finding suggests that with a standard backbone network, visual UDA algorithms can be strong baselines for TS-UDA.
Methods with joint distribution alignment tend to perform consistently better. As shown in Table 1, methods that address the joint distributions (i.e., both marginal and conditional distributions concurrently) such as DIRT-T, MMDA, and DSAN, achieve the state-of-the-art performance on the four datasets, outperforming TS-UDA methods. For example, the best performing method, as selected by the target risk, is DIRT-T in both UCIHAR and WISDM datasets, and DSAN in SSC and HHAR datasets. Similarly, with respect to the different risks, DIRT-T, MMDA, and DSAN are interchangeably achieving performing best across the benchmarking datasets. Hence, considering the conditional distribution when aligning the source and target can be beneficial to the performance.
Model selection has a significant effect on the performance. Model selection strategies can yield different performance for the same UDA algorithm, as shown in Table 3. This reveals the significant contribution of the model selection approach to the overall performance. While target risk achieves superior performance, it is infeasible for practical scenarios with the absence of target
domain labels. Surprisingly, source risk can achieve a comparable performance to the target risk on UCIHAR, HHAR and SSC datasets. In addition, with affordable labeling efforts, our proposed few-shot target risk can also achieve a competitive performance to the target risk. However, for severely imbalanced WISDM dataset, different model selection approaches fail to follow the target risk performance.
Backbone network can be pivotal to the performance. To investigate the impact of the backbone network on the method’s performance, we evaluate all the UDA algorithms under two different backbone networks for both small-scale and large-scale datasets, i.e., UCIHAR and HHAR. Particularly, we employ 1D-CNN (described in Section 4.2) and 1D-ResNet-18 (i.e., a standard ResNet-18 with 1D convolutional kernel) as backbone networks. The two architectures are different in terms of their complexity and the number of trainable parameters. Figure 3 shows a comparison of different UDA algorithms under two backbone architectures. Clearly, the two backbone networks can have different absolute performance for each UDA methods. Nevertheless, the relative performance between different UDA methods can still be consistent when fixing the backbone network. For instance in Figure 3b, the DSAN and DIRT approaches perform best for both 1D-CNN and 1D-ResNet-18 backbone networks. Besides, it is still clear that visual UDA methods perform better than TS-UDA methods with the two different backbone networks.
Accuracy metric should not be used to measure performance for imbalanced data. While it is well-known that accuracy is not a representative metric for class-imbalanced datasets, existing TSUDA methods are still using it to report their performance (Chang et al., 2020; Wilson et al., 2020; Liu & Xue, 2021). We aim to re-emphasize that the accuracy metric can give over-optimistic results when considering the imbalanced nature in most time series data. For example, in Figure 2(b), although CDAN achieves higher accuracy than some other methods such as DDC, MMDA and DSAN, it performs the worst in terms of F1-score. In contrast, with a balanced dataset (i.e., UCIHAR), accuracy can still be representative and achieve similar performance to F1-Score, as shown in Figure 3(a).

heading: 6 CONCLUSIONS AND RECOMMENDATIONS
text: In this work, we provided ADATIME, a systematic evaluation methodology for evaluating the existing domain adaptation methods on time series data. To ensure fair and realistic evaluation, we standardized the benchmarking dataset, evaluation schemes, and backbone networks among different domain adaptation methods. Moreover, we explored more realistic model selection approaches that can work with without any target domain labels or only few-shot labeled samples. Based on our systematic study and unveiled findings, we suggest the following recommendations:
• Future research on time series domain adaptation should consider visual UDA methods as strong baselines.
• Considerable amount of target labels should not be used for the UDA model selection. Instead, source risk or few-shot target risk can be better candidates for a more realistic model selection approach.
• The backbone network should be fixed when comparing between different UDA methods. • Larger datasets should be considered when comparing baselines to obtain reliable results. • Accuracy metric should no longer be used to evaluate imbalance datasets, as it can yield
misleading results.

heading: A DETAILED RESULTS OF OUR FRAMEWORK
text: Here, we provide the detailed results including mean and standard deviation for each cross-domain scenario. It can be clearly seen, that datasets with small sample size suffers from high variance among the different cross-domain scenarios.

heading: B CLASS DISTRIBUTION OF DIFFERENT SUBJECTS
text: In this section, we visualize the class distribution of each selected subjects for all the datasets.
6.0 1.0 5.0 4.0 2.0 3.0 Classes
0 10 20 30 40 50 nu m be r o f s am pl es
User:6.0
1.0 5.0 6.0 2.0 4.0 3.0 Classes
0 10 20 30 40 50 nu m be r o f s am pl es
User:7.0
1.0 6.0 2.0 4.0 5.0 3.0 Classes
0
20
40
60
nu m
be r o
f s am
pl es
User:11.0
5.0 6.0 4.0 2.0 1.0 3.0 Classes
0
20
40
60
80
nu m
be r o
f s am
pl es
User:16.0
5.0 6.0 4.0 1.0 2.0 3.0 Classes
0
20
40
60
80
nu m
be r o
f s am
pl es
User:17.0
6.0 4.0 5.0 1.0 3.0 2.0 Classes
0
20
40
60
nu m
be r o
f s am
pl es
User:23.0
1.0 5.0 6.0 2.0 4.0 3.0 Classes
0
20
40
60
nu m
be r o
f s am
pl es
User:25.0
6.0 5.0 4.0 1.0 2.0 3.0 Classes
0
20
40
60
80
nu m
be r o
f s am
pl es
User:28.0
1.0 5.0 6.0 2.0 3.0 4.0 Classes
0
20
40
60
nu m
be r o
f s am
pl es
User:2.0
1.0 6.0 4.0 2.0 5.0 3.0 Classes
0
10
20
30
40
50
nu m
be r o
f s am
pl es
User:9.0
5.0 6.0 2.0 4.0 1.0 3.0 Classes
0
20
40
60
nu m
be r o
f s am
pl es
User:12.0
6.0 1.0 5.0 2.0 4.0 3.0 Classes
0
20
40
60
nu m
be r o
f s am
pl es
User:13.0
(a) HAR dataset

heading: C HYPER-PARAMETER IMPORTANCE
text: The importance of each hyper-parameter can be valuable when you have low budget for hyperparameter tuning. As such you can tune the most important hyper-parameter while fixing others to specific value. In this work, We also study how different hyper-parameters can affect the model performance. We test the learning rate against other model specific performance for 5 different domain adaptation algorithms. To do so, we leverage random forest model and feed the corresponding hyper-parameters as input and the target metric as output (Probst et al., 2019). In our case, we averaged all the model selection risks and use them as a metric to calculate the importance of each parameter.
Figure 5, we study the effect of learning rate, as well as the weights of domain alignment loss (differs according to each method) and the source classification loss. We calculate the importance of these parameters while running the sweeps of three different methods i.e., DDC, Deep-CORAL and AdvSKM. We find that the learning rate is the most significant parameter especially with SSC dataset, as it contributes with more than 80% of the performance. We conclude that more effort should be put in finding the best learning rate that suites each dataset. In addition, we find that the source classification loss comes next in the importance, and hence, more weight should be assigned to it.

heading: D DETAILED PARAMETER RANGES FOR THE HYPER-PARAMETER SEARCH
text: Here, we provide the detailed ranges for each parameter among all selected domain adaptation methods.

heading: E BACKBONE NETWORK ARCHITECTURE
text: Here, we present the detailed architecture of ADATIME backbone network.

heading: F PERFORMANCE STUDY OF TWO DIFFERENT BACKBONE NETWORKS
text: We added 1D-ResNet18 [1] as an additional backbone network. We conducted the experiments on on both small- and large-scale datasets. Table below shows the results of 1D-CNN and 1D-Resnet18 on both HAR and HHAR datasets. We found that our conclusions are consistent among different backbone networks and datasets. Altgouh the absolute performance can be different, the relative performance between different UDA methods is preserved.

Review summaries:
This paper explores the unsupervised domain adaptation of time series data (TS-UDA). And it focuses on the benchmark construction. By standardizing the base model, datasets, and model selection, this paper provides a good benchmark of TS-UDA. This benchmark can facilitate future research. Also, the paper proposes some findings.

The paper proposes a systematic evaluation framework named ADATIME, which systematically evaluates different unsupervised domain adaptation methods on time series data. The whole framework consists of a feature extractor, a classifier, and a domain alignment component. The paper conducts large-scale experiments adapting the state-of-the-art visual domain adaptation methods to the proposed framework on time series classification tasks. The findings based on the experimental results reveal the key points of applying UDA to time series data.

This paper presents an empirical approach for unsupervised domain adaptation of time series data. The paper points out some of the drawbacks of existing approaches due to inconsistencies in evaluation schemes, datasets, model selection rules, and base neural network architectures. The paper then presents adaptations of visual domain adaptation methods for time series data. Experimental results using ten state-of-the-art methods on three benchmark datasets spanning fifteen cross-domain scenarios are presented.