A Systematic Evaluation of Domain Adaptation Algorithms On Time Series Data

1 INTRODUCTION . Time series classification problem is predominant in many real-world applications including healthcare and manufacturing . Yet , time series data with its inherit temporal dynamics and complex underlying pattern can be more challenging than static data . Recently , deep learning has gained more attention in time series classification tasks , assuming access to a vast amount of labeled data for training ( Fawaz et al. , 2019 ) . However , annotating time series data can be challenging and burdensome due to its complex underlying patterns and complicated domain knowledge ( Chang et al. , 2020 ) . One way to reduce labeling burden is to leverage annotated data ( e.g. , synthetic or public data ) from a relevant domain ( i.e. , source domain ) to train the model , while testing it on the domain of interest ( i.e. , target domain ) . Nevertheless , deep learning tends to perform poorly when tested on unseen data that have different distribution from the training data , which is well-known as the domain shift problem . Considering time series applications , source and target domains usually represent data from different subjects ( persons ) as in human activity recognition ( HAR ) ( Wilson et al. , 2020 ; Chang et al. , 2020 ) or sleep stage classification ( SSC ) tasks ( Phan et al. , 2020 ) . A considerable amount of literature has been developed on Unsupervised Domain Adaptation ( UDA ) for visual applications to mitigate the domain shift problem ( Long et al. , 2015 ; Ganin et al. , 2016 ; Long et al. , 2018 ; Chen et al. , 2020 ; Rahman et al. , 2020 ) . Recently , more attention has been paid to time series UDA ( TS-UDA ) ( Wilson et al. , 2020 ; Chang et al. , 2020 ; Ragab et al. , 2020 ; Liu & Xue , 2021 ) . However , the literature of TS-UDA methods suffers from the following limitations : • Existing TS-UDA works lack consistent evaluation schemes including benchmark datasets , preprocessing , and backbone networks . For instance , some methods that leverage recurrent neural network as backbone network ( Purushotham et al. , 2017 ) are compared against methods with convolutional based backbone networks ( Wilson et al. , 2020 ) . In addition , even with a similar architecture , training procedures can also vary among different algorithms including number of epochs , weight decay , and learning rate schedulers ( Tonutti et al. , 2019 ; Purushotham et al. , 2017 ) . • Existing TS-UDA works use labeled data from the target domain for model selection , violating the basic assumption of UDA ( Wilson et al. , 2020 ; Liu & Xue , 2021 ) , and providing an overoptimistic view of their performance . It is worth noting that model selection for domain adaptation in the absence of target domain labels is a long standing problem . • Most of existing algorithms are mainly application specific , and few works have been proposed for general TS-UDA . As a result , there is a shortage of baseline methods when applying domain adaptation on time series data . All the aforementioned challenges can highly affect the performance and can be mistakenly attributed to the proposed domain adaptation methods . In this work , we propose a systematic evaluation suite ( ADATIME ) to tackle the aforementioned obstacles and remove all extraneous factors to ensure a fair evaluation of different UDA algorithms on time series data . Specifically , to address the inconsistent evaluation schemes and backbone networks , we first standardize the preparation and processing of four benchmarking datasets from two classic real-world applications , including healthcare and human activity recognition . Besides , we unify the backbone network and the training procedures when comparing between different UDA methods . Second , to select the model hyper-parameters in the absence of target labels , we explore more realistic model selection strategies for TS-UDA problem that do not require target labels . Particularly , we investigate the model performance when selecting models based on source dataset ( i.e. , source risk ( Ganin et al. , 2016 ) ) , unlabeled target data ( i.e. , deep embedded evaluation ( DEV ) ( You et al. , 2019 ) ) , or only few-shot labeled samples from target data . Last , to address the lack of TSUDA baselines , we re-implement various state-of-the-art visual UDA methods that can be adapted to time series data while comparing to the existing TS-UDA methods Given our standard methodology , we aim to systematically study the following questions : ( 1 ) With standard backbone network and evaluation schemes , how will the visual UDA methods perform on time series data ; ( 2 ) Can we use realistic model selection methods —relying on only few or no target labels— and still achieve acceptable adaptation performance on time series data ; ( 3 ) How can the backbone network contribute to the performance . In this paper , we conduct comprehensive experiments to answer the aforementioned three questions . Some of our findings are summarized as follows : • Visual domain adaptation methods can achieve comparable or even better performance than the methods proposed for time series data . • Unlike image data , selecting models based on source risk can achieve reasonable performance and outperforms the DEV risk for time series data . Additionally , we find that our proposed few-shot target risk can achieve comparable performance to target risk , with affordable few labels samples . • Changing the base architecture can be pivotal to the performance . Moreover , complex architectures with time series data can have lower generalization performance on crossdomain experiments . 2 DOMAIN ADAPTATION PROBLEM . We start by defining the unsupervised domain adaptation problem . We assume the access to a labeled source domain XS = { ( xsi , ysi ) } NS i=1 that represents uni-variate or multivariate time series data , and unlabeled target domain XT = { ( xTi ) } NT i=1 where NS and NT denote the number of samples for XS and XT respectively . The source and target domains are sampled from different marginal distributions , i.e. , PS ( x ) 6= PT ( x ) , while the conditional distribution remains stable ( PS ( y|x ) = PT ( y|x ) ) . The main goal of UDA is to reduce the distribution shift between PS ( x ) and PT ( x ) , assuming they share the same label space . The mainstream of UDA algorithms is addressing the domain shift problem by finding domain invariant feature representation . Formally , given a backbone model fθ : X → Z , which transforms the input space to the feature space , the UDA algorithm mainly optimizes the backbone network to find new invariant representation for the target features such that Ps ( fθ ( x ) ) = Pt ( fθ ( x ) ) . As a results , a model trained on the source domain can generalize well to the adapted target domain . 3 ADATIME : A BENCHMARKING APPROACH FOR TIME SERIES DOMAIN ADAPTATION . In this work , we systematically evaluate domain adaptation algorithms on time series data , ensuring fair and realistic procedures . Figure 1 shows the details of ADATIME , which proceeds as follows . Given a dataset , we first apply our standard time series preparation schemes on the selected source and target domains . Subsequently , the source train data XStrain and target train data X T train are passed through the backbone network to extract source and target features , i.e. , ZStrain and Z T train respectively . The selected UDA algorithm is then applied to mitigate the distribution shift between the extracted features of two domains . Last , to set the hyper-parameters of the UDA algorithm , we consider three practical model selection approaches ( i.e. , without any target domain labels or only few-shot labeled samples ) namely source ( SRC ) risk , deep embedded evaluation ( DEV ) risk , and few-shot target ( FST ) risk . The proposed evaluation pipeline can prevent any extraneous factors and enable fair comparison between different UDA methods . The code of ADATIME will be made publicly available for researchers to enable seamless evaluation of different domain adaptation methods on time series data . 3.1 BACKBONE NETWORK . The general domain adaptation network consists of a feature extractor , a classifier , as well as different components to align the domains . We refer to the feature extractor and the classifier as the backbone network . The backbone network can have a pivotal contribution to the UDA model performance , as using different architectures may result in different performance . Hence , changing the backbone network when comparing different UDA methods can result in misleading outcomes , which can hinder the fair evaluation protocol . In most visual applications , ResNet-50 is the dominantly adopted feature extractor , as its complex architecture can extract representative features from images . However , in time series , using such complex architectures can lead to overfitting problems ( Fawaz et al. , 2019 ) . Therefore , we use a 1D-CNN architecture in ADATIME ( see Figure 6 ) . In addition , we study the effect of different backbone architectures on the performance of TS-UDA methods . 3.2 DOMAIN ADAPTATION ALGORITHMS . While numerous UDA approaches have been proposed to address the domain shift problem ( Zhao et al. , 2020b ) , a comprehensive review of existing UDA methods is out of our scope . Instead , we only included the most solid and recent baselines for visual UDA that can be adapted to time series . Besides , we included the state-of-the-art methods proposed for time series data . Overall , the selected algorithms in ADATIME can fit into two main categories according to the domain adaptation method , namely discrepancy based methods and adversarial based methods . The former aims to minimize a statistical distance between source and target features to mitigate the domain shift problem ( Tzeng et al. , 2014 ; Sun & Saenko , 2016 ; Chen et al. , 2020 ) . The latter leverages a domain discriminator network that enforces the feature extractor to produce domain invariant features ( Ganin et al. , 2016 ; Tzeng et al. , 2017 ) . Each method in the above categories can also be classified according to the aligned distribution . Specifically , some algorithms only align the marginal distribution of the feature space , while other algorithms jointly align the marginal and conditional distributions . The selected UDA algorithms are as follows : Deep Domain Confusion ( DDC Tzeng et al . ( 2014 ) ) ; Higher-order Moment Matching ( HoMM Zhao et al . ( 2020a ) ) ; Correlation Alignment via Deep Neural Networks ( Deep-CORAL Sun et al . ( 2017 ) ) ; Minimum Discrepancy Estimation ( MMDA Rahman et al . ( 2020 ) ) ; Domain-Adversarial Training of Neural Networks ( DANN Ganin et al . ( 2016 ) ) ; Conditional Adversarial Domain Adaptation ( CDAN Long et al . ( 2018 ) ) ; Deep Subdomain Adaptation ( DSAN Zhu et al . ( 2021 ) ) ; ( DIRT-T Shu et al . ( 2018 ) ) . In addition , we also include two UDA methods applied in time series classification , which are CoDATS ( Wilson et al. , 2020 ) and AdvSKM ( Liu & Xue , 2021 ) , noting that there are very few TS-UDA methods in the literature . We excluded the methods proposed for time series prediction/forecasting since they are out of our scope . Table 1 shows a summary of the selected methods and their corresponding categories .

Summary: This paper explores the unsupervised domain adaptation of time series data (TS-UDA). And it focuses on the benchmark construction. By standardizing the base model, datasets, and model selection, this paper provides a good benchmark of TS-UDA. This benchmark can facilitate future research. Also, the paper proposes some findings.