A Systematic Evaluation of Domain Adaptation Algorithms On Time Series Data

1 INTRODUCTION . Time series classification problem is predominant in many real-world applications including healthcare and manufacturing . Yet , time series data with its inherit temporal dynamics and complex underlying pattern can be more challenging than static data . Recently , deep learning has gained more attention in time series classification tasks , assuming access to a vast amount of labeled data for training ( Fawaz et al. , 2019 ) . However , annotating time series data can be challenging and burdensome due to its complex underlying patterns and complicated domain knowledge ( Chang et al. , 2020 ) . One way to reduce labeling burden is to leverage annotated data ( e.g. , synthetic or public data ) from a relevant domain ( i.e. , source domain ) to train the model , while testing it on the domain of interest ( i.e. , target domain ) . Nevertheless , deep learning tends to perform poorly when tested on unseen data that have different distribution from the training data , which is well-known as the domain shift problem . Considering time series applications , source and target domains usually represent data from different subjects ( persons ) as in human activity recognition ( HAR ) ( Wilson et al. , 2020 ; Chang et al. , 2020 ) or sleep stage classification ( SSC ) tasks ( Phan et al. , 2020 ) . A considerable amount of literature has been developed on Unsupervised Domain Adaptation ( UDA ) for visual applications to mitigate the domain shift problem ( Long et al. , 2015 ; Ganin et al. , 2016 ; Long et al. , 2018 ; Chen et al. , 2020 ; Rahman et al. , 2020 ) . Recently , more attention has been paid to time series UDA ( TS-UDA ) ( Wilson et al. , 2020 ; Chang et al. , 2020 ; Ragab et al. , 2020 ; Liu & Xue , 2021 ) . However , the literature of TS-UDA methods suffers from the following limitations : • Existing TS-UDA works lack consistent evaluation schemes including benchmark datasets , preprocessing , and backbone networks . For instance , some methods that leverage recurrent neural network as backbone network ( Purushotham et al. , 2017 ) are compared against methods with convolutional based backbone networks ( Wilson et al. , 2020 ) . In addition , even with a similar architecture , training procedures can also vary among different algorithms including number of epochs , weight decay , and learning rate schedulers ( Tonutti et al. , 2019 ; Purushotham et al. , 2017 ) . • Existing TS-UDA works use labeled data from the target domain for model selection , violating the basic assumption of UDA ( Wilson et al. , 2020 ; Liu & Xue , 2021 ) , and providing an overoptimistic view of their performance . It is worth noting that model selection for domain adaptation in the absence of target domain labels is a long standing problem . • Most of existing algorithms are mainly application specific , and few works have been proposed for general TS-UDA . As a result , there is a shortage of baseline methods when applying domain adaptation on time series data . All the aforementioned challenges can highly affect the performance and can be mistakenly attributed to the proposed domain adaptation methods . In this work , we propose a systematic evaluation suite ( ADATIME ) to tackle the aforementioned obstacles and remove all extraneous factors to ensure a fair evaluation of different UDA algorithms on time series data . Specifically , to address the inconsistent evaluation schemes and backbone networks , we first standardize the preparation and processing of four benchmarking datasets from two classic real-world applications , including healthcare and human activity recognition . Besides , we unify the backbone network and the training procedures when comparing between different UDA methods . Second , to select the model hyper-parameters in the absence of target labels , we explore more realistic model selection strategies for TS-UDA problem that do not require target labels . Particularly , we investigate the model performance when selecting models based on source dataset ( i.e. , source risk ( Ganin et al. , 2016 ) ) , unlabeled target data ( i.e. , deep embedded evaluation ( DEV ) ( You et al. , 2019 ) ) , or only few-shot labeled samples from target data . Last , to address the lack of TSUDA baselines , we re-implement various state-of-the-art visual UDA methods that can be adapted to time series data while comparing to the existing TS-UDA methods Given our standard methodology , we aim to systematically study the following questions : ( 1 ) With standard backbone network and evaluation schemes , how will the visual UDA methods perform on time series data ; ( 2 ) Can we use realistic model selection methods —relying on only few or no target labels— and still achieve acceptable adaptation performance on time series data ; ( 3 ) How can the backbone network contribute to the performance . In this paper , we conduct comprehensive experiments to answer the aforementioned three questions . Some of our findings are summarized as follows : • Visual domain adaptation methods can achieve comparable or even better performance than the methods proposed for time series data . • Unlike image data , selecting models based on source risk can achieve reasonable performance and outperforms the DEV risk for time series data . Additionally , we find that our proposed few-shot target risk can achieve comparable performance to target risk , with affordable few labels samples . • Changing the base architecture can be pivotal to the performance . Moreover , complex architectures with time series data can have lower generalization performance on crossdomain experiments . 2 DOMAIN ADAPTATION PROBLEM . We start by defining the unsupervised domain adaptation problem . We assume the access to a labeled source domain XS = { ( xsi , ysi ) } NS i=1 that represents uni-variate or multivariate time series data , and unlabeled target domain XT = { ( xTi ) } NT i=1 where NS and NT denote the number of samples for XS and XT respectively . The source and target domains are sampled from different marginal distributions , i.e. , PS ( x ) 6= PT ( x ) , while the conditional distribution remains stable ( PS ( y|x ) = PT ( y|x ) ) . The main goal of UDA is to reduce the distribution shift between PS ( x ) and PT ( x ) , assuming they share the same label space . The mainstream of UDA algorithms is addressing the domain shift problem by finding domain invariant feature representation . Formally , given a backbone model fθ : X → Z , which transforms the input space to the feature space , the UDA algorithm mainly optimizes the backbone network to find new invariant representation for the target features such that Ps ( fθ ( x ) ) = Pt ( fθ ( x ) ) . As a results , a model trained on the source domain can generalize well to the adapted target domain . 3 ADATIME : A BENCHMARKING APPROACH FOR TIME SERIES DOMAIN ADAPTATION . In this work , we systematically evaluate domain adaptation algorithms on time series data , ensuring fair and realistic procedures . Figure 1 shows the details of ADATIME , which proceeds as follows . Given a dataset , we first apply our standard time series preparation schemes on the selected source and target domains . Subsequently , the source train data XStrain and target train data X T train are passed through the backbone network to extract source and target features , i.e. , ZStrain and Z T train respectively . The selected UDA algorithm is then applied to mitigate the distribution shift between the extracted features of two domains . Last , to set the hyper-parameters of the UDA algorithm , we consider three practical model selection approaches ( i.e. , without any target domain labels or only few-shot labeled samples ) namely source ( SRC ) risk , deep embedded evaluation ( DEV ) risk , and few-shot target ( FST ) risk . The proposed evaluation pipeline can prevent any extraneous factors and enable fair comparison between different UDA methods . The code of ADATIME will be made publicly available for researchers to enable seamless evaluation of different domain adaptation methods on time series data . 3.1 BACKBONE NETWORK . The general domain adaptation network consists of a feature extractor , a classifier , as well as different components to align the domains . We refer to the feature extractor and the classifier as the backbone network . The backbone network can have a pivotal contribution to the UDA model performance , as using different architectures may result in different performance . Hence , changing the backbone network when comparing different UDA methods can result in misleading outcomes , which can hinder the fair evaluation protocol . In most visual applications , ResNet-50 is the dominantly adopted feature extractor , as its complex architecture can extract representative features from images . However , in time series , using such complex architectures can lead to overfitting problems ( Fawaz et al. , 2019 ) . Therefore , we use a 1D-CNN architecture in ADATIME ( see Figure 6 ) . In addition , we study the effect of different backbone architectures on the performance of TS-UDA methods . 3.2 DOMAIN ADAPTATION ALGORITHMS . While numerous UDA approaches have been proposed to address the domain shift problem ( Zhao et al. , 2020b ) , a comprehensive review of existing UDA methods is out of our scope . Instead , we only included the most solid and recent baselines for visual UDA that can be adapted to time series . Besides , we included the state-of-the-art methods proposed for time series data . Overall , the selected algorithms in ADATIME can fit into two main categories according to the domain adaptation method , namely discrepancy based methods and adversarial based methods . The former aims to minimize a statistical distance between source and target features to mitigate the domain shift problem ( Tzeng et al. , 2014 ; Sun & Saenko , 2016 ; Chen et al. , 2020 ) . The latter leverages a domain discriminator network that enforces the feature extractor to produce domain invariant features ( Ganin et al. , 2016 ; Tzeng et al. , 2017 ) . Each method in the above categories can also be classified according to the aligned distribution . Specifically , some algorithms only align the marginal distribution of the feature space , while other algorithms jointly align the marginal and conditional distributions . The selected UDA algorithms are as follows : Deep Domain Confusion ( DDC Tzeng et al . ( 2014 ) ) ; Higher-order Moment Matching ( HoMM Zhao et al . ( 2020a ) ) ; Correlation Alignment via Deep Neural Networks ( Deep-CORAL Sun et al . ( 2017 ) ) ; Minimum Discrepancy Estimation ( MMDA Rahman et al . ( 2020 ) ) ; Domain-Adversarial Training of Neural Networks ( DANN Ganin et al . ( 2016 ) ) ; Conditional Adversarial Domain Adaptation ( CDAN Long et al . ( 2018 ) ) ; Deep Subdomain Adaptation ( DSAN Zhu et al . ( 2021 ) ) ; ( DIRT-T Shu et al . ( 2018 ) ) . In addition , we also include two UDA methods applied in time series classification , which are CoDATS ( Wilson et al. , 2020 ) and AdvSKM ( Liu & Xue , 2021 ) , noting that there are very few TS-UDA methods in the literature . We excluded the methods proposed for time series prediction/forecasting since they are out of our scope . Table 1 shows a summary of the selected methods and their corresponding categories .

Summary: The paper proposes a systematic evaluation framework named ADATIME, which systematically evaluates different unsupervised domain adaptation methods on time series data. The whole framework consists of a feature extractor, a classifier, and a domain alignment component. The paper conducts large-scale experiments adapting the state-of-the-art visual domain adaptation methods to the proposed framework on time series classification tasks. The findings based on the experimental results reveal the key points of applying UDA to time series data.