Year: 2021

Abstract:
Time series classification is an important problem in real world. Due to its nonstationary property that the distribution changes over time, it remains challenging to build models for generalization to unseen distributions. In this paper, we propose to view the time series classification problem from the distribution perspective. We argue that the temporal complexity attributes to the unknown latent distributions within. To this end, we propose DIVERSIFY to learn generalized representations for time series classification. DIVERSIFY takes an iterative process: it first obtains the worst-case distribution scenario via adversarial training, then matches the distributions between all segments. We also present some theoretical insights. Extensive experiments on gesture recognition, speech commands recognition, and sensor-based human activity recognition demonstrate that DIVERSIFY significantly outperforms other baselines and effectively characterizes the latent distributions by qualitative and quantitative analysis.

heading: 1 INTRODUCTION
text: Time series classification is one of the most challenging problems in machine learning and statistics community. Example applications include sensor-based human activity recognition, Parkinson’s disease diagnosis, and electronic power consumption (Fawaz et al., 2019). One important nature of time series is non-stationary property, which means that its statistical features are changing over time. For years, there have been tremendous efforts to tackle the time series classification problem, such as hidden Markov models (Fulcher & Jones, 2014), RNN-based methods (Hüsken & Stagge, 2003), and Transformer-based approaches (Li et al., 2019).
In this paper, we are specifically interested in modeling time series from the distribution perspective. More precisely, we aim to learn representations for time series that can generalize to unseen distributions. Note that this scenario has been extensively studied in existing literature of domain generalization (Muandet et al., 2013; Wang et al., 2021a) and out-of-distribution generalization (Krueger et al., 2021), where researchers are keen to bridge the gap between known and unknown distributions, thus generalize well. While most of the efforts are done in image classification, few of them focus on the time series domain, which is more challenging. Although time series share a similar goal as image data in domain generalization, it naturally brings more challenges due to its nonstationary property: the distribution keeps changing over time, which contains diverse distribution information that should be harnessed well for better generalization.
We show an illustrative example in Figure 1. Domain generalization in image classification often involves several domains and the domain information is known (subfigure (a)). Thus, we can leverage such domain information to build generalization models. However, in Figure 1 (b), we see that in time series data, although the distribution is changing dynamically over time, its domain information is not available. This will dramatically impede the modeling of existing domain generalization algorithms as they typically assume access to domain information (subfigure (c)).
In order to learn a generalized time series model, we propose DIVERSIFY, a domain generalization algorithm to characterize the latent distributions inside the time series data. Concretely speaking, our method consists of a min-max adversarial game that: on one hand, it learns to segment the time series data into several latent sub-domains by maximizing the segment-wise distribution gap to preserve
diversities, i.e., the worst-case distribution scenario; on the other hand, it learns domain-invariant representations by reducing the distribution divergence for the worst-case scenario. Such diversification naturally exists in a non-stationary dataset where the data from multiple people naturally follow several latent distributions. Moreover, it is also surprising to find that even the data of one person still has such diversification: it can also be split into several latent distributions. Obviously, DIVERSIFY can effectively characterize the latent distributions (Figure 1 (d)).
To summarize, our contributions are three-fold:
• Novel problem: For deep learning-based time series classification, we identify the generalized representation problem, which is challenging than traditional image classification problem due to the existence of unidentified latent distributions.
• New methodology: We propose DIVERSIFY, a theoretically-motivated solution to solve the generalized representation learning problem to identify the latent distributions.
• Good performance: Our approach is extensively evaluated in three types of tasks: gesture recognition, speech command recognition, and sensor-based activity recognition. By qualitative and quantitative analysis, we demonstrate the superiority of DIVERSIFY on several challenging scenarios: on difficult tasks, significantly diverse datasets, and limited data.

heading: 2 METHODOLOGY

heading: 2.1 PROBLEM FORMULATION
text: We are given a time-series dataset Dtr = {(xi, yi)}Ni=1 as the training dataset, where N is the number of samples, xi ∈ X ⊂ Rp is the p-dimensional instance (sliding window) and yi ∈ Y = {1, . . . , C} is its label. We use Ptr(x, y) on X × Y to denote the joint distribution of the training dataset. Our goal is to learn a generalized model fromDtr to predict well on an unseen target dataset, Dte, which is inaccessible in training. Like Dtr, time series in Dte also be split into short series. In our problem, the training and test datasets have the same input and output spaces but different distributions, i.e., X tr = X te, Yte = Yte, but Ptr(x, y) 6= Pte(x, y). We aim to train a model h from Dtr to minimize the risk on Dte: minh E(x,y)∼Pte [h(x) 6= y]. Note that due to the non-stationary property, the training dataset may be composed of several unknown latent distributions, instead of one fixed distribution, i.e., Ptr(x, y) = ∑K i=1 πiPi(x, y), where Pi(x, y) is the distribution of the i-th sub-domains of the training data and πi is its weight. K is the number of sub-domains that is unknown, and ∑K i=1 πi = 1.

heading: 2.2 MOTIVATION
text: The labeled time series data can be composed of several latent distributions (domains) that are challenging to characterize, even if the dataset is fully labeled. For instance, data collected by sensors of
three persons may belong to two different distributions when considering their similarities. Moreover, even for data from one single person, different segments of one sequence may follow different distributions. In a nutshell, in reality, there often exist several sub-domains in one time series dataset.
To ensure good generalization performance on the test dataset, it is important to learn distributioninvariant, or domain-invariant representations from the training dataset by characterizing its latent distributions. These latent distributions may contain both benign and malignant knowledge that influences generalization on the target dataset. In Figure 1 (c), we assume the source domain contains two sub-domains (circle and plus points). Directly learning from the entire source domain by treating it as one distribution may generate the black margin. Although green star data points and red star points can be classified easily, red star data points are misclassified to the green class when predicting on the out-of-distribution domain (star points) with the learned model. While other methods fail, our method can characterize the latent distributions, which will be introduced later.

heading: 2.3 DIVERSIFY
text: In this paper, we propose DIVERSIFY to learn generalized representations for time series classification. The core of DIVERSIFY is to characterize the latent distributions in a time series dataset and then to minimize the distribution divergence between each two. To characterize the diverse information for better generalization, DIVERSIFY utilizes an iterative process: it first obtains the worst-case distribution scenario from a given dataset, then bridges the distribution gaps between each pair of latent distributions. Why the worst-case scenario? We argue that the worst-case scenario will maximally preserve the diverse information of each latent distribution, thus benefiting generalization.
Figure 2 describes the main procedures of DIVERSIFY:
1. Pre-processing: this step adopts the sliding window to split the entire training dataset into fixed-size windows. We believe that the data from one window is the smallest domain unit.
2. Fine-grained feature update: this step updates the feature extractor using the proposed pseudo domain-class labels as the supervision.
3. Latent distribution characterization: this step aims to identify the domain label for each instance to acquire the latent distribution information. It tries to maximize the different distribution gaps to enlarge diversity.
4. Domain-invariant representation learning: this step utilizes pseudo domain labels from the last step to learn domain-invariant representations and train a generalizable model.
Note that steps 2 ∼ 4 are iterative as shown in Figure 2. Now we elaborate their details.
Fine-grained Feature Update Before we characterize the latent distributions, we perform finegrained feature update to obtain useful representations. As shown in Figure 2 (blue), in order to fully utilize the knowledge contained in the domains and classes, we propose a new concept: pseudo domain-class label, which serves as the supervision for feature extractor. This will ensure that the feature extractor can learn fine-grained information which benefits the later steps.
At the first iteration, there is no domain label d′ and we simply initialize d′ = 0 for all samples. We treat per class per domain as a new class, and set the label to s ∈ {1, 2, · · · , S}. We have S = K×C where K is the pre-defined number of latent distributions that can be tuned in experiments. We perform the following pseudo domain-class label assignment to get discrete values for supervision: s = d′ × C + y. (Response to reviewer kiW8: this equation is put inline.)
Let h(2)f , h (2) b , h (2) c be feature extractor, bottleneck, and classifier, respectively (we use superscript to denote step number), then, the supervised loss is computed using the cross-entropy loss `:
Lsuper = E(x,y)∼Ptr` ( h(2)c (h (2) b (h (2) f (x))), s ) . (1)
Latent Distribution Characterization This step characterizes the latent distributions contained in one dataset. As shown in Figure 2 (green), we employ an adversarial training strategy to disentangle the domain labels from the class labels. However, there are no actual domain labels provided, which hinders such disentanglement. Inspired by (Liang et al., 2020), we adopt a self-supervised pseudolabeling strategy to obtain domain labels.
Firstly, we attain the centroid for each domain with class-invariant features:
µ̃k =
∑ xi∈X tr δk(h (3) c (h (3) b (h (3) f (xi))))h (3) b (h
(3) f (xi))∑
xi∈X tr δk(h (3) c (h (3) b (h (3) f (xi))))
, (2)
where h(3)f , h (3) b , h (3) c are feature extractor, bottleneck, and classifier, respectively. µ̃k is the initial centroid of the kth latent sub-domains while δk is the kth element of the logit soft-max output. Then, we obtain the pseudo domain labels via the nearest centroid classifier using a distance function D:
d̃′i = argmin k D(h (3) b (h (3) f (xi)), µ̃k). (3)
Then, we compute the centroids based on the new pseudo labels and obtain the updated pseudo domain labels:
µk =
∑ xi∈X tr I(d̃ ′ i = k)h (3) b (h
(3) f (x))∑
xi∈X tr I(d̃ ′ i = k)
,
d′i = argmin k D(h (3) b (h (3) f (xi)), µk),
(4)
where I(a) = 1 when a is true, otherwise 0. After we obtain d′, we can compute the goal of step 2:
Lself + Lcls = E(x,y)∼Ptr`(h(3)c (h (3) b (h (3) f (x))), d ′) + `(h (3) adv(Rλ1(h (3) b (h (3) f (x)))), y), (5)
where h(3)adv is the discriminator for step 3 that contains several linear layers and one classification layer. Rλ1 is the gradient reverse layer with hyperparameter λ1 (Ganin et al., 2016). After this step ends, we can obtain pseudo domain label d′ for x.
Domain-invariant Representation Learning After obtaining the latent distributions, we learn domain-invariant representations for generalization. In fact, this step (Figure 2 purple) is simple: we borrow the idea from DANN (Ganin et al., 2016) and directly use adversarial training to update the classification loss Lcls and domain classifier loss Ldom using GRL:
Lcls + Ldom = E(x,y)∼Ptr`(h(4)c (h (4) b (h (4) f (x))), y) + `(h (4) adv(Rλ2(h (4) b (h (4) f (x)))), d ′), (6)
where ` is the cross-entropy loss and Rλ2 is the gradient reverse layer with hyperparameter λ2 (Ganin et al., 2016). More details of GRL and adv. training are presented in appendix A.1.
We repeat the above three steps until convergence or max epochs. The final model is selected via a validation dataset split from the source domain (Gulrajani & Lopez-Paz, 2021).
As for inference, we predict the labels with the modules from the last step.

heading: 2.4 THEORETICAL INSIGHTS
text: We present some theoretical insights to show that our approach is well motivated in theory.
Proposition 2.1. (Proposition 3.1 in (Sicilia et al., 2021)) Let X be a space and letH be a class of hypotheses corresponding to this space. Let Q and the collection {Pi}ki=1 be distributions over X and let {ϕi}ki=1 be a collection of non-negative coefficients with ∑ i ϕi = 1. Let the object O be a set of distributions such that for every S ∈ O the following holds∑ i ϕidH∆H(Pi,S) ≤ max i,j dH∆H(Pi,Pj). (7)
Then, for any h ∈ H, the following equality holds:
εQ(h) ≤ λϕ + ∑ i ϕiεPi(h) + 1 2 min S∈O dH∆H(S,Q) + 1 2 max i,j dH∆H(Pi,Pj), (8)
where λϕ = ∑ i ϕiλi and each λi is the error of an ideal joint hypothesis for Q and Pi. εP(h) is the error for a hypothesis h on a distribution P. dH∆H(P,Q) is H-divergence which measure differences in distribution (Ben-David et al., 2010).
The first item in Eq. 8, λϕ, is often neglected since it is small in reality. The second item,∑ i ϕiεPi(h), exists in almost all methods and can be minimized via supervision from class labels with cross-entropy loss in Eq. 6. Our main purpose is to minimize the last two items in Eq. 8. Here Q corresponds to the unseen out-of-distribution target domain.
The last term 12 maxi,j dH∆H(Pi,Pj) is common in DA and DG which measures the maximum differences among source domains. This corresponds to step 4 in our approach.
Finally, the third item, 12 minS∈O dH∆H(S,Q), explains why we exploit sub-domains in step 3. Since our goal is to learn a model which can perform well on an unseen target domain, we cannot obtain Q. To minimize 12 minS∈O dH∆H(S,Q), we can only enlarge the range of O. We have to maxi,j dH∆H(Pi,Pj) according to Eq. 7, corresponding to step 3 in our method which tries to segment the time series data into several latent sub-domains by maximizing the segment-wise distribution gap to preserve diversities, i.e., the worst-case distribution scenario.

heading: 3 EXPERIMENTS
text: We extensively evaluate our approach in three time series classification tasks: gesture recognition, speech commands recognition, and sensor-based human activity recognition. These applications represent diverse situations of time series, thus can thoroughly reflect the advantage of our method. We will introduce their detail later. Per-segment accuracy is adopted as the evaluation metric.
For all experiments, we conduct the training-domain-validation strategy following DomainBed (Gulrajani & Lopez-Paz, 2021). The training data are split by 8 : 2 for training and validation. For comparison, we re-implement eight recent strong comparison methods following DomainBed: ERM, DANN (Ganin et al., 2016), CORAL (Sun & Saenko, 2016), Mixup (Zhang et al., 2018), GroupDRO (Sagawa et al., 2020), RSC (Huang et al., 2020), ANDMask (Parascandolo et al., 2021), and GILE (Qian et al., 2021). For fairness, all methods (except GILE) utilize a feature net with two blocks and each block has one convolution layer, one pooling layer, and one batch normalization layer. Moreover, the bottleneck and classification layer adopts a FC layer, respectively. See appendix C for more details. Note that most methods require the domain labels known in training while our does not, which is more challenging and practical. We tune all methods to report the averaged best performance of three trials.

heading: 3.1 GESTURE RECOGNITION
text: Electromyography (EMG) is a common type of time-series data that is based on bioelectric signals. We use EMG for gestures Data Set (Lobov et al., 2018) that contains raw EMG data recorded by MYO Thalmic bracelet. The bracelet is equipped with eight sensors equally spaced around the forearm that simultaneously acquire myographic signals. Data of 36 subjects are collected while they
performed series of static hand gestures and the number of instances is 40, 000−50, 000 recordings in each column. It contains 7 classes and we select 6 common classes for our experiments. We randomly divide 36 subjects into four domains without overlapping and each domain contains data of 9 persons. Experiments are done in three random trials, thus mitigating influence of unfair splits.
Table 1: Results on EMG dataset.
Target 0 1 2 3 AVG
ERM 62.6 69.9 67.9 69.3 67.4 DANN 62.9 70.0 66.5 68.2 66.9 CORAL 66.4 74.6 71.4 74.2 71.7 Mixup 60.7 69.9 70.5 68.2 67.3 GroupDRO 67.6 77.4 73.7 72.5 72.8 RSC 70.1 74.6 72.4 71.9 72.2 ANDMask 66.5 69.1 71.4 68.9 69.0 DIVERSIFY 71.7 82.4 76.9 77.3 77.1
Since EMG data comes from bioelectric signals, it is affected by many factors. EMG data are scene and devicedependent, which means the same person may generate different data when performing the same activity with the same device at a different time or with the different devices at the same time. Therefore, the benchmark of EMG is challenging. Table 1 shows that our method achieves the best average performance and is 4.3% better than the second-best method. Table 7 in appendix shows the advantage of our method in different model sizes.
3.2 SPEECH COMMANDS
Then, we adopt a regular speech recognition task, the Speech Commands dataset (Warden, 2018). The dataset consists of one-second audio recordings of both background noise and spoken words such as ’left’, ’right’, etc. It is collected from more than 2,000 persons, thus is more complicated. Following (Kidger et al., 2020), we use 34,975 time series corresponding to ten spoken words to produce a balanced classification problem. Since this dataset is collected from multiple persons, thus the training and test distributions are different, which is also an OOD problem with one training domain. There are many subjects and each subject only records few audios. Thus, an audio recording is short so we do not split each sample.
Figure 3 shows the recognition results on two backbones. Compared with GroupDRO, DIVERSIFY has over 1% improvement based on a basic CNN architecture and over 0.6% improvement based on MatchBoxNet3-1-64 (Majumdar & Ginsburg, 2020) (its baseline is already strong). The results demonstrate the superiority of our method on a regular time-series benchmark composed of massive distributions.

heading: 3.3 SENSOR-BASED HUMAN ACTIVITY RECOGNITION
text: Finally, we employ several popular sensor-based human activity recognition datasets for more detailed evaluation. UCI daily and sports dataset (DSADS) (Barshan & Yüksek, 2014) consists of 19 activities collected from 8 subjects wearing body-worn sensors on 5 body parts. USC-SIPI human activity dataset (USC-HAD) (Zhang & Sawchuk, 2012) is composed of 14 subjects (7 male, 7 female, aged from 21 to 49) executing 12 activities with a sensor tied on the front right hip. UCIHAR (Anguita et al., 2012) is collected by 30 subjects performing 6 daily living activities with a waist-mounted smartphone. PAMAP physical activity monitoring dataset (Reiss & Stricker, 2012) contains data of 18 activities, performed by 9 subjects wearing 3 sensors.
We construct four different settings representing different degrees of generalization: (1) Crossperson generalization: This setting utilizes DSADS, USC-HAD, PAMAP1 datasets to construct three benchmarks. Within each dataset, we randomly split the data into four groups2 and then use three groups as training data to learn a generalized model for the last group. (2) Cross-position generalization: this setting uses DSADS dataset and data from each position denotes a different domain. Each sample contains three sensors with nine dimensions. We treat one position as the test domain while the others are for training. (3) Cross-dataset generalization: this setting uses all four datasets, and each dataset corresponds to a different domain. Six common classes are selected. Two sensors from each dataset that belong to the same position are selected and data is down-sampled to ensure the same dimension of data. (4) One-Person-To-Another. This setting adopts DSADS,
1We do not use UCI-HAR in cross-person setting since its baseline is good enough. 2For more details on dataset information and domain splits, please refer to Appendix B.
USC-HAD, and PAMAP datasets. In each dataset, we randomly select four pairs of persons where one is the training and the other is the test. For simplicity, we use 0, 1, · · · to denote different domains. All experiments are done in three random trials, mitigating the influence of unfair splits.
Table 2 and Table 3 show the results on four settings for HAR3. For all settings for HAR, our method achieves the best average performances. For Cross-person, Cross-position, Cross-dataset, and One-Person-To-Another settings, our method significantly outperforms the second-best baseline by 2.4%, 1.4%, 9.9%, and 5.8% respectively. All results demonstrate the superiority of DIVERSIFY in different generalization settings.
We observe more insightful conclusions. (1) When the task is difficult: In the Cross-Person setting, USC-HAD may be the most difficult task. Although it includes more samples, it contains 14 subjects with only two sensors on one position, which may bring more difficulty in this situation. The experimental results prove the above argument that all methods perform terribly on this benchmark while ours has the largest improvement. (2) When datasets are significantly more diverse: Compared to Cross-Person and Cross-Position settings, Cross-Dataset may be more difficult since all datasets are totally different that samples are influenced by subjects, devices, sensor positions, and some other factors. In this setting, our method is substantially better than others. (3) Limited data: Compared with Cross-Person setting, One-Person-To-Another is more difficult since it has fewer data samples. Therefore, in this situation, enhancing diversity can bring a remarkable improvement and our method can boost the performance.

heading: 3.4 ANALYSIS
text: Ablation Study We present ablation study to answer the following three questions: (1) Why obtaining pseudo domain labels with class-invariant features in step 3? If we obtain pseudo domain labels with common features, domain labels may have correlations with class labels, which may introduce contradictions when learning domain-invariant representations and lead common performance. This is certified by the results in Figure 4(a). (2) Why using fine-grained domain-class labels in step 2? If we utilize pseudo domain labels to update the feature net, it may make the representations seriously biased towards domain-related features and thereby leads to terrible performance on classification, which is proved in Figure 4(b). If we only utilize class labels to update the feature net, it may make representations biased to class-related features, make DIVERSIFY unable to obtain true latent sub-domains, and thereby has unremarkable performance, which is proved in Figure 4(c).
3In One-Person-To-Another setting, we only report average accuracy of four tasks on each dataset. Since only one domain exists in training dataset for this setting, DANN and CORAL cannot be implemented here.
Therefore, we should utilize fine-grained domain-class labels to obtain representations with both domain and class information. (3) The more latent sub-domains, the better? More latent sub-domains may not bring better results as shown in Figure 4(d). A dataset may only contain a few sub-domains and introducing more may have a conflict with the intrinsic data property. Plus, more latent subdomains also brings difficulty to obtain pseudo domain labels and learn domain-invariant features, which may have negative effects.
Visualization Study We present some visualizations to show the rationales of our method and more results are in appendix. Data points with different initial domain labels are mixed together in Figure 5(a) while our DIVERSIFY can characterize different latent distributions to separate them well in Figure 5(b). It illustrates that our method can find better latent sub-domains. From Figure 5(d) and Figure 5(c), we can see that our method can learn better domain-invariant representations compared to the latest method ANDMask. Therefore, DIVERSIFY can find better latent sub-domains and then find domain-invariant representation from split domains, which can finally enhance generalization.
Existence of latent distributions Latent sub-domains widely exist in time series, even in data collected from one person when performing the same activity. In Figure 6(a), for a subject in USC-HAD dataset, there is more than one latent distribution for his walking activities. Figure 6(b) demonstrates similar phenomena exist in EMG dataset where our algorithm found three latent distributions.
Quantitative analysis for worst-case distributions We present quantitative analysis by computing the H-divergence (Ben-David et al., 2010) to show the effectiveness of our ‘worst-case distribution’. As shown in Figure 6(c) and Figure 6(d), compared with initial domain splits, latent sub-domains generated by our method have larger H-divergence among each other. According to Prop. 2.1, largerH-divergence among domains can bring better generalization.

heading: 4 RELATED WORK
text: Time series classification is a challenging problem for years. Existing researches mainly focus on the modeling of temporal relations using either RNN-based methods or the recently-proposed Transformer architecture. To our best knowledge, there is only one recent work (Du et al., 2021) that studied time series from the distribution level. However, their method is not end-to-end trainable since they developed a two-stage method that is non-differential.
Domain / OOD generalization (Wang et al., 2021a) mainly consists of data augmentation (Zhang et al., 2018; Yue et al., 2019), domain-invariant feature learning (Muandet et al., 2013; Ganin et al., 2016), and meta-learning (Li et al., 2018) techniques. The work of (Matsuura & Harada, 2020) also studied DG without knowing the domain labels by clustering with the style features for image data. However, that work is not applied to time series and it is not end-to-end trainable due to clustering. Disentanglement (Peng et al., 2019; Wang et al., 2021b) tries to disentangle the domain and label information, but they assume access to domain information. Single domain generalization is similar to our problem setting that also involves one training domain (Fan et al., 2021; Li et al., 2021; Qiao et al., 2020; Wang et al., 2021c). Most work adopted generative models and data augmentation strategies. However, they treated the single domain as one distribution and did not explore the latent sub-distributions. Specifically, Wang et al. (2021c) proposed a ‘learning-to-diversify’ algorithm that has a similar name to ours, but focused on mutual information maximization.
Mixture models (Rasmussen et al., 1999) are models representing the presence of subpopulations within an overall population, e.g., Gaussian mixture models. Our approach belongs to this category in general with focus on OOD. Moreover, we do not rely on generative models as most mixture models. Subpopulation shift is a new setting introduced in (Koh et al., 2021), which refers to the situation that the training and test domains overlap, but their relative proportions differ. Our problem does not belong to this setting since we assume that the training and test distributions do not overlap.
Distributionally robust optimization (Delage & Ye, 2010) shares a similar paradigm with our work, whose paradigm is also to seek a distribution that has the worst performance within a range of the raw distribution. However, we study the internal distribution shift instead of seeking a global distribution close to the original one. GroupDRO (Sagawa et al., 2020) studied the DRO at a group level, which is significantly different from our idea.
Pseudo-labeling is a common strategy in semi-supervised and unsupervised learning. SHOT (Liang et al., 2020) combined the pseudo-labeling strategy and IM (Information Maximization) to learn target adaptation features for source free domain adaptation. However, we mainly focus on domain generalization and thereby have different goals and updating styles. DANN (Ganin et al., 2016) has become the fundamental technology in domain adaptation and domain generalization, using adversarial training to obtain domain-invariant representations. In our method, we utilize it to obtain class-invariant and domain-invariant representations based on all features.

heading: 5 CONCLUSIONS AND FUTURE WORK
text: We proposed DIVERSIFY to learn generalized representations for time series classification. DIVERSIFY employs an adversarial game that maximizes the worst-case distribution scenario while minimizing their distribution divergence. We demonstrated its effectiveness in different types of applications via qualitative and quantitative analysis. We are surprised that not only a mixed dataset, but one dataset from a single person can also contain several latent distributions. Characterizing such latent distributions will greatly improve the generalization performance on unseen datasets.
In the future, we plan to extend our algorithm by seeking new architectures and developing techniques to estimate the value of domain numbers K. Moreover, we plan to apply our algorithm to other time series-related applications such as forecasting and video classification.

heading: A METHOD DETAILS
text: A.1 DOMAIN-INVARIANT REPRESENTATION LEARNING
Domain-invariant representation learning utilizes adversarial training which contains a feature network, a domain discriminator, and a classification network. The domain discriminator tries its best to discriminate domain labels of data while the feature network tries its best to generate features to confuse the domain discriminator, which thereby obtains domain-invariant representation. Therefore, it is an adversarial process, and in our setting, it can be expressed as following,
min h (4) b ,h (4) c
E(x,y)∼Ptr`(h(4)c (h (4) b (h (4) f (x))), y)− `(h (4) adv(h (4) b (h (4) f (x))), d ′),
min h (4) adv
E(x,y)∼Ptr`(h (4) adv(h (4) b (h (4) f (x))), d
′). (9)
To optimize Eq. 9, we need an iterative process to optimize h(4)b , h (4) c and h (4) adv iteratively, which is cumbersome. It is better to optimize h(4)b , h (4) c and h (4) adv at the same time. It is obvious that the key is to solve the problems caused by the negative sign in Eq. 9. Therefore, a special gradient reversal layer (GRL), a popular implementation of the adversarial training in training several domains as suggested by (Ganin et al, 2016), came. GRL acts as an identity transformation during the forward propagation while it takes the gradient from the subsequent level and changes its sign before passing it to the preceding layer during the backpropagation. During the forward propagation, the GRL can be ignored. During the backpropagation, the GRL makes the sign of gradient on h(4)b reverse, which solves the problems caused by the negative sign in Eq. 9.

heading: B DATASET
text: B.1 DATASETS INFORMATION
Table 4 shows the statistical information on each dataset.
B.2 DATA PREPROCESSING
We will introduce how we preprocess data and the final dimension of data for experiments here. We mainly utilize the sliding window technique, a common technique in time-series classification, to split data. As its name suggests, this technique involves taking a subset of data from a given array or sequence. Two main parameters of the sliding window technique are the window size, describing a subset length, and the step size, describing moving forward distance each time.
For EMG, we set the window size 200 and the step size 100, which means there exist 50% overlaps between two adjacent samples. We normalize each sample with x̃ = x−minXmaxX−minX . X contains all x. The final dimension is 8× 1× 200. For Speech Commands, we follow (Kidger et al., 2020).
Now we give details on all datasets in Cross-person setting. For DSADS, we directly utilize data split by the providers. The final dimension shape is 45 × 1 × 125. 45 = 5 × 3 × 3 where 5 means five positions, the first 3 means three sensors, and the second 3 means each sensor has three axes. For USC-HAD, the window size is 200 and the step size is 100. The final dimension shape is
6×1×200. For PAMAP, the window size is 200 and the step size is 100. The final dimension shape is 27× 1× 200. For UCI-HAR, we directly utilize data split by the providers. The final dimension shape is 6× 1× 128. For Cross-position, we directly utilize samples obtained from DSADS in Cross-person setting. Since each position corresponds to one domain, a sample is split into five samples in the first dimension. And the final dimension shape is 9× 1× 125. For Cross-dataset, we directly utilize samples obtained in Cross-person setting. To make all datasets share the same label space and input space, we select six common classes, including WALKING, WALKING UPSTAIRS, WALKING DOWNSTAIRS, SITTING, STANDING, LAYING. In addition, we down-sample data and select two sensors from each dataset that belong to the same position. The final dimension shape is 6× 1× 50. For One-Person-To-Another, we randomly select four pairs of persons from DSADS, USC-HAD, and PAMAP respectively. Four tasks are 1 → 0, 3 → 2, 5 → 4, and 7 → 6. Each number corresponds to one subject. And the final dimension shape is 45 × 1 × 125, 6 × 1 × 200, and 27× 1× 200 for DSADS, USC-HAD, and PAMAP respectively. As we can see, samples in EMG and HAR are all have more than one channels (the first dimension shape), which means they are all multivariate.
B.3 INITIAL DOMAIN SPLITS
We introduce how we split data here.
Since Speech Commands is a regular task, we just randomly split the entire dataset to a training dataset, a validation dataset, and a testing dataset.
We mainly focus on EMG and HAR, and we construct domains for OOD tasks. We denote subjects of a dataset with 0 − sn, where sn is the number of subjects in the dataset. For example, there are 36 subjects in EMG and we utilize 0, 1, 2, · · · , 35 to denote data of them respectively. Table 5 shows the initial domain splits of EMG and all datasets for HAR in Cross-person setting. We just want make each domain has the similar number of samples in one dataset. As noted in the main paper, we also utilize 0, 1, 2, and 3 to represent different domains but they have different meanings with subjects. When conducting experiments, we take one domain as the testing data and the others as the training data. Our method is not influenced by the splits of the training data since we do not need the domain labels.

heading: C NETWORK ARCHITECTURE AND HYPERPARAMETERS
text: For the architecture, the model contains two blocks, and each has one convolution layer, one pooling layer, and one batch normalization layer. A single-fully-connected layer is used as the bottleneck layer while another fully-connected layer serves as the classifier. All methods are implemented with PyTorch (Paszke et al., 2019). The maximum training epoch is set to 150. The Adam optimizer with weight decay 5 × 10−4 is used. The learning rate for GILE is 10−4. The learning rate for the rest methods is 10−2 or 10−3. (For new added Speech Commands with MatchBoxNet3-1-64, we also try the learning rate, 10−4.) We tune hyperparameters for each method.
For the pooling layer, we utilize MaxPool2d in PyTorch. The kernel size is (1, 2) ad the stride is 2. For the convolution layer, we utilize Conv2d in PyTorch. Different tasks have different kernel sizes and Table 6 shows the kernel sizes.

heading: D EVALUATION METRICS
text: We utilize average accuracy on the testing dataset as our evaluation metrics for all benchmarks. Average accuracy is the most common metric for DG and it can be computed as the following,
Acc =
∑ (x,y)∈Dte Iy(y∗)
#|Yte| ,
y∗ = argmaxh(x). (10)
Iy(y∗) is an indicator function. If y = y∗, it equals 1, otherwise it equals 0. #| · | represent the number of the set. h is the model to learn. Please note that X te has a different distribution from X tr for EMG and HAR. And x has been preprocessed and each sample is a segment.

heading: E MORE EXPERIMENTAL RESULTS
text: E.1 VISUALIZATION STUDY
We show more visualization study in this part. As shown in Figure 7(a) and Figure 7(b), both ERM and RSC also cannot obtain fine domain-invariant representations and our method still achieves the best domain-invariant representations. As shown in Figure 8, compare with initial domain splits, latent sub-domains generated by our method are better separated.
E.2 PARAMETER SENSITIVITY
There are mainly four hyperparameters in our method: K which is the number of latent sub-domains, λ1 for the adversarial part in step 3, λ2 for the adversarial part in step 4, and local epochs and total
rounds. For fairness, the product of local epochs and total rounds is the same value. We evaluate the parameter sensitivity of our method in Figure 9 where we change one parameter and fix the other to record the results. From these results, we can see that our method achieves better performance in a wide range, demonstrating that our method is robust.
E.3 H-DIVERGENCE
Figure 10 shows H-divergence among domains with initial splits and our splits on EMG, which demonstrates our splits have largerH-divergence and thereby can bring better generalization.
E.4 THE INFLUENCE OF ARCHITECTURES
To ensure that our method can work with different sizes of models, we add some more experiments with more complex or simpler architectures. As shown in Table 7, where small, medium, large indicates the different model sizes (our paper uses the medium), we see a clear picture that model sizes influence the results and our method also achieves the best performance. Small corresponds to the model with one convolutional layer, Medium corresponds to the model with two convolutional layers, and Large corresponds to the model with four convolutional layers. For most methods, more complex models bring better results. For all architectures, our method achieves the best performance.
E.5 RESULTS ON WESAD
We show results on a larger dataset, WESAD, here.
Wesad (Schmidt et al., 2018) is a publicly available dataset for wearable stress and affect detection, which contains physiological and motion data of 15 subjects. Wesad contains 63,000,000 instances. And we utilize techniques mentioned above to preprocess it. We utilize sensor modalities of chestworn device including electrocardiogram, electrodermal activity, electromyogram, respiration, body temperature, and three axis acceleration. We split 15 subjects into four domains, [(0-3), (4-7), (8- 11), (12-14)]. The results are shown in Table 8, and we can see that our method achieves the best performance compared to other state-of-the-art methods with an improvement over 8%.

Review summaries:
This paper proposed a time series classification method with loss function that simultaneously promotes diverse distribution among different sub-domains within each domain and domain invariant feature representation within the same class. Here the domain is considered in a rather granular manner such as different person may be considered as different domains in gesture recognition application. An iterative process is proposed to learn a classification model. Both theoratical analysis and empirical evaluation on three different applications are provided. The proposed method shows better accuracy especially on generalizing across different domains than other competing methods.

This paper focuses on time-series classification problems, such as gesture or speech command recognition, where the data for any given sequence contains information from one user, but for which a priori you don't know anything about that user. Their goal is to be able to train on data with one set of users ("domains") and evaluate on a different set of users ("domains"). The users (at train and test) are unknown and are modeled by discrete latent variables.   Contributions include an algorithm ("DIVERSIFY") for learning distributions of domains/users. Their approach ultimately results in a model for time-series classification and for latent segmentation of data into discrete domains (aka user profiles). The model is framed using a min-max adversarial formulation.   The paper documents results on three datasets (EMG signals, Speech Commands, and Human Activities) show improvements over other distributional algorithms.  

This work proposes a framework to learn a domain-invariant representation of time series. Using the idea of domain generalization, this paper removes the influence of varying domains/distributions across patients/datasets through pseudo domain classification. Although this kind of model has been rather studied in images, but not a lot in time series.