DIVERSIFY to Generalize: Learning Generalized Representations for Time Series Classification
Authors: Wang Lu
Jindong Wang
Yiqiang Chen
Xinwei Sun
Venue: ICLR 2022 Submitted (2022)
Keywords: Time series classification
domain generalization

Abstract: Time series classification is an important problem in real world. Due to its nonstationary property that the distribution changes over time, it remains challenging to build models for generalization to unseen distributions. In this paper, we propose to view the time series classification problem from the distribution perspective. We argue that the temporal complexity attributes to the unknown latent distributions within. To this end, we propose DIVERSIFY to learn generalized representations for time series classification. DIVERSIFY takes an iterative process: it first obtains the worst-case distribution scenario via adversarial training, then matches the distributions between all segments. We also present some theoretical insights. Extensive experiments on gesture recognition, speech commands recognition, and sensor-based human activity recognition demonstrate that DIVERSIFY significantly outperforms other baselines while effectively characterizing the latent distributions by qualitative and quantitative analysis.

Under review as a conference paper at ICLR 2022
DIVERSIFY TO GENERALIZE: LEARNING GENERAL- IZED REPRESENTATIONS FOR TIME SERIES CLASSIFI- CATION
Anonymous authors Paper under double-bling review
ABSTRACT
Time series classification is an important problem in real world. Due to its non- stationary property that the distribution changes over time, remains challeng- ing to build models for generalization to unseen distributions. In this paper, we propose to view the time series classification problem from the distribution per- spective. We argue that the temporal complexity attributes to the unknown latent distributions within. To this end, we propose DIVERSIFY to learn generalized rep- resentations for time series classification. DIVERSIFY takes iterative process: it first obtains the worst-case distribution scenario via adversarial training, then matches the distributions between all segments. We also present some theoretical insights. Extensive experiments on gesture recognition, speech commands recog- nition, and sensor-based human activity recognition demonstrate that DIVERSIFY significantly outperforms other baselines and effectively characterizes the latent distributions by qualitative and quantitative analysis.
INTRODUCTION
Time series classification one of the most challenging problems in machine learning and statisties community. Example applications include sensor-based human activity recognition, Parkinson's s disease diagnosis, and electronic power consumption (Fawaz et al. 2019). One important nature of time series is non-stationary property, which means that its statistical features are changing over time. For years, there have been tremendous efforts to tackle the time series classification problem, such hidden Markov models (Fulcher & Jones 2014). RNN-based methods (Hüsken & Stagge 2003), and Transformer-basee approaches (Li et al. 2019).
In this paper, are specifically interested in modeling time series from the distribution perspective. More precisely, we aim to learn representations for time series that can generalize to unseen distri- butions. Note that this scenario has been extensively studied in existing literature of domain gener- alization (Muandet et al.) 20133 Wang et al.) 2021a) and out-of-distribution generalization (Krueger) al. 2021), where researchers tions, thus generalize well. While most of the efforts are done in image classification, few of them focus on the time series domain, which is more challenging. Although time series share a similar goal as image data in domain generalization, it naturally brings more challenges due to its non- stationary property: the distribution keeps changing over time, which contains diverse distribution information that should be harnessed well for better generalization.
keen bridge the gap between known and unknown distribu-
We show an illustrative example in Figure Domain generalization in image classification often involves several domains and the domain information is known (subfigure (a)). Thus, we can lever- age such domain information to build generalization models. However, Figure (b), we see that in time series data, although the distribution is changing dynamically over time, its domain informa- tion is not available. This will dramatically impede the modeling existing domain generalization algorithms they typically assume access domain information (subfigure (c)).
In order to learn a generalized time series model, we propose DIVERSIFY, a domain generalization algorithm to characterize the latent distributions inside the time series data. Concretely speaking, method consists of a min-man max adversarial game that: on one hand, it learns to segment the time series data into several latent sub-domains by maximizing the segment-wise distribution gap preserve

---

Under review as a conference paper at ICLR 2022
(a) Image:
Domain bel
(b) Time series: Domain label unknown
(c) Misclassified sub-domains if we treat it as one distribution
(d) Latent distributions of domain leamed by our method
scributionn
Pa
Pialnting
Px
19
(a) Domain generalization for image data require known do- main labels. (b) Domain label is unknown for time series. (c) If we treat the time series data as one single domain, the sub-domains are misclassified. Different colors and shapes correspond differ- ent classes and domains. (d) Finally, our DIVERSIFY can effectively learn the latent distributions.
Figure 1: Illustration of DIVERSIFY:
diversities, i.e., the worst-case distribution scenario; the other hand, it learns domain-invarian representations by reducing the distribution divergence for the worst-case scenario. Such diversi- fication naturally exists in non-stationary dataset where the data from multiple people naturally follow several latent distributions. Moreover, is also surprising to find that even the data one person still has such diversification: it can also be split into several latent distributions. Obviously, DIVERSIFY can effectively characterize the latent distributions (Figure (d))
To summarize, contributions three-fold:
For deep learning-based time series classification, we identify the gener- alized representation problem, which is challenging than traditional image classification problem due to the existence of unidentified latent distributions.
Novel problem:
New methodology: We propose DIVERSIFY, a pnporeiicall-motiive solution to solve the generalized representation learning problem identify the latent distributions.
Good performance: Our approach is extensively evaluated in three types of tasks: gesture recognition, speech command recognition, and sensor- activity recognition. By qual- itative and quantitative analysis, demonstrate the superiority of DIVERSIFY several challenging scenarios: on difficult tasks, significantly diverse datasets, and limited data.
2 METHODOLOGY
2.1 PROBLEM FORMULATION
We are given a time-series dataset D {(x, vo) as the training dataset, where N is the number of samples,
X' C Rº is the p-dimensional instance (sliding window) and y: E ) (1...., C} is label. We use pa px"(x, y) on X x y denote the joint distribution of the training dataset. Our goal is to learn generalized model from D to predict well on an wnseen target dataset, Dee which is inaccessible in training. Like Der, time series in D also be split into short series. In problem, the training and test datasets have the same input and output spaces but different distributions, i.c., Xtr Lte. ytte yds, but [p'( (x, 3) + pdc y). We aim to train a model from D minimize the risk Die: min, (xx,))== (h(x) # v).
Note that due to the non-stational non-stationary property, the training dataset may be composed of several un- known latent distributions, instead of one fixed distribution, i.c., (x. y) = Σ T, (x, y), where PK (x, y) is the distribution of the i-th sub-domains of the training data and n, is its weight. K is the number of sub-domains that is unknown, and D i== I. = 1.
2.2 MOTIVATION
The labeled time series data can be composed of several latent distributions (domains) that are chal- lenging characterize, even if the dataset is fully labeled. For instance, data collected by sensors of
2

---

Under review as a conference paper at ICLR 2022
three persons may belong to two different distributions when considering their similarities. More- over, even for data from one single person, different segments of one sequence may follow different distributions. In nutshell, in reality, there often exist several sub-domains in one time series dataset.
To ensure good generalization performance on the test datasct, it is important to learn distribution- invariant, domain-invarian representations from the training dataset by characterizing its latent distributions. These latent distributions may contain both benign and malignant knowledge that in- fluences generalization the target dataset. In Figure (c), assume the source domain contains two sub-domains (circle and plus points). Directly learning from the entire source domain by treat- ing as one distribution may generate the black margin. Although green star data points and red star points can be classified easily, red star data points are misclassified to the green class when pre- dicting on the -uist--iiitiiuution domain (star points) with the learned model. While other methods fail, our method can characterize the latent distributions, which will be introduced later.
2.3 DIVERSIFY
In this paper, we propose DIVERSIFY to learn generalized representations for time series classifica- tion. The core of DIVERSIFY is to characterize the latent distributions in time series dataset and then to minimize the distribution divergence between each two. To characterize the diverse informa- tion for better generalization, DIVERSIFY utilizes iterative process: first obtains the worst-case distribution scenario from a given dataset, then bridges the distribution gaps between each pair of latent distributions. Why the worst-case scenario? We argue that the worst-case scenario will maxi- mally preserve the diverse information of each latent distribution, thus benefiting generalization.
Concept
Raw input & Preprocessing
Fine-grained feature update
Latent distribution
Domain-invariant feature learning
characterization
Algorithm
Copy
Copy
Feature extractor
Feature extractor
Feature extractor
Bottleneck
Bottleneck
Bottleneck
Classifler
Casstter
Linear
Clessiier
Linear
= K x
1,2,-.. .......
Cxel
Cole Class label
Classifer
Self Self-supervised pseudo domain labe!
Forward
Backward
engeer
Leta Class label
Loss
Pseudo domein-dass label
Domain label
Figure 2: The framework of DIVERSIFY.
Figure
describes the main procedures of DIVERSIFY:
1. Pre- precerocessng:
this step adopts the sliding window to split the entire training dataset into fixed-sizz windows. We believe that the data from one window is the smallest domain unit.
2. Fine-grained feature update: this step updates the feature extractor using the proposed pseudo domain-class labels the supervision.
3. Latent distribution characterization: this step aims to identify the domain label for each instance to acquire the latent distribution information. tries to maximize the different distribution gaps enlarge diversity.
4. last step to learn
representation learning: representations and train generalizable model.
this step utilizes pseudo domain labels from the
Note that steps 2 4 are iterative as shown in Figure 2 Now elaborate their details.
3

---

Under review as a conference paper at ICLR 2022
Fine-grained Feature Update Before we characterize the latent distributions, perform fine- grained feature update to obtain useful representations. As shown in Figure 2 (blue), in order to fully utilize the knowledge contained in the domains and classes, we propose new concept: pseudo domain-class label, which serves as the supervision for feature extractor. This will ensure that the feature extractor can learn fine-grained information which benefits the later steps.
At the first iteration, there is no domain label d' and we simply initialize d' = 0 for all samples. We treat per class per domain as new class, and set the label to s , We have K x where K is the pre-defined number of latent distributions that can be tuned in experiments. We perform the following pseudo domain-clast label assignment to get discrete values for supervision: = d' x C + (Response to reviewer kiW8: this equation is put inline.)
Let n ( h be feature extractor, bottleneck, and classifier, respectively (we use superseript to denote step number), then, the supervised loss is computed using the cross-entropy loss t
Sauper B(x (M!2) (2) (2) (x))),
(hi (hgp
Latent Distribution Characterization This step characterizes the latent distributions contained in one dataset. As shown in (green),, employ adversarial training strategy to disentangle the domain labels from the class labels. However, there are no actual domain labels provided, which hinders such disentanglement. Inspired by (Liang et al. 2020), we adopt self-supervised pseudo- labeling strategy to obtain domain labels.
Firstly, we attain the centroid for each domain with invariaav features:
B ExEexer da (3) (1.²)
( (h)))
(3) (x)))))
(3) (n (x,))
Exeex (he (3) (N) (1 (3)
(x:)))
where f 1b he are feature extractor, bottleneck, and classifier, respectively. eu is the initial centroid of the 4tth latent sub-domains while 5g is the 1tth element of the logit soft-max output. Then, we obtain the pseudo domain labels via the nearest centroid classifier using a distance function D:
d arg min (h
(3) (3) (h xx)),,x).
(3)
Then, we compute the centroids based on the new pseudo labels and obtain the updated pseudo domain labels:
14 Exexo I(d; = k)h (3) (n) (x))
Ex.xx I(d, = kk)
df = arg min D(h, (1l (xx)), mw),
where I(a)= = when a is true, otherwise O. After we obtain d', we can compute the goal of step 2: (3)
fisllf + Lats ((h (1) (n. (1 (x) e(h (Rx, (11 (1 (x)))) v),
(3)
(5)
where h (3) the discriminator for step 3 that contains several linear layers and one classification layer. Rx, is the gradient reverse layer with hyperparameter X (Ganin et al. 2016). After this step ends, we can obtain pseudo domain label d' for x.
invarian Representation Learning After obtaining the latent distributions, we learn comain-invarian representations for generalization. In fact, this step (FigureZ is simple: we borrow the idea from DANN (Ganin et al.] 2016) and directly use adversarial training to update the classification loss Lete and domain classifier loss Ldom using GRL:
Lote + Ldom E(: (h( (x))). (( (4) (Rx, (h.
(4) (n (1((((())) (x))), d'),
(6)
where L is the cross-entropy loss and Rx, is the gradient reverse layer with hyperparameter
X2 (Ganin et al. 2016). More details of GRL and adv. training are presented in appendix A.
We repeat the above three steps until convergence max epochs. The final model is selected via validation dataset split from the source domain (Gulrajani & Lopez-Paz 2021
As for inference, we predict the labels with the modules from the last step.
4

---

Under review as a conference paper at ICLR 2022
2.4 THEORETICAL INSIGHTS
We present some theoretical insights to show that our approach is well motivated in theory.
Proposition 2.1. (Proposition 3.1 in (Sicilia et al. 2021)) Ler be a space and let H be a class of
hypotheses corresponding this space. Let Q and the collection {P: (P.),= be distributions over X and let { be collection of non-negative coefficients with E, iP: = L. Let the object O be set of distributions such that for every & E O the following holds
Σ (P) S max dHAH (HIn(PPPPP)).
i.j
Then, for any H. the following equality holds:
EQ(h) S Ao Σ PIEE,
1
min (HAH Q) max HHHH (IP. SEO
1
(8)
where Xa = -E= siide and each X; is the error of an ideal joint hypothesis for Q and Pt. (T(h) is the error for a hypothesis h on distribution P. dHAH((P Q is H-diveegence which measure differences distribution (Ben-David al. 2010).
The first item in Eq. 8
Aup is often neglected since it is small in reality. The second item, Σ, sp:er, (h), exists in almost all methods and can be minimized via supervision from class la- bels with cross-entropy loss in Eq. 6 Our main purpose is minimize the last two items in Eq, Here Q corresponds to the unseen out-of-distribution target domain.
The last term & maxi, (HH; PP)) is common in DA and DG which measures the maximum differences among source domains. This corresponds to step 4 in our approach.
Finally, the third item, 1/ minseo (S, Q). explains why we exploit sub-domains in step 3. Since our goal is to learn a model which can perform well on an unseen target domain, we cannot obtain Q To minimize 1 minseo d=ARH (S. to max., (P, P2) according to Eq. 7 corresponding to step 3 in our method which tries segment the time series data into several latent sub-domains by maximizing the segment-wise distribution gap to preserve diversities, i.e., the worst-case distribution scenario.
(S, Q). we can only enlarge the range of O. We have
EXPERIMENTS
We extensively evaluate our approach in three time series classification tasks: gesture recognition, speech commands recognition, and sensor-based human activity recognition. 'hese applications represent diverse situations of time series, thus can thoroughly reflect the advantage of our method. We will introduce their detail later. Per-segment accuracy is adopted as the evaluation metric.
For all experiments, conduct the
strategy following DomainBed (Gul- rajani & Lopez-Paz 202T). The training data are split by 8 2 for training and validation. For com-
parison, we re-implement eight recent strong comparison methods following DomainBed: ERM,
DANN (Ganin et al., 2016), CORAL (Sun & Saenko (016).. Mixup (Zhang et al.) (0188). Group-
DRO (Sagawa et al. 2020), RSC (Huang et al.) 2020), ANDMask (Parascandolo
2021) } and
GILE (Qian et al. 2021). For fairesss, all methods (except GILEE) utilize a feature net with two blocks and each block has one convolution layer, one pooling layer, and one batch normalization layer. Moreover, the bottleneck and classification layer adopts FC layer, respectively. See ap- pendix C for more details. Note that most methods require the domain labels known in training while our does not, which is more challenging and practical. We tune all methods to report the averaged best performance of three trials.
3.1 GESTURE RECOGNITION
Electromyography (EMG) is a common type of time-series data that is based on bioelectric signals. EMG data recorded
We use EMG for gestures Data Set (Lobov al.) 2018) that contains by MYO Thalmic bracelet. The bracelet is equipped with eight sensors equally spaced around the forearm that simultaneously acquire myographic signals. Data of 36 subjects are collected while they
5

---

Under review as a conference paper at ICLR 2022
performed series of static hand gestures and the number of instances is 40, 10.000--0 50, 000 recordings in each column. contains 7 classes and we select 6 common classes for our experiments. We randomly divide 36 subjects into four domains without overlapping and each domain contains data of 9 persons. Experiments are done in three random trials, thus mitigating influence of unfair splits.
Since EMG data comes from bioelectric signals, it is af- fected by many factors. EMG data are scene and device- dependent, which means the same person may generate different data when performing the same activity with the same device a different time with the different de- vices at the same time. Therefore, the benchmark of EMG challenging. Table [ shows that our method achieves the best average performance and is 4.3% better than the second- best method. Table 7 in appendix shows the ad- vantage of our method in different model sizes.
Table 1: Results on EMG dataset.
Target
0
AVG
ERM
62.6
69.9
67.9
69.3
67.4
DANN
62.9
70.0
66.5
68.2
66.9
CORAL
66.4
74.6
71.4
71.7
Mixup
60.7
69.9
70.5
68.2
67.3
GroupDRO
67.6
77.4
73.7
72.5
72.8
RSC
70.1
74.6
72.4
71.9
72.2
ANDMask
66.5
69.1
71.4
68.9
69.0
DIVERSIFY
71.7
82.4
76.9
77.3
77.1
3.2 SPEECH COMMANDS
Then, we adopt a regular speech recognition task, the Speech Com- mands dataset [Warden 2018). The dataset consists of one-second audio recordings of both background noise and spoken words such as "left', 'right', etc. It is collected from more than 2,000 persons, thus is more complicated. Following (Kidger al. 2020), we use 34,975 time series corresponding to ten spoken words to produce a balanced classification problem. Since this dataset is collected from multiple persons, thus the training and test distributions are different, which is also an OOD problem with one training domain. There many subjects and each subject only records few audios. Thus, an audio recording is short so we do not split each sample.
2-CKN
MatthBoxNet5-1 -114
Accuracy 196)
93
GroupDRo
Figure 3: Results on Speech commands dataset with two different backbones.
Figure 3 shows the recognition results on backbones. Com- pared with GroupDRO, DIVERSIFY has over 1% improvement based on a basic CNN architecture and over 0.6% improvement based on MatchBoxNet3-1 (Majumdar Ginsburg 2020) (its base- line is already strong). The results demonstrate the superiority of our method on a regular time-series benchmark composed of massive distributions.
3.3 SENSOR DENSOR-BASED HUMAN ACTIVITY RECOGNITION
Finally, we employ several popular sensor-based human activity recognition datasets for more de- tailed evaluation. UCI daily and sports dataset (DSADS) (Barshan & Yüksek 2014) consists of 19 activities collected from 8 subjects wearing body-worn sensors on body parts. USC-SIPI hu- man activity dataset (USC-HAD) (Zhang & Sawchuk 2012) is composed of 14 subjects (7 male, female, aged from 21 to 49) executing 12 activities with sensor tied on the front right hip. UCI- HAR (Anguita et al. 2012) is collected by 30 subjects performing 6 daily living activities with waist-mounted smartphone. PAMAP physical activity monitoring dataset (Reiss & Stricker 2012)
contains data of 18 activities, performed by 9 subjects wearing sensors.
We construct four different settings representing different degrees of generalization: (1) Cross- person generalization: This setting utilizes DSADS, USC-HAD, PAMAP]] datasets to construct three benchmarks. Within each dataset, we randomly split the data into four groups3 and then use three groups as training data to Iearn a generalized model for the last group. (2) Cross-poss position generalization: this setting uses DSADS dataset and data from each position denotes different domain. Each sample contains three sensors with nine dimensions. We treat one position as the test domain while the others are for training. (3) Cross-dataset generalization: this setting uses all four datasets, and each dataset corresponds to a different domain. Six common classes are selected. Two sensors from each dataset that belong to the same position are selected and data down-sampled to ensure the same dimension of data. (4) -o---rrro-----nootiiee
This setting adopts DSADS,
We do not use UCI-HAR in cross-person setting since its haseline is good enough. Forr details on dataset information and domain splits, please refer Appendix B

---

Under review as a conference paper at ICLR 2022
Table 2: Classification accuracy on cross-person generalization.
DSADS
USC.HAD
PAMAP
ALL
Target
0
2
AVG
0
2
3
AVG
2
3
AVG
AVG
79.3
87.8
80.3
57.7
74.0
69.7
55.8
84.4
77.1
DANN
89.1
84,2
85.9
83.4
85.6
81.2
57.9
76.7
70.7
71.6
82.2
78.1
55.4
87.3
75.7
77.7
CORAL
91.0
85.8
86.6
78.2
85.4
78.8
58.9
75.0
53.7
66.6
86.2
77.8
49.0
87.8
75.2
75.7
Mixup
89.6
82.2
89.2
86.9
87.0
80.0
64.1
74.3
61.3
69.9
89.4
80.3
58.4
87.7
79.0
78.6
GRO
91.7
85.9
87.59
78.3
85.9
80.1
55.5
74.7
60.0
67.6
85.2
77.7
56.2
85.0
76.0
76.5
RSC
84.9
82.3
86.7
77.7
82,9
81.9
57.9
73.4
65.1
69.6
87.1
76.9
60.3
87.8
78.0
76.9
ANDMask
85.0
75.8
87.0
71.6
81.4
79.9
55.3
74.5
63.0
68.7
86.7
76.4
43.6
85.6
73.1
74.4
81.0
75.0
77.0
74.7
78.0
62.0
77.0
63.0
70.0
68.0
42.0
76.0
67.5
DIVERSIFY
90.4
86.5
90.0
86.1
88.2
82.6
63.5
78.7
71.3
74.0
91.0
84.3
60.5
87.7
80.8
81.0
Table 3: Classification accuracy on cross pross-position, cross- and one-to-another generalization.
Cros-pocition generalization
Crost-dataset generalization
One- Pree-Peson-To--- Tn-Anhhh
Targei
3
AVG
AVG
DSADS
USC-IIAD
PAMAP
AVG
ERM
41.5
26.7
35.8
21.4
27.3
30.6
26.4
29.6
44.4
32.9
33.3
51.3
46.2
53.1
50.2
DANN
45.4
25.3
38.1
28.9
25.1
32.6
29.7
45.3
46.1
43.8
41.2
CORAL
33.2
25.2
25.8
22.3
20.6
25.4
39.5
41.8
39.1
36.6
39.2
Mixup
48.8
34.2
37.5
29.5
29.9
36.0
37.3
47.4
40.2
23.1
37.0
62.7
46.3
58.6
55.8
GroupDRO
27.1
26.7
24.3
18.4
24.8
24.3
51.4
36.7
33.2
33.8
38.8
51.3
48.0
53.1
50.8
RSC
46.6
27.4
35.9
27.0
29.8
33.3
33.1
39.7
45.3
45.9
41.0
59.1
49.0
59.7
55.9
ANDMask
47.
31.1
39.2
30.2
29.9
35.6
41.7
33.8
43.2
40.2
57.2
54.3
DIVERSIPY
47.2
32,9
44.5
31,6
30.4
37.4
48.7
46.9
49.0
59.9
51.1
67.6
55.0
62.5
61.7
USC-HAD, and PAMAP datasets. In each dataset, we randomly select four pairs of persons where one the training and the other is test. For simplicity, we use 0, I, to denote different domains. All experiments done in three random trials, mitigating the influence of unfair splits.
Table and Table show the results on four settings for HAR For all settings for HAR, our method achieves the best average performances. For Cross-person, Cross-position, Cross-dataset, and One-Person- -ow--eeeeooo----oeeeeee To-Another settings, our method significantly outperforms the second- best baseline by 2.4%. 1.4%, 9.9%, and 5.8% respectively. All results demonstrate the superiority of DIVER- SIFY in different generalization settings.
We observe more insightful conclusions. (1) When the task is difficult: In the Cross-Person setting, USC-HAD may be the most difficult task. Although it includes more samples, it contains 14 subjects with only two sensors experimental results prove the above argument that all methods perform terribly on this benchmark while ours has the largest improvement. (2) When datasets are significantly more diverse: Compared to Cross Person and Possi-oosi settings, are totally different that samples are influenced by subjects, devices, sensor positions, and some other factors. In this setting, our method is substantially better than others. (3) Limited duta: Compared with Pross- setting, One- Pne-Person-To-- To-Another is more difficult since has fewer data samples. Therefore, in this situation, enhancing diversity can bring remarkable improvement and our method can boost the performance.
position, which may bring more difficulty in this situation. The
may be more difficult since all datasets
3.4 ANALYSIS
Ablation Study We present ablation study to answer the following three questions: (1) Why ob- taining pseudo domain labels with ciass-invariat features in step 3? If we obtain pseudo domain labels with common features, domain labels may have correlations with class labels, which may introduce contradictions when learning domain-invarian representations and lead common perfor- mance. This is certified by the results in Figure (() (2) Why using fireggfieggee fine-grained domain-class labels in step 2? If we utilize pseudo domain labels to update the feature net, it may make the represen- tations seriously biased towards domain Comain-related features and thereby leads to terrible performance on classification, which proved in Figure 4bb) If we only utilize class labels to update the feature net, it may make representations biased to class-related features, make DIVERSIFY unable to obtain true latent sub-domains, and thereby has unremarkable performance, which proved in Figure (c)
In Coisson-ToAAotiee To-Another setting, we only report average accuracy of four tasks each dataset. Since only domain exists in training dataset for this setting, DANN and CORAL cannot he implemented here.

---

Under review as a conference paper at ICLR 2022
Upokta Featurns sthy
Ous BAM
66
20
Senchman
Saxchrork
Baschmank
(a) Class-invariant feat.
(b) Update feat. with d'
(c) Update feature with y
(d) #Domain K (EMG)
Figure 4: Ablation study.
(a) Initial domain split
(b) Our domain split
(c) ANDMask classitica.
(d) Our classification
Figure 5: t-SNE visualizations for domain splits ((a) (b)) and classification ((c) (d)) on EMG data.
Therefore, we should utilize fine- rre-grained domain-class labels to obtain representations with both do- main and class information. (3) The more latent sub-domains, the better? More latent sub-domains may not bring better results as shown in Figure 4(d) A dataset may only contain a few sub-domains and introducing more may have a conflict with the intrinsic data property. Plus, latent sub-
domains also brings difficulty to obtain pseudo domain labels and learn Nomain-invarian features, which may have negative effects.
Visualization Study We present some visualizations to show the rationales of our method and more results are appendix. Data points with different initial domain labels are mixed together in Figure while our DIVERSIFY can characterize different latent distributions to separate them well in (ig It illustrates that our method can find better latent sub-domains. From Figure(5dd)ann Figure (()) we can see that our method can learn better domain-invarian representations compared to the latest method ANDMask. Therefore, DIVERSIFY can find better latent sub-domains and then find domain-invariant representation from split domains, which can finally enhance generalization.
Existence latent distributions Latent sub-domains widely exist in time series, even data col- lected from one person when performing the same activity. In Figure 6(a) for subject in USC-HA -HA dataset, there is more than one latent distribution for his walking activities. Figure 6(b) demonstrates similar phenomena exist in EMG dataset where algorithm found three latent distributions.
Quantitative analysis for worst-case distributions We present quantitative analysis by comput- ing the H-divergence (Ben-David al. 2010) to show the effectiveness of 'worst-case dis- tribution'. As shown in Figure [6C)) and Figure [(d) compared with initial domain splits, latent sub-domains generated by our method have larger H-divergence among each other. According to Prop. 2.1 larger H-divergence among domains can bring better generalization.
RELATED WORK
Time series classification is challenging problem for years. Existing researches mainly focus on the modeling temporal relations using either RNN-based methods or the recently-pron preenllyprooosed Transformer architecture. To best knowledge, there is only recent work (Du et al., 2021) that studied time series from the distribution level. However, their method is not end-to-end traiaale since they developed two-stage method that is non-differential.
8

---

Under review as a conference paper at ICLR 2022
3.
1:00
0.68
1.71
1.68
15
1 1.00
0.90
1
1.71
1.50
64
0.90
1.50
0.0
0
2
0
2
(a) USC-HAD
(b) EMG
(c) Initial splits
(d) Our splits
Figure 6: (a) (b) Latent distributions obtained by our approach on two datasets. (c) (d) H-divergence among domains with initial splits and our splits on PAMAP.
Domain / OOD generalization (Wang et al. 2021a) mainly consists of data augmentation (Zhagg
et al. 2018 Yue et al.) 2019), domain-invarian feature learning (Muandet et al. 2013 Ganin et al. 2016), and meta-learning (Li al. 2018) techniques. The work of (Matsuura & Harada 2020 also studied DG without knowing the domain labels by clustering with the style features for image data. However, that work is not applied to time series and is not end-to-end trainable due to clustering. Disentanglement (Peng et al. 2019 Wang et al.) (0216) tries to disentangle the domain and label information, but they assume access to domain information. Single domain generalization is sim- ilar to our problem setting that also involves one training domain (Fan et al. [021] Li et al.) [202]] Qiao et al. 2020: Wang et al. 2021c). Most work adopted generative models and data augmentation strategies. However, they treated the single domain as distribution and did not explore the latent sub-distributions. Specifically, Wang et al. (202cc) proposed a 'learning-to-de diversiyy algorithm that has similar name ours, but focused on mutual information maximization.
Mixture models (Rasmussen et al. 1999) are models representing the presence of subpopulations within an overall population, e.g., Gaussian mixture models. Our approach belongs to this category in general with focus on OOD. Moreover, we do not rely on generative models as most mixture models. Subpopulation shift is new setting introduced in (Koh et al.) 2021), which refers the situation that the training and test domains overlap, but their relative proportions differ. Our problem does not belong this setting since assume that the training and test distributions do not overlap.
Distributionally robust optimization (Delage & Ye 2010) shares a similar paradigm with our work, whose paradigm is also seek a distribution that has the worst performance within range of the raw distribution. However, we study the internal distribution shift instead of seeking a global distribution close to the original one. GroupDRO (Sagawa et al. 2020) studied the DRO group level, which is significantly different from our idea.
Pseudo-labeling is common strategy in semi- supervuee and unsupervised learning. SHOT (Liang al., (2020) combined the pseudo-labeling strategy and IM (Information Maximization) to learn target adaptation features for source free domain adaptation. However, we mainly focus on domain generalization and thereby have different goals and updating styles. DANN (Ganin al. (016) has become the fundamental technology in domain adaptation and domain generalization, using adversarial training to obtain Nomain-invarian representations. In our method, we utilize it to obtain class-invariant and domain-invariant representations based on all features.
CONCLUSIONS AND FUTURE WORK
We proposed DIVERSIFY to learn generalized representations for time series classification. DI- VERSIFY employs an adversarial game that maximizes the worst-case distribution scenario while minimizing their distribution divergence. We demonstrated its effectiveness in different types of applications via qualitative and quantitative analysis. We are surprised that not only a mixed dataset, but one dataset from single person can also contain several latent distributions. Characterizing such latent distributions will greatly improve the generalization performance on unseen datasets.
In the future, plan to extend algorithm by seeking new architectures and developing tech- niques to estimate the value of domain numbers K. Moreover, we plan to apply our algorithm to other time series-related applications such forecasting and video classification.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge I. Reyes-Ortiz. Human activity recognition on smartphones using multiclass hardware-friendly support vector machine. In International workshop on ambient assisted living, pp. 216-223. Springer, 2012.
Billur Barshan and Murat Cihan Yüksek. Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units. The Computer Journal, 57(1)) (1):: 1649-1667, 2014.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175, 2010.
Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research, 58(3):595 612, 2010.
Yuntao Du, Jindong Wang, Wenjie Feng. Sinno Pan, Tao Qin, and Chongjun Wang. Adarnn: Adap- tive learning and forecasting of time series. arXiv preprint arXiv:2108.044:: 2021.
Xinjie Fan, Qifei Wang, Junjie Ke, Feng Yang, Boqing Gong, and Mingyuan Zhou, Adversari- ally adaptive normalization for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8208-8211, 2021.
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep learning for time series classification: review. Data mining and knowledge discovery, 33(4):917-963, 2019.
Ben Fulcher and Nick S Jones. Highly comparative feature-based time-series classification. IEEE Transactions on Knowledge and Data Engineering, 26(12):30263306 2014.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François training of neural net-
Laviolette, Mario Marchand, and Victor Lempitsky. works. The journal of machine learning research, 17(1)2096-20300, 2016.
Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Zeyi Huang, Haohan Wang, Eric Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 124-140 Springer, 2020.
Michael Hüsken and Peter Stagge. Recurrent neural networks for time series classification. Neuro- computing, 50:223-235, 2003.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural Controlled Differential Equa- tions for Irregular Time Series. Advances in Neural Information Processing Systems, 2020.
Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani. Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637- 5664. PMLR, 2021.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapo- lation (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Lei Li, Ke Gao, Juan Cao, Ziyao Huang, Yepeng Weng, Xiaoyue Mi, Zhengze Yu, Xiaoya Li, and Boyang Xia. Progressive domain expansion network for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 224- 233, 2021.
10

---

Under review as a conference paper at ICLR 2022
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, Advances in Neural Information Processing Systems, 32:5243 2019.
Jian Liang. Dapeng Hu, and Jiashi Feng. Do really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning. pp. 6028-600 0039. PMLR, 2020.
Sergey Lobov, Nadia Krilova, Innokentiy Kastalskiy, Victor Kazantsev, and Valeri Makarov. La- tent factors limiting the performance of semg- semg-intert interfacess... Sensors, 18(4):1122, 2018.
Somshubra Majumdar and Boris Ginsburg. Matchboxnet: 1d time-channel separable convolutional neural network architecture for speech commands recognition. arXiv preprint arXiv:2004.08531, 2020.
Toshihiko Matsuura and Tatsuya Harada. Domain generalization using mixture of multiple latent domains. In AAAI, 2020.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. International Conference on Machine Learning, pp. 10-18. PMLR,
2013.
Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. In ICLR, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. volume 32, pp. 8026-8037, 2019.
Xingchao Peng. Zijun Huang, Ximeng Sun, and Kate Saenko. Domain agnostic learning with dis- entangled representations. In International Conference on Machine Learning, pp. 5102-5112. PMLR, 2019.
Hangwei Qian, Sinno Jialin Pan, Chunyan Miao, H Qian, SJ Pan, and C Miao. Latent independent excitation for generalizable sensor-based cross-person activity recognition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 11921-11929, 2021.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12556 12565, 2020.
Carl Edward Rasmussen et al. The infinite gaussian mixture model. In NIPS, volume 12, pp. 554- 560. Citeseer, 1999.
Attila Reiss and Didier Stricker. Introducing new benchmarked dataset for activity monitoring, In 2012 16th international symposium on wearable computers, pp. 108-109. IEEE, 2012.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. In International Conference on Learning Representations (ICLR), 2020.
Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger, and Kristof Van Laerhoven. In- troducing wesad, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM international conference on multimodal interaction, pp. 400-408, 2018.
Anthony Sicilia, Xingchen Zhao, and Seong Jae Hwang. Domain adversarial neural networks for domain generalization: When it works and how to improve. arXiv preprint arXiv:2102.03924, 2021.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 443-450. Springer, 2016.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin. Generalizing to unseen domains: A survey on domain generalization. arXiv preprint arXiv: 103.
2 03097, 2021a.

---

Under review as a conference paper at ICLR 2022
Yufei Wang, Haoliang Li, Lap-Pui Chau, and Alex C Kot. Variational disentanglement for domain generalization. arXiv preprint arXiv:2109.05826, 2021b.
Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang, and Mahsa Baktashmotlagh. Learning to diversify for single domain generalization. In ICCV, 2021c.
Pete Warden. Speech commands: dataset for limited-vocabulan speech recognition. arXiv preprint arXiv: 03200,, 2018.
Xiangyu Yue. Yang Zhang, Sicheng Zhao, Alberto Sangiovanni SungiovaniiVVincenellii Kurt Keutzer, and Bo- qing Gong. Domain randomization and pyramid consistency: Simulation-to-rcal generalization without accessing target domain data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2100-21111 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Mi Zhang and Alexander Sawchuk. Usc-had: a daily activity dataset for ubiquitous activity recognition using wearable sensors. In Proceedings of the 2012 ACM conference on ubiquitous computing, pp. 1036-1043, 2012.
12

---

where Under review as a conference paper at ICLR 2022
A METHOD DETAILS
A. DOMAIN-INVARIANT REPRESENTATION LEARNING
innainninvariant representation learning utilizes adversarial training which contains a feature net- work, domain discriminator, and classification network. The domain discriminator tries its best discriminate domain labels of data while the feature network tries its best to generate features to confuse the domain discriminator, which thereby obtains domain-invariant representation. There- fore, it is an adversarial process, and our setting, it can be expressed as following,
(4) (4) (4) (1))((n))))))))))))))) (x))). - E(h (hg (hy (x))), (x..)) ((h (1g (1)
min (4) Ab6"
(4) E (x.g)) (lh (11 (n) (x))),
min (4)
To optimize Eq. we need an iterative process to optimize h, ((()) and h hade iteratively, which is cumbersome. It better to optimize h . and (4(4) is to solve the problems caused by the negative sign Eq. 9 Therefore, special gradient reversal layer (GRL), a popular implementation of the adversarial training in training several domains as suggested by (Ganin et al, 2016), came. GRL acts propagation while it takes the gradient from the subsequent level and changes its sign before passing to the preceding layer during the backpropagation. During the forward propagation, the GRL can be ignored. During the backpropagation, the GRL makes the sign of gradient on h reverse, which solves the problems caused by the negative sign in Eq.
ade at the same time. obvious that the key
identity transformation during the forward
B DATASET
B. DATASETS INFORMATION
Table shows the statistical information on each dataset.
Table 4: Information on HAR datasets.
Dataset
Subjects
Sensors
Classes
Samples
EMG
33,903,472
SPCMD
2618
105,829
DSADS
8
3
1.140,000
USC-HAD
14
2
12
5,441,000
UCI-HAR
30
2
6
1,310,000
PAMAP
9
3
18
3,850,505
B.2 DATA PREPROCESSING
We will introduce how we preprocess data and the final dimension of data for experiments here. We mainly utilize the sliding window technique, common technique in time-series classification, to split data. As its name suggests, this technique involves taking a subset of data from a given array sequence. Two main parameters of the sliding window technique are the window size, describing a subset length, and the step size, describing moving forward distance each time.
For EMG, set the window size 200 and the step size 100, which means there exist 50% overlaps between two adjacent samples. We normalize each sample with x x x-min
X contains all
min X
x. The final dimension is x 1 x 200.
For Speech Commands, we follow KKdger al. 2020).
Now we give details on all datasets in Cross-person setting. For DSADS, we directly utilize data split by the providers. The final dimension shape is 45 x 1 125. 45
means five positions, the first 3 means three sensors, and the second 3 means each sensor has three axes. For USC-HAD, the window size is 200 and the step size is 100. The final dimension shape is
13

---

also Under review as a conference paper at ICLR 2022
x 1 200. For PAMAP, the window size is 200 and the step size is 100. The final dimension shape 27 x x 200. For UCI-HAR, we directly utilize data split by the providers. The final dimension shape is 6 x 1 X 128.
For Cross-position each position corresponds to domain, sample is split into five samples in the first dimension. And the final dimension shape 9 x x 125.
we directly utilize samples obtained from DSADS in Cross-person person setting. Since
For datase,, directly utilize samples obtained in Cross-person setting. To make all datasets share the same label space and input space, we select six common classes, including WALKING, WALKING UPSTAIRS, WALKING DOWNSTAIRS, SITTING, STANDING, LAYING. In addi- tion, we down- sample data and select two sensors from each dataset that belong to the same position. 50.
The final dimension shape is 6
For Coe-Person-TooAnnthee we randomly select four pairs of persons from DSADS, USC-HAD,
and PAMAP respectively. Four tasks are 1 - 0.3 2, + 4, and + 6. Each number
corresponds to one subject. And the final dimension shape is 45 x 1 x 125, 6 x 1 200, and 27 x 200 for DSADS, USC-HAD, and PAMAP respectively.
As we can see, samples in EMG and HAR are all have more than one channels (the first dimension shape), which means they are all multivariate.
B.3 INITIAL DOMAIN SPLITS
We introduce how we split data here.
Since Speech Commands is a regular task, we just randomly split the entire dataset to training dataset, validation dataset, and testing dataset.
We mainly focus on EMG and HAR, and we construct domains for OOD tasks. We denote subjects of dataset with Sn, where 8n the number of subjects in the dataset. For example, there are 36 subjects in EMG and we utilize 0, 1, ... 35 to denote data of them respectively.
Table the initial domain splits of EMG and all datasets for HAR in Cross-person setting. We just want make each domain has the similar number of samples in one dataset. As noted in the main paper, we also utilize 0, 1, 2, and 3 to represent different domains but they have different meanings with subjects. When conducting experiments, we take one domain as the testing data and the others as the training data. Our method not influenced by the splits of the training data since we do not need the domain labels.
Table Initial domain splits.
Dataset
2
3
EMG
0-8
9-17
18-26
27-35
DSADS
2,3
4.5
6.7
USC- HAC-
0,1,2,11
3,5,6,9 1,5
1,8,
4.12
PAMAP
2,3,8
0,7
4.6
C NETWORK ARCHITECTURE AND HYPERPARAMETERS
For the architecture, the model contains two blocks, and each has one convolution layer, one pooling layer, and one batch normalization layer. A imgle-fully-conneeter layer is used as the bottleneck layer while another fully-connected layer serves as the classifier. All methods are implemented with PyTorch (Paszke et al. 2019). The maximum training epoch is set to 150. The Adam optimizer with weight decay x 10 is used. The learning rate for GILE is 10 10-4, The learning rate for the rest methods is 1 10 (For new added Speech Commands with MatchBoxNert311444
try the learning rate, 10-4.) We tune hyperparameters for each method.
For the pooling layer, we utilize MaxPool2d in PyTorch. The kernel size is (1, 2) ad the stride is 2. For the convolution layer, we utilize Conv2d in PyTorch. Different tasks have different kernel sizes and T the kernel sizes.
14

---

Under review as a conference paper at ICLR 2022
Table 6: The kernel size of each benchmark.
EMG
(1.9)
DSADS
(1.9)
SPEECHCOMMANDS
Cross-Person
USC-HAD (1.6)
Cross-position
(1.9)
PAMAP2
(1.9)
Cross-dataset
(1.6)
One-Person- Sin-Person-To-Amoole
The same as Cross-Peroe
D EVALUATION METRICS
We utilize average accuracy the testing dataset evaluation metrics for all benchmarks. Average accuracy is the most common metric for DG and it can be computed as the following.
Acc E(..)eD' Iz(U*)
#|)
(10)
y* = arg max h(x).
(y++ is an indicator function. If y number set. is the model to learn. Please note that X has different distribution from x for EMG and HAR. And x has been preprocessed and each sample is segment.
ys, it equals 1, otherwise it equals O. #1 represent the
(a) ERM
RSC
(c) ANDMask
Ours
Figure 7: Visualization of the t-SNE embeddings for classification EMG. Different colors spond to different classes while different shapes correspond to different domains.
(a) Initial splits USC
(b) Our splits USC (c) Initial splits EMG (d) Our splits on EMG
Figure 8: Visualization of the t-SNE embeddings for domain splits where different colors represent different domains.
E MORE EXPERIMENTAL RESULTS
E. VISUALIZATION STUDY
We show more visualization study in this part. As shown in Figure 7(a) and Figure 7(b) both ERM and RSC also cannot obtain fine domain-invarian representations and our method still achieves the best domain-invariant representations. As shown in Figure g compare with initial domain splits, latent sub-domains generated by our method are better separated.
E.2 PARAMETER SENSITIVITY
There are mainly four hyperparameters in our method: which is the number latent nob-domains, A1 for the adversarial part in step X for the adversarial part in step and local epochs and total
15

---

Under review as a conference paper at ICLR 2022
50
90
Outs BMV
Our EHM
Cart
374
ERM
87
70
10
93
65
1:150
3:50
5:30
1015
365
(a) #Latent domains
(b) A1
(c) A2
(d) Local Round
epoch
and
Figure 9: Parameter sensitivity analysis (EMG).
rounds. For fairness, the product local epochs and total rounds the same value. We evaluate the parameter sensitivity of our method in Figure where we change one parameter and fix the other to record the results. From these results, we can see that our method achieves better performance in wide range, demonstrating that our method is robust.
E.3 H-DIVERGENCE
0-
0.45
0.48
0.78
0.60
1
0.45
0.38
1
0.78
0.78
0.48
0.38
0.60
0.78
2
(a) Initial splits
(b) Our splits
Figure 10: H-divergence among domains with initial splits and our splits on EMG.
Figure 10 shows H-divergence among domains with initial splits and our splits EMG, which demonstrates our splits have larger H-divergence and thereby can bring better generalization.
E.4 THE INFLUENCE OF ARCHITECTURES
Table 7: Results on EMG dataset with different model sizes.
Model Size
Small
Medium
Large
Target
AVG
AVG
2
ERM
56.6
65.7
65.3
61.8
62.3
62.6
69.9
67.9
69.3
67.4
61.2
78.8
68.8
64.6
68.4
DANN
65.3
69.3
63.6
62.9
65.3
62.9
70.0
66.5
68.2
66.9
63.0
72.7
69.4
68.5
68.4
CORAL
66.9
74.9
70.8
73.2
71.4
66.4
74.6
71.4
74.2
71.7
67.7
77.0
72.7
71.8
72.3
Mixup
56,8
61.0
68.1
67.2
63.2
60.7
69.9
70.5
68.2
67.3
66.3
81.1
71.2
69.6
72.0
GroupDRO
71.6
69.1
77.5
73.7
72.8
79.3
73.0
RSC
62.7
73.2
67.6
64.0
66.9
74.6
72.4
71.9
72.2
65.1
76.8
72.2
67.7
70.4
ANDMask
62.4
66.0
66.3
65.6
65.1
66.6
69.1
71.4
68.9
69.0
65.7
78.1
72.1
71.9
71.9
DIVERSIFY
69.8
77.3
74.4
74.4
74.0
71.7
82,4
76.9
77.3
77.1
72.0
86.6
78.5
78.9
79.0
To ensure that our method can work with different sizes models, we add some more experiments with more complex or simpler architectures. As shown in Table 7 where small, medium, large indicates the different model sizes (our paper uses the medium), we see a clear picture that model sizes influence the results and our method also achieves the best performance. Small corresponds to the model with one convolutional layer, Medium corresponds to the model with two convolutional layers, and Large corresponds to the model with four convolutional layers. For most methods, more complex models bring better results. For all architectures, our method achieves the best perfor- mance.
16

---

Under review as a conference paper at ICLR 2022
E.5 RESULTS ON WESAD
We show results on a larger dataset, WESAD, here.
Wesad (Schmidt al. 2018 publicly available dataset for wearable stress and affect detection, which contains physiological and motion data 15 subjects. Wesad contains 63,000 63,000,000 instances. And we utilize techniques mentioned above to preprocess it. We utilize sensor modalities of chest- worn device including clectrocardiogram, electrodermal activity, electromyogram, respiration, body temperature, and three axis acceleration. We split 15 subjects into four domains, [(0-3), (4-7), (8- 11), (12-14)]. The results are shown in Table B and we can see that our method achieves the best performance compared other state-of- of-the-arr art methods with an improvement over 8%.
Table 8: Results on WESAD dataset.
Target
AVG
ERM
36.4
38.9
42.7
67.2
46.3
GroupDRO
44.3
58.3
43.2
64.1
52.5
ANDMask
42.5
45.3
51.2
62.8
50.5
DIVERSIFY
46.5
67.4
57.5
73.2
61.1
17


Reviews:
[{"review_id": "tLFN4pCrPn", "reviewer": "Reviewer_kiW8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Good representation.\n\n- The motivation and model design makes sense to me. \n\n- Extensive experiments. This work used 6 datasets from 3 different scenarios, worked on 4 different challenging tasks. I also appreciate the visualizations. \n\n\nWeaknesses:\n\nNo obvious weakness to me. Here are some suggestions for further revision:\n\n- This is an important point. Section 3.3, regarding cross-person generalization setting: I read the descriptions in the first set and noticed the footnote 2, but how to split 14  (USC-HAD) and 9 (PAMAP2) subjects into 4 groups? So I guess you are not splitting the subjects but splitting the segments, right? In detail, you shuffled all segments from all subjects, and split them into 4 groups (each group contains segments from all subjects). \n\nI also understand that the term of 'cross-person' is used in some papers (especially in EEG related publications) referring to shuffling all segments from all subjects. However, in their case, 'cross-person' is in contrast to 'person dependent' (use the segments from the same subject for both training and test) and 'person independent' (use several subjects for training and others for test). \n\nIn your case, it's not good to call it 'cross-person generalization'. Because, in the context of 'cross-position' and 'cross-dataset' where the training set and test set are from different positions/datasets, so readers suppose 'cross-person' means the training set and test set are from different subjects. Thus, my suggestion is to change the setting from splitting segments to splitting subjects. For example, in PAMAP2, you can use 7 subjects for training and 2 residuals for test (it's kind of like setting 4 but not the same). At least, please provide the results of splitting subjects over one dataset. \n\n- BTW, I didn't find the results for setting one-person-to-another (not in Table 2 nor Table 3), am I missed something? \n\n- Please enlarge panel c of Fig 1, the point shapes are hardly readable.\n\n- This paper takes univariate time series as an example, please discuss how to deal with multivariate time series? Specifically, are the signals from different sensors (in the same segment) belong to the same domain label? If not, how to solve the problem?\n\n- Please add more descriptions and details of domain-invariant representation learning (bottom in Page 4 or add to appendix). It's not clear to me that why the GRL learned representation is invariant to domains?\n\n- Please make Eq 1 an in-line equation. It's not important for the whole paper.\n\n- Please clarify, are the h_f, h_b, h_c are all functions? Implemented by neural networks? Please let me know if I missed some related information in your experimental setup.", "summary_of_the_paper": "This work proposes a framework to learn a domain-invariant representation of time series. Using the idea of domain generalization, this paper removes the influence of varying domains/distributions across patients/datasets through pseudo domain classification. Although this kind of model has been rather studied in images, but not a lot in time series. ", "summary_of_the_review": "This paper focuses on one meaningful topic, targets the challenge, and proposed an interesting solution to solve it. The presentation is easy to follow, the model is good (maybe the pseudo self-classification is not very novel, but still ok to me), and the experiments are sound to me. So I suggest accepting it. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "mQcF0Fbi-Nk", "reviewer": "Reviewer_8pYF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting paper and potentially the start to something really interesting. There are a number of questions/concerns I have as described below. Sorry if this comes off as overly negative. I enjoyed reading the paper and thought this is interesting work. \n\nContributions/Arguments/Writing:\n* There is a lot of discussion on stationarity early in the paper, with claims seeming to indicate that all time-series signals are non-stationary and other modalities, like images, are stationary. Strictly speaking, non-stationarity is a problem with practically all real-world classification problems, including images and other areas that \"distributional\" and \"domain\" work has been done. Perhaps the argument here should be around user-specific effects (e.g., the way one person wears the Myo EMG band may make gestures look different than another person? Or the same person wearing the device on two different days may require different calibration?).\n* One thing I didn't understand was why \"viewing time-series classification from a distributional perspective,\" at least using the methods in this paper, is important. Perhaps the authors can address this? Based on my cursory understanding of related work like GroupDRO, their goal is to minimize worst-case errors on specific distributions of data. You could imagine uses in ML fairness where you want the worst case performance for minority classes to be sufficiently good. However, the distributions used in this paper are latent and seemingly don't have semantic meaning. \n* Given the distributional slant, are the metrics described the this paper the right ones for the task? (caveat: as noted below, I don't believe the metrics are named). Results are provided holistically for all distributions whereas perhaps we should be more curious to see results on specific distributions. Based on the theory it seems like worst case performance should be improved. \n* The claims around what is or isn't a domain were not consistent between modalities. In the image modality (Fig 1a) a sketch is one domain whereas a cartoon is another. However, in Fig 1b it seems that two subsequences within one series are considered separate domains? \n\nModels / algorithms / experiments\n* The models used (as described in the appendix) are very small, especially for the Speech Commands dataset. It would be interesting to see state of the art (or at least recent) architectures for each problem (e.g., MatchBox or similar on Google Speech Commands) and to apply the paper's adversarial algorithm to those models. Based on a quick search it seems that this [1] is state of the art on this dataset achieves 97.41% compared to around 95% in this paper. \n* Metrics: It would be useful to describe the metrics being used for each experiment. I can try to infer (I assume it's mostly per-clip accuracy?) but I don't know for sure. More broadly it's unclear how metrics are compute computed, whether or not they're the \"right\" metrics for each dataset, or if they're the right metrics for evaluating distributional approaches like the one described in the paper. \n* It's hard to understand how substantial performance improvements are, especially on the EMG and Human Activity datasets. I could only find one other recent paper using the EMG dataset and performance was in the 90%s [2]. Assumably these are not apples-to-apples comparisons with the results in this paper... but having some external validation would be appreciated (especially on Speech Commands).\n* Re: Ablation study. I don't know what dataset these experiments are being performed on. Is there a reason the performance decreases with more domains? I was surprised by the visualizations in Fig 8 and Fig 9. These seem to be contrary to what is described in the rest of the paper: it seems that the domains actually don't do a good job of capturing latent structure? Perhaps I'm reading these incorrectly. \n* The EMG and HAR datasets are very small. I suppose this is philosophical, but would it be more impactful to develop algorithms that better fit this data or to collect 10x more data from each set of sensors? It's hard to know how well performance generalizes on datasets like this. \n\n[1] Mordido et al. \"Compressing 1D Time-Channel Separable Convolutions using Sparse Random Ternary Matrices.\" May 2021.\n[2] Rubio et al. \"Identification of hand movements from electromyographic signals using Machine Learning\" 2020.\n\n", "summary_of_the_paper": "This paper focuses on time-series classification problems, such as gesture or speech command recognition, where the data for any given sequence contains information from one user, but for which a priori you don't know anything about that user. Their goal is to be able to train on data with one set of users (\"domains\") and evaluate on a different set of users (\"domains\"). The users (at train and test) are unknown and are modeled by discrete latent variables. \n\nContributions include an algorithm (\"DIVERSIFY\") for learning distributions of domains/users. Their approach ultimately results in a model for time-series classification and for latent segmentation of data into discrete domains (aka user profiles). The model is framed using a min-max adversarial formulation. \n\nThe paper documents results on three datasets (EMG signals, Speech Commands, and Human Activities) show improvements over other distributional algorithms.  ", "summary_of_the_review": "I could be swayed to change my ratings fro \"correctness\" and \"empirical novelty\" but in general there are too many unanswered questions for me to vote to accept this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R3CHKY-QAdq", "reviewer": "Reviewer_7NdQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n1. This paper extended the domain generalization on time series classification problem by modeling the distribution of sub-domains within each domain and demonstrate its benefit in learning better representation.\n2. The proposed approach shows superior performance on three different benchmarks compared to state-of-the-art domain generalization methods. Multiple generalization settings are provided in human activity recognition task and the improvement is consistent across different settings.\n\n\n### Weakness\n1. The overall technical novelty is marginal. First, the pseudo-labeling strategy is adopted directly from (Liang et al. 2020). Secondly, the adversarial training strategy is adopted from (Ganin et al., 2016). Finally, the determination of number of sub-domains is based on simple heuristics i.e. cross-validation on different K values, which is not really a new concept compared to decoding time series using a state-space model such as hidden Markov model. The authors need to emphasize their contributions on existing development of domain generalization.\n2. The theoretical insight is based on existing conclusion from (Sicilia et al., 2021). It does not add to the novel contribution of the paper other than providing an interpretation of loss function.\n3. The discussion on classification model is not sufficient. Is the same model architecture as described in Appendix B used for all datasets and experiments? What is the impact of using more complex or simpler architectures? The ablation study can be strengthened by comparing different choices of model artitechtures. \n\n\n### Additional Comments\n1. It would be good to elaborate more implementation details such as how data is pre-processed, how convolution is performed for multi-variate time series, what is the kernel size, and what is the criteria for convergence in repeating step 2-4. The discussion on model architecture in Appendix B should be placed in main paper instead.\n2. Figure 6(d) shows even when K=2, the accuracy is much higher than ERM. I'm wondering what is the performance when K=1, which can be considered as degenerated baseline of the proposed approach. This can provide additional evidence on how effective is characterizing sub-domains and promote diversity.", "summary_of_the_paper": "This paper proposed a time series classification method with loss function that simultaneously promotes diverse distribution among different sub-domains within each domain and domain invariant feature representation within the same class. Here the domain is considered in a rather granular manner such as different person may be considered as different domains in gesture recognition application. An iterative process is proposed to learn a classification model. Both theoratical analysis and empirical evaluation on three different applications are provided. The proposed method shows better accuracy especially on generalizing across different domains than other competing methods.", "summary_of_the_review": "Overall, the proposed approach showed quite promising results on several different time series classification applications. The problem formulation and techniques are mostly adopted from existing work, which reduce the technical novelty. The authors need better justification on the performance improvement achieved by proposed approach, and highlight their contributions in domain generalization methodology. Please refer to the main review for more suggestions on improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has been reviewed by three reviewers, two scoring it borderline leaning towards an accept, and one scoring it as accept. The key criticism from reviewers is the lack of novelty (*the technical novelty is limited due to the adoption of existing methods without substantial changes (pseudolabeling for latent distribution, adversarial learning for domain generalization)*) and limited technical analysis (*Theoretical analysis is weak due to the use of existing conclusion from Sicilia et al., 2021*). On the other hand, authors argue that the problem they address is new (*learn the generalized representation for time series data, which is a new and challenging problem*).

As it stands, AC sees this paper as borderline leaning towards reject for the above reasons even though the evaluations are interesting.
Decision: Reject