Semantic-Discriminative Mixup for Generalizable Sensor-based
Cross-domain Activity Recognition
WANG LU, Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. of Computing Tech., CAS, China
JINDONG WANG, Microsoft Research Asia, China
YIQIANG CHEN∗ , Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. of Computing Tech.,arXiv:2206.06629v1 [cs.AI] 14 Jun 2022

CAS, China

SINNO JIALIN PAN, Nanyang Technological University, Singapore
CHUNYU HU, Qilu University of Technology (Shandong Academy of Sciences), China
XIN QIN, Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. of Computing Tech., CAS, China
It is expensive and time-consuming to collect sufficient labeled data to build human activity recognition (HAR) models.
Training on existing data often makes the model biased towards the distribution of the training data, thus the model might
perform terribly on test data with different distributions. Although existing efforts on transfer learning and domain adaptation
try to solve the above problem, they still need access to unlabeled data on the target domain, which may not be possible
in real scenarios. Few works pay attention to training a model that can generalize well to unseen target domains for HAR.
In this paper, we propose a novel method called Semantic-Discriminative Mixup (SDMix) for generalizable cross-domain
HAR. Firstly, we introduce semantic-aware Mixup that considers the activity semantic ranges to overcome the semantic
inconsistency brought by domain differences. Secondly, we introduce the large margin loss to enhance the discrimination of
Mixup to prevent misclassification brought by noisy virtual labels. Comprehensive generalization experiments on five public
datasets demonstrate that our SDMix substantially outperforms the state-of-the-art approaches with 6% average accuracy
improvement on cross-person, cross-dataset, and cross-position HAR.
CCS Concepts: • Human-centered computing → Ubiquitous computing; • Computing methodologies → Transfer
learning.
Additional Key Words and Phrases: Human Activity Recognition, Transfer Learning, Domain Generalization
ACM Reference Format:
Wang Lu, Jindong Wang, Yiqiang Chen, Sinno Jialin Pan, Chunyu Hu, and Xin Qin. 2022. Semantic-Discriminative Mixup for
Generalizable Sensor-based Cross-domain Activity Recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 2,
Article 65 (June 2022), 19 pages. https://doi.org/10.1145/3534589∗ Wang Lu, Yiqiang Chen, and Xin Qin are also with University of Chinese Academy of Sciences. Yiqiang Chen is also with Pengcheng

Laboratory. Correspondence to: Jindong Wang and Yiqiang Chen.
Authors’ addresses: Wang Lu, Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. of Computing Tech., CAS, Beijing, China,
luwang@ict.ac.cn; Jindong Wang, jindong.wang@microsoft.com, Microsoft Research Asia, Beijing, China; Yiqiang Chen, yqchen@ict.ac.cn,
Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. of Computing Tech., CAS, China; Sinno Jialin Pan, sinnopan@ntu.edu.sg,
Nanyang Technological University, Singapore; Chunyu Hu, hcy@qlu.edu.cn, Qilu University of Technology (Shandong Academy of Sciences),
China; Xin Qin, qinxin18b@ict.ac.cn, Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. of Computing Tech., CAS, Beijing,
China.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.
Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy
otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions frompermissions@acm.org.
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
2474-9567/2022/6-ART65
https://doi.org/10.1145/3534589

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

65

65:2

•

Lu et al.

1

INTRODUCTION

Sensor-based human activity recognition (HAR) aims to train machine learning models to recognize humanactivities based on different sensor data such as accelerometer and gyroscope. HAR has wide applications in many
areas including senior care, rehabilitation, and personal fitness [10, 31]. Machine learning HAR models, especiallydeep learning-based models, often need large amounts of well-labeled data for training. However, it is expensive
and time-consuming to collect or annotate massive labeled data in real applications. Even if there exist sufficient
data for training, the performance of models may still deteriorate when applied to a new environment withdifferent data distributions. For instance, a model trained on the activity data collected from the elderly cannot bedirectly used for adults because of the difference in their body size, activity patterns, and other characteristics.

(a) Different sensor readings on different subjects.

(b) Different sensor readings on different positions.

Fig. 1. Different sensor reading distributions on different subjects and different positions when walking.

As shown in Figure 1, the sensor data collected during walking follow different distributions. From Figure 1(a),we can see data of the adult is more stable than data of the elderly when walking. And Figure 1(b) demonstrates
that there exist differences between data collected from two positions of one person at the same time. Therefore,
directly applying a model trained from existing data to a new environment may suffer from dramatic deterioration
caused by distribution shifts.
Domain adaptation (DA) [19, 36] is a popular technique to transfer the knowledge from a well-labeled source
domain to a target domain while reducing their distribution discrepancies. Over the years, DA has been successfully
applied to HAR by either aligning feature distributions via different distances [34] or adversarial training [5].
Nevertheless, DA needs access to the target data in training. This may be less realistic in modern applications
where we often expect a trained HAR model to perform well in different situations such as different environments,
people, and datasets. Recently, domain generalization (DG) [33] is gaining increasing attention. In contrast to DA,
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition

•

65:3

the goal of DG is to learn a model from one or several different but related domains that will generalize well on
unseen target domains. For example, we expect a DG algorithm that trains on activity data from diverse ages tobe able to generalize well to new persons of different ages. Existing literature on DG is mainly based on data
augmentation [35], domain-invariant learning [8], or meta-learning [2]. Since HAR applications often requirecomputation-restricted devices, we are interested in data augmentation-based DG methods for their simplicity
and effectiveness.
Mixup [42] is a popular data augmentation method and has been applied to DG [35, 40]. Mixup enlarges thediversity of training data by interpolating between different domains. However, the generalization performanceof Mixup-based DG can be undermined because of two challenges: semantic inconsistency and discriminative
slackness. Firstly, the distributions of different activities are lying on different semantic spaces while direct
Mixup will overlook their different semantic characteristics, resulting in semantic inconsistency between classes.
Secondly, the interpolation of mixup can easily generate noisy data, which will decrease discriminations of theactivity classifier.
In this paper, we propose SDMix, a Semantic-Discriminative Mixup approach for generalizable sensor-based
HAR. To prevent semantic inconsistency, SDMix utilizes a semantic-aware Mixup technique to prevent classification boundaries from approaching the activity categories with large distribution shifts. We introduce activity
semantic range for HAR that corresponds to the statistical values of activity shifts in the original space or feature
space. Therefore, this technique can eliminate negative effects brought by different semantic ranges of different
activities. Then, to handle discriminative slackness, SDMix introduces the large margin loss to replace the original
cross-entropy loss. Through the large margin loss, each class is far away from each other, which means mostvirtual data points will locate between classes and induce less noise.
To sum up, our contributions are as follows:(1) We explore Mixup for generalizable human activity recognition and empirically observe two limitations ofMixup: the semantic inconsistency and discriminative slackness, which undermine the performance ofMixup for domain generalization.
(2) We propose SDMix to handle the semantic inconsistency and discriminative slackness challenges ingeneralizable HAR by introducing semantic-aware Mixup and enhancing its discrimination using large
margin loss. SDMix is conceptually simple and easy to implement.
(3) Extensive experiments demonstrate that our proposed SDMix significantly outperforms the state-of-theart approaches (6% average accuracy improvement on cross-person, cross-dataset, and cross-position
scenarios).
The remainder of this paper is organized as follows. Section 2 provides a brief review of the related work. Section 3 presents problem formulation and background. Then, Section 4 proposes SDMix. In Section 5, experimental
evaluation and analysis on five public HAR datasets in three settings. Finally, we present the conclusions and thefuture work in Section 6.

2 RELATED WORK
2.1 Human Activity Recognition
Sensor-based HAR is receiving increasing attention over the years [31]. To alleviate the cost for data collection,
there is growing interest in applying transfer learning and domain adaptation for HAR [5, 6, 34]. All existingdomain adaptation and transfer learning-based HAR are assuming the availability of target domain data during
training. Very little attention is paid to generalizable HAR, i.e., when the test distribution is unavailable for training.
To our best knowledge, GILE [22] is the first work for generalizable HAR that disentangled domain-agnostic anddomain-specific features in the latent feature space. However, GILE is based on a variational auto-encoder which
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

65:4

•

Lu et al.

is more complicated in real applications. Our method does not use generative models and it can perform well on
simple CNNs based on Mixup data augmentation.

2.2

Domain Adaptation

Domain adaptation has developed for many years and detailed surveys can be found in [19, 36]. There is muchprior work focusing on HAR with domain adaptation and a detailed survey can be found in [6]. For cross-domain
activity recognition, Wang et al. [32] proposed a stratified transfer learning (STL) algorithm that learns theclass-wise feature transformation. Wang et al. [34] designed a unified framework for source domain selection
and activity transfer. Later, to select the most similar source domain to the target domain and perform accuratelytransfer activity, Qin et al. [24] proposed an adaptive spatial-temporal transfer learning (ASTTL) approach. Most
recently, Lu et al. [15] tried to make full use of data structures and matched substructures of data via optimaltransport. Although the above work can relieve distribution shifts, it cannot cope with situations when targets
data is inaccessible.

2.3

Domain Generalization

Domain generalization (DG) tries to learn a model from one or several different but related domains that will
generalize well on unseen testing domains. DG is different from leave-one-out-cross-validation (LOOCV) [37]since LOOCV is a model selection technique which selects models via parts of training data while DG is toenhance generalization on unseen targets. Existing domain generalization methods can be grouped into three
categories [33]: data manipulation, representation learning, and learning strategy. Data manipulation contains two
kinds of popular techniques: data augmentation [9, 18] which is mainly based on augmentation, randomization,transformation of input data, and data generation [14, 23] which generates diverse samples to help generalization.
Representation learning also includes two kinds of representative techniques: domain-invariant representation
learning [13, 25] which performs kernel, adversarial training, explicitly feature alignment between domains,
or invariant risk minimization to learn domain-invariant representations and feature disentanglement [12, 41]
which tries to disentangle the features into domain-shared or domain-specific parts for better generalization.
Learning strategy mainly has three kinds of methods: ensemble learning [16] which relies on the power ofensemble to learn a unified and generalized predictive function, meta-learning [2, 30] which is based on thelearning-to-learn mechanism to learn general knowledge by constructing meta-learning tasks to simulate domain
shift, and gradient operation [11, 28] which tries to learn generalized representations by directly operating on
gradients. For more details, please refer to [33]. Despite the popularity of DG in recent years, most of the methods
are proposed for computer vision tasks that do not consider the application of HAR.

2.4

Mixup

Mixup [42] is a simple but effective technique for data augmentation. It extends the training distribution byincorporating the prior knowledge that linear interpolations of feature vectors should lead to linear interpolationsof the associated targets. Although vanilla Mixup has shown its ability for domain generalization, many adapted
versions are proposed for better performances. Recent work [38, 39] used the vanilla Mixup for domain adaptation
without modifications. Wang et al. [35] mixed up samples across multiple source domains with two different
sampling strategies to improve the generalization performance across different tasks. They perform well on
computer vision while there is no Mixup method designed specifically for HAR.

3

PROBLEM FORMULATION AND PRELIMINARIESWe are given training data from several source domains. Our goal is to train a model that can perform well on an
unseen target domain with these available source domain data. Since the target data have different distributions
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition

•

65:5

compared to the source domains, we expect that the trained model is generalizable. A common technique toenhance generalization capability is data augmentation and Mixup is a common way. Mixup may encounter two
challenges: semantic inconsistency and discriminative slackness. To cope with these two challenges, SDMix is
proposed which utilizes a semantic-aware Mixup technique to eliminate negative effects brought by different
semantic ranges of different activities and introduces the large margin loss to reduce virtual noisy data.

3.1

Problem Formulation

Following the definition of generalizable cross-domain activity recognition from existing work [22], we are given
𝑆 . We use 𝑃 𝑖 (x, 𝑦) on X × Y to denote the joint
𝑆 labeled source domains as the training dataset: D 𝑡𝑟 = {D 𝑖 }𝑖=1
distribution of one domain, where x ∈ X ⊂ R𝑚 denotes the input and 𝑦 ∈ Y = {1, · · · , 𝐶} corresponds to output.
𝑚 and 𝐶 denote the input dimension and number of classes. Our goal is to learn a generalized model from D 𝑡𝑟 to
predict well on an unlabeled target domain, D𝑇 , which is unseen in training. In our problem, the training andtest domains have the same input and output spaces but different distributions, i.e., 𝑃 𝑖 (x, 𝑦) ≠ 𝑃 𝑗 (x, 𝑦), ∀𝑖, 𝑗 ∈{1, 2, · · · , 𝑆,𝑇 }. We aim to train a model ℎ from D 𝑡𝑟 to minimize the risk on D𝑇 : minℎ E (x,𝑦)∼𝑃𝑇 [ℎ(x) ≠ 𝑦].

3.2

Background

Mixup [42] is a popular approach for data augmentation. Given two random samples (x1, 𝑦1 ), (x2, 𝑦2 ), Mixup
extends the training distribution by incorporating the prior knowledge: linear interpolations of feature vectors
should lead to linear interpolations of the associated targets, formulated as 1 :x̃ = 𝜆x1 + (1 − 𝜆)x2,𝑦˜ = 𝜆𝑦1 + (1 − 𝜆)𝑦2,(1)

where 𝜆 ∼ 𝐵𝑒𝑡𝑎(𝛼, 𝛼) and 𝛼 ∈ (0, ∞) is a hyperparameter. As a powerful data augmentation technique, Mixup
plays a vital role to enlarge the distribution diversity in existing domain generalization research [35, 40].
We are specifically interested in applying Mixup to generalizable HAR because of its simplicity. Unfortunately
in HAR, Mixup faces the semantic inconsistency and discriminative slackness challenges that will dramaticallyundermine its performance.

4

SDMIX

In this paper, we propose SDMix (Semantic-Discriminative Mixup) to solve these two challenges in generalizable
HAR. In the following, we will introduce the key components of SDMix.

4.1

Semantic-aware Mixup

In generalizable HAR, the distributions of sensor reading from two domains are different due to different lifestyles,
habits, body shapes, and device positions. While Mixup tries to increase the diversity of distributions, we argue
that the activity semantic range will largely impede Mixup.
Definition 1 (Activity semantic range). The activity semantic range of a domain D refers to the maximum
shift towards the activity center, denoted as:
𝑅𝑐 = max 𝑑 (𝜇x|𝑦=𝑐 , x : x ∈ 𝑃 (x|𝑦 = 𝑐)),(2)where 𝜇x |𝑦=𝑐 = E[x|𝑦 = 𝑐] denotes the activity semantic center for class 𝑐 and 𝑑 (·, ·) is a certain distance function.
Definition 2 (Semantic inconsistency). If data of two classes have different activity semantic ranges, i.e.,
𝑅𝑐 1 ≠ 𝑅𝑐 2 , then we say that there exists a semantic inconsistency between these two classes.
1 When performing Mixup, labels are often in the form of one-hot.

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

65:6

•

Lu et al.

Different activities or even the same activity performed by different persons (domains) tend to have different
activity semantic ranges. For instance, a person that walks in a parking lot may have larger activity semantic
range than another person that walks on a treadmill with a similar speed in flat since walking in a parking lot ismore unstable and contains more degree of freedom. Another example is that two persons both playing basketball
may have different activity semantic ranges due to different habits and positions. The existing Mixup cannot
consider the influence of activity semantic range from two different domains, resulting in semantic inconsistency.
Activity semantic range varies between any two classes from any different domains. This situation cannot be
avoided in both the feature space (Figure 7) and the original space. As shown in Figure 2(a), when the above
situation occurs that two classes have different activity semantic ranges, the vanilla Mixup generates wrong
classifications near the decision boundary. Note that vanilla Mixup utilizes the same 𝜆 for data and categories,
which neglects the influence of activity semantic ranges.
In this paper, we propose semantic-aware Mixup to reduce the influence of different activity semantic ranges
from different domains. The samples generated by semantic-aware Mixup can be formulated as:x̃ = 𝜆x𝑖1 + (1 − 𝜆)x2𝑗 ,𝑦˜ = 𝑡𝑦𝑖1 + (1 − 𝑡)𝑦2𝑗 ,(3)

𝜆 ∼ 𝐵𝑒𝑡𝑎(𝛼, 𝛼),where (x𝑖1, 𝑦𝑖1 ), (x2𝑗 , 𝑦2𝑗 ) denote samples from domain 𝑖 and 𝑗. And 𝑡 is the activity semantic factor, computed as:𝑡=

𝜆 × 𝑅𝑐𝑖 1,(4)

𝜆 × 𝑅𝑐𝑖 1 + (1 − 𝜆) × 𝑅𝑐𝑗2

where 𝑅𝑐𝑖 1 and 𝑅𝑐𝑗2 are the activity semantic ranges for the class 𝑐 1 and 𝑐 2 of domains D 𝑖 and D 𝑗 , respectively.
Note that 𝑐 1 and 𝑐 2 may or may not be the same, indicating that this works for both the same or different classes.
We only apply 𝑡 for 𝑦, which is equivalent to calibrating mixed 𝑦˜ with semantic range for a mixed x̃.
What does Eq. (4) do? We illustrate its effects in Figure 2, where the larger 𝑅 makes the corresponding 𝑦play a more important role. In Figure 2(a), vanilla Mixup pays no attention to the activity semantic range andtreats x𝑖 equally. Therefore, its decision boundary is biased towards the class with larger activity semantic range.
While in Figure 2(c), the boundary is balanced by considering the effects of different activity semantic ranges. In
Figure 2(b) and Figure 2(d), a toy example where simple models learned from simulation data following Gaussian
distributions is to demonstrate that SDMix can learn better decision boundaries.
We present two alternatives to compute the value of activity semantic range 𝑅𝑐𝑖 by either using the largest
distance from samples to centroid, or the mean of all distances:
𝑅𝑐𝑖 = max 𝑑 (x, 𝜇𝑐𝑖 ),

(5a)

x∈D𝑐𝑖

Í
𝑅𝑐𝑖 =

𝑖
x∈D𝑐𝑖 𝑑 (x, 𝜇𝑐 ),|D𝑐𝑖 |(5b)where D𝑐𝑖 represents the samples associated with class 𝑐 from domain D 𝑖 and |·| is the cardinality. For computation,𝑑 (·, ·) can be ℓ1 , ℓ2 -distances or cosine distance. We choose different computing functions to achieve better
performance. 𝜇𝑐𝑖 denotes the 𝑐-th class center of domain D 𝑖 :
1 ∑︁
𝜇𝑐𝑖 =
x.
(6)
|D𝑐𝑖 |
𝑖x∈D𝑐

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition

𝑗

𝑅𝑐𝑖 1
𝑅𝑐𝑖 1𝑖 𝑖(𝑥1 , 𝑦1 )(𝑥1𝑖 , 𝑦1𝑖 )

1−𝜆
1
(𝑥,
෤ 𝑦)
෤− 𝜆
𝑥,
෤ 𝑦)
෤
𝜆 ∼ (𝐵𝑒𝑡𝑎(𝛼,
𝛼)

𝑅𝑐2
𝑗
𝑅𝑐2

𝜆
𝜆

𝜆 ∼ 𝐵𝑒𝑡𝑎(𝛼, 𝛼)𝑗𝑗(𝑥2 , 𝑦2 )𝑗 𝑗
(𝑥2 , 𝑦2 )Classification boundary
Classification boundary(a) Illustration (vanilla Mixup)

𝑅𝑐𝑖𝑖1
𝑅𝑐1
(𝑥1𝑖𝑖 , 𝑦1𝑖𝑖 )(𝑥1 , 𝑦1 )

𝑡
1−𝑡
𝑡
1−𝑡
(𝑥,
෤ 𝑦)
෤
(𝑥,
෤ 𝑦)
෤
𝑖

10
8
6
4
2
0
2
4
6
88

6

4

2 0

2

4

6

•

65:7

8

(b) Decision boundary (vanilla Mixup)
𝑗
𝑅𝑗 𝑐
𝑅𝑐2 2

𝜆𝑅𝑐
𝑗 𝑗
(𝑥𝑗 2 , 𝑦𝑗 2 )𝜆𝑅𝑐𝑖 1𝑡=𝑡 = 𝜆𝑅𝑐𝑖 + (11− 𝜆𝑅𝑗𝑐𝑗 ) (𝑥2 , 𝑦2 )𝜆𝑅𝑐𝑖 11 + (1 − 𝜆𝑅𝑐2 2)Classification boundary
Classification boundary

10
8
6
4
2
0
2
4
6
88

(c) Illustration (ours)

6

4

2 0

2

4

6

8

(d) Decision boundary (ours)

Fig. 2. The semantic inconsistency of Mixup. Different colors correspond to different classes. (a) Vanilla Mixup pays no
attention to activity semantic range. (b) Decision boundary for vanilla Mixup leads to semantic inconsistency. (c) Our method
considers the effects of activity semantic range. (d) Our semantic-aware Mixup mitigates such an issue.

4.2

Enhancing Discrimination of Mixup

Other than activity semantic range that describes the activity shift in HAR, we also notice another limitation of
Mixup: the virtual noisy activity instance will influence the discrimination of Mixup.
˜ generated by the Mixup of (x1, 𝑦1 )Definition 3 (Virtual noisy activity instance). The interpolation ( x̃, 𝑦)and (x2, 𝑦2 ) is a virtual noisy activity instance, if
˜ 1 ≥ 𝜖,||ℎ( x̃) − 𝑦||(7)where ℎ denotes the classification function and 𝜖 > 0 is a threshold.
Definition 4 (Discriminative slackness). When trained the model with virtual noisy activity instances via
Vanilla Mixup, the model may fail to discriminate different classes. We say that the model meets the discriminative
slackness problem.
Specially speaking, when x1 is close to the ideal decision boundary of class 𝑦2 (or x2 to 𝑦1 ), x̃ could be extremely
˜ And from the point of location, x̃ mayclose to the decision boundary of 𝑦1 or 𝑦2 instead of their interpolation 𝑦.
belong to 𝑦ˆ which is different from 𝑦. For instance, due to different lifestyles, the running data of one person
may have distributions similar to the walking data of another person. Thus, his data is easily misclassified. This
causes discriminative slackness of Mixup, as shown in Figure 3(a).
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

65:8

•

Lu et al.

6
4
2
0
2
4

6
4
2
0
2
4

(x, y)
4

2

0

2

4

6

(a) Discrimination (vanilla Mixup)(x, y)
4

2

0

2

4

6

(b) Discrimination (ours)

Fig. 3. Discriminative slackness of Mixup. The generated data in (a) can easily be misclassified. Different colors correspondto different classes, and different shapes correspond to different domains. Stars in blue or yellow are selected data points for
mixing. The red star is the virtual data point. Best viewed in color.

In this paper, we introduce large margin loss to enhance the discrimination of Mixup, which is formulated as:ℓ𝑦 (x𝑘 , 𝑦𝑘 ) = 𝒜𝑐≠𝑦𝑘 max{0, 𝛾 + 𝑢ℎ,x𝑘 ,{𝑐,𝑦𝑘 } sign(ℎ𝑐 (x𝑘 ) − ℎ 𝑦𝑘 (x𝑘 )},(8)

where ℓ𝑦 is the large margin loss, 𝒜 is an aggregation operator for multi-class setting, and sign(·) adjusts thepolarity of the distance. ℎ𝑐 : X → R or ℎ 𝑦𝑘 is a function that generates a prediction score for classifying the input
vector x ∈ X to class 𝑐 or 𝑦𝑘 . 𝛾 is the distance to the boundary which we expect. 𝑢ℎ,x,{𝑐 1,𝑐 2 } is the distance of apoint x to the decision boundary of class 𝑐 1 and 𝑐 2 , which is computed as:
𝑢ℎ,x,{𝑐 1,𝑐 2 } = min ||𝛿 ||𝑝 ,𝑠.𝑡 . ℎ𝑐 1 (x + 𝛿) = ℎ𝑐 2 (x + 𝛿),

(9)

𝛿
𝑝

where || · ||𝑝 is 𝑙𝑝 norm. As shown in [7], define 𝑞 = 𝑝−1 , then Eq. (8) can be computed as:)(ℎ𝑐 (x𝑘 ) − ℎ 𝑦𝑘 (x𝑘 ).
ℓ𝑦 (x𝑘 , 𝑦𝑘 ) = 𝒜𝑐≠𝑦𝑘 max 0, 𝛾 +∇xℎ𝑐 (x𝑘 ) − ∇xℎ 𝑦𝑘 (x𝑘 ) 𝑞(10)

We illustrate the effect of large margin in Figure 3. In Figure 3(a), two classes with shape circle are far awayfrom each other, but yellow circle data points are close to blue plus points. Direct adoption of the vanilla Mixup
with the blue star and the yellow star will generate the red star, which is a virtual noisy data point (according
to the location of data points). This means that a small class margin may induce terrible virtual labels when
performing Mixup. In Figure 3(b), we can see large margin can alleviate this issue.

4.3

SDMix for HAR

Similar to the vanilla Mixup, our proposed SDMix remains conceptually simple and can be easily trained in an
end-to-end manner. More importantly, our SDMix considers the semantics for different domains and also reduces
noisy data when performing Mixup. Therefore, it makes the model perform better on unseen target data.
Figure 4 shows the network architecture of the proposed method. Mixed data goes through two blocks andeach block contains one convolution layer and one pooling layer acting as feature extractor 𝐺 𝑓 . We view 1D timeseries as 2D data with the height being 1, following Fig. 2. in [31]. Then, it goes through one fully-connected layer
which serves as the classification layer 𝐺 𝑦 . Semantics are fused via weighted label Mixup. In Figure 4, we only
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition

Feature extractor (𝐺𝑓 )Classification Layer (𝐺𝑦 )

•

65:9

Large margin loss

𝑥𝑖
FC

Semantic

Mixup

𝑥𝑗
𝑥෤

Conv.

Max
poolingConv.

Max
pooling

Fig. 4. The network architecture of the proposed method.

show conducting Mixup in the original space. We also can perform Mixup on the outputs of the feature extractor
which can bring better results in some circumstances. In addition, we do not show BN layers in Figure 4.
The loss of SDMix can be expressed as:ℓSDMix (x1, 𝑦1 ; x2, 𝑦2 ) = ℓ𝑦 (Mix𝜆 (x1, x2 ), Mix𝜆 (𝑦1, 𝑦2 )),(11)

where Mix𝜆 (·, ·) represents the semantic-aware mix operation computed by Eq. (3). 𝒛 is the outputs of the feature
extractor. ℓ𝑦 is the large margin loss.
Therefore, our learning objective is formulated as:
min E𝑃1,𝑃2 ∼𝑃 D E (x1,𝑦1 )∼𝑃1,(x2,𝑦2 )∼𝑃2 E𝜆∼𝐵𝑒𝑡𝑎 (𝛼,𝛼) [𝑙 SDMix (x1, 𝑦1 ; x2, 𝑦2 )] ,(12)

where 𝑃 D is the distribution of domains and 𝑃1, 𝑃2 are distributions of data in selected domains. Note that suchan objective is optimized in a mini-batch style in deep networks and the details will be introduced in experiments.
Specifically, we choose two source domains randomly each time and choose two samples from these two selectedsource domains respectively to perform semantic-aware Mixup. Semantic factors are updated every several
iterations. For fairness (the same number of optimization iterations), the model only utilizes interpolated examples
and does not utilize the original training examples. We believe that our method can achieve better performancewith both original data and virtual data generated via mixing.

5

EXPERIMENTS

We evaluate the proposed method by constructing different generalization scenarios on five public datasets.

5.1

Datasets

The statistical information of the five public datasets is shown in Table 1, which includes sensor types, sampling
frequency, and activity classes. Now, we briefly introduce them in the following.
The UniMib SHAR dataset (SHAR) [17] records 9 types of activities of daily living and 8 types of falls. 30
subjects (aged from 18 to 60) conduct the 17 fine-grained activities with data collected by an acceleration
sensor embedded in an Android phone. 17 activities include 9 types of activities of daily living (StandingUpFL,
LyingDownFS, StandingUpFS, Running, SittingDown, GoingDownS, GoingUpS, Walking, Null) and 8 types of falls(FallingBackSC, FallingBack, FallingWithPS, FallingForw, FallingLeft, FallingRight, HittingObstacle, Syncope).
UCI daily and sports dataset (DSADS) [3] consists of 19 activities collected from 8 subjects wearing body-worn
sensors on 5 body parts. 19 activities include sitting, standing, lying on back and on right side, ascending andProc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

65:10•Lu et al.

descending stairs, standing in an elevator still, moving around in an elevator, walking in a parking lot, walking
on a treadmill with a speed of 4 km/h, running on a treadmill with a speed of 8 km/h, exercising on a stepper,
exercising on a cross trainer, cycling on an exercise bike in horizontal and vertical positions, rowing, jumping,and playing basketball. Each subject wears three sensors: accelerometer, gyroscope, and magnetometer.
PAMAP2 physical activity monitoring dataset (PAMAP2) [26] contains data of 18 different physical activities,
performed by 9 subjects wearing 3 sensors. 18 activities include lying, sitting, standing, walking, running, cycling,
Nordic walking, watching TV, computer work, car driving, ascending stairs, descending stairs, vacuum cleaning,ironing, folding laundry, house cleaning, playing soccer, rope jumping, and other (transient activities). The
sampling frequency is 100Hz and the data dimension is 27 (3 inertial measurement units).
USC-SIPI human activity dataset (USC-HAD) [43] is composed of 14 subjects (7 male, 7 female, aged from 21
to 49) executing 12 activities with a sensor tied on the front right hip. The data dimension is 6 and the sample
rate is 100Hz. 12 activities include Walking Forward, Walking Left, Walking Right, Walking Upstairs, Walking
Downstairs, Running Forward, Jumping Up, Sitting, Standing, Sleeping, Elevator Up, and Elevator Down.
Human activity recognition using smartphones dataset (UCI-HAR) [1] is collected by 30 subjects performing
6 daily living activities with a waist-mounted smartphone. 6 activities include walking, sitting, lying, standing,walking upstairs, walking downstairs. The data dimension is 6 and the sample rate is 50Hz.
Table 1. Statistical information of five datasets.

DatasetSHAR
DSADSUCI-HAR
PAMAP2
USC-HAD

5.2#Subject #Sensor
30
8
30
9
14

1
3
6
3
2

#Class#Sample

17
19
6
18
12

236,919
1,140,000
1,310,000
3,850,505
5,441,000

Experimental Setup

We construct three types of experiments to thoroughly evaluate all methods for generalizable HAR: (1) Crossperson HAR similar to existing research [22], (2) Cross-dataset HAR, which is our construction that is morechallenging than cross-person HAR since cross-dataset setting implies that not only the persons are different, butalso the sensor types and positions, and (3) Cross-position HAR, which utilizes data from different positions withsame persons. In all three settings, we adopt the sliding window technique [4] with 50% overlap to constructtraining samples following common practice in HAR [31]. The inputs should be cut into individual inputs
according to the sampling rate and each new input serve as one instance.
We give a more detailed introduction of three settings in the following.
• Cross-Person: We conduct experiments on SHAR, DSADS, PAMAP2, and USC-HAD for this setting. We
divide each dataset into 4 domains, e.g., 8 people are evenly divided into 4 domains where each domain
consists of data from 2 people and there is no overlap between domains. For SHAR, we follow GILE tochoose four persons. For the other datasets, we divide all persons into 4 groups. For PAMAP2, we choose
12 classes to ensure that each class has a certain number of data.
• Cross-Dataset: We conduct experiments on DSADS, PAMAP2, USC-HAD, and UCI-HAR for this setting.
We do not split domains; instead, each dataset can be treated as one domain and we take the common
classes from all datasets. To perform the Cross-Dataset setting, we first need to unify X and Y. For X, two
sensors that belong to the almost same position are selected and each sample contains six dimensions. Then,
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition

•

65:11

data is down-sampled and the sampling rate is 25Hz. For Y, six common classes are selected, including
walking, walking upstairs, walking downstairs, sitting, standing, and lying.
• Cross-Position: We conduct experiments on DSADS for this setting. 3 sensors are located in five parts ofa body in DSADS. Each body part corresponds to a different domain. Each sample contains three sensors
with nine dimensions.
Table 2. Detailed information on cross-person, cross-dataset, and cross-position experiments.

Cross-Person
Dataset#Domain#Sensor#Class#Domain Sample#TotalSHARDSADSPAMAP2
USC-HAD

4
4
4
4

1
3
3
2

17
19
12
12

(57,984;88,033;45,904;44,998)(285,000)×4(592,600; 622,200; 620,000; 623,400)(1,401,400;1,478,000;1,522,800;1,038,800)

236,919
1,140,000
2,458,200
5,441,000

Cross-Dataset
Dataset#Domain#Sensor#Class#Domain Sample#Total4

2

6(672,000;810,550;514,950;470,850)

2,468,350

Cross-Position
Dataset#Domain#Sensor#Class#Domain Sample#Total

DSADS

5

3

19

(1,140,000)*5

5,700,000

Table 2 shows the information on cross-person, cross-dataset and cross-position scenarios. It is worth notingthat since SHAR dataset contains much fewer samples than others, we replace it with UCI-HAR dataset [1] forcross-dataset experiments. The reason we did not run cross-person experiments on UCI-HAR is that the resultson this dataset are all near 100%. In total, we constructed 16, 4, and 5 tasks for cross-person, cross-dataset, andcross-position HAR, respectively. We use 0, 1, 2, 3 to denote the four divided domains for cross-person experiments.
We use 0, 1, 2, 3, 4 to denote the five divided domains for cross-position experiments.
Baselines. The latest method for generalizable HAR is GILE (Generalizable Independent Latent Excitation) [22].
Since there are no other methods designed for HAR except GILE, we turn to comparing with several populardomain generalization methods. DeepAll combines all source data together and trains a model according to ERM.
DANN [8] and Coral [29] are two traditional DG methods, and we extend them to fit the domain generalization
setting. Mixup [42], GroupDRO [27], RSC [11], and ANDMask [20] are four state-of-the-art universal domain
generalization methods which can be directly used for HAR. Except GILE, we reproduced all other methods
with the same network architecture in Pytorch [21] for fairness. GILE is based on VAE, and we directly use theirreleased code. Table 3 shows information on the input size and kernel size of the models in all settings and thefinal dimension of input size represents the window size. We do not compare other Mixup-based DG methods
since they are built for computer vision tasks that are not applicable to HAR problems.
To obtain the final model, we split data of the source domains into the training splits and validation splits. The
training splits are utilized for training the model while the validation splits are utilized to guide the selection of
the model. For all benchmarks, we select the best model via validation accuracy. In practice, we leave 80% of
source domain data as training splits while the rest data are for validation. For testing, we evaluate the selectedmodels on all data of the target domain. For our architecture, the model contains two blocks, and each has one
convolution layer, one pool layer, and one batch normalization layer. A single fully-connected layer serves as
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

65:12•Lu et al.

Table 3. Information on the architectures of the models.

Setting

Cross-Person

Cross-position
Cross-dataset

DatasetInput

Kernel Size

SHAR
DSADS
PAMAP2
USC-HAD-(3,1,151)(45,1,125)(27,1,200)(6,1,200)(9,1,125)(6,1,50)

(1,9)
(1,9)
(1,9)
(1,6)
(1,9)
(1,6)

the classifier. In each step, each domain selects 32 samples. The maximum training epoch is set to 150. For our
method, we search best hyperparameters within [0.1, 0.2, 0.5, 1, 10] for 𝛼, [1, 2, 5] for top 𝑐 in large margin loss,
and [10, 100, 10000, 100000] for 𝛾. For all methods except GILE2 , the Adam optimizer with a learning rate 10−2
and weight decay 5 × 10−4 is used. We use outputs of the feature extractor to compute radii and utilize different
computing ways for different datasets. For fair study, we extensively tune hyperparameters for each method andreport their average results of three trials. Code for SDMix will be available at http://transferlearning.xyz.

5.3

Experimental Results

The results of cross-person, cross-dataset, and cross-position HAR are shown in Table 4, 5, and 6 respectively. On
average, our proposed SDMix substantially outperforms the second-best methods: 6.0% for cross-person, 10.33%
for cross-dataset HAR, and 2.34% for cross-position HAR. Moreover, from Figure 5, we can see that our method
also achieves the best F1 score compared with other state-of-the-art methods whether in a balanced or unbalanced
situation. These indicate that our method is effective for generalizable cross-domain HAR applications.

Accuracy (%)

90

Precision
RecallF1 score

60
Precision
55
RecallF1 score
50
45
40
35
30
25
20 DeepALL DANN

Accuracy (%)

95

85
80
75
70 DeepALL DANNCORAL ANDMask Mixup

Task

RSC GroupDRO Ours(a) Cross-person: nearly balanced classification

CORAL ANDMask Mixup

Task

RSC GroupDRO Ours(b) Cross-dataset: imbalanced classification

Fig. 5. Precision, recall, and F1 score.Fig. 5(a) contains results from the first task in DSADS where data of the first two persons
are used for testing and data of the rest persons are used for training. Each class has an equal number of instances (120). In
Fig. 5(b), data in USC-HAD are used for testing while data in other datasets are used for training. Classes in USC-HAD have
2545, 2290, 3608, 2048, 1904, 3744 instances respectively.

We observe more insightful conclusions. (1) When are Mixup-based methods most effective? When the dataset
volume is rather small with more classes, i.e., on difficult tasks. This can be verified on SHAR and DSADS datasetsthat have fewer samples but more classes. In these datasets, Mixup-based methods dramatically increase the2 The source code for GILE is available at https://github.com/Hangwei12358/cross-person-HAR

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition

•

65:13

Src

Tar

DeepAll

DANN

CORAL

ANDMask

GroupDRO

RSC

Mixup

GILE

SDMix

SHAR

1,2,3
0,2,3
0,1,3
0,1,2
AVG

0
1
2
3
-

57.98±5.90
57.29±3.26
69.96±2.20
40.16±1.2356.35±1.7855.99±1.8250.20±3.5463.92±3.72
42.06±2.4653.04±2.1657.90±1.65
59.92±1.60
69.41±1.65
40.60±0.33
56.96±0.57

58.33±5.21
48.89±2.75
66.34±2.8542.39±3.1353.99±1.9354.60±1.74
58.72±5.20
68.09±2.6342.17±0.56
55.89±1.33

57.81±3.1255.06±6.17
67.87±2.08
40.27±1.3455.25±3.1758.68±1.9155.69±3.89
69.96±4.1741.83±1.2356.53±2.2149.57±3.57
48.02±4.02
65.44±5.4439.27±3.2750.54±3.04

65.80±0.70
59.86±2.4074.45±0.77
47.99±0.67
62.03±0.47

DSADS

1,2,3
0,2,3
0,1,3
0,1,2
AVG

0
1
2
3
-

83.26±4.2277.79±0.95
84.68±3.0174.74±3.7880.12±1.08

88.09±4.84
79.37±2.57
82.48±3.1476.05±6.18
81.50±2.8490.51±2.00
83.58±5.20
83.04±4.8475.45±1.37
83.15±1.3285.17±1.05
77.73±1.94
83.32±5.38
78.17±3.8781.10±3.03

91.77±1.1684.30±1.58
82.06±8.51
78.48±0.23
84.15±2.15

84.21±1.9378.79±2.47
81.48±4.4677.12±1.9480.40±1.88

88.08±2.1280.95±1.65
88.00±2.52
84.20±1.9685.31±1.76

79.67±1.67
75.00±0.00
77.00±1.00
67.00±1.00
74.65±0.15

94.20±1.04
91.07±0.5493.61±0.80
87.00±1.39
91.47±0.27

PAMAP2

1,2,3
0,2,3
0,1,3
0,1,2
AVG

0
1
2
3
-

89.02±0.60 85.78±3.60
75.80±1.84 75.86±1.61
51.83±6.77 49.47±10.89
82.98±3.06 84.64±3.41
74.91±1.56 73.94±2.4482.82±6.8577.43±1.4448.14±7.20
84.47±3.6273.21±3.1887.54±0.80
77.97±1.5345.36±7.94
84.01±3.4573.72±1.1283.42±2.08
74.95±2.37
50.59±8.20
82.66±1.9772.90±2.1587.14±2.29 80.24±10.41
77.31±5.95 78.77±3.8155.11±3.40 55.53±4.37
83.64±5.30 86.63±0.68
75.80±2.67 75.29±3.28

83.33±0.33
68.67±0.67
44.00±2.00
76.67±0.67
68.25±1.00

89.14±0.51
84.99±0.29
63.11±2.2788.77±0.2281.50±0.53

USC-HAD

Table 4. Classification accuracy (± std. error) for cross-person HAR. The bold and underline items denote the best andsecond-best results, respectively.

1,2,3
0,2,3
0,1,3
0,1,2
AVG

0
1
2
3
-

79.90±3.80
59.04±2.50
72.57±1.2767.15±1.29
69.67±1.3880.54±1.3961.36±3.4875.90±0.45
69.33±4.06
71.78±1.97

79.02±3.1159.87±3.89
74.41±1.6460.42±6.70
68.43±1.81

79.46±0.94
60.01±4.69
73.74±1.1965.58±0.5469.70±1.02

81.57±1.45
58.96±3.45
71.59±1.6159.47±2.4467.90±0.51

80.05±2.5757.04±2.60
74.25±0.86
66.99±1.8669.58±0.19

79.70±2.66
57.43±4.1473.83±1.1959.91±2.9867.72±1.60

78.67±0.67 84.09±0.66
63.00±1.00 62.86±0.74
77.00±1.00 76.46±0.97
61.67±0.67 71.90±0.05
70.08±0.08 73.83±0.45

AVG all

-

70.26

70.07

70.44

69.63

70.21

70.26

71.21

77.21

65.88

Table 5. Classification accuracy for cross-dataset HAR. The bold and underline items are the best and second-best results.
TargetDeepAll

DANN

CORAL

ANDMask

GroupDRO

RSC

Mixup

SDMix

DSADS, USC, PAMAP2
UCI-HAR
USC, UCI-HAR, PAMAP2DSADSDSADS, USC, UCI-HAR
PAMAP2DSADS, UCI-HAR, PAMAP2
USC

Source

46.06
29.73
43.84
45.33

39.10
39.46
36.61
41.82

44.44
26.35
32.93
29.58

43.22
41.66
40.17
33.83

33.20
51.41
33.80
36.74

45.28
33.10
45.94
39.70

40.24
37.35
23.12
47.39

46.41
52.66
53.65
53.54

41.24

39.25

33.32

39.72

38.79

41.01

37.03

51.57

AVG

-

Table 6. Classification accuracy for cross-position HAR. The bold and underline items are the best and second-best results.

SourceTargetDeepALLDANN CORAL

ANDMask GroupDRO

1,2,3,4
0,2,3,4
0,1,3,4
0,1,2,4
0,1,2,3

0
1
2
3
4

41.52
26.73
35.81
21.45
27.28

45.45
25.36
38.06
28.89
25.05

33.22
25.18
25.81
22.32
20.64

47.51
31.06
39.17
30.22
29.90

AVG

-

30.56

32.56

25.43

35.57

RSC

Mixup

SDMix

27.12
26.66
24.34
18.39
24.82

46.56
27.37
35.93
27.04
29.82

48.77
34.19
37.49
29.50
29.95

47.50
36.10
42.53
34.52
30.93

24.27

33.34

35.98

38.32

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

65:14•Lu et al.

055Actual010(a) Vanilla Mixup

Predicted

15

5

10

Predicted

15

10

5

15

0

15

1
6
8
9

10

0

Actualdiversity of data distributions that benefits generalization. On the larger PAMAP2 and USC-HAD datasets thatinvolve more diversity, the vanilla Mixup only slightly surpasses DeepAll or even worse. (2) When is SDMix lesseffective? When the training domains are much larger than the test domains, i.e., the training domains are alreadycontaining diverse3 data, where SDMix may bring smaller improvements. In our statistics, source domains 1 ∼ 3
in PAMAP2 have more samples that make it enough to learn models directly using DeepAll. The same goes forcross-dataset HAR when UCIHAR is the target. In addition, when activity semantic range is hard to compute,
SDMix only slightly surpasses other methods or even worse. For example, there exist little information and many
categories in the cross-position setting, which makes it hard to estimate activity semantic range. SDMix only
has a slight improvement compared with other methods and even performs worse on the first task. (3) What
influences stability in HAR generalization? Algorithms, random states, data splits, and so on. Whichever algorithm
is used, there exist fluctuations among the results of three trials. Even with DeepAll, results are also different.
From Table 4, we can see that our method achieves the smallest STD compared to other state-of-the-art methods
except GILE4 . The results demonstrate that our method is still reliable and stable with three trials compared toother state-of-the-art methods. (4) What influences generalization in HAR? Dataset quantity, diversity, and thecross-domain distribution discrepancy. There is no doubt that the generalization ability of a model will increase
when the dataset becomes larger and diverse (rf. Table 4). More importantly, we see a significant performancedrop from cross-person to cross-position HAR, indicating the importance of cross-domain discrepancies [5]. The
cross-dataset and position scenario is more challenging than cross-person since the sensor devices, positions,subjects are all different. In this scenario, our SDMix achieves the best performance by harnessing their diversity.
Note that there is still room for improvement in this setting.

(b) Our method(c) Visualization

Fig. 6. Confusion matrices and Visualization on DSADS dataset for target 0 in the Cross-Person setting. Fig. 6(a) and Fig. 6(b)are confusion matrices of vanilla Mixup and our method. Deeper colors denote larger values. Fig. 6(c) is the visualization of
the t-SNE embeddings with features learned by a simple model. Each class is denoted by a color and each shape corresponds
to a domain. Best viewed in color and zoom in.

5.4

Analysis

Why is SDMix effective? Do semantic inconsistency and discriminative slackness really exist? To analyze therationale behind the effectiveness of SDMix, we plot the confusion matrices for the vanilla Mixup and ours fortarget 0 on DSADS dataset in Figure 6. While most classes are correctly classified, we are more interested in the
3 Diversity is relative. Few instances may be enough for simple situations (e.g. iid) while it may need more instances for difficult cases.
4 GILE utilizes a different network architecture from other methods. Moreover, GILE selects the best model according to the target, which isunrealistic in reality and may be contrary to the principle of domain generalization.

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition

•

65:15

misclassified classes: class 8 (walking in a parking lot) and class 9 (walking on a treadmill with a speed of 4km/h inflat); class 1 (standing) and class 6 (standing in an elevator still). Firstly, as shown in Figure 6(a), some samples inclass 8 are misclassified as class 9 (Please see the row labeled 8). It is caused by different activity semantic ranges
since walking in a parking lot has more variance than walking on a treadmill. Our method can correctly classify
them. Secondly, some samples in class 6 are misclassified as class 1 (Please see the row labeled 6) since standing
and standing in an elevator still are two similar activities with small differences that weaken the discriminationof Mixup. Again, our method can mitigate these two issues to achieve the best results as shown in Figure 6(b).
As shown in Figure 6(c), class 8 in color 8 with shape circle has a larger semantic range than class 9 while class
1 almost compasses class 6 which illustrates they are close to each other. Figure 6(c) describes the phenomena
while confusion matrices are the results brought by the phenomena, indicating the claim mentioned above.
In Figure 7(a), different classes (points with different colors) have different semantic ranges, which demonstrates
that semantic inconsistency exists in reality and we should use semantic-aware Mixup. From Figure 7(b), we
can see that data points in color 17 with shape + are close to data points in color 9, which proves the existenceof the second issue. And from the right figure, we can see our method mitigates this issue. Figure 7 describes(a) Semantic Range(b) Discrimination

Fig. 7. Visualization of the t-SNE embeddings of DSADS dataset for target 3 in the Cross-Person setting. Each class is denotedby a color and each shape corresponds to a domain. Best viewed in color and zoom in.

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

65:16•Lu et al.

the phenomena and the improvements on the fourth task of DSADS in Cross-person setting also suggests thatpaying attention to semantic range and discrimination can bring better performance.5

5.5

Ablation Study

We perform ablation study in Figure 8. Figure 8(a) shows Mixup has a better performance than DeepAll while our
Semantic-aware Mixup further improves accuracy on SHAR in the Cross-Person setting. Moreover, with large
margin loss to enhance discrimination, there exists another improvement compared with semantic-aware Mixup,which demonstrates that our method achieves the best average accuracy and the two components of our SDMix
are both effective. Results of ablation study on DSADS, PAMAP2, and USC-HAD in the Cross-Person setting
and results in the Cross-Position setting (Figure 8(b), Figure 8(c), Figure 8(d), and Figure 8(f) respectively) all
demonstrate that increasing diversity of data can bring better performance and two components of our SDMix
are both effective. In Figure 8(e), Vanilla Mixup performs worse than DeepAll while ours achieve better results,
which demonstrates that it is better to pay attention to semantic inconsistency and discrimination slackness
when performing Mixup.

Mixup+ Sem. +Sem.+E.D.

MethodDeepAll(a) SHAR

Mixup+ Sem. +Sem.+E.D.

Method(b) DSADS

52
50
48
46
44
42
40
38
36 DeepAll

Mixup+ Sem. +Sem.+E.D.

Method(d) USC-HAD

Mixup+ Sem. +Sem.+E.D.

Method(c) PAMAP2

38
36

Accuracy (%)

Accuracy (%)

Accuracy (%)

76
75
74
73
72
71
70
69
68 DeepAll

83
82
81
80
79
78
77
76 DeepAll

Accuracy (%)

92
90
88
86
84
82
80

Accuracy (%)

Accuracy (%)

63
62
61
60
59
58
57
56
55 DeepAll

34
32
30

Mixup

+ Sem. +Sem.+E.D.

Method(e) Cross-Dataset

28 DeepAll

Mixup+ Sem. +Sem.+E.D.

Method(f) Cross-Position

Fig. 8. Ablation study. Fig. 8(a)-8(d) are in the Cross-Person setting. Fig. 8(e) is in the Cross-Dataset setting while Fig. 8(f) isin the Cross-Position setting. ‘Sem.’ and ‘E.D.’ denotes semantic and enhancing discrimination, respectively.

5.6

Parameter Sensitivity Analysis

We evaluate the parameter sensitivity of SDMix in Figure 9. There are mainly four hyperparameters in our
method: the Beta distribution in Mixup (𝛼), the aggregation class number in Eq. (10) (top 𝑐), the required distance
5 In contrast, paying no attention to semantic range and discrimination may suffer from some problems and have worse performance.

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition

•

65:17

to boundaries in Eq. (10) (𝛾), and alternatives for computing activity semantic range in Eq. (5a) or Eq. (5b). From
Figure 9(a)-9(c), we can see that the results with parameters around the highest points are all better than DeepAll.
Figure 9(d) demonstrates that different approaches of computing activity semantic ranges obtain different results
and we should choose the right one for better results through validation. Please note that our method achieves
the best performance no matter which approach of computing is selected. In a nutshell, It demonstrates that
SDMix is effective and robust that can be easily applied to real applications.

0.1

mixup:

(a) 𝛼

1

10

1

2

margin: top c

5(b) Top c

avg-cos
avg-1-norm
max-2-norm

95

accuracy (%)

Ours
DeepAll
0.2
0.5

90
85
80
75
70
65
60
55
50 10

Ours
DeepAll

Accuracy (%)

90.0
87.5
85.0
82.5
80.0
77.5
75.0
72.5

accuracy (%)

accuracy (%)

90
80
70
60
50
40
30

90

100

Ours
DeepAll
10000 100000

margin:
(c) 𝛾

85
80

0

1

2

Task

3

avg

(d) Alternatives to compute 𝑅𝑐

Fig. 9. Parameter sensitivity analysis (DSADS in the Cross-Person setting).

5.7

Extensibility

To demonstrate that our method is still a remark method in the common setting where training data and testingdata share the same distribution, we conduct experiments on DSADS. We split DSADS into two parts, 20% for
testing and the rest for training. When utilizing all rest data for training, the baseline accuracy is almost 100%.
100

Accuracy (%)

98

DeepAll
Mixup
SDMix

96
94
92
90

10%

25%

50%

Training data proportion

100%

Fig. 10. Results on DSADS when training and testing data have the same distribution.

To better show performance differences between our method and other state-of-the-art methods, we utilize
10%, 25%, 50%, 100% of rest data for training. To select the best model during training, 20% of data serve as
validation data while the rest data serve as the real training data. Therefore, when utilizing 10% of rest data
for training, only 6.4% of all data are for training. The results are shown in Figure 10 where we can see that
our method achieves the best performance compared to ERM and Mixup. Our method has improvements with
4.88%, 2.36%, 1.7%, 0.17% compared to ERM while it has improvements with 3.17%, 0.66%, 0.72%, 0.22% compared toMixup using 10%, 25%, 50%, 100% training data respectively. As we can see, all methods achieve better performancewith more training data used and our method achieves the best performance no matter how many training data
are used. The results demonstrate that our method is still a remark method in a common setting.
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

65:186•Lu et al.

CONCLUSION

In this paper, we proposed SDMix for generalizable sensor-based human activity recognition. SDMix solves two
critical challenges: semantic inconsistency and discriminative slackness. Specifically, SDMix took the activity
semantic range into consideration to learn more flexible interpolations. Moreover, SDMix introduced the large
margin loss to Mixup that enhances the discrimination. Extensive experiments on cross-person, cross-dataset,and cross-position scenarios demonstrated the effectiveness of our method. In the future, we plan to extend
SDMix to more challenging HAR settings including cross-category and incremental learning.

ACKNOWLEDGMENTS
This work is supported by the National Key Research and Development Plan of China No.2021YFC2501202,
Natural Science Foundation of China (No. 61972383, No. 61902377, No. 61902379, No.62002187), Beijing Municipal
Science & Technology Commission No.Z211100002121171.

REFERENCES[1] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge L Reyes-Ortiz. 2012. Human activity recognition on smartphones
using a multiclass hardware-friendly support vector machine. In International workshop on ambient assisted living. Springer, 216–223.
[2] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. 2018. Metareg: Towards domain generalization using meta-regularization.
Advances in neural information processing systems 31 (2018).
[3] Billur Barshan and Murat Cihan Yüksek. 2014. Recognizing daily and sports activities in two open source machine learning environments
using body-worn sensor units. Comput. J. 57, 11 (2014), 1649–1667.
[4] Andreas Bulling, Ulf Blanke, and Bernt Schiele. 2014. A tutorial on human activity recognition using body-worn inertial sensors. ACM
Computing Surveys (CSUR) 46, 3 (2014), 1–33.
[5] Youngjae Chang, Akhil Mathur, Anton Isopoussu, Junehwa Song, and Fahim Kawsar. 2020. A systematic study of unsupervised domain
adaptation for robust human-activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4,
1 (2020), 1–30.
[6] Diane Cook, Kyle D Feuz, and Narayanan C Krishnan. 2013. Transfer learning for activity recognition: A survey. Knowledge andinformation systems 36, 3 (2013), 537–556.
[7] Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. 2018. Large Margin Deep Networks forClassification. In Advances in Neural Information Processing Systems, Vol. 31. 842–852.
[8] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor
Lempitsky. 2016. Domain-adversarial training of neural networks. The journal of machine learning research 17, 1 (2016), 2096–2030.
[9] Vikas Garg, Adam Tauman Kalai, Katrina Ligett, and Steven Wu. 2021. Learn to expect the unexpected: Probably approximately correct
domain generalization. In International Conference on Artificial Intelligence and Statistics. PMLR, 3574–3582.
[10] Negar Golestani and Mahta Moghaddam. 2020. Human activity recognition using magnetic induction-based motion signals and deep
recurrent neural networks. Nature communications 11, 1 (2020), 1–11.
[11] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. 2020. Self-challenging improves cross-domain generalization. In Computer
Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer, 124–140.
[12] Maximilian Ilse, Jakub M Tomczak, Christos Louizos, and Max Welling. 2020. Diva: Domain invariant variational autoencoders. In
Medical Imaging with Deep Learning. PMLR, 322–348.
[13] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. 2018. Deep domain generalization via
conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV). 624–639.
[14] Alexander H Liu, Yen-Cheng Liu, Yu-Ying Yeh, and Yu-Chiang Frank Wang. 2018. A unified feature disentangler for multi-domain imagetranslation and manipulation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2595–2604.
[15] Wang Lu, Yiqiang Chen, Jindong Wang, and Xin Qin. 2021. Cross-domain activity recognition via substructural optimal transport.
Neurocomputing 454 (2021), 65–75.
[16] Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. 2018. Best sources forward: domain generalization throughsource-specific nets. In 2018 25th IEEE international conference on image processing (ICIP). IEEE, 1353–1357.
[17] Daniela Micucci, Marco Mobilio, and Paolo Napoletano. 2017. Unimib shar: A dataset for human activity recognition using acceleration
data from smartphones. Applied Sciences 7, 10 (2017), 1101.
[18] Narges Honarvar Nazari and Adriana Kovashka. 2020. Domain generalization using shape representation. In European Conference on
Computer Vision. Springer, 666–670.

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.

Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition

•

65:19

[19] Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22, 10 (2009),
1345–1359.
[20] Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. 2021. Learning explanations
that are hard to vary. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
OpenReview.net. https://openreview.net/forum?id=hb1sDDSLbV[21] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia
Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural
information processing systems, Vol. 32. 8026–8037.
[22] Hangwei Qian, Sinno Jialin Pan, and Chunyan Miao. 2021. Latent Independent Excitation for Generalizable Sensor-based Cross-Person
Activity Recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 11921–11929.
[23] Fengchun Qiao, Long Zhao, and Xi Peng. 2020. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition. 12556–12565.
[24] Xin Qin, Yiqiang Chen, Jindong Wang, and Chaohui Yu. 2019. Cross-dataset activity recognition via adaptive spatial-temporal transfer
learning. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4 (2019), 1–25.
[25] Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. 2020. Correlation-aware adversarial
domain adaptation and generalization. Pattern Recognition 100 (2020), 107124.
[26] Attila Reiss and Didier Stricker. 2012. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international
symposium on wearable computers. IEEE, 108–109.
[27] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. Distributionally Robust Neural Networks. In 8thInternational Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=ryxGuJrFvS[28] Yuge Shi, Jeffrey Seely, Philip H. S. Torr, N. Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. 2021. Gradient Matchingfor Domain Generalization. CoRR abs/2104.09937 (2021). arXiv:2104.09937 https://arxiv.org/abs/2104.09937[29] Baochen Sun and Kate Saenko. 2016. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer
vision. Springer, 443–450.
[30] Bailin Wang, Mirella Lapata, and Ivan Titov. 2021. Meta-learning for domain generalization in semantic parsing. In NAACL.
[31] Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. 2019. Deep learning for sensor-based activity recognition: A
survey. Pattern Recognition Letters 119 (2019), 3–11.
[32] Jindong Wang, Yiqiang Chen, Lisha Hu, Xiaohui Peng, and S Yu Philip. 2018. Stratified transfer learning for cross-domain activity
recognition. In 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 1–10.
[33] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin. 2021. Generalizing to Unseen Domains: A Survey on Domain
Generalization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. 4627–4635. Survey Track.
[34] Jindong Wang, Vincent W Zheng, Yiqiang Chen, and Meiyu Huang. 2018. Deep transfer learning for cross-domain activity recognition.
In proceedings of the 3rd International Conference on Crowd Science and Engineering. 1–8.
[35] Yufei Wang, Haoliang Li, and Alex C Kot. 2020. Heterogeneous domain generalization via domain mixup. In ICASSP 2020-2020 IEEE
International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3622–3626.
[36] Garrett Wilson and Diane J Cook. 2020. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems andTechnology (TIST) 11, 5 (2020), 1–46.
[37] Tzu-Tsung Wong. 2015. Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation. Pattern
Recognition 48, 9 (2015), 2839–2846.
[38] Yuan Wu, Diana Inkpen, and Ahmed El-Roby. 2020. Dual mixup regularized learning for adversarial domain adaptation. In European
Conference on Computer Vision. Springer, 540–555.
[39] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. 2020. Adversarial domain adaptation withdomain mixup. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 6502–6509.
[40] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. 2021. A Fourier-based Framework for Domain Generalization. In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14383–14392.
[41] Zheng Xu, Wen Li, Li Niu, and Dong Xu. 2014. Exploiting low-rank structure from latent domains for domain generalization. In European
Conference on Computer Vision. Springer, 628–643.
[42] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond Empirical Risk Minimization. In
6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track
Proceedings. OpenReview.net. https://openreview.net/forum?id=r1Ddp1-Rb[43] Mi Zhang and Alexander A Sawchuk. 2012. USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable
sensors. In Proceedings of the 2012 ACM conference on ubiquitous computing. 1036–1043.

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 2, Article 65. Publication date: June 2022.