Domain Generalization for Activity Recognition via
Adaptive Feature Fusion

arXiv:2207.11221v1 [cs.CV] 21 Jul 2022

XIN QIN, Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. of Computing Tech., CAS,
University of Chinese Academy of Sciences, China

JINDONG WANG∗ , Microsoft Research Asia, China
YIQIANG CHEN∗ , Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. of Computing Tech.,
CAS, University of Chinese Academy of Sciences, Pengcheng Laboratory, Shenzhen, China

WANG LU, Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. of Computing Tech., CAS,
University of Chinese Academy of Sciences, China

XINLONG JIANG, Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. of Computing Tech.,
CAS, China
Human activity recognition requires the efforts to build a generalizable model using the training datasetswith the hope to achieve good performance in test datasets. However, in real applications, the training andtesting datasets may have totally different distributions due to various reasons such as different body shapes,
acting styles, and habits, damaging the model’s generalization performance. While such a distribution gap
can be reduced by existing domain adaptation approaches, they typically assume that the test data canbe accessed in the training stage, which is not realistic. In this paper, we consider a more practical andchallenging scenario: domain-generalized activity recognition (DGAR) where the test dataset cannot be
accessed during training. To this end, we propose Adaptive Feature Fusion for Activity Recognition (AFFAR), a
domain generalization approach that learns to fuse the domain-invariant and domain-specific representations
to improve the model’s generalization performance. AFFAR takes the best of both worlds where domaininvariant representations enhance the transferability across domains and domain-specific representations
leverage the model discrimination power from each domain. Extensive experiments on three public HAR
datasets show its effectiveness. Furthermore, we apply AFFAR to a real application, i.e., the diagnosis ofChildren’s Attention Deficit Hyperactivity Disorder (ADHD), which also demonstrates the superiority of ourapproach.
CCS Concepts: • Human-centered computing → Ubiquitous computing; • Computing methodologies→ Transfer learning.
Additional Key Words and Phrases: Human Activity Recognition, Domain Generalization, Transfer Learning∗ Correspondence to: J. Wang and Y. Chen.

Authors’ addresses: Xin Qin, qinxin18b@ict.ac.cn, Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. ofComputing Tech., CAS, University of Chinese Academy of Sciences, Beijing, China; Jindong Wang, jindong.wang@microsoft.
com, Microsoft Research Asia, Beijing, China; Yiqiang Chen, yqchen@ict.ac.cn, Beijing Key Lab. of Mobile Computing andPervasive Devices, Inst. of Computing Tech., CAS, University of Chinese Academy of Sciences, Pengcheng Laboratory,Shenzhen, China; Wang Lu, luwang@ict.ac.cn, Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. ofComputing Tech., CAS, University of Chinese Academy of Sciences, Beijing, China; Xinlong Jiang, jiangxinlong@ict.ac.cn,
Beijing Key Lab. of Mobile Computing and Pervasive Devices, Inst. of Computing Tech., CAS, Beijing, China.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice andthe full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee. Request permissions from permissions@acm.org.
© 2021 Association for Computing Machinery.
2157-6904/2021/5-ART000 $15.00
https://doi.org/00.0000/0000000.0000000

ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

000:2

Qin, et al.

ACM Reference Format:
Xin Qin, Jindong Wang, Yiqiang Chen, Wang Lu, and Xinlong Jiang. 2021. Domain Generalization for Activity
Recognition via Adaptive Feature Fusion. ACM Trans. Intell. Syst. Technol. 0, 0, Article 000 (May 2021), 21 pages.
https://doi.org/00.0000/0000000.0000000

1

INTRODUCTION

Human activity recognition (HAR) is an active research topic in ubiquitous computing. HAR aims
at recognizing people’s activities by building machine learning models on the activity data. HAR
has been wildly applied in smart-home [15], fatigue detection [47], fall detection for the elder [32],attention deficit hyperactivity disorder (ADHD) [11], and other fields. Therefore, accurate HAR isof vital importance in real-world applications. Many machine learning methods have been adopted
to improve the performance of HAR, such as Support Vector Machines (SVM), K-Nearest Neighbor(KNN), Random Forest (RF), and the deep learning models including Convolutional Neural Networks(CNN) [25], Long-short Term Memory (LSTM) [21] and others [53].
Despite the great success in the past, one critical challenge is the generalization ability of the
HAR models, i.e., the performance of applying the trained models to a new, unseen dataset. In
real applications, the sensor signals are easily influenced by the diverse personalities of end-users
such as acting styles, habits, or different body shapes. When testing on a new end-user whose
activity data are never seen in the training set, the performance of the model is likely to drop. For
instance, Figure 1 shows the sensor readings of two users from DSADS dataset [3] collected using
the same device, where the distributions of sensor readings are different, i.e., 𝑃 (D 1 ) ≠ 𝑃 (D 2 ).
When deployed to an unseen test user D 𝑡𝑒 whose distribution is different from the training set, theperformance of the activity recognition model will be likely to drop. This is due to the domain shift
caused by the non-IID (independently and identically distributed) distributions between training
and testing datasets [48].
Train on users 𝒟1 and 𝒟 2

Test on user 𝒟 𝑡𝑒

Problem setting

1. 𝑃(𝒟1 ) ≠ 𝑃 𝒟 2≠ 𝑃(𝒟 𝑡𝑒 )2. 𝒟 𝑡𝑒 is unseen in training.
Test
acc.

𝒟1 𝒟 2 𝒟 𝑡𝑒

Fig. 1. The distributions of accelerometer readings vary among different users. When the test dataset has
different distributions and cannot be accessed during training, the performance of existing methods will drop
while our method can reach better performance.

Such a domain shift issue can be mitigated by transfer learning and domain adaptation techniques [10, 36], which have been applied to HAR over the years. Transfer learning first pre-trainsa model from the source dataset and then fine-tunes it on the new test data. Domain adaptation performs instance reweighting or feature transformation between the training and testingdatasets to learn domain-invariant representations where their distribution gap can be minimized [7, 10, 27, 39, 43, 54]. Unfortunately, both transfer learning and domain adaptation require
access to target domain training data, which is often not realistic in real applications as we aim toachieve “Train once, deploy everywhere”. In real applications, it is hard to obtain the test data [55].
For example, it is impractical to collect each patient’s data for medical healthcare, and to collectdata on a variety of fall poses for fall detection in advance.
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

Domain Generalization for Activity Recognition via Adaptive Feature Fusion

000:3

Domain generalization (DG) [55] is an emerging research topic in recent years. DG focuses
on utilizing the knowledge from several different domains to build a model that can generalize
well to unknown domains. Many works have been done and make a good performance in thecomputer vision field. Unfortunately, we cannot directly apply existing DG methods to our problem
due to the characteristics of wearable-sensor-based activity data. To the best of our knowledge,
few existing works focus on the domain generalization problems in HAR and we refer to this asdomain-generalized HAR (DGAR).
In this paper, we propose Adaptive Feature Fusion for Activity Recognition (AFFAR) to
improve the generalization ability of HAR models (refer to Figure 2). The key of AFFAR is tolearn both domain-specific representation and domain-invariant representation and fuse them
dynamically in a unified deep neural network. Specifically, domain-invariant representation learning
is to capture the general and transferable knowledge from the training domains, and domainspecific representation learning is to learn the specific characteristics of each domain to preserve
the diversity of features to improve generalization ability. Our key assumption is that although
we cannot get access to the test data and the sensor readings from different persons are different,
they still share some similarities that can be utilized to learn domain-invariant representations.
Therefore, we can learn transferable knowledge while preserving their diversities for generalization.
Our method can be optimized in an end-to-end neural network. We show the superiority of AFFAR
by experimenting on both public HAR datasets and a real application to the diagnosis of attention
deficit hyperactivity disorder (ADHD). Experiments demonstrate that our method significantly
outperforms the comparison methods.
The main contributions of this paper are four-fold:(1) We propose and study a more practical and challenging problem scenario: domain-generalized
activity recognition (DGAR), for robust and generalized activity recognition. We thoroughly
analyze the reason for this problem, indicating a new research direction.
(2) To solve the DGAR problem, we propose a novel algorithm: Adaptive Feature Fusion for
Activity Recognition (AFFAR), to learn both domain-invariant and domain-specific deep
representations to enhance the generalization capability of the model to unseen datasets.
(3) We evaluate AFFAR on three public HAR datasets. Experiments on cross-person activity recognition demonstrate that the proposed AFFAR can significantly outperform the comparison
methods.
(4) Finally, we apply our AFFAR algorithm to a real-world ADHD problem where it also achieves
the best performance.
2 RELATED WORK
2.1 Human Activity Recognition
Human activity recognition (HAR) [40] aims at recognizing the activities of people by training
machine learning models on the data collected during performing some specific activities. In
HAR, the wearable sensor-based human activity recognition has occupied an important position
as it is superior in pervasiveness, computational consumption, and privacy preservation withwearable sensors as interface [9]. So in this paper, we mainly focus on wearable sensor-based
activity recognition. Over these years, many efforts have been done to achieve accurate and robust
activity recognition including traditional machine learning methods[28] combined with feature
extraction, and deep learning methods [8, 53]. However, the conventional methods usually focus
on the i.i.d. data situation, i.e. the training data set and testing data set follow the same distribution,
this may result in performance degradation when faced with various tasks in real-life applications.
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

000:4

2.2

Qin, et al.

Transfer Learning and Domain Adaptation

In sensor-based HAR, domain shift is a common and must be solved problem due to the variation ofdevices, locations, personalities, and so on. During the data collection, any change in these factors
may result in distribution divergence. Large distribution divergence results in the performancedegradation of the trained model. Transfer learning, as a representative method of machine learning,
is an effective paradigm to solve the domain shift problems, which makes it possible to reuse existingknowledge. The purpose of transfer learning is to apply the knowledge learned in the existingdomains to related but different accessible domains during the training process, to improve theability to solve new tasks [36, 57].
The commonly used approach of transfer learning is pre-training and fine-tuning, i.e., pretraining on the source dataset to get a pre-trained model, and then fine-tuning this model on
the new target dataset. Such a simple and effective approach has been successfully applied tocomputer vision [46, 60], natural language processing [12], and speech recognition [24, 49]. In
order to tackle the domain shift challenge, transfer learning has become an effective method andapplied in HAR [7, 10, 27, 39, 43]. Ma M et.al [33] proposed a twin stream network architecture
and jointly fine-tuned the two networks to recognize objects, actions, and activities. Wang H Bet.al [52] verified that fine-tuning and regular constraints can increase the training efficiency, andfine-tuning is valid in practical HAR applications.
In the scope of transfer learning, domain adaptation (DA) is a major technique and has attracted
much attention from researchers in recent years. Domain adaptation aims at improving the performance on the given less annotated or no annotated target domain by exploiting the source
domains. Domain adaptation is a popular topic and has been applied in HAR to solve the domain
shift problem [7]. Khan et al. [27] proposed a feature-based model HDCNN, which adapts thesource and target features after every convolutional and fully connected layer. Chang et al. [7]made a comparison of adaptation techniques to give a guideline on applying unsupervised domain
adaptation algorithms to cross-position HAR problems.
However, both transfer learning and domain adaptation assume the target domain can beaccessed in the model training process. While in real HAR applications, the target tasks are noveland various and can not be accessed during the training. This requires the trained model has strong
generalization capability so that it can perform well in unknown fields, which goes beyond thescope of conventional transfer learning and domain adaptation problem settings. Specifically, themulti-source domain adaptation (MSDA) also has several source domains for training, but its goal
is to adapt the trained model to the target domain, which is accessible during training.
2.3

Domain Generalization

Domain generalization (DG) is an emerging topic and is attracting increasing attention in recent
years. The goal of DG is to learn a robust and well-generalized prediction function on several given
source domains to obtain the minimum prediction error on any possible unknown domain [55].
The most striking difference between DA and DG is that whether the target domain can beaccessed during the training, the target data can only be used for the model test in DG. Thus,
DG is more suitable for the situation of various unknown target HAR tasks in real life. DG has
been widely applied in the computer vision field and many DG methods have been proposed andevaluated on image datasets. The DG methods can be mainly divided into three branches, i.e. data
manipulation [38, 45], which works on the input data to assist the general representation learning;
representation learning [17, 18, 30], which aims at learning domain-invariant representation ordisentangling the features; and learning strategy [5, 23, 29, 42], which exploits the general learning
strategy such as meta-learning and ensemble learning for generalization. Yan et al. [58] solved the
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

Domain Generalization for Activity Recognition via Adaptive Feature Fusion

000:5

DG problem from the perspective of data generation by linear interpolation between instances andtheir labels. However, unlike image data, it is hard to intuitively assess the semantics and diversityof sensor data. Although some works assess sensor data by training a post-hoc classification or
prediction model or using other techniques, these assessments are still less intuitive and explainable
to some extent [31, 59]. Data generation methods may be not straightly applicable to sensor-based
HAR since there is still a lack of the intuitive quantitative assessment of quality for generated sensor
data. Li et al. [29] proposed a model agnostic training procedure by leveraging the meta-learning for
DG, i.e. MLDG, while this kind of method may not straightly applicable due to the high dependenceof sensor data. Many methods follow distribution alignment in DA to minimize the distribution
discrepancy between domains by adversarial training [19], Wasserstein distance [62] etc. to learn
domain-invariant representations, and the spirit is followed in DG. In the other aspect, someensemble-learning-based methods focus on the domain-specific, such as domain-specific neuralnetworks [34, 56], domain-specific batch normalization [44], weight averaging [6] etc., more details
can be found in [55]. However, few works both consider domain-invariant and domain-specificrepresentations, especially for HAR applications.
Last, it is worth noting that domain generalization is not Leave-One-Out-Cross-Validation(LOOCV) in traditional machine learning. For domain generalization, we sequentially leave one
domain for the final test to construct several domain generalization tasks, and these tasks are
independent of each other. LOOCV is mainly used for selecting models where all domains are usedfor validation in turn and it also has an inaccessible test dataset for the final testing. Therefore,
LOOCV can be seen as a typical model selection method that can also be used in DG.
3

OUR METHOD: ADAPTIVE FEATURE FUSION

In this section, we present our Adaptive Feature Fusion method for activity recognition in detail.
3.1

Problem Definition: Domain-generalized Activity Recognition

In a typical human activity recognition (HAR) problem, we are given a training dataset D 𝑡𝑟 ={(x𝑖 , 𝑦𝑖 )}𝑛𝑖=1 , where x ∈ R𝑑 denotes its 𝑑-dimensional features and 𝑦 ∈ {1, 2, · · · , 𝐶} denotes its
corresponding activity categories, such as walking or running. 𝑛 denotes the total number ofsamples. The goal is to build a machine learning model ℎ : x ↦→ 𝑦 such that it can accurately
recognize the activities in the training data, i.e., achieving the minimum training error:𝑛ℎ ∗ = arg minℎ1 ∑︁ℓ (ℎ(x𝑖 ), 𝑦𝑖 ),
𝑛 𝑖=1

(1)

where ℎ ∗ denotes the optimal model and ℓ (·, ·) is the loss function such as cross-entropy loss.
However, achieving the minimum error on the training data D 𝑡𝑟 does not necessarily guarantee
optimal performance when we apply the model to the unseen test data D 𝑡𝑒 . For instance, a welltrained HAR model can perform poorly when deployed to recognize different persons’ activities withdifferent body shapes or activity styles. Moreover, we can never collect all the possible training data
to build a generalized HAR model. While transfer learning and domain adaptation [10, 36, 39, 54]
are popular to perform cross-domain learning, they can not be used in our problem since theyrequire the availability of the test domain.
We aim to solve this practical and challenging problem, which we refer to as Domain-generalized
Activity Recognition, or DGAR. Here, “domain” is a general notion of “dataset”, i.e., a dataset
is a domain, or it can be split into several domains [55]. In DGAR, we assume there are several
training
i.e., there are 𝐾 different but related training domains D 𝑡𝑟 =
 1 2 (source)𝐾domains available,𝑛𝑘
D , D , · · · , D available. D 𝑘 = (x𝑘𝑖 , 𝑦𝑖𝑘 ) 𝑖=1
denotes the 𝑘𝑡ℎ training domain with 𝑛𝑘 samples.
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

000:6

Qin, et al.

Our goal is to learn a generalized model ℎ on the 𝐾 training domains such that it can achieve𝑡𝑒minimum error on the unseen test domain D 𝑡𝑒 = {(x𝑖 , 𝑦𝑖 )}𝑛𝑖=1. We often assume that all domains
share the same kinds of sensors and activities, i.e. the feature space and the label space are the same:X𝑡𝑟 = X𝑡𝑒 and Y 𝑡𝑟 = Y 𝑡𝑒 . In real applications, different domains tend to have different probability
distributions, i.e. 𝑃 𝑖 (x) ≠ 𝑃 𝑗 (x) ≠ 𝑃 𝑡𝑒 (x), 1 ≤ 𝑖 ≤ 𝑗 ≤ 𝐾. Since the probability distribution of thetest dataset is different from the training domains, DGAR is a practical setting to evaluate thegeneralization ability of activity recognition algorithms.
3.2

Motivation and Main Idea

Domain generalization (DG) [4, 55] is the general learning setting of DGAR. Over the past few years,
domain generalization has attracted the increasing attention of researchers, and a large number of
DG methods have been proposed [55]. Most of these methods are developed for general-purpose
learning and they are often evaluated on image classification tasks, with data augmentation [5],domain-invariant representation learning [35], or meta-learning methods [29]. Intuitively, it isnatural to ask: can we directly apply these existing DG methods to our DGAR problem? The shortanswer is yes but no. “Yes” means we can always do that, “no” means this ignores the characteristicsof activity recognition and we can do it better. The reasons are as follows.
First, to achieve strong generalization capability, some of them focus on manipulation of data
including data augmentation and data generation which are specific to the image domain [5].
However, such methods may not be straightly applicable to sensor-based HAR due to the lack of aquantitative assessment of quality for generated sensor data. Second, some focus on representation
learning, which aims to learn a better feature representation for better generalization. However,activity recognition is a special area where we should not only care about domain-invariant features
but also domain-specific features to capture the individually-specific features from each domain,which can preserve the diversity of different persons. Third, although meta-learning-based DG
methods could be used for our problem, our empirical experiments (ref. Table 2) indicate that its
performance is even worse than the empirical risk minimization method. The reason may be thesecond-order gradient optimization can have mode collapse for the special activity data. To sum
up, we need to develop special algorithms for this DGAR problem.
In this paper, we proposed a novel Adaptive Feature Fusion method for domain-generalized
Activity Recognition, abbreviated as AFFAR. Our key assumption is that although we cannot get
access to the test data in training, we can still learn to represent the test data using the aggregationof existing training data. This is reasonable since different persons may generate different sensor
readings while performing the same activities, but they could share some similarities such as body
shapes and activity styles. Thus, we can learn to represent each data as the weighted aggregation ofexisting training domains. Meanwhile, since the activity data from different domains have different
probability distributions, we also need to learn domain-invariant features to regularize the model
to facilitate knowledge transfer.
The core of AFFAR is to learn both domain-invariant and domain-specific feature representations.
Specifically, domain-invariant representation learning is to capture the general and transferable
knowledge from the training domains, and domain-specific representation learning is to learn thespecific characteristics of each source domain to enhance the generalization ability. We depict theoverall learning process of AFFAR in Figure 2.
AFFAR conceptually consists of four modules: feature extraction module (purple blocks), activity
classification module (blue blocks), domain-specific representation learning module (yellow blocks),and domain-invariant learning module (green blocks). The feature extraction module is used to
extract features from the raw sensor data. In this paper, we leverage two convolutional neuralnetwork (CNN) layers along with max-pooling operations to extract features. These layers are
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

Domain Generalization for Activity Recognition via Adaptive Feature Fusionℒ = ℒ 𝑐𝑐𝑐𝑐𝑐𝑐 + 𝜆𝜆ℒ 𝑑𝑑𝑑𝑑𝑑𝑑 + 𝛽𝛽ℒ 𝑑𝑑𝑖𝑖𝑖𝑖

Domain 1

𝑓𝑓1

𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑(𝒟𝒟 𝑖𝑖 , 𝒟𝒟 𝑗𝑗 )

𝑓𝑓2

�

…

…

Domain 2
Input
Domain 𝐾𝐾

000:7

Conv.

Max
poolingConv.

Max
pooling

𝑓𝑓𝐾𝐾

Feature extractor

Domain-specific feature

Activity classification

Domain-invariant feature𝑤𝑤ℒ𝑑𝑑𝑑𝑑𝑑𝑑
Walking
Running
Upstairs
Sitting

𝑓𝑓𝑐𝑐
𝒟𝒟1
𝒟𝒟 2
…

𝑓𝑓𝑑𝑑

𝒟𝒟 𝐾𝐾

ℒ𝑐𝑐𝑐𝑐𝑐𝑐ℒ𝑑𝑑𝑑𝑑𝑑𝑑

Fig. 2. Illustration of the AFFAR framework.

shared by all training domains to reduce parameter amounts. The domain-specific representation
learning module is used to learn domain-specific features, thus this is not shared, but specific foreach domain. We implement it by adding 𝐾 fully connected (FC) layers for each domain after feature
extraction. To aggregate the specific information of each domain, we design a weighting function.
The domain-invariant representation learning module is used to reduce the distribution discrepancy
between each domain D 𝑖 and domain D 𝑗 to learn domain-invariant features. Finally, the activity
classification module is an FC layer. Since the classification uses the fusion of domain-specific anddomain-invariant features, we call our method adaptive feature fusion.
The learning objective of AFFAR can be formulated as:
L = L𝑐𝑙𝑠 + 𝜆L𝑑𝑠𝑟 + 𝛽L𝑑𝑖𝑟 ,(2)where L𝑐𝑙𝑠 is the classification loss, L𝑑𝑠𝑟 is the loss of domain-specific representation learning, L𝑑𝑖𝑟is the loss of domain-invariant representation learning. 𝜆 and 𝛽 are the tradeoff hyperparameters.
For classification, we take cross entropy as the classification loss:
𝑁

L𝑐𝑙𝑠 = −1 ∑︁𝑦𝑖 log 𝑃 (𝑦𝑖 | 𝑥𝑖 ) ,
𝑁 𝑖=1

(3)

Í𝐾
where 𝑁 = 𝑘=1
𝑛𝑘 is the amount of the samples from all the training domains.
In the next sections, we will elaborate on the details of domain-specific and domain-invariant
representation learning modules.
3.3

Domain-specific Representation Learning

The domain-specific representation learning module aims to learn domain-specific features andthen aggregate them for unified feature representations by fusing the features from multiple sources
for the unseen target feature. More formally, given a new test data x, its feature z is formulated as:
z=

𝐾
∑︁
𝑘=1

𝑤𝑘 𝑓𝑘 (𝑓𝑒 (x)), where 𝑤𝑘 > 0 and𝐾∑︁

𝑤𝑘 = 1,(4)

𝑘=1where 𝑤𝑘 is the weight on domain D 𝑘 , indicating the similarity between the data on domain D 𝑘
and the true data. 𝑓𝑘 is the feature learning function of domain D 𝑘 and 𝑓𝑒 is the shared feature
extraction function, i.e., CNNs.
This process can be viewed as a certain type of ensemble learning [13], where each specific baselearner is trained to ensemble a stable model which can perform well in all aspects. In our problem,the base learner is composed of two parts: the shared CNN feature extractor 𝑓𝑒 and the specific
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

000:8

Qin, et al.

feature extraction layer 𝑓𝑘 for each domain D 𝑘 . This split is motivated by existing research on thetransfer learning ability of deep networks [60] that the lower layers tend to learn low-level andgeneral features while the higher layers tend to learn domain-specific features.
To learn the weights 𝑤𝑘 for each domain D 𝑘 , we build a domain classifier 𝑓𝑑 : x ↦→ R𝐾 thattakes as input the features after CNN layers, and then use softmax to satisfy Equation 4. At training
time, the domain label 𝑑𝑘 ≡ 𝑘 for each sample is known a priori, thus, the domain-specific loss foreach domain 𝑘 can be computed as:
𝑛𝑘
1 ∑︁
𝑘
L𝑑𝑠𝑟 =ℓ (𝑓𝑑 (𝑓𝑒 (x𝑖 )), 𝑑𝑘 ).
𝑛𝑘 𝑖=1(5)Then, we can get the total domain-specific loss by averaging the losses on all domains asÍ𝐾𝑘 .
L𝑑𝑠𝑟 = 𝐾1 𝑘=1
L𝑑𝑠𝑟
When testing on the target data, we cannot access the domain label of the unseen target data.
After the network extracts the features, the output of the domain weight branch is used as theweight to fuse the target features extracted by each domain-specific branch. It can be understoodthat the learned domain branch network adaptively fuses the domain-specific features extracted
from each source-specific branch to construct the feature representation of the target domain. In
this way, AFFAR can adaptively learn feature representation of any unseen domains.
3.4

Domain-Invariant Representation Learning

While domain-specific feature learning encourages the model to learn specific information for eachdomain, the feature distribution gap could also be enlarged due to their diverse representations.
Thus, to enhance the generalization capability, we further design a domain-invariant representation
learning module to seek balance with domain-specific representation learning.
Recall that features in the lower level of deep neural networks focus on learning common andlow-level features, while the higher layers focus more on specific tasks [60]. Thus, we make feature
adaptation to reduce the distribution discrepancy between source domains in the domain-specificlayers. In the domain adaptation field, the strategy is to reduce the distribution discrepancy between
the source domain and the target domain so that the model can be robust on the target. Underdomain generalization scenarios, we cannot access the target domain data during the training
process, so it is of vital importance to learn domain-invariant feature representation so that any
unseen target can be represented. To enable feature adaptation in DGAR, we turn to reducing the
distribution divergence between each domain pair D 𝑖 and D 𝑗 , i.e., to minimize 𝑑𝑖𝑠𝑡 (D 𝑖 , D 𝑗 ) where
𝑑𝑖𝑠𝑡 (·, ·) is a distribution distance measurement.
Specifically, we adopt the widely used distance metric Maximum Mean Discrepancy (MMD) [20]to help reduce the distribution divergence. MMD embeds distributions in Reproducing Kernel
Hilbert Space (RKHS) and calculates the distance between these embeddings as the test statistic.
Thus, it is often adopted to justify whether two distributions are the same or used to measure howsimilar two distributions are. The MMD loss between domains D 𝑖 and D 𝑗 is formulated as:
𝑖𝑗
L𝑑𝑖𝑟
=

2
1 ∑︁
1 ∑︁
𝜙 (x) −
𝜙 (x) ,
𝑛𝑖
𝑛𝑗
𝑖
𝑗
x∈ D

x∈D

(6)

H

where 𝑖 and 𝑗 are indexes of domains, 𝜙 (·) is the feature map which maps the original instances
into the RKHS H . Then, the total loss for domain-invariant learning can be computed by taking
Í𝑖𝑗
2
average on all possible domain pairs L𝑑𝑖𝑟 = 𝐾 (𝐾−1)
𝑖,𝑗 L𝑑𝑖𝑟 .
Although we adopt MMD as a metric for distribution divergence in this paper, AFFAR is a generalapproach that can embed other metrics instead of MMD.
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

Domain Generalization for Activity Recognition via Adaptive Feature Fusion

3.5

000:9

Training and Inference

As for training, after loading the training data, the feature extractor extracts the lower-level features
and then inputs them to each domain-specific branch. Each branch learns the higher-level domainspecific representations. Meanwhile, the domain classifier takes the lower-level features as input
and outputs the softmax weights to fuse the domain-specific features. At the same time, it makes
the in-between-source domain adaptation to learn domain-invariant representation. Then, it takes
the fused features as input of the final activity classifier to make activity classification.
As for inference, we fix the model parameters and learn domain-specific and domain-invariant
representations for the target. Different from the training, without prior knowledge of the domainlabel of target data, the output of the domain classifier can be regarded as the similarity between
the target and each source domain-specific branch.
The complete learning process of AFFAR is summarized in Algorithm 1.
Algorithm 1 AFFAR for domain-generalized activity recognition
Input: 𝐾 training domains D𝑠1, · · ·, D𝑠𝐾 , and 𝜆, 𝛽.
Output: Classification results on test domain.
1: Randomly initialize the model parameters 𝜃 ;2: while not converge do
3:
Sample a mini-batch B = {B 1, · · · , B 𝐾 } from 𝐾 domains;
4:
Extract the lower-level features 𝑓𝑒 (x) by the feature extractor;
5:
Extract the domain-specific features 𝑓𝑘 (𝑓𝑒 (x)) by 𝐾 domain-specific FC layers;
6:
Calculate the domain-specific loss L𝑑𝑠𝑟 and output the weight for each source branch;
7:
Calculate the domain-invariant loss L𝑑𝑖𝑟 .
8:
Fuse the domain-specific features with weight according to Eq. (4);9:Calculate the total loss of AFFAR according to Eq. (2);10:Update the model parameter 𝜃 using SGD.
11: end while
12: Make inference on the target HAR data.
13: return Classification results on target HAR data.

3.6

Discussions

The proposed AFFAR learns domain-specific and domain-invariant representations in a unified
framework to seek their balance in feature learning, which can take other distribution discrepancies
as the domain-invariant learning loss. We show two possible losses: the domain-adversarial neuralnetworks (DANN) [17] and the COReration ALignment (CORAL) loss [50]. DANN introduced anadversarial training objective where it used a min-max optimization to maximize the loss of domain
discriminator and minimize the loss of both feature extractor and classification. However, DANN
only aims at learning domain-invariant representations, which is good for domain adaptation tasks(that explains why DANN is the base model for modern domain adaptation models). DANN ignores
the specific features for each domain that is useful for domain generalization tasks, making it lessfavorable for our DGAR problem. We will empirically show this argument in later experiments (ref.
Section 4.4.2). On the other hand, AFFAR is a general framework for DGAR tasks where we canalso employ the domain discriminator loss of DANN to replace MMD:
𝑑𝑎𝑛𝑛
L𝑑𝑖𝑟
= E1≤𝑖≠𝑗 ≤𝐾 Ex𝑖 ∈ D𝑖 ,x 𝑗 ∈ D 𝑗 log[𝐷 (𝑓𝑖 (𝑓𝑒 (x𝑖 )))] + log[1 − 𝐷 (𝑓 𝑗 (𝑓𝑒 (x 𝑗 )))],(7)

ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

000:10

Qin, et al.

where E denotes expectation operation and 𝐷 is a domain discriminator (typically a two-layer
feed-forward network). Thus, DANN for domain generalization requires to build 𝐾 (𝐾−1)domain2
discriminators, which is not efficient. We can also replace MMD with CORAL loss as:
𝑐𝑜𝑟𝑎𝑙
L𝑑𝑖𝑟
=

1
||𝐶 𝑖 − 𝐶 𝑗 || 2𝐹 ,
4𝑑 2

(8)

where 𝑑 denotes the feature dimension, 𝐶 𝑖 , 𝐶 𝑗 denotes the covariance matrices for two domains,and || · || 𝐹 denotes the Frobenius norm. In later experiments (Section 4.4.2), we will show that our
AFFAR can also achieve competitive performance with these two losses.
3.7

Theoretical Insights

Finally, we show that our algorithm is theoretically-motivated using the theory proposed in [1].
Theorem 1 (Risk upper bound on unseen domain [1]). Let 𝛾 = 𝑑 H (D 𝑡𝑒 , D̄ 𝑡𝑒 ) denote the
H -divergence between target domain and its nearest neighbor in source domain convex hull, then, therisk on unseen domain D 𝑡𝑒 of hypothesis ℎ is upper-bounded by the weighted risk on source set 𝑆:
𝑅𝑡𝑒 [ℎ] ≤

𝑁𝑆
∑︁






𝜋𝑖 𝑅𝑆𝑖 [ℎ] + 𝛾 + 𝜖 + min E D̄𝑡𝑒 𝑓𝑆𝜋 − 𝑓 𝑡𝑒 , E D𝑡𝑒 𝑓 𝑡𝑒 − 𝑓𝑆𝜋 ,

(9)

𝑖=1

where 𝜖 is the largest
distribution
divergence
betweenunseen target domain and any source domainand min E D̄𝑡𝑒 𝑓𝑆𝜋 − 𝑓 𝑡𝑒 , E D𝑡𝑒 𝑓 𝑡𝑒 − 𝑓𝑆𝜋denotes the difference between labeling functions.
In our problem, the categories between training and testing are the same, the main distributiondifference between training and testing data is the activity patterns (i.e., 𝑃 (x)). So it is close to
the covariate shift assumption: the labeling function error (min{·, ·}) and 𝛾 are both relatively
small [1]. In this way, the risk on unseen domain is bounded by two terms: the weighted sourceÍ𝑁 𝑆
risk 𝑖=1
𝜋𝑖 𝑅𝑆𝑖 [ℎ] and the source-target distribution divergence 𝜖. Obviously, our domain-specificlearning module (Eq. (4)) corresponds to minimizing the weighted source risk and the domaininvariant learning module (Eq. (6)) minimizes the risk 𝜖. Thus, our algorithm can also be interpreted
from the theory. Additionally, we also provide a visualization study to help better analyze thealgorithm in Section 4.4.
4

EXPERIMENTAL EVALUATIONIn this section, we evaluate the performance of the proposed AFFAR approach via extensive
experiments on domain-generalized activity recognition.
4.1

Datasets and Preprocessing

We adopt three large public activity datasets as summarized in Table 1. In the following, we briefly
introduce their basic information, and detailed descriptions are in their original papers.
Table 1. Statistical information of three public activity recognition datasets
DatasetSubject

Activity

SampleBody Position

Sampling rate

Training : Test sizeDSADS
USC-HADPAMAP2

8
14
9

19
12
18

1.14M
2.81M
2.84M

5
1
3

25Hz
∼100Hz
100Hz

∼15:1
∼16:1
∼15:1

ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

Domain Generalization for Activity Recognition via Adaptive Feature Fusion

000:11DSADS. UCI Daily and Sports Data Set [3] collects 19 activities through 8 subjects (four males
and four females between the ages of 20 and 30) wearing body-worn sensor units including triaxial
accelerometer, triaxial gyroscope, and triaxial magnetometer on 5 body parts: torso, right arm, leftarm, right leg and left leg. To construct the domain-generalized activity recognition scenario, we
divide the 8 subjects into 4 groups where each group consists of 2 different subjects, and data fromeach group is regarded as a domain, leading to 4 domains in total.
USC-HAD. USC Human Activity Dataset [61] consists of data collected from 14 subjects (7 males
and 7 females) performing 12 activities. A motion mode is equipped at the front right hip of subjects
to capture triaxial accelerometer and triaxial gyroscope sensor readings. In order to construct a
domain-generalized activity recognition scenario, we divide the 14 subjects into 5 groups where
each of the first four groups consists of 3 different subjects, and the last group consists of two
subjects. That leads to 5 domains in total.
PAMAP2. PAMAP2 [41] consists of data collected from 9 subjects performing 18 activities.
Each subject wears 3 inertial measurement units (IMU) and a heart rate monitor. We use thedata from IMU in the experiments. Each IMU consists of two 3-axis accelerometers, one 3-axis
magnetometer, and one 3-axis gyroscope. To construct a domain-generalized activity scenario,
we choose 8 subjects (subjects IDS 1-8) and their common eight activities: lying, sitting, standing,walking, ascending stairs, descending stairs, vacuum cleaning, and ironing. Data are divided into 4
domains.
For each task in one dataset, we select one domain as the test domain while other domains serveas the training domains. We further split a validation set from the training set with a ratio of 0.2
for hyperparameter tuning. Our main focus is to test the performance on cross-person settings in adataset (i.e., different person, same sensor device).
4.2

Comparison Methods and Implementation Details

We compare AFFAR with several state-of-the-art domain generalization methods:• Empirical Risk Minimization (ERM [51], i.e., CNN baseline): minimizes the sum of errors overdata. We regard it as the naive baseline to learn a single model on all source domains.
• Meta-Learning Domain Generalization (MLDG [29]): learns how to generalize cross domains
by leveraging Model-Agnostic Meta Learning (MAML [16]).
• Domain Adversarial Neural Network (DANN [17]): employs an adversarial network thatconsists of a generator and a discriminator to adapt feature distribution. Under the domain
generalization setting, we perform DANN across multiple source domains as no target canbe accessed during the training process.
• Group Distributionally Robust Optimization (GroupDRO [42]): couples group DRO models
with increased regularization to increase the importance of the worst-group loss.
• Representation Self-Challenging (RSC [23]): discards the dominant features i.e. representations associated with the higher gradients at each epoch, and forces the model to predict
with the remaining information.
• AND-mask [37]: learning explanations that are hard to vary, which uses AND-mask to
improve the consistency in gradients for better generalization.
In addition, we compare the results of all methods with the results trained on the target domain(split the target data into train and test set with a rate around 8:2), which are ideal cases since ourproblem does not access the target domain data:• ERM-t: directly trains models on the target domain using ERM.
• Fine-tune: trains a model using ERM on the training set and then fine-tunes it on the target.
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

000:12

Qin, et al.

Experimental settings are as follows. In order to evaluate the classification performance of AFFAR,
we construct non-iid cross-person HAR tasks under the domain generalization scenario. First, we
divide the data of subjects into several groups as illustrated in Section 4.1, data from each groupis regarded as a domain. Each domain plays the role of the unseen target domain and remains assource domains. We use 2-D convolutions for our implementations with the kernel size of (1, 6)
and (1, 9), depending on different datasets.
For the comparison methods, we adopt the implementations from DomainBed [22] while wechange their network structures to be the same as ours for the comparison study. We perform
hyperparameter tuning for each comparison method to achieve its best performance on each task.
Specifically, we tune the following hyperparameters: learning rate is selected in {0.0003, .., 0.001},
batch size is set as 128. We set the total training epochs as 500 and early stop patience to 30. We run
the experiments five times and report the average results. F1 score is the main evaluation metric,
and we also analyze the accuracy, precision, recall, and ROC curves in detailed analysis.
4.3

Classification Performance

The test weighted F1 score on three datasets are shown in Table 2, 3, and 4, respectively. Fromthese results, we can make some observations that: 1) The proposed AFFAR can achieve thebest classification on almost all tasks. Concretely speaking, AFFAR significantly outperforms thesecond-best comparison methods by 2.5%, 1.7%, and 3.7% on three datasets, respectively. 2) Other
comparison methods such as AND-mask and RSC can achieve good classification on some tasks
while behaving less satisfied on others, this may be because they neglect the domain-specificknowledge, which may neglect some latent information between the distributions. 3) ERM isregarded as the naive baseline and the experimental results are worse than other methods on
several tasks, it is because it only minimizes the empirical risk on the training source data without
reducing the distribution discrepancy and investigating the latent information, the generalization
capability is less satisfied due to the large distribution discrepancy. Thus, it is necessary to explore
and utilize domain generalization approaches in real-life cross-domain HAR application tasks.
4) The performance of DANN is also not comparable to ours. It aims to learn domain-invariant
representations. However, it does not consider the domain-specific information, making it lesseffective for generalization tasks. 5) The average performance of MLDG in Table 2 is significantly
worse than ERM, indicating it is not feasible to directly apply meta-learning-based DG algorithms
to activity recognition problems. This may be because the split of meta-train and meta-test datasets
depends heavily on the independence of data, while the activity sensor data are highly dependent,
thus making the results worse. The same conclusions go for data augmentation methods whose
results are also not comparable, thus we did not list them. 6) Finally, the results of the two ideal-casemethods ERM-t and Fine-tune which are trained on the target domain are better than all DG
methods, indicating the importance of target train data. While AFFAR achieves the best results,
there is still room for improvement.
Table 2. Weighted F1 score (%) on DSADS dataset. The bold is the best result except for two ideal conditions.
Target
T-1
T-2
T-3
T-4
Average

ERM
82.58
80.04
82.87
82.82
82.10

MLDG
64.19
72.32
80.79
52.81
67.53

DANN
83.94
79.52
83.36
84.76
82.90

GroupDRO
81.58
80.75
83.24
82.69
82.07

RSC
81.16
79.86
83.53
84.03
82.15

AND-mask
81.86
80.78
83.24
80.83
81.68

AFFAR (ours) ERM-t Fine-tune
82.74
98.90
99.12
84.92
98.90
98.90
87.60
99.34
100
86.45
98.22
98.22
85.43
98.84
99.06

ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

Domain Generalization for Activity Recognition via Adaptive Feature Fusion

000:13

Table 3. Weighted F1 score (%) on USC-HAD. The bold is the best result except for two ideal conditions.
Target
T-1
T-2
T-3
T-4
T-5
Average

ERM
72.79
76.87
72.82
59.21
65.83
69.50

DANN
74.68
77.64
72.00
60.05
68.18
70.51

GroupDRO
73.32
75.82
69.27
57.67
60.68
67.35

RSC
73.75
77.12
73.70
59.66
70.17
70.88

AND-mask
72.35
75.17
70.86
58.88
67.38
68.93

AFFAR (ours) ERM-t
75.11
90.80
78.57
89.90
74.59
87.73
62.20
82.67
72.41
87.72
72.58
87.76

Fine-tune
91.22
90.99
89.74
84.72
90.33
89.40

Table 4. Weighted F1 score (%) on PAMAP2. The bold is the best result except for two ideal conditions.
ERM DANN
56.25 57.92
84.06 86.11
85.21 86.21
82.72 83.32
77.06 78.39

90

75
70

Accuracy (%)

Accuracy (%)

ALL

80

RSC AND-mask
57.63
56.74
86.58
86.54
84.85
84.93
84.12
82.28
78.29
77.62

86

cls
cls + dircls + dsr

85

65GroupDRO
57.55
86.91
85.24
84.02
78.43

AFFAR (ours)
65.37
89.35
87.43
86.46
82.15

82
78

PAMAP

DatasetUSC-HAD(a) Ablation study

ERM-t Fine-tune
94.88
95.58
93.83
94.95
94.04
94.70
93.90
95.21
94.16
95.11

Domain 1
Domain 2

Domain 3
Domain 4(USC-HAD only)

0.5

74
70

DSADS

1.0

ERM
Ours(DANN)
Ours(CORAL)
Ours(MMD)

WeightTarget
T-1
T-2
T-3
T-4
Average

DSADS

PAMAP

DatasetUSC-HAD(b) AFFAR with DANN and CORAL

0.0

DSADS

PAMAP

DatasetUSC-HAD(c) Learned weights

Fig. 3. Detailed analysis of AFFAR. (a) Ablation study to show the effectiveness of domain-invariant anddomain-specific representation learning modules. (b) Replace the MMD loss with DANN and CORAL loss. (c)Weights of a randomly selected target test sample for each dataset.

4.4

Ablation Study

4.4.1 Domain-invariant and domain-specific learning modules. AFFAR consists of two important
modules: (1) domain-invariant learning module and (2) domain-specific learning module. In thissection, we conduct an ablation study by evaluating the importance of each module. We compare
four variants of our method: (1) L𝑐𝑙𝑠 , (2) L𝑐𝑙𝑠 + L𝑑𝑖𝑟 , (3) L𝑐𝑙𝑠 + L𝑑𝑠𝑟 , and (4) the full version of
AFFAR. Figure 3(a) reports the average classification accuracy of these variants on all tasks. It canbe observed that by combining domain-invariant and domain-specific learning modules, the whole
AFFAR can achieve the best performance. It evaluates that both the modules are very importantand make contributions to the accurate classification of HAR tasks.
We also show the feature embeddings of each module of our method in Figure 4. (1) Compare
Figure 4(a) and Figure 4(b), we see that adding domain-specific learning to the model will enlarge
the domain margin and the classes are far from each other. This is because domain-specific learning
focuses on separating classes in each domain. However, the domains are not aligned well (in eachclass, different domains denoted by shapes are still far). (2) Compare Figure 4(a) with Figure 4(c),we see that domains (denoted by shapes) are more invariant in each class since domain-invariant
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

000:14

Qin, et al.

60

60

0
1
2
3
4
8

0
1
2
3
4
8

-60-6060-60-60(a) Only L𝑐𝑙𝑠

60

60

0
1
2
3
4
8

-60-60(c) Adding domain-invariant learning

60(b) Adding domain-specific learning

0
1
2
3
4
8

60

-60-6060(d) Domain-specific + domain-invariant learning

Fig. 4. Visualization of the t-SNE embeddings of USC-HAD dataset. Each class is denoted by color and eachdomain is denoted by a shape. The classes denoted by numbers are walking forward, walking left, walking
right, walking upstairs, walking downstairs, and standing. Best viewed in color and zoom in.

learning focuses on learning general features that can transfer across domains. But the classification
is worse than the whole version (class margins are small, making it easy to misclassify samples). (3)Finally, we see from Figure 4(d) that adding two modules can not only make the domains moreinvariant but also enhance the classification results.
4.4.2 Extending domain-invariant learning with other distances. In Section 3.6, we show that our
AFFAR can also take other distribution matching techniques such as domain-adversarial learning(DANN, Eq. (7)) and CORAL loss (Eq. (8)). In this section, we replace the original MMD measurewith the DANN and CORAL loss to evaluate the performance of AFFAR.
We thoroughly test the performance in three different datasets and record their average accuracy
in Figure 3(b). It shows that our method is general and flexible that can use other distribution
matching metrics to achieve competitive performance, which is better than the original ERM.
We also observe that AFFAR with MMD loss gives the best performance. On the other hand,comparing their computational complexity (O𝑀𝑀𝐷 ≈ O𝐶𝑂𝑅𝐴𝐿 < O𝐷𝐴𝑁 𝑁 ), we use MMD as our
main distribution matching loss.
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

Domain Generalization for Activity Recognition via Adaptive Feature Fusion

000:15(a) Target-1(b) Target-2(c) Target-3(d) Target-4

Fig. 5. Micro-average ROC curves of AFFAR and other comparison methods for each task on PAMAP2.

4.4.3 Analysis of domain-specific module. In this section, we analyze the domain-specific module
by investigating the learned domain weights to the target domain. The weights can act as thesimilarity between the training domains and the test data, representing how much information canbe transferred from these domains. Figure 3(c) shows the (normalized) weights given to a target test
sample for each dataset. Note that only the USC-HAD dataset has four training domains and theother two have three training domains. The weights reflect the similarity between the target dataset
and each training domain. Thus, it shows that our AFFAR can effectively learn such similarity,
which acts as the contribution of each domain to the target dataset for better generalization.
4.5

Detailed Analysis

To further evaluate the performance of AFFAR on each class, we provide the fine-grained analysis
by the micro-average Receiver Operating Characteristics (ROC) curves in Figure 5. ROC is a more
effective metric for the areas of cost-sensitive learning and unbalanced class issues [14]. We also
calculate the AUC (Area Under Curve) on the four tasks of the PAMAP2 dataset. Figure 5 illustrates
the following conclusions: 1) Our AFFAR can achieve good performance on four tasks. The microaverage results of four target tasks are all higher than 0.91 which evaluates the classification
effectiveness of AFFAR. 2) Results of all methods on Target-1 are less satisfying, which may be
because the distribution discrepancy of the other domains is smaller so the generalization capability
is reduced. The improvement of AFFAR is more obvious on Target-1 than on the other three target
tasks, which shows that our approach is more robust on hard tasks.
Meanwhile, we randomly select a task of PAMAP2 to further make the fine-grained evaluationfrom the perspective of comparison with some state-of-the-art methods by utilizing the metrics ofmulti-class precision (P), recall (R), F1 scores (F1) in Table 5 and the visualization of the confusion
matrix for each category in Figure 6. From these experimental results, it can be observed that
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

000:16

Qin, et al.

Table 5. Precision, recall, and F1 score of AFFAR and other comparison methods for each class on PAMAP2.
Numbers in each cell denote the percentage of the prediction for each class.
DANN
Activity
Lying
SittingStanding
Walking
Ascending stairs
Descending stairs
Vacuum cleaning
Ironing
Average

P
1.00
0.83
0.85
0.88
0.82
0.85
0.73
0.91
0.86

R
0.94
0.87
0.78
0.95
0.73
0.69
0.85
0.93
0.84

GroupDRO
F1
P
0.97 1.00
0.85 0.83
0.81 0.89
0.91 0.90
0.77 0.82
0.76 0.80
0.79 0.74
0.92 0.93
0.85 0.86

RF1
0.96 0.98
0.87 0.85
0.81 0.85
0.94 0.92
0.61 0.70
0.73 0.77
0.89 0.81
0.97 0.95
0.85 0.85

RSC
P
0.99
0.82
0.82
0.95
0.84
0.88
0.69
0.93
0.87

R
0.96
0.83
0.82
0.93
0.70
0.76
0.82
0.97
0.85

AND-mask
F1
P
0.97 0.99
0.82 0.89
0.82 0.82
0.94 0.91
0.77 0.80
0.81 0.83
0.75 0.70
0.95 0.94
0.85 0.86

R
0.96
0.77
0.81
0.94
0.80
0.71
0.88
0.96
0.85

AFFAR

F1
0.97
0.82
0.81
0.92
0.80
0.77
0.78
0.95
0.85

P
1.00
0.92
0.83
0.95
0.84
0.97
0.72
0.94
0.90

(a) ERM(b) DANN(c) GroupDRO(d) RSC(e) AND-mask(f) Ours

R
0.96
0.80
0.92
0.96
0.73
0.76
0.92
0.97
0.88

F1
0.98
0.86
0.87
0.95
0.78
0.85
0.81
0.95
0.88

Fig. 6. Confusion matrices of all methods on PAMAP2. Classes 0-7 denote the following activities: lying,sitting, standing, walking, ascending stairs, descending stairs, vacuum cleaning, and ironing.

AFFAR can get the most number of the best precision-recall-F1 results on all categories thanother comparison methods, and can get the best average precision-recall-F1 across each class.
Besides, from the confusion matrix, we can see that AFFAR can get more balance results on eachcategory. Most methods get less satisfying results on the fourth and fifth class, AFFAR can reduce
the performance degradation. Although AND-mask can get the best performance on the fourthclass, it gets less satisfying results on other classes than AFFAR except for the first class where all
the comparative methods can achieve a satisfying performance.
4.6

Parameter Sensitivity, Convergence, and Time Analysis

We empirically evaluate the sensitivity of two parameters 𝜆 and 𝛽 by setting their values from{0.005, 0.01, 0.1, 1, 5, 10} and {0.05, 0.1, 0.5, 1, 5, 10}, respectively . The results are shown in Figure 7(a)and 7(b). It indicates that our AFFAR stays robust to a wide range of parameter choices.
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

Domain Generalization for Activity Recognition via Adaptive Feature Fusion

000:17280

Training Time (s)

240
200
160
120
80
40

(a) Sensitivity of 𝜆(b) Sensitivity of 𝛽

ERM
DANN
GroupDRO
RSC
AND-mask
Ours(c) Convergence

DSADS

DatasetUSC-HAD(d) training Time

Fig. 7. Parameter sensitivity of (a) 𝜆 and (b) 𝛽. (c) is convergence analysis. (d) is training time analysis.
Table 6. Comparison of inference time and weighted F1-score

MethodERM

DANNGroupDRO

RSC

AND-mask

AFFAR (Ours)

Inference time (s) .000010
Weighted F1 (%)
82.10.00001282.90.00001182.07.00001182.15.00001281.68.000057
85.43

We also empirically analyze the convergence of our method and draw the loss curve on arandomly chosen task of the PAMAP2 dataset in Figure 7(c). Other tasks follow similar observations.
The results show that AFFAR can converge in dozens of epochs, indicating that it is easy to train.
Furthermore, we show the training time of each method on one task in Figure 7(d). We see thatthe training time of our method is almost the same as others, while slightly takes more time insome circumstances. This is reasonable because our method is an ensemble-based learning processwith several domain-specific branches to be learned, which is comparable with other methods. We
obtain similar observations for the inference time.
As inference time is also very crucial to make quick classification while performing accurate
activity recognition. Table 6 shows the comparison of the average inference time of each sample
and weighted F1 score on the DSADS dataset. We can observe that the inference time of different
methods are similar and the proposed method is slightly longer, while the recognition performanceis the best among the comparison methods. This indicates the method can make better activity
recognition in applications with a little more time. And we will make efforts to further reduce theinference time for better applications in future work.
5 APPLICATION TO ADHD RECOGNITION
Attention Deficit Hyperactivity Disorder (ADHD) is one of the most common mental disorders inchildren [26]. ADHD is characterized by inappropriate inattention, hyperactivity, and impulsivity [2]. It is often accompanied by some motor abnormality, thus it is possible and necessary toutilize the HAR method to assist the diagnosis. In this section, we apply the proposed AFFAR to
the ADHD application to further evaluate its effectiveness.
We apply our algorithm to a real-world ADHD dataset [26]. This dataset is collected with adesigned wearable diagnostic assessment system in the room environment with little tension.
Ten diagnostic tasks including six interactions with the screen tasks and four physical objects
interaction tasks (Schulte grid, Multi-ball tracking, Catching grasshopper, Drinking birds, Limb
reaction, Reading, Finger holes, Shape-color conflicting, Catching worms, and Keeping balance)
are designed for children for ADHD symptoms according to DSM-5. Six wearable sensors areattached to children, one on each wrist and each ankle, one on the head, one on the waist during
the ten assessment tasks and accelerometer data are collected. 54-dimensional motion features are
extracted according to [26]. For more detailed information about this dataset, please refer to [26].
ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

000:18

Qin, et al.

Accuracy (%)

In our experiment, features from 83 normal
76
children and 83 children with ADHD are involved. Meanwhile, the children with ADHD
are diagnosed by doctors to confirm they meetthe diagnostic criteria for ADHD. Furthermore,
we have more fine-grained labels, i.e., chil72
dren with ADHD are diagnosed with subtypes,
i.e. predominantly inattentive (50 children),predominantly hyperactive-impulsive (14 children), and combination (19 children). This is a
68
ERM GroupDRO DANN RSC AND-mask Ours
multi-class classification task.
MethodWe divide the data into the source training
set and target test set, with a number of participants of 125 and 41, respectively. To harness Fig. 8. Multi-class classification in ADHD application.
the different distributions in the training set,
we further divide it into 3 domains. The classification results of all methods are shown in Figure 8.
We can observe that AFFAR improves the classification accuracy with a rate of around 2.44%. It
indicates the effectiveness of AFFAR in classification with ADHD and the effectiveness of finegrained classification on subtypes. Experimental results also show that AFFAR has the potential tothe applied in real-life wearable healthcare.
6

CONCLUSIONS AND FUTURE WORK

Generalization to the unseen test data has always been the key research and application problem
in human activity recognition. While transfer learning and domain adaptation approaches rely on
the availability of test data during the training stage, in this paper, we propose AFFAR to solve
this problem by learning both the domain-invariant and domain-specific features. The key of ouralgorithm is to preserve the specific representations of the training data while learning transferable
representations, which could be informative to the generalization on unseen test data. Experiments
on both public datasets and the real application have demonstrated the superiority of our method.
In the future, we plan to extend AFFAR in the following two directions. First, apply it to more
healthcare applications such as the diagnosis of Parkinson’s disease. Second, experiments showthat there is still a gap between our method and fine-tuning, which motivates us to improve theperformance of our method by introducing other domain-invariant learning modules.
ACKNOWLEDGMENTS
This work is supported by National Key Research and Development Plan of China (No. 2021YFC2501202),Natural Science Foundation of China (No. 61972383, No. 61902377, No. 62101530, No. 61902379),
Science and Technology Service Network Initiative, Chinese Academy of Sciences (No. KFJ-STSQYZD-2021-11-001).
REFERENCES[1] Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. 2019. Generalizing
to unseen domains via distribution matching. arXiv preprint arXiv:1911.00804 (2019).
[2] Russell A Barkley. 1998. Attention-deficit hyperactivity disorder. Scientific American 279, 3 (1998), 66–71.
[3] Billur Barshan and Murat Cihan Yüksek. 2014. Recognizing daily and sports activities in two open source machinelearning environments using body-worn sensor units. Comput. J. 57, 11 (2014), 1649–1667.
[4] Gilles Blanchard, Gyemin Lee, and Clayton Scott. 2011. Generalizing from several related classification tasks to a new
unlabeled sample. Advances in neural information processing systems 24 (2011), 2178–2186.

ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

Domain Generalization for Activity Recognition via Adaptive Feature Fusion

000:19[5] Fabio M Carlucci, Antonio D’Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. 2019. Domain generalization by solving jigsaw puzzles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
2229–2238.
[6] Junbum Cha, Hancheol Cho, Kyungjae Lee, Seunghyun Park, Yunsung Lee, and Sungrae Park. 2021. Domain generalization needs stochastic weight averaging for robustness on domain shifts. arXiv 2102.08604 (2021).
[7] Youngjae Chang, Akhil Mathur, Anton Isopoussu, Junehwa Song, and Fahim Kawsar. 2020. A systematic study of
unsupervised domain adaptation for robust human-activity recognition. Proceedings of the ACM on Interactive, Mobile,
Wearable and Ubiquitous Technologies 4, 1 (2020), 1–30.
[8] Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. 2021. Deep learning for sensor-based
human activity recognition: Overview, challenges, and opportunities. ACM Computing Surveys (CSUR) 54, 4 (2021),
1–40.
[9] Liming Chen, Jesse Hoey, Chris D Nugent, Diane J Cook, and Zhiwen Yu. 2012. Sensor-based activity recognition.
IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 42, 6 (2012), 790–808.
[10] Diane Cook, Kyle D Feuz, and Narayanan C Krishnan. 2013. Transfer learning for activity recognition: A survey.
Knowledge and information systems 36, 3 (2013), 537–556.
[11] Ditte Demontis, Raymond K Walters, Joanna Martin, Manuel Mattheisen, Thomas D Als, Esben Agerbo, Gísli Baldursson,
Rich Belliveau, Jonas Bybjerg-Grauholm, Marie Bækvad-Hansen, et al. 2019. Discovery of the first genome-wide
significant risk loci for attention deficit/hyperactivity disorder. Nature genetics 51, 1 (2019), 63–75.
[12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectionaltransformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
[13] Thomas G Dietterich et al. 2002. Ensemble learning. The handbook of brain theory and neural networks 2 (2002),
110–125.
[14] Tom Fawcett. 2006. An introduction to ROC analysis. Pattern recognition letters 27, 8 (2006), 861–874.
[15] Kyle Dillon Feuz and Diane J Cook. 2014. Heterogeneous transfer learning for activity recognition using heuristic
search techniques. International Journal of Pervasive Computing and Communications (2014).
[16] Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep
networks. In International Conference on Machine Learning. PMLR, 1126–1135.
[17] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario
Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The journal of machinelearning research 17, 1 (2016), 2096–2030.
[18] Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. 2015. Domain generalization for object
recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision.
2551–2559.
[19] Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool. 2019. Dlow: Domain flow for adaptation and generalization. In
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2477–2486.
[20] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. 2012. A kernel
two-sample test. Journal of Machine Learning Research 13, Mar (2012), 723–773.
[21] Yu Guan and Thomas Plötz. 2017. Ensembles of deep lstm learners for activity recognition using wearables. Proceedingsof the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (2017), 1–28.
[22] Ishaan Gulrajani and David Lopez-Paz. 2021. In search of lost domain generalization. In International Conference on
Learning Representations (ICLR).
[23] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. 2020. Self-challenging improves cross-domain generalization.
In European Conference on Computer Vision (ECCV), Vol. 2.
[24] Navdeep Jaitly, Patrick Nguyen, Andrew Senior, and Vincent Vanhoucke. 2012. Application of pretrained deep neuralnetworks to large vocabulary speech recognition. (2012).
[25] Wenchao Jiang and Zhaozheng Yin. 2015. Human activity recognition using wearable sensors by deep convolutional
neural networks. In Proceedings of the 23rd ACM international conference on Multimedia. 1307–1310.
[26] Xinlong Jiang, Yiqiang Chen, Wuliang Huang, Teng Zhang, Chenlong Gao, Yunbing Xing, and Yi Zheng. 2020. WeDA:
Designing and Evaluating A Scale-driven Wearable Diagnostic Assessment System for Children with ADHD. In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12.
[27] Md Abdullah Al Hafiz Khan, Nirmalya Roy, and Archan Misra. 2018. Scaling human activity recognition via deeplearning-based domain adaptation. In 2018 IEEE international conference on pervasive computing and communications(PerCom). IEEE, 1–9.
[28] Oscar D Lara and Miguel A Labrador. 2012. A survey on human activity recognition using wearable sensors. IEEE
communications surveys & tutorials 15, 3 (2012), 1192–1209.
[29] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. 2018. Learning to generalize: Meta-learning for domain
generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.

ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

000:20

Qin, et al.

[30] Wen Li, Zheng Xu, Dong Xu, Dengxin Dai, and Luc Van Gool. 2017. Domain generalization and adaptation using low
rank exemplar svms. IEEE transactions on pattern analysis and machine intelligence 40, 5 (2017), 1114–1127.
[31] Xi’ang Li, Jinqi Luo, and Rabih Younes. 2020. ActivityGAN: generative adversarial networks for data augmentation insensor-based human activity recognition. In ACM International Joint Conference on Pervasive and Ubiquitous Computing
and Proceedings of the 2020 ACM International Symposium on Wearable Computers. 249–254.
[32] Na Lu, Yidan Wu, Li Feng, and Jinbo Song. 2018. Deep learning for fall detection: Three-dimensional CNN combinedwith LSTM on video kinematic data. IEEE journal of biomedical and health informatics 23, 1 (2018), 314–323.
[33] Minghuang Ma, Haoqi Fan, and Kris M Kitani. 2016. Going deeper into first-person activity recognition. In Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition. 1894–1903.
[34] Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. 2018. Best sources forward: domain
generalization through source-specific nets. In 2018 25th IEEE international conference on image processing (ICIP). IEEE,
1353–1357.
[35] Toshihiko Matsuura and Tatsuya Harada. 2020. Domain generalization using a mixture of multiple latent domains. In
Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 11749–11756.
[36] Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. IEEE Transactions on knowledge and dataengineering 22, 10 (2009), 1345–1359.
[37] Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. 2020. Learning
explanations that are hard to vary. In International Conference on Learning Representations (ICLR).
[38] Fengchun Qiao, Long Zhao, and Xi Peng. 2020. Learning to learn single domain generalization. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition. 12556–12565.
[39] Xin Qin, Yiqiang Chen, Jindong Wang, and Chaohui Yu. 2019. Cross-dataset activity recognition via adaptive spatialtemporal transfer learning. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4(2019), 1–25.
[40] Nishkam Ravi, Nikhil Dandekar, Preetham Mysore, and Michael L Littman. 2005. Activity recognition from accelerometer data. In Aaai, Vol. 5. Pittsburgh, PA, 1541–1546.
[41] Attila Reiss and Didier Stricker. 2012. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th
International Symposium on Wearable Computers. IEEE, 108–109.
[42] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. 2020. Distributionally robust neural networks
for group shifts: On the importance of regularization for worst-case generalization. In International Conference on
Learning Representations (ICLR).
[43] Andrea Rosales Sanabria and Juan Ye. 2020. Unsupervised domain adaptation for activity recognition across heterogeneous datasets. Pervasive and Mobile Computing 64 (2020), 101147.
[44] Mattia Segu, Alessio Tonioni, and Federico Tombari. 2020. Batch normalization embeddings for deep domain generalization. arXiv preprint arXiv:2011.12672 (2020).
[45] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. 2018.
Generalizing across domains via cross-gradient training. ICLR (2018).
[46] Ling Shao, Fan Zhu, and Xuelong Li. 2014. Transfer learning for visual categorization: A survey. IEEE transactions onneural networks and learning systems 26, 5 (2014), 1019–1034.
[47] Gulbadan Sikander and Shahzad Anwar. 2018. Driver fatigue detection systems: A review. IEEE Transactions onIntelligent Transportation Systems 20, 6 (2018), 2339–2352.
[48] Elnaz Soleimani and Ehsan Nazerfard. 2021. Cross-subject transfer learning in human activity recognition systems
using generative adversarial networks. Neurocomputing 426 (2021), 26–34.
[49] Mihaela C Stoian, Sameer Bansal, and Sharon Goldwater. 2020. Analyzing ASR pretraining for low-resource speech-totext translation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
IEEE, 7909–7913.
[50] Baochen Sun and Kate Saenko. 2016. Deep coral: Correlation alignment for deep domain adaptation. In European
conference on computer vision. Springer, 443–450.
[51] Vladimir Vapnik. 1992. Principles of risk minimization for learning theory. In Advances in neural information processing
systems. 831–838.
[52] Hong-Bo Wang, Yanze Xue, Xiaoxiao Zhen, and Xuyan Tu. 2018. Domain Specific Learning for Sentiment Classification
and Activity Recognition. IEEE Access 6 (2018), 53611–53619.
[53] Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. 2019. Deep learning for sensor-based activity
recognition: A survey. Pattern Recognition Letters 119 (2019), 3–11.
[54] Jindong Wang, Yiqiang Chen, Lisha Hu, Xiaohui Peng, and S Yu Philip. 2018. Stratified transfer learning for crossdomain activity recognition. In 2018 IEEE International Conference on Pervasive Computing and Communications(PerCom). IEEE, 1–10.

ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.

Domain Generalization for Activity Recognition via Adaptive Feature Fusion

000:21[55] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip
Yu. 2022. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge andData Engineering (2022).
[56] Shujun Wang, Lequan Yu, Kang Li, Xin Yang, Chi-Wing Fu, and Pheng-Ann Heng. 2020. Dofe: Domain-oriented
feature embedding for generalizable fundus image segmentation on unseen datasets. IEEE Transactions on Medical
Imaging 39, 12 (2020), 4237–4248.
[57] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. 2016. A survey of transfer learning. Journal of Big data 3, 1
(2016), 1–40.
[58] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. 2020. Improve unsupervised domain adaptation withmixup training. arXiv preprint arXiv:2001.00677 (2020).
[59] Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. 2019. Time-series generative adversarial networks. Advancesin Neural Information Processing Systems 32 (2019).
[60] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neuralnetworks?. In Advances in Neural Information Processing Systems (NeurIPS).
[61] Mi Zhang and Alexander A Sawchuk. 2012. USC-HAD: a daily activity dataset for ubiquitous activity recognition
using wearable sensors. In ACM UbiComp. ACM, 1036–1043.
[62] Fan Zhou, Zhuqing Jiang, Changjian Shui, Boyu Wang, and Brahim Chaib-draa. 2020. Domain generalization with
optimal transport and metric learning. arXiv preprint arXiv:2007.10573 (2020).

ACM Trans. Intell. Syst. Technol., Vol. 0, No. 0, Article 000. Publication date: May 2021.