This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MIS.2020.2988604, IEEE Intelligent
Systems
IEEE INTELLIGENT SYSTEMS, VOL. XX, NO. X, 20191

FedHealth: A Federated Transfer Learning
Framework for Wearable Healthcare
Yiqiang Chen, Xin Qin, Jindong Wang, Chaohui Yu, Wen Gao
Abstract—With the rapid development of computing technology, wearable devices make it easy to get access to people’s healthinformation. Smart healthcare achieves great success by training machine learning models on a large quantity of user personal data.
However, there are two critical challenges. Firstly, user data often exists in the form of isolated islands, making it difficult to performaggregation without compromising privacy security. Secondly, the models trained on the cloud fail on personalization. In this paper, we
propose FedHealth, the first federated transfer learning framework for wearable healthcare to tackle these challenges. FedHealthperforms data aggregation through federated learning, and then builds relatively personalized models by transfer learning. Wearable
activity recognition experiments and real Parkinson’s disease auxiliary diagnosis application have evaluated that FedHealth is able toachieve accurate and personalized healthcare without compromising privacy and security. FedHealth is general and extensible in many
healthcare applications.
Index Terms—Federated learning, Transfer learning, Wearable healthcare.

F

1

I NTRODUCTIONA

CTIVITIES of daily living (ADL) are highly related topeople’s health. Recently, the development of wearable technologies helps people to understand their healthstatus by tracking activities using wearable devices suchas smartphone, wristband, and smart glasses. Wearable
healthcare has the potential to provide early warnings toseveral cognitive diseases such as Parkinson’s [1] and small
vessel diseases [2]. Other applications include mental healthassessment, fall detection, and sports monitoring [3]. In fact,
there is a growing trend for wearable healthcare over theyears.
In healthcare applications, machine learning models areoften trained on sufficient user data to track health status.
Unfortunately, there are two critical challenges in today’s
wearable healthcare (Figure 1). First of all, in real life, data
often exists in the form of isolated islands. Although there isplenty of data in different organizations, it is not possible toshare them due to privacy and security concerns as shown in
Figure 1. This makes it hard to train powerful models using
the valuable data. Additionally, recently, China, the UnitedStates, and the European Union enforced the protectionof user data via different regularizations [4], [5]. Hence,
the acquisition of massive user data is not possible in realapplications.
The other important issue is personalization. Most of themethods are based on a common server model for nearly allusers. After acquiring sufficient user data to train a satisfac-

•

•
•
•

Yiqiang Chen is the corresponding author with Beijing Key Lab. of
Mobile Computing and Pervasive Devices, Inst. of Computing Tech.,
CAS and University of Chinese Academy of Sciences, Beijing, China, andPengcheng Laboratory, Shenzhen, China.
E-mail: yqchen@ict.ac.cn
Xin Qin and Chaohui Yu are with Beijing Key Lab. of Mobile Computing
and Pervasive Devices, Inst. of Computing Tech., CAS and University ofChinese Academy of Sciences, Beijing, China.
Jindong Wang is with Microsoft Research Asia, Beijing, China.
Wen Gao is with Pengcheng Laboratory, Shenzhen, China

Cloud A

Cloud B

Cloud C

Institution A

Institution B

Institution C

Smartphone

Watch

ActigraphFig. 1. The data islanding and personalization problems in wearable
healthcare

tory machine learning model, the model is then distributed
to all user devices on which the daily health information canbe tracked. This process lacks personalization. As can beseen, different users have different physical characteristics
and daily activity patterns. Therefore, the common model
fails to perform personalized healthcare.
In this paper, we propose FedHealth, the first federated transfer learning framework for wearable healthcare.
FedHealth can solve both of the data islanding and personalization problems. Through federated learning [6] andhomomorphic encryption [7], FedHealth aggregates the datafrom separate organizations to build powerful machinelearning models with the users’ privacy well preserved.
After the cloud model is built, FedHealth utilizes transfer
learning methods to achieve personalized model learningfor each organization. The framework can incrementally
update. FedHealth is extensible and can be deployed tomany healthcare applications to continuously enhance theirlearning abilities in real life.
Our contributions are as follows:

1541-1672 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: MICROSOFT. Downloaded on April 24,2020 at 01:28:49 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MIS.2020.2988604, IEEE Intelligent
Systems
IEEE INTELLIGENT SYSTEMS, VOL. XX, NO. X, 2019

1. We propose FedHealth, the first federated transfer
learning framework for wearable healthcare, which aggregates the data from different organizations without compromising privacy security, and achieves relatively personalized model learning through knowledge transfer.
2. We show the excellent performance achieved by FedHealth in smartphone-based human activity recognition.
Experiments show that FedHealth dramatically improves
the recognition accuracy compared to traditional learning
approaches.
3. FedHealth is extensible and can be the standard framework for many healthcare applications. Specifically, we design a FedHealth system and apply it in Parkinson’s disease
auxiliary diagnosis with the users’ privacy well preserved
and good performance achieved in the real situation.

2

R ELATED W ORK

2.1

Wearable Healthcare

Certain activities in daily life reflect early signals of somecognitive diseases [8]. For instance, the change of gait
may result from small vessel disease or stroke. A lot of
researchers pay attention to monitor users’ activities using
body-worn sensors [9], through which daily activities andsports activities can be recognized. With the developmentof wearable technology, smartphone, wristbands, and smart
glasses provide easy access to this information.
It is noteworthy that traditional healthcare applications
often build the model by aggregating all the user data.
However, in real applications, data are often separate andcannot be easily shared due to privacy issues [4]. Moreover,the models built by applications lack the ability of personalization.
2.2

Federated Transfer Learning

Federated machine learning was firstly proposed by
Google [10], where they trained machine learning models
based on distributed mobile phones all over the world. The
key idea is to protect user data during the process. Federated
learning has the ability to resolve the data islanding problems by privacy-preserving model training in the network.
Transfer learning aims at transferring knowledge fromexisting domains to a new domain. The key idea is to reducethe distribution divergence between different domains. To
this end, there are mainly two kinds of approaches: instancereweighting [11] and feature matching [12]. Recently, deeptransfer learning methods have made considerable successin many application fields. FedHealth is mainly related todeep transfer learning. Most of the methods assume theavailability of training data, which is not realistic. FedHealthmakes it possible to do deep transfer learning in the federated learning framework without accessing the raw user
data. Therefore, it is more secure.
Federated transfer learning considers scenarios where
neither samples nor features have much in common. Recently, more researchers start to focus on this field. According to [13], Liu et al. proposed a secure federated transfer
learning system in a two-party privacy preserving setting,which more focused on data security. And some researchers
proposed federated domain adaptation approaches which2extended domain adaptation to federated setting constraints
to tackle data privacy and domain shift. Although theresearch work is fast growing, federated transfer learning
faces many challenges to apply in practical application [13].
Our work is the first federated transfer learning framework
tailored for wearable healthcare applications and it can beextended with various transfer learning methods.

3

T HE P ROPOSED F ED H EALTH F RAMEWORK

3.1

Problem Definition

We are given data from N different users (organizations), denote the users by {S1 , S2 , · · · , SN } and the sensor
readings they provide by {D1 , D2 , · · · , DN }. Conventional
methods train a model MALL by combining all the data
D = D1 ∪ D2 ∪ · · · ∪ DN . All the data have different
distributions. In our problem, we want to collaborate all thedata to train a federated transfer learning model MF ED ,where any user Si does not expose its data Di to eachother. If we denote the accuracy as A, then the objective of
FedHealth is to ensure the accuracy of federated learning isclose to or superior to that of conventional learning denotedby:

AF ED − AALL > ∆,(1)

where ∆ is an extremely small non-negative real number.

3.2

Overview of the Framework

FedHealth aims to achieve accurate personal healthcare
through federated transfer learning without compromising
privacy security. Figure 2 gives an overview of the framework. Without loss of generality, we assume there are 3
users (organizations) and 1 server, which can be extended
to the more general case. The framework mainly consistsof four procedures. Firstly, the cloud model on the serveris trained based on public datasets. Then, the cloud model
is distributed to all users where each of them can train
their own model on their data. Subsequently, the user model
can be uploaded to the cloud to help training a new cloudmodel by model aggregation. Finally, each user can train
personalized models by utilizing the cloud model and data
and local data. In this step, since there is large distribution
divergence between server data and user data, transfer
learning is performed to make the model more tailored tothe user (right part in Figure 2). It is noteworthy that all theparameter sharing processes do not involve any leakage ofuser data through homomorphic encryption [7].
The federated learning paradigm is the main computing
model for the whole FedHealth framework. It deals withmodel building and parameter sharing during the entire
process. After the server model is learned, it can be directly
applied to the user. This is just what traditional healthcare
applications do for model learning. It is obvious that the
samples in the server are having highly different probability
distribution with the data generated by each user. Therefore,
the common model fails in personalization. Additionally,user models cannot easily be updated continuously due tothe privacy security issue.

1541-1672 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: MICROSOFT. Downloaded on April 24,2020 at 01:28:49 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MIS.2020.2988604, IEEE Intelligent
Systems
IEEE INTELLIGENT SYSTEMS, VOL. XX, NO. X, 20193

Cloud Model

Cloud

User
Data

Model B
Model A

Model N

Data A

Data B

User A

User B……

Model

Data C

Model
Transfer

User N

New User Model

Fig. 2. Overview of the FedHealth framework. “User” represents organizations

3.3

Federated Learning

FedHealth adopts the federated learning paradigm [6] toachieve encrypted model training and sharing. This step
mainly consists of two critical parts: cloud and user model
learning.
In FedHealth, we adopt deep neural networks to learn
both the cloud and user models. Deep neural networks
perform end-to-end feature learning and classifier training
by taking the raw inputs of the user data as inputs. Let fS
denote the server model to be learned, then the learning
objective becomes:arg min L =ΘnX`(yi , fS (xi )),

(2)

i=1

where `(·, ·) denotes the loss for the network, e.g. crossentropy loss for classification tasks. {xi , yi }ni=1 are samples
from the server data with n their sizes. Θ denotes all theparameters to be learned, i.e. the weight and bias.
After acquiring the cloud model, it is distributed to all
the users. As we can see from the “wall” in Figure 2, thedirect sharing of user information is forbidden. This process uses homomorphic encryption [7] to avoid information
leakage. Since the encryption is not our main contribution,
we will show the process of additively homomorphic encryption using real numbers. The encryption scheme of theweight matrix and bias vector are following the same idea.
The additively homomorphic encryption of a real number
a is denoted as hai. In additively homomorphic encryption,
for any two numbers a and b, we have hai + hbi = ha + bi.
Therefore, the parameter sharing can be done without leaking any information from the users. Through federated
learning, we can aggregate user data without compromising
privacy security.
Technically, the learning objective for user u is denotedas:nuXarg minLu =`(yiu , fu (xui )).
(3)
u
Θ

i=1

After all the user model fu is trained based on the sharedcloud model, it is uploaded to the server for aggregation. It

has been evaluated that with shared initialization, averaging
the models can achieve good performance in loss reductionin the approach of FederatedAveraging [14]. Thus, following [14], we use model average to align user models, K usermodels are averaged for the cloud model update in eachtraining round. Note that we simply average user models
here, we will further study the influence of specific model
parameters on the average in the future. The updated cloudmodel is denoted as:K

fS0 (w) =1 X
fu (w),
K k=1 k(4)

where w are parameters of the network, K is the number ofusers. After adequate rounds of iteration, the updated server
model fS0 has better generalization ability. Subsequently,
new users can take part in the next training round of server
model, thus FedHealth has the capability for incremental
learning.
3.4

Transfer Learning

Federated learning solves the data islanding problem.
Therefore, we can build models using all the user data.
Another important factor is the personalization. Even if we
can directly use the cloud model, it still performs poor ona particular user. This is due to the distribution difference
between the user and the cloud data. The common modelin the server only learns the coarse features from all users,
while it fails in learning the fine-grained information on aparticular user.
In this paper, FedHealth uses transfer learning to build apersonalized model for each user (organization). Recall thatfeatures in deep neural networks are highly transferable in
the lower levels of the network since they focus on learningcommon and low-level features. The higher layers learn
more specific features to the task [15]. In this way, after
obtaining the parameters of the cloud model, we can perform transfer learning on the user to learn their personalized
models.
Figure 3 presents the process of transfer learning for a
specific convolutional neural network (CNN). Suppose the1541-1672 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: MICROSOFT. Downloaded on April 24,2020 at 01:28:49 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MIS.2020.2988604, IEEE Intelligent
Systems
IEEE INTELLIGENT SYSTEMS, VOL. XX, NO. X, 2019

4

FC

User
model

FC

FC

Maxpool

Conv 1D

Maxpool

Conv 1D

Algorithm 1 The learning procedure of FedHealthServermodel

Alignment

FC

FrozenMaxpool

Conv 1D

Maxpool

Conv 1D

Data

Train

Fig. 3. The transfer learning process of FedHealthnetwork is composed of two convolutions layers (conv1,
conv2), two max-pooling layers (pool1, pool2), two
fully connected layers (fc1, fc2), and one softmax layer
for classification. The network is designed for human activity recognition where the input data is the activity signals
for a user and the output is his/her activity classes.
In model transfer, we think that the convolution layers
aims at extracting low-level features about activity recognition. Thus we keep these layers along with the maxpooling layers frozen, which means we do not update theirparameters in backpropagation. As for the fully connected
layers fc1 and fc2, since they are higher level, we believe
they focus on learning specific features for the task and user.
Therefore, we update their parameters during training. The
softmax serves as the classification function, which can beformulated as:
ezc
yj = PC
,
(5)
zc
c=1 e
where zc denotes the learned probability for class C , and yj
is the final classification result.
FedHealth adapts the inputs from different domains byreplacing fc2 with an alignment layer. We regard the public datasets as the source domain. Given the network fromthe server and user, we add a correlation alignment [16]layer before the softmax layer to further adapt the domains.
This alignment function is used to align the second-order
statistics between the inputs. Formally, the loss of correlation alignment is computed as follows:`CORAL =2

1
2
kCS − CT kF
4d2

(6)

where k·kF denotes the squared matrix Frobenius norm andd is the dimension of the embedding features. CS and CT
are the covariance matrices of the source and target features
computed by [16]. Let η denote the trade-off parameter. The
cross-entropy loss is calculated with the source and target
data. Therefore, the loss for the user model is computed by:

Input: Data from different users {D1 , D2 , · · · , DN }, η
Output: Personalized user model fu
1: Construct an initial cloud model fS with public datasets
using Eq. (2)2: Distribute fS to all users
3: Train user models using Eq. (3)4: Update all user models to the server using homomorphic encryption. Make models aggregation by using
Eq. (4). Then the server takes this aggregation modelas the updated cloud model fS0 .
5: Distribute fS0 to all users, then perform transfer learningon each user to get their personalized model fu using
Eq. (7)
6: Repeat the above procedures with the continuously
emerging user data

product, the more personalized the model can be. Other
than transfer learning, other popular methods such as incremental learning can also be embedded in FedHealth forpersonalization.
The entire framework can also adopt other machinelearning methods other than deep neural networks. For
instance, the gradient boosting decision tree can be integrated into the framework to harness the power of ensemble
learning. These lightweight models can be deployed to computation restricted wearable devices. This makes FedHealthmore general to real applications.

4

E XPERIMENTS

4.1

Datasets

We adopt a public human activity recognition dataset called
UCI Smartphone [17]. This dataset contains 6 activities
collected from 30 users with smartphones on the waist. 9
channels of accelerometer and gyroscope data is collected at
a constant rate of 50Hz. There are 10,299 instances in total.
In order to construct the problem situation in FedHealth,
we extracted 5 subjects (Subject IDS 26 ∼ 30) and regarded
them as the isolated users which cannot share data due toprivacy security. Data on the remaining 25 users are used
to train the cloud model. Henceforth, the objective is to use
the cloud model and all the 5 isolated subjects to improve
the activity recognition accuracy on the 5 subjects withoutcompromising the privacy. In short, it is a variant of theframework in Figure 2 where there are 5 users.

4.2 Implementation DetailsnuX`(yi , fu (xi ))+`(yiu , fu (xui ))+η`CORAL , On both the server and the user end, we adopt a CNN
Θu
i=1
i=1
(7) for training and prediction. The network is composed of 2
convolutional layers, 2 pooling layers, and 3 fully connected
layers. The network adopts a convolution size of 1 × 9.
3.5 Learning Process
It uses mini-batch Stochastic Gradient Descent (SGD) for
The learning procedure of FedHealth is presented in optimization. During training, we use 70% of the training
Algorithm 1. Note that this framework works continuously data for model training, while the rest 30% is for modelwith the new emerging user data. FedHealth can update the evaluation. We fix η = 0.01 and K = 5. We set theuser model and cloud model simultaneously when facing learning rate to be 0.01 with batch size of 64 and training
new user data. Therefore, the longer the user uses the epochs fixed to 80. The accuracy of user u is computed asarg min Lu =n
X

1541-1672 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: MICROSOFT. Downloaded on April 24,2020 at 01:28:49 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MIS.2020.2988604, IEEE Intelligent
Systems
IEEE INTELLIGENT SYSTEMS, VOL. XX, NO. X, 2019

5

TABLE 1
Classification accuracy (%) of the test subject

Subject
P1
P2
P3
P4
P5
AVG

KNN
83.8
86.5
92.2
83.1
90.5
87.2

SVM
81.9
96.9
97.2
95.9
98.6
94.1

RF
87.5
93.3
88.9
91.0
91.6
90.5

NoFed
94.5
94.5
93.4
95.5
92.6
94.1

FedHealth
98.8
98.8
100.0
99.4
100.0
99.4

100

Accuracy (%)

98

96

94

92

90

P1

NoFed

+Fine-tune

P2

P3

+MMD

+CORALP4

P5

Subject

Fig. 4. Extending FedHealth with other transfer learning methodsu ∧ŷ(x)=y(x)|, where y(x) and ŷ(x) denote the
Au = |x:x∈D|x:x∈Du|true and predicted labels on sample x, respectively.
We follow [7] for homomorphic encryption in federated learning. During transfer learning, we freeze all the
convolutional and pooling layers in the network. Only theparameters of the fully connected layers are updated with
SGD. To show the effectiveness of FedHealth, we compare
its performance with traditional deep learning (NoFed),
where we record the performances on each subject using
the initial server model only, and other traditional machinelearning methods. The hyperparameters of all the comparison methods are tuned using cross-validation. For the fairstudy, we run all the experiments 5 times to record theaverage accuracies.

4.3

Classification Accuracy

The activity classification accuracy for each subject is shownin Table 1. FedHealth achieves the best classification accuracy on all users. Compared to NoFed, it significantly
improves the average results by 5.3%. This is because
federated learning can indirectly utilize more informationfrom distributed data to train a better model, and throughtransfer learning, models become more personalized to thecharacteristics of each user. Compared to the traditional
methods (KNN, SVM, and RF), FedHealth also greatly improves the recognition results. In short, it demonstrates theeffectiveness of the FedHealth framework.
The results also show that for activity recognition, thedeep methods (NoFed and FedHealth) achieve better results
than traditional methods. This is due to the representation capability of deep neural networks, while traditional

methods rely on hand-crafted feature learning. Anotheradvantage of deep learning is that the models can be updated online and incrementally without retraining, while
traditional methods require further incremental algorithms.
This property is extremely valuable in federated transfer
learning where model reuse is important and helpful.

4.4

Evaluation of Extensibility

In this section, we analyze the extensibility of FedHealthwith different transfer learning approaches. We compare its
performance with two methods: 1) fine-tuning, which only
fine-tunes the network on each subject without explicitly
reducing the distribution divergence between domains; and2) transfer with MMD (Maximum Mean Discrepancy) [12],which replaces the alignment loss with MMD loss. The
comparison results are shown in Figure 4.
We can see that other than the alignment loss, FedHealthcan also achieve promising results using fine-tuning or
MMD. The results of transfer learning significantly outperform no transfer by 4% on average accuracy. This indicatesthat the transfer learning procedure of FedHealth is highly
effective and extensible. Therefore, FedHealth is general andcan be extended in many applications by integrating othertransfer learning algorithms. Moreover, the federated learning procedure can also be extended using other encryption
methods, which can be the future research.

5 A PPLICATION IN AUXILIARY D IAGNOSIS OF
PARKINSON ’ S D ISEASE
Parkinson’s disease usually distinguished with some motor
symptoms thus it is possible to utilize wearable healthcare
methods to help diagnosis [1]. Besides, patient data is a
privacy-sensitive issue that requires federated learning toaddress. Thus, we make an application of FedHealth in
the auxiliary diagnosis of Parkinson’s disease, which canbe deployed in hospitals. After the user model is trainedin hospital (user side), patients can download it into theirsmartphones and update in the next visit. They can makeself-test and get real-time feedback to acquire disease statusconveniently.

5.1

Parkinson’s Disease Dataset

We developed a smartphone application to collect patients’
acceleration and gyroscope signals in the symptom tests at
rate of 50Hz. Five symptom tests including Arm Droop,
Balance, Gait, Postural Tremor and Resting Tremor are provided in series. For each test, symptoms can be ranked into
five levels from normal to severe. The detailed description
and the interfaces of these tests are shown in Figure 5.
During the collection process, a Parkinson’s disease therapist is around to rate the symptoms. We collected sensor
data with more than 130 patients age from 19 to 87 years
old participated. In the following evaluation, we carriedexperiments on test data of Arm Droop and Postural Tremor
from where two classes with relatively adequate data are
selected.

1541-1672 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: MICROSOFT. Downloaded on April 24,2020 at 01:28:49 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MIS.2020.2988604, IEEE Intelligent
Systems
IEEE INTELLIGENT SYSTEMS, VOL. XX, NO. X, 2019

6

KNN
36.0 (0.32)
64.0 (0.64)
85.7 (0.86)
61.9 (0.61)

Subject
P1
P2
P3
AVG

KNN
50.4 (0.44)
59.5 (0.58)
64.3 (0.64)
58.1 (0.55)

FedHealth75.0 (0.75)
93.2 (0.93)
84.8 (0.84)
84.3 (0.84)

Upper Bound
88.6 (0.88)
100.0 (1.00)
89.1 (0.89)
92.6 (0.92)

FedHealth85.3 (0.81)
70.9 (0.67)
68.4 (0.68)
74.9 (0.72)

Upper Bound
87.0 (0.86)
87.3 (0.86)
75.1 (0.75)
83.1 (0.82)

100

90

90

80

80

70

70

Accuracy (%)

100

60
50
40
30

10

NoFed

+Fine-tune+MMD

5.3 Extensibility with Alternative Transfer Learning
Methods
Consistent with the experiment settings in Section 4.4, extensibility results on Arm Droop and Postural Tremor test
data are shown in Figure 6 (a) and (b). We can see that
FedHealth can achieve satisfying results with fine-tune or
MMD in most cases, this indicates that FedHealth is effective
and extensible with other transfer learning algorithms in thereal application.
5.4

Ablation Study

We further provide the ablation study to evaluate the twomain components of federated learning and transfer learn-

30

P2

NoFed

+Fine-tune

P1

P3

+MMD

+CORALP2

P3

Subject

Subject(a) Extensibility On Arm Droop Data(b) Extensibility On Postural Tremor Data

100

100

90

90

80

80

70

70

60
50
40
30

60
50
40
30
20

10

We evaluate the classification accuracy and macro F1 scoreon the collected dataset. Data is collected from three hospitals, we randomly selectly 70% from each hospital as public
dataset, and 30% as three users and we fixed K = 3.
Comparison results are shown in Table 2. Also, we provide
the results of ideal scenario with the proposed approach
but where all data been stored in one location to see theupper bound of the model performance. From the results
we can see that FedHealth achieves the best classification
accuracy which significantly outperforms the best comparison method by 21.6% and 16.6% in two datasets and thebest mean-F1 score on all users, and it narrows the gap with
the ideal situation. This indicates that through federated
transfer learning FedHealth can achieve effective symptom
classification in the real application.

40

0
P1

Classification Performance

50

10+CORAL

0

20

5.2

60

20

20

Accuracy (%)

Subject
P1
P2
P3
AVG

Arm Droop test
SVM
RF
NoFed
40.4 (0.38) 44.9 (0.41) 49.4 (0.49)
61.8 (0.62) 57.3 (0.47) 68.5 (0.68)
72.5 (0.72) 67.0 (0.64) 70.3 (0.70)
58.2 (0.57) 56.4 (0.51) 62.7 (0.62)
Postural Tremor test
SVM
RF
NoFed
47.4 (0.37) 59.0 (0.58) 47.5 (0.41)
58.3 (0.58) 51.8 (0.52) 61.2 (0.61)
52.3 (0.50) 51.1 (0.51) 58.0 (0.58)
52.7 (0.48) 54.0 (0.54) 55.6 (0.53)

Accuracy (%)

TABLE 2
Classification accuracy (%) and mean-F1 (shown in the bracket) of
each subject in Arm Droop and Postural Tremor test

Accuracy (%)

Fig. 5. Main interfaces of symptom tests in smartphone application

NoFed

Fed

10

FedHealth

0

NoFed

Fed

FedHealth

0
P1

P2

P3

Subject(c) Ablation Study On Arm Droop Data

P1

P2

P3

Subject(d) Ablation Study On Postural Tremor Data

Fig. 6. Extensibility with alternative transfer learning methods and ablation study on Parkinson’s disease data

ing. We use Fed to denote the averaged model without
personalized transfer learning. Results are shown in Figure 6(c) and (d). Results show that both federated learning andtransfer learning make important contributions to the performance of FedHealth. Comparing Fed with NoFed, we cansee model with federated settings can improve the classification accuracy, which indicate the effectiveness of federated
learning. Further comparing Fed with the proposed federated transfer learning framework FedHealth, we can seethat combined with transfer learning, each user model canachieve better performance on classification. This is because:1) with federated learning, the server can take advantage
of more information from multiple users indirectly to geta more generalized cloud model; 2) with transfer learning,users can utilize the cloud model to get a more personalizeduser model.

6

D ISCUSSIONSIn this section, we discuss its potential to be extended anddeployed to other situations with possible solutions.
1. FedHealth with incremental learning. Incremental
learning has the ability to update the model with the gradually changing time, environment, and users. In contrast to

1541-1672 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: MICROSOFT. Downloaded on April 24,2020 at 01:28:49 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MIS.2020.2988604, IEEE Intelligent
Systems
IEEE INTELLIGENT SYSTEMS, VOL. XX, NO. X, 2019transfer learning that focuses on model adaptation, incremental learning makes it possible to update the model inreal-time without much computation when new user data
arrive.
2. FedHealth to be applied in more applications. This
work mainly focuses on the possibility of federated transfer
learning in healthcare via activity recognition and Parkinson’s disease auxiliary diagnosis. In more real situations,
FedHealth can be deployed to more healthcare applications
such as elderly care, fall detection, cognitive disease detection, etc. We hope that through FedHealth, federated learning can become federated computing which can become anew computing model in the future.

7

C ONCLUSIONS AND F UTURE W ORK

In this paper, we propose FedHealth, the first federated
transfer learning framework for wearable healthcare. FedHealth aggregates the data from different organizations
without compromising privacy security and achieves relatively personalized model learning through knowledgetransfer. Experiments and applications have evaluated theeffectiveness of the framework. We also present a detailed
discussion for its potential from specific technical improvements to healthcare applications. FedHealth opens a newdoor for future research in wearable healthcare. In the future, we plan to extend FedHealth with incremental learning
to achieve more personalized and flexible healthcare.

ACKNOWLEDGMENTS
This paper is supported in part by National Key R & D
Plan of China (No. 2017YFB1002802), NSFC (No. 61972383),
R & D Plan in Key Field of Guangdong Province (No.
2019B010109001), and Beijing Municipal Science & Technology Commission (No.Z171100000117017).

R EFERENCES
[1]

[2]

[3]
[4]
[5]
[6]
[7]
[8]

Y. Chen, X. Yang, B. Chen, C. Miao, and H. Yu, “Pdassist: Objective
and quantified symptom assessment of parkinson’s disease via
smartphone,” in 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2017, pp. 939–945.
Y. Chen, C. Hu, B. Hu, L. Hu, H. Yu, and C. Miao, “Inferring
cognitive wellness from motor patterns,” IEEE Transactions on
Knowledge and Data Engineering, vol. 30, no. 12, pp. 2340–2353,
2018.
J. Wang, Y. Chen, S. Hao, X. Peng, and L. Hu, “Deep learning forsensor-based activity recognition: A survey,” Pattern Recognition
Letters, vol. 119, pp. 3–11, 2019.
N. Inkster, China’s Cyber Power. Routledge, 2018.
P. Voigt and A. Von dem Bussche, “The eu general data protection regulation (gdpr),” A Practical Guide, 1st Ed., Cham: Springer
International Publishing, 2017.
Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” ACM Transactions on Intelligent
Systems and Technology (TIST), vol. 10, no. 2, p. 12, 2019.
R. L. Rivest, L. Adleman, M. L. Dertouzos et al., “On data banks
and privacy homomorphisms,” Foundations of secure computation,
vol. 4, no. 11, pp. 169–180, 1978.
H. H. Atkinson, C. Rosano, E. M. Simonsick, J. D. Williamson,
C. Davis, W. T. Ambrosius, S. R. Rapp, M. Cesari, A. B. Newman,
T. B. Harris et al., “Cognitive function, gait speed decline, andcomorbidities: the health, aging and body composition study,”
The Journals of Gerontology Series A: Biological Sciences and Medical
Sciences, vol. 62, no. 8, pp. 844–850, 2007.

7[9]

Y. Chen, J. Wang, M. Huang, and H. Yu, “Cross-position activity
recognition with stratified transfer learning,” Pervasive and Mobile
Computing, vol. 57, pp. 1–13, 2019.
[10] J. Konečnỳ, H. B. McMahan, D. Ramage, and P. Richtárik, “Federated optimization: Distributed machine learning for on-device
intelligence,” arXiv preprint arXiv:1610.02527, 2016.
[11] P. Huang, G. Wang, and S. Qin, “Boosting for transfer learningfrom multiple data sources,” Pattern Recognition Letters, vol. 33,
no. 5, pp. 568–579, 2012.
[12] X. Qin, Y. Chen, J. Wang, and C. Yu, “Cross-dataset activity
recognition via adaptive spatial-temporal transfer learning,” Proceedings of the ACM on Interactive, Mobile, Wearable and UbiquitousTechnologies, vol. 3, no. 4, pp. 1–25, 2019.
[13] Q. Yang, Y. Liu, Y. Cheng, Y. Kang, T. Chen, and H. Yu, “Federated
learning,” Synthesis Lectures on Artificial Intelligence and Machine
Learning, vol. 13, no. 3, pp. 1–207, 2019.
[14] H. B. McMahan, E. Moore, D. Ramage, S. Hampson et al.,“Communication-efficient learning of deep networks from decentralized data,” arXiv preprint arXiv:1602.05629, 2016.
[15] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable
are features in deep neural networks?” in Advances in neural
information processing systems, 2014, pp. 3320–3328.
[16] B. Sun, J. Feng, and K. Saenko, “Return of frustratingly easy
domain adaptation,” in Thirtieth AAAI Conference on Artificial
Intelligence, 2016.
[17] D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz,“Human activity recognition on smartphones using a multiclass
hardware-friendly support vector machine,” in International workshop on ambient assisted living. Springer, 2012, pp. 216–223.

Yiqiang Chen (S’01 M’03 SM’19) received the PH.D. degree in computer science from the Institute of Computing Technology, Chinese
Academy of Sciences, Beijing, China, in 2003. In 2004, he was a Visiting
Scholar Researcher with the Department of Computer Science, Hong
Kong University of Science and Technology (HKUST), Hong Kong. He
is currently a professor and the director of the Pervasive Computing
Research Center at the Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS). He serves as the AE of IEEE TETCI
and IEEE Access. His research interests include artificial intelligence,
pervasive computing, and human computer interaction.

Xin Qin received the B.S. and M.S. degrees in computer science andtechnology from Shandong Normal University, Jinan, China, in 2015
and 2018, respectively. Now she is a Ph.D. candidate in the Institute ofComputing Technology, Chinese Academy of Sciences, Beijing, China.
Her current research interests include machine learning, pervasive computing, and activity recognition.

Jindong Wang is currently a researcher at Microsoft Research, Beijing,
China. He received his Ph.D degree from Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China in 2019. His research interest mainly includes transfer learning, machine learning, data
mining, and artificial intelligence. He is the publicity co-chair of IJCAI
2019, and serves as the reviewers of many journals and conferences,including TPAMI, CSUR, AAAI, ACM CHI, and Neurocomputing. He is
also a member of IEEE, ACM, AAAI, and CCF.

Chaohui Yu received the bachelor degree in communication engineering from Shandong University, Shandong, China in 2017. He is currently
pursuing the master degree from Institute of Computing Technology,
Chinese Academy of Sciences, Beijing, China in 2019. His researchinterest mainly includes transfer learning, deep learning, and artificial
intelligence.

1541-1672 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: MICROSOFT. Downloaded on April 24,2020 at 01:28:49 UTC from IEEE Xplore. Restrictions apply.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MIS.2020.2988604, IEEE Intelligent
Systems
IEEE INTELLIGENT SYSTEMS, VOL. XX, NO. X, 20198

Wen Gao (M’92 SM’05 F’09) received the Ph.D. degree from the
University of Tokyo, Tokyo, Japan. He has been the Director of thePeng Cheng Laboratory, Shenzhen, China, since 2018. He is currently
a Professor of computer science with Peking University, Beijing. He has
authored extensively, including five books and over 600 technical articles
in refereed journals and conference proceedings in image processing,video coding and communication, pattern recognition, etc. He is a member of the China Engineering Academy. He has been the Chair of theIEEE ICME and ACM Multimedia. He served or serves on the Editorial
Board of several journals, such as the IEEE TCSVT, the IEEE TMM, etc.

1541-1672 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: MICROSOFT. Downloaded on April 24,2020 at 01:28:49 UTC from IEEE Xplore. Restrictions apply.