Co-MEAL: Cost-Optimal Multi-Expert Active Learning
Architecture for Mobile Health Monitoring
Ramyar Saeedi

Keyvan Sasani

Assefaw H. Gebremedhin

SCADS Lab, School of EECSWashington State University
Pullman, Washington 99164
rsaeedi@eecs.wsu.edu

SCADS Lab, School of EECSWashington State University
Pullman, Washington 99164
ksasani@eecs.wsu.edu

SCADS Lab, School of EECSWashington State University
Pullman, Washington 99164
assefaw@eecs.wsu.edu

ABSTRACT
Mobile health monitoring plays a central role in a variety of healthcare applications. Using mobile technology, health-care providers
can access clinical information and communicate with subjects
in real-time. Due to the sensitive nature of health-care applications, these systems need to process physiological signals highly
accurately. However, as mobile devices are employed in dynamic
environments, the accuracy of a machine learning model drops
whenever a change in configuration of the system occurs. Therefore, data mining and machine learning techniques that specificallyaddress challenges associated with dynamic environments (e.g. different users, signal heterogeneity) are needed. In this paper, using
active learning as an organizing principle, we propose a cost-optimal
multiple-expert architecture to adapt a machine learning model (e.g.
classifier) developed in a given context to a new context or configuration. More specifically, in our architecture, a system’s machinelearning model learns from experts available to the system (e.g.
another mobile device, human annotator) while minimizing thecost of data labeling. Our architecture also exploits collaboration
between experts to enrich their knowledge which in turn decreases
both cost and uncertainty of data labeling in future steps.
We demonstrate the efficacy of the architecture using a publicly
available dataset on human activity. We show that the accuracyof activity recognition reaches over 85% by labeling only 15% of
unlabeled data. At the same time, the number of queries fromhuman expert is reduced by up to 82%.

KEYWORDS
Mobile Health Monitoring; Wearables; Active Learning; Cost-Optimal;
Time Series; Uncertainty; Query Strategy; Physical Activity Recognition; Collaborative; Multi-Expert; Accelerometer.

1

INTRODUCTION

Mobile-health (M-health) is receiving growing attention due to its
capacity to provide better and more efficient patient care. M-healthtechnology refers to the use of smartphones, tablets, wearables, andPermission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
ACM-BCB’17, August 20–23, 2017, Boston, MA, USA.
© 2016 ACM. 978-x-xxxx-xxxx-x/YY/MM. . . $15.00
DOI: 10.1145/nnnnnnn.nnnnnnn

implantable devices to deliver health care and preventive health services. The World Health Organization (WHO) describes M-healthas “medical and public health practice supported by mobile devices”[1]. According to recent studies, 83 percent of physicians in the U.S.
already use M-health technology [2, 3].
Health-care providers use M-health technology for a varietyof goals: to access clinical information; to collaborate with careteams; to communicate with patients; and to conduct real-time
health monitoring. On the other end, patients/users can use an
M-health system to track their own health-data (e.g. Fitbit activity
tracker), to access clinical records through patient/user portal, andto communicate with health-care providers [3–6].
Mobile devices considerably improve the quality and effectiveness of service, bringing especially high value for the elderly, patients with chronic conditions, athletes, and those requiring constant monitoring of their daily state [7–12]. Despite these enormous
promises and potential use of M-health technology however, theutilization of mobile devices is currently largely limited to systems
designed for controlled environments using passive supervised
computational models. This has several undesirable consequences.
First, the accuracy of machine learning algorithms drops whenever the system’s configuration changes (e.g. sensor calibration,
device orientation, sensor location). For example, the training data
may be collected using a smartphone located on waist, while thefinal data processing may be conducted on a Fitbit wristband. Second, sensor data, generated during personalized exercises or tasks,
may be too specific or inadequate to be used as a training set for
a new user/patient. Moreover, even for the same user, the physiological patterns may change over time. Third, a user’s needs (e.g.
number of events to detect) may change over time. For example,
in a physical activity monitoring system, we may need to detect a
new set of activities. Finally, supervised machine learning models
are passive and as a result the system would necessarily fail orprovide inaccurate outputs whenever a sensor failure or removal
happens. Furthermore, a passive model is unable to leverage additional knowledge from a new point of view to detect events ofinterest in the mobile system.
Next generation mobile devices need to be context-aware andcapable of adapting a machine learning model to the context at
hand in order to generate appropriate feedbacks and make decisions(e.g. on-time alert, patient status). Such an adaptation is generallyachieved in one or more of these ways: (i) using available knowledge
including related data (e.g. other subjects), (ii) employing otherpoints of view (e.g. another sensor), and (iii) using feedback froma human expert (e.g. physician). However, these ways may cause

disturbance or interference in user’s life and may incur expenses
related to knowledge extraction [13–17].
Our aim in this work is to enable next generation mobile devices
to actively update their machine learning models while minimizing
the side-effects of system adaptation. In particular, we seek todesign a system that exploits available sensory systems, smart
environments, feedback from human experts, and related data. Sucha system needs to be able to handle multiple sources of knowledge
and update the associated computational models accordingly. Here,
we call each source of knowledge an expert (a human expert is one
type of expert).
In order to actively update a computational model whenevera new context is faced, the system needs to interact with the setof experts to access new labeled data [18–20]. The system should
strive to minimize the cost of active learning from the experts
while maintaining the system’s accuracy. The cost associated witheach expert may be defined in terms of user inconvenience, humanexpert’s feedback cost, equipment’s cost, power consumption, etc.
Furthermore, an expert typically has only limited knowledge ofuser status or is uncertain in its knowledge. Considering expert
uncertainty, a multi-expert system that implements collaboration
between experts can result in lower overall cost and more accurate
knowledge.
In this paper, we propose a cost-optimal multi-expert active learning architecture (Co-MEAL) targeted for M-health systems. The
architecture is designed to meet a variety of requirements: a) avoid
dependence on collecting lots of labeled data for each new configuration or context; b) minimize the total number of queries as well as
queries from human experts in order to reduce the active learning
cost; c) use the knowledge of different experts at the same time; d)allow for experts to collaborate to reduce both annotation cost anddata labeling uncertainty; e) be able to add a new mobile device tothe system for a more accurate decision making. We outline our
specific contributions as follows.
a) Architecture. We propose a novel architecture for collaborative active learning for M-health systems. A key idea of thearchitecture is to minimize overall query cost while maintaining
system’s accuracy. Fig. 1 gives a high level overview of the architecture. The architecture is unpacked and discussed in detail in
Section 2.
b) Problem Formulation. We formulate an optimization problem to model cost-optimal active learning in the multi-expert architecture. This formulation minimizes the cost of data annotation for
informative instances. The formulation is presented in Section 3
along with formal definitions.
c) Algorithms. We have developed three algorithms corresponding to different aspects of the architecture. The first algorithm
implements the overall functionality of the Co-MEAL architecture– it calls the other two algorithms as subroutines. The second
algorithm is a knowledge transfer algorithm whose purpose is toinitialize the learner of the system. The third algorithm is for finding
the cost-optimal expert in each query phase of the active learning
process. The algorithms are presented in Section 4.
d) Experimental Evaluation. We use a human physical activity dataset as a case study to validate the Co-MEAL architecture. We
developed several scenarios to show the effects of active learningon the accuracy of a mobile monitoring system. These scenarios

are used to show the applicability of the architecture for systemreconfiguration as well as sensor integration to the system. We
show that using the architecture increases the accuracy of physicalmonitoring by up to 45%. We also show that annotation cost is
substantially reduced because of cost-optimal expert selection as
well as collaboration between experts in the active learning phase.
Our experimental evaluation along with details of the experimental
set up are presented in Section 5.

2

SYSTEM ARCHITECTURE

A conventional mobile monitoring system is generally composedof several sensor nodes, a base station, and a back-end server. Eachsensor node is attached to the body (a wearable or an implant device) to sample and preprocess physiological signals and transmit
partial results (e.g. features) to the gateway. Moreover, sometimes
additional sensors are placed in the environment (e.g. smarthome)
to capture the context as well as the decision making process. The
gateway is a more powerful unit, such as a smartphone, that performs data fusion for the health analytic system. The results arefurther transmitted, typically through the Internet, to a back-end
server for storage, further processing, and clinical decision support
from physicians and health-care experts.
With this description of conventional architecture of M-healthsystems as a background, our proposed reconfigurable architecture
is constituted as depicted in Fig. 1. It consists of three main components: 1) databases (DB), 2) a data processing unit (DPU), and 3) anexpert management unit (EMU). The DPU is the core intelligenceof the M-health system. The DPU may be located entirely on thegateway or on both the gateway and the back-end server dependingon the computational power of the gateway. The DB, the EMU, andthe experts can be considered part of the back-end server.
The DPU communicates with the DB and the EMU whenever
there is a need for system reconfiguration. The incoming data issampled by the DPU, and then the DPU decides whether or not
to request labels. If the learner is capable of classifying the data
accurately using the labeled data in the current context, the active
learning step is deactivated. The DPU has multiple submodules
each of which is responsible for a different functionality in thereconfiguration phase. We discuss in the remainder of this section
the details of each submodule, the Databases, and the EMU.

2.1

Learner Initialization

In active learning, a learner often begins with a small numberof instances in the labeled training set L. However, an M-healthsystem quite often may have to start with no labeled data at all. Thisrequires more sophisticated strategies in the initialization phase of
the active learning process. Meanwhile, for many applications ofM-health, there may be data for related contexts. For example, in aphysical activity monitoring system, the data from other subjects
could be used to initialize the classifier of the system. In Section 4,
we present an initialization algorithm which uses the related labeleddata to build the first learner model.

2.2

Query Strategy

In order to reduce annotation cost, we need a query strategy to select the best subset of unlabeled instances. The decision of whether

Classification

Query
Strategy

Expert1

Database

Sensors…

Feature Extraction

Segmentation

Data

Preprocessing

Control

Learner

TargetUnlabeled
Data

RelatedLabeled
Data

TargetLabeled
Data

Expert Selector

Event of interest

Broadcast the label to the learner and other experts

Data to update classifier

Expert 2…

Expert K
Learner Initializaiton

Data Processing Unit

Expert Management Unit

Figure 1: High-level illustration of the cost-optimal multi-expert active learning (Co-MEAL) architecture for M-health monitoring systems.

or not to query an instance can be framed in several ways. One
approach is to evaluate samples using some measure that shows
how informative an instance is, since more informative instances
are more likely to be queried [18]. For such a purpose, an entropybased measure can be used. Entropy is an information-theoretic
measure that represents the amount of information needed to encode a distribution. An informative data instance is defined asfollows [18].
Definition 2.1 (Informative Instance). Let x be a vector that stands
for an unlabeled data sample and P M (`i |x) be the posterior proba∗ expressedbility under prediction model M. Then the data point x H
below is the most informative data point:∗xH= argminxmÕ

P M (`i |x) log P M (`i |x)

(1)

i=1

where `i ranges over all possible labels.
Another query strategy is a committee-based algorithm. This approach involves maintaining a committee of models (e.g classifiers)
which are all trained on the current labeled set L, but represent
competing hypotheses. Each committee member is then allowed
to vote on the labels of query candidates. The most informative
instance is considered to be the one about which they most disagree.
Different models should represent different regions of the feature
space, and have some measure of disagreement among committee
members [18].
In this paper, we use a hybrid query strategy of the above approaches that is based on a random forest classifier. The informativeness of an input instance is computed as the mean informativenessof the trees in the forest. The informativeness measure of a single
tree is the fraction of samples of the same class in a leaf.

2.3

The Databases

A key component in our active learning architecture is a set of
databases. Each database represents knowledge for a unique case.
Below, we formally define a database.
Definition 2.2 (Database). A database (or dictionary) DB is anunordered set of instances. The i th instance is a q-dimensional
vector denoted by DBi , where q is number of features extracted
from each intance. Each instance may or may not be labeled in the
dictionary.
In our architecture, there are three types of dictionaries. The
target unlabeled data dictionary stores the incoming streaming data
in real-time. This data will be used both to initialize the system andto modify the classifier. The second database, target labeled data,
gradually collects two kinds of information: (i) instances from thethird database, related labeled data, and (ii) queries from the EMU.
The database referred to as related labeled data is built based onrelated contexts and domains useful for initializing the classifierand decreasing the cost of queries.

2.4

The Expert Management Unit

In real-world health-care applications, labels may come from various sources, including different health-care experts, other mobile
devices or even a smart environment. Furthermore, the knowledge
provided by each expert is subject to uncertainty. For example,
the performance of human annotators depends on factors such aslevel of expertise, experience, and concentration/distraction [21].
Additionally, experts may have more or less knowledge for different
parts of the problem that has to be solved (e.g., different classes,
different features). Another factor is the cost of data annotation byeach expert. Therefore, it is necessary to design a system that is
capable of interacting with multiple experts efficiently.

The multi-expert architecture we envision satisfies several uniqueproperties. 1) The learning process relies on the collective intelligence of a group of experts. 2) The architecture could also provide
a learning opportunity for experts. In particular, our system allows
for each expert to update their knowledge based on the feedback
from other experts in each round of query. 3) The expert selection submodule chooses the expert(s) in such a way that the query
cost is minimized, while keeping the provided label for the queried
instance accurate. 4) The architecture leverages different pointsof view from the experts to provide a more comprehensive modelthrough the system’s life time. For example, new class labels may
arise during the life time, or a new sensor may be integrated into
the mobile health system. 5) An expert can be part of the dataprocessing unit. For example, in a multi-node mobile system, one
node shares the label with the DPU whenever the node is confident. We present the algorithm we developed for expert selection
in Section 4.

3

PROBLEM STATEMENT

In real applications, it is impossible to collect training data for every
context. Furthermore, data collected for one setting (e.g. a subject,a sensor location) may no longer be suitable for a new setting.
Therefore, there is a need to modify the machine learning algorithm(e.g. classifier) for accurate data processing. Given the initial related
labeled database DB R , the most relevant data instances can beselected to initiate the learner model. However, the initialized
model may be inaccurate due the difference between the related
data distribution and the current context. Whenever the user starts
using the health monitoring system, context-specific unlabeled datais collected gradually throughout the system’s life time and may
well be different from the related labeled data.
The key idea behind our architecture is that the learner achieves
continuously greater accuracy while minimizing the cost of annotation from the set of experts. The knowledge of the learner is used
to select the cost-optimal expert for each query. Therefore, ourgoal is to actively update the machine learning algorithms using
knowledge from experts.
An expert can be another sensor, a smart environment, a physician or the user. The query strategy submodule selects the top K
informative instances in each query round, for a specified parameter K. The selected instances are passed to the expert selection
submodule to find the cost-optimal, confident expert for each data
instance. The expert selection needs access to the cost and uncertainty of each expert for data annotation. Also, the architecture isdesigned to update the expert knowledge on each round of query.
We provide a formal definition of cost-optimal multi-expert active
learning in the remainder of this section.

Definition 3.1 (Uncertainty Score). An uncertainty score vector
U = (u 1 , u 2 , . . . , uk ) for a given expert is a vector whose i th entry
stores a value in the range [0, 100] (i.e. percentage) indicating
the uncertainty of the expert on its prediction result on the label`i . A number close to 100 indicates highest uncertainty (lowest
confidence) in prediction of a specific label.
In addition to uncertainty score, each expert has a query cost foreach label. Our final goal is to minimized the overall cost of queries.
We also assume that experts collaborate with each other to update
their knowledge in the active learning process. We formally define
the collaborative active learning (CAL) problem as follows.
Problem 1 (CAL Problem). Suppose each instance of data is aq-dimensional vector, where q is the number of features. Let XU ={x 1 , x 2 , . . . , x n } be the sequence of observations made by a learner
device (e.g. smartphone), the labels of which are not yet known withcertainty. Let X L = {(x n+1 , yn+1 ), (x n+2 , yn+2 ), . . . , (x n+m , yn+m )}be the set of labeled data which are used to initialize the learner.
This labeled data may come from related labeled data or partiallytrained learner. The collaborative active learning problem is to selecta subset X S ⊂ XU to query from the expert set E such that the overall
cost of query is minimized and the overall knowledge of experts is
maximized.

3.2

Problem Formulation

The CAL problem can be formulated as an optimization problem asfollows. Let βi be a binary variable that determines whether or not
the query strategy submodule selects x i as an informative instance:
βi

=



1,
0,

if label for x i is needed
otherwise.

(2)

Let α i j be a binary variable indicating whether or not the expert
selection model requests the label for instance x i from expert e j .
That is:
αi j

=



1,
0,

if label for x i is asked from e jotherwise(3)

And let c i j be the query cost for instance x i from expert e j andp be the number of experts. Then, the Integer Linear Programming(ILP) formulation for the CAL problem is as follows:

Minimizepn ÕÕβi α i j c i j(4)∀ i ∈ {i | βi = 1}(5)i=1 j=1

subject to:pÕ

αi j ≥ 1

j=1

3.1

Problem Definition

Let L = {`1 , `2 , ..., `k } be a set of k labels that the system aims torecognize. Let E = {e 1 , e 2 , . . . , ep } be a set of p possible experts eachof which has full or partial knowledge on the current labeled set L.
We model the knowledge of each expert on each `i ∈ L using an
uncertainty score vector. The formal definition of uncertainty scorefor each expert is as follows.

βi ∈ {0, 1}, α i j ∈ {0, 1}(6)

The objective function in (4) aims to minimize the total cost of
the queries from the set of experts. The constraint (5) guarantees
that each selected unlabeled data is asked from at least one expert
and the constraint in (6) ensures that the variable βi and α i j takeonly binary values.

Algorithm 1 CollaborativeActiveLearning

Algorithm 2 LearnerInitialization

Input: M, DB R , X L , XU , L = {`1 , `2 , . . . , `k }, K, ut h , E
Output: Updated Learner M
1: (DB L , M(x)) ← LearnerInitialization(X U , DB R , L)
2: X L ← X L ∪ DB L , X S ← ∅3: while u M > u t h do
4:
(X , u) ← getInformativeInstances(K, XU )5:
for all x i ∈ X do
6:yi ← ExpertSelection(x i , E, M(x i ), ut h )7:
X S ← X S ∪ (x i , yi )8:
end for9:
broadcastLabeledData(X S ) /*collaborative*/
10:
updateUncertaintyAll(X S , E) /*collaborative*/
11:
XL ← XL ∪ XS
12:
XS ← ∅
13:
M ← UpdateLearner(X L )14: end while

Input: DB R , XU , L = {`1 , `2 , . . . , `k }Output: DB L , Learner M(x)1: DB L ← ∅, SIM ← ∅2: (Centersu , Clustersu ) = Clustering(X U , k)
3: for all DB R i ∈ DB R do
4:
SIMi = cosSim(Centers DB R i , Centersu )5:SIM ← SIM ∪ SIMi
6: end for
7: for all `i ∈ L do8:
DB `i ← arдmax `i SIM
9:
DB L ← DB L ∪ DB `i
10: end for
11: M(X ) ← Classifier(DB L )4

ALGORITHMS

The CAL problem is an ILP, which in general is known to be NPHard. We develop a greedy algorithm that finds an approximate
solution.
Algorithm 1. The driver routine of our algorithm for solving
the CAL problem is presented in psuedocode form in Algorithm 1.
The algorithm takes as input the learner model M(x), the dictionaryof related labeled data DB R , the dictionary of target labeled data
X L , the dictionary of target unlabeled data XU , the label set L, thenumber of instances for each round of query K, the uncertainty
threshold ut h , and the set E of experts. It gives as output an updated
version of the model M(x).
Algorithm 1 calls the routine LearnerInitialization (Algorithm 2) to initialize the learner based on the closest data instances
to the current context in DB R . It then adds this to the target labeled
database X L . The while-loop after the learner initialization represents the active learning phase. In each iteration of the while-loop,
the most informative instances are selected via the function getInformativeInstances (recall the discussion about informative
instances in Section 2.2), which returns the K most informative
instances from the database of unlabeled data. Subsequently, theroutine ExpertSelection (Algorithm 3) is called to find the label
for each instance from the most-relevant, lowest-cost expert.
Then, the new labeled data are added to the target labeled database and the learner model M(x) is updated. The function BroadcastLabeledData shares the new labeled data with the other experts. The function UpdateUncertaintyAll is run on each expert
to update their knowledge and accordingly their uncertainty score.
This process is repeated until we iterate over all the selected instances or the learner’s uncertainty score falls below the thresholduth for all `i ∈ L. The learner built based on the enhanced target
labeled data database is returned as the output of the algorithm(CAL).
Algorithm 2. The LearnerInitialization algorithm takes
as input the dictionary of related labeled data DB R , the dictionaryof target unlabeled data XU , and the label set L consisting of kelements. The algorithm finds k clusters for the unlabeled data. It

Algorithm 3 ExpertSelection
Input: x, E, semi-label L, uth
Output: label `1: u = 100
2: while u > u th do
3:
Econf ← ∅, Cconf ← ∅4:
for all ei ∈ E do
5:
if Ui (L) ≤ uth then6:
add ei to the Econf /*confident experts*/
7:
add cost(ei ) to Cconf /*cost of experts*/
8:
end if
9:end for10:expert ← arдminei ∈Econf Cconf
11:
L ← label from expert for instance x
12:u ← uncertainty of expert in labeling x
13: end while
14: ` = Lthen searches the entire database of related data DB R to find theclosest set of labeled data to each cluster. We used cosine similarity
to measure the closeness of clusters and labeled data. The details
are shown in Algorithm 2.
Algorithm 3. The ExpertSelection algorithm is a greedy
algorithm. It takes as input the instance x, the set of experts E, asemi-label L, and the uncertainty threshold uth . The semi-label L
is the most probable label for instance x. The goal of the algorithm
is to find the least costly and most confident expert for the instancex. The set Econf is the set of experts that are confident enough todetect the semi-label L at a given moment during the course of
the algorithm. The semi-label L is updated in the course of thealgorithm, and is deemed acceptable if the uncertainty score of theselected expert is lower than the threshold uth . Further details areshown in Algorithm 3.

5

EXPERIMENTAL VALIDATION

We experimentally assess the efficacy of the proposed architectureusing real human activity data. We implemented all of the algorithms we developed and simulated the proposed architecture in
Python. We present our findings in this section. The section isTable 1: Statistical features extracted from each data segment.

5.2

Architecture Design Settings

In this section, we describe the design settings of the Co-MEAL
architecture we employed in our experiments.
Feature
Amp
Med
Mean
Max
P2P
Var
Std
RMS
S2E

Description
Amplitude of signal segment
Median of signal segment
Mean value of signal segment
Maximum amplitude of signal segment
Peak to peak amplitude
Variance of signal segment
Standard deviation
Root mean square powerStart to end value

5.2.1 Data Preparation. We applied a moving average filter on
the activity data to remove high frequency noises from sensor data.
After data filtering, a two-second sliding window was used for data
segmentation. We consider 50% overlap between consecutive data
segments.
A variety of features can be extracted from human activity data
segments. Previous studies have identified statistical features thatare effective for activity recognition [25, 26]. Table 1 lists the set offeatures that we extracted from individual data segments.

Table 2: Uncertainty score of experts in E for all label `i ∈ L.

E1
E2
E3
E4
E5
E6
E7
E8

`1
00.01
57.58
100.0
65.51
60.69
67.56
54.55
62.52

`2
00.02
56.60
80.36
60.30
60.69
53.28
75.76
07.89

`3
00.01
02.00
100.0
50.20
56.84
100.0
02.33
00.02

`4
00.01
0.020
04.33
13.56
82.61
02.66
01.20
10.66

`5
00.01
04.61
01.19
42.54
02.41
01.93
30.48
72.05

`6
00.03
01.09
02.60
01.77
13.86
36.28
04.09
02.54

`7
00.01
46.48
100.0
04.88
30.31
27.32
12.50
02.33

`8
00.20
100.0
100.0
47.18
28.00
48.99
74.12
22.42

organized as follows. We discuss our case study—physical activity monitoring—and describe the dataset we used in Section 5.1.
We present the architecture design settings in Section 5.2. In Section 5.3, we show the benefits of using active learning on physicalactivity monitoring. In Section 5.4, we demonstrate results on cost
evaluation of the architecture.

5.1

Case Study: Activity Monitoring

Activity recognition is an important research area in health monitoring systems because of its potential to enable novel contextaware applications in elderly care, fitness, education, sports, andpatients with chronicle diseases. Various types of sensors havebeen proposed for such purposes, ranging from motion sensors
such as accelerometers, gyroscopes, GPS sensors to combinationsof different sensor modalities [22–24]. Because of the importanceof physical activity monitoring, we chose an activity recognition
dataset to show the efficacy of our proposed architecture.
Dataset Description. The dataset we used is publicly available
at Data and Web Science Research Group, University of Mannheim[24]. Fifteen subjects, seven females and eight males, were asked toperform eight different daily activities: Walking, Running, Sitting,Standing, Stair down, Stair up, Jumping, and Lying. Subjects wereasked to perform the activity set while carrying seven mobile devices located on chest, shin, forearm, thigh, upper-arm, head, andwaist. Each subject performed each activity roughly 10 minutes
except for jumping which is performed only for 100 seconds. Eachmobile device is equipped with different types of sensors including
tri-axial accelerometer and gyroscopes.

5.2.2 Classifiers and Query Strategy. A variety of classifiers canbe used for activity detection. Previous studies have shown that
K-NN, SVM, decision tree (DT), random forest (RF) classifiers are
effective for activity recognition [27, 28]. In this work, we used RF
for the learner model and sensor-based experts for two reasons.
First, RF fits a number of decision tree classifiers on various subsamples of the training data and uses averaging to improve thepredictive accuracy to control over-fitting. Secondly, using RF, wealready have a built-in query strategy for the active learning phaseas we explained in Section 2.2.
5.2.3 Expert Initialization. For the purpose of simulation, we
built a multi-expert EMU using the available sensor data for different locations. We also used the ground truth label as the least
uncertain (most confident) expert in the EMU. In a real-world scenario, this expert can be a highly confident physician or the user inthe monitoring system. We used the labeled data of other locations
to simulate other experts.
To have more realistic scenario, each expert is trained in sucha way that it is confidence over a subset of label set L. Therefore,
each expert has imperfect knowledge over some of the labels. We
initialize the uncertainty score of each expert based on their missclassification for their own test data. Table 2 shows the expert set
that we used along with their uncertainty score over the label set
L. We set the uncertainty threshold to 20% for each expert in theexpert selection procedure in all of the below experiments except
for Section 5.4.1. We use the K-means clustering algorithm for thelearner initialization.

5.3

Performance Evaluation Scenarios

To evaluate the performance of the Co-MEAL architecture, we used
two different scenarios, a single sensor learning scenario and amultiple sensor learning scenario.
5.3.1 Single Sensor Learning. In this scenario, the assumption
is that a single sensor is placed on human body as a learner (e.g.
mobile phone on the waist). Our goal is to learn the sensor for a newcontext that is different compared to the available labeled data. We
assume that we have training data for all the subjects in the related
database except one subject. Therefore, the new context would bea new subject. The architecture actively retrains the learner for the
excluded subject in the reconfiguration phase. We use the label ofthe excluded subject for accuracy evaluation.

0.8

Accuracy (%)

Accuracy (%)

0.7
0.6
0.5
0.4

STL
RAL
CAL

0.3
0.24

6

8

10 12 14
Train Size (%)

16

18

20

0.90
0.85
0.80
0.75
0.70
0.65
0.60
0.55
0.504

(a) Chest sensor learning.

0.85
0.80

Accuracy (%)

0.9

6

8

10 12 14
Train Size (%)

16

STL
RAL
CAL

18

20

0.75
0.70
0.65
0.60
0.554

(b) Waist sensor learning.

68

10 12 14
Train Size (%)

16

STL
RAL
CAL

18

20

(c) Thigh sensor learning.

Figure 2: Single sensor learning scenario for different sensor locations.

0.9

Accuracy (%)

Accuracy (%)

0.8
0.7
0.6
0.5
0.4
0.30

20

40

CAL
CAL*
Upper Bound60
80 100
Number of Queries(a) No missing label.

120

140

1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.20

1.0
0.9
Accuracy (%)

1.0

20

CAL
CAL*
Upper Bound

40 60 80 100 120 140
Number of Queries(b) One missing label.

0.8
0.7
0.6

CAL
CAL*
Upper Bound

0.5
0.40

20

40 60 80 100 120 140
Number of Queries(c) Two missing label.

Figure 3: Multiple sensors (two sensors located on chest and waist) learning scenario.

After the learner initialization step, the query strategy submodule selects the top K = 10 informative instances in each queryround and the EMU labels the selected instances. We compare theresults of activity recognition accuracy with two learner models: 1)
randomly select samples in the query procedure instead of informative samples (RAL) and 2) employ a simple transfer learning-basedlearner (STL) which uses the most relevant training data of othersubjects. We run the experiment for all subjects and plot the average
accuracy for all three algorithms. The results on are summarized inFig. 2. The training data refers to the percentage of queries and thepercentage of used training data for CAL/RAL and STL algorithms
respectively.
Fig. 2 shows that using only 15% query, the accuracy of the
proposed collaborative active learning algorithm CAL goes up to88%. The accuracy of the STL algorithm fluctuates for most cases
after adding a certain amount of training data from other subjects.
After a point, the difference in distribution of training data and ourcontext result in an uncertain learner for the STL algorithm. The
RAL algorithm accuracy increases in the active learning processuntil a point that the added labeled data has no impact on thelearner accuracy. The comparison of accuracy is shown in Fig. 2
for three different learner locations (Chest, Waist, and Thigh). The
reported accuracy is the average for all subject in different runs.
5.3.2 Multiple Sensor Learning. Sometimes a user may needto integrate several sensors to the system for better performanceor context-related data collection. Furthermore, one may need toclassify more labels. We consider a learner with two sensor nodes
located on different body parts. Each node is already trained on asubset of labels with high confidence. The union of the two label
subsets may contain all or part of the label set L. Since in this casewe additionally use previous knowledge of the learners, we call theset up CAL*, to distinguish it from our standard CAL algorithm.
The way data is collected for CAL* requires a bit of clarification.
The subject will start using the system and the unlabeled data arelabeled by both sensors in the system. Those samples for which oneof the sensors is confident with their labeling decision are added
to the target labeled database (X L ). The rest of unlabeled data areadded to the target unlabeled data (XU ). The system also enhances
the target labeled data using Algorithm 2.
We compare the results of the activity recognition accuracy
obtained using CAL* with two other setups. (1) An active learning
procedure without the knowledge of sensor nodes and where thesystem is built solely on actively selected queries (CAL). (2) The
case in which the upper bound on accuracy is set using the samecontext training data (Upper Bound).
To compare the three setups, we consider three cases on theinitial overall knowledge of the two learners: 1) cover all labels, 2)
cover all labels except one, and 3) cover all labels except two labels.
Fig. 3 shows using available knowledge of learners increases theinitial activity recognition accuracy by up to 92%, 81%, and 88% for

1.0

50
00.0

Human Queries
All Queries
Learner Accuracy
0.1

0.5

0.2
0.3
Expert Uncertainty Threshold (%)

0.7

100

0.50.4

0.4

0.8

150

50
00.0

0.9

Human Queries
All Queries
Learner Accuracy(a) Chest sensor

0.1

0.2
0.3
Expert Uncertainty Threshold (%)

0.4

0.8

150

Accuracy (%)

0.6

1.0

200
Number of Queries

Accuracy (%)

0.7
100

200
Number of Queries

0.8

150

250
0.9

0.9

200
Number of Queries

250

Accuracy (%)

250

0.7
100

0.6

50

0.50.500.0

0.6

Human Queries
All Queries
Learner Accuracy
0.1(b) Upper arm sensor

0.5

0.2
0.3
Expert Uncertainty Threshold (%)

0.4

0.50.4(c) Waist sensor

Figure 4: Expert uncertainty threshold versus activity recognition accuracy and number of queries (blue axes: number ofqueries, red axes: accuracy).

E1
E2

700

E3
E4

E5
E6

E7
NCAL

CAL

E3
E4

E5
E7

E8
NCAL

CAL

400

500

Number of Queries

Number of Queries

600

400
300
200

300
200
100

100
0

E1
E2

500

5

10

Train Size (%)1520(a) The Waist and Ankle Sensor for Subject-1

0

5

10

Train Size (%)1520(b) The Waist and Ankle Sensor for Subject-9

Figure 5: Comparison of non-collaborative and collaborative active learning in terms of total number queries. Each color
corresponds to a different expert.

having knowledge on all labels, all labels except one label, and alllabels except two labels, respectively. It can be seen that deficiency
in knowledge on the label set raises the number of queries to gethigher performances.

5.4

Query Cost Evaluation

In this section, we present results assessing the effects of uncertainty
and the benefits of collaboration in the Co-MEAL architecture.
5.4.1 Effects of Uncertainty. Depending on how critical accurate
decision making is to the application, we can control the qualityof selected expert in the EMU unit. To show how uncertainty ofeach expert affects the number of queries in a real world application, we run the architecture with different uncertainty thresholds.
Fig. 4 shows the effects of uncertainty on (i) the number of queries
employed and (ii) the accuracy of activity recognition. Note thatthe plots in Fig. 4 use two vertical axes to show the two quantities simultaneously. In these experiments, we assumed that themost expensive data annotator is the subject, and we compared thenumber of human expert queries against all queries for different
uncertainty score thresholds.
The lower the uncertainty score threshold is, the higher thenumber of expensive queries. However, even choosing uncertain
experts, we still reach an acceptable data classification accuracy. We
repeated the experiment for three different locations. We used 20%of the data in active learning process. We see that the EMU reduces
the number of expensive queries by up to 40% while attaining an
accuracy of greater than 80% for uncertainty score threshold lower
than 0.2.
5.4.2 Benefits of Collaboration in Active Learning. As was shownin Table 2, we simulated a real world scenario using the available
sensors on different locations as well as the ground truth labels.
We defined a cost on each expert proportional to their confidence
on the label set L. The expert set is sorted in a descending order
based on their cost (E1 has the highest cost).
Fig. 5 shows the number of queries for both collaborative (CAL)
and non-collaborative (NCAL) active leaning. NCAL is simply avariant of Algorithm 1 in which the lines 9 and 10 corresponding
to collaboration between experts are deleted.
The training data size is the number of instance that are labeledby experts. As we can see, the total number of queries decreases
for CAL compared to NCAL due to the use of the active learning
process. As we go forward in time, experts’ confidence increases in
CAL, while the knowledge of experts remains the same for NCAL.
We also see that the number of queries from expensive experts
decreases for CAL. Therefore, the queries from inexpensive experts(e.g. E7, E8) increase, while using less queries from expensive
experts (e.g. E1, E2).

6

RELATED WORK

There is no existing work against which a direct comparison with
our approach can be made. But in a broader sense, reliable and reconfigurable design of health monitoring systems has been studied
in a number of related works.
The conventional approach is to design reliable machine learning algorithms with vast amount of training data. In [29], a setof heuristics is presented for activity recognition with sensors displaced within a single body segment. The authors show that withincertain limits and with modest quality degradation, activity recognition is tolerant to displacement. In [23], a location-aware activity
monitoring is proposed for the purpose of accurate activity detection. Zappi et. al. investigate the effects of exploiting redundantsensors distributed on the body to use a subset of sensors in thepresence of disorientation [28]. Allan et. al. investigate sensor
heterogeneity in activity recognition systems [30]. A clustering
based method is proposed to categorize different devices and thenactivity recognition is performed using a specific computationalmodel for each category.
A shortcoming of the conventional approach is that it provides
either limited degree of freedom (e.g. a certain amount of sensor
displacement) or requires very large amount of training data tocover every situations (e.g displaced data, misplaced data) [28, 30,
31].
Transfer learning is another approach that recently has beenused for system reconfiguration of health monitoring systems [32,33]. Kyle et. al. propose three heterogeneous transfer learningtechniques that reverse the typical transfer model and map thetarget feature space to the source feature space and apply them toactivity recognition in a smart apartment [34]. In [35], the authors
presente different algorithms for knowledge transfer in wearable
systems. It is shown that knowledge transfer from the source
domain to the target domain improves accuracy by 15%. In anotherwork, a multi-view learning approach is presented for wearables
with multiple sensors in which one of them acts as the source
domain [36]. It is shown that by using an additional wearable node
performance can be increased by up to 15%. Fallahzadeh et. al. [16]propose an uninformed cross-subject transfer learning algorithm
that leverages cross-user similarities by constructing a network fordata in source and target users and perform community detection
to extract the core observations in the target data.
While transfer learning approaches offer advantages compared
to conventional approaches, they still have several limitations: 1)
they typically rely on sending the entire sensors’ data to the backend station to perform the knowledge transfer, 2) they are applicablefor highly correlated contexts (e.g. similar user), and 3) even forsimilar contexts, knowledge transfer is inefficient for accurate dataprocessing. Therefore, a system solely designed based on related
knowledge can be inadequate for M-health monitoring systems.
Active learning approaches could help mitigate some of theselimitations, and they have been used extensively in the broader
literature of machine learning [37, 38]. However, active learning
is rarely used in the context of health monitoring systems [39].
Longstaff et. al. [40] investigate the applicability of active learning
for activity recognition. Active learning is compared with several
semi-supervised learning methods including self-learning and colearning. It is shown that active learning improves the system’s
accuracy the most. In [41], several active learning methods aredesigned and implemented for their applicability to the activity
recognition task. The authors evaluated the methods using the
CASAS smart home sensor data and present a crowd-sourcing
application for annotation. Another active learning for activity
recognition systems in single person smart homes is presented in[42]. This approach exploits clustering approaches to categorize inhome activities. Users only need to label each cluster as an activity
as opposed to labeling all instances of all activities.

7

CONCLUSIONS

We explored the use of active learning for system reconfigurationof mobile sensory devices, focusing on mobile health monitoring.
We targeted a design process that simultaneously addresses several
challenges and scenarios: 1) The cost of data labeling throughthe learning process may be expensive. This cost may relate tointerference in user’s life, feedbacks from health-care experts, or
equipment cost. 2) The suitable expert for each data instance may be
unknown. And 3) the labels may be subject to uncertainty becauseof limited knowledge of experts.
To help address these challenges, we proposed a new architecturethat manages a multi-expert M-health system. The architecture
finds a cost-optimal and most-confident expert each time a queryneeds to be issued. At the same time, we considered collaboration
among the experts in order to gain more accurate data labelingwhile minimizing the cost. We proposed new algorithms for the
purpose of expert selection as well as system initialization for Mhealth systems. As part of the algorithms, we employed available
clustering algorithms as well as ensemble classification methods.
We demonstrated the efficacy of the architecture and the developed algorithms using activity monitoring as a case study. Specifically, we considered accelerometer-based human activity recognition dataset. We showed that an accuracy of activity recognition
as high as 85% can be achieved by labeling only 15% of unlabeled
data. We also showed that annotation cost is substantially reduced
because of cost-optimal expert selection as well as collaboration
between experts in the proposed Co-MEAL architecture.
In future work, we plan to enhance the architecture to handle
multi-modality sensory systems. We also plan to use the system ina real pilot study of an intervention system. In a separate future
direction, we plan to integrate transfer learning and active learning
approaches in order to further reduce the number of queries andincrease learner’s accuracy.

ACKNOWLEDGMENTS
This work was supported by a Research Fellowship from the Community Health Analytics Initiative (CHAI) at Washington State
University and by the National Science Foundation CAREER Award
IIS 1553528.

REFERENCES[1] Misha Kay, Jonathan Santos, and Marina Takane. mHealth: New horizons forhealth through mobile technologies. World Health Organization, 3:66–71, 2011.
[2] Abu Saleh Mohammad Mosa, Illhoi Yoo, and Lincoln Sheets. A systematic review
of healthcare applications for smartphones. BMC medical informatics and decision
making, 12(1):67, 2012.
[3] Zachary W Adams, Erin A McClure, Kevin M Gray, Carla Kmett Danielson,
Frank A Treiber, and Kenneth J Ruggiero. Mobile devices for the remote acquisition of physiological and behavioral biomarkers in psychiatric clinical research.
Journal of psychiatric research, 85:1–14, 2017.
[4] Caroline Free, Gemma Phillips, Louise Watson, Leandro Galli, Lambert Felix, Phil
Edwards, Vikram Patel, and Andy Haines. The effectiveness of mobile-healthtechnologies to improve health care service delivery processes: a systematic
review and meta-analysis. PLoS Med, 10(1):e1001363, 2013.
[5] Daniel F Hayes, Hugh S Markus, R David Leslie, and Eric J Topol. Personalized
medicine: risk prediction, targeted therapies and mobile health technology. BMC
Medicine, 12(1):37, 2014.
[6] Armin Shahrokni, Sanam Mahmoudzadeh, Ramyar Saeedi, and Hassan
Ghasemzadeh. Older people with access to hand-held devices: Who are they?
Telemedicine and e-Health, 21(7):550–556, 2015.
[7] Emanuela Marcelli, Alessandro Capucci, Gabriele Minardi, and Laura Cercenelli.
Multi-sense cardiopatch: A wearable patch for remote monitoring of electromechanical cardiac activity. Asaio Journal, 63(1):73–79, 2017.
[8] Philip E Eggers, Eric A Eggers, and Benjamin Z Bailey. Wearable apparatus, system and method for detection of cardiac arrest and alerting emergency response,
June 23 2016. US Patent 20,160,174,857.
[9] Javad Birjandtalab, Maziyar Baran Pouyan, Diana Cogan, Mehrdad Nourani,
and Jay Harvey. Automated seizure detection using limited-channel eeg andnon-linear dimension reduction. Computers in Biology and Medicine, 82:49–58,
2017.
[10] David Zakim and Matthias Schwab. Data collection as a barrier to personalized
medicine. Trends in pharmacological sciences, 36(2):68–71, 2015.
[11] Ramin Fallahzadeh, Mahdi Pedram, Ramyar Saeedi, Bahman Sadeghil, Michael
Ongl, and Hassan Ghasemzadeh. Smart-cuff: A wearable bio-sensing platform
with activity-sensitive information quality assessment for monitoring ankle
edema. 2015.
[12] Pio Alfredo Di Tore. Situation awareness and complexity: the role of wearable technologies in sports science. Journal of Human Sport and Exercise,10(1proc):500–506, 2015.
[13] Rachel Lomasky, Carla E Brodley, Matthew Aernecke, David Walt, and MarkFriedl. Active class selection. In European Conference on Machine Learning, pages
640–647. Springer, 2007.
[14] Burr Settles, Mark Craven, and Lewis Friedland. Active learning with real
annotation costs. In Proceedings of the NIPS workshop on cost-sensitive learning,
pages 1–10, 2008.
[15] Hesam Sagha, José del R Millán, and Ricardo Chavarriaga. A probabilistic
approach to handle missing data for multi-sensory activity recognition. In Workshop on Context Awareness and Information Processing in Opportunistic Ubiquitous
Systems at 12th ACM International Conference on Ubiquitous Computing, number
EPFL-CONF-151974, 2010.
[16] Ramin Fallahzadeh and Hassan Ghasemzadeh. Personalization without userinterruption: Boosting activity recognition in new subjects using unlabeleddata (an uninformed cross-subject transfer learning algorithm). In International
Conference on Cyber-Physical Systems (ICCPS). ACM/IEEE, 2017.
[17] Vincent Wenchen Zheng, Derek Hao Hu, and Qiang Yang. Cross-domain activity
recognition. In Proceedings of the 11th international conference on Ubiquitous
computing, pages 61–70. ACM, 2009.
[18] Burr Settles. Active learning literature survey. Computer Science Technical Report
1648, University of Wisconsin, Madison, (1-67):67, 2010.
[19] Wei Chu, Martin Zinkevich, Lihong Li, Achint Thomas, and Belle Tseng. Unbiased
online active learning in data streams. In Proceedings of the 17th ACM SIGKDD
international conference on Knowledge discovery and data mining, pages 195–203.
ACM, 2011.
[20] Bo Wang and John Tsotsos. Dynamic label propagation for semi-supervised
multi-class multi-label classification. Pattern Recognition, 52:75–84, 2016.

[21] Adrian Calma, Jan Marco Leimeister, Paul Lukowicz, Sarah Oeste-Reiß, Tobias
Reitmaier, Albrecht Schmidt, Bernhard Sick, Gerd Stumme, and Katharina Anna
Zweig. From active learning to dedicated collaborative interactive learning. In
ARCS 2016, pages 1–8. VDE, 2016.
[22] Zelun Zhang and Stefan Poslad. Improved use of foot force sensors and mobile
phone gps for mobility activity recognition. IEEE Sensors Journal, 14(12):4340–
4347, 2014.
[23] Ramyar Saeedi, Navid Amini, and Hassan Ghasemzadeh. Patient-centric on-bodysensor localization in smart health systems. In Signals, Systems and Computers,
2014 48th Asilomar Conference on, pages 2081–2085. IEEE, 2014.
[24] Timo Sztyler and Heiner Stuckenschmidt. On-body localization of wearable
devices: an investigation of position-aware activity recognition. In Pervasive
Computing and Communications (PerCom), 2016 IEEE International Conferenceon, pages 1–9. IEEE, 2016.
[25] H. Ghasemzadeh, E. Guenterberg, and R. Jafari. Energy-Efficient InformationDriven Coverage for Physical Movement Monitoring in Body Sensor Networks.
IEEE Journal on Selected Areas in Communications, 27:58–69, 2009.
[26] Hassan Ghasemezadeh, Sarah Ostadabbas, Eric Guenterberg, and Alexandros
Pantelopoulos. Wireless medical embedded systems: A review of signal processing techniques for classification. IEEE Sensors Journal, 2012, In Press.
[27] Yufei Chen and Chao Shen. Performance analysis of smartphone-sensor behavior
for human activity recognition. IEEE Access, 5:3095–3110, 2017.
[28] Piero Zappi, Thomas Stiefmeier, Elisabetta Farella, Daniel Roggen, Luca Benini,and Gerhard Troster. Activity recognition from on-body sensors by classifier
fusion: sensor scalability and robustness. In Intelligent Sensors, Sensor Networks
and Information, 2007. ISSNIP 2007. 3rd International Conference on, pages 281–286.
IEEE, 2007.
[29] Kai Kunze and Paul Lukowicz. Using acceleration signatures from everyday
activities for on-body device location. In Wearable Computers, 2007 11th IEEE
International Symposium on, pages 115–116. IEEE, 2007.
[30] Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow,
Mikkel Baun Kjærgaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen.
Smart devices are different: Assessing and mitigating mobile sensing heterogeneities for activity recognition. In Proceedings of the 13th ACM Conference on
Embedded Networked Sensor Systems, pages 127–140. ACM, 2015.
[31] Ramyar Saeedi, Ramin Fallahzadeh, Parastoo Alinia, and Hassan Ghasemzadeh.
An energy-efficient computational model for uncertainty management in dynamically changing networked wearables. In International Symposium on Low
Power Electronics and Design (ISLPED). IEEE/ACM, 2016.
[32] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions
on knowledge and data engineering, 22(10):1345–1359, 2010.
[33] Sandeepkumar Satpal and Sunita Sarawagi. Domain adaptation of conditional
probability models via feature subsetting. In European Conference on Principles
of Data Mining and Knowledge Discovery, pages 224–235. Springer, 2007.
[34] Kyle Dillon Feuz and Diane J. Cook. Heterogeneous transfer learning for activity
recognition using heuristic search techniques. International Journal of Pervasive
Computing and Communications, 10(4):393–418, 2014.
[35] Ramyar Saeedi, Hassan Ghasemzadeh, and Assefaw H. Gebremedhin. Transfer
learning algorithms for autonomous reconfiguration of wearable systems. In
IEEE International Conference on Big Data. IEEE, 2016.
[36] Seyed Ali Rokni and Hassan Ghasemzadeh. Plug-n-learn: automatic learning ofcomputational algorithms in human-centered internet-of-things applications. In
Proceedings of the 53rd Annual Design Automation Conference, page 139. ACM,
2016.
[37] Akshay Gadde, Aamir Anis, and Antonio Ortega. Active semi-supervised learning using sampling theory for graph signals. In Proceedings of the 20th ACM
SIGKDD international conference on Knowledge discovery and data mining, pages
492–501. ACM, 2014.
[38] Yi Yang, Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann.
Multi-class active learning by uncertainty sampling with diversity maximization.
International Journal of Computer Vision, 113(2):113–127, 2015.
[39] Andreas Holzinger. Interactive machine learning for health informatics: when
do we need the human-in-the-loop? Brain Informatics, 3(2):119–131, 2016.
[40] Brent Longstaff, Sasank Reddy, and Deborah Estrin. Improving activity classification for health applications on mobile devices using active and semi-supervised
learning. In Pervasive Computing Technologies for Healthcare (PervasiveHealth),
2010 4th International Conference on-NO PERMISSIONS, pages 1–7. IEEE, 2010.
[41] Salikh Bagaveyev and Diane J Cook. Designing and evaluating active learning
methods for activity recognition. In Proceedings of the 2014 ACM International
Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication,
pages 469–478. ACM, 2014.
[42] Enamul Hoque and John Stankovic. AALO: Activity recognition in smart homes
using active learning in the presence of overlapped activities. In Pervasive
Computing Technologies for Healthcare (PervasiveHealth), 2012 6th International
Conference on, pages 139–146. IEEE, 2012.