UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Towards A Unified System for
Multimodal Activity Spotting:
Challenges and A Proposal
Long-Van Nguyen-Dinh
Wearable Computing Lab
ETH Zürich
Zürich, Switzerland
longvan@ife.ee.ethz.ch

Gerhard Tröster
Wearable Computing Lab
ETH Zürich
Zürich, Switzerland
troester@ife.ee.ethz.ch

Alberto Calatroni
Wearable Computing Lab
ETH Zürich
Zürich, Switzerland
alberto.calatroni@ife.ee.ethz.ch

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for componentsof this work owned by others than the author(s) must be honored. Abstractingwith credit is permitted. To copy otherwise, or republish, to post on servers or
to redistribute to lists, requires prior specific permission and/or a fee. Request
permissions from Permissions@acm.org.
UbiComp ’14 , September 13 - 17 2014, Seattle, WA, USA
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-3047-3/14/09$15.00.
http://dx.doi.org/10.1145/2638728.2641301

807

Abstract
In the existing multimodal systems for activity
recognition, there is no single method to process different
sensor modalities at different on-body positions.
Moreover, sensor types are often selected and optimized
so as to accord with the goal of application. The
complexity makes those systems infeasible to be deployed
for new settings. This paper proposes a unified system
which works with any available wearable sensors placed onuser’s body to spot activities. Each data stream is treateduniformly through our proposed template matching
WarpingLCSS to spot activities. With the uniformity inextracting activity-specific patterns from raw sensor
signals, our proposed system is compatible with respect tomodalities and body-worn positions.
We evaluate our system on the Opportunity dataset offour subjects consisting of 17 hard-to-classify classes (e.g.,
open/close drawers at different heights) with 17 sensors
belonging to three modalities (accelerometer, gyroscopeand magnetic field) attached at different on-body
positions. The system achieves good performances (63%to 84% in F1 score). Moreover, the robustness andefficiency to addition and removal of sensors as well asactivity classes are also investigated.

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Author Keywords

since each modality allows to recognize some restrictedsets of activities, system designers still tend to solve
specific activity recognition problems with specific sensors.

Continuous Activity Recognition; Multi-modality;
Multi-sensor; Body-worn Sensors; WarpingLCSSACM Classification Keywords
H.1.2 [User/Machine Systems]; I.5.2 [Pattern Recognition]

Introduction
Continuous activity recognition (activity spotting ) is acore component in context-aware systems. It enables avariety of applications such as ambient assisted living,
human computer interaction. In activity spotting, actionsof interest and their temporal boundaries are detected in acontinuous data stream in which they are randomly mixed
with arbitrary non-interest actions (null class).
In the past few years, promising results from body-worn
sensors for activity spotting have been presented [1, 13].
Many modalities, such as motion-related ones
(acceleration, rate of turn, magnetic field) [1, 5],temperature [6] or sound [7] have been explored as inputs
to activity recognition systems. Nevertheless, with theincreasing availability of commercial wearable sensor
devices (such as smartphones, watches, glasses and, in anear future, sensor-equipped garments), the multimodal
aspect is ready for being fully exploited.
Why are there no multimodal activity spotting systems
readily deployed in commercial applications? One
challenge is that recognition chains for different sensor
modalities need still quite some hand-crafting for being
deployed. Features and classifiers used for accelerometerdata differ substantially from the ones needed forgyroscope or compass data, or audio. Even for the same
modality, the needed features can differ for sensors
mounted on different parts of the body. Furthermore,

808

Other challenges of a unified multimodal framework thatmake it easily deployed in any settings are the following:• If new sensors are worn by the user (e.g., the user wears
a new smart watch), these should be integrated into thesystem smoothly, without asking the user where on the
body the devices have been mounted or which classifiershould be used for new data from those sensors.
• The system should handle missing sensors in run-time(e.g., the user gets off his sensor-equipped shoes)
without interfering with other sensors.
• The system should also adapt new activity classes ofinterest smoothly without retraining the whole system.
In this paper we make one step towards a unified
framework for multimodal activity recognition whichattempts to overcome the challenges addressed above.
Our system treats different modalities, and sensors withthe same modalities at different on-body placements in a
homogeneous way. Specifically, each data stream is firstquantized into strings of symbols by using k-means (hereserving as a vector quantization step) and then substring
matching is performed by the fast and efficient templatematching method WarpingLCSS [8] to spot activities. Our
system recognizes activities by extracting activity-specificpatterns from raw sensor signals, hence it is agnostic withrespect to modalities and on-body positions. Thus, it can
combine any available wearable sensors placed on user’s
body to spot activities. Besides that, our system takes thebenefit of template matching methods in which different

WORKSHOP: HASCA

classes can be trained and spotted separately so that it
can handle new activity classes easily.
We investigate two multimodal frameworks to fusedifferent data sources either at the signal level (signal
fusion) or at a decision level (classifier fusion). In theclassifier fusion framework, a novel fusion technique fortemplate matching is proposed to combine all spotting
results from different sensors. The two frameworks will becompared throughout the paper in terms of recognition
performance, speed, ease to add or remove sensors, andease to add or remove activity classes.
In this work, we test the proposed frameworks with therecognition of hand actions (i.e., gestures), but theframeworks apply with no loss of generality to otheractivities. The proposed system is evaluated with thecomplex Opportunity dataset [10], which includes 17 handactions using 17sensors belonging to three modalities(accelerometer, gyroscope and magnetic field) at different
on-body positions. The performances of all subsets ofsensors in the classifier fusion framework are also given to
demonstrate its flexibility to sensor addition and removal.

Related Work
Various techniques for online gesture recognition can befound in literature; they include Hidden Markov Models(HMM) [12], support vector machines (SVMs) [14],template matching methods (TMM) using dynamic time
warping (DTW) [13] or using longest common
subsequence (LCSS) [8].
With recent advances in the development of inexpensive
wearable sensors, researchers have investigated activity
recognition systems using multimodal sensors or multiple
single-modal sensors to improve the performance. In [1],five accelerometers were attached at different on-body809

positions to recognize physical activities. [9] used motion
sensors and force sensing resistors to recognize handactions for quality inspection in car production. The
fusion of multiple data sources can be performed either
early at signal level, feature level [6], or late at decisionlevel (i.e., classifier fusion) [1, 9].
For each modality, a wide range of features and supervised
learning techniques for activity recognition has been
explored [4]. As one example, accelerometer data can beclassified with Naive Bayes [1], SVMs [14], C4.5 decision
trees [1], HMM [12], TMM [8] and a variety of features inboth time and frequency domains can be extracted [4].
Due to the diversity of methods and features, the existing
multimodal systems selected different methods andfeatures for different modalities or sensors mounted at
different on-body positions [9, 13]. For example, in theapplication of car quality inspection [9], inertial sensors
must be attached at arms and torso in order to acquire
the trajectories of wrists and elbows, force sensing
resistors attached at lower arms to monitor muscle. Fourdifferent methods including K-Nearest-Neighbor (KNN),
TMM, k-means classifier, and Bayes classification wereused in that work. They also trained different classifiers
linked to the different modalities with different set oflabels for the best possible performance purpose.
WarpingLCSS was first presented in our previous work [8]as a fast and efficient method to spot gestures using one
3D accelerometer on arm. The method showed robustness
against noisy annotations and high variances in activity
execution. In the work by Chen et al. [2], they
investigated WarpingLCSS with multi-sensor fusion
combining 6 different accelerometers at wrists and arms.
However, the performance was not improved. According
to the best of our knowledge, there is no previous work

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

representative points which can capture body movements
at sensor-attached positions regarding to specificactivities.

Multi-modality System

Training

In our system, we use the recently proposed templatematching WarpingLCSS [8] as a core module for dataprocessing, training and activity spotting. Data isrecorded continuously and synchronously from multiple
sensors. The training data is manually labeled with a listof activity classes of interest. Activities which are not in
the list are considered as null class.

- Rejection Threshold (RT)

Template Matching
The TM module processes input data, generates
templates for activity classes in the training phase andrecognizes activities in the spotting phase. An overview of
the TM module is shown in Figure 1.
First, the TM module applies k-means to all training data
points and quantizes the signal input to their closest
cluster centroids. Thus, signal data are then represented
as a string of symbols (i.e., the indices of the centroids).
The number of symbols k depends on the variation of the
input signal. Accordingly, the cluster centroids are810

Quantize tocentroids

Weight (X, class)RT- Weight (X, class)
Templates

centroids

KmeansX

We propose two frameworks for fusing multimodal sensors
at two different processing levels: classifier fusion andsignal fusion. In the classifier fusion framework, signal
data from each sensor are processed separately by atemplate matching (TM) module and then the spotting
outputs from all sensors will be fused to have a final
recognition output. Different TM sessions can be run inparallel. In contrast, in the signal fusion framework,
signals from all sensor modalities are combined into one
data stream before being fed into the TM module.
Figures 2 and 4 gives an overview of the two frameworks.

- Select Templates

Symbol stringthat investigates the use of WarpingLCSS towards a
unified multimodal system, especially with othermodalities such as gyroscopes, magnetic sensors.

WarpingLCSS
Symbol string

Spotted Activities[ts,te,label,score]

Spotting

Figure 1: Template Matching Module. X is data input fromone or multiple combined sensors.

In the training process, one or more templates are createdfor each activity of interest to represent the typical
patterns for that class. The templates are chosen as
instances that have the highest average longest common
subsequence (LCSS) scores [3] to all other instances ofthe same class. Additionally, a rejection threshold needs
to be calculated for each activity class in the training
phase to be able to reject signals not belonging to thatclass upon recognition. Let µ(c) and and σ (c) be the mean
and the standard deviation, respectively, of LCSS values
between the template of a class c and any string
belonging to the same class. We calculate the rejection
threshold to be below µ(c) by some standard deviations:µ(c) − h(c) ∗ σ (c) , with h(c) = 0,1,2,.... The value of h(c)is determined by testing the recognition of class c on the
training data and selected as the one which yields the best
F1 score performance. Specifically,
F 1c = 2 ∗precisionc ∗ recallc,
precisionc + recallc(1)

WORKSHOP: HASCA

where precisionc is the proportion of samples of class cpredicted correctly over the total samples predicted asclass c; recallc is the proportion of samples of class cpredicted correctly over the total samples of class c. Notethat the value of F 1c can also be used to indicate how
well sensor data fed into the TM module can recognize
the specific class c.
When spotting, the same process of quantization isapplied to the streaming sensor data, with the cluster
centroids identified during training. Then, for each activity
class, the WarpingLCSS method [8] is used to match thetemplate within the online string to spot activities
belonging to that class. Given the activity template forclass c, s̄(c) , the WarpingLCSS score W(s̄(c) ,s) (i, j)between the first i symbols of the template s̄(c) and thefirst j symbols of the string s is obtained as follows:0, if i = 0 or j = 0








W(s̄(c) ,s) (i − 1, j − 1) + 1, if s̄(c) (i) = s(j)W(s̄(c) ,s) (i, j) =
 W(s̄(c) ,s) (i, j) − p ∗ d(s̄(c) (i), s(j))



max

W (c) (i − 1, j) − p ∗ d(s̄(c) (i), s̄(c) (i − 1)) (s̄ ,s)W(s̄(c) ,s) (i, j − 1) − p ∗ d(s(j), s(j − 1)), otherwise,

where p is a penalty parameter of the dissimilarity andd(·, ·) is the normalized Euclidean distance between two
symbols (i.e., two corresponding centroids) in a range[0,1] . When a new symbol arrives, the WarpingLCSS
processes and updates the score immediately. Hence thecomputational cost of WarpingLCSS is low for online
recognition. The WarpingLCSS score grows (i.e., symbols
are matched) when an instance of the examined class isperformed and drops significantly if other classes are
performed due to the penalty terms. Additionally, thepenalty is accumulated in a time-warping manner as in
dynamic time warping (DTW) [11], hence WarpingLCSS

811

penalizes the same consecutive symbols which aremismatched only once. An activity of class c is recognizedfor each local maximum of W that exceeds the rejection
threshold.
Outputs of the TM module are spotted activities with aformat [start-time, end-time, label, simScore] to indicate
when the activity occurs and the similarity score(simScore) between the activity and the template of thatclass. In the TM module, the spotting of different activity
classes can be processed concurrently in parallel. A moredetailed explanation, complexity analysis and illustrationof WarpingLCSS can be found in [8].
Classifier Fusion Framework
In the classifier fusion framework, each sensor is treateduniformly via the same process in the Template Matchingmodule. The spotting outputs from all sensors are
combined in the Classifier Fusion module as shown in
Figure 2. Then the Decision Making (DM) module
resolves conflicts for spotted instances belonging tomultiple classes and output the results.
Let Φ and |Φ| be the set of sensors and the number of
sensors in the system, respectively. We represent the
spotting output from a sensor S ∈ Φ in a spotting matrix
M(S) of size C ∗ N , with C is the number of activity
classes of interest and N is the number of samples
processed. M(S)(c, i) represents the entry at the ith
sample and the row of class c in the matrix M(S). Eachrow c in the matrix, indicated as M(S)(c) stores theinformation of spotted instances of an activity class c fromthe sensor S. Specifically, if the sensor outputs an activity
instance of class c from start-time to end-time with asimilarity score simScore (i.e., [start-time, end-time, c,
simScore]), then M(S)(c, i) = simScore for all i-th
samples in the interval from start-time to end-time at the

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

row c. Figure 3 gives an example of the spotting matrix.
Spotted from S1

Template

S1

Matching

Weight (S1,class)….

….

Decision

FusionMakingt2
P

Spotted from Sn

Template
Matching

Sn

Classifier

Given the fused spotting matrix M̄, for each spotted
activity [t1, t2, c, simScore], the similarity score simScore
is updated as the average score in the interval from thetime t1 to the time t2 at the row c in M̄. Specifically, theupdated simScore is computed as follows.
¯
simScore=

Weight (Sn,class)(t2 − t1) [samples].

(3)

Consequently, the similarity score of an activity isboosted if more sensors predict that activity performed.

Figure 2: Classifier fusion framework
samples

Activity classM̄(c, i)

i=t1

10

…

1

2

3

4

5

6

7

8

9

drink

0

0.8

0.8

0.8

0

0

0

0

0

0

…

open_door

0

0

0

0

0

0

0.6

0.6

0.6

0

…

close_door

0

0

0

0

0

0

0

0

0

0

…

Figure 3: An example of the spotting matrix with three
activity classes (drink, open door and close door) and two
spotted activities: [2, 4, drink, 0.8] and [7, 9, open door, 0.6].

Signal Fusion Framework
In the signal fusion framework, the Signal Fusion module
combines signals from all sensors into one data stream asshown in Figure 4.
S1
S2

Signal….

Classifier Fusion We propose Weighted Fusion method
to fuse the spotting results from all sensors. Let
Weight(S,c) be a prior weight to indicate how well sensor
S can recognize the specific class c. We set Weight(S,c)
as the best F 1c performance (see Equation 1) whenselecting the rejection threshold for activity class c in theprocessing of sensor s. The weighted summed spotting
matrix is computed as follows.

Decision Making If an activity is spotted as belonging
to multiple classes (i.e., boundaries of spotted instances
are overlapping), the DM module will resolve conflicts bydeciding the class with highest similarity score as the best
match. If an activity is classified into only one class, theDM will output the class. Otherwise, if no activity class isspotted, the DM will output null.

FusionTemplate

Spotted Activities

Matching

Decision
MakingSn

M̄(c) =|Φ|X

W eight(Si , c) ∗ M(S i )(c) ∀c.

(2)

i=1
Si ∈Φ

The similarity score of an activity in the spotting matrix
degrades if the prior performance of the sensor torecognize the corresponding activity class is low.

812

Figure 4: Signal fusion framework

Let di be the dimension of signal data generated fromsensor Si ∈ Φ. The combined data stream from the Signal|Φ|P
Fusion module has a dimension of
di . The TM
i=1,Si ∈ΦWORKSHOP: HASCA

module then processes data and outputs spotted activities.
Finally, the DM module handles spotting conflicts andoutputs recognized activities as discussed above.

Experiments= accelerometer, gyro,
magnetic= accelerometer
RUA_A, RUA_G, RUA_M
LUA_A, LUA_G, LUA_M

RLA_A, RLA_G, RLA_M
LLA_A, LLA_G, LLA_M

BACK_A, BACK_G, BACK_M

RSHO_ALSHO_A

Figure 5: Sensors attached at
different places on body and theirmodalities in Opportunity
dataset.

We present the activity dataset, evaluation metrics andthe conducted experiments to evaluate the proposedsystem in this section.
DatasetWe evaluate the system on the Opportunity dataset 1 [10]which is a rich multimodal multi-sensor dataset collected
in a naturalistic environment akin to an apartment, whereusers execute 17 daily gestures. The dataset contains alarge variability in the execution of the activities and nullclass is predominant (37%). We use the subset ofrecording corresponding to four subjects in which eachsubject wears 17 sensors belonging to three modalities(3D accelerometer, 3D gyroscope and 3D magnetic field)
attached at different on-body positions. Each subject
performs 20-40 repetitions of each gesture class. Totally,
the dataset contains 1485 activity instances. Table 1
shows the list of activity classes in the Opportunity
dataset. Note that there are three drawers located at
different heights and two different doors in the dataset.
Figure 5 shows locations of sensors on body (i.e., rightupper arm (RUA), right lower arm (RLA), left upper arm(LUA), left lower arm (LLA), back (BACK), right shoe
(RSHO) and left shoe (LSHO)) and their modalities. The
signals of all sensors are recorded at a frequency of 30Hz.
Nullclose Drawer 1-2-3 (CDr1-2-3)
open Fridge (OF)open Dishwasher (ODi)clean Table (CT)open Door 1-2 (OD1-2)
close Fridge (CF)close Dishwasher (CDi)open Drawer 1-2-3 (ODr1-2-3)
close Door 1-2 (CD1-2)
drink Cup (D)Toggle Switch (TS)

Table 1: Activities in Opportunity dataset.
1 http://www.opportunity-project.eu/challengeDownload

813

Evaluation Metrics
Generally, activity classes may occur non-uniformly inreal-life datasets. In the Opportunity dataset, null class ispredominant (37%). Therefore, we use the weighted
average sample-based F1 score to assess the performanceof activity recognition. It is computed as the sum of theF1 scores of all classes, each weighted according to the
proportion of samples in that particular class. Specifically,XF1 =wc ∗ F 1c ,cwhere c is the class index, wc is the proportion of samples
of class c, and F 1c is computed as in Equation 1.
We present two ways of computing the F1 score, either
including (F1-Null) or excluding the null class(F1-NoNull). F1-NoNull does not consider the null class,but still takes into account false predictions of gesturesamples or instances misclassified as null class or viceversa. F1-NoNull value represents how well therecognition system detects activity classes of interest. The
recognition system that has high values of both F1-Nulland F1-NoNull predicts well both activities and null class.
Experiments on Multimodal System
For each subject, we perform experiments in 5-fold crossvalidation. All raw signals (30Hz sampling rate) are
down-sampled for a faster computation. Specifically, anaverage value of each sliding window of size 6 samples andoverlap 3 is extracted to represent the corresponding setof data points in the window. In our experiments, the TM
module generates only one template for each activity class.
In the classifier fusion framework, the number of symbols(i.e., number of clusters in k-means) is selected
empirically k = 20 for each 3D sensor. Note that k can beselected by using cross-validation on the training data.

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Subject 1
0.8

F1−Null

0.7
0.6
0.5

Results and Discussion

0.4
0.3
1 2 3 4 5 6 7 8 9 1011121314151617
Number of sensors

0.8
0.7
0.6
F1−NoNullIn the signal fusion framework, the Signal Fusion module
combines all 17 sensors into a data stream with a highdimension of 51. Consequently, the number of symbols is
selected much higher to capture variants in the combined
movements at seven on-body positions (see Figure 5). We
select empirically k = 200.

0.5
0.4
0.3
0.2
0.1
1 2 3 4 5 6 7 8 9 1011121314151617
Number of sensors

Figure 6: Performance ofclassifier fusion framework withall subset combinations of 17
sensors for subject 1. The red
middle line shows the average
performance.

Performance of One Sensor
The performance of a sensor reflects how well that sensor
recognizes the activities. Figure 6 shows the performanceof each sensor (i.e., number of sensors = 1) and their
combinations in the classifier fusion framework. Due to a
space limit, we report only the result from subject 1,
however, those performances for other subjects have asimilar trend. As seen in Figure 6, the performances ofdifferent sensors vary significantly. The sensors on shoes(LSHO A and RSHO A) give the worst performance since
their signals are not distinguishable for different gesture
executions (e.g., open doors and open drawers have thesimilar patterns of foot movements). The accelerometer atlower dominant arm (RLA A) gives the best performancefor subject 1. Meanwhile, the magnetic sensors at lower
dominant arm give the best results for subjects 2-4.
Comparison between Two Frameworks
Table 2 show the results on the use of all 17 sensors in thetwo proposed fusion frameworks. They both achieve agood performance for the four subjects (63% to 84%
F1-Null). In average, the performance of the classifier
fusion framework on 17 sensors increases by 16% F1-Nulland 21% F1-NoNull compared with the average
performance of one sensor. It also increases by 4%
F1-Null and 15% F1-NoNull compared against the
average performance of the best one-sensor.

814

The signal fusion outperforms the classifier fusion about
7% F1-Null and only 1% F1-NoNull in average. It means
the signal fusion can detect the null class better than theclassifier fusion. The rationale is that the signal fusion has
a global view of data from all sensors at once before
processing; meanwhile the classifier fusion framework has
only a local view of data from each sensor. The handactions of concern and the null class may have the similarfoot movements (e.g., walking, standing). Hence, datafrom the shoe sensor may detect the activities when thenull class actually occurs. Even the other sensors candetect the null instance, the classifier fusion still outputs
the false detected activities. By contrast, the signal fusion
framework outputs an activity only when the combined
pattern of that activity from different sensors is matched.
Besides the recognition accuracy, we compare theadvantages and limitations of the two frameworks withregards to speed, ease to remove or add sensors to thesystem, and ease to add or remove activity classes. They
are summarized Table 3.
Parallelism at sensor
Parallelism at activity class
Sensor Addition or RemovalClass Addition or RemovalClassifier fusion
* Yes
* Yes
* Easy* EasySignal fusion* Do not care
* Yes
* Hard* EasyTable 3: Comparison between classifier fusion framework andsignal fusion framework.

The running time of our proposed system depends on how
many sessions of the TM module run to spot activities. In
the signal fusion framework, the number of TM session is
only one. In the classifier fusion framework, it equivalents
to the number of sensors deployed in the system.
However, those TM sessions for different sensors can be
executed in parallel. Therefore, if the parallelism is

WORKSHOP: HASCA

MethodClassifier Fusion
Signal Fusion

Subject 1
F1-Null F1-NoNull
0.74
0.79
0.77
0.77

Subject 2
F1-Null F1-NoNull
0.63
0.67
0.67
0.68

Subject 3
F1-Null F1-NoNull
0.75
0.80
0.84
0.81

Subject 4
F1-Null F1-NoNull
0.65
0.71
0.74
0.73

Average
F1-Null F1-NoNull
0.69
0.74
0.76
0.75

Table 2: Performance of two frameworks on 17 sensors in the Opportunity dataset.

maximized, the running time of the two frameworks tospot activities is equivalent (i.e., time to run the TM
module once).
Each sensor is processed separately in a uniform way inthe classifier fusion framework. Therefore, it enables easy
addition and removal of sensors without interfering withthe use of other sensors to spot activities. Meanwhile, inthe signal fusion framework, the combination of all sensor
signals into one data stream before being processed will
require the same sensor settings in the training andspotting phases so that the quantization step can proceed
properly. Hence, adding or removing sensors in the signal
fusion framework requires retraining the whole system.
The TM module spots each activity class separately.
Therefore, the spotting for different activity classes can be
executed independently in parallel. Hence, our proposed
recognition system with the core Template Matching isvery flexible in adding or removing activity classes.
Sensor Combinations in Classifier Fusion Framework
We show the performances of the classifier fusion
framework on all subset combinations of 17 sensors in
Figure 6. The results show that the performances amonggroups of sensors differ less when the number of sensors
increase. This indicates that adding a better performance
sensor into a group increases the average performance.
The average performance increases significantly in both
F1-Null and F1-NoNull as the number of sensors increases
from 1 to 6 and then keeps stable. Generally, the815

combination of more sensors does not always yield thebetter performance (e.g., two accelerometers at lower arm
and upper arm may not improve the detection of OD1
and OD2). The best-combination performance increases
dramatically as the group size increases from 1 to 4.
F1-NoNull is almost unchanged after reaching the best
performance. Meanwhile, the F1-Null of 17 sensors is lessthan the best performance by 10% in average. As
discussed above, the presence of not-so-distinguishable
sensors (e.g., shoe sensors) in the classifier fusion makes
the recognition more confused in detecting the null class.
Table 4 gives the subset combination of 17 sensors thatgets the best result. The orientation sensors on the backand arm (gyroscope and magnetic sensors) distinguish wellthe hard-to-classify activities in the Opportunity dataset.
Subject 1
Subject 2
Subject 3
Subject 4

Sensors (Number of sensors)
BACK M, RUA G, RLA G, RLA M,
LUA A, LLA A (6)BACK G, BACK M, RUA G, RUA M,
RLA G, LUA A (6)BACK G, RUA M, RLA G, RLA M (4)BACK M, RUA G, RLA A, RLA M (4)F1-Null
0.82

F1-NoNull
0.83

0.71

0.73

0.87
0.75

0.85
0.74

Table 4: The combination of sensors giving the best
performance in the classifier fusion framework.

Conclusion and Future Work
We have introduced the unified multimodal system foractivity spotting by processing different sensors in a
homogeneous way based on the template matching
WarpingLCSS. Two fusion frameworks are investigated:

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

the classifier fusion and the signal fusion. The results ofthe experiments show the flexibility and efficiency of oursystem in handling multimodal sensors. The more sensors
are added into the system, the equal or better
performance is achieved in average. Moreover, the system
is flexible in adding or removing activity classes. The
classifier fusion framework provides the ease to add or
remove sensors. Meanwhile, the signal fusion framework
yields the better performance in classifying null classes
due to a global view of data. In future, we plan to apply
sensor selection algorithms in the classifier fusion
framework to achieve the best performance. We also plan
to investigate other modalities in our system.

Acknowledgments
This work has been supported by the Swiss Hasler
Foundation project Smart-DAYS.

References[1] Bao, L., and Intille, S. S. Activity recognition fromuser-annotated acceleration data. In Pervasive
Computing: Proc. of the 2nd Int’l Conference (2004).
[2] Chen, C., and Shen, H. Improving online gesturerecognition with WarpingLCSS by multi-sensor
fusion. In Computer Engineering and Networking,Lecture Notes in Electrical Engineering. 2014.
[3] Cormen, T. H., Stein, C., Rivest, R. L., andLeiserson, C. E. Introduction to Algorithms. 2001.
[4] Huynh, D. T. G. Human Activity Recognition with
Wearable Sensors. PhD thesis, TU Darmstadt, 2008.
[5] Kunze, K., Bahle, G., Lukowicz, P., and Partridge, K.
Can magnetic field sensors replace gyroscopes inwearable sensing applications? In International
Symposium on Wearable Computers (ISWC) (2010).
[6] Lee, Y.-S., and Cho, S.-B. Recognizing multi-modal
sensor signals using evolutionary learning of dynamic

816

bayesian networks. Pattern Analysis and Applications(2012).
[7] Nguyen-Dinh, L.-V., Blanke, U., and Tröster, G.
Towards scalable activity recognition: Adapting
zero-effort crowdsourced acoustic models. In
Proceedings of the 12th International Conference on
Mobile and Ubiquitous Multimedia, MUM (2013).
[8] Nguyen-Dinh, L.-V., Roggen, D., Calatroni, A., andTröster, G. Improving online gesture recognition with
template matching methods in accelerometer data.
In the 12th International Conference on Intelligent
Systems Design and Applications, ISDA (2012).
[9] Ogris, G., Stiefmeier, T., Lukowicz, P., and Troster,
G. Using a complex multi-modal on-body sensor
system for activity spotting. In IEEE International
Symposium on Wearable Computers, ISWC (2008).
[10] Roggen, D., and et. al. Collecting complex activity
data sets in highly rich networked sensor
environments. In 7th Int. Conf. on Networked
Sensing Systems (2010).
[11] Sakoe, H. Dynamic programming algorithm
optimization for spoken word recognition. IEEE
Transactions on Acoustics, Speech, and Signal
Processing 26 (1978), 43–49.
[12] Schlömer, T., Poppinga, B., Henze, N., and Boll, S.
Gesture recognition with a Wii controller. In
Proceedings of the 2nd international conference on
Tangible and embedded interaction (2008).
[13] Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P.,
and Tröster, G. Wearable activity tracking in car
manufacturing. IEEE Pervasive Computing Magazine(2008).
[14] Wu, J., Pan, G., Zhang, D., Qi, G., and Li, S.
Gesture recognition with a 3-D accelerometer. In
Ubiquitous Intelligence and Computing. 2009.